id leave it ,you dont have a problem leaving your husband for what ever reason, and so you need someone who is as ready as you are and who isnt in two minds, your 100% he is 50% committed, this thing youv got could drag on for years in secret and thats not a way to have a real relashionship.
My advice would be first ditch your husband if your not happy ,get a divorce then ditch this other guy and put it all behind you ,then go find other single men who are not tied down or with any obligations so they are committed 100% to you.