I'm going to tell you what you need to hear, because I've been in your shoes, and you're not doing anyone any favors with what you are doing, including yourself.
Affairs suck ass. They are a great high in the beginning the first time you slip your cock in her, then they fall into exactly the flip flopping, heart breaking, emotional mess that you're in right now.
My advise to you is to read the following links. If you're in an affair, it's because your marriage has issues and you're dealing with it in a piss poor way. I know, I've been there and done that. Fuck the affair.
Figure out if you want to stay married, and if so, what the terms are. Maybe you're a horny guy and need some room to fuck other women even though you're married. Maybe you want an open relationship in other ways. Maybe swinging will solve your needs and make your wife open up sexually. Maybe your wife and you can't agree to terms and so you divorce. In any case, it's pointless being in more than one serious relationship at a time. It's far different than being in several physical only relationships at a time.
Chances are you're unhappy with yourself because you know it's BS to two time any woman and fuck around on her without her knowing it all up front. But your affair partner makes you feel like you're a great guy, even though you're cheating on your wife. But then she feels bad for the same reasons, and needs you to build her back up. It's a fucked up deal, every time. They all are the same. I know, because I read about a TON of them when I had mine. Finally the light bulb went off. The only one who can really make you happy about yourself without all the drama is you. All you gotta do is decide what it is you want, and be that, and be willing to tell chicks up front, this is me. I think it is a piss poor idea to end your marriage specifically to be with your affair partner. How long before you wonder when she isn't home if she's fucking some guy from work while you watch her kids? Until you fix your issues, and she fixes hers, I wouldn't want anything to do with a relationship with her. Red flags all over the place.
Read the following links on infidelity. Might shed some light on your situation.
http://womensinfidelity.com/
http://forum.marriagebuilders.com/ubbt/ ... 315&page=1
Good luck. I hope you see the light, fix yourself and get to a better place. It's not easy, but it's the only way. Let me know if I can help any more. I have been where you are at, and it sucks.