You don't wanna fuck up the kids life too then just because you want to fuck other girls...

Sounds like you rushed into things and now you're stuck. You just need to suck it up and deal with it.