I'd say a mistress is a lady you have in secret while you have other things going on.

I think secret is the key word.