if you are with somebody for a short period of time and they still have numbers around, they are keeping them just in case you two don't work out. if you two have been together for a while there is a SLIGHT chance they forgot(like i forgot i had female pictures in my wallet behind my nephew's pictures) and i explained to my fiancee that it was an honest mistake. as soon as she saw them i took them out ripped it up and threw them away and told her what the deal was. or they may just be, again, holding on to that number for some undercover type shit.

either way allow for explanation, if it seems valid, then do what you do.