It really depends on what gen s4/a4 you're looking at.
If you're looking at getting a B5 S4, I would try to find one with relatively low miles and one that hasn't been abused...
Have the car checked out before you consider buying.
Transmissions and turbos tend to be the bigger problems for the S4 as are electronics, which is the case with most VW's.

As far as the A4 is concerned, I would get a B6 or B7 model.
Oil sludge and the electronics seem to be an issue though so have the car checked out.
I haven't heard of too many issues concerning the B6 S4 as it is NA.