But wouldn't a spacer make the wheel not touching the hub anyway and the space touching flat?

I get spacers for free so that's not a concern and have bored wheels out before it's just a total PITA.