Nah, you can't let 'em do that. If it's on a two lane road you speed up so they can't get by. And then when they get behind you slow down again. Unless of course it's on a road where passing is legal.
If someone wants to get ugly, they can call the police and you can be charged with road rage, though. That can be a drawback of what you're saying here. Later, QD.