My friend has an 07 Si Sedan. I have a Spec V. The Spec V is more fun to drive on a curvy back road. The Spec V has low end torque. The Si has top end. The QR25 does not produce as much horsepower with the mods such as I/H/E as compared to the Si with I/H/E. The K20 will show more gains with those modifications. If you want the Spec V to be 'fast' you have to boost it no questions. If you put I/H/E and cams and a tune (SAFC2) you should be looking at like 14.5 ET in the quarter mile just to give you an idea. If you get a 2006 Spec V they fixed the butterfly screw issue by using better screws that don't come off. I would still buy a header asap just in case the pre-cat decides to take a dump. I would also say get a Spec V because they are not as common as a Si.
Either way they are both nice cars.