Hi John,
Basically speaking, your intercooler lowers your charge temperature. It's particularly critical in forced induction engines (turbo/supercharged) as inlet air going through these devices raises the temperature (and therefore lowers the density) of the air going into your engine. This is also why your car might seem to pull harder on a frosty February morning, than a spanking hot mid-August afternoon.
Taking out your intercooler will raise the charge temperature and you'll loose some power. How much is hard to say (what ya got to start with?!) Possibly 10%-15% but that's a guess, there are so many factors to consider (and someone on here or another forum has probably done some research into this and has some data) but detonation may also be possible (although your knock sensor should back things off if it gets too dicey.)
Best to leave it on. This is why most people increase the size & maximise air flow across the intercooler face - for cooler, denser air.
And for the record, I agree with Andy too.