I couldn't find out what exactly lies under the stock coolers of the Radeon X800XL's. Are there memory chips or something I have to be concerned with after I stick a VGA block on it? I have very little airflow in my case.

And what about the Radeon X850XT PE? Do I need any extra cooling for that one?

Yes, there are little RAM chips, usually on the front and back. I would suggest sticking some RAM sinks on there (like the ones that come with a Zalman VF700 cooler)... won't cost a lot and it might save you from some freeze-ups and/or give you more OC potential.

I've been running an X800XL without ramsinks for ... ~six months now in an essentially fanless case (no airflow around the graphics card that's for sure, 'cos I watercool it). I've not experienced any problems at all with lock-ups and so on. I don't overclock though, so YMMV if you do.