Installing a top down cooler with the heatpipes coming out below the CPU socket or the GPU, the performance might suffer a bit. I'd have to look up the reviews, but the difference was only ~3 deg centigrade.
In a CPU tower cooler the heatpipes are vertical to the floor regardless of the mainboard orientation, same as in a regular tower. There should be no performance penalty at all.
Older heatpipe coolers, older than 5 years, where a bit more sensitive.
Is there a quiet enough (at idle) high performance AMD reference card that works well in the FT02?
What's a high performance card to you and what idle state do you mean?
The Radeon 7950 offers the same performance as a Geforce 580 GTX and there are several Radeon 7950 cards with aftermarket coolers. The quietest coolers are Asus Direct CU II, Gigabyte Windforce and MSI Twin Frozr, probably in that order.
If you use an Intel Z68 or H67 (or the upcoming Z77 or H77) chipset you can use Lucid Virtu. You connect your monitor to the mainboard and the graphic card is used for games only, everything elss is powered through the CPU's graphic. With Virtu an AMD 7950 or 7970 card can go into low idle mode consuming about 3W and power it's fan of completely.
I don't know which of these cards support low idle mode through their bios.