is there a way to make them fit together in a 1U rack case? Perhaps a mainboard like the DQ77KB could be so thin that the PCIe card could fit directly above it (with the HTS1155LP mounted on the CPU between the PCIe card and the mainboard)? Or is the HTS1155LP thin enough that the PCIe card could fit above it?

They say the heatsink is 26mm and the PCIe card can't be more than 15mm thick and there should be about 41mm of space in a 1U case...

No.
If you ever opened a 1U case you'd know. A rack case is 425mm wide, why would you want to put a card on top of the CPU if there's room to the right. 1U Rack cases usually have right hand riser for a single card, so the card covers all the slots of an ATX mainboard.
Mainboard standoffs are 5mm, add 2mm for the PCB, 7mm for CPU socket CPU and heatspreader, that's 15mm. Inside the case there's 41mm room, guess how tall passive 1U heatsinks are: 26mm. There's just no room in a 1U case above the CPU area.
The HST1155LP is a special heatsink, but it still adds at least 8mm height. That leaves you with 18mm at best. even if you can cram the card in, there's no room for the back connectors, because the mainbord's connector will get in the way of the tall sockets on the [email protected] card.
It might be doable in a 1.3U case. Why would you want to use a rack case in the first place (unless it goes in a rack) ?

Exactly. *IF* there's room to the right... So is there? Would the HTS1155LP heatsink&fan and the card fit together on the side?

If I could find a 1.XU case, I would gladly explore that possibility and would probably use Big Shuriken 2 Rev. B without a fan or Kozuti and vent it out of the top of the rack case panel. There's some room for that in my rack, but 2U would be too big

Not with that heatsink, the more I look at pictures the more I think the placement is meant to discourage putting it in a 1U rack case like you want to.
Why does it have to be that board and heatsink?
Even better why don't you rotate the haetsink 90degrees so it's in front of the board insteadof the right? That would leave the right side clear.

Simple answer: If you use this cpu cooler then you won't get any PCI card working because of not enough room.



Pick a different cooler and things will work out. Take a look at the pictures from
http://www.casetronic.com/corporates/37-c146.html