So Xyvotha, what solution did you end up with? Are you still looking 8 months later?
With 12 HDDs, vibration, resonance and heat would be your biggest concerns.
I would have just got an Antec Twelve Hundred, used dampening maperials for both side panels & top and bottom case. I would have cutted the fans cable (because they're included with some kind of switch at the back) and use Nexus 120mm reference fans all the way. Top fan would wimply be unpluggued.
Using Green (5400rpm, IntelliPower) drives would be a MUST for less heat as possible, because those fans won't move the amount of air you would need otherwise. I would strongly advise you to check Hitachi 5400rpm drives, they heat less than my WD Green and are less noisy.
To mount the HDDs, I would use these silicone brackets
http://www.frozencpu.com/products/3040/ ... 1182#blank . Since they require 5.25 bbays, you would have to remove the Twelve Hundred supplied HDD cages and use all 12 5.25 bays. So no internal optical drive option. (anyways, who needs this, I user external on rare occasion I need to use CD/DVD or USB keys now)
I would use integrated graphics to prevent more heat generation with a video card. I would suggest Ivy Bridge platform for less heat generation. (or Haswell if you can wait) I would also suggest the CPU tower cooler for heat dissipation ASAP. If you only use it as a file server, get a low power Core i-something processor, they're 35W TDP (for ivy bridge). Also use RAM with heatspreaders, and use only 2 sticks if it's enough for your needs. Don't go on socket LGA2011, chipset heats too much, kepp it simple on LGA1155 with a Z77, H77, Q77.
Anyways, you don't need a crazy motherboard, A Micro-ATX or even a mini-ITX would fit the Twelve Hundred and your needs. The smaller the board, the less chips, the less heat If you go the mini-ITX route, get at least 2 onboard Gigabit Ethernet port, so in case one toast, the second is backup.
All you need is a good x8 PCI-Express slot for your RAID/storage controller card. In this regards, I do not have any suggestion, as I have now on SSD and use AHCI on motherboard chipset. I never ran that much hard drives, my file server is an 2-bays NAS from D-Link that has been running since 2007. I've read good reviews about the LSi MegaRAID SAS9280-16i4e.
Nexus PCI-300 venting PCI-slot-bracket would be of great help, they prevent dust from coming in and they let the air circulate.
Finally the PSU, you would need a very poweful PSU with a 120mm fan. A modular PSU like the Seasonic X-850W. With the setup I've listed above, the system should use about ~300W at full load, so a 850W PSU would handle the job quite easily without event turning it fan speed up. All you need to look for, is the number of SATA power connector included with the PSU.