I've turned on the PC for 2-3 hours now and the temps are:
CPU Idle: 52C
Case Idle: 45C
This is with the fans taken out from the exhaust. So the only fans present are the ones in the PSU and the CPU fan. Enermax Noisetaker fan speed knob set to "low".
I'm located in sunny California so weather could be much hotter in the next few months! Could I get away with just the WhisperRock heatsink?
If I were in your shoes I'd be worried about those temps.
The delta between idle temps and what you'll get after hours of gaming can be more than you might guess with aircooling, and much more so with low-noise air cooling.
Though many people consider CPU load temps in the 60's or 70's (celcius) to be acceptable, the hotter you run your equipment the more chance of it breaking down (usually when you need it most!) or the more it's lifespan will be reduced.
You might try running with the side panel of your case removed. If the temps drop dramatically then case airflow is your main problem. If they don't, a better CPU heatsink & fan combo would be your best bet.
With a decent heatsink/fan combo you should be able to drop your case/psu fans to 5 volt and still be assured of safe (but maybe not great) temps.
IMHO silence isn't worth it if you're too worried about frying your components.
Best of luck!