Short answer: Don't do it.
Longer answer:
The manufacturer doesn't list this PSU's efficiency and it doesn't appear to be 80 Plus rated. So, chances are, it's 80% efficient at best and they just didn't bother spending the money and time to get it 80 Plus rated. At worst, it's an old school 70% efficient supply.
50W load under what scenario? Idle? video playback? h.264 encoding? You don't mention the other system components, so it's hard to know what your true idle/load scenarios will be. If you take the
components in Hardware.info's
mobo roundup, the best case idle, 3DMark Vantage, and Cinebench measurements are 24W (AC), 61W, and 84W. Replace the quad with your Celeron, and it would go to roughly 24W, 55W, and 62W. If your PSU met 80% at low loads (doubtful), the internal heat is ~5W at idle to ~12.5W running Cinebench.
If your PSU is closer to 70%, this could go up to ~ 26W, 61W, and 68W. That's
7.5W to 20W of heat dissipation inside the PSU. If your mobo uses more power than the most efficient one I used here, then this could make your idle and load go up another 20-30W with commensurate heat load.
So, no, I wouldn't kill the fan. If you want quiet and SFX form factor, consider getting an 80+ Gold rated PSU.