Radeon HD5750 [...] I think it is the source of the really loud noise and it doesn't perform very well anyway. Then there is the case which is kind of unfavorable: as you can also see from the picture I linked it has an opening on the top that has no air filter so it's getting quite dusty in there all the time making the fans louder etc.
So, my plan was to get a new case and a new graphics card, thing is I'm not very much into computer building so I can't really tell what's good and what not.
I would spend money on silent upgrades for your core system. It would be foolish to get rid of any of your components, as they are not slow (just loud).GPU & GPU Cooling:
Your monitor max resolution (1680x1050) is a good match for your video card speed. In video games, AMD and Nvidia are a close match at the low and medium price points. A low-noise cooler for your 5750 should only cost $20. ARCTIC COOLING Accelero L2 ProCPU Heatsink and Fan:
The next thing to do is to replace the stock cooler on your CPU. Xigmatek makes a lot of low-noise coolers that are not expensive. In the low-priced market, the Xigmatek Gaia SD1283 is a good performer that can be bought at many stores for about $30.Xigmatek Gaia SD1283Power Supply:
If your power supply is going to be out of warranty, then you should not be afraid to open it up and replace the fan. Their are many fans that can be used, but many avoid normal sleeve bearing fans in this area that can get hot under load. You will need to buy a 80, 92 or 120mm fan that is low-noise. A quality non-sleeve 120 mm fan will cost $10-25.Computer Power Supply Fan ReplacementCase Fans:
You should replace your intake and output fans on the case with low-noise fans. I can not find exact information about your Thermaltake Armor case; some places claim 92 mm fans are used, while other sites claim 120mm fans are used. The people here at SilentPCReview like the Nexus 120mm low-speed units. They are $8 to 12 each, depending on where you buy them.
Unused top holes in the case should be covered to avoid dust and noise problems. Lian Li and Puget Systems both make 120mm & 200mm fan hole covers that sell for $8-12 each. You can also use strong packing tape to cover the holes. The tape only works if you completely clean the area so that the tape will stick.Lian Li PC-343120mm Fan Mount Cover (Black)200mm Fan Mount Cover (Black)Hard Drives:
Finally, your hard drives may be a source of noise if they are 7200 RPM or faster. You can replace them with a Green/Eco/LP drive that spin at a slower RPM. You could also get some anti-vibration mounts for your existing drives if they are already slower RPM units.
Some people are happy with the XIGMATEK CCA-EMFCB-U01
and it costs about $25.Conclusion:
It "sounds" like you can silence your system for $125 plus shipping to your home. You also should add labor costs, but I have no idea how much you are paying. Installing all of these parts will take 2 or 3 hours.