I am just curious whether sending a fan that requires 6.5V to run a lower voltage (say ~4.7V, the fan would essentially be turned off) if it will damage either the fan or the controller, or worse the PSU.

Thanks!

Also would be awesome if someone could recommend a 'cheap' 0v-12v controller without annoying LEDs

It will not damage a PSU, because 12V will still be sent to the fan controller. I doubt it would damage the fan, and have never heard of that happening, but I am not fan engineer. It will not damage the controller either, unless it is designed very poorly, or is overloaded (trying to control a fan(s) with wattage over the rated handling power of the controller, which is pretty hard to do if using the type of fans used in quiet computing).

Cheapest i know is Zalman fanmate, but it doesn't go down to 0v, think lowest is 5v

Had a look around, don't know if you can get them where you are but Sunbeamtech Rheobus
goes down to 0v