tl;dr: partway there. Thermal runaway solved, hunting down coil whine.
...I think this is the most time I spent with building and troubleshooting any system, and I'm not new to it by any stretch.
A bit of an update:
0. Core temp vs CPU (aka CPUTIN) temp: after much googling, it seems they are wildly different. The first is a maximum value calculated by measuring input current of a core, the second is a measured value of the cpu socket. The maximum value of 72°C set by Intel (Tcasemax) refers to the CPUTIN temp. My initial problem was that speedfan does not see the CPUTIN, that it seems running CPUID HW Monitor will get it stuck until reboot, and that Core temperatures seem to be higher (5-15°C, a bit hard to measure with the HW Monitor glitch) than CPUTIN. At least the BIOS Silent mode won't raise fans to the max till the core temps reach ~85°C, and thermal throttling does not set in even even at 80-85°C core temps.
1. Found out what caused my thermal runaways of 90+°C: The AsRock board seems to employ a bit of a turbo hack. After about 30-60-120 seconds (seems to be varying) of prime95, frequency will jump from the official multicore turbo frequency of 4200Mhz to the official single-core only turbo of 4400Mhz. (Officially, AFAIK this could only be employed if one core is stressed, and the rest are idling.) This also pushes the HD Monitor reported power consumption from the ~80W range to 90-100W, also increasing the temperatures by ~5-20°C. Using this thread
I set my long/short power limits to 85W, and expermientally found the current limit of 104 to work well.
2. Got the Noctua NH-U9B SE2 (single tower with dual 92mm fans) heatsink, put it to tests just on my table, without the case a couple days ago, no problems. I'm impressed with the package and what went into it. I might even ditch Scythe as my main supplier for CPU heatsinks. What I did not check was that the fans are 3-pin ones. Whoopsie.
3. Previously I said the NH-U12S might fit. I'm revising this: if the fan clips are anthing like that of the NH-U9B SE2, I doubt it would. It might fit without the clips, but as stated before, I'll be transporting this more frequently than a desktop case, so that is not an option.
4. It seems the "Silent Mode" fan setting in the bios gets the noise level quite a bit lower (~1100-1200rpm or ~70-75%) on prime95, provided you don't run into the turbo-hack. If you do run into the 4400Mhz, it brings the fans up to max, even outside the case.
5. Maxing out the Noctua fans manually on prime95 will not bring the temperatures down. I still get 68-70°C amongst the cores. Note that these are _core_ temperatures, as stated in #1.
6. I also got the NF-S12B redux, and tried it (without the case, just putting it approximately into place). It seems that on air cooling performance is really independent on whether I have the back 92mm on the heatsink, the S12B, or both. Inside the case is a bit more complicated, see below. What is interesting is that the fan seems to hum at certain PWM percentages: it has no sound at ~40%, and over 60%, but at 25-30%, 55-60% and ~75%, there is definitely a low pitched, very apparent hum. Unfortunately the Silen BIOS setting idles the fan at 30%.
6.b: Aaaand my old Noctua 120mm started sputtering after repeated loads, so I went and grabbed an older Scythe SY1225SL12LM-P that came with my Mugen 2.revB. It seems to work so far.
7. edh: it appears the m.2 slot is accessible even with an unmodified backplate. Although unfortunately AFAIK the only PCIe m.2 cards that offer a huge speed boost are too big for this motherboard (it supports only 2230 and 2242 sizes). There are some SSDs available in this size locally, but they are generally slower than a regular 2.5", and a bit more expensive.
8. I enacted some cable management to get rid of cables in the way of the airflow, attaching velcro cable ties to the top of the case, and putting in an unsleeved PSU cable extender that allowed me to have a much tighter turn radius.
9. There is a new nasty coil whine coming from the PC. It might be the result of using the PSU cable extender, so I'll try removing it, although I will have to disassemble the whole thing again. I'm too tired right now for that.
Not-so-Fun fact: I have disassembled the sides so many times that one of the screw holes is partially stripped now.
I assembled the machine again, and after many tries, went with a combo of one 92mm fan between the PSU and the heatsink, and the old Scythe as the exhaust fan. I've tried having the 92mm with ducting, and also replacing the front 92mm with my Nexus 120mm. As the 92mm Noctua has the same airflow and higher pressure, it is not that much surprising that the best combo seems to be having a 92mm at the front and the scythe at the back.
m.2 slot + some visible cable ties at the top
Through the window: not much cabling visible
...unless you are viewing from the same level
WIP shot, still with 3 fans and not final cable management (I tightened them and moved that USB3 double cable out of the way)