Supermicro X10DAi with big quiet fan coolers, new processors

Writing papers and studying for comprehensive exams has been eating most of my time recently, but I've done a few homelab-y things.

I've been trying to sell the X10DAi motherboard for a few months now to no avail. Super low demand for them for some reason. Anyways, it's a great motherboard, so I decided to use it to test some new (used) processors and CPU coolers.

I got a great deal on 2x E5-2690 V4 ES (might actually be QS). These are 14c/28t, base frequency 2.4GHz. They're in perfect condition, which is rare for ES processors. I haven't benchmarked them yet, but the all core turbo is probably about 3.0GHz, which is the same as the 10c/10t E5-4627 v3 QS. They can also use the 2400MHz memory I have at full speed. All in all, I should be getting a speed boost of somewhere between 10% (memory only) and 50% (memory and extra cores). This will make the headnode significantly faster than the compute nodes, which generally isn't useful, but if I allocate more tasks to it, then it should balance ok. There's also the possibility that the extra cores will end up being useless due to the memory bottleneck. With the E5-4627 v3's and the openfoam benchmark, going from 16 to 20 cores only improved performance by about 5%. The extra memory bandwidth will help this some, but I expect that the performance difference between 24 and 28 cores will be ~0.

Anyways, on to the coolers. The headnode workstation isn't really loud per se, but it isn't quiet either. This hasn't really mattered until now because the noise generated by the compute nodes + infiniband switch is on par with an R/C turbojet engine. Since the upgraded fans for the soundproof server cabinet are also loud, I haven't actually closed the doors on it yet. However, I'm planning to fix that soon, so I decided to try a quieter CPU cooling solution for the headnode. I looked into water coolers, but I only had space for two 140x140mm radiators, which means that they won't cool better than a good fan cooler. That, coupled with a price comparison, led me to fan coolers. It's a large case, and there's plenty of head space, but since it's a dual socket motherboard, I can't fit two gigantic fan coolers. I also wanted to be able to access the RAM with the coolers installed, which limited me to 120 or 140mm single fan coolers. I purchased two Cooler Master Hyper 212 Evo's (brand new from eBay was cheaper than amazon), which, at <$30 each, have an incredible $/performance ratio. I installed one CPU and one CPU cooler in the X10DAi to test both out. When I turned it on, I got the dreaded boot error beeps, 5 to be precise, which either means it can't detect a keyboard or a graphics card. The X10DAi does not have onboard graphics, so it requires a graphics card, which I had installed. I figured out I had a bad DVI cable, but that didn't fix the problem. After about an hour of head scratching, I realized that I hadn't cleared the CMOS. This is necessary when changing CPU's. I removed power, removed the CMOS battery, shorted the CMOS clear pins, put the CMOS battery back in, powered it on, and it booted right up. I then ran memtest86 on it. Then installed the second CPU and cooler, then ran memtest86 again. blah blah blah...

Anyways, the Cooler Master Hyper 212 Evo fan coolers work great on this dual socket motherboard. Note: you have to modify the instructions. The holes for the cooler are not drilled all the way through this motherboard, so you can't install the back plate. Instead, you have to screw the shorter standoffs into the threaded holes (same thread! that was lucky), then the CPU cooler bracket into those. Make sure not to over-tighten the CPU bracket screws because they are pulling up on the CPU plate, which is only attached to the surface of the motherboard PCB. If you tighten them too much, it could flex the plate enough to break it off the PCB.

Short standoffs installed on socket 2. You can see a post from socket 1 in upper right.

Both installed and running!

In case you're interested: the coolers could have been installed rotated 90 degrees. They're tall enough to clear the RAM, and I'm pretty sure tall enough to allow RAM access even if rotated 90 degrees. Since it's roughly the same size, I think they'll work on the ASUS Z10PE-D8 as well, which is where they'll ultimately be installed.



Coming soon:

• Quieter cabinet air extraction fans
• FDR Infiniband (oooo, shiny)