Venturi Flow Meter and Phi Testing

I hinted at another big project I was undertaking in my first homelab post. I purchased 140 used Xeon Phi cards recently, from the same source I got the original 12 from. The plan is to test them all, keep some of the working ones, and sell the rest.

Of course, it's not that easy. These are all passively cooled server cards, and I sold the server I used to test the original ones with. Not that that server was ideal anyways, because it only technically supported one variant of the Phi family. Thus, I'm assembling a new desktop machine to test them with. But because it's a desktop, it doesn't have a high enough cooling air flow rate for the cards. There are some examples on the intel forums of people building their own coolers out of foam board and strapping a fan to them. Some people 3D printed fan adapters, and you can even buy one for a blower fan on eBay from Australia for ~$20. I looked at what was available and thought, "I could do better." I wanted to engineer the fuck out of this.

The Xeon Phi datasheet contains a lot of useful information, including exact flow rate requirements for the primary and secondary side of the card vs. inlet temperature and pressure drop across the card. I extracted the worst case condition, i.e. highest inlet temperature, maximum flow rate. I figured out the total flow rate required, the pressure drop across the card, and started researching fans powerful enough to meet that requirement.

But first, what am I testing exactly? The experiment has two main goals:

1. Verify that the fan adapter designs I make split the flow in the correct proportion between primary and secondary sides to meet flow rate requirements
2. Determine how much more pressure the fan needs to develop over the published Phi pressure drop spec due to the fan adapters. This will be used to select fans for one and two card adapters based on their published fan curves. 

Examining the goals, I could tell I'd need the following:

• Two Xeon Phi cards
• Fan(s)
• A way to power and control the fan(s)
• Fan adapter for one Phi
• Fan adapter for two Phis
• A way to measure relative and absolute pressure
• A way to measure flow rate

I realized early on that a 3D printer could help supply most of those things. In fact, this is why I purchased the Wanhao i3 about a month ago now. 

Before I could design a fan adapter, I needed to select a fan. In general, axial fans are more efficient than blowers. I chose an 80x80mm axial fan size for a few reasons. 1. It fits inside the width of the card, so it's unlikely to interfere with anything on the motherboard. 2. Its flow can be funneled into one card in a fairly short space (fan+ adapter can be shorter than a blower + adapter), 3. It's the perfect height for cooling two cards at once, 4. There are many 12V 80mm square fans to choose from, 5. It's more efficient than a blower fan (about 30-50% less power). I probably went through about 50 fan datasheets looking at fan pressure vs. flow rate curves to find one that could deliver ~20% more pressure at the desired flow rate. I looked for ones that would be good for 1 or for 2 cards. Cooling two cards in parallel doubles the flow rate requirement, though the pressure drop stays the same, so fans good for two cards would be able to handle one card, as well. For testing purposes, I decided to go with a far overpowered fan and use a PWM controller to throttle it. In fact, I already had a fan controller in my homelab cabinet, so I could just use that. I chose a fan I could obtain from eBay for cheap, which happened to be a Nidec v80e12bs2a5-57 from a supermicro fan-0148L4 kit.

It turns out most 80x80mm axial fans have similar exit geometry, so designing one that works for one fan will likely work for most fans. I decided to start with a single Phi adapter because I only have one Phi.

Fan adapter attached to fan

The fan adapter is the same one from this post. I designed the primary and secondary flow areas to roughly match the proportion of flow for each. I'll probably have to adjust it some for the final version. The plan is to superglue in the M3 nuts for the Phi screws. Threads in 3D prints are difficult. I went with nuts because I didn't want to deal with heated press-in inserts, and thread-in inserts are huge. You can design nuts to be pressed in, but there really isn't room do a good press fit inside of the adapter.

I designed a mock Phi with a similar airflow path as the real one, but shortened, and with a restriction plate so I can tune the flow loss. 

Took 16 hours because the printer isn't doing so well right now...can see some under extrusion near top right.

Hard to see, but poor bridging inside.

Mounted to fan-Phi adapter. Restriction plate goes in the slot near the top

The plan is to do the testing on the single Phi, then do testing on the mock Phi and adjust it until its pressure drop matches the pressure drop of the real one. Then I can test a two Phi adapter by measuring the flow output of the real phi in both positions (top and bottom). Anyways, that'll become more clear when I actually do it and write it up. 

Now I have the fan, fan controller, fan power (PC molex to fan connector), fan-to-Phi adapter, and Phi. I still need a way to measure pressure and flow rate.

I started looking at pressure transducers and gauges...things I'd used in the past back in lab in my undergrad and masters. While they would work, they're more accurate than I need and quite expensive for a project this simple. So I stopped, took a step back, and thought about what the simplest way to measure pressure is. A water manometer! These are super simple to make, and cheap, too. All you need is a piece of flexible clear tubing, some barbed tube fittings, a ruler, and water. I bought some 6mm aquarium tubing and straight 4mm connectors for it off eBay for ~$5.

I also need to measure flow rate. I started looking up flow meters that could measure the flow rate I needed to, and they're all fairly expensive. There are numerous methods for measuring flow rate, but one of the simplest is an orifice plate. Just stick an orifice in your pipe with a known Cd, measure the pressure drop across it, do some math, and then you know the flow rate. The problem is knowing Cd accurately. A possibly more accurate way is with a venturi flow meter. A smooth venturi will have a fairly constant Cd around 0.98. They're also pretty easy to design and 3D print. I figured out that a 50 mm to 20 mm diameter venturi would give me ~150 mm H2O, which is large enough to be easy to measure with a ruler. Bonus: I can use the same water manometer for this! I'll show the math for this in the next post along with the results.

I also needed an exit adapter to funnel the Phi's exit flow into a tube, which would then go to the venturi flow meter. I designed and printed that, too, but in two parts because the Phi's used in desktops have a PCI bracket that impedes the flow some and I wanted to model that.

I also needed a tube to go between the exit adapter and the venturi meter, and then after the venturi meter. You need a pipe after the meter to eliminate exit effects. If I was in the US, I'd just go to Home Depot or Lowes and get some PVC. In fact, you can make a reasonably good venturi meter out of PVC reducers and pipe. But this is the UK, and they don't have anything equivalent. B&Q is close, but not the same. I'd have to order the pipe to be shipped here, which seemed dumb. Instead, I bought two 50 mm diameter mailer tubes off eBay for ~$2.

Most of the parts. Venturi on right

The printer messed up right before the venturi print. I managed to get it printing again, but the quality is degraded. I ended up sanding the venturi pretty heavily to remove ridges, but I think I did a good enough job.

I then drilled and put in the static pressure ports (cut up aquarium tube connectors).  I'll also put one in the tube right before the entrance to the venturi. These will be glued in and sanded flush on the inside. I chose locations where the wall was fairly flat relative to the central axis to minimize flow components along the axis of the port, which would throw of the static pressure reading because it would be reading part of the dynamic pressure. The unused ports will be plugged during testing.

Measuring from fan to ambient (with nothing after the Phi) will give me the pressure developed by the fan, which will be the pressure drop across the Phi and fan adapter. However, I won't know what flow rate that is. Measuring from fan to exit adapter should also be the pressure drop across the Phi and fan adapter. Measuring from before venturi to venturi throat will be the dP required for the flow rate calculations. Those measurements will give me everything I need to know to select fans.

Measuring primary and secondary flow rates in the Phi will be tricky. I can measure the primary flow rate fairly easily, but the secondary flow is open to ambient in multiple places. If I can't figure out a good way to measure full flow rate (maybe taping off all the exit paths?), then there's no way for me to calculate the secondary flow rate (full-primary = secondary). I'll just have to estimate at that point.

One of the main assumptions I'm making here is that the heating of the flow doesn't affect pressure drop. This is not strictly true, but for incompressible flows, it's not a terrible assumption. Another assumption is that the tubes and venturi flow meter don't influence the flow in the Phi. If the Phi was a closed tube, then this would be a good assumption. However, the Phi has holes in it. Since the pipes and venturi add pressure drop to the system, there will be a higher pressure relative to ambient in the Phi with them than without them, which will cause more air to escape from the holes. I forgot to mention this, but high quality (good layer adhesion) 3D prints are pretty air and water tight. This wasn't at all obvious to me when I started out with 3D printing, so I tested a small 3D printed cylinder by sealing one end and blowing into the other. The pressures in this test will be fairly low, but I bet you could 3D print parts that hold a fair amount of pressure. 

That's all I have for now. Next post on this will include the testing results. 

Oh, I never fully answered why I'm doing all this when I could just fudge something and use a bigger fan. 1. This is fun for an enginerd. 2. I think I can sell these Phi adapters with the Phi cards and hopefully earn enough to cover the cost of the 3D printer (minus time). I'll probably list them on eBay for a few years, too.