OpenVPN on OpenBSD

It seems like I have a habit of not doing things the easy way. A great example of this is, instead of using something like Rocks, I built all of the cluster's software up from CentOS. This post will be another example of forgoing the easy for the sake of learning.

VPN's are a great tool to prevent ISPs and other internet companies from collecting valuable data on you. Internet companies, e.g. Google (owners of Blogger, the irony of which is not lost on me), collect data on what you do online in order to target you with ads. They also bundle the data and sell it to other companies, such as political organizations specializing in political manipulation. Government agencies collect data, i.e. spy, on their own citizens' internet usage. Saying you have nothing to hide is a terrible argument. "Arguing that you don't care about the right to privacy because you have nothing to hide is no different than saying you don't care about free speech because you have nothing to say." Enough of that rant.

While the free VPN's are mostly crap if not malicious, a few of the more prominent paid for VPN's certainly appear to do what they advertise. Many of these VPN services provide a VPN client for download. This client abstracts all of the background details of how the VPN is configured, is established, working, etc, and often include useful tools, such as routines for forcing all internet traffic over the VPN and for preventing DNS leaks. As a learning experience, I signed up for a month of one of the more prominent ones: PIA, or Private Internet Access. PIA has an excellent VPN client available for many OS's, including Windows, Mac, most flavors of linux, routers, etc. However, they do not have one that supports OpenBSD. You can probably guess where this is going...

Enter OpenVPN. OpenVPN is a very popular open-source VPN server-client program. It also has a port for OpenBSD. PIA's servers have OpenVPN servers setup, and they provide sample .ovpn configuration files. Now begins the adventure of getting OpenVPN working on OpenBSD.

Installing OpenVPN was simple enough using the pkg_add command. As of OpenBSD 6.4, the openvpn version number is 2.4+, which is good. The first step though was configuring the firewall using pf and pf.conf. I needed the firewall to do a few things: 1. Block all traffic except: 2. Let communications, SSH in particular, on my private LAN pass, 3. allow inbound and outbound traffic on the VPN's virtual interface. The OpenVPN FAQ and pf man pages were excellent resources, as was this blog, which I unfortunately didn't find until a couple hours into this. The following is my pf.conf file (with some stuff obscured):

#....header lines

# external interface:
ext_if = "interface0"
opvn_if = "tun0"
lan_net = "192.168.X.0/24"

# don't filter on loopback interface
set skip on lo0

# scrub incoming packets
match in all scrub (no-df)

block drop all

#antispoofing:
# antispoof for $opvn_if inet
block in quick from urpf-failed

# only allow ssh connections from LAN. Can restrict to a few IP's later.
# as it is, not necessary due to previous and following rules.
# block return in quick on $ext_if proto tcp from ! $lan_net to $ext_if port ssh
# Alternatively:
# pass in on $ext_if inet proto tcp from $lan_net to $ext_if port ssh flags S/SA synproxy state

# pass all traffic to and from the LAN
# these rules will create state entries due to the default
# "keep state" option which will automatically be applied.
pass in on $ext_if from $lan_net
pass out on $ext_if to $lan_net

# At this point, can only communicate on LAN.

# pass tcp, udp, and icmp out on the interface to internet
pass out on $opvn_if proto { tcp udp icmp } all modulate state 

# must open port udp 1197:
pass out on $ext_if proto udp from self to any port 1197
pass in on $ext_if proto udp from any to self port 1197

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

This successfully blocks internet traffic if the tun0 interface is down, i.e. if the VPN is not connected.  The next step was to configure the openvpn client. This is done with a .ovpn file. There are lots of useful guides for this online: 1 2 3 4 5 . Link 5 includes information for setting up an openvpn server on openbsd, too. You can download a PIA OpenVPN configuration file from the generator (log into your account) or from here. I used the "strong" settings, which include better cryptographic algorithms. I modified the PIA files slightly. Here's an example:

client
dev tun0
proto udp
remote belgium.privateinternetaccess.com 1197
remote de-berlin.privateinternetaccess.com 1197
remote lu.privateinternetaccess.com 1197
remote france.privateinternetaccess.com 1197
remote-random
resolv-retry infinite
nobind 

# add downgrade privileges after initialization:
# requires sudo package
# user _openvpn
# group _openvpn
# chroot /var/openvpn 

#prevent passwords from being cached in memory:
auth-nocache 

persist-key
persist-tun
cipher aes-256-cbc
auth sha256
tls-client
remote-cert-tls server
auth-user-pass
compress
verb 1
reneg-sec 0
...crl verify....(removed from this post for brevity)
...ca certificate ...(removed from this post for brevity)
# disable-occ
# being polite:
explicit-exit-notify
writepid /etc/openvpn/curpid
#prevent dns leaks. must use pia dns servers
# doesn't work on linux: push "dhcp-option DNS 209.222.18.218"
script-security 2
up "/bin/sh /etc/openvpn/change-resolve-conf"
down "/bin/sh /etc/openvpn/undo-resolve-conf"
# add to force all client traffic through VPN
redirect-gateway def1

I won't explain all of the settings. I removed the crl verify and ca certificates because they're very long and not necessary for this post, but they would be where the ...'s are. I commented out the disable-occ so that I could see warnings. OpenVPN with PIA gives warnings about the type of cipher or auth being used being inconsistent between the client and server. It turns out that these are false warnings and can be ignored (I did output at verb 4 once to check). I write the pid out to a file. Originally, I was trying to use that in a alias that could kill the openvpn process, but I found it was easier to use "pkill openvpn". The last line "redirect-gateway def1" is very important: that does a few things to your routing table (and undoes them when openvpn exits), but its primary purpose is to force all of your internet traffic over the VPN. On the rare occasions where that isn't the desired behavior, then you can do some fancy routing instead.

Now all of the traffic should be going through the VPN, which means that, if the VPN provider doesn't keep logs (they say they don't), then there's no way for your ISP/anyone to know what you are doing over the VPN (unless you expose yourself by logging into a website or something like that). However, there is still a way for your ISP to know which websites you visit...this is called a DNS leak. There are a few ways to check for DNS leaks. One of the easiest is to go to dnsleaktest.com. If your ISP is listed as the server, then you have a DNS leak. Another way is to do: nslookup -type=NS somewebsite.com  or  dig www.somewebsite.com . The "Server" entry will be the DNS server...if its your local gateway or your ISP's, then you have a DNS leak. This is why I added the script-security 2 and the following two lines to the .ovpn configuration file. Before I added those, I expected to (and did) have a DNS leak. The contents of the scripts are as follows:

#change-resolve-conf
#!/bin/sh
mv /etc/resolv.conf /etc/resolv_conf.orig
mv /etc/resolv_ovpn_conf /etc/resolv.conf  

#undo-resolve-conf
#!/bin/sh
mv /etc/resolv.conf /etc/resolv_ovpn_conf
mv /etc/resolv_conf.orig /etc/resolv.conf

Basically, one (up script) changes the resolv.conf file from the one with my router's IP in it (the original) to one with the PIA DNS servers, and the down script (run when openvpn closes) changes it back. This works great and prevents the DNS leak.

The last thing to do was figure out how to start openvpn without having to type out the full command: !/usr/local/sbin/openvpn --daemon --config /etc/openvpn/openbsd.ovpn (note: this has to be done as root). There are a few options: 1. create a /etc/hostname.tun0 file containing "up" on the first line and the above command on the second line, 2. create a rc.d script (see helpful link 5 above), 3. create aliases. Option 1 and 2 will auto-start the VPN on boot, and of the two, option 1 is recommended. Option 3 allows you to start (and stop) the VPN at will, and is what I implemented first, though it wasn't straightforward. OpenBSD uses ksh by default and doesn't have something like a .bashrc or .chsrc. It has a .profile. However, digging into the documentation, it's recommended that you create a $HOME/.kshrc file for your aliases and put a line in the .profile that calls it. However, if I log in as a normal user, then all of my shells start out as user shells, which means the .kshrc of root is never read even when I do su root. I have to do su root, then ksh -l, in order to force the .kshrc (and thus aliases) to be read. Kind of a pain, but it works. I created an alias for pkill openvpn, which allows me to stop vpn. I will probably switch to Option 1 shortly. 

After all this, I decided to download the PIA client on my windows machine just to try it. It took ~10 minutes to setup vs. the ~10 hours this took, haha. But at least I have an understanding of how all of this works now.

Update: See this post for updates to the PF rules (more secure) and ovpn configuration.

Installing OpenBSD on an old laptop

Up until this point, I've only had experience with Windows, Ubuntu, and RHEL/CentOS. BSD's are another type of Unix OS common in the server/IT/security world, and I thought it'd be a fun exercise to learn how to install and use one. While FreeBSD has more packages/support, OpenBSD supposedly "just works", and it's core is continuously security audited, so I decided to try that one.

Downloading and Verifying

Useful links: 1

The first step is to download the installer. Since my old laptop is a 64bit intel processor, I needed the "amd64" version of installXX.fs, where XX is the version number (6.4 at the time of this writing). You also need to get the SHA256 file, which contains checksums, and the SHA256.sig signature file. If you copy the link address of the installer, paste it into the browser, then navigate to the parent folder you should see those two files are available for download. Since I'm going to create the install media on a USB flash drive from another (Ubuntu) computer, I don't have the OpenBSD sha256 or signify tools installed. On Ubuntu, the corresponding tools are sha256sum and signify-openbsd, though typically you would use gpg on most linux systems for verifying signatures. To do the checksum for the installXX.fs file, do: sha256sum installXX.fs . The output should match the corresponding checksum in the SHA256 file you downloaded. If not, then it did not download correctly. The next thing to check is the signature. This helps prevent downloading of potentially malicious software: if the signature doesn't match, then whatever you downloaded is not what the person who originally signed it meant for you to download. In order to do this, you have to find the public key for the openbsd version/type you are using. The release notes have the signing keys published at the top. Another place to look is the openbsd signify repository, here or here. You want to cross check multiple signing keys because it much harder for a malicious party to compromise the public key in multiple locations. In my case, I needed the openbsd-64-base.pub. After downloading that, I ran the following: signify-openbsd -Cp openbsd-64-base.pub -x SHA256.sig install64.fs . If this doesn't say that the signature is verified, then either the key is wrong, the sig file is wrong, or the installer download file is wrong. After all of that, you can be fairly confident that the installer downloaded correctly.

Create Installation Media

Start with a blank USB flash drive. If it has partitions on it, delete them with fdisk and write 0's to the first ~MB using dd to zero out the partition table (many online tutorials for how to do this). Then run the command: sudo dd if=installXX.fs of=/dev/sdX bs=1M . On openbsd, the bs=1M would be bs=1m, but I'm doing the installation media creation from a Linux computer. sdX should be the USB drive device (whole device, not a partition on the device). This will create the installation media.

Installation Process

fdisk,disklabel,dd

Boot the old laptop with the OpenBSD installation media. The installation guide (or this one) does a decent job of explaining how to do a simple install, which should work for most people. I want to do a simple install on an encrypted drive. 

OpenBSD uses its softraid RAID utility to provide disk encryption. The first step is to figure out which disk you want to encrypt. From the boot menu, select "s" for shell. You can do sysctl hw.disknames to get the devices and disk names, as well as dmesg | egrep '^([cswf]d). ' . For my laptop, the sysctl command outputs 4 devices: sd0:,cd0:,rd0:....,sd1:..... Oddly, the sd0 and cd0 did not have associated disknames. cd0 is the cd drive, and since nothing is in it, that kind of makes sense. But it turns out sd0 is the primary HDD (see below), so I expected it to have a disk name. rd devices are for if you boot into ram, so that can be ignored for our purposes. The dmesg command is a little more useful. It says that sd0 is a drive connected to the scsi bus, and the part number is the primary HDD. It says sd1 is a also a drive connected to the scsi bus, but it's part number is the USB drive I put the installer on. The last check that needs to be done is with fdisk. fdisk sd0 returns the partition table on sd0. Multiplying the number of sectors by the sector size (512 bytes for most HDDs) yields the HDD size, so sd0 is definitely the drive I want. The partition table shows the old linux installation I had on it, which I plan to overwrite. fdisk sd1 returns nothing, I'm guessing because openbsd doesn't want you to accidentally mess up the installer's partitioning while you're using it. One more useful command is disklabel -h sd0 . This outputs more useful information. 

The next step is to prepare the drive (assumed to be sd0 from here on). If you're worried about an attacker figuring out how much of your drive you actually use, then its a good idea to write random bits to the whole drive. This is accomplished with the following: dd if=/dev/urandom of=/dev/rsd0c bs=1m . Note the output device name. OpenBSD devices are named differently than linux devices. The prefix "r" means raw, and the suffix "c" actually refers to the complete disk instead of a partition. If you try to use sd0 instead of rsd0c, it will complain that the disk is full and not do anything. This command will take a long time to run for the full disk. I just let it run for a few minutes, then killed it because this is not a security-critical system. I then ran dd if=/dev/zero of=/dev/rsd0c bs=1m for a few seconds to zero out the first chunk of the disk so that fdisk doesn't think there is a (random) partition table present. Now do disklabel -h sd0 and fdisk sd0 . You should now see that the c partition is "unused" and the partition table is all zeros. 

Since this is an old laptop with an old BIOS, I don't have UEFI, and so I don't need a GPT. I need a MBR instead, so I run: fdisk -iy sd0 . The disk encryption guide shows the command for a GPT. Next, create the partitions using disklabel: disklabel -E sd0 . This launches the command driven editor. Following the guide, I created an "a" partition, default offset, full size, FS type RAID, then "w" for write, and "q" for quit the editor. disklabel sd0 now shows the new a partition, which will become the new encrypted device. The next command uses bioctl, which is the RAID management interface: bioctl -c C -l sd0a softraid0 . The -c C means use RAID type CRYPTO, which basically means encrypt the chunk of device -l sd0a, then name it softraid0. It will ask for a passphrase, then attach the new encrypted volume to a pseudo-device, in my case sd2 (because sd1 was the usb drive). Before you can do anything else, you have to run the following command: cd /dev && sh MAKEDEV sd2  . This creates the device nodes in /dev. The way I understand it is that OpenBSD doesn't automatically created the /dev/ nodes for every device, so you have to run the MAKEDEV script for each new one. 

Now, lets say you want to change the password of the encrypted volume. To do that, run: bioctl -P sd2 , enter your old pass phrase, then your new one twice.  

Finally, overwrite the first 1MB of the encrypted volume with zeros for the new MBR: dd if=/dev/zero of=/dev/rsd2c bs=1m count=1 . Now restart your computer. I found that the restart was necessary or the installer would try to install to the ramdisk device instead of another device, resulting in write errors complaining about not enough space on the disk. Go back into the shell and run the bioctl -c C -l sd0a softraid0 and cd /dev && sh MAKEDEV sd2  commands again to discover and mount the previously made crypto volume. Type exit and hit enter to return to the main menu. Now type I and hit enter and follow the installation guide.

I didn't bother to set up the network devices yet. I selected yes for start sshd by default and yes to star the X window system (since I want a gui). I setup a user, said no to allow root ssh login. I selected sd2 (the encrypted volume) as the root disk, then "w" for whole disk to be partitioned with MBR, then "a" for the automatic default partition scheme. The INSTALL.amd64 guide has more information on how to do custom partitioning. **check this. I didn't notice until much later, but it left about half the volume unused**. It will then take a few seconds to build all of the partitions. You have to option to put partitions on other disks as well, but if you just hit enter, then it assumes you're done. 

The next step is not in the instructions anywhere. If you downloaded the installXX.fs, then you also downloaded the "file sets", which are the main packages for openbsd. This means that they are now on your usb drive. When it asks which device the file sets are on, the default is cd because the file sets used to be a separate (cd) disk from the installer. What you need to do now is type "disk" and hit enter, then select the usb disk (sd1 for me), select it isn't mounted, and select the partition to be mounted, which should be the largest partition (where the file sets are located), which for me was partition a. Then it asks for the pathname to the file sets: just hit enter (the default is correct). If the default is not correct, you can type "!" and hit enter to drop into a shell, and then navigate around until you find the filesets, then type "exit" and hit enter to jump back to the installer. Follow the instructions for selecting/deselecting sets to install. The installer then mentions that the directory does not contain a .sig file and asks if you want to continue without verifying. This is intentional: if someone compromised the installer, then of course they could compromise the verification process. This is why we did the verification process on the entire installXX.fs file (which includes the file sets) at the beginning of these instructions. Thus, we can safely type yes and enter. It then performs the extraction/installation of the file sets. Set your time zone. Finally, reboot. 

After Installation

Boot your system (remember to remove the USB installer first). If you setup an encrypted volume, it will ask you for the passphrase. Then it will boot.

The first thing you need to do is get a network interface up and running (if you didn't during the install). I'm going to use an ethernet interface for this computer, so I connected a cable and did "ifconfig" to find the name of the ethernet adapter, which was msk0 for me, so I created a /etc/hostname.msk0 with the line "dhcp". Then I ran the command sh /etc/netstart . This will initialize the interface. I'll switch it to a static IP (in router and hostname.msk0 file) later. 

The next thing to do is update the date and time. Run: date . If it's way off, the default ntpd won't update/fix the time. Mine was way way off. So I had to run the "date YYYYMMDDHHMM" command to get it close, then ntpd -s to fix the time.

Now run "syspatch", which will patch errata in the current release. 

Run "fw_update" to automatically install missing firmware. It is likely that some firmware will be missing; that's the nature of obscure OS's. 

Read "man afterboot" for more tips, as well as the other openbsd FAQs. 

To change to a static ip address, I reserved an IP address for the computer in my router, the edited the hostname.msk0 file to have "inet 192.168.1.X 255.255.255.0 NONE" as the only line, where X is the IP address I reserved, then I created the file /etc/mygate and put the default gateway in it: 192.168.1.1 (IP of router). Then I ran sh /etc/netstat msk0 to restart the interface. If you did it right, you should see the adapter connected in your router administrator interface and you should be able to ping 8.8.8.8. The dhclient (when dhcp was active before) should have created a resolv.conf file with your gateway as the DNS (meaning: pass DNS queries through the router to your ISP). If not, check the resolv.conf man pages for how to create this.

The default window manager is fvwm. You can create a configuration file for it to make the display nicer, and there are guides online for doing this. There are also better window managers that I might look into later. Tip: to open the window manager menu, left click in a blank area of a screen. You can open multiple windows this way. I ran into an issue where if I log in as a user, then do su root, I can't open anything graphical because XAUTHORITY wasn't set. I had to add export XAUTHORITY=/home/user1/.Xauthority to my root's .profile in order to make it work, but I have no idea if that's a security problem or not.

Installing packages is done with pkg_add. I installed a graphical text editor nedit, rsync which has a progress feature for copying files, and ntfs-3g for mounting/accessing ntfs formatted media.

Shutting down is accomplished with "shutdown -p now".

I restricted SSH access to the local network (using pf firewall rules) and to key authentication only.  I turned off a few things in the ssh config file to harden it. I created key pairs (with a password on the private keys) on the computers that I wanted to be able to SSH with into the openbsd laptop, then put their public keys in the authorized keys file on the openbsd laptop.

The /etc/myname file contains the hostname of the computer, e.g. bob.my.domain. If that's the hostname, then you could add "bob" and "bob.my.domain" to the /etc/hosts file next to localhost on the 127.0.0.1 and ::1 lines. Then add "lookup file bind" to the top of /etc/resolv.conf (which should have your DNS server, the IP of my router in this case) so that any name queries, such as "bob", will result in the resolver checking the /etc/hosts file first before asking the DNS. I did the first part, but not the /etc/resolv.conf part because I don't have anything that needs to resolve my hostname. If I do in the future, I'll finish this setup.

This is a significantly more involved tutorial. Have a look there.

More fun with systemd services

I obtained some software that I wanted to install on the cluster. This software requires the installation of a license server as a systemd service. Unfortunately, following the instructions that came with this software resulted in a failed installation of the license server. I'm going to detail the steps I took to troubleshoot this problem here.

These commands were very useful for figuring out what was going on:

• systemctl stop [or start or status] (name).service
• journalctl -u (name).service
• ps aux | grep (command)
• gedit /var/log/messages
• Also just running the command with its arguments outside of the service

The license server came with a .sh install script. This particular script had very limited debug help info. It installs the program, then creates the systemd service file in /usr/lib/systemd/system and then creates a symlink from there to /etc/systemd/system/multi-user.target.wants/. It then attempts to start the service, which failed.

As with all debugging with stuff that involves networking, I started by disabling the firewall: systemctl stop firewalld

The first errors I got were related to missing /lib/ld-lsb.so.3. This is because the license server is a 32bit program, and CentOS 7 is 64bit. The 32bit libraries aren't installed by default. Theoretically, you should be able to just install redhat-lsb.i686, but this caused a bunch of conflicts on my system that were unresovable, particularly with the nvidia drivers. Instead of that, I did: yum provides /lib/ld-lsb.so.3 , then read the output to figure out which package contains that. It will be different for different systems. For me, I then had to install: yum install redhat-lsb-core-4.1-27.el7.centos.1.i686 . That should install a bunch of other packages, including libstdc++.i686, glibc.i686, and libgcc.i686. No unresovable dependencies, yay. This solved that problem.

The next error is because the systemd service file that the install sh script creates was not formatted correctly. This took a few hours to figure out. In a previous post, I mentioned creating a service file for a simple script that runs at start up to force the cpu mode to be "performance". The service file was a "oneshot" type, i.e. it runs once and is done, and the ExecStart command was a simple call to a bash script. In this service file, the type is "forking" because it creates more processes, and the ExecStart command requires multiple arguments. It turns out that command lines in systemd service files interpret spaces as separate commands, which means that if there is a space between the command and the argument, it will think the argument is another command. So how do you run commands with arguments in a service file? Well, whoever wrote the install script for this program thought that you could just enclose the whole command in quotes, which honestly makes a lot of sense...but nope, can't do that either because the quotes are just removed by the interpreter. Here's what you actually have to do in the [Service] block:

Environment='ARGS=-c whatever -l whatever2'
ExecStart=/absolute/path/to/command $ARGS

It has to be in that exact format. Adding quotes after the = around the arguments causes failure, quotes around the command fails, etc. Another option is to create an EnvironmentFile with the text in the ' ' in it, but that's a waste for only a couple arguments.

The next problem was a host communication error. This was uncovered by attempting to ping localhost, 127.0.0.1, and "headnode", which is the name of the node I was attempting to install this software one. I had my network adapters switched off. I could still ping localhost and 127.0.0.1, but due to the way my /etc/hosts file was setup for openmpi, I can't have "headnode" as an alias. Instead, "headnode" is an alias for the first node of the 192.168.2.X subnet, which is what I have openmpi use for administrative messages, and what I use for ssh, internode commands, etc. Because I had the network adapter associated with that subnet disabled, ping couldn't resolve "headnode". Also, because this license server uses the name of the computer as the host name, it has to be resolvable. When I switched that adapter on, headnode was then ping-able and that solved the communication error.

Finally, it's time to add the firewall back in. I stopped the license server service, turned on the firewall, and surprisingly the license server service didn't fail. I think it's using the subnet that "headnode" is on for communication, which I had to whitelist for slurm/openmpi (talked about in a previous post) because there was no way to specify a port range. This isn't necessarily a security problem because that subnet is on an adapter isolated from the internet (private internal network). Looking at the license server log, it listens on one fixed tcp port (though doesn't mention other networking info) and has another process that spawns with a random tcp port. This means that, if I hadn't already whitelisted that subnet, I'd probably have to for this because I don't think there's a way to specify the currently random tcp port. I ran the commands netstat -plnt and lsof -c (program) -a -i. Examining the results showed that it was listening on :::XXXXX (where XXXXX is the port number). The :: means the whole ipv6 space, and since Linux is "single stack" and maps between ipv4 and ipv6, it also means the whole ipv4 space. Looking at the lsof outputs, it has established connections to its spawned process on localhost:XXXXX as well as other ports. After some digging, it seems that firewalld does not police the loopback interface (lo) by default, i.e. it's not even assigned to a zone, so any localhost (or 127.0.0.1) communications will not be subject to firewall rules. THAT is why turning the firewall on didn't affect the license server program, because it only had localhost connections. It also means that no requests to the ports its listening on from external sources should get through the firewall, except for the whitelisted subnet (which is the compute node's subnet).

The next step was to install the program that used the license server. The installation seemed to go fine, but when I tried to run the program, it said that it could not find the license. During installation, I specified the license server as 127.0.0.1, which should be localhost. I tried changing it to localhost, too, but that didn't work. I had to change it to the (port)@localhost in order to get it to work, where (port) is the listening port number discovered above, e.g. 11111@localhost. For the compute nodes, I expect that I will have to use (port)@headnode because the license server will be on the headnode. Anyways, netstat and lsof don't show any new connections, so I guess it just connects to the server once at startup to check to make sure the license is there.

Hopefully this helps someone in the future with debugging.

I created an environment module for the program that uses the license server. I decided not to enable the license server systemd service, which means I'll have to start it before launching the program. If I was planning to use the program a lot, then I would enable it.

It turns out that the version of the software I have is not multi-node (MPI) capable. It has a remote server option that's convenient if you don't want to run on the computer you launch from, but it can't link multiple computers/servers into a cluster. I think I'll just leave it on the headnode for now.

3D printed jet engine

I 3D printed a jet engine. It's not a scale of any particular engine, and the mixer is wacky, but it has all of the major components and was free, so I can't complain too much. Definitely the most complicated thing I've printed to date.

Took a few months of off and on printing to finish all of the parts. Most parts had to be printed multiple times to get them to come out nicely. It took about 5 hours to assemble, much of which was spent trying to get it to turn smoothly without any rubbing. The final product is pretty nice.

I also printed a bunch of other things recently, and updated this post with them.

Internet Upgrade

We recently upgraded from BT 50Mbps internet to Virgin 105Mbps internet. Twice the speed for exactly half the cost. Virgin's customer service is worse, and the wifi router built into their modem was much worse than BT's "Smarthub", but it is much faster internet. The virgin quickstart kit is bullshit by the way: unless that property has had virgin internet recently, you'll have to have an engineer come out to get everything working: I think you can select that you don't have the right ports at checkout on their website in order to schedule the engineer. Their customer service agents are all in India and useless.

Anyways, I purchased a TPLink C1200 router to solve the poor range problem. You have to be careful buying routers: there are still many out there with 100Mbps ethernet ports. I'd consider modern to be 4x 1Gbe ports and AC wifi, and the C1200 seemed to have the best range/cost ratio with those options. It works great. I later noticed that there was a cable hook up right behind the server cabinet, so I moved the modem and router to the shelf above my homelab.

Originally, the headnode was connected to the wifi via a usb wifi adapter. This worked ok, but obviously wasn't ideal.

Network diagram drawn with draw.io

Now that I have the router close to the homelab, it just makes sense to use a wired ethernet connection for internet. However, I'm out of ethernet ports on the headnode's motherboard. As a reminder, there are two ethernet ports on the motherboard, each connects to one 8 port unmanaged switch and has a static IP assigned. The ethernet ports/switches are on different subnets. One switch is connected to the management ports on all four SM compute nodes for IPMI (administration). The other switch is connected to an ethernet port on all four compute nodes (for MPI, ssh, data transfer, etc). This works well: the headnode can talk to the management controllers over the ipmi interface and ssh to the compute nodes' OSs' over the intranet interface.

I'd like to replace the wifi link with an ethernet link. Since the IPMI network usage is tiny, and I have extra ports on that switch, I realized I could connect the router to that switch without a performance hit. Doing that means that the ethernet adapter on the motherboard connected to the ipmi switch has to have two different static IP addresses and subnets: one for the original ipmi, and one for the router. Turns out this is fairly simple to do using nmtui. Just edit the connection you want to add an IP address to, go to add address, add in the new IP address and subnet mask, e.g. 192.168.1.100/24, put the router's internal network gateway (usually just the router's IP, which is usually XXX.XXX.XXX.1) as the "gateway" and "dns", then make sure "never use this network for default route" is unchecked, bring the interface up and down, and you should now be able to access both networks from one interface. The route command is useful for troubleshooting. Here are some links I found useful: 1 2 3 4 . And if you need to cross the router between subnets: 1. I also set the static IP for the headnode in the router using the router's administration page (accessed by typing the router's IP into a browser and entering the administrative password) so that the router's DHCP wouldn't try to double assign the headnode's IP.

I also: 1. edited my firewall and moved the ipmi interface to the public zone, 2. updated the ddns (router) and noip settings so I can access it remotely, 3. got all of the wireless devices on the new wifi network.

Yay for more stable internet.

Follow up: homelab thermal solution

In a few prior posts, I mentioned that I replaced the soundproof cabinet's exhaust fans with quieter ones and added a temperature controller for the fans' speed. I sized the new fans well...much quieter, and they can handle the steady state heat extraction when the cluster is running at full power. However, what I didn't account for was heating of the room. The cluster is in a small office, and the office heats up after a few hours at full power. This makes sense...it's thermally equivalent to leaving a 1500W space heater on for hours/days. The reason that's a problem is that the inlet air is now hotter, which causes the exhaust to be even hotter, which causes the inlet air to be hotter, etc. The cabinet has a barrier between the outlet and inlet (both on the bottom of the cabinet) to minimize re-circulation, but if the whole room is hot, that doesn't matter. Leaving the door fully open and putting a large fan in the doorway seems to help some. It also offsets the house's gas heating needs, especially for the upper floor (where the office is). However, it's still getting too hot inside the room and cabinet. I don't have a great solution for this yet...



To do:
1. Fix room heating problem
2. Replace ntpd with chronyd

Automatic mulit-threading with python numpy

This came up while running python codes on the cluster. It turns out that numpy vector operations are automatically parallelized if numpy is linked against certain libraries, e.g. openBLAS or MKL, during compilation. Those linear algebra libraries will automatically use the max number of available cores (or if your processor is HT, 2x number of physical cores) for matrix operations. While that might seem convenient, it actually made a lot of people unhappy because of the overhead involved with multithreading lots of tiny matrix operations. Fortunately, there is a way to control the max number of threads used, and some devs are working on a way of dynamic control via numpy.

I created the following basic test script. It generates two random matrices, then multiplies them together. The random number generation is a serial operation, but the dot product is parallelized by default.

import os
#must set these before loading numpy:
os.environ["OMP_NUM_THREADS"] = '8' # export OMP_NUM_THREADS=4
os.environ["OPENBLAS_NUM_THREADS"] = '8' # export OPENBLAS_NUM_THREADS=4
os.environ["MKL_NUM_THREADS"] = '8' # export MKL_NUM_THREADS=6
#os.environ["VECLIB_MAXIMUM_THREADS"] = '4' # export VECLIB_MAXIMUM_THREADS=4
#os.environ["NUMEXPR_NUM_THREADS"] = '4' # export NUMEXPR_NUM_THREADS=6

import numpy as np
import time

#np.__config__.show() #looks like I have MKL and blas
np.show_config()

start_time=time.time()
#test script:
a = np.random.randn(5000, 50000)
b = np.random.randn(50000, 5000)
ran_time=time.time()-start_time
print("time to complete random matrix generation was %s seconds" % ran_time)
np.dot(a, b) #this line should be multi-threaded
print("time to complete dot was %s seconds" % (time.time() - start_time - ran_time))

The lines under import os set environment variables. The one(s) you need to set depend on what your numpy is linked against, as shown by np.show_config(). Note that those must be set before importing numpy.

I ran some experiments on one of the compute nodes (dual e5-2690v2) using slurm execution. Software was anaconda 5.2, so anyone with a recent anaconda should have similar behavior. My np.show_config() returned information about MKL and openBLAS, so I think those are the relevant variables to set.

Test 1: slurm cpus-per-task not set, ntasks=1, no thread limiting variables set.
Results: No multi-threading because slurm defaults to one cpu per task.

Test 2: slurm cpus-per-task=10, ntasks=1, no thread limiting variable set.
Results: dot used 10 threads (10.4s)

Test 3: slurm cpus-per-task=20, ntasks=1, no thread limiting variable set.
Results: dot used 20 threads (5.4s)

Test 4: slurm cpus-per-task=4, ntasks=1, no thread limiting variable set.
Results: dot used 4 threads (24.8s)

Test 5: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, OMP_NUM_THREADS=4
Results: dot used 4 threads (24.8s)

Test 6: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, OPENBLAS_NUM_THREADS=4
Results: dot used 10 threads (10.4s)

Test 7: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, MKL_NUM_THREADS=4
Results: dot used 4 threads (24.9s)

Test 8: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, VECLIB_MAXIMUM_THREADS=4
Results: dot used 10 threads (10.4s)

Test 9: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, NUMEXPR_NUM_THREADS=4
Results: dot used 10 threads (10.4s)

Test 10: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, OMP_NUM_THREADS=8, OPENBLAS_NUM_THREADS=8, MKL_NUM_THREADS=8
Results: dot used 8 threads (12.5s)

As you can see above, setting either MKL or OMP_NUM_THREADS will limit the number of threads, though apparently openBLAS is not being used, at least for dot. Also, limiting the number of cpus available will also limit the number of threads.

For this code, which has to run on 100's of different cases that can be run simultaneously, it looks like giving one full socket per case (10 cores) is optimal. The environment variables don't need to be set because the default behavior is to use all available cores (limited by slurm). That's assuming np.dot is a good indicator, which it might not be because her code is far more complicated.

Anyways, I hope someone finds this useful.