OpenBSD and growfs: growing a partition

I didn't notice until recently, but the automatic partitioning scheme I used during installation left about half of the volume unused. I decided I want to add all of that space to my /home partition. If you have more than two partitions, e.g. root and swap, adding space to a partition that isn't the last is almost impossible without reinstalling. Luckily, and probably by design, /home is the last partition on the disklabel, so this should be simple.

First, backup all of your files. If your disk is sd0, run: disklabel sd0 . Compute how much space you have (units are sectors, which are 512 bytes) between the end of the last partition (probably /home) and the end of the volume. Now compute the new total size of the partition. Write those numbers down. Note the letter of the last partition, e.g. sd0k. Make sure nothing is running in the /home directory. cd to /, then run umount /home or umount /dev/sd0k (where k is the letter of your last partition). You may have to use the -f option to force it to unmount. Now run disklabel -E sd0 . This opens the interactive disklabel editor. Type "c" and enter (you can also use "m" instead of "c", see the man pages). It will show you the maximum size, which you can type in, or you can type in how much ever you want to grow the partition with a + number of sectors. Check the numbers it gives against what you calculated: the number it gives should be close, but slightly smaller. After that, type "q", enter, "yes", enter. That will exit the interactive editor and save changes. Now do: growfs sd0k . This will grow the filesystem to the size of the partition. Next, check the partition: fsck /dev/sd0k . Let this run until it's finished. Now mount the partition again: mount /dev/sd0k /home . Make sure your files are still there. Run disklabel sd0 one more time just to check. That's it!

Note: These steps works just fine for partitions on an encrypted volume: that's how mine is setup.

Unfortunately, there isn't an easy tool for shrinking partitions.

Links: 1 2 3

OpenBSD USB transfer speed woes, and a hack to get around that

One of the major problems with OpenBSD is how slow file transfers are to non-openbsd file systems. It has ports of various file system adapters, like ntfs-3g (which is based on FUSE) for reading/writing to NTFS formatted media. However, these have known speed issues that no one has bothered to fix yet.

As a test, I tried to copy files from the internal HDD to an external 8TB USB 3.0 HDD formatted NTFS. In Windows and various Linux distros, the sustained write speed is 70 MB/s+. In Openbsd, I plugged it into a confirmed usb 3 port (usbdevs -v), mounted it with ntfs-3g /dev/sd2j /mnt/usb1, and then tried to copy a large (>1GB) file to it with cp or rsync. Rsync has a progress option which shows write speeds. cp wasn't much if any faster. I couldn't get over 2.2 MB/s write speed with a USB 3.0 port, or over 1.6 MB/s with a USB 2.0 port. I tried the ntfs-3g mount options noatime and big_writes, but they had no affect on write speed. Openbsd's port doesn't have an async option. I also tried a USB 3.0 flash drive, also formatted NTFS, that can get higher write speeds than the external HDD in Windows, but it's even slower on the Openbsd laptop.

From what I've read online, there's literally nothing you can do unless you feel like re-writing ntfs-3g or something like that.

I found a hack around this limitation. I setup a ftp server (ftpd) on the openbsd computer and added the following pf firewall rules in an anchor:

pass in on $ext_if proto tcp from $lan_net to port 21
pass in on $ext_if proto tcp from $lan_net to port > 49151

I then enabled and started the ftpd service. I connected the USB HDD to my Windows machine (on same LAN as the openbsd computer), and did "add network location" in windows, ftp://xxx.xxx.xxx.xxx (IP of my openbsd computer) and logged in with the user account on the openbsd computer. This allowed me to browse files on the openbsd computer. I copied (ctrl+c ctrl+v) a large (>1GB) file from the openbsd computer to the usb hdd. Interestingly, window's transfer time estimate was too long...calculating speed from that, it was estimating ~2 MB/s. However, the actual average transfer speed was ~8.5MB/s. Thus, it's about 4x faster to transfer files from openbsd->FTP->Windows->USB HDD than openbsd->USB HDD. I didn't attempt any ways to speed this up, and the openbsd laptop's network adapter is limited to 100Mbps, so it might be possible to get higher transfer speeds.

Update: Hit ~13 MiB/s (12 continuous) running 2 file transfers with the filezilla client in windows. That's right at the upper limit of the 100 Mbit/s ethernet port on the openbsd machine. It could probably handle much faster with a gigabit connection.

If the X windows system didn't convince me, this example really showcases why openbsd is meant to be a server OS, and not a personal computer OS.

Headnode Wake-on-lan Support

In a previous post, I mentioned setting up DDNS with NoIP, configuring my router with the DDNS name, and configuring it to forward SSH traffic. This effectively lets me control the headnode, and thus the cluster, from anywhere in the world over SSH. However, I still have to leave the headnode on 24/7 in order to have access to it.

This is where wake-on-lan (wol) comes in. The most common approach is the "magic packet" approach, where a packet with the network adapter's mac address in a certain pattern is sent to the network adapter, thus telling it to wake the computer up. Your network adapter, motherboard, and PSU have to support it to work, which is where my first problem was. The ASUS Z10PE-D8 has two ethernet ports on board, but neither support wake-on-lan. This is not an oversight: ASUS specifically excluded that capability to try to get you to buy their management chip (ASMB8-iKVM), which is fairly expensive, though comes with a bunch of nice remote management features. Since I needed another port for internet anyways, I purchased a 4 port gigabit pcie card (IBM rebrand of Intel i340-t4) that supports wake-on-lan on port 1. After configuring the router and static IP for the new adapter, I could run ethtool (interface name) to examine its wake-on-lan capability. Sure enough, there was a "g" next to wake-on-lan, meaning it supports magic packets. Not all ethernet adapters support wake-on-lan...make sure the one you are planning to buy does before you buy it.

I followed various guides for implementing wake-on-lan. Links: 1 2 3. Originally, it was only meant to be used within a LAN. I found a windows powershell script (there are a bunch, just google them) that creates a magic packet based on the headnode internet interface's MAC address and then broadcasts it to 255.255.255.255. This successfully woke the headnode. However, more work needed to be done to be able to wake-on-lan over the internet.

I created another port forwarding rule ("virtual server" in my router) to forward UDP traffic from an external port to an internal port with the headnode's static IP address. Typical ports for wake on lan are 7 or 9, but I think you can use any. The headnode already has a static IP address, but I also had to create a static ARP assignment in the router, which binds a MAC address to that IP address. I already had DDNS setup with a hostname for my public facing IP. I modified the windows powershell script (on my laptop) to look up the IP address of a given hostname, gave it the cluster's DDNS hostname, changed the port to the external port I forwarded, logged into my VPN so that I would send the wake-on-lan packet from the external internet, and ran the script. Boom, wake-on-lan over internet. I also tested this using the "wake-on-lan" android app when connected to my cell network. There's a wakeonlan linux package, too. Many options to get this to work.

Security disclaimer: opening inbound ports in a router firewall is never without risks. Make sure your computer's firewall rules are setup correctly.

OpenBSD, OpenVPN, and a bittorrent client

Foreword

I do not use or condone the use of torrenting software for the illegal sharing of copyrighted material, i.e. piracy. This post is for promoting the use of torrents for legal file sharing and for informational purposes.

Most of this post will be useful to other operating systems and VPN clients. The only part specific to OpenBSD is the PF firewall rules.

Bittorrent

Despite the nefarious reputation of torrenting, it is a pretty amazing technology that has legitimate purposes. Bittorrent is a file transfer protocol. The typical way you download a file from the internet is as follows: you find the file download link of the file you want, you click it which sends a request to the server, a connection is established, and the file server serves you the file. There are many protocols, e.g. FTP, but regardless of the protocol used the whole file is hosted on a server somewhere, and you download the whole file from that server. If the person/company hosting the file server is risk-averse, they have backup file servers in case the primary one goes down, but that's not always the case. For "free" files like Linux distros, a common practice is to have multiple mirrors, which are file servers that have copies of the file, so even if the primary source and some of the mirrors go down, you can still download the file from another mirror. Bittorrent is different. In order to improve robustness, a file that is served with bittorrent is broken up into many smaller pieces. The computers that are currently downloading ("peers") or have fully downloaded the file serve ("seed") these pieces to other computers that are downloading ("leechers"). This is called a "swarm". A computer downloading might pull parts of a file from 100 different sources. A torrent file (.torrent) contains how the pieces go together and information about the "tracker", but doesn't contain any of the actual file to be downloaded. A tracker is a server that handles creating connections between peers, and it doesn't store or process any of the file to be transferred. A more modern approach uses distributed hash tables (DHT), removing the single failure point of the tracker for establishing connections. The bittorrent protocol reduces strain on the file server (no single central file server that has to do all of the serving), vastly improves redundancy/robustness, and can work well with low network bandwidths and dropouts. It's also tamper resistant. If a hacker gains access to a central file server, the hacker can compromise all of the files being served from that server. With bittorrent, each piece of the file to be downloaded has a cryptographic hash computed for it when the torrent was created and these are stored in the .torrent file. If a piece is modified, the hash changes, and the bittorrent client (the piece of software running on a computer doing the downloading and seeding) knows to reject that piece. A computer wishing to download a file has to obtain a copy of the file's .torrent file, which the bittorrent client interprets, communicates with the tracker/DHT thus creating connections, downloads pieces, and assembles the file, all without a central file server. Pretty cool, right? 

Unfortunately, it has not been widely adopted for legal purposes, likely due to its association with illegal file sharing of copyrighted material. It's also partially due to the greedy nature of humans: in order for a bittorrent system to work, the files have to be seeded by multiple sources, which implies leaving the fully downloaded file on your machine for a long time and seeding it (letting others use your upload network bandwidth). This is, of course, rare, and so a common problem with both legal and illegal torrents is a lack of seeders. In the illegal realm, there are "private trackers" that enforce strict rules on the ratio of seeded data to downloaded data, but as their name suggests, they are private, hidden, hard-to-get-into communities. In the legal realm, what often happens is similar to the central file server with mirrors topology mentioned earlier: a central server that initiated the torrent shares the file with its mirrors, which then also seed the torrent. Users (leechers) download the file, but don't necessarily have to seed it (though it would help with speeds if everyone did). The various advantages over a single server serving the file are still present, but it feels less democratic because the people using the file aren't sharing it. An example of this would be ubuntu distro downloads via torrents. Another reason widespread adoption hasn't occurred is the relative simplicity on the user end of http or ftp file transfers: click a link to download, vs. installing and configuring torrent client, finding torrent files with seeders, etc. 

Many network operators outright block the normal ports used for bittorrent, though due to its ability to operate on udp or tcp on any port, it's impossible to completely block without packet inspection techniques. That's where a VPN comes in: since VPNs encrypt the network traffic going through them, packet inspection techniques don't work. As mentioned in a previous post, VPNs are also useful for maintaining privacy while online.

Using a Bittorrent Client To Share Files

As mentioned earlier, a bittorrent client is a piece of software running on your computer that handles the uploading and downloading of files. There are many different ones, but for the purposes of the rest of this post, I will use qBittorrent, which is an open source version of uTorrent. 

In order to start a file download, you need to obtain through normal download channels a .torrent file or magnet link of the file you wish to download and open it with the torrent client. The client then connects to the tracker(s) listed in the .torrent file to help find connections, or directly finds other peers via DHT. Then the client connects directly to the peers in order to request pieces and send pieces, thus becoming part of a swarm. The client may also report progress to trackers to help the tracker with its peer recommendations. When the client has all of the pieces of the file, it assembles them, and then starts sharing the file as a seeder instead of a peer.

Many clients have a built in function for creating your own .torrent file from a file you wish to share. In qbittorrent, go to Tools->Torrent Creator. Navigate to the file or folder you want to share. It will automatically calculate the number of pieces to split it into. If you want it to be private, i.e. you send the .torrent file to the people you want to share the file with, then you can select don't share with DHT network and don't put any public tracker urls in the box. If you want to share it publicly, then don't check the private box and put a few public tracker announcement urls in the corresponding box. You can add comments and other data, too. Then click create torrent. That's all there is to it. It's definitely easier than setting up your own file server, and I think it takes less than time than uploading to a file share site, e.g. google drive, and creating a sharing link...it certainly isn't harder. 

Another cool thing: some clients, including qbittorrent, can be used as trackers. This is useful for sharing files among a small group of people.

Hopefully you now have a good understanding of how torrenting works. The following sections will show how to set up qbittorrent on openbsd with an openvpn vpn. 

Firewall Rules 

Properly implement firewalls help reduce security risks by controlling network traffic. The next two sections detail the general firewall requirements when not use a VPN and when using a VPN. These should be applicable to all operating systems. I assume that your setup is a single host (computer) behind a router. More complicated network architectures will require more complicated settings. I discuss OpenBSD-specific firewall rules further down.

Generally, unless you're using a security hardened OS or firewall, all outbound connections on a host are allowed by default. This kind of makes sense: if a program is making an unwanted outbound connection request from your computer, then it has already compromised your system. Out going connections will use a random network port, and you can block outbound ports and connections using firewall rules. On the contrary, most inbound ports are blocked by default. Some common ones, like the ones used for ssh, http, etc. might be open by default. Inbound ports can be opened using firewall rules, which allow inbound connections to be made. Depending on your firewall, you can restrict the a rule to apply to a specific protocol (tcp, udp) network device, IP address, or program. Note that once a connection is established, data can flow either direction, regardless of whether the connection originated from an outbound or inbound request.

Without a VPN

In this example, the bittorrent client is on a host (computer) with a firewall that is connected to a router that has its own firewall, and the router is connected to the internet. The network interface device on the host that connects to the router will be called "ext1" that has an IP address of 10.0.1.1. for this example, and the computer has another network device connected to the router called "ext2" with an IP address of 10.0.1.2. These IP address are static IP addresses are assigned to the host network devices in the host OS and by the router. DHCP (dynamic IP addresses) are not ideal for firewall rule creation because if the IP address of the network device changes, and a firewall rule was created for the previous IP address, it will no longer apply.

If your host firewall allows outbound connections by default, then host firewall rules for the bittorrent client aren't necessary because it will be able to create outbound connections by default. If you want to restrict the client's outbound connections to a certain network device, e.g. ext1, or IP address, e.g. 10.0.1.1, then you can do that with firewall outbound rules. This is useful if you want the bittorrent traffic to only go over a certain network interface. Some bittorrent clients allow you to select the network interface to use, but they might switch to another if that one becomes unavailable for whatever reason, so firewall rules are safer.

If your host firewall blocks outbound connections by default, you'll need to allow the bittorrent client to make any outbound tcp and udp connection. If you want to restrict the client's outbound connections to a certain network device, e.g. ext1, or IP address, e.g. 10.0.1.1, then you only allow connections over that device or from that IP address.

Your router's firewall will probably allow outbound connections by default. If so, you don't need to do anything with the router's firewall configuration. If not, in the case of custom router firewalls, then you will need allow all outbound udp and tcp traffic from the desired host device's IP, e.g. 10.0.1.1. Router firewalls will block inbound connections by default.

The bittorrent client should now work fine. It will be able to make outbound connections to peers and seeds. Once the connections are established, you can both download and seed. However, seeds and peers will not be able to find you because your router firewall and host firewall block incoming connections by default. This limits the number of connections you can make, which limits your net download and upload speeds. If upnp is enabled on your router firewall and host, the client might be able to accept inbound connections. However, UPnP has security flaws (think about it...allows remote connections to bypass a firewall...not good) and should almost always be disabled if any part of the network interfaces with the internet.

So, without upnp, how do we allow inbound connections to the bittorrent client? The first step is to select an inbound port number. You want to choose one in the range 49152-65535 because these are "unregistered" to any specific piece of software. I will call this the "listening port". Next, create an inbound host firewall rule allowing tcp and udp connections over the listening port. It's good practice to restrict this "hole" in the firewall as much as possible. For example, you can restrict the open inbound port to connections made via the ext1 interface, or to ext1's IP address (10.0.1.1), and (if possible) only for the bittorrent client program. Next, set the listening port in the bittorrent client to the one you chose; if your client doesn't allow you to pick it, you need to find out which port it listens on for inbound connections and use that as the listening port. The last step is to allow the inbound connection through your router firewall. This is where "port forwarding" comes in. In your firewall's advanced settings, look for "virtual server" or "port forwarding". You want your router to forward inbound traffic aimed at your public facing IP address on the listening port to your internal IP address, e.g. 10.0.1.1., on the listening port. Create the port forwarding rule for both TCP and UDP protocols with the external and internal ports the same as the listening port, and the internal IP as the IP of the host network device, e.g. ext1's 10.0.1.1. That should be all of the necessary firewall rules.

To test this setup, start up our bittorrent client (with the listening port you chose earlier) and have it start downloading or seeding a file. This means the client will be actively listening on the listening port. Go to canyouseeme.org. It should find your public facing IP address. Change the port to your listening port and click "check port". It should say success: this means you can accept incoming connections, which should vastly improve torrent performance. If you get an error, your firewall rules are probably not correct.

Bittorrent client tips: deselect any of the following connection options: "use upnp/NAT", "use different port on each start up", "random port". If you can select the network device to use (might be in advanced options), do so, so it doesn't use a random device that you don't have firewall rules setup for.

With a VPN

Using a torrent client with a VPN is recommended. In fact, I recommend using a VPN all of the time. Whoever is watching the torrents can see which IP address the pieces are going to, and thus see who is downloading the file...not necessarily a problem for legal file transfers, but this is the most common way people who download illegal files get caught. However, if you use a VPN, the VPN server's IP address is the one the pieces are seen going to. If the VPN provider is bad, they keep internal logs that correlate their users' IP addresses with their server IP addresses. If the VPN provider is in the US, they can be court-ordered to keep logs, which can then be subpoenad. So, rules to follow: get a VPN with servers outside of the US, and use one that (at least says) doesn't keep logs. This doesn't prevent people from doing something dumb and getting caught, though. For example, if you sign in to a website, e.g. gmail, when accessing the VPN, then your VPN IP is now associated with everything you do during that session. Anyways, OPSEC is not the point of this post.

The setup is almost the same as the case without the a VPN (read first paragraph, previous section), except now a VPN tunnel is created through the router to the VPN server. A virtual network device will be created that has its own IP address, and the VPN server's firewall will take the job of your router's firewall. Thus, your router settings and firewall are irrelevant now, i.e. don't do any port forwarding in your router settings.

The host firewall rules will be the same as the without a vpn example, except applied to the new VPN virtual interface device and IP address instead of the physical interface device (ext1 in the last example). It's a good idea to block all connections for the client program on all devices except for the VPN virtual device and its IP address. You'll also want to change your default gateway to the IP address of your VPN interface. These rules force all traffic through the VPN. If this is not desired, you can implement fancy routing rules to route certain traffic through the VPN and certain traffic on the normal interface to your router.

The VPN server's firewall will allow all outbound traffic and block all inbound traffic (except maybe for certain commonly used ports).

The bittorrent client should now work fine. It will be able to make outbound connections to peers and seeds. Once the connections are established, you can both download and seed. However, seeds and peers will not be able to find you because the VPN server's firewall and host firewall block incoming connections by default. This limits the number of connections you can make, which limits your net download and upload speeds. VPN servers do not enable upnp, so make sure you disable that in your bittorrent client.

Setting up port forwarding is different than the previous example. The first step is to get a forwarded port from your VPN provider (make sure they offer this service, not all do). If they do offer this service, you usually can't choose which port is forwarded, they just assign you one. I will call this the "listening port". Next, create an inbound host firewall rule allowing tcp and udp connections over the listening port. It's good practice to restrict this "hole" in the firewall as much as possible. For example, you can restrict the open inbound port to connections made via the virtual VPN interface, or to its IP address, and (if possible) only for the bittorrent client program. Next, set the listening port in the bittorrent client to the one given to you by the VPN provider; if your client doesn't allow you to change it, you should use a different client. Don't setup any port forwarding rules in your router...they aren't necessary due to the VPN tunnel. That should be all of the necessary firewall rules.

To test this setup, start up our bittorrent client (with the listening port you chose earlier) and have it start downloading or seeding a file. This means the client will be actively listening on the listening port. Go to canyouseeme.org. It should find your public facing IP address. Change the port to your listening port and click "check port". It should say success: this means you can accept incoming connections, which should vastly improve torrent performance. If you get an error, your firewall rules are probably not correct.

Bittorrent client tips: deselect any of the following connection options: "use upnp/NAT", "use different port on each start up", "random port". If you can select the network device to use (might be in advanced options), set it to the virtual VPN device, so it doesn't use a random device that you don't have firewall rules setup for.

OpenVPN Setup

I cover most of the steps required to get OpenVPN + PIA working in a previous post. I'll cover the changes I've made since then in this section.

PIA only offers port fowarding on certain servers. I copied my previous ovpn configuration to a new one, then deleted all but one of the servers from that list. I changed the auth-user-pass line to "auth-user-pass /path/to/pia-login.txt", where the pia-login.txt contains my PIA username on one line and my password on the second line. This link shows how to encrypt and decrypt the credentials file, but I didn't implement that. Instead, I just made it readable and writeable by root only, and since openvpn has to be launched by root, this works fine. My reasoning is that if someone compromises the computer enough to get root access, your PIA credentials will be the least of your concerns. I added logging with "log-append /var/log/openvpn/openvpn", which is useful for debugging, though I'll probably switch that off in the future. I modified the up and down script lines to point to new scripts, which I'll talk about momentarily, and changed the "up" to a "route-up".

The old scripts only changed the DNS settings by replacing the resolv.conf file. The new up and down scripts still do that, but also do a lot more.

Note: gateway changes are still handled by the openvpn's "redirect-gateway def1" option.

Up Script 

First, the up script checks if the physical interface is active. If it isn't, then the VPN can't be active, so it exits. Next, it waits for the VPN to connect. If it takes longer than 120s to connect, then even if it connects later, PIA won't assign a port number, so it exits. Once it's connected, it changes the DNS settings. Then it asks a PIA server for forward port assignment via curl. Finally, it changes the firewall settings (discussed below).

The up script has to be called by a helper script that runs the up script in the background and then exits (exit 0). The reason for this is that, as a security measure (therefore by design), OpenVPN will not allow any packets to traverse the tunnel while any script which was started by the openvpn binary is still executing, which causes curl to fail. The helper script sort of tricks openvpn into thinking its done running setup scripts, thus allowing traffic through the connection. The helper script has to be called by the "route-up" openvpn option instead of the normal "up" because the former waits for the connection to be fully established.

Script link below.

Down Script

First, the down script undoes the changes the up script made to the DNS. Then it changes the firewall settings back to the way they were before the up script ran. Script link below.

Improved PF Settings

I made a lot of changes to my PF firewall settings to improve functionality and security. Here's my new pf.conf (with some stuff obscured, and many comments removed):

#header lines...
wi_if = "iwn0"
ext_if = "ext0"
ovpn_if = "tun0"
lan_net = "192.168.X.X/24" 

# don't filter on loopback interface
set skip on lo 

# scrub incoming packets
match in all scrub (no-df) 

#default: block everything
block 

#antispoofing:
block in quick from urpf-failed 

# only allow ssh connections from LAN. Can restrict to a few IP's later.
#pass in on $ext_if inet proto tcp from $lan_net to $ext_if port ssh flags S/SA synproxy state 

pass out on $ovpn_if proto { tcp udp icmp } all keep state 

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010 

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild 

anchor "piaopenvpnsetup"{
    pass out on $ext_if to 192.168.X.X #this is the router gateway IP
    pass out on $ext_if proto udp from self to any port 1197
} 

anchor piaportforward

Anchors allow for dynamic rule changes. Inline anchors (anchors with their rules specified in the pf.conf file), e.g. piaopenvpnsetup, are read and implemented at boot The piaopenvpnsetup anchor's rules are only necessary during the setup of the openvpn connection. The first rule is necessary so that DNS can resolve the PIA server's (listed in the .ovpn file) IP address. The second rule is necessary for PIA to establish the connection. They are flushed (removed) by the openvpn up script after the connection is established. They are added back in by the openvpn down script.

Anchors that don't have the rules inline, nor specify a path to a file with rules in it, e.g. piaportforward, are not implemented at boot. This rule is only loaded when the anchor-piaportforward file is created and the command pfctl -a piaportforward -f /etc/anchor-piaportforward is run by the up script. The down script flushes this anchor's rule. The rule is: pass in on $ovpn_if proto { tcp udp } from any to $ovpn_if port 12345 , where the port number is the port assigned by PIA in the up script. Even though the rules aren't defined in pf.conf, the "anchor piaportforward" line in pf.conf is critical. I spent about 6 hours trying to figure out why this anchor's rules weren't being applied before finding that in a tutorial somewhere. You can actually load the anchor without that line in the pf.conf, and pfctl -a piaportforward -s rules will show the rules as active, but they won't actually be active, so don't forget that line.

qBittorrent Installation and Launcher Script

Installing qbittorrent on openbsd is easy: pkg_add qbittorrent . You then launch it and its GUI manually with qbittorrent (gui-less version is called "qbittorrent-nox" I think). You can then go in and make the settings changes recommended above.

If your forwarded port changes, then it'd be nice to be able to update qbittorrent's listening port automatically. I created a launching script that does that. First, it checks to see if qbittorrent is running or not. If it is not, it launches it. Then it uses the Web API and curl to update the listening port.

Scripts

Link to the scripts.

OSs that have PIA Client Support

The PIA client supports many operating systems. It works similar to OpenVPN, but is much easier to setup and use. If your system is supported, I suggest using it. Requesting a forwarded port is as simple as click a check box in the settings, and the port is displayed on the client's gui. You should still make your own firewall rules, and you'll have to adjust the qbittorrent listening port, but it handles all of the VPN configuration stuff for you.

Possibly Useful Links

Windows powershell PIA/qbittorrent script

qbittorrent web api doc

qbittorrent options

OpenBSD PF FAQ

Transmission and PIA script

PIA and openvpn scripts

Original PIA port forwarding script thread

Modified script for a router linked in above thread

Enhanced version of script from above thread

curl and qbittorrent
PIA help desk port forwarding

Final Words

Have fun. Don't do illegal shit.

OpenBSD and wifi

Recently decided to attempt to use wifi + dhcp instead of static ethernet just for the learning experience. The openbsd networking FAQ, and the ifconfig man pages are pretty good resources for this.

There are many different wireless adapter brands/lines, and in bsd systems, they get their own name. So instead of all wireless adapters getting "wan0, wan1, etc", there could be ral0, ap0, etc. I had to create a hostname.if file for the wireless interface that included the nwid of the network I wanted to connect to (this can be scanned for with ifconfig) and the password in the format: "nwid MYWIFI wpakey MYPASSWORD", and then a second line with dhcp. Make sure you back up your original resolv.conf file: dhclient will overwrite it, though can control the dhclient's behavior with a /etc/dhclient.conf. For example, say you wanted to use a different name server, you could add a line to the dhclient.conf: "supersede domain-name-servers XXX.XXX.XXX.XXX". I have an openvpn up script that replaces this anyways, so I'm not concerned about it.

I then had to edit my pf.conf. I made a backup of the old one for the ethernet interface, then did the following: created a $wi_if variable for the wireless interface device, commented out the $ext_if lines, added "pass out on $wi_if to $lan_net" since the wireless network uses the same 192.168.1.X subnet, then changed the pass in and out rules on udp port 1197 (for VPN to establish) from the $ext_if to the $wi_fi. That's all. The "pass out on $wi_if to $lan_net" rule is necessary because the openvpn needs to be able to communicate over the original internet channel before establishing a connection (for DNS for example). I can comment this out and reload after the VPN channel is established. In fact, I should probably figure out some way cancel that rule once the VPN is established and then to only allow traffic to certain $lan_net IPs for ssh, scp, etc. That'd be more secure. pf can't discriminate based on program, but it can based on user. If a program was only used by a particular user, then it'd be even better to limit allowed connections to that user. Anyways, clearly there are some general improvements I could make to the firewall. I'll probably end up having to experiment with them.

Finally, I needed to run "sh /etc/netstart", which restarts all of the network connections, including starting openvpn. This brought the wifi interface up and started the openvpn connection, which takes longer because it keeps retrying until the wifi connection is established. After getting the firewall rules above correct, it worked. If the wireless connection drops, the vpn connection doesn't register it has been disconnected. I'm not sure how to go about fixing that. Every time it happens, I have to stop openvpn, then run "sh /etc/netstart" again. If you plan to use your openbsd laptop + openvpn with a wireless connection, I suggest writing some sort of script that monitors the wireless connection and does the above automatically.

"tcpdump -i wifi0 -y IEEE802_11_RADIO" (where wifi0 is your wireless adapter) is a nifty one-liner that will output current signal strength and noise of the wireless connection. Useful for finding a good place in the room to use your laptop.

OpenVPN on OpenBSD

It seems like I have a habit of not doing things the easy way. A great example of this is, instead of using something like Rocks, I built all of the cluster's software up from CentOS. This post will be another example of forgoing the easy for the sake of learning.

VPN's are a great tool to prevent ISPs and other internet companies from collecting valuable data on you. Internet companies, e.g. Google (owners of Blogger, the irony of which is not lost on me), collect data on what you do online in order to target you with ads. They also bundle the data and sell it to other companies, such as political organizations specializing in political manipulation. Government agencies collect data, i.e. spy, on their own citizens' internet usage. Saying you have nothing to hide is a terrible argument. "Arguing that you don't care about the right to privacy because you have nothing to hide is no different than saying you don't care about free speech because you have nothing to say." Enough of that rant.

While the free VPN's are mostly crap if not malicious, a few of the more prominent paid for VPN's certainly appear to do what they advertise. Many of these VPN services provide a VPN client for download. This client abstracts all of the background details of how the VPN is configured, is established, working, etc, and often include useful tools, such as routines for forcing all internet traffic over the VPN and for preventing DNS leaks. As a learning experience, I signed up for a month of one of the more prominent ones: PIA, or Private Internet Access. PIA has an excellent VPN client available for many OS's, including Windows, Mac, most flavors of linux, routers, etc. However, they do not have one that supports OpenBSD. You can probably guess where this is going...

Enter OpenVPN. OpenVPN is a very popular open-source VPN server-client program. It also has a port for OpenBSD. PIA's servers have OpenVPN servers setup, and they provide sample .ovpn configuration files. Now begins the adventure of getting OpenVPN working on OpenBSD.

Installing OpenVPN was simple enough using the pkg_add command. As of OpenBSD 6.4, the openvpn version number is 2.4+, which is good. The first step though was configuring the firewall using pf and pf.conf. I needed the firewall to do a few things: 1. Block all traffic except: 2. Let communications, SSH in particular, on my private LAN pass, 3. allow inbound and outbound traffic on the VPN's virtual interface. The OpenVPN FAQ and pf man pages were excellent resources, as was this blog, which I unfortunately didn't find until a couple hours into this. The following is my pf.conf file (with some stuff obscured):

#....header lines

# external interface:
ext_if = "interface0"
opvn_if = "tun0"
lan_net = "192.168.X.0/24"

# don't filter on loopback interface
set skip on lo0

# scrub incoming packets
match in all scrub (no-df)

block drop all

#antispoofing:
# antispoof for $opvn_if inet
block in quick from urpf-failed

# only allow ssh connections from LAN. Can restrict to a few IP's later.
# as it is, not necessary due to previous and following rules.
# block return in quick on $ext_if proto tcp from ! $lan_net to $ext_if port ssh
# Alternatively:
# pass in on $ext_if inet proto tcp from $lan_net to $ext_if port ssh flags S/SA synproxy state

# pass all traffic to and from the LAN
# these rules will create state entries due to the default
# "keep state" option which will automatically be applied.
pass in on $ext_if from $lan_net
pass out on $ext_if to $lan_net

# At this point, can only communicate on LAN.

# pass tcp, udp, and icmp out on the interface to internet
pass out on $opvn_if proto { tcp udp icmp } all modulate state 

# must open port udp 1197:
pass out on $ext_if proto udp from self to any port 1197
pass in on $ext_if proto udp from any to self port 1197

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

This successfully blocks internet traffic if the tun0 interface is down, i.e. if the VPN is not connected.  The next step was to configure the openvpn client. This is done with a .ovpn file. There are lots of useful guides for this online: 1 2 3 4 5 . Link 5 includes information for setting up an openvpn server on openbsd, too. You can download a PIA OpenVPN configuration file from the generator (log into your account) or from here. I used the "strong" settings, which include better cryptographic algorithms. I modified the PIA files slightly. Here's an example:

client
dev tun0
proto udp
remote belgium.privateinternetaccess.com 1197
remote de-berlin.privateinternetaccess.com 1197
remote lu.privateinternetaccess.com 1197
remote france.privateinternetaccess.com 1197
remote-random
resolv-retry infinite
nobind 

# add downgrade privileges after initialization:
# requires sudo package
# user _openvpn
# group _openvpn
# chroot /var/openvpn 

#prevent passwords from being cached in memory:
auth-nocache 

persist-key
persist-tun
cipher aes-256-cbc
auth sha256
tls-client
remote-cert-tls server
auth-user-pass
compress
verb 1
reneg-sec 0
...crl verify....(removed from this post for brevity)
...ca certificate ...(removed from this post for brevity)
# disable-occ
# being polite:
explicit-exit-notify
writepid /etc/openvpn/curpid
#prevent dns leaks. must use pia dns servers
# doesn't work on linux: push "dhcp-option DNS 209.222.18.218"
script-security 2
up "/bin/sh /etc/openvpn/change-resolve-conf"
down "/bin/sh /etc/openvpn/undo-resolve-conf"
# add to force all client traffic through VPN
redirect-gateway def1

I won't explain all of the settings. I removed the crl verify and ca certificates because they're very long and not necessary for this post, but they would be where the ...'s are. I commented out the disable-occ so that I could see warnings. OpenVPN with PIA gives warnings about the type of cipher or auth being used being inconsistent between the client and server. It turns out that these are false warnings and can be ignored (I did output at verb 4 once to check). I write the pid out to a file. Originally, I was trying to use that in a alias that could kill the openvpn process, but I found it was easier to use "pkill openvpn". The last line "redirect-gateway def1" is very important: that does a few things to your routing table (and undoes them when openvpn exits), but its primary purpose is to force all of your internet traffic over the VPN. On the rare occasions where that isn't the desired behavior, then you can do some fancy routing instead.

Now all of the traffic should be going through the VPN, which means that, if the VPN provider doesn't keep logs (they say they don't), then there's no way for your ISP/anyone to know what you are doing over the VPN (unless you expose yourself by logging into a website or something like that). However, there is still a way for your ISP to know which websites you visit...this is called a DNS leak. There are a few ways to check for DNS leaks. One of the easiest is to go to dnsleaktest.com. If your ISP is listed as the server, then you have a DNS leak. Another way is to do: nslookup -type=NS somewebsite.com  or  dig www.somewebsite.com . The "Server" entry will be the DNS server...if its your local gateway or your ISP's, then you have a DNS leak. This is why I added the script-security 2 and the following two lines to the .ovpn configuration file. Before I added those, I expected to (and did) have a DNS leak. The contents of the scripts are as follows:

#change-resolve-conf
#!/bin/sh
mv /etc/resolv.conf /etc/resolv_conf.orig
mv /etc/resolv_ovpn_conf /etc/resolv.conf  

#undo-resolve-conf
#!/bin/sh
mv /etc/resolv.conf /etc/resolv_ovpn_conf
mv /etc/resolv_conf.orig /etc/resolv.conf

Basically, one (up script) changes the resolv.conf file from the one with my router's IP in it (the original) to one with the PIA DNS servers, and the down script (run when openvpn closes) changes it back. This works great and prevents the DNS leak.

The last thing to do was figure out how to start openvpn without having to type out the full command: !/usr/local/sbin/openvpn --daemon --config /etc/openvpn/openbsd.ovpn (note: this has to be done as root). There are a few options: 1. create a /etc/hostname.tun0 file containing "up" on the first line and the above command on the second line, 2. create a rc.d script (see helpful link 5 above), 3. create aliases. Option 1 and 2 will auto-start the VPN on boot, and of the two, option 1 is recommended. Option 3 allows you to start (and stop) the VPN at will, and is what I implemented first, though it wasn't straightforward. OpenBSD uses ksh by default and doesn't have something like a .bashrc or .chsrc. It has a .profile. However, digging into the documentation, it's recommended that you create a $HOME/.kshrc file for your aliases and put a line in the .profile that calls it. However, if I log in as a normal user, then all of my shells start out as user shells, which means the .kshrc of root is never read even when I do su root. I have to do su root, then ksh -l, in order to force the .kshrc (and thus aliases) to be read. Kind of a pain, but it works. I created an alias for pkill openvpn, which allows me to stop vpn. I will probably switch to Option 1 shortly. 

After all this, I decided to download the PIA client on my windows machine just to try it. It took ~10 minutes to setup vs. the ~10 hours this took, haha. But at least I have an understanding of how all of this works now.

Update: See this post for updates to the PF rules (more secure) and ovpn configuration.

Installing OpenBSD on an old laptop

Up until this point, I've only had experience with Windows, Ubuntu, and RHEL/CentOS. BSD's are another type of Unix OS common in the server/IT/security world, and I thought it'd be a fun exercise to learn how to install and use one. While FreeBSD has more packages/support, OpenBSD supposedly "just works", and it's core is continuously security audited, so I decided to try that one.

Downloading and Verifying

Useful links: 1

The first step is to download the installer. Since my old laptop is a 64bit intel processor, I needed the "amd64" version of installXX.fs, where XX is the version number (6.4 at the time of this writing). You also need to get the SHA256 file, which contains checksums, and the SHA256.sig signature file. If you copy the link address of the installer, paste it into the browser, then navigate to the parent folder you should see those two files are available for download. Since I'm going to create the install media on a USB flash drive from another (Ubuntu) computer, I don't have the OpenBSD sha256 or signify tools installed. On Ubuntu, the corresponding tools are sha256sum and signify-openbsd, though typically you would use gpg on most linux systems for verifying signatures. To do the checksum for the installXX.fs file, do: sha256sum installXX.fs . The output should match the corresponding checksum in the SHA256 file you downloaded. If not, then it did not download correctly. The next thing to check is the signature. This helps prevent downloading of potentially malicious software: if the signature doesn't match, then whatever you downloaded is not what the person who originally signed it meant for you to download. In order to do this, you have to find the public key for the openbsd version/type you are using. The release notes have the signing keys published at the top. Another place to look is the openbsd signify repository, here or here. You want to cross check multiple signing keys because it much harder for a malicious party to compromise the public key in multiple locations. In my case, I needed the openbsd-64-base.pub. After downloading that, I ran the following: signify-openbsd -Cp openbsd-64-base.pub -x SHA256.sig install64.fs . If this doesn't say that the signature is verified, then either the key is wrong, the sig file is wrong, or the installer download file is wrong. After all of that, you can be fairly confident that the installer downloaded correctly.

Create Installation Media

Start with a blank USB flash drive. If it has partitions on it, delete them with fdisk and write 0's to the first ~MB using dd to zero out the partition table (many online tutorials for how to do this). Then run the command: sudo dd if=installXX.fs of=/dev/sdX bs=1M . On openbsd, the bs=1M would be bs=1m, but I'm doing the installation media creation from a Linux computer. sdX should be the USB drive device (whole device, not a partition on the device). This will create the installation media.

Installation Process

fdisk,disklabel,dd

Boot the old laptop with the OpenBSD installation media. The installation guide (or this one) does a decent job of explaining how to do a simple install, which should work for most people. I want to do a simple install on an encrypted drive. 

OpenBSD uses its softraid RAID utility to provide disk encryption. The first step is to figure out which disk you want to encrypt. From the boot menu, select "s" for shell. You can do sysctl hw.disknames to get the devices and disk names, as well as dmesg | egrep '^([cswf]d). ' . For my laptop, the sysctl command outputs 4 devices: sd0:,cd0:,rd0:....,sd1:..... Oddly, the sd0 and cd0 did not have associated disknames. cd0 is the cd drive, and since nothing is in it, that kind of makes sense. But it turns out sd0 is the primary HDD (see below), so I expected it to have a disk name. rd devices are for if you boot into ram, so that can be ignored for our purposes. The dmesg command is a little more useful. It says that sd0 is a drive connected to the scsi bus, and the part number is the primary HDD. It says sd1 is a also a drive connected to the scsi bus, but it's part number is the USB drive I put the installer on. The last check that needs to be done is with fdisk. fdisk sd0 returns the partition table on sd0. Multiplying the number of sectors by the sector size (512 bytes for most HDDs) yields the HDD size, so sd0 is definitely the drive I want. The partition table shows the old linux installation I had on it, which I plan to overwrite. fdisk sd1 returns nothing, I'm guessing because openbsd doesn't want you to accidentally mess up the installer's partitioning while you're using it. One more useful command is disklabel -h sd0 . This outputs more useful information. 

The next step is to prepare the drive (assumed to be sd0 from here on). If you're worried about an attacker figuring out how much of your drive you actually use, then its a good idea to write random bits to the whole drive. This is accomplished with the following: dd if=/dev/urandom of=/dev/rsd0c bs=1m . Note the output device name. OpenBSD devices are named differently than linux devices. The prefix "r" means raw, and the suffix "c" actually refers to the complete disk instead of a partition. If you try to use sd0 instead of rsd0c, it will complain that the disk is full and not do anything. This command will take a long time to run for the full disk. I just let it run for a few minutes, then killed it because this is not a security-critical system. I then ran dd if=/dev/zero of=/dev/rsd0c bs=1m for a few seconds to zero out the first chunk of the disk so that fdisk doesn't think there is a (random) partition table present. Now do disklabel -h sd0 and fdisk sd0 . You should now see that the c partition is "unused" and the partition table is all zeros. 

Since this is an old laptop with an old BIOS, I don't have UEFI, and so I don't need a GPT. I need a MBR instead, so I run: fdisk -iy sd0 . The disk encryption guide shows the command for a GPT. Next, create the partitions using disklabel: disklabel -E sd0 . This launches the command driven editor. Following the guide, I created an "a" partition, default offset, full size, FS type RAID, then "w" for write, and "q" for quit the editor. disklabel sd0 now shows the new a partition, which will become the new encrypted device. The next command uses bioctl, which is the RAID management interface: bioctl -c C -l sd0a softraid0 . The -c C means use RAID type CRYPTO, which basically means encrypt the chunk of device -l sd0a, then name it softraid0. It will ask for a passphrase, then attach the new encrypted volume to a pseudo-device, in my case sd2 (because sd1 was the usb drive). Before you can do anything else, you have to run the following command: cd /dev && sh MAKEDEV sd2  . This creates the device nodes in /dev. The way I understand it is that OpenBSD doesn't automatically created the /dev/ nodes for every device, so you have to run the MAKEDEV script for each new one. 

Now, lets say you want to change the password of the encrypted volume. To do that, run: bioctl -P sd2 , enter your old pass phrase, then your new one twice.  

Finally, overwrite the first 1MB of the encrypted volume with zeros for the new MBR: dd if=/dev/zero of=/dev/rsd2c bs=1m count=1 . Now restart your computer. I found that the restart was necessary or the installer would try to install to the ramdisk device instead of another device, resulting in write errors complaining about not enough space on the disk. Go back into the shell and run the bioctl -c C -l sd0a softraid0 and cd /dev && sh MAKEDEV sd2  commands again to discover and mount the previously made crypto volume. Type exit and hit enter to return to the main menu. Now type I and hit enter and follow the installation guide.

I didn't bother to set up the network devices yet. I selected yes for start sshd by default and yes to star the X window system (since I want a gui). I setup a user, said no to allow root ssh login. I selected sd2 (the encrypted volume) as the root disk, then "w" for whole disk to be partitioned with MBR, then "a" for the automatic default partition scheme. The INSTALL.amd64 guide has more information on how to do custom partitioning. **check this. I didn't notice until much later, but it left about half the volume unused**. It will then take a few seconds to build all of the partitions. You have to option to put partitions on other disks as well, but if you just hit enter, then it assumes you're done. 

The next step is not in the instructions anywhere. If you downloaded the installXX.fs, then you also downloaded the "file sets", which are the main packages for openbsd. This means that they are now on your usb drive. When it asks which device the file sets are on, the default is cd because the file sets used to be a separate (cd) disk from the installer. What you need to do now is type "disk" and hit enter, then select the usb disk (sd1 for me), select it isn't mounted, and select the partition to be mounted, which should be the largest partition (where the file sets are located), which for me was partition a. Then it asks for the pathname to the file sets: just hit enter (the default is correct). If the default is not correct, you can type "!" and hit enter to drop into a shell, and then navigate around until you find the filesets, then type "exit" and hit enter to jump back to the installer. Follow the instructions for selecting/deselecting sets to install. The installer then mentions that the directory does not contain a .sig file and asks if you want to continue without verifying. This is intentional: if someone compromised the installer, then of course they could compromise the verification process. This is why we did the verification process on the entire installXX.fs file (which includes the file sets) at the beginning of these instructions. Thus, we can safely type yes and enter. It then performs the extraction/installation of the file sets. Set your time zone. Finally, reboot. 

After Installation

Boot your system (remember to remove the USB installer first). If you setup an encrypted volume, it will ask you for the passphrase. Then it will boot.

The first thing you need to do is get a network interface up and running (if you didn't during the install). I'm going to use an ethernet interface for this computer, so I connected a cable and did "ifconfig" to find the name of the ethernet adapter, which was msk0 for me, so I created a /etc/hostname.msk0 with the line "dhcp". Then I ran the command sh /etc/netstart . This will initialize the interface. I'll switch it to a static IP (in router and hostname.msk0 file) later. 

The next thing to do is update the date and time. Run: date . If it's way off, the default ntpd won't update/fix the time. Mine was way way off. So I had to run the "date YYYYMMDDHHMM" command to get it close, then ntpd -s to fix the time.

Now run "syspatch", which will patch errata in the current release. 

Run "fw_update" to automatically install missing firmware. It is likely that some firmware will be missing; that's the nature of obscure OS's. 

Read "man afterboot" for more tips, as well as the other openbsd FAQs. 

To change to a static ip address, I reserved an IP address for the computer in my router, the edited the hostname.msk0 file to have "inet 192.168.1.X 255.255.255.0 NONE" as the only line, where X is the IP address I reserved, then I created the file /etc/mygate and put the default gateway in it: 192.168.1.1 (IP of router). Then I ran sh /etc/netstat msk0 to restart the interface. If you did it right, you should see the adapter connected in your router administrator interface and you should be able to ping 8.8.8.8. The dhclient (when dhcp was active before) should have created a resolv.conf file with your gateway as the DNS (meaning: pass DNS queries through the router to your ISP). If not, check the resolv.conf man pages for how to create this.

The default window manager is fvwm. You can create a configuration file for it to make the display nicer, and there are guides online for doing this. There are also better window managers that I might look into later. Tip: to open the window manager menu, left click in a blank area of a screen. You can open multiple windows this way. I ran into an issue where if I log in as a user, then do su root, I can't open anything graphical because XAUTHORITY wasn't set. I had to add export XAUTHORITY=/home/user1/.Xauthority to my root's .profile in order to make it work, but I have no idea if that's a security problem or not.

Installing packages is done with pkg_add. I installed a graphical text editor nedit, rsync which has a progress feature for copying files, and ntfs-3g for mounting/accessing ntfs formatted media.

Shutting down is accomplished with "shutdown -p now".

I restricted SSH access to the local network (using pf firewall rules) and to key authentication only.  I turned off a few things in the ssh config file to harden it. I created key pairs (with a password on the private keys) on the computers that I wanted to be able to SSH with into the openbsd laptop, then put their public keys in the authorized keys file on the openbsd laptop.

The /etc/myname file contains the hostname of the computer, e.g. bob.my.domain. If that's the hostname, then you could add "bob" and "bob.my.domain" to the /etc/hosts file next to localhost on the 127.0.0.1 and ::1 lines. Then add "lookup file bind" to the top of /etc/resolv.conf (which should have your DNS server, the IP of my router in this case) so that any name queries, such as "bob", will result in the resolver checking the /etc/hosts file first before asking the DNS. I did the first part, but not the /etc/resolv.conf part because I don't have anything that needs to resolve my hostname. If I do in the future, I'll finish this setup.

This is a significantly more involved tutorial. Have a look there.

More fun with systemd services

I obtained some software that I wanted to install on the cluster. This software requires the installation of a license server as a systemd service. Unfortunately, following the instructions that came with this software resulted in a failed installation of the license server. I'm going to detail the steps I took to troubleshoot this problem here.

These commands were very useful for figuring out what was going on:

• systemctl stop [or start or status] (name).service
• journalctl -u (name).service
• ps aux | grep (command)
• gedit /var/log/messages
• Also just running the command with its arguments outside of the service

The license server came with a .sh install script. This particular script had very limited debug help info. It installs the program, then creates the systemd service file in /usr/lib/systemd/system and then creates a symlink from there to /etc/systemd/system/multi-user.target.wants/. It then attempts to start the service, which failed.

As with all debugging with stuff that involves networking, I started by disabling the firewall: systemctl stop firewalld

The first errors I got were related to missing /lib/ld-lsb.so.3. This is because the license server is a 32bit program, and CentOS 7 is 64bit. The 32bit libraries aren't installed by default. Theoretically, you should be able to just install redhat-lsb.i686, but this caused a bunch of conflicts on my system that were unresovable, particularly with the nvidia drivers. Instead of that, I did: yum provides /lib/ld-lsb.so.3 , then read the output to figure out which package contains that. It will be different for different systems. For me, I then had to install: yum install redhat-lsb-core-4.1-27.el7.centos.1.i686 . That should install a bunch of other packages, including libstdc++.i686, glibc.i686, and libgcc.i686. No unresovable dependencies, yay. This solved that problem.

The next error is because the systemd service file that the install sh script creates was not formatted correctly. This took a few hours to figure out. In a previous post, I mentioned creating a service file for a simple script that runs at start up to force the cpu mode to be "performance". The service file was a "oneshot" type, i.e. it runs once and is done, and the ExecStart command was a simple call to a bash script. In this service file, the type is "forking" because it creates more processes, and the ExecStart command requires multiple arguments. It turns out that command lines in systemd service files interpret spaces as separate commands, which means that if there is a space between the command and the argument, it will think the argument is another command. So how do you run commands with arguments in a service file? Well, whoever wrote the install script for this program thought that you could just enclose the whole command in quotes, which honestly makes a lot of sense...but nope, can't do that either because the quotes are just removed by the interpreter. Here's what you actually have to do in the [Service] block:

Environment='ARGS=-c whatever -l whatever2'
ExecStart=/absolute/path/to/command $ARGS

It has to be in that exact format. Adding quotes after the = around the arguments causes failure, quotes around the command fails, etc. Another option is to create an EnvironmentFile with the text in the ' ' in it, but that's a waste for only a couple arguments.

The next problem was a host communication error. This was uncovered by attempting to ping localhost, 127.0.0.1, and "headnode", which is the name of the node I was attempting to install this software one. I had my network adapters switched off. I could still ping localhost and 127.0.0.1, but due to the way my /etc/hosts file was setup for openmpi, I can't have "headnode" as an alias. Instead, "headnode" is an alias for the first node of the 192.168.2.X subnet, which is what I have openmpi use for administrative messages, and what I use for ssh, internode commands, etc. Because I had the network adapter associated with that subnet disabled, ping couldn't resolve "headnode". Also, because this license server uses the name of the computer as the host name, it has to be resolvable. When I switched that adapter on, headnode was then ping-able and that solved the communication error.

Finally, it's time to add the firewall back in. I stopped the license server service, turned on the firewall, and surprisingly the license server service didn't fail. I think it's using the subnet that "headnode" is on for communication, which I had to whitelist for slurm/openmpi (talked about in a previous post) because there was no way to specify a port range. This isn't necessarily a security problem because that subnet is on an adapter isolated from the internet (private internal network). Looking at the license server log, it listens on one fixed tcp port (though doesn't mention other networking info) and has another process that spawns with a random tcp port. This means that, if I hadn't already whitelisted that subnet, I'd probably have to for this because I don't think there's a way to specify the currently random tcp port. I ran the commands netstat -plnt and lsof -c (program) -a -i. Examining the results showed that it was listening on :::XXXXX (where XXXXX is the port number). The :: means the whole ipv6 space, and since Linux is "single stack" and maps between ipv4 and ipv6, it also means the whole ipv4 space. Looking at the lsof outputs, it has established connections to its spawned process on localhost:XXXXX as well as other ports. After some digging, it seems that firewalld does not police the loopback interface (lo) by default, i.e. it's not even assigned to a zone, so any localhost (or 127.0.0.1) communications will not be subject to firewall rules. THAT is why turning the firewall on didn't affect the license server program, because it only had localhost connections. It also means that no requests to the ports its listening on from external sources should get through the firewall, except for the whitelisted subnet (which is the compute node's subnet).

The next step was to install the program that used the license server. The installation seemed to go fine, but when I tried to run the program, it said that it could not find the license. During installation, I specified the license server as 127.0.0.1, which should be localhost. I tried changing it to localhost, too, but that didn't work. I had to change it to the (port)@localhost in order to get it to work, where (port) is the listening port number discovered above, e.g. 11111@localhost. For the compute nodes, I expect that I will have to use (port)@headnode because the license server will be on the headnode. Anyways, netstat and lsof don't show any new connections, so I guess it just connects to the server once at startup to check to make sure the license is there.

Hopefully this helps someone in the future with debugging.

I created an environment module for the program that uses the license server. I decided not to enable the license server systemd service, which means I'll have to start it before launching the program. If I was planning to use the program a lot, then I would enable it.

It turns out that the version of the software I have is not multi-node (MPI) capable. It has a remote server option that's convenient if you don't want to run on the computer you launch from, but it can't link multiple computers/servers into a cluster. I think I'll just leave it on the headnode for now.