UniFi Controller, Ubuntu 18.04, and oVirt

Next on my long list of things to get running in the house is a more professional WiFi setup. I had heard great things about Ubiquiti and saw that one of their UniFi AP-AC-PRO units was on a pretty good sale over at Amazon. I picked one up and in keeping with all of my other projects aimed to set it up on my VM cluster.

VM Setup

All of the Ubiquiti hardware requires a control plane application running somewhere to manage and configure the site. Thankfully, the application is available for Linux as well as Windows and Mac; plus it is packaged conveniently for Debian derived distributions. The install guide recommends Ubuntu 16.04 or 18.04 to host the application and since the built in oVirt image repository has an image for 18.04 I went for the newer version.

After pulling down the template I set up a new VM with 2CPUs, 3GB of RAM, and 20GB of disk space as per the user guide. I also only included a single NIC with a static IP, but I may add a second at some point in the future if I implement a dedicated management network fabric instead of just running everything through the same switches. All pretty simple so far.

I boot up the VM and notice after a while that it seems to be hung during the boot process. I pulled up the console window and saw that the boot process was stuck at initializing BTRFS for some reason. A quick search brought me to this bug report which suggested adding a VirtIO serial console to the VM configuration to resolve the issue. A quick configuration change and reboot later and we are off to the races.

ovirt-guest-agent and freeipa-client

The next hurdle for integrating this system with my larger architecture was to add both ovirt-guest-agent and freeipa-client so that everything integrates nicely. The install was simple:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install ovirt-guest-agent freeipa-client

But when I tried to start up the guest agent I ran into the same permission error I had when installing SecurityOnion. Thankfully the log directory was already owned by the correct user and I just needed to reload the udev rules to get the permissioning issue sorted out.

$ sudo udevadm trigger --subsystem-match="virtio-ports"

Next up was getting the VM added to FreeIPA and again we hit a snag. Apparently, even when I set the full FQDN for the server through cloud-init Ubuntu only takes the host portion and sets that as the hostname. Unfortunately, the IPA client installer assumes that the hostname is set to the FQDN and fails if it isn’t. Again a pretty easy fix:

$ sudo hostnamectl status
$ sudo hostnamectl set-hostname controller.example.com
$ ipa-client-install --mkhomedir

Interestingly, this version of ipa-client-install doesn’t provide options to force NTPd usage or automatically create DNS records for the new server. That is an easy fix though and I just added the A and PTR records by hand into FreeIPA. With that, a quick reboot and we were ready to go installing the UniFi Controller.

UniFi Controller Install

The UniFi Controller is a Java application and ever so slightly involved to get running. Thankfully, a member of the Ubiquiti community has provided a handy little script which automates all of the installation. Now usually I run screaming from anything wget X | sudo bash but after pulling this script down manually and reading through everything that it is doing it looks free of anything shocking. I might try re-implementing it with Ansible if I get the time but for now the initial install was as simple as running it and taking all of the defaults.

Once the install finished I pulled up the web UI as described in the user guide and ran through the setup process there as well. When that finished I tried to adopt my AP but I noticed that while my AP was connected to the network and I could see it’s DHCP lease; the controller wasn’t discovering it. That sent me down yet another troubleshooting rabbit hole.

Device Discovery and Adoption

The first problem was that the controller VM and AP actually live on different subnets within my home network and there is a fairly restrictive firewall between the two to prevent anything nasty coming in over the WiFi. The controller and any managed devices need several TCP and UDP ports opened so I implemented those firewall changes and tried looking for the device again. No dice. A quick nmap from my laptop of the controller VM showed that the required ports were accessible through both the router and host firewall so it was on to the next theory.

Moving on, the APs are actually full Linux systems under the hood so the next debugging step is to actually log in and view the logfiles. Until the AP is adopted by a controller the default SSH credentials are ubnt:ubnt which will let you poke around the AP’s filesystem. I dug around in the router’s DHCP lease list until I found the address the AP had pulled and used that to SSH in.

After poking around in /var/log/messages I saw that the AP seemed to be trying to connect to http://unifi:8080/inform and was unable to resolve the name. I also found this forum post that said to reset the inform URL to have the device connect correctly, so I tried that next using the DNS name I had configured for the controller earlier.

$ set-inform http://controller.example.com:8080/inform

Immediately, the AP showed up in the controller interface allowing me to adopt it and provision. Once the initial provision was done I modified the AP settings to use a static IP and my FreeIPA DNS servers before allowing it to reconfigure again. Once that was done I added DNS entries for the AP and started setting up my WiFi networks.

Random Device Disconnects

While I was working through my network configuration I noticed that the AP was displaying as Heartbeat Missed and then Disconnected even while I could see it was still ping-able and passing traffic correctly. It seems that this is a common issue and that generally the solution is to remove and re-adopt the device. Unfortunately for me, that did not work as hoped and shortly after re-configuring the device it was again in the Disconnected state.

The next time the AP disconnected I SSHed into it using the post-adoption credentials provided in the controller UI to look at the log files. This time I didn’t see a failure to resolve unifi:8080 but rather the controller.example.com:8080 I had configured when setting up the device. After reconfirming that the DNS servers were correctly configured for the AP and that it could curl the controller manually, I tried running set-inform again. This allowed me to re-adopt the AP and reconnect it, but only temporarily. Pretty soon it was Disconnected again and the logs still said that it was unable to resolve the controller. The solution, frustratingly, was to simply use the controller’s IP address instead of it’s DNS hostname when running set-inform and that seems to have the AP connected properly now. I suppose that there is either flakiness in the DNS resolution on the AP or in how my FreeIPA servers are returning responses, but that’s a debug for another day.

An Aside About mDNS

The last thing that I noticed while getting the WiFi networks up and running was that Avahi/Bonjour/mDNS packets were not getting sent properly across the WiFi. Due to this only other devices on wireless were able to see each other and they couldn’t see any of the other devices connected to the LAN. This also seems to be a problem other people have faced with Ubiquiti equipment. My solution ended up being a combination of enabling Multicast Enhancement (IGMPv3) for the WiFi networks where I wanted mDNS working and restarting the Avahi mDNS reflector running on my router which was configured to pass mDNS packets between the different subnets it manages. With those changes devices started popping up again on the Macs and everything looked like it should.

Conclusion

After all the work of getting it set up the AP has actually worked fairly well and looks to provide what I need in an access point. Plus I must admit I am looking forward to a more centralized and streamlined configuration story as opposed to the consumer router turned AP route. The next couple of weeks will tell how well the controller connection holds up and how it handles multiple users streaming at the same time.

Update - Custom SSL Certificate

So I finally got tired of seeing the SSL error every time I wanted to connect to the controller interface and I decided to issue a certificate for it from my personal CA. I generated the CSR as described in the official documentation but ran into issues when I tried to import the certificate and CA from FreeIPA. The first error was just generally not being able to import the certificate with ace.jar and was resolved by stripping all newlines from both the certificate and CA files.

$ tr -d '\n' < host.crt > host.strip.crt
$ tr -d '\n' < ca.crt > ca.strip.crt

With that I was able to get ace.jar to at least parse my host certificate but it failed to recognize the CA certificate and would throw an error saying that it lacked an appropriate Certificate Authority in the keystore. I managed to fix that by directly importing my CA certificate into the UniFi keystore using the following command and the password aircontrolenterprise which is the UniFi default.

$ keytool -import -trustcacerts \
  -keystore /varlib/unifi/keystore \
  -alias custom-root \
  -import -file ca.crt

Since my controller host is already registered with FreeIPA the certificate was available in the global keystore but for some reason the controller wasn’t picking it up. When keytool prompted to add the key anyway I said yes and the key was installed. From there the documented install method worked just fine.

$ java -jar lib/ace.jar import_cert host.strip.crt ca.strip.crt
$ systemctl restart unifi