Fixing iptables-persistent On a Debian Linode

Over the last couple of years I’ve been a very happy customer of Linode for their low cost and low hassle Linux VPS boxes. More recently I had made the move from a single monolithic server hosting mail, databases, and web applications to a more distributed system where those services were spread across several hosts and then firewalled off from each other save for the minimum required access. That way, for example, my MySQL instance would only accept connections to port 3306 from my webserver on it’s private IP address. Great in theory, except when I ran an nmap scan recently and saw my server’s ports hanging out in the breeze.

User Error: Persistence Is Not Baked In

Part of the decentralization move was to change my hosts over from Arch Linux to Debian Jessie and it’s clear now I didn’t do my homework. I had configured my rules using a bit of Puppet (I’m now working on moving that configuration over to Ansible) but they were not persisting across reboots. I suppose that when I had originally configured things on Arch my memory glossed over the “save the rules” part of the exercise.

Intent upon not making that foolish mistake again, a quick Google search lead me to the iptables-persistent package in Debian. A call to sudo apt-get install iptables-persistent should have had me up and running. Should have…

System Error: Don’t Try to Load Modules That Don’t Exist

One of iptables-persistent dependencies is the package netfilter-persistent but when apt tried to install the package it failed saying that it couldn’t start the service and to check the logs. Pulling out sudo journalctl -xn I poked through the logs and found an entry very similar to this one, copied here from the Debian bug describing the error.

> Oct 17 17:46:56 prod sudo[5176]: puppet : TTY=pts/0 ; PWD=/home/puppet ; USER=root ; COMMAND=/bin/systemctl start netfilter-persistent
> Oct 17 17:46:56 prod sudo[5176]: pam_unix(sudo:session): session opened for user root by puppet(uid=0)
> Oct 17 17:46:56 prod systemd[1]: Starting Load Kernel Modules...
> -- Subject: Unit systemd-modules-load.service has begun with start-up
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit systemd-modules-load.service has begun starting up.
> Oct 17 17:46:56 prod systemd-modules-load[5179]: could not open builtin file '/lib/modules/3.15.4-x86_64-linode45/modules.builtin.bin'
> Oct 17 17:46:56 prod systemd-modules-load[5179]: Failed to find module 'loop'
> Oct 17 17:46:56 prod systemd[1]: systemd-modules-load.service: main process exited, code=exited, status=1/FAILURE
> Oct 17 17:46:56 prod systemd[1]: Failed to start Load Kernel Modules.
> -- Subject: Unit systemd-modules-load.service has failed
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit systemd-modules-load.service has failed.
> --
> -- The result is failed.
> Oct 17 17:46:56 prod systemd[1]: Dependency failed for netfilter persistent configuration.
> -- Subject: Unit netfilter-persistent.service has failed
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit netfilter-persistent.service has failed.
> --
> -- The result is dependency.
> Oct 17 17:46:56 prod sudo[5176]: pam_unix(sudo:session): session closed for user root
> Oct 17 17:46:56 prod systemd[1]: Unit systemd-modules-load.service entered failed state.

From that it was clear that the loop module wasn’t being found for some reason. A quick poke around /lib/modules made it fairly clear that if it existed anywhere it had to be built into the kernel. Another hour later I came to the conclusion that figuring out what modules are built into your kernel when you weren’t the one that did the building is a bit of a tricky prospect. If anyone knows how to pull it off, please let me know, I’m quite curious.

After spending a little more time with Google and the bug report I stumbled across a Linode forum discussion about the same bug but in relation to the fuse module instead of loop. There a user suggested placing fuse in the modules.builtin file inside of /lib/modules and rebuilding the module index using the following commands.

sudo echo "fuse" >> /lib/modules/4.1.5-x86_64-linode61/modules.builtin
sudo touch /lib/modules/4.1.5-x86_64-linode61/modules.order
sudo depmod

Unfortunately, that and several reboots did not give me any more success in getting netfilter-persistent to install properly. It seems that contrary to what I thought, the loop module isn’t built into the custom kernel that the Linodes boot so obviously adding it as a builtin is useless.

Simple Solutions Work

After some more searching, I found someone who had come to the simplest but surprisingly effective solution, just comment out the loop module in /etc/modules. In his post he lays out exactly the same problem I encountered and suggests the removal of the offending module if nothing in your system currently relies on it. A quick comment and reboot later and iptables-persistent is happily installed along with all of its dependencies. Be sure when installing though to tell it to dump your current rule set to a file if you’ve already got it configured. If you don’t you’ll need to write them up and save them to /etc/iptables/rules.v4 for IPv4 and /etc/iptables/rules.v6 for IPv6 so that the service can load and save to them.

Give the server another few reboots and a couple of scans with nmap to make sure everything has taken properly. Finally, sit back to enjoy the improved security of your servers and ponder what problem is next on your list to fix. Don’t kid yourself, there’s always something new to fix.