Over the last couple of years I’ve been a very happy customer of Linode for their low cost and low hassle Linux VPS boxes. More recently I had made the move from a single monolithic server hosting mail, databases, and web applications to a more distributed system where those services were spread across several hosts and then firewalled off from each other save for the minimum required access. That way, for example, my MySQL instance would only accept connections to port 3306 from my webserver on it’s private IP address. Great in theory, except when I ran an nmap scan recently and saw my server’s ports hanging out in the breeze.
User Error: Persistence Is Not Baked In
Part of the decentralization move was to change my hosts over from Arch Linux to Debian Jessie and it’s clear now I didn’t do my homework. I had configured my rules using a bit of Puppet (I’m now working on moving that configuration over to Ansible) but they were not persisting across reboots. I suppose that when I had originally configured things on Arch my memory glossed over the “save the rules” part of the exercise.
Intent upon not making that foolish mistake again, a quick Google search lead
me to the
iptables-persistent package in Debian. A call to
install iptables-persistent should have had me up and running. Should have…
System Error: Don’t Try to Load Modules That Don’t Exist
iptables-persistent dependencies is the package
but when apt tried to install the package it failed saying that it couldn’t
start the service and to check the logs. Pulling out
sudo journalctl -xn I
poked through the logs and found an entry very similar to this one, copied here
from the Debian bug describing the error.
> Oct 17 17:46:56 prod sudo: puppet : TTY=pts/0 ; PWD=/home/puppet ; USER=root ; COMMAND=/bin/systemctl start netfilter-persistent > Oct 17 17:46:56 prod sudo: pam_unix(sudo:session): session opened for user root by puppet(uid=0) > Oct 17 17:46:56 prod systemd: Starting Load Kernel Modules... > -- Subject: Unit systemd-modules-load.service has begun with start-up > -- Defined-By: systemd > -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel > -- > -- Unit systemd-modules-load.service has begun starting up. > Oct 17 17:46:56 prod systemd-modules-load: could not open builtin file '/lib/modules/3.15.4-x86_64-linode45/modules.builtin.bin' > Oct 17 17:46:56 prod systemd-modules-load: Failed to find module 'loop' > Oct 17 17:46:56 prod systemd: systemd-modules-load.service: main process exited, code=exited, status=1/FAILURE > Oct 17 17:46:56 prod systemd: Failed to start Load Kernel Modules. > -- Subject: Unit systemd-modules-load.service has failed > -- Defined-By: systemd > -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel > -- > -- Unit systemd-modules-load.service has failed. > -- > -- The result is failed. > Oct 17 17:46:56 prod systemd: Dependency failed for netfilter persistent configuration. > -- Subject: Unit netfilter-persistent.service has failed > -- Defined-By: systemd > -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel > -- > -- Unit netfilter-persistent.service has failed. > -- > -- The result is dependency. > Oct 17 17:46:56 prod sudo: pam_unix(sudo:session): session closed for user root > Oct 17 17:46:56 prod systemd: Unit systemd-modules-load.service entered failed state.
From that it was clear that the
loop module wasn’t being found for some
reason. A quick poke around
/lib/modules made it fairly clear that if it
existed anywhere it had to be built into the kernel. Another hour later I came
to the conclusion that figuring out what modules are built into your kernel when
you weren’t the one that did the building is a bit of a tricky prospect. If
anyone knows how to pull it off, please let me know, I’m quite curious.
After spending a little more time with Google and the bug report I stumbled
across a Linode forum discussion about the same bug but
in relation to the
fuse module instead of
loop. There a user suggested
fuse in the
modules.builtin file inside of
rebuilding the module index using the following commands.
sudo echo "fuse" >> /lib/modules/4.1.5-x86_64-linode61/modules.builtin sudo touch /lib/modules/4.1.5-x86_64-linode61/modules.order sudo depmod
Unfortunately, that and several reboots did not give me any more success in
netfilter-persistent to install properly. It seems that contrary to
what I thought, the
loop module isn’t built into the custom kernel that the
Linodes boot so obviously adding it as a builtin is useless.
Simple Solutions Work
After some more searching, I found someone who had come to the simplest but
surprisingly effective solution, just comment out the
loop module in
/etc/modules. In his post he lays out exactly the same
problem I encountered and suggests the removal of the offending module if
nothing in your system currently relies on it. A quick comment and reboot later
iptables-persistent is happily installed along with all of its
dependencies. Be sure when installing though to tell it to dump your current
rule set to a file if you’ve already got it configured. If you don’t you’ll
need to write them up and save them to
/etc/iptables/rules.v4 for IPv4 and
/etc/iptables/rules.v6 for IPv6 so that the service can load and save to them.
Give the server another few reboots and a couple of scans with nmap to make sure everything has taken properly. Finally, sit back to enjoy the improved security of your servers and ponder what problem is next on your list to fix. Don’t kid yourself, there’s always something new to fix.