Docker on an Ancient Platform

TL;DR at the end.

My home has an Odroid C1+, a single-board computer the size of a credit card, based on an ARM CPU and made by Korean company Hardkernel. It’s always on and runs a few essentials like DNS, DHCP, and Samba.

I decided to add two services to it, a photo gallery and a password vault. Installing them natively would be a lot of work — prerequisite software has to be installed, web servers and databases need to be configured. Besides, installation instructions assume heavy stuff like Apache HTTPd and MySQL, probably too heavy for this tiny computer.

It would be so much easier if those services came in the shrink-wrapped form, for example as Docker containers. And indeed, dockerized versions do exist!

However, Docker has certain prerequisites that my current Odroid configuration doesn’t provide. This is so because it runs an old Linux kernel, 3.10 to be specific. To upgrade the kernel, I would have to use ARMbian, which I am not too happy to install, since it is not supported by Hardkernel. For now, I stick to the original Ubuntu 16.04 installed on this server.

So far, I have discovered two problems.

Problem 1: Docker detects Apparmor but can’t use it

The first problem comes up right at the beginning, when I try to run the Docker hello-world container. I get:

docker: Error response from daemon: Could not check if docker-default AppArmor profile was loaded: open /sys/kernel/security/apparmor/profiles: no such file or directory

Yes, it’s true: /sys/kernel/security/apparmor exists, but it has no subdirectory named profiles. When I check the status of Apparmor, I get the same complaint:

$ aa-status
apparmor module is loaded.
Could not open /sys/kernel/security/apparmor/profiles: No such file or directory

After some searching, my verdict is that Apparmor is probably not fully supported by that kernel. To fix the problem, I deinstall it and disable the kernel module by adding apparmor=0 to the kernel parameter list. Now, Docker works.

Problem 2: Spotty network access to containers

The next problem is networking. I install two dockerized applications, Lychee and Bitwarden_RS. They launch all right, but I don’t get network access to them. The browser just times out. However, as soon as I launch tcpdump on the docker bridge, docker0, access is established. But after a few minutes, again timeouts. Strangely, it seems that whenever I try to analyze network traffic with tcpdump, the problem goes away. How can a simple probe impact networking?

I install the containers on a virtual machine running the x64_86 version of Ubuntu 20. No network problems there.

Eventually I discover messages in the Odroid’s kernel log:

$ sudo journalctl -k |grep docker0
Nov 26 19:46:03 odroid kernel: device docker0 left promiscuous mode
Nov 26 19:47:49 odroid kernel: device docker0 entered promiscuous mode
Nov 26 20:16:50 odroid kernel: device docker0 left promiscuous mode
Nov 26 20:16:59 odroid kernel: device docker0 entered promiscuous mode

Whenever the docker0 bridge is promiscuous, I can access the containers. By default, tcpdump puts NICs in promiscuous mode, but I can also do it manually with

ip link set docker0 promisc on

The precise cause of this behaviour is still not clear to me, but after a few web searches I come to the conclusion it must have something to do with the br_netfilter kernel module, which doesn’t exist on Linux 3.10. The Ubuntu 20 virtual machine doesn’t require promiscuity for Docker’s networking to work.

Summary

To make Docker work on older Linux kernels, consider disabling and/or removing Apparmor and make the docker0 bridge promiscuous.

Should I run into more problems, I will update this post.