Installing Docker, Tailscale, K3s, and Cilium on Rockchip Arm Boards

Overview

I bought several cheap Purple PI OH development boards^[1] (I got 3 for just over 500 yuan🤑). This development board is similar to the Raspberry Pi and is based on the Rockchip (瑞芯微) rx3566 arm64 chip. Here it is:

Installing Docker, Tailscale, K3s, and Cilium on Rockchip Arm Boards — Purple PI OH

I bought it to use as a home server or home lab. The main considerations were:

1.Cheap2.High playability3.Low power consumption4.Minimal heat, quiet operation5.The Arm64 ecosystem is still decent

The configuration is as follows (selected specifications):

•SOC: RockChip RK3566•CPU: Quad-core 64-bit Cortex-A55 processor, with a maximum frequency of 1.8GHz•Supports up to 8GB high-speed LPDDR4, with a rate of up to 1066Mbps (I bought the 2G version)•Storage: eMMC defaults to 8GB (options for 16GB/32GB/64GB) (I bought the 16G version)•1 HDMI2.0 port supporting 4K@60Hz or 1080P@120Hz•1 adaptive gigabit Ethernet port•WIFI Bluetooth wireless communication•Onboard 1 USB3.0 port, 3 USB2.0 ports•Very small PCBA size, 85mm*56mm

Supported systems:

•Android 11•Debian 10•Buildroot + QT•HarmonyOS OpenHarmony3.2•Ubuntu•Kylin OS

After flashing the official Debian 10, I found that I couldn’t install Docker/Tailscale/K3s/Cilium, why is that?

Reason Analysis

Many development board operating systems do not have UEFI and cannot directly use the ISO installation media provided by the Debian Linux official website. Instead, they use a self-compiled Debian Linux.

The compiled Debian will include uboot as the bootloader and integrate the corresponding chip/interface drivers for the development board.

The officially provided self-compiled Debian Linux only enables a small portion of the kernel parameters. However, Docker/Tailscale/K3s/Cilium require functionalities that are tightly related to the kernel, but these functionalities are not available in the officially provided Debian Linux. Therefore, we need to compile it ourselves according to the official “Linux SDK Compilation Manual”.

What Kernel Configurations Are Needed

Kernel Configurations Required for Docker

The kernel configurations required for Docker can be checked using the script at check-config.sh. The running example is as follows:

# ./check_config.shinfo: reading kernel config from ./kernel/.config ...
Generally Necessary:- cgroup hierarchy: properly mounted [/sys/fs/cgroup]- apparmor: enabled and tools installed- CONFIG_NAMESPACES: enabled- CONFIG_NET_NS: enabled- CONFIG_PID_NS: enabled- CONFIG_IPC_NS: enabled- CONFIG_UTS_NS: enabled- CONFIG_CGROUPS: enabled- CONFIG_CGROUP_CPUACCT: enabled- CONFIG_CGROUP_DEVICE: enabled- CONFIG_CGROUP_FREEZER: enabled- CONFIG_CGROUP_SCHED: enabled- CONFIG_CPUSETS: enabled- CONFIG_MEMCG: missing- CONFIG_KEYS: enabled- CONFIG_VETH: missing- CONFIG_BRIDGE: missing- CONFIG_BRIDGE_NETFILTER: missing- CONFIG_IP_NF_FILTER: missing- CONFIG_IP_NF_TARGET_MASQUERADE: missing- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: missing- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: missing- CONFIG_NETFILTER_XT_MATCH_IPVS: missing- CONFIG_IP_NF_NAT: missing- CONFIG_NF_NAT: missing- CONFIG_POSIX_MQUEUE: missing
Optional Features:- CONFIG_USER_NS: enabled- CONFIG_SECCOMP: enabled- CONFIG_SECCOMP_FILTER: enabled- CONFIG_CGROUP_PIDS: missing- CONFIG_MEMCG_SWAP: missing- CONFIG_MEMCG_SWAP_ENABLED: missing- CONFIG_BLK_CGROUP: missing- CONFIG_BLK_DEV_THROTTLING: missing- CONFIG_CGROUP_PERF: missing- CONFIG_CGROUP_HUGETLB: missing- CONFIG_NET_CLS_CGROUP: missing- CONFIG_CGROUP_NET_PRIO: missing- CONFIG_CFS_BANDWIDTH: enabled- CONFIG_FAIR_GROUP_SCHED: enabled- CONFIG_RT_GROUP_SCHED: missing- CONFIG_IP_NF_TARGET_REDIRECT: missing- CONFIG_IP_VS: missing- CONFIG_IP_VS_NFCT: missing- CONFIG_IP_VS_PROTO_TCP: missing- CONFIG_IP_VS_PROTO_UDP: missing- CONFIG_IP_VS_RR: missing- CONFIG_SECURITY_SELINUX: missing- CONFIG_SECURITY_APPARMOR: missing

Therefore, the required kernel config for Docker is:

# Docker Generally NecessaryCONFIG_NAMESPACES=yCONFIG_NET_NS=yCONFIG_PID_NS=yCONFIG_IPC_NS=yCONFIG_UTS_NS=yCONFIG_CGROUPS=yCONFIG_CGROUP_CPUACCT=yCONFIG_CGROUP_DEVICE=yCONFIG_CGROUP_FREEZER=yCONFIG_CGROUP_SCHED=yCONFIG_CPUSETS=yCONFIG_MEMCG=yCONFIG_KEYS=yCONFIG_VETH=yCONFIG_BRIDGE=yCONFIG_BRIDGE_NETFILTER=yCONFIG_IP_NF_FILTER=yCONFIG_IP_NF_TARGET_MASQUERADE=yCONFIG_NETFILTER_XT_MATCH_ADDRTYPE=yCONFIG_NETFILTER_XT_MATCH_CONNTRACK=yCONFIG_NETFILTER_XT_MATCH_IPVS=yCONFIG_IP_NF_NAT=yCONFIG_NF_NAT=yCONFIG_POSIX_MQUEUE=y
# Optional Features:=yCONFIG_USER_NS=yCONFIG_SECCOMP=yCONFIG_SECCOMP_FILTER=yCONFIG_CGROUP_PIDS=yCONFIG_MEMCG_SWAP=yCONFIG_MEMCG_SWAP_ENABLED=yCONFIG_BLK_CGROUP=yCONFIG_BLK_DEV_THROTTLING=yCONFIG_CGROUP_PERF=yCONFIG_CGROUP_HUGETLB=yCONFIG_NET_CLS_CGROUP=yCONFIG_CGROUP_NET_PRIO=yCONFIG_CFS_BANDWIDTH=yCONFIG_FAIR_GROUP_SCHED=yCONFIG_RT_GROUP_SCHED=yCONFIG_IP_NF_TARGET_REDIRECT=yCONFIG_IP_VS=yCONFIG_IP_VS_NFCT=yCONFIG_IP_VS_PROTO_TCP=yCONFIG_IP_VS_PROTO_UDP=yCONFIG_IP_VS_RR=yCONFIG_SECURITY_SELINUX=yCONFIG_SECURITY_APPARMOR=y

Kernel Configurations Required for Tailscale

Tailscale is primarily based on user-space implementations for broad applicability and can run without any kernel configurations using a sock5 proxy. However, to run in a normal state, it only relies on one kernel configuration:

# TailscaleCONFIG_TUN=y

If you are using Wireguard or other software that relies heavily on the kernel, please check the relevant kernel configuration requirements yourself.

Kernel Configurations Required for K3s

K3s has a well-designed CLI that can directly check the kernel configuration requirements. On my compiled Debian 10, it runs as follows:

$ k3s check-config
Verifying binaries in /var/lib/rancher/k3s/data/ef31d9f1b153134534c2b9664540479f3071940e08ee95dd2877e102a31d235e/bin:- sha256sum: good- aux/ip6tables: symlink to xtables-legacy-multi- aux/ip6tables-restore: symlink to xtables-legacy-multi- aux/ip6tables-save: symlink to xtables-legacy-multi- aux/iptables: symlink to xtables-legacy-multi- aux/iptables-restore: symlink to xtables-legacy-multi- aux/iptables-save: symlink to xtables-legacy-multi- links: good
System:- /var/lib/rancher/k3s/data/ef31d9f1b153134534c2b9664540479f3071940e08ee95dd2877e102a31d235e/bin/aux iptables v1.8.8 (legacy): ok- swap: disabled- routes: ok
Limits:- /proc/sys/kernel/keys/root_maxkeys: 1000000
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.232/modules.dep.bin'modprobe: FATAL: Module configs not found in directory /lib/modules/4.19.232info: reading kernel config from /proc/config.gz ...
Generally Necessary:- cgroup hierarchy: cgroups Hybrid mounted, cpuset|memory controllers status: good- CONFIG_NAMESPACES: enabled- CONFIG_NET_NS: enabled- CONFIG_PID_NS: enabled- CONFIG_IPC_NS: enabled- CONFIG_UTS_NS: enabled- CONFIG_CGROUPS: enabled- CONFIG_CGROUP_PIDS: enabled- CONFIG_CGROUP_CPUACCT: enabled- CONFIG_CGROUP_DEVICE: enabled- CONFIG_CGROUP_FREEZER: enabled- CONFIG_CGROUP_SCHED: enabled- CONFIG_CPUSETS: enabled- CONFIG_MEMCG: enabled- CONFIG_KEYS: enabled- CONFIG_VETH: enabled- CONFIG_BRIDGE: enabled- CONFIG_BRIDGE_NETFILTER: enabled- CONFIG_IP_NF_FILTER: enabled- CONFIG_IP_NF_TARGET_MASQUERADE: enabled- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled- CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled- CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled- CONFIG_IP_NF_NAT: enabled- CONFIG_NF_NAT: enabled- CONFIG_POSIX_MQUEUE: enabled
Optional Features:- CONFIG_USER_NS: enabled- CONFIG_SECCOMP: enabled- CONFIG_BLK_CGROUP: enabled- CONFIG_BLK_DEV_THROTTLING: enabled- CONFIG_CGROUP_PERF: enabled- CONFIG_CGROUP_HUGETLB: enabled- CONFIG_NET_CLS_CGROUP: enabled- CONFIG_CGROUP_NET_PRIO: enabled- CONFIG_CFS_BANDWIDTH: enabled- CONFIG_FAIR_GROUP_SCHED: enabled- CONFIG_RT_GROUP_SCHED: enabled- CONFIG_IP_NF_TARGET_REDIRECT: enabled- CONFIG_IP_SET: enabled- CONFIG_IP_VS: enabled- CONFIG_IP_VS_NFCT: enabled- CONFIG_IP_VS_PROTO_TCP: enabled- CONFIG_IP_VS_PROTO_UDP: enabled- CONFIG_IP_VS_RR: enabled- CONFIG_EXT4_FS: enabled- CONFIG_EXT4_FS_POSIX_ACL: enabled- CONFIG_EXT4_FS_SECURITY: enabled- Network Drivers:  - "overlay":    - CONFIG_VXLAN: enabled      Optional (for encrypted networks):      - CONFIG_CRYPTO: enabled      - CONFIG_CRYPTO_AEAD: enabled      - CONFIG_CRYPTO_GCM: enabled      - CONFIG_CRYPTO_SEQIV: enabled      - CONFIG_CRYPTO_GHASH: enabled      - CONFIG_XFRM: enabled      - CONFIG_XFRM_USER: enabled      - CONFIG_XFRM_ALGO: enabled      - CONFIG_INET_ESP: enabled      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled- Storage Drivers:  - "overlay":    - CONFIG_OVERLAY_FS: enabled
STATUS: pass

Based on the above output, the kernel configurations required for K3s are:

# K3s Generally Necessary:CONFIG_NAMESPACES=yCONFIG_NET_NS=yCONFIG_PID_NS=yCONFIG_IPC_NS=yCONFIG_UTS_NS=yCONFIG_CGROUPS=yCONFIG_CGROUP_PIDS=yCONFIG_CGROUP_CPUACCT=yCONFIG_CGROUP_DEVICE=yCONFIG_CGROUP_FREEZER=yCONFIG_CGROUP_SCHED=yCONFIG_CPUSETS=yCONFIG_MEMCG=yCONFIG_KEYS=yCONFIG_VETH=yCONFIG_BRIDGE=yCONFIG_BRIDGE_NETFILTER=yCONFIG_IP_NF_FILTER=yCONFIG_IP_NF_TARGET_MASQUERADE=yCONFIG_NETFILTER_XT_MATCH_ADDRTYPE=yCONFIG_NETFILTER_XT_MATCH_CONNTRACK=yCONFIG_NETFILTER_XT_MATCH_IPVS=yCONFIG_NETFILTER_XT_MATCH_COMMENT=yCONFIG_NETFILTER_XT_MATCH_MULTIPORT=yCONFIG_IP_NF_NAT=yCONFIG_NF_NAT=yCONFIG_POSIX_MQUEUE=y
# Optional Features:CONFIG_USER_NS=yCONFIG_SECCOMP=yCONFIG_BLK_CGROUP=yCONFIG_BLK_DEV_THROTTLING=yCONFIG_CGROUP_PERF=yCONFIG_CGROUP_HUGETLB=yCONFIG_NET_CLS_CGROUP=yCONFIG_CGROUP_NET_PRIO=yCONFIG_CFS_BANDWIDTH=yCONFIG_FAIR_GROUP_SCHED=yCONFIG_RT_GROUP_SCHED=yCONFIG_IP_NF_TARGET_REDIRECT=yCONFIG_IP_SET=yCONFIG_IP_VS=yCONFIG_IP_VS_NFCT=yCONFIG_IP_VS_PROTO_TCP=yCONFIG_IP_VS_PROTO_UDP=yCONFIG_IP_VS_RR=yCONFIG_EXT4_FS=yCONFIG_EXT4_FS_POSIX_ACL=yCONFIG_EXT4_FS_SECURITY=y# Network DriversCONFIG_VXLAN=y# Optional (for encrypted networks):CONFIG_CRYPTO=yCONFIG_CRYPTO_AEAD=yCONFIG_CRYPTO_GCM=yCONFIG_CRYPTO_SEQIV=yCONFIG_CRYPTO_GHASH=yCONFIG_XFRM=yCONFIG_XFRM_USER=yCONFIG_XFRM_ALGO=yCONFIG_INET_ESP=yCONFIG_INET_XFRM_MODE_TRANSPORT=y# Storage DriversCONFIG_OVERLAY_FS=y

Compared to Docker, K3s has the following additional kernel requirements:

•Overlay network•Overlay storage

Kernel Configurations Required for Cilium

The kernel configurations required for Cilium can be found here: System Requirements — Cilium 1.13.4 documentation^[2]

# Cilium Base RequirementsCONFIG_BPF=yCONFIG_BPF_SYSCALL=yCONFIG_NET_CLS_BPF=yCONFIG_BPF_JIT=yCONFIG_NET_CLS_ACT=yCONFIG_NET_SCH_INGRESS=yCONFIG_CRYPTO_SHA1=yCONFIG_CRYPTO_USER_API_HASH=yCONFIG_CGROUPS=yCONFIG_CGROUP_BPF=yCONFIG_PERF_EVENTS=y# Optional: Iptables-based MasqueradingCONFIG_NETFILTER_XT_SET=mCONFIG_IP_SET=mCONFIG_IP_SET_HASH_IP=m# Optional: L7 and FQDN PoliciesCONFIG_NETFILTER_XT_TARGET_TPROXY=mCONFIG_NETFILTER_XT_TARGET_CT=mCONFIG_NETFILTER_XT_MATCH_MARK=mCONFIG_NETFILTER_XT_MATCH_SOCKET=m# Optional: IPSecCONFIG_XFRM=yCONFIG_XFRM_OFFLOAD=yCONFIG_XFRM_STATISTICS=yCONFIG_XFRM_ALGO=mCONFIG_XFRM_USER=mCONFIG_INET{,6}_ESP=mCONFIG_INET{,6}_IPCOMP=mCONFIG_INET{,6}_XFRM_TUNNEL=mCONFIG_INET{,6}_TUNNEL=mCONFIG_INET_XFRM_MODE_TUNNEL=mCONFIG_CRYPTO_AEAD=mCONFIG_CRYPTO_AEAD2=mCONFIG_CRYPTO_GCM=mCONFIG_CRYPTO_SEQIV=mCONFIG_CRYPTO_CBC=mCONFIG_CRYPTO_HMAC=mCONFIG_CRYPTO_SHA256=mCONFIG_CRYPTO_AES=m# Optional: Bandwidth ManagerCONFIG_NET_SCH_FQ=m

Additionally, Cilium has strict requirements for the kernel version, as some of its functionalities depend on a higher version of the kernel. Please check accordingly.

Modifying Kernel Configurations and Compiling

The Linux SDK is generally provided, and the directory structure for the Linux SDK files based on the Rockchip chip series is largely similar. Here, I take the Purple Pi OH as an example, and I believe if you have a related need and are using other Rockchip development boards, you can quickly find the corresponding directory.

The directory for kernel configurations is generally: xxxxxxx\kernel\arch\arm64\configs\xxxxx_linux_defconfig

Add the provided kernel configurations to the end of that file (be sure to avoid duplicates) and save it.

Then compile directly:

./build.sh kernel

After compiling, the boot.img image will be generated in the ./rockdev directory.

Then use the Rockchip provided flashing tool: RKDevTool.exe, to flash the boot.img separately.

Compiling really consumes resources, time, CPU, and disk space 😂

Possible Exceptions

Partition Exceeded

During the compilation process, I encountered an error stating that the recovery partition exceeded the limit, as shown below:

rk356x_linux_sdk/buildroot/output/rockchip_rk356x_recovery/images/recovery.img's size exceed parameter.txt's limit!ERROR: Running build_firmware failed!

The solution is to modify the partition size, specifically by modifying the corresponding parameter.txt file. After modifying, recompile completely:

./build.sh

And re-flash the entire firmware.

Docker Fails to Start After Installation

According to the official documentation: Install Docker Engine on Debian | Docker Documentation^[3], Docker fails to start after installation.

The official documentation clearly states that the latest version requires the following Debian versions:

Debian Bookworm 12 (stable) Debian Bullseye 11 (oldstable)

It may be due to compatibility issues with Debian 10. However, there is a workaround.

The specific error is related to nftables.

The solution is:

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy

After rebooting, Docker can run normally.

Conclusion

After a weekend of playing with the Rockchip arm64 development board, I can finally run Docker, K3s, Tailscale, and Cilium on the 2G memory board. 🎉🎉🎉

As I mentioned before, Arm development boards have the following advantages:

1.Cheap2.High playability3.Low power consumption4.Minimal heat, quiet operation5.The Arm64 ecosystem is still decent

However, compared to x86, these Arm development boards have a poor ecosystem, lacking BIOS/UEFI. Installing the aforementioned software requires adjusting kernel parameters and then compiling, which is quite exhausting.

x86 also has these advantages:

1.Cheap2.High playability

But the x86 ecosystem is much better, making it easier to play around with.

Compared to x86, Arm is actually only stronger in terms of low power consumption and minimal heat. However, some Intel chips, such as the N100, have a TDP of 6W, which is also very low, and can be kept silent through passive cooling. The advantages of Arm are diminishing.

Therefore, my suggestion is: if you want a hassle-free, quiet, and low-power option, the x86 mini host is the first choice; if you want to tinker, be quiet, and have low power consumption, various Arm Pi development boards or boxes are recommended; if you don’t care about power consumption, noise, or high specifications, you can try out second-hand x86 servers. 😂😂😂

That’s all.

📚️Reference Documents

•runc/script/check-config.sh at main · opencontainers/runc (github.com)^[4]•System Requirements — Cilium 1.13.4 documentation^[5]•Install Docker Engine on Debian | Docker Documentation^[6]•Userspace networking mode (for containers) · Tailscale^[7]•【Technical Sharing】RK356X Debian/Ubuntu System Installation Docker (qq.com)•Purple-Pi-OH Linux SDK Compilation Manual (yuque.com)^[8]•Purple-Pi-OH Android/Linux Firmware and Flashing Manual (yuque.com)^[9]•Purple Pi OH – Shenzhen Touch Intelligent Technology Co., Ltd. (industio.cn)^[10]•RK ParameterTool_v1.2 Partition Tool Usage^[11]

References

[1] Purple PI OH development board: http://www.industio.cn/product-item-37.html[2] System Requirements — Cilium 1.13.4 documentation: https://docs.cilium.io/en/stable/operations/system_requirements/#linux-kernel[3] Install Docker Engine on Debian | Docker Documentation: https://docs.docker.com/engine/install/debian/[4] runc/script/check-config.sh at main · opencontainers/runc (github.com): https://github.com/opencontainers/runc/blob/main/script/check-config.sh[5] System Requirements — Cilium 1.13.4 documentation: https://docs.cilium.io/en/stable/operations/system_requirements/#linux-kernel[6] Install Docker Engine on Debian | Docker Documentation: https://docs.docker.com/engine/install/debian/[7] Userspace networking mode (for containers) · Tailscale: https://tailscale.com/kb/1112/userspace-networking/[8] Purple-Pi-OH Linux SDK Compilation Manual (yuque.com): https://industio.yuque.com/industio/gyzv1h/dkboqc87if6m91ad?singleDoc#zZ8PT[9] Purple-Pi-OH Android/Linux Firmware and Flashing Manual (yuque.com): https://industio.yuque.com/industio/gyzv1h/eozksh14txdyb3mr?singleDoc#%20%E3%80%8APurple-Pi-OH%20RK3566%20-%20%E5%9B%BA%E4%BB%B6%E5%8F%8A%E7%83%A7%E5%BD%95%E8%AF%B4%E6%98%8E%E3%80%8B[10] Purple Pi OH – Shenzhen Touch Intelligent Technology Co., Ltd. (industio.cn): http://www.industio.cn/product-item-37.html[11] RK ParameterTool_v1.2 Partition Tool Usage: https://note.youdao.com/ynoteshare/index.html?id=e821b67d0e86a883784be34cfc26231f&type=note&_time=1689754806594