A Better Intranet Penetration Solution Than Tailscale: Headscale

Tailscale is a virtual networking tool based on WireGuard. It implements the WireGuard protocol in user space, which may result in some performance loss compared to kernel-space WireGuard, but it has made significant efforts in functionality and usability:

• Ready to use
• No firewall configuration required
• No additional configuration needed
• High security/privacy
• Automatic key rotation
• Peer-to-peer connections
• Supports user review of end-to-end access logs
• In addition to existing UDP protocols like ICE and STUN, it implements the DERP TCP protocol for NAT traversal
• The control server based on the public network issues ACLs and configurations, enabling dynamic updates of nodes
• Generates users and private keys through third-party (e.g., Google) SSO services for identity authentication

In short, we can think of Tailscale as a more user-friendly and feature-rich WireGuard.

Having these features is not enough; as someone who enjoys free services, we care more about open source and self-hosting.

Tailscale is a commercial product, but individual users can use it for free as long as they do not connect more than 20 devices (although there are some limitations, such as not being able to customize subnet segments and not being able to set multiple subnets). Except for the graphical applications for Windows and macOS, other Tailscale client components (including the Android client) are developed as open-source projects under the BSD license, and you can find the source code for clients of various operating systems in their GitHub repository^[1].

For most users, using Tailscale for free is sufficient, but if you have higher demands, such as customizing subnet segments, you can choose to pay.

Can I self-host? Yes, but you need to read on.

What is Headscale

The control server for Tailscale is not open-source and has many restrictions for free users, which is understandable as it is their cash cow. Fortunately, there is currently an open-source implementation called Headscale^[2], which is the only one of its kind and we hope it can grow.

Headscale is developed by Juan Font at the European Space Agency using the Go language and is released under the BSD license. It implements all the main functions of the Tailscale control server and can be deployed internally within enterprises, with no limits on the number of devices, and all network traffic is controlled by yourself.

Headscale Deployment

One-Click Deployment with Sealos

If you find the following steps too lengthy, you can choose to directly use the Sealos application template for one-click deployment; all you need is a hand, and no settings are required.

Simply copy the link below and paste it into your browser to open the deployment interface for the Sealos application template:

• https://template.cloud.sealos.io/deploy?templateName=headscale

❝

If this is your first time opening Sealos, you need to register and log in first.

Then click the “Deploy Application” button to start the deployment. Once the deployment is complete, you will see two applications: one is Headscale, and the other is the Headscale Visualization Interface^[3].

Click on the “Details” of the Headscale application to enter the details page. The internal network port 8080 corresponds to the external network address, which is the public domain name of Headscale.

To access the visualization interface, append /admin/ to the public domain name of Headscale.

Deploying on Linux

The deployment steps on Linux are a bit more complex.

❝

In theory, as long as your Headscale service can be exposed to the public network exit, it is fine, but it is best not to have NAT, so it is recommended to deploy Headscale on a cloud host with a public IP.

First, you need to download the latest binary file from its GitHub repository’s Release page.

$ wget --output-document=/usr/local/bin/headscale 
   https://github.com/juanfont/headscale/releases/download/v<HEADSCALE VERSION>/headscale_<HEADSCALE VERSION>_linux_<ARCH>

$ chmod +x /usr/local/bin/headscale

Create the configuration directory:

$ mkdir -p /etc/headscale

Create directories to store data and certificates:

$ mkdir -p /var/lib/headscale

Create an empty SQLite database file:

$ touch /var/lib/headscale/db.sqlite

Create the Headscale configuration file:

$ wget https://github.com/juanfont/headscale/raw/main/config-example.yaml -O /etc/headscale/config.yaml

• Modify the configuration file to change server_url to your public IP or domain name. If it is a domestic server, the domain name must be registered. My domain cannot be registered, so I just used the public IP directly.
• If you do not need DNS functionality temporarily, you can set magic_dns to false.
• Set server_url to http://<PUBLIC_ENDPOINT>:8080, replacing <PUBLIC_ENDPOINT> with your public IP or domain name.
• It is recommended to enable random ports by setting randomize_client_port to true.
• You can customize the private subnet and enable both IPv4 and IPv6:

  ip_prefixes:
    # - fd7a:115c:a1e0::/48
    - 100.64.0.0/16
  

Create the SystemD service configuration file:

# /etc/systemd/system/headscale.service
[Unit]
Description=headscale controller
After=syslog.target
After=network.target

[Service]
Type=simple
User=headscale
Group=headscale
ExecStart=/usr/local/bin/headscale serve
Restart=always
RestartSec=5

# Optional security enhancements
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/headscale /var/run/headscale
AmbientCapabilities=CAP_NET_BIND_SERVICE
RuntimeDirectory=headscale

[Install]
WantedBy=multi-user.target

Create the headscale user:

$ useradd headscale -d /home/headscale -m

Change the owner of the /var/lib/headscale directory:

$ chown -R headscale:headscale /var/lib/headscale

Modify the unix_socket in the configuration file:

unix_socket: /var/run/headscale/headscale.sock

Reload SystemD to load the new configuration file:

$ systemctl daemon-reload

Start the Headscale service and set it to start on boot:

$ systemctl enable --now headscale

Check the running status:

$ systemctl status headscale

Check the occupied ports:

$ ss -tulnp|grep headscale

tcp LISTEN 0 1024 [::]:9090 [::]:* users:(("headscale",pi

 d=10899,fd=13))
tcp LISTEN 0 1024 [::]:50443 [::]:* users:(("headscale",pi

 d=10899,fd=10))
tcp LISTEN 0 1024 [::]:8080 [::]:* users:(("headscale",pi

 d=10899,fd=12))

Create User

Command Line

Tailscale has a concept called tailnet, which you can understand as a tenant. Tenants are isolated from each other. For details, see Tailscale’s official documentation: What is a tailnet^[4]. Headscale also has a similar implementation called user. We need to create a user first to allow subsequent client access, for example:

$ headscale user create default

View the namespace:

$ headscale user list

ID | Name | Created

1 | default | 2022-03-09 06:12:06

If you deployed Headscale using Sealos, you can click the “Terminal” button on the right side of the Headscale application’s detail page to enter the Headscale container’s terminal:

Then execute the above command in the terminal to create the user.

Visualization Interface

Headscale-Admin^[5] requires an API Key to connect to Headscale, so we need to create an API key first. Click the “Terminal” button on the right side of the Headscale application’s detail page to enter the Headscale container’s terminal:

Then execute the following command to create the API Key:

$ headscale apikey create

Fill in the Headscale public domain name and API Key in the settings page of Headscale-Admin, uncheck Legacy API, and then click “Save”:

Once connected successfully, click on the “Users” in the left sidebar, then click “Create” to start creating users:

Tailscale Client Access

Currently, except for the iOS client, other platform clients can customize the Tailscale control server.

Let’s first look at the access process for the Linux platform.

Linux

Tailscale officially provides software packages for various Linux distributions, but you know the network in China; the software sources are simply unusable. Fortunately, the official also provides statically compiled binaries^[7], which we can download directly. For example:

$ wget https://pkgs.tailscale.com/stable/tailscale_1.22.2_amd64.tgz

Extract:

$ tar zxvf tailscale_1.22.2_amd64.tgz
x tailscale_1.22.2_amd64/
x tailscale_1.22.2_amd64/tailscale
x tailscale_1.22.2_amd64/tailscaled
x tailscale_1.22.2_amd64/systemd/
x tailscale_1.22.2_amd64/systemd/tailscaled.defaults
x tailscale_1.22.2_amd64/systemd/tailscaled.service

Copy the binary files to the official package’s default path:

$ cp tailscale_1.22.2_amd64/tailscaled /usr/sbin/tailscaled
$ cp tailscale_1.22.2_amd64/tailscale /usr/bin/tailscale

Copy the systemD service configuration file to the system path:

$ cp tailscale_1.22.2_amd64/systemd/tailscaled.service /lib/systemd/system/tailscaled.service

Copy the environment variable configuration file to the system path:

$ cp tailscale_1.22.2_amd64/systemd/tailscaled.defaults /etc/default/tailscaled

Start the tailscaled.service and set it to start on boot:

$ systemctl enable --now tailscaled

Check the service status:

$ systemctl status tailscaled

Tailscale access Headscale:

# If you are deploying on your own server, replace <HEADSCALE_PUB_ENDPOINT> with your Headscale public IP or domain name
$ tailscale up --login-server=http://<HEADSCALE_PUB_ENDPOINT>:8080 --accept-routes=true --accept-dns=false

# If you are using Sealos for one-click deployment, replace <HEADSCALE_PUB_ENDPOINT> with the public domain name of Headscale mentioned above
$ tailscale up --login-server=https://<HEADSCALE_PUB_ENDPOINT> --accept-routes=true --accept-dns=false

You can also get the access command from the Deploy interface in Headscale-Admin:

It is recommended to turn off the DNS function, as it will override the system’s default DNS. If you need DNS, you can research the official documentation; I will not go into detail here.

After executing the above command, you will see the following information:

To authenticate, visit:

 https://qgemohpy.cloud.sealos.io/register/mkey:e13651ddbfc269513723f1afd6f42465e56922b67ecea8f37d61a35b1b357e0c

Open this link in your browser, and you will see the following interface:

Copy the command from the browser and paste it into the terminal of the machine where Headscale is located, replacing USERNAME with the previously created user.

$ headscale nodes register --user default --key 905cf165204800247fbd33989dbc22be95c987286c45aac3033937041150d846
Machine register

Registration successful, view the registered nodes:

$ headscale nodes list

ID | Name | NodeKey | Namespace | IP addresses | Ephemeral | Last seen | Online | Expired

1 | coredns | [Ew3RB] | default | 100.64.0.1 | false | 2022-03-20 09:08:58 | online | no

Return to the Linux host where the Tailscale client is located, and you can see that Tailscale will automatically create the relevant routing table and iptables rules. You can view the routing table with the following command:

$ ip route show table 52

Check the iptables rules:

$ iptables -S
-P INPUT DROP
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N ts-forward
-N ts-input
-A INPUT -j ts-input
-A FORWARD -j ts-forward
-A ts-forward -i tailscale0 -j MARK --set-xmark 0x40000/0xffffffff
-A ts-forward -m mark --mark 0x40000 -j ACCEPT
-A ts-forward -s 100.64.0.0/10 -o tailscale0 -j DROP
-A ts-forward -o tailscale0 -j ACCEPT
-A ts-input -s 100.64.0.5/32 -i lo -j ACCEPT
-A ts-input -s 100.115.92.0/23 ! -i tailscale0 -j RETURN
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP

$ iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A ts-postrouting -m mark --mark 0x40000 -j MASQUERADE

macOS

There are 3 installation methods for macOS:

• Install directly from the App Store at: https://apps.apple.com/ca/app/tailscale/id1475387142^[8]. The prerequisite is that you need a US ID…
• Download the installation package^[9] and install it directly, bypassing the App Store.
• Install the open-source command-line tools tailscale and tailscaled. Related link: https://github.com/tailscale/tailscale/wiki/Tailscaled-on-macOS^[10].

The core data packet processing code of these three installation packages is the same; the only difference lies in the packaging method and interaction with the system.

Applications in the App Store run in a sandbox^[11], isolated from other parts of the system. In the sandbox, the application can be a network extension^[12], enabling VPN or VPN-like functionality. The functionality implemented by network extensions cannot be effective for applications outside the App Store.

Starting from macOS 10.15, system extensions^[13] have been added, which are essentially user-space kernel extensions that enhance many features compared to traditional network extensions, such as content filtering, transparent proxies, DNS proxies, etc. The installation package independent of the App Store uses system extensions, distributed via DMG or zip archives.

{{< alert >}}
Do not install both the App Store version and the independently distributed version at the same time; only one can be installed at a time.
{{< /alert >}}

The command-line tools do not use network extensions or system extensions; instead, they use the utun interface^[14], which lacks some functionalities compared to the GUI version, such as MagicDNS and Taildrop.

Overview:

After installing the GUI version of the application, you need to perform some additional operations to allow Tailscale to use Headscale as the control server. Of course, Headscale has provided us with detailed operation steps; you just need to open the URL in your browser: https://<HEADSCALE_PUB_ENDPOINT>/apple, and the following interface will appear:

For Tailscale versions 1.34.0 and above, you can operate as follows:

1. Hold down the “ALT” key, then click the Tailscale icon in the top menu bar, and hover the mouse pointer over the “Debug” menu.

   

2. Select “Add Account…” under “Custom Login Server”.

3. In the pop-up window, enter the public domain name of Headscale, then click “Add Account”.

   

4. You will immediately be redirected to a browser and a page will open.

   

5. Then, as with the Linux client, go back to the machine where Headscale is located and execute the command in the browser to register successfully:

   

Return to the machine where Headscale is located, and check the registered nodes:

$ headscale nodes list

ID | Name | NodeKey | Namespace | IP addresses | Ephemeral | Last seen | Online | Expired

1 | coredns | [Ew3RB] | default | 100.64.0.1 | false | 2022-03-20 09:08:58 | online | no
2 | carsondemacbook-pro | [k7bzX] | default   | 100.64.0.2     | false     | 2022-03-20 09:48:30 | online  | no

Back on macOS, test whether you can ping the other node:

$ ping -c 2 100.64.0.1
PING 100.64.0.1 (100.64.0.1): 56 data bytes
64 bytes from 100.64.0.1: icmp_seq=0 ttl=64 time=37.025 ms
64 bytes from 100.64.0.1: icmp_seq=1 ttl=64 time=38.181 ms

--- 100.64.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 37.025/37.603/38.181/0.578 ms

You can also use the Tailscale CLI to test:

$ /Applications/Tailscale.app/Contents/MacOS/Tailscale ping 100.64.0.1
pong from coredns (100.64.0.1) via xxxx:41641 in 36ms

For Tailscale clients with version numbers lower than 1.32.0, you only need to follow the steps described in the image; I will not elaborate further.

Android

The Android client has supported custom control servers (i.e., coordination servers) since version 1.30.0. You can download the latest version of the client from Google Play^[19] or F-Droid^[20].

After installation, open the Tailscale App, and you will see the following interface:

Click the three dots in the upper right corner, and you will only see an About option:

Next, you will need to perform some operations repeatedly; you need to keep tapping and closing the three dots in the upper right corner. After repeating this three or four times, a Change server option will appear:

Click Change server and fill in the address of the headscale control server:

Then click Save and restart to restart, click Sign in with other, and you will see this page:

Paste the command into the terminal of the machine where Headscale is located, replacing USER with the user you created earlier, and execute the command. After successful registration, you can close this page and return to the App homepage, as shown in the picture:

Windows

To use Headscale as the control server for the Windows Tailscale client, simply open the URL in your browser: https://<HEADSCALE_PUB_ENDPOINT>/windows, and the following interface will appear:

Follow the steps in the interface to complete the setup.

Other Linux Distributions

In addition to regular Linux distributions, there are some special Linux distributions for specific scenarios, such as OpenWrt, QNAP, Synology, etc. The installation methods for these distributions have been documented by others, so I will not describe them in detail here. I will just provide the relevant GitHub repositories, and if you have specific needs, you can check the documentation in those repositories.

• OpenWrt: https://github.com/adyanth/openwrt-tailscale-enabler^[21]
• Synology: https://github.com/tailscale/tailscale-synology^[22]
• QNAP: https://github.com/tailscale/tailscale-qpkg^[23]

iOS

The iOS system can be installed directly from the App Store, of course, provided you have a US ID.

1. After installation, open Tailscale to confirm that you are not logged into any account. Then go to “Settings”, scroll down, find “Tailscale” under “Game Center” or “TV Provider”, and click in.

   

   If your device has previously logged into the Tailscale server, you need to enable the “Reset Keychain” option.

2. Enter your Headscale public domain name under “Alternate Coordination Server URL”.

3. Close Tailscale from the iOS app switcher and reopen it, then select “Log in”, which will pop up a Headscale authentication page.

   

4. Copy the command from the Headscale authentication page into the terminal of the Headscale container, replacing USERNAME with the user you created earlier.

   $ headscale nodes register --user default --key mkey:1fbd9696ebb03b9394033949514345bc5dba0e570bc0d778f15f92a02d2dcb66
   2023-12-29T09:55:38Z TRC DNS configuration loaded dns_config={"Nameservers":["1.1.1.1"],"Proxied":true,"Resolvers":[{"Addr":"1.1.1.1"}]}
   Node localhost registered
   

5. Registration successful.

   

Access via Pre-Authkeys

All the previous access methods require server consent, which can be cumbersome. In fact, there is a simpler way to access directly without server consent.

First, generate a pre-authkey token on the server, with an expiration time that can be set to 24 hours:

$ headscale preauthkeys create -e 24h --user default

View the generated keys:

$ headscale --user default preauthkeys list
ID | Key                                              | Reusable | Ephemeral | Used  | Expiration          | Created            
1  | 57e419c40e30b0dxxxxxxxf15562c18a8c6xxxx28ae76f57 | false    | false     | false | 2022-05-30 07:14:17 | 2022-05-29 07:14:17

You can also generate it in Headscale-Admin. Click on the User you want the client to join:

In the pop-up interface, click “PreAuth Keys” on the right side and click Create, set an expiration time (e.g., 100 years~), and if you want to reuse this key, check Reusable, then click ✅:

Once created, you can copy the PreAuth Key from the highlighted area:

Now the new node can connect directly without server consent:

$ tailscale up --login-server=http://<HEADSCALE_PUB_ENDPOINT>:8080 --accept-routes=true --accept-dns=false --authkey $KEY

Connecting LANs

So far, we have only created a point-to-point mesh network where each node can connect directly via WireGuard’s private network IP. But we can be bolder; remember I mentioned accessing resources on the home intranet at the beginning of the article? We can configure it appropriately to allow each node to access other nodes’ LAN IPs. This use case is quite extensive; you can directly access the NAS at home or any service on the intranet. Advanced users can use this method to access the Pod IPs and Service IPs of Kubernetes clusters in the cloud.

Suppose your home intranet has a Linux host (like OpenWrt) with the Tailscale client installed, and we want other Tailscale clients to access any device on the home intranet directly through the home LAN IP (for example, 192.168.100.0/24).

The configuration method is simple. First, you need to set IPv4 and IPv6 routing forwarding:

$ echo 'net.ipv4.ip_forward = 1' | tee /etc/sysctl.d/ipforwarding.conf
$ echo 'net.ipv6.conf.all.forwarding = 1' | tee -a /etc/sysctl.d/ipforwarding.conf
$ sysctl -p /etc/sysctl.d/ipforwarding.conf

Modify the command for registering the node to add the parameter --advertise-routes=192.168.100.0/24, telling the Headscale server, “This node can forward the routes of these addresses.”

$ tailscale up --login-server=http://<HEADSCALE_PUB_ENDPOINT>:8080 --accept-routes=true --accept-dns=false --advertise-routes=192.168.100.0/24 --reset

On the Headscale side, check the routes, and you will see that the related routes are disabled.

$ headscale nodes list|grep openwrt

6 | openwrt | [7LdVc] | default | 100.64.0.6 | false | 2022-03-20 15:50:46 | online | no

$ headscale routes list -i 6

Route | Enabled

192.168.100.0/24 | false

Enable the route:

$ headscale routes enable -i 6 -r "192.168.100.0/24"

Route | Enabled

192.168.100.0/24 | true

If you have multiple routes, separate them with ,:

$ headscale routes enable -i 6 -r "192.168.100.0/24,xxxx"

You can also enable all routes using the -a parameter:

$ headscale routes enable -i 6 -a

Other nodes can check the routing results:

$ ip route show table 52|grep "192.168.100.0/24"
192.168.100.0/24 dev tailscale0

Other nodes need to add the --accept-routes=true option when starting to declare, “I accept routes published by other external nodes.”

Now, you can ping any machine at home from any node where the Tailscale client is located. You can access any device in your home just like at home, whether in the office or Starbucks. Isn’t that great?

Conclusion

Currently, in terms of stability, Tailscale is slightly better than other networking tools that use WireGuard, as it generally does not experience ping failures. This is due to Tailscale’s various optimizations for NAT traversal in user space. They even wrote an article explaining NAT traversal principles^[24], which has been translated into Chinese by domestic eBPF expert Zhao Yanan. I highly recommend everyone to read it. Here’s a picture to give you a feel:

References

Previous Recommendations