default route in "vpn" mode is incorrect on FreeBSD

[dch]

setup

• FreeBSD 12.0-CURRENT amd64 & zerotier 1.2.4 installed

• 1 zt network with allowDefault=0 connecting to a working ZT VPN gateway

• iphone, imac work via this vpn gateway as default route

• connecting successfully to various hosts

• when zerotier-cli set 12345678 allowDefault=1 is run, all network connectivity to/from laptop is lost (no ping in/out from local network nor zt ones, external devices like firewalls also cannot connect

• when this is reversed the system returns to normal

# allowdefault=0 (off/working zt config)
# netstat -rn4

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.16.2.1         UGS       wlan0
10.144.0.0/16      link#3             U      zt1flo98
10.144.49.109      link#3             UHS         lo0
127.0.0.1          link#1             UH          lo0
172.16.2.0/24      link#2             U         wlan0
172.16.2.15        link#2             UHS         lo0

# allowdefault=1 (on/broken zt config)
# netstat -rn4

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
0.0.0.0/1          10.144.0.1         UGS    zt1flo98
default            172.16.2.1         UGS       wlan0
10.144.0.0/16      link#3             U      zt1flo98
10.144.49.109      link#3             UHS         lo0
127.0.0.1          link#1             UH          lo0
128.0.0.0/1        10.144.0.1         UGS    zt1flo98
172.16.2.0/24      link#2             U         wlan0
172.16.2.15        link#2             UHS         lo0

more logs & debugging or remote access available as needed.

[dch]

So today I tried adding the 2 routes that @adamierymenko mentioned in a previous issue, but manually.

zt1flo98dm0peka: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 5000 mtu 2800
	options=80000<LINKSTATE>
	ether 8a:58:82:3c:2f:91
	hwaddr 00:bd:75:ec:52:09
	inet 10.244.23.143 netmask 0xffff0000 broadcast 10.244.255.255 
	inet6 fc7b:dbb3:c9e2:8e50:6c98::1 prefixlen 40 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
	groups: tap 
	Opened by PID 75070

root@wintermute /t/zt# netstat -rn4 > before.lst
root@wintermute /t/zt# route add 0.0.0.0/1 10.244.0.1
add net 0.0.0.0: gateway 10.244.0.1
root@wintermute /t/zt# route add 128.0.0.0/1 10.244.0.1
add net 128.0.0.0: gateway 10.244.0.1
root@wintermute /t/zt# netstat -rn4 | tee after.lst
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
0.0.0.0/1          10.244.0.1         UGS    zt1flo98
default            172.16.1.1         UGS        igb0
10.144.0.0/16      link#6             U      zt1flo98
10.144.0.2         link#6             UHS         lo0
10.241.0.0         link#4             UH          lo1
10.241.0.1         link#4             UH          lo1
10.241.0.2         link#4             UH          lo1
10.241.0.3         link#4             UH          lo1
10.241.0.4         link#4             UH          lo1
10.241.0.5         link#4             UH          lo1
10.241.0.6         link#4             UH          lo1
10.241.0.7         link#4             UH          lo1
10.241.0.8         link#4             UH          lo1
10.241.0.9         link#4             UH          lo1
10.241.0.10        link#4             UH          lo1
10.241.0.11        link#4             UH          lo1
10.241.0.12        link#4             UH          lo1
10.241.0.13        link#4             UH          lo1
10.241.0.14        link#4             UH          lo1
10.241.0.15        link#4             UH          lo1
10.244.0.0/16      link#5             U      zt1flo98
10.244.23.143      link#5             UHS         lo0
127.0.0.1          link#3             UH          lo0
128.0.0.0/1        10.244.0.1         UGS    zt1flo98
172.16.1.0/24      link#1             U          igb0
172.16.1.14        link#1             UHS         lo0
192.168.0.0        link#2             UHS         lo0
192.168.0.0/16     link#2             U          igb1



root@wintermute /t/zt# patdiff before.lst after.lst 
------ before.lst
++++++ after.lst
@|-1,10 +1,11 ============================================================
 |Routing tables
 |
 |Internet:
 |Destination        Gateway            Flags     Netif Expire
+|0.0.0.0/1          10.244.0.1         UGS    zt1flo98
 |default            172.16.1.1         UGS        igb0
 |10.144.0.0/16      link#6             U      zt1flo98
 |10.144.0.2         link#6             UHS         lo0
 |10.241.0.0         link#4             UH          lo1
 |10.241.0.1         link#4             UH          lo1
 |10.241.0.2         link#4             UH          lo1
@|-21,12 +22,13 ============================================================
 |10.241.0.13        link#4             UH          lo1
 |10.241.0.14        link#4             UH          lo1
 |10.241.0.15        link#4             UH          lo1
 |10.244.0.0/16      link#5             U      zt1flo98
 |10.244.23.143      link#5             UHS         lo0
 |127.0.0.1          link#3             UH          lo0
+|128.0.0.0/1        10.244.0.1         UGS    zt1flo98
 |172.16.1.0/24      link#1             U          igb0
 |172.16.1.14        link#1             UHS         lo0
 |192.168.0.0        link#2             UHS         lo0
 |192.168.0.0/16     link#2             U          igb1

NB this is a different box (ethernet, no wifi, 2 NICs, 2 zt networks, 10.241.x assigned to jails) but I get the same result.

It's not clear to me yet whether this is a FreeBSD-specific issue in zerotier, or zerotier using FreeBSD routing incorrectly, or something else entirely. Apologies if I'm barking up the wrong tree.

[dch]

also huge thanks to @dharrigan for working through this with me!

[kkdzo]

I'm quite interested in seeing this fixed as well, in the mean time I got a workaround working by using multiple routing tables.

Since the usual routing table change doesn't work in FreeBSD (and OpenBSD), I looked into using a separate routing table for zerotier.
The idea is to have one table for zerotier only which has a default route to a regular gateway and another one for the rest of the system and applications, this one pointing to the zerotier gateway.

In order to use multiple routing tables, net.fibs=2 needs to be added to /boot/loader.conf (2 is the number of routing tables needed, up to 16). A reboot is required after making this change.
FreeBSD allows us to tell a process to use a specific routing table with the setfib utility.
I wrote a small rc.d script that takes care of the routing tables changes at boot time and also starts zerotier.
This replaces the rc.d script that comes packaged with zerotier.

The steps in the script are:

• Add the regular default route taken from the first / default routing table to the second one.
  This second routing table is only used by zerotier and will allow it to reach the outside world.
• Start zerotier and tell it to use the second routing table
  Upon automatically joining the network we will receive the managed routes defined for that network.
  Wait 3 seconds to allow for zerotier to do what it has to do.
• Grab the gateway for default route out of the zerotier managed routes and make it the next hop in the default routing table.
  I used jq to extract the ip from the json output, this needs to be installed.
  This effectively forces any process using the default routing table (all except zerotier-one) to go through zerotier

Since zerotier is started inside the script, it should not be enabled on its own, my /etc/rc.conf has the following entries:

zerotier_enable="NO"
ztroute_enable="YES"

And the script itself that I put in /usr/local/etc/rc.d/ztroute, with executable permission.

#!/bin/sh

# $FreeBSD$
#
# PROVIDE: ztroute
# REQUIRE: zerotier
#
# Add these lines to /etc/rc.conf.local or /etc/rc.conf
# to enable this service:
#
# ztroute_enable (bool):        Set to NO by default.
#                               Set it to YES to enable zerotier.

. /etc/rc.subr

name=ztroute
rcvar=ztroute_enable

load_rc_config $name

: ${ztroute_enable:="NO"}

start_cmd="ztroute_start"

ztroute_start()
{
        echo "Adding default route to fib 1"
        setfib 1 route add default $(netstat -rnf inet |grep default | awk '{ print $2}')
        echo "Starting Zerotier"
        setfib 1 /usr/local/sbin/zerotier-one -d
        sleep 3
        echo "Replacing fib 0 default route with ZT route"
        setfib 0 route change default $(/usr/local/bin/zerotier-cli listnetworks -j |/usr/local/bin/jq '.[].routes[] | select(.target=="0.0.0.0/0") | .via' | sed -e 's/"//g')
}

run_rc_command "$1"

This can obviously be improved in many ways like adding logic to disable and re-enable the route or having a cron entry that checks if the managed route has changed and update the table accordingly but for now it works for me.

One last thing, sshd should be started after this script is done, without that, it will not bind to the zerotier interface and sshing to the zerotier ip will not be possible.

Hope this helps.

[janjaapbos]

Thank you very much @kkdzo for nailing this down. Since I don't use freebsd I'll set up a vm with freebsd and test it myself as well.

[dch]

@kkdzo .a.w.e.s.o.m.e. with your permission I'll roll these changes into the port itself. Let me know how you'd like to be credited.

[dharrigan]

I look forward to this as well :-)