High Availability Router/Firewall Using OpenBSD, CARP, pfsync, and ifstated
Learn more about using OpenBSD, CARP, and more on your firewall.
·
Join the DZone community and get the full member experience.
Join For FreeIntroduction
I have been running OpenBSD on a Soekris net5501 for my router/firewall since early 2012. Because I run a multitude of services on this system (more on that later), the meager 500Mhz AMD Geode + 512MB SDRAM was starting to get a little sluggish while trying to do anything via the terminal. Despite the perceived performance hit during interactive SSH sessions, it still supported a full 100Mbit connection with NAT, so I wasn’t overly eager to change anything. Luckily though, my ISP increased the bandwidth available on my plan tier to 150Mbit+. Unfortunately, the Soekris only contained 4xVIA Rhine Fast Ethernet. So now, I was using a slow system and wasting money by not being able to fully utilize my connection.
Naturally, I looked back to Soekris for an upgrade that would allow me to take advantage of this new speed since it served me so well for so long, but I soon discovered that Soekris stopped innovating and closed US operations a few years ago. After widening the search, I decided to try the PC Engines APU4C4. This included a 4 Core 1Ghz AMD GX-412TC CPU, 4GB of DDR3-1333 DRAM and 4xIntel PRO/1000 Gigabit Ethernet. A huge improvement.
Now that I had two appliances, I figured why not try setting them up in failover mode with CARP and pfsync. At the very least, this would come in handy during the post-patching reboots and while down for an upgrade every six months. A small win, but at least I could upgrade whenever I wanted without impacting my family’s Internet connectivity.
Note: While there are a lot of things I still have planned to do with this setup and may already have noted any deficiencies, I welcome suggestions and feedback.
New Network Topology
The following diagram details out how everything is wired together. I split my LAN and Wi-Fi into separate networks to add an additional layer of separation between the secured servers on the hardwire and potential rogues coming in through the wireless. I may eventually flatten this once I feel all services are locked down better, but for now, it gives me a better (unwarranted?) sense of security. I plan to eventually replace the three unmanaged switches with a single managed switch with vlans, but that is for a future article.
Description
As I mentioned above, a number of services for the network are provided directly from the router/firewall. While I realize this isn’t necessarily best practice, it is for my home network where I am constrained by budget and compelled by convenience to put them all on the router. Those daemons include:
-
dhcpd- Listens on vr2, vr3, em2, em3
- Since I’m running dhcpd on both routers, which operate on the same subnets, I chose to split the addresses each serves (r2 has last octet of 100-150, r1 has 151-250) rather than trying to do anything magical.
-
tftpd- Currently listening on vr2 only
- Need to do more work here
-
opensmtpd(vr2, em2)- Listens on vr2, em2
- Acts merely as a relay for all LAN servers to personal email server
-
openntpd- Listens on vr2, em2
- Created as a pool in nsd for round-robin client access
-
unbound- Listens on vr2, vr3, em2, em3
- Configured for DNSSEC validation and upstream dns-over-tls to Quad9
-
nsd- Listens on vr2, vr3, em2, em3
- Hosts personal domain for easy access of local services
-
avahi_daemon- Listens on vr2, vr3, em2, em3
- Exposes ZFS array on LAN server to OSX clients for TimeMachine backups
It should go without saying, but each system, of course, runs openssh and requires keys for access (i.e. no password auth). I also use ddclient to keep a domain I own up to date if the ISP DCHP-provided address ever changes. In addition, I am using munin for monitoring system trends and pflow for connection tracking with nfsen. Data from the latter two are pushed to and accessible from the server on the LAN.
While much of the above falls outside the scope of this post, I will detail most of the OpenBSD-specific configuration files below.
Configuration
I’m using Ansible to manage all of these files, so anything highlighted is controlled with jinja2 to ensure that the proper config is pushed to the correct system.
Caveat: This is accurate for OpenBSD 6.4
Caveat 2: I modified some of the configs below during the creation of this post, so there may be mistakes since they are no longer copypasta.
Both R1 and R2
/etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.carp.preempt=1
ddb.panic=0Soekris (R1)
/etc/boot.conf
stty com0 19200
set tty com0/etc/mygate
192.168.50.3I want to give the backup system Internet access at boot or my pf.conf rules won’t load, so I give it the IP of the other host’s physical link (not the CARP device) via the mygate file. Note: You must ensure your pf rules permit this communication.
/etc/hostname.vr0
I chose not to add this file, but you could (and probably should) add one with just the down command
/etc/hostname.vr1
inet 10.0.0.2 255.255.255.0 10.0.0.255/etc/hostname.vr2
inet 192.168.0.2 255.255.255.0 192.168.0.255/etc/hostname.vr3
inet 172.16.0.2 255.255.255.0 172.16.0.255/etc/hostname.carp2
vhid 2 carpdev vr2 pass <carp2 password> advskew 100 192.168.0.1 255.255.255.0/etc/hostname.carp3
vhid 3 carpdev vr3 pass <carp3 password> advskew 100 172.16.0.1 255.255.255.0/etc/hostname.pfsync0
syncdev vr1/etc/hostname.pflow0
flowsrc 192.168.0.2 flowdst <ip of nfsen host>:<port nfsen is listen to> pflowproto 10/etc/ifstated.conf
# Initial State
init-state auto
# Macros
if_carp_up="carp2.link.up && carp3.link.up"
if_carp_down="!carp2.link.up || !carp3.link.up"
if_wan_up="vr0.link.up"
state auto {
if $if_carp_up {
set-state master
}
if $if_carp_down {
set-state backup
}
}
state master {
init {
# WAN hostname.if(5) should be started as 'down' with no ipaddr.
# Spoof MAC so it can get an IP from ISP without a modem reboot
run "/sbin/ifconfig vr0 lladdr 00:de:b9:be:11:ad up"
# Clean up stale routes and arp cache; dhclient will create default route.
run "/sbin/route -qn flush".
# Renew the ip lease - hopefully stays the same, for pfsync.
run "/sbin/dhclient vr0"
# notify root whenever master changes
run "echo master firewall is now `hostname` | mail -s ‘carp master changed’ root@localhost”
# Need to create pfctl -a carp rules to be loaded
# so other firewall can route through this firewall when this becomes the master
}
if $if_carp_down {
set-state backup
}
}
state backup {
init {
# This process should be terminated, first.
run "/usr/bin/pkill -9 dhclient"
# Delete IP, reset mac and bring wan if down with alternate MAC
run "/sbin/ifconfig vr0 delete lladdr de:ad:00:00:be:ef down"
# Clean up stale routes and arp cache.
run "/sbin/route -qn flush”
# Create default route out to internet via the master host.
run "/sbin/route -qn add default 192.168.0.3"
# Need to create pfctl -a carp -F rules to be unloaded
# so other firewall can only route through this firewall when this becomes the master
}
if $if_carp_up {
set-state master
}
}/etc/pf.if
ext_if="vr0"
sync_if="vr1"
lan_if="vr2"
wifi_if="vr3"/etc/pf.conf
#Include Ansible configured interface file so pf.conf can be the same on both routers
include "/etc/pf.if"
#Allow pfsync on link connecting both routers
pass quick on $sync_if proto pfsync
#Allow carp traffic on physical devices carp uses
pass quick on { $wifi_if $lan_if } proto carp
#This will eventually become an anchor so pf.conf can be the same on both systems
#so only an achor file will be different on each host
pass quick on $lan_if from $lan_if:0 to 192.168.50.3
Note: You should have more rules than this. I am just explicitly noting the additional rules added for this project.
PC Engines (R2)
/etc/boot.conf
stty com0 115200
set tty com0/etc/hostname.vr0
dhcp/etc/hostname.vr1
inet 10.0.0.3 255.255.255.0 10.0.0.255/etc/hostname.vr2
inet 192.168.0.3 255.255.255.0 192.168.0.255/etc/hostname.vr3
inet 172.16.0.3 255.255.255.0 172.16.0.255/etc/hostname.carp2
vhid 2 carpdev em2 pass <carp2 password> 192.168.0.1 255.255.255.0/etc/hostname.carp3
vhid 3 carpdev em3 pass <carp3 password> 172.16.0.1 255.255.255.0/etc/hostname.pfsync0
syncdev em1/etc/hostname.pflow0
flowsrc 192.168.0.2 flowdst <ip of nfsen host>:<port nfsen is listen to> pflowproto 10/etc/ifstated.conf
# Initial State
init-state auto
# Macros
if_carp_up="carp2.link.up && carp3.link.up"
if_carp_down="!carp2.link.up || !carp3.link.up"
if_wan_up="em0.link.up"
state auto {
if $if_carp_up {
set-state master
}
if $if_carp_down {
set-state backup
}
}
state master {
init {
# WAN hostname.if(5) started with dhcp upon boot so pf.conf rules will load so this will reissue request.
# This is actually the MAC of this machine, but since backup replaces with spoof
# we set it here explicitly so it can get an IP from ISP without a modem reboot
run "/sbin/ifconfig em0 lladdr 00:de:b9:be:11:ad up"
# Clean up stale routes and arp cache; dhclient will create default route.
run "/sbin/route -qn flush”
# Renew the ip lease - hopefully stays the same, for pfsync.
run "/sbin/dhclient em0"
# notify root whenever master changes
run “run “echo master is now `hostname` | mail -s ‘carp master changed’ root@localhost”
# Need to create pfctl -a carp rules to be loaded
#so other firewall can route through this firewall when this becomes the master
}
if $if_carp_down {
set-state backup
}
}
state backup {
init {
# This process should be terminated, first.
run "/usr/bin/pkill -9 dhclient"
# Delete IP, reset mac and bring wan if down with alternate MAC
run "/sbin/ifconfig em0 delete lladdr de:ad:00:00:be:ef down"
# Clean up stale routes and arp cache.
run "/sbin/route -qn flush" # flushes arp cache too.
# Create default route out to internet via the master host.
run "/sbin/route -qn add default 192.168.0.2"
# Need to create pfctl -a carp -F rules to be unloaded
# so other firewall can only route through this firewall when this becomes the master
}
if $if_carp_up {
set-state master
}
}/etc/pf.if
ext_if="em0"
sync_if="em1"
lan_if="em2"
wifi_if="em3"/etc/pf.conf
#Include Ansible configured interface file so pf.conf can be the same on both routers
include "/etc/pf.if"
#Allow pfsync on link connecting both routers
pass quick on $sync_if proto pfsync
#Allow carp traffic on physical devices carp uses
pass quick on { $wifi_if $lan_if } proto carp
#This will eventually become an anchor so pf.conf can be the same on both systems
#so only an achor file will be different on each host
pass quick on $lan_if from $lan_if:0 to 192.168.50.2Again, you should have more rules than this. I am just explicitly noting the additional rules added for this project.
References
Opinions expressed by DZone contributors are their own.
Comments