Home Network Overview

Today, the core of my home network looks like this.

Yes, I know what your first question is. The cable labels are these ones. Write with a felt tip marker, erase with warm water.

The core of the local network needs to handle most isolation tasks (firewalls, VLANs, etc.), plus managing uplinks (fail-over, shaping traffic, etc). Therefore, we need a smart device there. In my old days, I would go and set up a spare Linux box that would do everything. In fact, some of my "lab servers" today are smaller devices that I used to run at the network core. Over the years, I have gradually replaced many Linux network boxes with Mikrotik devices.

RB 4011 is overkill for most uses, and it is quite expensive. Most home networks do not need this kind of gear. My own choice was driven by several considerations: IPSec performance for tunneling between different home/VPN networks, decent amount of Ethernet ports that I can plug different-ranked devices into, the integrated wireless AP that simplifies deployment, enough CPU capacity to drive complex firewall and bridging rules without bottlenecking on router CPU.

I should say this again: Mikrotiks are not for everyone. Their powerful features come as the double-edged sword: you have to know what you are doing, or at least be ready to read a lot. In most cases, the experience with networking itself, and probably with Linux networking specifically, would come as the huge help. This is how I look at these boxes: they are Linux boxes in disguise. When it comes to doing firewall rules, you can clearly see iptables, when doing shaping rules, tc, etc.

When building the network, especially for working from home, it is important to consider what happens when one piece of the network breaks. Yes, you can technically grab a laptop and go to the nearest library until it blows over, but what if you need to survive the outage lasting full days?

In the case of network core, I have the backup hEX (RB 750 Gr3) router that used to be the old core. Its configuration closely resembles the configuration of the current core, albeit at lower Ethernet port count. So if core RB 4011 breaks, I can get a major part of my home network back just by re-plugging half of the Ethernet cables back to hEX, and rewiring the fiber links back to Ethernet.

Since I live in Germany, VDSL2 is still pretty much a thing, so the other part of the core is the DSL modem. I run DrayTek Vigor 165 in bridge mode, where it basically converts the DSL media to the Ethernet that core RB 4011 terminates with appropriate VLANs and PPPoE. I chose Vigor because it gives me more verbose line stats, and allows fine-tuning the SNR delta. Being able to trade in synchronization rate for better SNR margin is a nice fallback when/if DSL misbehaves.

Sample status log, negotiated at 12 dB SNR:

It helps to diagnose wiring performance. For example, it was proven experimentally that ziptying the long unshielded DSL twisted pair cable with other cables is not a good idea, d’uh. Also, proving that DSL engineers have not screwed up the connection when moving the sockets from one room to the other by demonstrating SNR had not degraded is a nice touch.

Deep down I lament the absence of plain Ethernet or fiber connection in my apartment building. Oh well. In principle, I can go and replace the Vigor 165 device with VDSL2 SFP+ module that plugs directly into RB 4011, but that does not seem to buy me anything substantial in comparison to a separate device. That SPF+ port is better used to uplink the 10G switch.

The VDSL2 modems also have backups. The ISP’s own Fritzbox router is still usable in the same bridge mode to get the uplink running if Vigor 130 fails. ALLNet MC115 that I used before as the reliable bridge works too, but only at ADSL2+ speeds (there is a newer model that supports VDSL2 vectoring, but why bother when Vigor works). The advantage for running those modems in bridge mode is interchangeability: plug it between WAN port on RB 4011 and DSL socket, done. RB 4011 config does not care what actually translates DSL for it.

If DSL link fails altogether, one of RB 4011’s wireless interfaces is repurposed to connect to the outside Vodaphone Hotspot that gives daily/weekly/monthly vouchers for accessing the internet with decent speeds. If that fails, the same thing tethers to mobile 4G devices. The good thing to handle this on core RB 4011 is that the rest of the network does not need any reconfiguration, and can proceed as usual, albeit at lower outside bandwidth. When tethering to mobile devices, centralized traffic shaping configuration allows hard rationing for the entire fleet of devices.

The network core is in the study room where most of our hardware resides. Connecting that hardware is easy: wires. Normally, you would do a Gigabit Ethernet, and that should be enough. In my updated network, most of the backhaul is fiber. We will get to that soon, but we need to consider a few other options.

The story gets interesting when you need to extend the network beyond one room. Let me get one thing very straight: for in-house connectivity, your best cost/performance bet is still Gigabit Ethernet. If I ever renovate a home/apartment, I would resist laying down laying Ethernet cables. I would lay conduits: that way, you could route either Cat6, Cat7, or fiber, depending on your needs. if you don’t have this opportunity, for example you live in the old house (like me), or in a rented apartment that does not have cable channel infrastructure and/or does not take kindly to drilling walls (like mine), or you are generally lazy (like me), then you have several options.

At this point, we have built the core and the backhaul. Now we have to terminate the backhaul with the actual access points. Those access points have to do both Ethernet and wireless. PLC gives about 100 Mbps in good conditions here, so getting the access routers with Gigabit ports might be a bit of overkill. With MoCA, you might want to terminate with Gigabit, but then you would not like to overload the MoCA network. With fiber, you can pretty much do whatever, either SFP or SFP+ uplinks. You would also like to have an access point per room, which means the routers should not be very expensive.

Therefore, in my network the access points are Mikrotik hAP AC-s, bought on second-hand market for peanuts.

The most privileged network is the study network that runs our work computers. Those can access almost all devices on all other subnetworks, but the reverse is not true. This subnetwork is not even carried by wireless, so if I wanted to have the same privilege level as my desktop on my laptop, I need to physically plug it in.

Main subnetwork is what most deployments have as the only network: it handles the normal devices, and it is a default subnetwork for wireless users and Ethernet ports on access routers. These can access Internet and all other same-rank devices, but not much else. Which means that even if you know the WiFi password or you plug into Ethernet in other rooms, you are not able to access the sensitive parts of the network.

When running lab servers that carry experimental hardware and software, firewalling them individually is not convenient. This is doubly so for virtual machines that run on lab server. Because of this, the entire lab subnetwork is isolated, and the bulk of smaller Pi hosts are carried by its own separate switch.

The access to lab subnetwork is possible from a selected few devices, mostly our work machines.

The lab network also has no access to Internet, apart from a few designated version control hosts that they routinely communicate with (for example, hg.openjdk.java.net). So, the lab servers would not be able to auto-update, or push telemetry, or do something else that might in-/ex-filtrate the sensitive data. When build servers need to push the builds, they don’t even go to Internet, but rather push them to a designated file server that mirrors it to Internet (say, builds.shipilev.net) in one go.

Once updates are needed, I can selectively enable the firewall rule that allows lab servers to go outside, update them in a batch, and then disable the rule back.

Sensors run a number of Pi Zero nodes powered by PoE via 24-5V stepdown converters. These sensors run the seldom-updated Linux distros, sometimes they run software that was written by some unknown IoT enthusiasts. And generically, I don’t want them in the main network, so they get their own.

The sensors subnetwork does not have access to anything beyond a particular internal host/port where they feed the data to. Connecting to sensors is allowed from the set of designated hosts as well. Overall, this is the most fortified subnetwork here: completely untrustworthy in both directions.

Another special device in the network is the RIPE Atlas probe node.

While RIPE people get my respect and trust with configuring their nodes correctly, that is still 3rd-party node running within the home network, and it can in principle be used as the intermediate node to get into the home network. It also auto-updates itself, so there is however small possibility of receiving the malicious code drop. Therefore, while this probe naturally has access to Internet, it is firewalled hard from the rest of any network.

On the upside, it can be routed through different uplinks, depending on which AS has lesser density of probes (ask RIPE Atlas folks for guidance here).

As I mentioned before, two Freifunk nodes are using TP-Link CPE 210 routers.

Freifunk VLANs are numerous, because they are carrying different layers of the network data.

For starters, there is "freifunk-uplink" VLAN that is used by both nodes to access the Internet. They are terminated at the core router and both nodes share their own /24 subnetwork. Firewalls isolate the routing from that network towards the rest of the subnetworks, plus only allow Internet exiting towards the Freifunk VPN servers and a handful of monitoring hosts. This provides the additional layer of security in case the botched configurations on FF nodes would start leaking the traffic from the unauthenticated users via my uplink.

Then, there is "freifunk-users" VLAN that carries Freifunk users. Normally, FF users connect directly to Freifunk nodes, but that "freifunk-users" VLAN also allows me to carry Freifunk services on every other access router in my home. So, when someone connects to one of the access routers with freifunk SSIDs, it would be carried via that VLAN to the destination Freifunk node, that would route it as if it was (it actually is) the regular Freifunk user.

Then, there is "freifunk-mesh" VLAN that allows Freifunk nodes to mesh together through the local Ethernet network. The Freifunk network itself runs the OLSR, and it is technically possible to tell the nodes to discover themselves over the Ethernet. In practice, that means if one node’s uplink is down, then it would redirect the traffic through the mesh to another node. And, in the city-wide mesh network, it provides one of the repeater bridges: one CPE210 looks one way, meshes with another CPE210, which looks the other way.

The core router also shapes the network traffic. This is rarely important for internal traffic, because internal network bandwidth is quite abundant, but it is important for rationing the uplink. It is easy to classify the traffic with a few simple rules. For performance reasons, we would like to leverage connection tracking for this.

So, first mark all new connections with connection marks:

Then translate connection marks to packet marks for every packet — this saves more complicated match against the port sets, etc:

Then use Mikrotik’s simple queues to chop up the downlink/uplink:

With this, interactive traffic gets prioritized first, then "usual" web traffic gets the slice, then lab servers get a part, then Freifunk users pick up what is left.

It is useful to throttle some subnetworks too, so that no automated system can saturate the network:

This shaping thing becomes very important when ISP uplink fails. Then, after tethering to external hotspot or 4G, we can trim down the limits in the shaper to ration the bandwidth correctly.

Or, as it proved useful recently, cap the labs Internet traffic when you are delivering the online talk.

The usual thing is local DNS forwarder to optimize DNS lookups. It also comes with two nifty opportunities.

First, it allows naming the local network nodes without maintaining the separate DNS server or messing with public DNS zones:

Second, it allows a crude Adblock by resolving offending hosts to a private IP:

…​and then rejecting the connection to it:

The benefit of doing that on the core router is implicitly covering every device in the managed network.

One of the understandable but still surprising things about Pi Zero is the absence of always-on real-time clock. This means when my sensor node starts up and tries to submit the sensor data, it is rather confused about the current time. The way out is to sync the time with NTP at startup. But since sensor nodes are prohibited from exiting to Internet, they need the local NTP.