Network resilience and scaling
Networks that survive a fault and bear a surge: the single point of failure (SPOF) and high availability (N-1), the HA firewall pair (active/standby vs active/active, failover, state synchronization, split-brain), load balancing (scale-out, VIP, algorithms, sticky sessions, health check), the redundancy map (STP, HSRP/VRRP, EtherChannel/LAG, multi-WAN) and designing a resilient network by risk analysis.
Lesson 1: Why resilience: SPOF and N-1
The single point of failure (SPOF)
In earlier courses you built networks that work. Now we teach them to survive when something breaks – because eventually it always does. Power fails, someone trips over a cable, a device dies after years of service. The key term is SPOF (single point of failure): an element whose failure takes down the whole network because no backup path exists. Look at the scene as a packet’s journey from internet users (NET) to a web server: NET → firewall → load balancer → server. If each element in that row existed only once, it would be a chain of pure SPOFs – one link snaps and the whole service is down. That is why the scene shows elements doubled: two firewalls (FW-A, FW-B), two web servers (WEB-1, WEB-2) and a load balancer between them. That is not waste – it is insurance. The rest of the course is about a single question: where in the network is a SPOF and how to remove it, without turning a simple network into an unmanageable monster.
Step by step
- Traffic flows from internet users through the firewall and load balancer to the web server. So far so good.
- If the firewall existed only once, its failure would cut off everything. That is a SPOF: a single point of failure.
- So elements are doubled: if FW-A fails, traffic goes via FW-B; if WEB-1 fails, WEB-2 serves. The SPOF is gone.
- But doubling one element can reveal the next SPOF: what if the load balancer is the only one? Hunting SPOFs is a whole-course job.
High availability and N-1
When you remove a SPOF, you gain high availability (HA): the ability of the network to keep running even when something breaks. Engineers have a short rule for it – N-1: the network must withstand the failure of any one element and still serve traffic. Two firewalls? One may fail (N-1) and the other carries it. Three servers where two can handle the load? Also N-1. The point is not to have two of everything ‘for comfort’, but to deliberately compute how many elements may fail at once without the service dropping. How good is availability ‘enough’? It is measured as the percentage of time the service runs. The often-quoted target of ‘five nines’ (99.999%) means only a few minutes of outage over a whole year – and it is enormously expensive. Two numbers that will haunt you: MTBF (mean time between failures – how long an element lasts on average without a fault) and RTO (recovery time objective – how quickly you must recover once a fault does occur). Remember the meaning, not the formulas: N-1 says how much may fail at once; RTO says how fast the network recovers from it.
Step by step
- High availability = the network keeps running through a fault. The N-1 rule: it must withstand the loss of any single element.
- FW-A failed – the second firewall takes over and the service keeps running. That is N-1 in practice.
- MTBF says how long an element lasts on average without a fault – but no number ever means ‘never fails’.
- And RTO says how fast the network recovers after a fault. ‘Five nines’ (99.999%) = a few minutes of outage a year – and it is expensive.
Find the SPOF: audit and resilience test
You will now get hands-on with SPOF and N-1 right in the simulator – and two tools help nicely. The first is the topology check: it walks your network and flags weak spots, including elements and links that are the only one of their kind on a critical path. The second is the resilience test (an N-1 check): the simulator ‘switches off’ each element in turn and checks whether traffic from source to destination still gets through. An element whose removal makes the connection drop is your SPOF – exactly the one the audit pointed at. Try it: build a simple row (client → one firewall → server), run the resilience test, and you will see that switching off the firewall breaks the connection. Then add a second firewall as an HA pair (lesson 2 shows exactly how), run the test again – and the service now survives one firewall failing. This cycle of ‘find the SPOF → double it → verify by test’ is the heart of resilient network design. Learn one discipline along the way: do not scale blindly. First find where it hurts most (audit), only then add – and after every change verify that the SPOF is really gone and no new one appeared.
Step by step
- Build a simple row in the simulator and run a topology check over it – it lists the critical weak spots.
- The resilience test switches off the firewall and retries – it drops. This element is a SPOF.
- Add a second firewall as an HA pair (lesson 2). Now a backup path via FW-B exists.
- The design cycle: find the SPOF → double it → verify by test. And after each change check no new one appeared.
Lesson 2: High availability: the HA pair
Active/standby vs active/active
The most common way to remove a SPOF at a firewall or router is an HA pair: two identical devices that agree to act as one logical unit, each ready to stand in for the other. There are two modes. Active/standby: one device (FW-A) handles all traffic, the other (FW-B) waits quietly as a backup and continuously checks the active one is alive. When the active drops, the standby wakes and takes over its role. Upside: simple, predictable, no arguments about who does what. Downside: the standby does nothing most of the time – you pay for two units and use one. Active/active: both devices handle traffic at once and share the load; when one drops, the other picks up its share too. Upside: you use both units and get more capacity. Downside: more complex – you must keep both seeing the same connection state, and after a failure the survivor must carry the full load (so even an active/active pair must not be loaded to 100%, or the survivor is overloaded after a failure). In the simulator, a firewall or L3 element has an HA section with the mode choice: active/active, or active/standby. Exactly how that ‘takeover’ happens is the next module.
Step by step
- Active/standby: FW-A serves all traffic. FW-B waits quietly as a backup.
- The price of simplicity: standby FW-B mostly just waits – you pay for two units and use one.
- Active/active: both devices serve at once and share the load. You use both units and get more capacity.
- Caution: even an active/active pair must not run at 100%. After a failure the survivor carries the full load. In the simulator: the HA section + mode.
How failover happens
How exactly does the ‘takeover’ happen? It is called failover and has three phases. 1) Detection. The devices in an HA pair send each other short ‘I’m alive’ messages (a heartbeat) over a dedicated link. As long as FW-B hears FW-A’s pulse, it does nothing. When the pulse goes silent for a moment, FW-B concludes the active one has failed. 2) Takeover. The standby declares itself the new active and assumes the identity the rest of the network sends traffic to – chiefly the shared virtual IP and MAC address of the gateway (you know this trick from course 10 as HSRP/VRRP: clients talk to one unchanging gateway address even as the physical device beneath it swaps). 3) Redirection. Neighboring elements learn the address now belongs to a different device and traffic starts flowing through it. The key number is how long all this takes – the failover time. A well-tuned pair does it in seconds or fractions of a second: the user notices at most a brief stutter, an existing download may need to resume, but the service as a whole keeps going. You will try it directly in the simulator: build an HA firewall pair and in the active one’s context menu choose ‘Simulate failure (HA)’. You will watch the standby take over and traffic find the new path – exactly what you verified with the resilience test in lesson 1.
Step by step
- Detection: the firewalls send ‘I’m alive’ (a heartbeat) over a dedicated link. While FW-B hears the pulse, it does nothing.
- FW-A fails, the pulse goes silent. After a moment FW-B concludes the active device is gone.
- Takeover: FW-B declares itself active and assumes the gateway’s virtual IP/MAC – clients keep the same address (like HSRP/VRRP).
- Redirection: traffic flows via FW-B, the whole thing takes seconds. In the simulator: choose ‘Simulate failure (HA)’ in the active one’s menu.
State synchronization and split-brain
The failover from the previous module has a catch worth thinking through. A firewall is stateful (recall course 5): it remembers a table of open connections so it can let replies back through. When the active FW-A fails and FW-B takes over, what happens to the thousands of connections in progress? If FW-B knows nothing about them, it treats them as ‘unsolicited’ and drops them – users lose calls, transfers, logins, even though the network itself is ‘up’. The fix is state synchronization: the active device continuously copies its connection table to the backup, so at the moment of takeover FW-B knows every connection and continues them seamlessly. Failover then becomes almost invisible to users. The second catch is called split-brain: what if the active firewall did not fail, but only the link between them carrying the heartbeat did? FW-A lives on and keeps serving – but FW-B hears no pulse, concludes FW-A died, and also declares itself active. Suddenly there are two actives, both claiming the same virtual IP, and the network descends into chaos. The fix is beyond a beginner course (a second independent heartbeat link, a ‘vote’ on who may be active) – it is enough for you to know split-brain exists and to understand why HA must never rest on a single interconnect link.
Step by step
- A firewall is stateful: it remembers open connections. The active copies them to the backup continuously – state synchronization.
- Thanks to sync, FW-B knows every connection and continues them seamlessly – failover is almost invisible to users.
- The catch: what if only the heartbeat link fails, not the firewall? FW-A lives, but FW-B cannot hear it.
- Split-brain: both declare themselves active and claim the same virtual IP. So never build HA on a single link.
Lesson 3: Load balancing
Scale-up vs scale-out and the VIP
An HA pair solves availability – keeping the service alive through a failure. But what if the service cannot keep up because it gained users? You have two paths. Scale-up: take the existing server and give it more power – a faster CPU, more memory. Simple, but it hits a ceiling: the strongest machine has finite capacity, is expensive, and is again one big SPOF. Scale-out: instead of one big server, deploy several smaller ones and split the load among them. This approach has two magics at once: it handles almost any growth (add another server) and it is inherently resilient – if one server dies, the others keep running. That is exactly why the scene shows two web servers. But how do clients know which one to connect to? They must not have to – that is the job of the load balancer. Clients talk to a single address called a VIP (virtual IP of the service). The load balancer sits behind that VIP, assigns each incoming connection to one of the servers in the pool, and sends the reply back. The user has no idea how many servers stand behind the VIP, whether it is five or fifty – they see one service. Exactly how the LB picks a server, and how it knows one is dead, is in the next two modules.
Step by step
- Scale-up: give one server more power. Simple, but it has a ceiling – and is again one big SPOF.
- Scale-out: deploy more smaller servers and split the load. It grows almost freely and is inherently resilient.
- The client talks to a single address – the VIP. It has no idea how many servers stand behind it; it sees one service.
- Behind the VIP sits the load balancer: it assigns each connection to a server in the pool and sends the reply back.
Balancing algorithms
By what does a load balancer choose which server a new connection goes to? It has several algorithms, and they are worth understanding because each fits a different case. Round-robin: it hands out connections in turn – the first to WEB-1, the second to WEB-2, the third to WEB-1 again… Fair and simple, ideal when the servers are equally strong and requests are roughly equally heavy. Least-connections: it sends a new connection where there are currently the fewest open connections. Better when requests vary in length – a server stuck on a few heavy transfers gets no more until it frees up. Weighted: you assign servers weights by power; a stronger machine gets more connections than a weaker one. Handy when your servers are unequal. Source-IP (by the client’s address): the same client always goes to the same server. Why would anyone want that? Because of sticky sessions: some applications keep login state or cart contents locally on the server where you started. If the LB sent you elsewhere next time, you would lose the cart. Source-IP (or a cookie) therefore sticks you to ‘your’ server. A side note: stickiness is a crutch – it is cleaner when servers keep the state in shared storage outside themselves (a common database or cache), so it does not matter which one you land on. In the simulator, a load balancer offers exactly these algorithms: rr, leastconn, weighted, srcip.
Step by step
- Round-robin: hands connections out in turn – WEB-1, WEB-2, WEB-1… Ideal when servers are equally strong.
- Least-connections: a new connection goes where the fewest are open now. Better for varied-length requests.
- Weighted: assign weights by power – the stronger machine gets more connections. For unequal servers.
- Source-IP: the same client keeps hitting the same server – a sticky session, because the app holds state there (cart, login).
Health check and the dead backend
A load balancer that splits load blindly has a hidden trap: what if one server in the pool fails? If the LB did not know, it would keep dutifully sending every Nth connection to it – and each such user would hit a dead server and get an error. Load balancing would thus paradoxically worsen availability. That is why every serious load balancer runs a health check: it regularly asks each server in the pool whether it is alive – from a plain ‘do you answer on the network?’ to a smarter ‘return me a test page and I will check the answer makes sense’. A server that stops answering is removed from the pool by the LB, which stops sending it traffic. Users never notice – connections flow smoothly to the remaining healthy servers. And once the server recovers and starts answering again, the LB puts it back. Notice how both halves of the course meet here: the load balancer not only scales out (more capacity), but thanks to the health check it also provides resilience – it removes the ‘dead backend’ SPOF by itself. In the simulator, build a VIP on the load balancer with a pool of two servers, send traffic through the VIP and watch it split. Then ‘switch off’ one backend and confirm traffic keeps flowing only to the healthy one. That is scale-out and resilience in one picture.
Step by step
- The load balancer regularly asks each server in the pool whether it is alive – that is the health check.
- WEB-1 fails and stops answering. A blind LB would keep sending it every Nth connection – all errors.
- The health check removes WEB-1 from the pool; traffic flows smoothly only to the healthy WEB-2. Users notice nothing.
- Once WEB-1 answers again, the LB returns it to the pool. So the LB scales out and removes the ‘dead backend’ SPOF.
Lesson 4: The redundancy map
L2 and gateway redundancy
You have two big resilience techniques behind you – the HA pair and load balancing. Now it is time to assemble the map: a network has several places where a SPOF lurks, and each calls for a different tool. You have met most of them in earlier courses; here we gather them in one place. Start from the bottom. The L2 layer – loops: when you run a backup cable between switches you gain redundancy, but also a dangerous loop in which frames circle forever (a broadcast storm). The fix is STP (Spanning Tree Protocol, course 6): it keeps the backup link blocked, and when the main one fails it safely opens the loop. Redundancy without the storm. The gateway layer (default gateway): computers have one gateway out configured – and if that router failed, the whole subnet would go mute even if another path existed. The fix is HSRP/VRRP (course 10): two routers share one virtual gateway IP; clients keep it unchanged, and when the active router leaves, the other assumes it – exactly the same trick as the firewall HA pair from lesson 2, just at the gateway level. You see the pattern running through the whole course: one shared identity, two devices, a quiet takeover. The next module adds redundancy of capacity and of the internet connection.
Step by step
- From the bottom: a backup cable between switches gives redundancy but also a dangerous loop (a broadcast storm).
- STP keeps the backup link blocked and opens it only when the main fails – redundancy without the storm (course 6).
- The gateway layer: if the only default router failed, the whole subnet goes mute even if another path exists.
- HSRP/VRRP: two routers share one virtual gateway IP; the other assumes it on failure – the same trick as the HA pair (course 10).
Capacity and WAN redundancy
We continue the redundancy map two floors higher. Link capacity: what if one link between switches cannot carry the traffic – or you want a single cable failing to limit no one? You can bundle several physical links into one logical link. This is called EtherChannel (or generally LAG – link aggregation, course 6): four cables act as one thick one, their speed adds up and it survives any one of them failing – the rest keep going, just at lower capacity. Two birds with one stone: more throughput and resilience. The internet connection (WAN): the most precious SPOF of all – if your single link to the provider drops, no internal redundancy helps, you cannot get out. The fix is multi-WAN: have two connections from two providers (ideally on different technologies too – fiber and mobile). In normal times you can even split traffic across both; when one fails, failover redirects everything to the other (here the floating static route from courses 4 and 10 returns – a backup route with a worse metric that activates once the main one disappears). Remember the logic: EtherChannel handles capacity and internal failure, multi-WAN handles failure of the path out. The next module condenses the whole map into a single ‘where to use what’ table.
Step by step
- EtherChannel/LAG: bundles several physical links into one – adds speed and survives any one failing (course 6).
- The most precious SPOF: the single link to the provider. It drops – and no internal redundancy gets you out.
- Multi-WAN: two connections from two providers, ideally on different technologies (fiber + mobile).
- When one WAN drops, failover redirects everything to the other – the floating static route from courses 4 and 10 returns.
The whole map: where to use what
Let us gather the redundancy map in one place – this is the module you will come back to. Each network layer has its own tool, and the key is to use the right one, not to force the same thing everywhere. L2 (links between switches): STP/RSTP against loops (keeps the backup blocked), EtherChannel/LAG for capacity and resilience (bundles links). Gateway (first hop out): HSRP/VRRP – two routers, one virtual IP. Firewall and L3 elements: an HA pair (active/standby or active/active) with state synchronization. Servers and applications: a load balancer with a VIP, pool and health check – it scales out and removes a dead backend by itself. Path to the internet: multi-WAN with two providers and failover. Notice two patterns recur across all of it. First: shared identity + quiet takeover (the virtual IP in HSRP, the HA pair and multi-WAN failover). Second: split the load across several units (the load balancer, EtherChannel, active/active). Resilience is thus not one magic box – it is the discipline of hunting SPOFs layer by layer and applying each layer’s proven tool. And beware the last, most often forgotten layer that no protocol solves: power and physics. Two power supplies, two feeds, a backup battery (UPS), servers in two different rooms or buildings. The cleverest HA pair is useless when both units hang on one socket.
Step by step
- The redundancy map: L2 → STP/LAG, gateway → HSRP/VRRP, firewall → HA pair, servers → load balancer, internet → multi-WAN.
- Pattern 1: shared identity + quiet takeover – the virtual IP in HSRP, the HA pair and multi-WAN failover.
- Pattern 2: split the load across units – the load balancer, EtherChannel, active/active pair.
- The most forgotten layer: power and physics – two supplies, a UPS, two rooms. An HA pair on one socket is useless.
Lesson 5: Designing a resilient network
Layer your redundancy
The last lesson is about design: how to assemble the individual tools into a network that truly holds. The first and most important rule: no single redundancy is enough on its own. You can have the finest HA firewall pair – but if a single server hangs behind it, the service drops anyway. You can have ten servers behind a load balancer – but if the load balancer itself is just one, you have just made it the new SPOF. Resilience is therefore designed in layers, and a chain is only as strong as its weakest link. Walk the packet’s path end to end and at every hop ask ‘and what if this piece dies?’: the internet (→ multi-WAN), the gateway (→ HSRP), the firewall (→ HA pair), the balancer and the servers (→ a load balancer with a pool, and even two LBs in HA – sharing a virtual IP with the same trick as HSRP or the firewall HA pair from lesson 4), and beneath it all the switches and links (→ STP, LAG) and the power (→ two supplies, a UPS). Only when every link of the chain has a backup do you truly have a resilient network – and that is exactly what the resilience test from lesson 1 tells you after each change: it runs N-1 across the whole topology and shows whether a lonely element remains somewhere whose failure takes everything down. Resilience design is thus not a one-off act but a loop: build a layer, test N-1, find the weakest link, strengthen it, test again.
Step by step
- You can have the finest HA firewall pair – two devices, quiet takeover, state synchronization.
- …but if a single server hangs behind it, the service drops anyway. No single redundancy is enough alone.
- Resilience is designed in layers: at every hop ask ‘what if this piece dies?’. The chain is only as strong as its weakest link.
- Design is a loop: build a layer, test N-1, find the weakest link, strengthen it, test again.
Cost vs benefit
If resilience is so great, why not double absolutely everything, always? Because redundancy costs something – and blind ‘two of everything’ is the same mistake as no redundancy, just pricier. Each doubled element means double the purchase, more space, more power, and above all more complexity: more configuration, more things that can break or be set up wrong. And beware – complexity itself is a risk: a misconfigured HA pair can take down a network in ways a single device never could (recall split-brain). So how to decide where you want redundancy and where it is overkill? With the same tool as in the OT course: risk analysis. Ask two questions. How often does it fail (probability) and what happens when it does (impact)? High impact → redundancy almost always pays off: a company e-shop that loses hundreds of thousands per hour of outage wants an HA pair, a load balancer and multi-WAN. Low impact → a plain backup or nothing suffices: a home network survives the router dropping for five minutes once a year – buying a second one is overkill. And a key idea often forgotten: before reaching for expensive real-time redundancy, weigh cheaper backups and fast recovery. For many services a few minutes of outage and a quick restore from backup is acceptable – at a fraction of the cost of full HA. Resilience is not a contest of ‘who has more nines’, but a considered choice of how much of that cost you are willing to pay for how much certainty.
Step by step
- Redundancy costs: double purchase, more space and power, and above all more complexity. Blind ‘two of everything’ is a pricier mistake.
- And beware: complexity is a risk. A misconfigured HA pair can crash a network worse than a single device.
- Risk analysis decides (probability × impact). An e-shop losing hundreds of thousands per hour of outage wants HA and multi-WAN.
- Low impact → overkill. Before expensive HA, weigh cheap backups and fast recovery – a few minutes of outage is often fine.
Summary: resilience together
The course ends – let us assemble resilience and scaling into one picture. Find the weak spot: a SPOF is an element whose failure takes down the whole network; the goal is N-1 (withstand any single piece failing), and the tools are the topology check and the resilience test (lesson 1). Ensure availability: an HA pair doubles a firewall or router – active/standby or active/active, with quiet failover and state synchronization so takeover does not drop connections; beware split-brain (lesson 2). Scale out: a load balancer hides a server pool behind one VIP, splits the load (round-robin, least-conn, weighted, sticky) and removes a dead backend by itself via the health check – scale-out and resilience in one (lesson 3). Use the right tool on the right layer: STP and LAG on L2, HSRP/VRRP on the gateway, an HA pair on the firewall, a load balancer on the servers, multi-WAN on the internet – and do not forget the power (lesson 4). Design with restraint: no single redundancy suffices, a chain is only as strong as its weakest link, and how much of the cost you pay is decided by risk analysis (lesson 5). Notice the common thread: almost none of this was new magic – the virtual IP, floating static, STP, the stateful firewall you know from earlier courses. Resilience is largely a way of thinking: at every element ask ‘and what if this dies?’ – and have the answer ready before it happens.