Redundancy and failover
Deciding which failures the system must survive, then building redundancy at the layer that actually addresses them.
What it is
Redundancy is a set of specific answers to a specific question: what fails, and what happens next. It can live at several layers. Physical path redundancy with a loop protected by RSTP or a vendor ring protocol. Link aggregation across two uplinks. Dual network interfaces on end devices, which behave differently depending on whether the device does hot standby or genuinely transmits on both. Protocol-level redundancy, where sACN priority lets a backup source sit live and be arbitrated by receivers. Application redundancy in the tracking or show control layer, with a second server tracking the same state. And the least glamorous form, a configured cold spare in the rack with a documented swap procedure.
Why it matters
Eight shows a week is a duty cycle that finds the weak component, and it finds it during the run rather than during tech. A design that says it is redundant without saying what it is redundant against usually protects the cheap failure and leaves the expensive one alone: dual uplinks between switches while a single tracking server feeds every consumer, or a redundant core with one power feed. The other half of the problem is recovery time. Some mechanisms reconverge in a way a control stream can absorb, and some produce a visible gap. Knowing which is which, and having rehearsed the failover rather than assumed it, is the difference between a moment nobody notices and a moment everybody does.
Name the failure first: a cable, a switch, a power feed, a server, or someone unplugging the wrong thing
RSTP reconverges quickly but not instantly; a stream that cannot tolerate any gap needs a different mechanism
sACN priority gives defined source arbitration; equivalent behavior is not guaranteed for every tracking protocol
A redundant network in front of a single tracking server has moved the single point of failure, not removed it
Is spanning tree enough?
It depends on what the streams tolerate. RSTP prevents loops and restores a path after a link failure, but restoration takes time and consumers see an interruption during it. For monitoring and management traffic that is fine. For a live position stream feeding automation or moving light targeting, decide explicitly whether a gap of that order is acceptable, and if it is not, look at redundancy that does not depend on reconvergence.
Do dual NICs on a console give real redundancy?
That depends entirely on what the device does with them, and implementations vary widely. Some transmit identical data on both interfaces continuously, some hold one as a standby and switch on link loss, some use the second interface for an entirely different purpose. Read what the manufacturer actually specifies, then test it by pulling a cable and watching downstream, rather than inferring behavior from the presence of two ports.
How do we test failover without risking a show?
Schedule it, script it, and do it when the building is not full. Pull one element at a time, note what the consumers do and how long recovery takes, and put those observations in the documentation as expected behavior. Untested failover is a belief. The value of the test is often the surprise it produces, which is exactly the surprise you do not want to have during a performance.