25  System Architecture: ROS 2 Nodes, Services, and Topics

Part IV closed with a promise: the complete system — every node, every topic, every authority transfer from perception to PX4 — laid out as one architecture and interrogated as one design. This chapter keeps it. Everything the book has built so far arrived one file at a time: the Coordinate Frames chapter read frames.py, the Geofencing and Fail-Safes chapter read failsafe_monitor.py and the mission manager’s _enter_failsafe, the Harassment Pattern Generation chapter read harassment_node.py, and Part III built the detection pipeline that feeds them all. What no chapter has yet done is put the shipped Goose Management System code on the table whole — seven cooperating processes, ten service endpoints, eight mission topics — and ask how the pieces actually talk to each other, in what order, and under whose authority.

Doing that has a consequence worth stating up front, because it is the most instructive thing about the chapter. Assembling the real system end to end exposes the seams between what shipped and what the book taught while building toward it. The real mission machine has eleven states, not the five of the Finite State Machines chapter. The real search leg is flown by PX4’s own onboard mission executor, not by the Offboard setpoint streaming the Path Planning on the Edge chapter used. And the repository’s own README drifts from its own code in several places. None of these is an error to be papered over; each is an architectural decision (or a documentation decay) with reasons a systems engineer should be able to articulate — and articulating them is precisely the kind of work this Part, on integration and diagnostics, exists to teach. The pedagogical builds were not wrong; they taught mechanics the real system genuinely uses — just not always where a reader might have assumed. This chapter says exactly where.

25.1 Learning Objectives

  • Explain the ROS 2 distinction between a Service (request–response, blocking until acknowledged) and a Topic (continuous publish/subscribe streaming), and give one real example of each from the Goose Management System — a flight-controller service such as /goose/fc/arm that returns only after PX4’s own telemetry confirms the change, versus a continuously published topic such as /goose/active_controller or /goose/target.
  • Draw the full node graph of the running example — Mission Manager, Flight Controller, Navigation, Harass, Detector, Failsafe, and the Drone itself — labeling each edge as a service or a topic with its real name, rate, and message type.
  • Trace one complete mission execution flow through the graph, from the operator’s /goose/start_mission call through the real eleven-state sequence (IDLE → PREFLIGHT → ARMING → START_MISSION → MISSION → ENGAGE_PREP → ENGAGE → HARASS → RESUME → … → DONE or FAILSAFE), stating for each transition whether a service call succeeding or a topic crossing a threshold triggers it.
  • Reconcile the shipped system with the book’s pedagogical builds: explain why the real search leg uses PX4’s native AUTO.MISSION mode while the engagement phase uses Offboard streaming, and why the real state machine is more granular than the five-state teaching machine.
  • Explain the valid/confirmed distinction in the Detector’s Target message, and why a target must clear a higher bar to start an engagement than to be considered still tracked during one.
  • Argue, with the real _enter_failsafe mechanism as evidence, why an independent Failsafe node with override authority improves fault tolerance over folding safety checks into each behavior node individually.

25.2 Key Vocabulary

  • ROS 2 node — one independently running process in a ROS 2 system, owning its own subscriptions, publications, services, timers, and failure modes; the unit of modularity the whole architecture is built from, and the reason a crash in one subsystem does not take down its supervisor.
  • Topic (ROS 2) — a named, typed channel for continuous one-to-many data streaming: publishers send messages at their own rate, subscribers receive whatever arrives, and no reply ever flows back; the right shape for state that is continuously true — position, status, a target estimate.
  • Service (ROS 2) — a named request–response endpoint: a client sends one request, exactly one server processes it and returns a result, and the client can wait on that result; the right shape for actions that must be acknowledged before the mission logic dares proceed — arm, change flight mode, run a preflight check.
  • Publish/subscribe — the decoupled communication pattern behind topics: publishers do not know who is listening and subscribers do not know who is talking, so nodes can be added, restarted, or removed without any other node’s code changing.
  • Active controller — the running example’s arbitration token: a String on /goose/active_controller, written only by the Mission Manager, naming which behavior node ('none', 'navigation', or 'harassment') currently owns the aircraft’s setpoint stream; every behavior node’s tick begins by checking whether it is the one named.
  • Override authority — the property that one node can revoke another’s control of the vehicle without the revoked node’s cooperation or consent; in this system it is exercised at the token (set to 'none') and at the flight-mode services, and it flows strictly downward as the Geofencing and Fail-Safes chapter’s authority ladder established.
  • Mission execution flow — the end-to-end sequence of states, service calls, and topic-triggered transitions that carries one mission from an operator’s start request to a landed, disarmed aircraft; the system-level artifact a full architecture walkthrough traces from start to finish.
  • Confirmed target — the higher of the Target message’s two truth levels: valid means a detection arrived recently (within a 2.0 s staleness window); confirmed means the full confirmation criteria — enough recent detections, spatially clustered — are also met, and only a confirmed target may begin a new engagement.
  • Command verification — the flight-controller discipline of never reporting a service successful merely because a command was sent: the node polls PX4’s own telemetry until the vehicle’s state actually reflects the change, or times out and reports failure.

25.3 Two Verbs: Services and Topics

ROS 2 gives a distributed system two verbs, and the running example’s repository opens its own README with the distinction because everything else in the architecture follows from it. A Service is a request–response transaction: one node asks another to perform a specific action and waits for confirmation before proceeding. A Topic is a continuous stream: a publisher asserts a piece of state at its own cadence, and any number of subscribers consume it without ever replying. The design question for every edge in the node graph is which verb it needs, and the test is simple: does the sender need to know the action happened before it can safely do the next thing?

The system’s own arm sequence is the canonical service. When the Mission Manager reaches its ARMING state, it calls the Flight Controller’s /goose/fc/arm service and blocks its mission logic on the answer — because commanding a takeoff on an aircraft that silently failed to arm is not a degraded mission, it is an incident. And the Flight Controller’s implementation shows why the request–response shape earns its keep end to end: it does not return success when the VehicleCommand is transmitted to PX4. It sends the command, then polls the vehicle’s own status telemetry — up to two seconds per attempt — until PX4’s reported state actually shows the vehicle armed, retrying the whole send-and-poll cycle up to three times before giving up, and only then does the service return success. The module’s docstring states the discipline in seven words: every command is verified against vehicle state. A service, used this way, is not just a function call across processes; it is a contract — “when this returns true, the world has changed, and here is the autopilot’s own telemetry to prove it.”

The active-controller token is the canonical topic. Which node currently owns the aircraft is not an action to acknowledge; it is a fact that must be continuously true and visible — to nodes that were up before it changed, to nodes that restarted after, to anyone running ros2 topic echo at the field bench. So the Mission Manager publishes the token as a String on /goose/active_controller, re-asserting it at 1 Hz even when nothing changed, the same publish-while-true protocol the Geofencing and Fail-Safes chapter’s watchdog uses on /goose/failsafe. The Detector’s /goose/target stream is the same verb at a faster cadence: a Target message at 10 Hz, whether or not anyone is listening, whether or not a goose exists — because “what does perception currently believe?” is a question whose answer must always be one subscription away. Nothing waits on a topic; nothing acknowledges it. That is not a weakness of topics — it is their specification, and matching each edge of the graph to the right verb is most of what this chapter’s architecture is.

25.4 The Node Graph

The whole shipped system is seven nodes, and Figure 25.1 draws them with every edge labeled by its real name from the repository’s px4_defs.py constants file. The table beneath anchors each node’s one-line responsibility, its real interface, and — because this chapter is the integration view, not a re-derivation — the chapter that already covers its internals.

Node graph diagram titled the Goose Management System as one ROS 2 graph, services command, topics stream. Seven boxes: at top left a grey operator and ground station box; top center a teal Mission Manager box annotated 10 hertz tick, sequences the mission, sole writer of the token; top right an amber Failsafe monitor box annotated 1 hertz independent watchdog with a cross reference to the Geofencing and Fail-Safes chapter. A middle row holds three white behavior and perception boxes: Navigation, annotated 20 hertz approach-point streamer; Harass, annotated orbit generator from the Harassment Pattern Generation chapter; and Detector, annotated 10 hertz Target publisher fed by the Part III vision pipeline via the detections underscore 3d topic. Below center sits a teal Flight Controller box annotated the only node that issues vehicle commands, every command verified against vehicle state, 5 hertz FcStatus publisher; at the bottom a dark Drone box representing PX4 and the physical aircraft publishing fmu out telemetry. Amber service arrows: operator to Mission Manager labeled goose start underscore mission and goose abort; Mission Manager to Flight Controller labeled a bundle of Trigger services goose fc preflight, arm, mode underscore mission, mode underscore offboard, mode underscore hold, rtl, land. Teal topic arrows: Mission Manager to Navigation and Harass labeled goose active underscore controller, a String naming none, navigation, or harassment, republished at 1 hertz; Detector to Mission Manager, Navigation, and Harass labeled goose target, a Target message at 10 hertz carrying valid and confirmed flags; Navigation to Mission Manager labeled goose nav underscore arrived Bool and goose nav underscore distance Float32; Harass to Mission Manager labeled goose harass underscore done Bool; Failsafe to Mission Manager labeled goose failsafe, a String encoding severity colon reason republished while active; Flight Controller to Mission Manager labeled goose fc underscore status, an FcStatus aggregate at 5 hertz including in underscore mission, in underscore loiter, in underscore offboard booleans. Grey arrows connect the Drone's fmu out telemetry to Flight Controller, Detector, Navigation, Harass, and Failsafe, and carry Navigation and Harass setpoint streams, offboard control mode plus trajectory setpoint, gated by the token, and the Flight Controller's verified vehicle commands down to the Drone. A legend distinguishes service, topic, and PX4 bridge edges, and a footer reads: a service edge means the caller waits for proof; a topic edge means the publisher asserts and moves on; the token is the only thing standing between three setpoint authors and one autopilot.
Figure 25.1: The Goose Management System’s full ROS 2 node graph: amber edges are services (request–response, returning only on confirmed completion), teal edges are topics (continuous streams at the labeled rates), and grey edges are PX4’s own uORB-bridge telemetry. The Mission Manager holds mission logic; the Flight Controller alone commands PX4; the behavior nodes act only while the active-controller token names them.
Table 25.1: The seven nodes of the shipped system, their real interfaces, and where each node’s internals are treated in depth.
Node Responsibility Services Key topics Internals covered in
Mission Manager (mission_manager.py) Sequences the whole mission; sole writer of the token; owns every state transition Offers /goose/start_mission, /goose/abort; holds seven flight-controller service clients (all but disarm) Publishes /goose/active_controller; subscribes to /goose/fc_status, /goose/target, /goose/nav_arrived, /goose/harass_done, /goose/failsafe This chapter; failsafe path in Geofencing and Fail-Safes
Flight Controller (flight_controller_node.py) Sole PX4 command author; executes and reports, never decides Offers /goose/fc/preflight, arm, disarm, mode_mission, mode_offboard, mode_hold, rtl, land Publishes /goose/fc_status (5 Hz); subscribes to PX4’s /fmu/out/* telemetry PX4 and the Autopilot Stack (the bridge); this chapter (the service pattern)
Navigation (navigation_node.py) Flies the approach to a confirmed target at a standoff — Publishes setpoints (20 Hz, token-gated), /goose/nav_arrived, /goose/nav_distance; subscribes to /goose/target, token Path Planning on the Edge (the mechanics); this chapter (the approach role)
Harass (harassment_node.py) Orbits the target on station; reports pattern completion — Publishes setpoints (token-gated), /goose/harass_done; subscribes to /goose/target, token Harassment Pattern Generation
Detector (detection_node.py) Turns raw 3D detections into a confirmed, world-frame target estimate — Publishes /goose/target (10 Hz); subscribes to /detections_3d, vehicle attitude and position Part III (the vision pipeline); Coordinate Frames (the conversion)
Failsafe (failsafe_monitor.py) Independent mission-rule watchdog; detects, never acts — Publishes /goose/failsafe (1 Hz while a condition holds) Geofencing and Fail-Safes
Drone (PX4 + airframe) Flies; publishes telemetry; enforces its own native failsafes — Publishes /fmu/out/*; consumes commands and setpoints on /fmu/in/* Parts I–II

A few integration-level observations the per-node chapters could not make, because they only appear when the graph is whole.

The Mission Manager is a client of everything and a server of almost nothing. It offers exactly two services — /goose/start_mission and /goose/abort, the operator’s two buttons — and consumes everything else: five topic subscriptions and seven service clients into the Flight Controller. That asymmetry is the design. The design document behind the shipped code calls this node “the brain… it decides what the drone should be doing and who’s in charge of steering,” and a brain’s interface should be thin: two ways for a human to address it, and one topic (the token) by which it delegates. It runs a 10 Hz decision tick — the production counterpart of the Finite State Machines chapter’s 20 Hz teaching tick — plus the 1 Hz token republish.

The Flight Controller is the mirror image: a server of everything and a decider of nothing. Eight Trigger services, one aggregated status topic, and — by its own design document’s emphatic rule — no mission logic: “it executes and reports, never decides.” Everyone who wants the aircraft to do something asks this node; this node asks PX4; and PX4’s telemetry, cached from the /fmu/out/* subscriptions (vehicle status, local position, battery, GPS, home position, land-detected, attitude), is the only evidence it will accept that anything happened. One implementation detail matters at the graph level: the node runs a MultiThreadedExecutor with four threads and a reentrant callback group, so a service call that spends up to several seconds retrying and polling for mode confirmation does not stall the 5 Hz FcStatus publisher — the request–response verb and the streaming verb coexist in one node only because the node was built to service both concurrently. The FcStatus aggregate it publishes distills PX4’s raw nav_state enum into booleans the mission logic can read directly — in_mission, in_loiter, in_offboard — and one of those booleans, as the mission walk below shows, is how the system knows its search is finished.

The behavior nodes are deliberately interchangeable in shape. Navigation and Harass have the same silhouette: subscribe to the target and the token, publish Offboard setpoints only while the token names them, and report their one fact upward — /goose/nav_arrived (plus a continuous /goose/nav_distance, the “how far to target” stream) for Navigation, /goose/harass_done for Harass. Neither offers a service; neither is asked to do anything. They are delegated to via the token and report via topics, and the Harassment Pattern Generation chapter’s rule — the node reports and keeps flying until authority is taken from it — is now visible as a property of the whole graph: behavior nodes have no verb for “stop yourself,” on purpose. Navigation’s internals are the Offboard streaming machinery the Path Planning on the Edge chapter taught — stream a setpoint, test arrival against a radius — pointed at a computed approach point (a 3.0 m standoff along the target-to-drone line, at 3.0 m engage altitude, arrival declared inside 2.0 m horizontal and 1.5 m vertical, at 20 Hz) rather than at a lawnmower list.

The Detector is a translator, not a controller. Its input is /detections_3d — the Detection3DArray stream from the Part III vision pipeline, already carrying real depth from the RealSense camera — plus the vehicle attitude and position it needs to run frames.optical_to_ned_point(), the Coordinate Frames chapter’s own function, in production, exactly as that chapter said it would be. Its output is one Target message at 10 Hz. It never touches the token, never streams a setpoint, and keeps publishing regardless of who is flying — perception, in this architecture, is a continuous assertion about the world, and control is somebody else’s job. Per frame it keeps only the single highest-confidence detection of the target class above a 0.6 confidence floor, feeding a 64-entry rolling buffer of timestamped world-frame hits; what it does with that buffer is the next section’s subject.

The Failsafe node detects and the Mission Manager acts — the split the Geofencing and Fail-Safes chapter built in full. Here it suffices to place the edge: /goose/failsafe, a String encoding SEVERITY:reason, republished at 1 Hz while a condition holds, consumed by exactly one subscriber. Why this is the right topology is the argument of this chapter’s penultimate section.

25.5 Two Booleans on One Topic: valid and confirmed

The Target message carries the goose’s estimated NED position — and two separate booleans, and the distinction between them is one of the nicest small designs in the repository. valid means something was detected recently: at least one hit landed in the buffer within the last stale_timeout_s (default 2.0 s). confirmed means the full confirmation criteria are met on top of validity: at least confirm_count (default 5) hits within the last confirm_window_s (default 2.0 s), and the most recent five of them clustered — every one within cluster_radius_m (default 3.0 m) of their own median. The position the message reports is itself a running median of the recent hits, continuously updated as the target moves rather than locked to the first confirmed fix. A single glimpse makes a target valid; only a short, spatially consistent track makes it confirmed.

The payoff is in who reads which flag. The Mission Manager’s MISSION state checks confirmed to decide whether to begin a new engagement — committing the aircraft to leave its search, switch flight modes, and close on a wild animal is exactly the kind of decision that should require five clustered detections in two seconds, not one hopeful bounding box. But its engagement watchdogs — the shared logic that runs during ENGAGE and HARASS — check valid to decide whether the target has been lost: once an engagement is underway, the question is no longer “are we sure enough to start?” but “is there still anything there at all?”, and demanding re-confirmation every window would end engagements every time the detector’s cluster test hiccupped during a fast, banking approach. A target must clear a high bar to start an engagement and only a low bar to sustain one — hysteresis, the Finite State Machines chapter’s own remedy for chattering guards, implemented here not as two thresholds on one number but as two booleans on one message. And even the low bar has teeth: if valid stays false for more than target_lost_timeout_s (default 5.0 s) mid-engagement, the watchdog fires and the mission resumes its search rather than orbiting an empty point.

25.6 One Mission, End to End

Here is the chapter’s central artifact: one complete mission execution flow, traced through the real mission_manager.py’s state machine with the actual trigger for every transition. The real St enum has eleven states — IDLE, PREFLIGHT, ARMING, START_MISSION, MISSION, ENGAGE_PREP, ENGAGE, HARASS, RESUME, FAILSAFE, DONE — and Figure 25.2 draws the flow with each edge labeled by what fires it: a service call succeeding, or a topic crossing a threshold. Those are the only two trigger species in the whole machine, which is itself a lesson — the state machine is exactly as expressive as the communication graph beneath it.

Flow diagram titled one mission, end to end, the real eleven-state machine of mission manager dot py. A top row of five rounded boxes runs left to right: IDLE, PREFLIGHT, ARMING, START underscore MISSION, and MISSION, connected by amber arrows labeled with their triggers: operator calls the goose start underscore mission service; the preflight service returns success, GPS fix with at least eight satellites, battery at least fifty percent, valid local position, home position set; the arm service returns success, confirmed by PX4 telemetry; the mode underscore mission service returns success, PX4 enters AUTO.MISSION. From MISSION a teal arrow exits right to DONE labeled fc underscore status shows in underscore mission seen, then in underscore loiter: grid complete, loiter after mission. A second row shows the engagement loop: MISSION drops via a teal arrow labeled target dot confirmed true on goose target and the 20 second cooldown elapsed, token set to navigation, into ENGAGE underscore PREP; an amber arrow labeled after a 1 second offboard warmup the mode underscore offboard service returns success leads to ENGAGE; a teal arrow labeled goose nav underscore arrived reads true, token set to harassment leads to HARASS; a teal arrow labeled goose harass underscore done reads true, token set to none leads to RESUME; and an amber arrow labeled the mode underscore mission service returns success, a 20 second cooldown starts curves back up into MISSION. A watchdog note attached to ENGAGE and HARASS reads: shared engagement watchdogs divert to RESUME if 90 seconds elapse or the target stays invalid more than 5 seconds. An amber FAILSAFE box sits at the lower right with a dashed border, entered from any state on a goose failsafe message of severity colon reason, or by escalation when RESUME fails three times, reason RTL colon cannot resume mission; inside it the enter failsafe logic parses the severity, sets the token to none, and calls the matching land, rtl, or hold service, standing down entirely on an unrecognized severity. Two shaded background bands label the PX4 flight mode per phase: AUTO.MISSION behind MISSION and RESUME, and OFFBOARD behind ENGAGE underscore PREP, ENGAGE, and HARASS. A footer maps the five pedagogical states of the Finite State Machines chapter onto the real machine: IDLE slash TAKEOFF unfolds into IDLE, PREFLIGHT, ARMING, and START underscore MISSION; SEARCH underscore GRID corresponds to MISSION plus RESUME; HARASS underscore TARGET unfolds into ENGAGE underscore PREP, ENGAGE, and HARASS; LAND slash RETURN-TO-HOME corresponds to FAILSAFE and DONE; and EVADE underscore THREAT has no counterpart in the shipped code, it is the book's own scaffolding on the failsafe mechanism.
Figure 25.2: The real mission execution flow: eleven states, each transition triggered by a service call succeeding (amber) or a topic crossing a threshold (teal). Shaded bands show which PX4 flight mode holds during each phase — the autopilot’s native AUTO.MISSION flies the search, and Offboard streaming takes over only for the reactive engagement loop.

Startup is a ladder of services. The machine idles until an operator calls /goose/start_mission. PREFLIGHT then calls the Flight Controller’s preflight service — which checks PX4’s own preflight flags, a GPS fix (fix type ≥ 3, at least 8 satellites by default), a valid local position, battery at or above a 50% launch floor, and a recorded home position — and advances only on success. ARMING calls arm and waits for PX4-confirmed armed state. START_MISSION calls mode_mission, and when PX4’s telemetry confirms the vehicle is in AUTO.MISSION, the machine settles into MISSION. Three states, three service calls, and this is the first place the real machine’s extra granularity over the Finite State Machines chapter’s teaching machine earns its keep: that chapter’s IDLE/TAKEOFF folded “getting airborne” into one box, which was the right pedagogical altitude — but operationally, preflight validation, arming, and the mode switch are three distinct requests with three distinct failure modes, each of which can independently time out or be refused, and each therefore needs its own state so its failure has its own name in the log and its own recovery path. The teaching machine merged what the mission never decides differently between; the production machine splits what fails differently.

The search is a topic-watch. In MISSION, PX4’s onboard executor flies the pre-uploaded survey — the companion computer streams nothing (the next section dwells on this). The Mission Manager just watches two streams. On /goose/fc_status, it looks for the end-of-search signature: in_mission has been true at some point and in_loiter has become true — PX4 auto-loiters when an uploaded mission completes, so “mission mode gave way to loiter” is “grid complete,” and the code logs it in exactly those words. That path leads to DONE and the flight’s wind-down. On /goose/target, it watches for confirmed — and when a confirmed target arrives and the post-engagement cooldown window (default 20.0 s) has elapsed, it sets the token to 'navigation' and enters ENGAGE_PREP.

The engagement is a loop of both verbs. ENGAGE_PREP exists because of a PX4 rule the Mission Scripting chapter established: Offboard mode requires a live setpoint stream before the mode switch. So the state waits out an offboard_warmup_s grace period (default 1.0 s) — Navigation, newly named by the token, is already streaming its approach setpoints — then calls mode_offboard, and on PX4-confirmed success advances to ENGAGE. This split is the second place the real machine is finer-grained than the teaching one: the pedagogical HARASS_TARGET folded “get control in Offboard” and “actively track” into one state, but warming up a setpoint stream and closing on a live target are different jobs with different failure modes, and the shipped code gives each its own box. ENGAGE then watches /goose/nav_arrived; when Navigation reports arrival at the standoff point, the token flips to 'harassment' and the machine enters HARASS. HARASS watches /goose/harass_done; when the orbit’s fixed duration elapses (the Harassment Pattern Generation chapter’s time-latch), the machine enters RESUME. Throughout ENGAGE and HARASS, the shared engagement watchdogs run: 90 seconds of total engagement, or 5 seconds of the target being invalid, diverts the machine to RESUME early — bounded commitment, enforced by topic-watching.

RESUME closes the loop, and its failure path is the escalation. RESUME sets the token back to 'none', re-calls mode_mission to hand the aircraft back to PX4’s onboard executor mid-survey, and on success starts the 20-second cooldown that keeps a resumed search from instantly re-engaging the same goose it just harassed. If the resume service fails three times running, the machine escalates through the same path any emergency takes: _enter_failsafe('RTL:cannot resume mission') — parse the severity, zero the token, call the matching flight-controller service — the exact function, and the exact severity protocol, the Geofencing and Fail-Safes chapter read in full.

One box deserves an explicit disclaimer, because the book put it there. The shipped machine has no EVADE_THREAT state. The Vector-Based Threat Evasion chapter’s escape machinery, and the five-state machine’s dashed override arrows into it, are the book’s own synthesis — original scaffolding built on top of the real system’s failsafe skeleton, teaching reactive-safety design the shipped code handles more bluntly (its FAILSAFE state is terminal and severity-driven, not a computed-evasion-then-resume state). The real machine’s eleven states are the ground truth of what flies; EVADE_THREAT is what the book added to teach what a richer reactive layer would look like. A reader who greps the repository for it should not be surprised to find nothing — and should now be able to say precisely where it would attach if it were built: a new severity in the SEVERITY:reason protocol, or a new state between the engagement watchdogs and RESUME.

25.8 One Failsafe Node, Not N Safety Checks

The third learning objective asks for an argument, so here is the design question stated plainly: the system has three nodes that can command the aircraft’s motion — Navigation, Harass, and (through mode changes) the Mission Manager. Why is safety monitoring none of their jobs? Why one independent Failsafe node, when each behavior node could simply check the battery and the fence before each setpoint?

Start with what the centralized mechanism actually is, because its smallness is the argument’s first half. The Failsafe node — the Geofencing and Fail-Safes chapter’s failsafe_monitor.py, unchanged — watches telemetry and publishes SEVERITY:reason strings; it holds no mission state and flies nothing. The Mission Manager’s _enter_failsafe does the acting, in three moves that chapter traced: parse the severity prefix, set the active controller to 'none' — which silences every behavior node within one message, because their ticks begin by checking the token — and dispatch to the matching flight-controller service (land, rtl, or hold); an unrecognized severity dispatches to nothing, the manager logging that it is standing down because a PX4-native failsafe outranks it. Detection in one node, execution in another, actuation in a third: the entire safety-override machinery of a seven-node system is one string protocol, one token write, and one service call.

Now price the alternative. Fold safety into each behavior node and the system needs the battery-floor comparison, the fence check, the severity ordering, and the “am I outranked by PX4?” deference implemented in Navigation, and in Harass, and in every behavior node anyone adds later — \(N\) implementations of logic whose whole value is being correct, which means \(N\) places to get the threshold comparison backwards, \(N\) parameter sets to keep synchronized when the field crew tightens the fence radius, and \(N\) opportunities for the next contributor’s new node to simply forget. Worse than the duplication is the correlation: a behavior node checking its own safety shares every failure mode with the behavior it is supposed to police — the hung process that is streaming stale setpoints is the same hung process that was supposed to notice the fence, so the check is least alive exactly when it is most needed. And worst is what scattering does to override authority: safety enforced inside each node is safety enforced by consent — it works only if the node executes its own check and acts on it — where safety enforced at the token is enforced by decree: the Mission Manager revokes control at a channel the behavior nodes cannot ignore, because not-being-named is tested before their tick does anything at all. The centralized design means every behavior node gets to be simple — Navigation’s entire safety obligation is one line, check the token — and the system’s fault tolerance concentrates in one small, independent, separately-crashable process whose logic is written once, reviewed once, and parameterized once. The Geofencing and Fail-Safes chapter built the ladder; seen from the whole graph, the ladder is also an economy: one place for override logic to be right beats \(N\) places for it to be wrong.

25.9 When the README Disagrees with the Code

One more thing surfaces only at integration time, and it opens this Part’s diagnostics theme perfectly: the repository’s own README, whose Services-versus-Topics framing this chapter happily adopted, drifts from the code it describes. The drift is ordinary, instructive, and worth cataloguing, because diffing documentation against implementation is a diagnostic skill every systems engineer needs.

The README lists the Mission Manager’s services as “Start Mission, End Mission” — the code implements /goose/start_mission and /goose/abort; there is no “end mission” service, and the README’s phrase is best read as a loose description of the abort. It lists a Flight Controller service called “Start Flight” — the real service is preflight, which validates readiness and reports; it commands no flight at all, and the rename matters because a reader who believed the README might think calling it launches the aircraft. Its “Change Flight Mode” is really mode_mission specifically — the other mode changes (mode_offboard, mode_hold, rtl, land) are each their own separate service, as the README’s own list elsewhere admits. The README’s “Navigation Status” and “Distance to Target” are two real topics, not one (/goose/nav_arrived and /goose/nav_distance). Its mission-flow narrative says the Flight Controller “arms the drone and switches to Offboard mode” as step two — which describes the engagement phase; the search phase, as this chapter established, flies in AUTO.MISSION. Even the code comments drift: px4_defs.py’s comment on the active-controller topic says its values are none|detection|harassment, but the values the Mission Manager actually publishes are 'none', 'navigation', and 'harassment' — the Detector never takes the token, because perception in this architecture never controls anything. And the design documents drift furthest of all, in the honest direction the Harassment Pattern Generation chapter already flagged for the orbit parameters: the detection design discussion proposed an 80% confidence bar over 3 frames; the shipped defaults are 0.6 over 5 hits in 2 seconds.

None of this is scandal — it is entropy, the normal decay of prose that nobody’s tests exercise. The lesson is methodological, and this book has practiced it all along without naming it: the code is the authority for what exists; documentation is the authority only for intent. The README’s conceptual framing (services command, topics stream) is accurate and valuable; its inventory is stale. When this chapter states a topic name, a default, or a rate as fact, the fact came from px4_defs.py and the node sources — and when the reader stands up the real graph in the Try It below, ros2 topic list and ros2 service list will arbitrate every claim in this chapter the same way.

25.10 Check Your Understanding

  1. Classify each of the following as a Service or a Topic in the running example’s architecture, and justify each with the chapter’s test (“does the sender need to know the action happened before safely proceeding?”): (a) arming the vehicle; (b) the Detector’s target estimate; (c) the operator’s abort; (d) the Flight Controller’s aggregated status.
  1. Service — /goose/fc/arm: the Mission Manager must not proceed to takeoff logic until arming is confirmed, so it needs a blocking, acknowledged transaction, and the Flight Controller strengthens the acknowledgement by polling PX4 telemetry before returning success. (b) Topic — /goose/target at 10 Hz: perception is a continuously true assertion about the world with multiple independent consumers (Mission Manager, Navigation, Harass), none of which should reply, and it must keep flowing regardless of who is flying. (c) Service — /goose/abort: an abort demands acknowledged execution; an operator must know it was received and acted on. (d) Topic — /goose/fc_status at 5 Hz: vehicle state is streamed, not requested, so any node (or a late-joining diagnostic tool) can consume it without the Flight Controller knowing or caring who listens.
  1. During MISSION, a goose walks into frame and the Detector begins landing hits at 5 Hz. Using the shipped defaults — confirm_count = 5 within confirm_window_s = 2.0 s, Target published at 10 Hz, Mission Manager tick at 10 Hz, offboard_warmup_s = 1.0 s — estimate the minimum time from the first detection hit to the mode_offboard service call, assuming the cluster test passes, the cooldown has long expired, and services respond instantly. Which term dominates?

Five hits at 5 Hz span four inter-hit intervals: \(4 \times 0.2 = 0.8\) s from first hit to fifth (comfortably inside the 2.0 s window). The Detector’s next 10 Hz publish adds up to 0.1 s; the Mission Manager’s next 10 Hz tick adds up to another 0.1 s before it reads confirmed, flips the token, and enters ENGAGE_PREP. ENGAGE_PREP then waits out the full 1.0 s offboard warmup before calling mode_offboard. Total: roughly \(0.8 + 0.1 + 0.1 + 1.0 \approx 2.0\) s. The confirmation accumulation (0.8 s) and the warmup (1.0 s) dominate, and both are deliberate: the first buys false-positive immunity before committing the aircraft, the second satisfies PX4’s requirement that a live setpoint stream precede the Offboard mode switch. The messaging latencies (~0.2 s combined) are noise by comparison — tuning tick rates would buy almost nothing.

  1. The engagement watchdogs read target.valid, but the MISSION state reads target.confirmed. Predict the two concrete misbehaviors that would appear if a maintainer “simplified” the code to use a single boolean — first if everything read confirmed, then if everything read valid.

Everything reads confirmed: engagements become fragile. During a fast approach the detector’s view geometry changes quickly, and the five-hit cluster test can momentarily fail (hits spread beyond the 3.0 m cluster radius, or fewer than five land in the window) even though the goose is plainly still there; with the watchdog demanding confirmed, such lapses lasting over 5 s would abort healthy engagements, and shorter ones would erode the timeout margin — the aircraft gives up on targets it is still tracking. Everything reads valid: engagements become trigger-happy. A single spurious detection — one misclassified rock — makes the target valid for 2.0 s, and the MISSION state would launch a full engagement (mode switch, approach, orbit) on one frame of evidence, exactly the false-positive commitment the five-clustered-hits bar exists to prevent. The two-boolean design is hysteresis: a high bar to start, a low bar to continue, each read by the decision that needs it.

  1. The Flight Controller’s mode_mission service sends VEHICLE_CMD_DO_SET_MODE, polls for up to 2 s per attempt until VehicleStatus.nav_state actually reads AUTO_MISSION, and retries the whole cycle up to three times before giving up. A teammate proposes returning success as soon as the command is transmitted, to make the mission “snappier.” Using the RESUME state as your example, explain what specifically breaks.

RESUME’s job is to hand the aircraft back to PX4’s onboard executor: it zeroes the token (silencing the behavior nodes’ setpoint streams) and calls mode_mission. With send-equals-success, a command PX4 rejects or drops — radio hiccup, mode-transition refusal — still returns true: the Mission Manager believes the search has resumed, starts its 20 s cooldown, and settles into MISSION watching fc_status for completion, while the actual aircraft sits in whatever mode it fell into with nobody streaming setpoints and no state responsible for noticing. The escalation path breaks too: the real code counts three failed resume attempts and then enters failsafe with RTL:cannot resume mission — but a service that cannot fail can never trip that counter, so the safety net for exactly this scenario is unreachable. Command verification is what makes a service’s return value mean something; without it, every state transition triggered by “service succeeded” is triggered by hope.

  1. The companion computer’s mission process hangs for eight seconds. Compare the outcome if this happens (a) mid-MISSION, during the search sweep, and (b) mid-ENGAGE, during an approach — and connect the difference to the architectural choice of AUTO.MISSION for the search phase.
  1. Mid-MISSION: nothing happens to the flight. PX4’s onboard executor is flying the uploaded survey and neither needs nor notices the companion computer; when the process recovers, fc_status and target are streams it simply resumes consuming, and the mission continues. (b) Mid-ENGAGE: the aircraft is in Offboard mode, fed by Navigation’s 20 Hz setpoint stream. Eight seconds of silence trips PX4’s Offboard timeout — the dead-man’s switch from Part II — and the autopilot’s native failsafe takes the aircraft (per configuration: hold, RTL). That is a safe failure, but it is a flight-mode event with the mission lost. The asymmetry is the argument for the split: the search occupies most of each flight, so flying it in AUTO.MISSION means the system spends most of its airborne time immune to companion-computer failure, and accepts Offboard’s fragility only for the short engagement windows where per-tick reactivity to a moving animal is the entire point and no onboard mode can provide it.
  1. A new contributor adds a “Herding” behavior node (the Multi-Agent Systems chapter’s herding-vector hold) to the shipped graph. List everything the node must implement to be a well-behaved citizen of this architecture — and everything it does not need to implement, with the design feature that excuses each omission.

Must implement: subscribe to /goose/active_controller and gate every tick on being named (the token contract); subscribe to /goose/target for its herding geometry; stream Offboard setpoints only while named, and keep streaming until authority is taken (never stop itself); publish one completion/progress topic upward (a herding_done in the style of /goose/harass_done) for the Mission Manager to watch; and the Mission Manager needs a new token value plus the state logic to grant and revoke it. Does not need: any safety logic — battery, fence, severity ordering — because the Failsafe node detects and _enter_failsafe revokes at the token, which the new node already honors by construction; any service server — behavior nodes are delegated to via the token, not asked; any PX4 command authority — mode changes stay the Flight Controller’s monopoly; and any knowledge of the other behavior nodes — publish/subscribe decoupling means Navigation and Harass never learn a sibling exists. The short must-list and the long need-not-list are the architecture’s dividend: the token, the failsafe split, and the service monopoly let every new behavior be almost as simple as its geometry.

25.11 Try It

Part 1 — stand the graph up and interrogate it. Bring up the running example’s node graph against PX4 SITL in Gazebo — the shipped nodes if you have the repository, or your own Part II–IV implementations of each role (the interfaces in Table 25.1 are the spec). Before flying anything, audit the architecture with ROS 2’s own introspection: ros2 node list should show seven-ish processes; ros2 service list should show /goose/start_mission, /goose/abort, and the eight /goose/fc/* endpoints; ros2 topic list should show the eight mission topics — and every name should match this chapter’s claims, not the README’s (this is the documentation-diff exercise run live). Then fly one full mission: call /goose/start_mission from the command line with ros2 service call, and keep ros2 topic echo /goose/active_controller and the Mission Manager’s log side by side. The acceptance test is the trace: the log should walk IDLE → PREFLIGHT → ARMING → START_MISSION → MISSION, the token should read none throughout the sweep; script a goose into the detector’s view and watch the full engagement loop — token to navigation at ENGAGE_PREP, mode_offboard succeeding after the warmup, token to harassment on nav_arrived, RESUME re-entering AUTO.MISSION with the token back to none. For every transition in the log, write down which of the two trigger species fired it — a service returning success, or a topic crossing a threshold — and check it against Figure 25.2. Finish with the timeout tour: hide the goose mid-approach and confirm RESUME fires at the 5 s target-lost watchdog; let an engagement run long and confirm the 90 s ceiling; re-present the goose immediately after RESUME and confirm the 20 s cooldown holds the machine in MISSION.

Part 2 — build the field diagnostic. Write a small mission_dashboard node — the tool a field crew would actually want — that subscribes to /goose/fc_status, /goose/active_controller, /goose/target, /goose/nav_distance, /goose/harass_done, and /goose/failsafe, and renders one status line per second: current controller token, in_mission/in_loiter/in_offboard flags, target valid/confirmed with its age, distance to target, and any active failsafe string. Two requirements carry the chapter’s lessons. First, staleness is a display state: stamp every message on arrival and render any stream older than three of its own publish periods as STALE in place of its last value — a dashboard that shows dead data as live data is the “wrong with confidence” failure of the Finite State Machines chapter, in miniature. Second, the dashboard must be a pure subscriber: no services called, no setpoints, nothing published — verify with ros2 node info /mission_dashboard that it could not command the aircraft if it wanted to, and note that publish/subscribe decoupling is what let you bolt a whole new observer onto a running system without touching one line of the seven nodes it watches. Then use it: replay Part 1’s mission and confirm the dashboard tells the same story as the logs — and kill the Detector process mid-engagement to watch valid go stale on your display roughly 5 s before the Mission Manager’s watchdog acts on the same silence.

Related Lab: See Appendix B for the lab assignment that applies this chapter’s concepts.

The system is now assembled — and this Part can begin its real work, which is living with it. A graph this size fails in ways no single node’s chapter could show: a topic that streams stale truth, a service that times out only in the field, a log that must be read backwards from an incident. And a system that autonomously pursues wild animals owes an account of itself that no architecture diagram can give. Diagnosing the first and reckoning with the second are where Part V goes from here — with, at last, the whole machine on the bench.