# ConnectionMonitoring

Data model plugin (`libconnmon.so`) for TR-181 `Device.ConnectionMonitoring.`,
loaded by bbfdm's `dm-service` as a micro-service. For the object tree and
parameter semantics, see the TR-181 2.21 spec — this document covers design and
implementation, not the spec itself.

## Split between this repo and the engine

This repository is the data model side only: config ownership, AddObject/
DeleteObject, and status readback. It contains no probing or recovery-action
logic. That runs as a separate process — the connmon *engine*, a shell daemon
plus per-method/per-action scripts — shipped by the `connmon` feed package
(`feeds/iopsys/connmon`), not by this repository.

The two processes never call each other directly. They share only UCI:

- `/etc/config/connmon` — persistent config, owned by the data model, read by
  the engine.
- `/var/state/connmon` — a varstate package, owned and written exclusively by
  the engine every tick; the data model only reads it
  (`dmuci_get_option_value_string_varstate`). A missing/not-yet-written file is
  not an error — reads fall back to a per-parameter default, so there's no
  boot-ordering dependency between the two processes.

This split means the actual probe methods (ARP/ND/Ping/BFDEcho/DNS) and
recovery actions (RestartDHCP/RestartInterface/Reboot) can be replaced per
deployment by swapping shell scripts, without touching the data model or
rebuilding anything.

## Engine design

The engine (`/usr/sbin/connmon`, in the feed package) is one long-running
shell script, not a cron job that gets invoked from scratch every so often.
It wakes up, checks whatever connections are due, goes back to sleep for as
long as it can, and repeats — forever. Being one continuously-running process
(rather than a fresh process per check) is what lets it remember things like
"how many times has `wan` failed in a row" from one check to the next.

- **What a "tick" is, and the state machine per connection.** Each pass
  through the loop is called a tick. On every tick, the engine looks at each
  enabled connection and asks "is it due for a check yet?" If not, it leaves
  it alone. If it is due, it runs that connection's test (e.g. ping the
  gateway) and reacts:
  - Test passes -> mark it "Up", reset its failure counter, and don't check
    it again until the next full `MonitorInterval` (e.g. 30s).
  - Test fails -> mark it "Unresponsive", bump its failure counter by one,
    and check it again much sooner, after `FailInterval` (e.g. 15s) instead
    of the full interval — this is what "escalation" means here: the more a
    connection is failing, the more closely it gets watched.
  - Once the failure counter reaches an Action's configured threshold
    (`AttemptCount`), that Action's recovery script runs (e.g. restart the
    interface, or reboot the device).

  All of these numbers (the failure counter, when each connection is next
  due, lifetime totals) are not just kept in the script's memory — they are
  written out to `/var/state/connmon` at the end of every tick and read back
  in at the start of the next one. That sounds redundant for a process that
  never stops running, but it means if the engine ever crashes and procd
  restarts it, it picks up exactly where it left off instead of forgetting
  everything and starting from zero.

  This is what the engine decides for one connection, once per tick:

  ```mermaid
  flowchart TD
      A["Tick: look at one connection"] --> B{"Enabled?"}
      B -- "no" --> C["Status: Disabled<br/>attempts = 0"]
      B -- "yes" --> D{"Due for a check yet?<br/>(has MonitorInterval /<br/>FailInterval passed?)"}
      D -- "not yet" --> E["Leave it as-is,<br/>check again next tick"]
      D -- "yes, run the test" --> F["Probe the connection<br/>(ARP / ND / Ping / BFDEcho / DNS)"]
      F -- "reachable" --> G["Status: Up<br/>attempts reset to 0<br/>next check in MonitorInterval"]
      F -- "unreachable" --> H["Status: Unresponsive<br/>attempts += 1<br/>next check sooner, in FailInterval"]
      H --> I{"attempts reached an<br/>Action's AttemptCount?"}
      I -- "yes" --> J["Run that Action<br/>(e.g. restart interface, reboot)"]
      I -- "no" --> K(("Wait for next tick"))
      J --> K
      C --> K
      E --> K
      G --> K
  ```

  So `Up` and `Unresponsive` aren't separate modes so much as "last test
  passed" vs. "last test failed" — the only real difference in behavior is
  the failed side re-checks sooner (`FailInterval` instead of the longer
  `MonitorInterval`) and counts up towards triggering an Action.

- **Noticing a settings change without being told.** A user can change a
  connection's settings (say, switch its `Method` from Ping to DNS) at any
  time via `uci set`, completely independently of the engine's own tick loop.
  The engine has no direct notification for "this one setting changed" —
  instead, on every tick it recomputes a short fingerprint string from all
  the settings that affect how a connection is tested (interface, method,
  destination, timeout, etc.). If that fingerprint doesn't match the one it
  saved last tick, it knows something relevant changed underneath it, and it
  resets that connection's failure counter and schedule so old results don't
  get judged against new settings. Lifetime counters (total failures, total
  actions taken) are deliberately *not* reset by this, since those are meant
  to survive until the next reboot regardless of config changes.

- **Sleeping smartly instead of polling every second.** The simplest possible
  design would be "wake up once a second, forever, and check whatever's due."
  That wastes CPU (and, on a battery/thermally-constrained router, power) for
  no benefit, since most seconds nothing is due. Instead, after each tick the
  engine works out the soonest moment any connection will next need
  checking, and sleeps exactly until then (capped between a floor and a
  ceiling, so it never sleeps too briefly or too long even if something odd
  happens with the numbers). This is the outer loop the per-connection
  flowchart above sits inside:

  ```mermaid
  flowchart TD
      T["run_tick():<br/>check every due connection<br/>(the flowchart above,<br/>run once per connection)"] --> N["Find the soonest time any<br/>*enabled* connection is next due"]
      N --> M["Clamp that to between<br/>a minimum and a maximum wait"]
      M --> S["Sleep in the background<br/>for that long"]
      S -- "timer runs out" --> T
      S -- "config changed —<br/>SIGHUP arrives" --> Q["Sleep abandoned early"]
      Q --> T
  ```

  Two details behind that diagram:

  - Disabled connections are deliberately left out of "find the soonest time
    any enabled connection is next due". They're internally marked as due
    "immediately", so including them would make the engine think something
    is always due right now and check every second forever, defeating the
    whole point of sleeping smartly.
  - The "config changed" shortcut is a Unix signal, `SIGHUP`, sent to the
    engine by the init script whenever the config is changed. The engine
    reacts to it with a `trap` — a handler that runs no matter where in the
    loop the engine currently is — which just sets a flag and lets the
    current sleep end immediately instead of waiting it out, so a config
    change (e.g. re-enabling a connection, or changing its `Method`) takes
    effect on the very next tick rather than after however long was left to
    sleep.

- **Cleaning up after deleted connections.** When a connection (or one of
  its actions) is removed from the config, its old entry in
  `/var/state/connmon` doesn't disappear by itself — nothing else deletes it
  automatically. Rather than checking for this on every single tick (which
  would mean re-scanning that whole file every second just in case), the
  engine remembers the list of connection names it saw last tick and only
  bothers doing that cleanup scan on a tick where the list has actually
  changed.
