Two telemetry paths, one truth

A dead host is not a witness

In-band monitoring has a structural flaw: the reporter dies with the patient. When a kernel panics, a bus hangs, or power drops, the agent on the node stops reporting at the exact moment its report matters most. Any monitoring story that starts and ends with an agent is a story about healthy nodes.

So Lilac watches every node over two independent paths.

Out-of-band, the BMC speaks Redfish over a dedicated management network, in its own power domain. It reports power draw, temperatures, fan speeds, PSU and DIMM health, and boot progress. It keeps answering while the host is crashed, hung, mid-reimage, or powered off. During a power event it is the only voice a node has.

In-band, an agent on the node reports what the BMC cannot see: GPU utilization and clocks, Xid events as they happen, ECC error trends, NCCL health, disk errors, NIC counters, driver state. This is the workload's view of the machine. A tenant does not experience fan speeds. A tenant experiences a GPU that fell off the bus mid-step.

The two paths also audit each other. If the agent goes silent while the BMC reports a powered, booted host, that is an agent or network problem, and it gets fixed without touching the tenant. If both go silent, that is a node problem, and the clock on it starts immediately. When both paths report the same event, a thermal climb from the BMC and sagging clocks from the agent, the record has two independent witnesses. SLA-grade operations need that corroboration, because each path alone always has an alibi.

Three layers of detection

Two transports carry three kinds of signal:

• Active signals. The hardware announces its own failure: an Xid event, an uncorrectable ECC error, a PSU fault, a link down. This is the fast path. Detection is immediate if someone is listening on the right channel.
• Passive degradation. Nothing announces anything. Fans work harder each week for the same load. Link retrains get more frequent. Sustained clocks sag a little more each month. These faults only appear as trends across retained time series, never as a single bad reading. This is why counters are stored, not sampled and discarded.
• Scheduled validation. Some faults emit no signal at all until load finds them. Periodic health sweeps run the same checks the node passed at acceptance and compare the results against the node's own history. What we measure before a GPU reaches a customer describes that harness.

An alert that dies in a dashboard was never detected

Detection has a definition, and it is not a panel turning red. An alert counts when it reaches a human pager, 24x7, and opens a ticket, and the ticket arrives carrying the node's full timeline: both telemetry paths, recent workflow history including updates, reboots, and tenant changes, prior incidents, and current validation state. The responder at 3am should never have to ask the node what happened. The node may not be able to answer.

The same timeline feeds the SLA evidence ledger: when the fault started on each path, when it was detected, which workflow ran, and when the node returned to service through revalidation. Claims about reliability are only as good as the records behind them, so the records are the product.

Two paths, cross-checked. Trends retained. Sweeps scheduled. Escalation that ends at a human with the history attached. That is the difference between owning GPUs and operating a cloud.