THE FIELD GUIDE / FLEET MANAGEMENT
Event delivery, spool health & recovery
Keep collected observations durable, diagnose delivery pressure and retain an honest loss history.
Read health from the server dashboard
The server Overview and fleet summary show queue bytes and capacity, utilization, pending batches/observations, oldest observation age, delivery lag and last successful delivery. Intake/flush rates, retries, failures and last-check time explain whether a queue is catching up. Unknown or stale telemetry stays distinct from an empty healthy queue. Follow a problem status into delivery diagnostics without opening private observation payloads.
Bound payloads and flush deliberately
Observations are serialized into byte-bounded batches without splitting an individual record. Normal-load snapshots use 256 KiB batches and the established target spool baseline is 48 MB. Configure payload limits, periodic/size/shutdown flush behavior and resource-appropriate capacity for the deployment. Adaptive sizing and pressure-aware draining handle bursts; individually oversized observations receive an explicit diagnostic rather than repeatedly disappearing with their peers.
Recover undelivered work
A protected disk-backed spool keeps unacknowledged observations across restarts and temporary connectivity loss. Central persistence precedes acknowledgement. Durable checkpoints and deduplication make replay inspectable; exponential backoff with jitter avoids a reconnect storm. Graceful shutdown flushes eligible work, while abrupt loss recovers retained undelivered records.
Respond to pressure before capacity runs out
Set warning and critical utilization thresholds suited to each target. Inspect producer/consumer imbalance, backlog age, connectivity and controller capacity before increasing disk use. Bounded backpressure and explicit event-priority policy explain which security-critical work takes precedence and how fairness is retained. Any unavoidable policy discard or capacity eviction records its reason and measured loss.
Tell old loss from new loss
Separate durable counters track lost batches and lost observations where measurable, with reasons such as an oversized record, exhausted capacity or delivery failure. Older unclassified loss remains visible as historical evidence. Only newly observed loss creates a current integrity finding; a recovered queue clears the active condition without resetting cumulative history.
Check disk and transport health
Disk-full, permission, partial-write and corrupt-state failures remain explicit and fail visibly. Keep spool files private, match retention/capacity to available resources and verify replay after an outage or restart. Version-negotiated compression reduces transport bytes only when both peers support it; compare CPU and bandwidth cost before enabling it across a fleet.
Keep collection and delivery upgrades together
Update the controller and matching workers, then use Finish / verify installation to deploy and verify collectors. Review fresh telemetry after a representative burst, restart and outage. The batching baseline and the wider configurable reliability, pressure-control and dashboard capabilities are documented in the project’s batching, durable-spool and observability requirements.
Open the application reference for capability scope and technical details.
Open Open SECURITY_IMPLEMENTATION.md on GitHub ↗