THE FIELD GUIDE / OPERATIONS
Reboot, rollback & backups
Evidence-based recovery, encrypted backups and staged restore.
Review recovery policy
Reboot and rollback default off. Enable them only for a reviewed host policy with tested prerequisites. Snapshot rollback requires a root-owned adapter and a bootable snapshot layout. Snapper/Btrfs support does not add snapshot recovery to arbitrary ext4 hosts.
Verify the original system
Before mutation, preserve inventory, configuration ownership and modes and service evidence. After reboot, verify a changed boot ID and healthy services. Rollback must restore the original evidence; a healthy boot alone is insufficient.
Reconcile interruptions
Running or recovery-required orphaned jobs block further host mutation while monitoring continues. Inspect the target through a trusted console, then use Activity → Review interrupted operation when central and target locks are idle. Reconciliation releases the queue without replaying the old job.
An ordinary patch can need recovery too
A lost, truncated or invalid completion reply, or a failed post-patch inventory read, can leave package changes uncertain even with reboot and rollback disabled. The server stays held until an administrator reviews the interrupted operation. A retained target receipt remains available when a later step fails. A known-unstarted attempt is distinct from an operation that may already have changed the target; neither is blindly replayed.
Review interrupted paid work
An interrupted incident/startup advisory plan retains its AI run, call receipts and spend reservations for administrator review. Its hold does not require an unrelated target status query. Release the queue only after reviewing whether a provider call may have been billed; see AI waits and recovery.
Back up the whole deployment
Use scripts/recovery.py for encrypted restic backup and staged restore of the database, encryption key, per-server SSH identities/rotation state and TLS/Caddy volumes. Complete coverage also includes controller evidence archives and receipts, trusted vendor packages/keys, YARA rules/keys and local GCP credential state. Stop every app and worker replica first. The backup takes the database maintenance lock exclusively. Protect .env and external identity/database/restic credentials separately.
Verify the bundle before activation
Format-2 manifests record exact file coverage and hashes. Staged verification rejects linked or special files and missing or tampered required state. Restore can use pre-existing empty volume roots after validating their destinations, and it restores private permissions and service ownership. Empty-fleet backups include valid empty SSH state; a missing required server key blocks backup or activation. Restore into empty destinations only after verification. Older format-1 bundles remain readable with an explicit missing-coverage warning; use the current recovery script for format-2 bundles.
Open the application reference for capability scope and technical details.
Open Open ENTERPRISE_RECOVERY.md on GitHub ↗