Clustering, availability, and recovery

AtomOS can run many hosts that discover one another and accept workloads from Electros. That alone is not high availability (HA). HA is an outcome you design from storage layout, network paths, operational procedures, and—where verified on your release—automation that reacts to failure.

Use this page to separate concepts operators often conflate, plan realistic recovery, and link the pieces that must work together: C4, Electros, Storage & vaults, and Networking & certificates.

Multi-host ≠ HA

Registering several AtomOS targets in Electros and enabling them under Active Connections gives you capacity and scheduling options. It does not automatically eliminate downtime, split-brain safety, or data loss. Define RPO/RTO per workload and validate behavior on your topology before calling a design “HA.”

Terms operators should not mix up

Concept	What it is	What it is not
Clustering	AtomOS hosts that participate in C4 discovery and cluster services on the network paths you provide	A guarantee that VMs survive any host failure without intervention
Resource scheduling	Matching a portable pledge (from a specfile) to a host that advertises suitable CPU, memory, and device constraints via C4	Load-aware rebalancing of running VMs; see Dynamic load balancing where applicable
Load balancing	Distribution of new placement or traffic according to policy and product features on your release	Transparent failover of state on a failed node unless storage and automation support it
Planned maintenance	Operator-controlled drain, stop, migrate, or reschedule before changing hardware or software	The same code path as unplanned host death
Live migration	Moving a running eligible VM from one hypervisor to another while guest CPU/RAM state transfers over the network	Possible for every guest; blocked by passthrough devices, incompatible CPU features, or unreachable/shared storage
VM restart	Starting a guest again on a host (same or different) after stop or crash	Always equivalent to restoring application consistency without backup or quiescing
Host failure recovery	Procedures and automation that attempt to restore service after a compute node is lost	Automatic unless storage, networking, and release behavior are verified for your case
Storage redundancy	RAID, replication, ZFS, GlusterFS, shared SAN/NFS, or other backends that protect volume data	Hypervisor HA by itself
Automatic failover	Unattended switch of workload to surviving infrastructure when prerequisites are met	Implied by every multi-node cluster — validate on your topology before relying on it

flowchart LR
  subgraph control["Control plane"]
    E[Electros]
  end
  subgraph coord["Coordination"]
    C4[C4 discovery and scheduling]
  end
  subgraph hosts["AtomOS hosts"]
    H1[Compute / full]
    H2[Compute / full]
    S[Storage tier]
  end
  E -->|HTTPS pledges and ops| H1
  E --> H2
  H1 <-->|C4 / cluster services| H2
  H1 -->|volume I/O| S
  H2 -->|volume I/O| S

Clustering and scheduling (foundation)

Clustering in AtomOS terms means peer-capable hosts running cluster and compute services, advertising capacity through C4, and trusting one another for inter-host APIs where configured. Electros registration and Active Connections authorise which hosts your organisation may use; C4 helps find capacity that matches a pledge.

When a host is unavailable, the same pledge can be submitted again so scheduling can consider other reachable hosts—if volumes are accessible from those hosts and constraints (CPU features, devices, networks) are still satisfiable. See C4 protocol.

Authorization vs discovery

Discovery makes capacity visible on the network. Electros registration and scenario enablement decide who may consume it. Both matter in production.

Load balancing

Dynamic load balancing (DLB)—when enabled and supported on your release—addresses a different problem than host-failure recovery: spreading or steering load according to policy. Do not assume DLB replaces backup, shared storage design, or runbooks for dead nodes.

Details: Dynamic load balancing.

Planned maintenance

For maintenance you control the sequence:

1. Notify consumers and set change windows.
2. Identify guests that cannot live-migrate (GPU/PCIe passthrough, pinned CPU features, local-only disks).
3. Quiesce applications where consistency matters (databases, transactional services).
4. Drain by live-migrating eligible VMs or shutting them down cleanly.
5. Perform OS patches, firmware, atomos-cli updates, certificate rotation, or hardware work.
6. Validate services with atomos-cli --healthcheck, storage checks, and test VMs before returning production load.

Use atomos-cli --status and service journals (atomos-cli --journal compute 1h) while watching Electros host health during the window.

Live migration

AtomOS supports live migration for eligible virtual machines between hosts in a cluster. Success typically requires:

• Compatible CPU features between source and destination (or a migration policy that allows your difference).
• Storage the destination can attach—shared vault/backing reachable from both hosts, or equivalent volume mobility you have validated.
• Stable network connectivity between hypervisors for migration traffic (often the same paths as cluster/compute services; see install reference).
• No passthrough devices that pin the guest to one machine (GPU / PCI passthrough).

Guests with assigned physical GPUs or other dedicated PCIe functions usually cannot live-migrate like fully virtualised workloads. Plan stop/migrate or maintenance windows for those guests.

Validate on your build

Migration APIs, TLS requirements, and UI entry points can differ by release. Exercise a non-production VM pair before relying on live migration in production maintenance.

VM restart vs recreate after failure

Action	Typical trigger	Data expectation
Restart	Guest OS or libvirt restart on the same host	Root disk unchanged if volume stayed local
Recreate on peer	Compute host lost; pledge or automation targets another node	Requires volume reset, migration, or shared access you have verified

After unplanned host loss, “bring the VM back” may mean Electros resubmitting a pledge, manual import/attach, or restore from backup—not a single universal button.

Host failure recovery (conceptual)

After a compute host fails, peer hosts may attempt to recreate affected VMs when your deployment provides all of the following:

• Peer compute capacity is available and reachable on cluster and storage networks.
• Affected volumes can be reset, migrated, or otherwise made attachable on the peer (depends on vault backend and how the volume was exported).
• Cluster and inter-host trust (certificates, secrets, firewall) remain intact.

Treat automatic recreate as a capability to validate, not a contractual RTO. Even when automation runs, confirm manually: guest boot, application recovery, correct vault attachment, and no stale instance on a partitioned node.

Quorum and fencing

Quorum, STONITH, and fencing are not documented AtomOS product guarantees in this guide. If your compliance framework requires them, design and validate them explicitly with Elemento — multi-host install alone does not provide them.

Storage redundancy and automatic failover

Hypervisor clustering does not replace storage design.

Storage pattern	Failure domain	HA implication
Local vault on one full node	That disk and host	Backups and spare capacity; recreate from backup after host loss
Replicated or shared backend (e.g. GlusterFS, ZFS, SAN, NFS you validated)	Backend-specific	Failover depends on how volumes export and attach across hosts—test attach from a second compute node
Separate storage role hosts	Storage service or network	Compute may survive storage outage only if backends are redundant and exports recover

See Storage & vaults. For disaster recovery across sites, pair storage design with Backup & Veeam and contact support@elemento.cloud for multi-site patterns.

Automatic failover of applications requires end-to-end proof: health checks, floating IPs or load balancers, application-level replication, and verified volume mobility. AtomOS multi-host features are building blocks, not a complete application HA stack.

Failure scenarios (planning table)

Use this table in runbooks. Expected behavior columns describe outcomes teams validate in PoC—not promises for every release without testing.

Failure	Impact	Prerequisites	Expected behavior if verified	Admin recovery	Limitations
Single compute host power loss	VMs on that host stop	Peers reachable; volumes mobile or recoverable; pledges/specfiles documented	Peer recreate may run when volumes can reset/migrate; else manual	Resubmit pledge in Electros; restore from backup; replace host	Passthrough GPUs, local-only disks, or network partition block clean recreate
Compute host hang (non-responsive)	Same as power loss after timeout	Monitoring detects hang; fencing not assumed unless confirmed for release	May match power-loss path after health timeout	Hard power cycle; validate storage; avoid dual attach	Without fencing, risk of two writers if host was not actually dead
Storage host or vault offline	Volume I/O fails for dependent VMs	Redundant storage path or replica	VMs pause or crash; no magic cross-vault failover unless designed	Repair backend; atomos-cli --restart storage; restore from backup	Ubuntu/Debian storage-only nodes still need validated backends
Network partition (cluster vs storage)	Split visibility; migration/recreate may fail	Segmented management, cluster, storage VLANs	Partial cluster; manual intervention likely	Restore routing/firewall; verify atomos-cli --verify-network	C4 discovery does not fix partitions
Electros unavailable	No fleet UI/API path	Hosts still running	Existing VMs may keep running; new orchestration blocked	Operate via host access, AtomOS GUI, runbooks	Not a substitute for host-level HA
Planned maintenance on one host	Controlled downtime for drained guests	Live migration or shutdown plan	Eligible VMs migrated; others stopped	Follow maintenance section above	GPU guests usually not live-migrated
Corrupt or deleted volume	Guest data loss	Backups	Restore vid.<id>.elimg from backup	Veeam/file restore per Backup	Live file-level copies of active disks are not consistency-safe — see Backup
Certificate or cluster secret mismatch	Peer/API auth failures	Consistent trust setup	Cluster ops fail closed	Regenerate/trust certs; align cluster secret config	Misconfiguration looks like “HA failure”

RPO/RTO and when to engage Elemento

• Define RPO (acceptable data loss window) and RTO (time to restore service) per tier.
• PoC every pattern you rely on: live migration, recreate-on-peer, backup restore, and Electros pledge resubmission.
• For production HA architecture reviews and cluster sizing: sales@elemento.cloud or support@elemento.cloud.

Related reading

• C4 protocol — discovery, pledges, scheduling
• Architecture — failure domains
• Use cases and topologies
• Troubleshooting