Troubleshooting
Structured runbooks for common AtomOS and Electros integration failures. Each section follows:
Symptom → Diagnostic checks → Expected → Likely causes → Corrective actions → Verification
Use Diagnostic commands for command details and Install & CLI reference for ports and flags. Escalate to support@elemento.cloud when data safety or cluster-wide auth is in question.
Electros does not discover or show the host
| Step | Detail |
|---|---|
| Symptom | Target missing, grey, or never appears in Connections |
| Diagnostic checks | Ping/traceroute from Electros egress to target IP; sudo atomos-cli --status; ss -tulpn on host for HTTPS listeners |
| Expected | Host reachable on registered IP; compute/storage services running |
| Likely causes | Wrong IP registered; firewall blocking TCP client/compute ports; services stopped; host not installed |
| Corrective actions | Fix IP/DNS; open required TCP paths from Electros subnet (reference); sudo atomos-cli --start-all or --restart compute; re-run --verify-install |
| Verification | Target visible; Active Connections can enable scenario |
Electros target connect / HTTPS failures
| Step | Detail |
|---|---|
| Symptom | Connect errors, SSL errors, or trust prompts that never succeed |
| Diagnostic checks | sudo atomos-cli --show-fingerprint; compare with Electros trust UI; curl -vkI https://TARGET_IP:7701/ from a jump host; cert SAN list vs registered IP |
| Expected | Fingerprint match; certificate includes target IP/DNS |
| Likely causes | 0.9+ HTTPS without cert; SAN omitting management IP; stale trust; clock skew |
| Corrective actions | sudo atomos-cli --generate-cert then -ra; re-trust in Electros; fix NTP |
| Verification | Electros target healthy; VM operations succeed |
From AtomOS 0.9, APIs use HTTPS by default — Networking.
TLS trust between peers (compute ↔ storage)
| Step | Detail |
|---|---|
| Symptom | Volume ops fail across hosts; cluster auth errors |
| Diagnostic checks | --status on both; --show-fingerprint on peer; existing pinned certs |
| Expected | Mutual trust for each IP + role (compute / storage) |
| Likely causes | Missing --get-certificate; cert rotation without re-pin; ELEMENTO_CLUSTER_SECRET mismatch |
| Corrective actions | sudo atomos-cli --get-certificate 10.0.0.11 storage (use the real peer IP; repeat reverse); align cluster secret per Security; -ra |
| Verification | Cross-host volume attach or migration test in staging |
AtomOS services down
| Step | Detail |
|---|---|
| Symptom | --status shows failed units; Electros operations timeout |
| Diagnostic checks | sudo atomos-cli --status; sudo atomos-cli --journal compute 2h (or storage); journalctl -p err -b |
| Expected | Core services active |
| Likely causes | Failed update; disk full; license issue; manual package drift |
| Corrective actions | --logs; fix host disk space; --license; --restart SERVICE or -ra in window |
| Verification | --verify-install; --healthcheck |
VM will not create
| Step | Detail |
|---|---|
| Symptom | Electros errors at create; pledge never schedules |
| Diagnostic checks | Target enabled under Active Connections; Electros org permissions; host --healthcheck; free RAM/CPU |
| Expected | C4 finds capacity; storage can allocate volume |
| Likely causes | Target disabled; insufficient resources; storage vault full; allow list blocking UID |
| Corrective actions | Enable connection; free capacity; --volume-info; review allow/ban lists (reference) |
| Verification | Test VM creates and reaches power state |
VM will not boot or power on
| Step | Detail |
|---|---|
| Symptom | Stuck off, immediate crash, or libvirt error in journal |
| Diagnostic checks | --journal compute; --volume-info; attached ISO/path exists |
| Expected | Guest starts without missing disk paths |
| Likely causes | Missing volume; corrupt image; passthrough device unavailable; host OOM |
| Corrective actions | Reattach volume; restore from backup/elemento-bin; remove unavailable PCI passthrough; add RAM |
| Verification | Console shows boot; guest OS healthy |
libvirt break-glass
Avoid ad-hoc virsh define/edit unless your runbook allows it — can desync Electros state (Diagnostics).
VM has no network
| Step | Detail |
|---|---|
| Symptom | Guest no DHCP or no routing |
| Diagnostic checks | bridge link show; ip -br addr; guest NIC on correct bridge; upstream switch VLAN |
| Expected | br0 (or chosen bridge) up with slave NIC; L2 path to DHCP/router |
| Likely causes | Bridge down; wrong NIC enslaved; VLAN mismatch; firewall on guest OS |
| Corrective actions | --bridge-setup or restore NM profile from console; fix VLANs; restart compute if needed |
| Verification | Guest gets IP; pings gateway |
Volumes missing in Electros or CLI
| Step | Detail |
|---|---|
| Symptom | Known volumes absent; create fails with storage errors |
| Diagnostic checks | mount | grep /mnt; sudo atomos-cli --volume-info; --journal storage; vault visible in --vault-editor |
| Expected | Vault mounted; storage service running |
| Likely causes | Unmounted backend; edited vault disabled; full disk; storage service crash |
| Corrective actions | Restore mounts; fix backend; --restart storage in approved window only |
| Verification | Volumes listed; test attach |
Export or storage client failures
| Step | Detail |
|---|---|
| Symptom | iSCSI/NFS/export path fails; REST errors on 27777 |
| Diagnostic checks | ss -tulpn; firewall; --fw-info; backend array/filer health |
| Expected | Listeners on documented TCP storage ports where used |
| Likely causes | Closed firewall; wrong initiator; backend offline |
| Corrective actions | Open paths between initiator and target subnets only; fix backend; --restart storage |
| Verification | Initiator login or loopback API probe per runbook |
Remote auth for storage client API: unknown — requires verification (Feature availability).
Bridge / host network loss
| Step | Detail |
|---|---|
| Symptom | SSH dropped after --bridge-setup; guests and host unreachable |
| Diagnostic checks | BMC/console: nmcli device status; ip route |
| Expected | Management IP on intended interface |
| Likely causes | Only NIC bridged; lost default route; cloud IPv6 route |
| Corrective actions | From console, revert bridge profile; run --verify-network after fix |
| Verification | SSH stable; --verify-network; guests on br0 |
See Networking.
VLAN tagging issues
| Step | Detail |
|---|---|
| Symptom | Guests on wrong segment or no L2 connectivity |
| Diagnostic checks | Switch port mode (access vs trunk); host VLAN iface names; VLAN guide |
| Expected | Tag alignment end-to-end |
| Likely causes | Native VLAN mismatch; missing VLAN subinterface on host |
| Corrective actions | Fix switch config; adjust NM VLAN on bridge/NIC |
| Verification | DHCP/domain on correct VLAN |
Live migration failures
| Step | Detail |
|---|---|
| Symptom | Migration hangs or aborts |
| Diagnostic checks | Guest has passthrough GPU/USB; CPU compatibility; shared storage visibility on both hosts; cluster connectivity |
| Expected | Eligible guest; both hosts reach volume paths |
| Likely causes | Passthrough device; local-only vault; network partition; TLS not pinned between hosts |
| Corrective actions | Migrate only eligible guests (Migration); fix storage networking; pin certs |
| Verification | Successful migration in maintenance window |
GPU passthrough failures
| Step | Detail |
|---|---|
| Symptom | VM start fails when GPU attached; device missing in guest |
| Diagnostic checks | sudo atomos-cli --pci-list; IOMMU in firmware/kernel; driver bound to vfio |
| Expected | GPU in isolated IOMMU group usable for passthrough |
| Likely causes | IOMMU off; wrong device; host driver holding GPU |
| Corrective actions | Enable IOMMU; follow GPU passthrough; release GPU from host |
| Verification | Guest sees device; workload runs |
Failed or partial updates
| Step | Detail |
|---|---|
| Symptom | --update errors; mixed package versions; GUI missing after upgrade |
| Diagnostic checks | atomos-cli --version; --status; --logs |
| Expected | Consistent package set from Elemento repos |
| Likely causes | Network; disk space; skipped cert step after 0.9; atomosgui never installed |
| Corrective actions | Refresh CLI via download-then-execute get-atomos.sh; retry --update; install GUI role or dnf install atomos-atomosgui if needed — Upgrades |
| Verification | --verify-install; Electros target healthy |
Production acceptance checklist
Use after install or major change:
- [ ] Versions recorded (
atomos-cli --version, firmware, NIC drivers) - [ ] IOMMU and NICs validated where GPU or SR-IOV planned
- [ ] Networking, firewall, certificates, admin access verified
- [ ] VM create, reboot, console, and networking tested
- [ ] Backup and restore tested on non-critical data
- [ ] Migration, GPU, and multi-host scenarios exercised if in scope — HA/automatic failover not assumed without verification (HA, Feature availability)
- [ ] Runbooks and Support & contacts documented