Troubleshooting | Proto Fleet Documentation

Start on ha-a or ha-b with the local HA status command. It checks the local Fleet runtime and the cluster control path without exposing credentials.

$ sudo /opt/proto-fleet/deployment/ha/fleet-ha status /etc/proto-fleet/ha/node.env

The command is available on the two full Fleet hosts, not the witness. When failover redundancy is degraded, it still prints the status JSON and exits nonzero.

Read HA status

A healthy active host reports output like this:

1 {
2   "runtime": {
3     "version": "v0.2.10",
4     "role": "active",
5     "observation": "current",
6     "observed_at": "2026-08-17T20:00:00Z",
7     "lease_expires_at": "2026-08-17T20:00:10Z",
8     "endpoint": "healthy",
9     "reason_codes": null
10   },
11   "control": {
12     "control_ready": true,
13     "failover_ready": true,
14     "reason_codes": null
15   }
16 }

Field	Meaning
`runtime.role`	`active`, `passive`, `initializing`, or `degraded` on this host
`runtime.observation`	Whether this host has a current view of cluster ownership
`runtime.endpoint`	Whether an active host owns and serves the virtual IP
`control_ready`	The current active service and control path are usable
`failover_ready`	The active service, this host’s runtime, and the expected redundant peers are ready

control_ready: true with failover_ready: false means Fleet may still be usable, but another loss could prevent automatic takeover.

Runtime reason codes

Reason code	What to check
`observation_pending`	The local runtime is still waiting for its first ownership observation.
`control_plane_unavailable`	This host cannot read the coordination or database-primary state.
`observation_stale`	The last ownership observation is too old to trust.
`activation_pending`	This host is becoming active but has not opened the application yet.
`endpoint_unhealthy`	The virtual IP or its local proxy health does not match the observed role.

Cluster reason codes

Reason code	What to check
`etcd_quorum_unavailable`	Fewer than two coordination members are reachable and agree on the cluster.
`etcd_redundancy_degraded`	Coordination has quorum, but one of the three members is unavailable.
`writer_unavailable`	Fleet cannot confirm the writable database primary.
`database_redundancy_degraded`	The synchronous database replica is unavailable or behind.
`fleet_redundancy_degraded`	This host’s Fleet runtime or the expected active/passive pair is not healthy.
`fleet_version_mismatch`	The two Fleet hosts run different versions. This is expected between the two steps of a rolling update.
`vip_unavailable`	The virtual IP is not serving the active health endpoint.

Installation remains at peer waiting

The two database-host commands can finish before the cluster has quorum. Their HA services remain installed and keep converging under systemd while the remaining hosts join. The ha-c command waits until Fleet is active through the virtual IP.

After installing all three hosts, reconnect to any host that is still waiting and inspect the local service:

$ sudo systemctl status proto-fleet-ha.service --no-pager
$ sudo journalctl -u proto-fleet-ha.service -n 200 --no-pager

Then run fleet-ha status on ha-a and ha-b.

Do not rerun the installer over a partially installed host. The installer does not maintain resume or rollback state. If local setup was interrupted after it changed the host and the service cannot start, reimage the dedicated host before trying the guided installation again.

Installer stops before service startup

The installer checks the host before it writes Fleet configuration. Use the message it prints to find the relevant condition.

Linux release or architecture is not supported

Use Debian 12 or 13, Ubuntu 22.04 or 24.04, or 64-bit Raspberry Pi OS based on Debian 12 or 13. A compatible Debian or Ubuntu derivative must report a release codename available from Docker’s package repository.

The installer supports amd64 and arm64. It does not support 32-bit hosts, RPM-based systems, or systems without systemd.

The host uses a 16K page size

Check the running kernel page size:

$ getconf PAGESIZE

The result must be 4096. If it is 16384, boot a kernel configured for a 4,096-byte page size, reboot, and run the installer again. The installer does not change boot configuration or reboot the host.

Existing software conflicts with the dedicated-host requirement

The installer can reuse a complete, unused Docker Engine and Docker Compose 2.24.4 or newer. Existing images, volumes, networks, and image cache are allowed.

Move other workloads before installing HA. Existing containers, custom Docker daemon configuration, configured or active keepalived, previous Proto Fleet paths or services, and existing HA data are treated as conflicts.

Existing input firewall rules are also a conflict. This includes input filtering created by tools such as ufw or firewalld. Remove those rules before installing HA. Unrelated nftables tables without input hooks are preserved.

A required port or address is already in use

Confirm that ports 2379 and 2380 are available on all three hosts. On ha-a and ha-b, also confirm that ports 80, 443, 4000, 5432, and 8008 are available.

Reserve each node address in DHCP. Keep the virtual IP outside the DHCP pool and confirm that no device currently owns it. The three hosts and virtual IP must be on the same local network.

A host bundle is rejected

ha-a transfers each role-specific bundle before it installs itself. Run the exact peer command printed by the same ha-a installation. Do not copy, rename, or edit the prepared file at /var/tmp/proto-fleet-ha-host.json.

If the command reports that the bundle is missing, confirm that you are connecting to the address entered for that role. If the bundle does not match the host address or release, correct the host networking or use the command from the matching cluster preparation run.

A successful peer installation consumes its bundle. Do not rerun the cluster installer over a partially installed host.

ha-a cannot connect to a peer

The cluster installer checks both peers before it changes ha-a. Confirm that the SSH username entered in the wizard works from ha-a to both peer addresses.

If you use an SSH agent on your operator machine, start the ha-a command with ssh -A -t and preserve SSH_AUTH_SOCK through sudo as shown on the installation page. If you use passwords, respond to the password and host-key prompts in the authenticated ha-a session.

The installer does not copy SSH keys or turn off host-key verification.

Fleet does not open at the virtual IP

Run fleet-ha status on both full Fleet hosts.
Confirm that exactly one host reports active and that control_ready is true.
Check for vip_unavailable or endpoint_unhealthy.
Open https://<virtual-ip>, not a node address.
Confirm that the client trusts proto-fleet-ha-service-ca.crt and that its fingerprint matches the value printed by the ha-a installer.

If the original installer session is unavailable, read the installed fingerprint on either full Fleet host:

$ sudo openssl x509 -in /etc/proto-fleet/ha/service-ca.crt -noout -fingerprint -sha256

If the browser still reports an untrusted certificate, import the public service CA into its own trust store or your managed browser policy. Some browsers do not use the operating system’s root store. Never import a host bundle.

If the virtual IP cannot be reached, confirm that it remains outside the DHCP pool and that the client is on, or routed to, the same local network.

Update recovery

During the interval between updating the passive host and completing the update, fleet_version_mismatch and failover_ready: false are expected. Complete the update before treating that reason as a cluster problem.

If an update reports pending application recovery, restart the local updater and inspect status:

$ sudo systemctl restart proto-fleet-updater.service
$ sudo /opt/proto-fleet/deployment/ha/fleet-ha status /etc/proto-fleet/ha/node.env

If the target version is already installed and the host is healthy, do not rerun the update.
If the old version remains and the host is passive, retry the ordinary update.
If the old version remains and the host is active, use --complete only after the passive peer is healthy on the target version.

The updater refuses to stop Fleet when the peer, ownership role, or control path is not safe for the requested update. Check both hosts before retrying.

To inspect updater activity:

$ sudo journalctl -u proto-fleet-updater.service -n 200 --no-pager

Uninstall reports an incomplete installation

The uninstaller requires the files created by the guided installer and a reachable Docker daemon. It does not remove arbitrary partial or damaged installations.

If you already uninstalled without --purge-data, the retained state cannot be removed by running uninstall again. Reimage the dedicated host before a fresh guided installation.

If installation-owned files disappeared before uninstall, keep the host offline from the HA cluster and reimage it. Do not delete individual database, credential, service, or firewall paths in an attempt to make the uninstaller continue.

For help interpreting the output, open a GitHub Discussion or report a bug without attaching host bundles, credentials, or private keys.