> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.proto.xyz/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.proto.xyz/_mcp/server.

# Troubleshoot high availability

Start on `ha-a` or `ha-b` with the local HA status command. It checks the local Fleet runtime and the cluster control path without exposing credentials.

```bash
sudo /opt/proto-fleet/deployment/ha/fleet-ha status /etc/proto-fleet/ha/node.env
```

The command is available on the two full Fleet hosts, not the witness. When failover redundancy is degraded, it still prints the status JSON and exits nonzero.

## Read HA status

A healthy active host reports output like this:

```json
{
  "runtime": {
    "version": "v0.2.10",
    "role": "active",
    "observation": "current",
    "observed_at": "2026-08-17T20:00:00Z",
    "lease_expires_at": "2026-08-17T20:00:10Z",
    "endpoint": "healthy",
    "reason_codes": null
  },
  "control": {
    "control_ready": true,
    "failover_ready": true,
    "reason_codes": null
  }
}
```

| Field                 | Meaning                                                                             |
| --------------------- | ----------------------------------------------------------------------------------- |
| `runtime.role`        | `active`, `passive`, `initializing`, or `degraded` on this host                     |
| `runtime.observation` | Whether this host has a current view of cluster ownership                           |
| `runtime.endpoint`    | Whether an active host owns and serves the virtual IP                               |
| `control_ready`       | The current active service and control path are usable                              |
| `failover_ready`      | The active service, this host's runtime, and the expected redundant peers are ready |

`control_ready: true` with `failover_ready: false` means Fleet may still be usable, but another loss could prevent automatic takeover.

#### Runtime reason codes

| Reason code                 | What to check                                                              |
| --------------------------- | -------------------------------------------------------------------------- |
| `observation_pending`       | The local runtime is still waiting for its first ownership observation.    |
| `control_plane_unavailable` | This host cannot read the coordination or database-primary state.          |
| `observation_stale`         | The last ownership observation is too old to trust.                        |
| `activation_pending`        | This host is becoming active but has not opened the application yet.       |
| `endpoint_unhealthy`        | The virtual IP or its local proxy health does not match the observed role. |

#### Cluster reason codes

| Reason code                    | What to check                                                                                           |
| ------------------------------ | ------------------------------------------------------------------------------------------------------- |
| `etcd_quorum_unavailable`      | Fewer than two coordination members are reachable and agree on the cluster.                             |
| `etcd_redundancy_degraded`     | Coordination has quorum, but one of the three members is unavailable.                                   |
| `writer_unavailable`           | Fleet cannot confirm the writable database primary.                                                     |
| `database_redundancy_degraded` | The synchronous database replica is unavailable or behind.                                              |
| `fleet_redundancy_degraded`    | This host's Fleet runtime or the expected active/passive pair is not healthy.                           |
| `fleet_version_mismatch`       | The two Fleet hosts run different versions. This is expected between the two steps of a rolling update. |
| `vip_unavailable`              | The virtual IP is not serving the active health endpoint.                                               |

## Installation remains at peer waiting

The two database-host commands can finish before the cluster has quorum. Their HA services remain installed and keep converging under systemd while the remaining hosts join. The `ha-c` command waits until Fleet is active through the virtual IP.

After installing all three hosts, reconnect to any host that is still waiting and inspect the local service:

```bash
sudo systemctl status proto-fleet-ha.service --no-pager
sudo journalctl -u proto-fleet-ha.service -n 200 --no-pager
```

Then run `fleet-ha status` on `ha-a` and `ha-b`.

Do not rerun the installer over a partially installed host. The installer does not maintain resume or rollback state. If local setup was interrupted after it changed the host and the service cannot start, reimage the dedicated host before trying the guided installation again.

## Installer stops before service startup

The installer checks the host before it writes Fleet configuration. Use the message it prints to find the relevant condition.

#### Linux release or architecture is not supported

Use Debian 12 or 13, Ubuntu 22.04 or 24.04, or 64-bit Raspberry Pi OS based on Debian 12 or 13. A compatible Debian or Ubuntu derivative must report a release codename available from Docker's package repository.

The installer supports `amd64` and `arm64`. It does not support 32-bit hosts, RPM-based systems, or systems without `systemd`.

#### The host uses a 16K page size

Check the running kernel page size:

```bash
getconf PAGESIZE
```

The result must be `4096`. If it is `16384`, boot a kernel configured for a 4,096-byte page size, reboot, and run the installer again. The installer does not change boot configuration or reboot the host.

#### Existing software conflicts with the dedicated-host requirement

The installer can reuse a complete, unused Docker Engine and Docker Compose 2.24.4 or newer. Existing images, volumes, networks, and image cache are allowed.

Move other workloads before installing HA. Existing containers, custom Docker daemon configuration, configured or active keepalived, previous Proto Fleet paths or services, and existing HA data are treated as conflicts.

Existing input firewall rules are also a conflict. This includes input filtering created by tools such as `ufw` or `firewalld`. Remove those rules before installing HA. Unrelated nftables tables without input hooks are preserved.

#### A required port or address is already in use

Confirm that ports `2379` and `2380` are available on all three hosts. On `ha-a` and `ha-b`, also confirm that ports `80`, `443`, `4000`, `5432`, and `8008` are available.

Reserve each node address in DHCP. Keep the virtual IP outside the DHCP pool and confirm that no device currently owns it. The three hosts and virtual IP must be on the same local network.

#### A host bundle is rejected

`ha-a` transfers each role-specific bundle before it installs itself. Run the exact peer command printed by the same `ha-a` installation. Do not copy, rename, or edit the prepared file at `/var/tmp/proto-fleet-ha-host.json`.

If the command reports that the bundle is missing, confirm that you are connecting to the address entered for that role. If the bundle does not match the host address or release, correct the host networking or use the command from the matching cluster preparation run.

A successful peer installation consumes its bundle. Do not rerun the cluster installer over a partially installed host.

#### ha-a cannot connect to a peer

The cluster installer checks both peers before it changes `ha-a`. Confirm that the SSH username entered in the wizard works from `ha-a` to both peer addresses.

If you use an SSH agent on your operator machine, start the `ha-a` command with `ssh -A -t` and preserve `SSH_AUTH_SOCK` through `sudo` as shown on the installation page. If you use passwords, respond to the password and host-key prompts in the authenticated `ha-a` session.

The installer does not copy SSH keys or turn off host-key verification.

## Fleet does not open at the virtual IP

1. Run `fleet-ha status` on both full Fleet hosts.
2. Confirm that exactly one host reports `active` and that `control_ready` is `true`.
3. Check for `vip_unavailable` or `endpoint_unhealthy`.
4. Open `https://<virtual-ip>`, not a node address.
5. Confirm that the client trusts `proto-fleet-ha-service-ca.crt` and that its fingerprint matches the value printed by the `ha-a` installer.

If the original installer session is unavailable, read the installed fingerprint on either full Fleet host:

```bash
sudo openssl x509 -in /etc/proto-fleet/ha/service-ca.crt -noout -fingerprint -sha256
```

If the browser still reports an untrusted certificate, import the public service CA into its own trust store or your managed browser policy. Some browsers do not use the operating system's root store. Never import a host bundle.

If the virtual IP cannot be reached, confirm that it remains outside the DHCP pool and that the client is on, or routed to, the same local network.

## Update recovery

During the interval between updating the passive host and completing the update, `fleet_version_mismatch` and `failover_ready: false` are expected. Complete the update before treating that reason as a cluster problem.

If an update reports pending application recovery, restart the local updater and inspect status:

```bash
sudo systemctl restart proto-fleet-updater.service
sudo /opt/proto-fleet/deployment/ha/fleet-ha status /etc/proto-fleet/ha/node.env
```

* If the target version is already installed and the host is healthy, do not rerun the update.
* If the old version remains and the host is passive, retry the ordinary update.
* If the old version remains and the host is active, use `--complete` only after the passive peer is healthy on the target version.

The updater refuses to stop Fleet when the peer, ownership role, or control path is not safe for the requested update. Check both hosts before retrying.

To inspect updater activity:

```bash
sudo journalctl -u proto-fleet-updater.service -n 200 --no-pager
```

## Uninstall reports an incomplete installation

The uninstaller requires the files created by the guided installer and a reachable Docker daemon. It does not remove arbitrary partial or damaged installations.

If you already uninstalled without `--purge-data`, the retained state cannot be removed by running uninstall again. Reimage the dedicated host before a fresh guided installation.

If installation-owned files disappeared before uninstall, keep the host offline from the HA cluster and reimage it. Do not delete individual database, credential, service, or firewall paths in an attempt to make the uninstaller continue.

For help interpreting the output, open a [GitHub Discussion](https://github.com/block/proto-fleet/discussions) or [report a bug](/fleet/support#report-a-bug-or-request-a-feature) without attaching host bundles, credentials, or private keys.