Update a high-availability cluster

View as Markdown

Update the passive Fleet host first. Then complete the update from the active host so the updated peer takes over the virtual IP.

Use the local fleet-ha command for HA updates. Do not update both Fleet hosts at the same time. Updates started from the Fleet application are not available in HA deployments.

HA deployments installed from v0.2.10 or later can use a newer stable, alpha, beta, or release-candidate tag. Use the exact tag from Proto Fleet releases, such as v0.2.11, v0.2.11-alpha.1, v0.2.11-beta.1, or v0.2.11-rc.1.

The target must be newer than the installed version. Downgrades and same-version updates are rejected. Before skipping versions, read the release notes and confirm that the application-only update is compatible with your installed database and coordination services.

1

Identify the active and passive hosts

Run status on both full Fleet hosts:

$ssh -t ha-a 'sudo /opt/proto-fleet/deployment/ha/fleet-ha status /etc/proto-fleet/ha/node.env'
$ssh -t ha-b 'sudo /opt/proto-fleet/deployment/ha/fleet-ha status /etc/proto-fleet/ha/node.env'

In the runtime object, find one host with "role": "active" and one with "role": "passive". Start only when the cluster reports "failover_ready": true.

2

Update the passive host

On the passive host, run:

$sudo /opt/proto-fleet/deployment/ha/fleet-ha update v0.2.11

Replace v0.2.11 with the exact newer release tag. The command downloads and verifies the release, replaces only the Fleet application services, and returns after the local host is healthy and passive.

3

Complete the update from the active host

On the host that still reports active, run:

$sudo /opt/proto-fleet/deployment/ha/fleet-ha update v0.2.11 --complete

The command prepares the release before stopping the local Fleet application. The updated peer takes ownership and begins serving the virtual IP. The old active host then installs the same release and returns as passive.

Plan for a brief interruption while the virtual IP moves. Takeover should complete within about 15 seconds in a qualified deployment, but the update does not provide zero downtime.

4

Handle a role change

The updated passive host may become active before you run the completion command. If that happens, the old active host is now passive. Run the ordinary update command on that passive host instead:

$sudo /opt/proto-fleet/deployment/ha/fleet-ha update v0.2.11

When the command returns, both Fleet hosts run the target release with one active and one passive.

5

Verify the update

Run status on both full Fleet hosts again:

$ssh -t ha-a 'sudo /opt/proto-fleet/deployment/ha/fleet-ha status /etc/proto-fleet/ha/node.env'
$ssh -t ha-b 'sudo /opt/proto-fleet/deployment/ha/fleet-ha status /etc/proto-fleet/ha/node.env'

Confirm that both hosts report the target runtime.version, one is active, one is passive, and both report failover_ready: true. The fleet_version_mismatch reason should be absent.

fleet-ha update does not change the witness or the database, coordination, and virtual IP services. There is no in-place update workflow for those components. Do not use this workflow if the target release notes require one of them to change.

If a command reports pending recovery, a version mismatch outside the update window, or a role problem, see troubleshoot an HA update.