Manage coordinators#

Use the following steps to manage coordinators in a high availability (HA) deployment.

Add a standby coordinator#

To add a standby coordinator:

  1. Deploy a Starburst Enterprise platform (SEP) coordinator with the same environment configuration.

  2. Configure the coordinator to announce itself to the Starburst control plane.

  3. The coordinator automatically registers as STANDBY.

  4. Health monitoring starts automatically.

Remove a coordinator#

To remove a coordinator from the HA pool:

  1. Ensure at least one other healthy coordinator is running in the environment.

  2. Stop the SEP coordinator, for example by scaling down a Kubernetes pod or stopping the service.

The coordinator record remains in the database with UNKNOWN status. To return the coordinator to the HA pool, restart the coordinator. After health checks succeed, the status changes to HEALTHY and the coordinator resumes in the STANDBY role.

Trigger a hard failover#

A hard failover promotes a standby coordinator immediately. The cluster must use discovery, and a healthy standby coordinator must be ready.

To trigger a hard failover in the UI, see Trigger a hard failover. Only the owner of the cluster can trigger a hard failover, in the UI or with the API. If the action is unavailable in the UI, a tooltip says why.

To trigger a hard failover with the public API, send a POST request to /public/api/v1/cluster/{clusterId}:failover. To check whether the cluster is ready first, send a GET request to /public/api/v1/cluster/{clusterId}/status. The response has the following fields:

  • failoverReadiness.code, with the value Ready or NotReady.

  • healthStatus of the active coordinator and of the standby coordinators, with the value Healthy, NotHealthy, or Unknown.

If the control plane rejects the failover with HTTP status 409, the code field of the response says why:

Code

Meaning

NoActiveCoordinator

The cluster has no active coordinator.

NoHealthyStandby

No healthy standby coordinator can take over.

FailoverInProgress

A failover is already running for the cluster.

DiscoveryDisabled

The cluster does not use discovery.

Conflict

The state of the cluster changed while the control plane processed the request.

Note

The API returns these values in PascalCase. Earlier releases returned READY, NOT_READY, HEALTHY, NOT_HEALTHY, UNKNOWN, NO_ACTIVE_COORDINATOR, NO_HEALTHY_STANDBY, FAILOVER_IN_PROGRESS, and CONFLICT. Update any script that compares them before you upgrade.

Trigger a failover by stopping the active coordinator#

To trigger a failover without a hard failover, stop the active SEP coordinator by scaling down a Kubernetes pod or stopping the service. The control plane detects the failure and promotes a healthy standby. This takes longer than a hard failover, because the control plane waits for three failed health checks. See How coordinator high availability works.