Configure coordinator high availability#
Use the following sections to configure coordinator high availability for your
Starburst Enterprise platform (SEP) clusters. The steps depend on the mode of the Starburst control
plane. In standalone mode, the coordinators and the workers announce themselves
with discovery.uri. In managed mode, they use deployment.mode and
deployment.manager.uri.
How coordinators announce themselves#
When you enable coordinator high availability, all SEP coordinators and workers send announcements to the control plane. The control plane uses these announcements to:
Register coordinators
Track coordinator availability and URI
Proxy worker announcements to the active coordinator
Note
The minimum required high availability configuration consists of one active coordinator and one standby coordinator.
Coordinators and workers must announce to the same address. Use an internal hostname or private IP address to keep announcements and health checks on your local network.
Configure SEP coordinators#
In standalone mode, set the following property in the config.properties file
of each coordinator:
discovery.uri=https://portal.example.com
In managed mode, set the following properties in the config.properties file
of each coordinator:
deployment.mode=MANAGED
deployment.manager.uri=https://control-plane.example.com
Managed mode sets discovery.uri to the value of deployment.manager.uri. Do
not set discovery.uri yourself, unless you use the Helm chart. See
Configure coordinator high availability with Helm. For the other steps to connect a managed cluster, see
Connect a cluster to control plane.
Configure SEP workers#
In standalone mode, set the following property in the config.properties file
of each worker:
discovery.uri=https://portal.example.com
In managed mode, set deployment.mode=MANAGED and the same
deployment.manager.uri in the config.properties file of each worker. A
worker without these properties uses the static catalog manager. It cannot load
the catalogs that you create in the control plane, and queries fail with an
error about missing catalogs.
The control plane forwards worker announcements to the active coordinator. During failover, the control plane automatically redirects workers to the new active coordinator.
Configure environments#
Each cluster you connect to the control plane must have a unique
node.environment value. This property identifies which cluster each node
belongs to. All nodes in the same cluster, which are the coordinators and the
workers, must use the same value. The value must equal the name of the cluster
in the control plane.
The control plane uses the node.environment value to:
Group nodes: When a node announces itself, the control plane reads the
X-Trino-EnvironmentHTTP header, which comes fromnode.environment, to determine which cluster it belongs to.Route queries: Each environment is one cluster in the control plane, and each cluster belongs to a cluster group. When routing sends a query to a cluster group, the control plane sends it to the active coordinator of a cluster in that group.
Isolate high availability state: Each environment maintains its own active coordinator independently. A failover in one environment does not affect other environments.
Register the cluster with discovery#
Coordinator high availability runs only for clusters that use discovery. In the Clusters pane, add the cluster with discovery turned on. In managed mode, the control plane requires discovery for every cluster. See Add a cluster and Discovery in managed mode.
If a cluster does not use discovery, the control plane does not manage its
coordinators, and a request for a failover returns the error code
DiscoveryDisabled. The coordinator of a cluster that announces itself without
discovery logs the error “Failed to announce node state”, with the HTTP status
409, about every five seconds.
Configure coordinator high availability with Helm#
To run two coordinators with the SEP Helm chart, set the number of coordinator replicas in your values file:
coordinator:
replicas: 2
The default values.yaml of the chart does not list replicas, so
helm show values does not show it. Use 2 for one active coordinator and one
standby coordinator. Make sure of the following:
Both coordinators have the same size.
The
environmentvalue of the chart equals the name of the cluster in the control plane. Both coordinators use it asnode.environment.The coordinators run on different nodes, so that the failure of one node does not stop both. For example, use a node group with at least two nodes.
internalTlsisfalse. See Limitations in How coordinator high availability works.
Helm values for a managed cluster#
The Helm chart sets its own discovery.uri, and managed mode needs
discovery.uri to equal deployment.manager.uri. Without this, the
coordinator does not start, and it logs the error “Property ‘discovery.uri’
must be equal to deployment.manager.uri when deployment.mode=MANAGED”. Set the
three properties in additionalProperties, for the coordinator and for the
workers:
environment: example-ha
internalTls: false
coordinator:
replicas: 2
additionalProperties: |
deployment.mode=MANAGED
deployment.manager.uri=https://control-plane.example.com
discovery.uri=https://control-plane.example.com
worker:
additionalProperties: |
deployment.mode=MANAGED
deployment.manager.uri=https://control-plane.example.com
discovery.uri=https://control-plane.example.com
Replace the environment and the address of the control plane with the values of your deployment. Use the public address of the control plane without a trailing slash.
Limits of high availability in a Helm installation#
Send queries through the control plane. The Kubernetes Service of the cluster selects both coordinators, and it cannot tell the active coordinator from the standby coordinator. A connection to the Service can reach the standby coordinator.
A Helm upgrade stops both coordinators. The coordinator Deployment uses the
Recreatestrategy. Coordinator high availability does not keep the cluster available during an upgrade.