Cluster requirements#

Before you add a cluster to Starburst control plane, make sure that it meets the requirements in this topic. The requirements depend on the mode that you use. For the modes, see Deployment modes.

Requirements for every cluster#

  • License and version. The cluster has an Elite SEP license and runs SEP version 480-e or later. Otherwise, control plane marks the cluster as UNSUPPORTED. Use the same SEP version for control plane and for its clusters. Control plane does not support a mix of versions.

  • Name. The node.environment property of the coordinator matches the cluster name that you use in control plane. Otherwise, control plane marks the cluster as UNSUPPORTED. Each cluster needs its own node.environment value, because control plane requires the name of every cluster to be unique.

  • Cluster group. The cluster belongs to exactly one cluster group.

Requirements in managed mode#

In managed mode, the cluster gets its catalogs from control plane, and control plane authenticates its users. Control plane routes only to clusters that run in managed mode. A cluster in standalone mode becomes UNSUPPORTED. Make sure of the following:

  • You set deployment.mode and deployment.manager.uri in the configuration of the cluster. See Connect a cluster to control plane.

  • You register the cluster in control plane with Use discovery turned on. Control plane does not accept a fixed URL for a cluster in managed mode. See Discovery in managed mode.

  • The cluster can reach control plane over the network. The coordinator gets the catalog definitions from control plane for each transaction, and queries fail when it cannot.

  • The cluster has no catalogs of its own. In managed mode, all catalogs live in control plane.

  • To use the users and groups that control plane provisions, the coordinator has the file etc/group-provider.properties with group-provider.name=portal. See Connect a cluster to control plane.

Requirements in standalone mode#

In standalone mode, each cluster keeps its own catalogs and access control. Make sure of the following:

  • The authentication configuration of the cluster matches control plane. This includes the authentication type, the user mapping rules, and the authentication signing secret. The secret is web-ui.shared-secret for password or insecure authentication, and http-server.authentication.oauth2.state-key for OAuth2. See Configure cluster authentication and Shared secrets.

  • You have the internal-communication.shared-secret value of the cluster. You enter it in the Shared secret field when you add the cluster.

  • All clusters in the same cluster group have the same catalog, access control, and authentication configurations.

Example configuration#

The following files show the settings of a cluster in standalone mode. They apply to the coordinators and to the workers of the cluster.

node.properties (coordinators and workers):

# node.properties
# The environment is unique for each cluster.
# It must equal the cluster name in control plane.
node.environment=cluster-a

config.properties (coordinators and workers):

# config.properties
# Do not set deployment.mode. A cluster without it runs in standalone mode.

# Coordinator high availability: announce to control plane.
# Use the same value on coordinators and workers.
discovery.uri=https://control-plane.internal.example.com

# Enter the same value in the Shared secret field
# when you add the cluster to control plane.
internal-communication.shared-secret=${ENV:INTERNAL_SHARED_SECRET}

# Authentication must match control plane exactly.
http-server.authentication.type=oauth2
web-ui.authentication.type=oauth2
http-server.authentication.oauth2.issuer=https://idp.example.com/
# Use the same value as on control plane.
http-server.authentication.oauth2.state-key=${ENV:OAUTH2_STATE_KEY}

# Catalogs and access control are identical on every cluster.
# static is the default.
catalog.management=static

Set discovery.uri only if the cluster uses coordinator high availability, and register the cluster in control plane with discovery. See Configure coordinator high availability. In a Helm installation with internalTls: true, the init container overrides a discovery.uri that you set in config.properties. See the limitations in How coordinator high availability works. For the secrets, see Shared secrets.

If control plane runs in managed mode, it marks a cluster that does not run in managed mode as UNSUPPORTED. So this configuration applies only when control plane runs in standalone mode. See Monitor cluster status.

Requirements for the clusters in a cluster group#

The clusters in a cluster group must be interchangeable, so that a query returns the same result on any of them. Make sure that all clusters in a group have:

  • the same execution modes, such as FTE or Warp Speed;

  • the same database objects, such as tables and materialized views. In standalone mode, the clusters also need the same data products and the same built-in access control, because each cluster keeps its own;

  • the same SEP version.

The clusters in a group can have different sizes and different numbers of worker nodes. Set the query limit of each cluster to the number of queries that it can run at once. See Set query limits.

Cluster groups#

Every cluster belongs to exactly one cluster group, and exactly one cluster group is the default. If you create a cluster and do not name a group, control plane puts the cluster in the default group. If you update a cluster and do not name a group, it stays in its group.

In managed mode, you need the CREATE_CLUSTER privilege to create a cluster. To choose a cluster group for a cluster, you also need one of the privileges MANAGE_SECURITY, CREATE_CLUSTER, or MANAGE_QUERY_ROUTING_RULES. With one of them, you can choose any group. Owning a group does not let you use it for a cluster, and the default group is not an exception. If you own a cluster, you can update it without any of these privileges, as long as it stays in its group.

If you create or update clusters through the API or with automation, the role that the automation uses needs one of those privileges to name a group. Name the group by its ID. The older routingGroup field, which carries the group name, still works but is deprecated.