Install and configure Icehouse LakeOps#

Icehouse LakeOps maintains the Iceberg tables in catalogs that you register with the Control Plane. To set it up, turn it on in the Control Plane, deploy its coordinator and workers with the starburst-icehouse Helm chart, give it access to storage, and create a catalog for it to maintain. Then register the catalog to start maintenance.

Before you begin, make sure that you meet the requirements.

Configure the Control Plane#

To turn on Icehouse LakeOps, add the following properties to the Control Plane config.properties file:

icehouse-tasks.enabled=true
icehouse.server.name=lakeops
icehouse.server.shared-secret=${ENV:ICEHOUSE_SHARED_SECRET}
starburst.event-bus.enabled=true
Icehouse LakeOps Control Plane properties#

Property name

Description

Default

icehouse-tasks.enabled

Enables Icehouse LakeOps. Requires control-plane.enabled=true.

false

icehouse.server.name

The name of the Icehouse LakeOps deployment. It must match the node.environment property of the LakeOps coordinator, and it must differ from the names of your SEP clusters.

None (required)

icehouse.server.shared-secret

The shared secret of the Icehouse LakeOps deployment. It must match the internal-communication.shared-secret property of the LakeOps coordinator.

None

starburst.event-bus.enabled

Sends Icehouse LakeOps usage events to the Control Plane. Required for Icehouse LakeOps.

false

Install the LakeOps Helm chart#

Icehouse LakeOps runs on its own coordinator and workers, which you deploy with the starburst-icehouse Helm chart. The workers run in one of two types of worker pool:

  • Static: A fixed number of worker pods.

  • Dynamic: The LakeOps coordinator starts workers on demand, up to a resource limit that you set.

Create the secrets#

Create the following Kubernetes secrets in the namespace of the LakeOps deployment:

  • icehouse-shared-secret: The secret that the LakeOps coordinator and workers use to authenticate to each other.

  • portal-icehouse-secret: The same value as the icehouse.server.shared-secret property of the Control Plane.

Configure a static worker pool#

Create an icehouse-values.yaml file. The following example configures a coordinator and two workers:

image:
  repository: <registry>/starburstdata/starburst-icehouse

sharedSecret:
  existingSecret: icehouse-shared-secret
  secretKey: shared-secret

coordinator:
  replicas: 1
  service:
    name: starburst-icehouse-coordinator
  resources:
    requests: {cpu: 500m, memory: 3Gi}
    limits: {cpu: 2, memory: 4Gi}
  env:
    - name: PORTAL_ICEHOUSE_SECRET
      valueFrom: {secretKeyRef: {name: portal-icehouse-secret, key: secret}}
  etcFiles:
    config.properties: |
      icehouse-server.type=COORDINATOR
      node.environment=lakeops
      node.id=${ENV:ICEHOUSE_NODE_ID}
      http-server.http.port=8080
      discovery.uri=http://localhost:8080
      communication.shared-secret-key.base64=${ENV:ICEHOUSE_SHARED_SECRET}
      task-coordinator.worker-class.id=lakeops-worker
      task-coordinator.worker-class.cpu-milli-cores=1000
      task-coordinator.worker-class.memory=4GB
      worker-pool.resource.total-allowed.cpu.milli-cores=4000
      worker-pool.resource.total-allowed.memory=4GB
      sep-control-plane.url=http://<control-plane-host>:8080
      starburst.event-bus.ingestion-base-uri=http://<control-plane-host>:8080
      internal-communication.shared-secret=${ENV:PORTAL_ICEHOUSE_SECRET}
      maintenance.compaction.table.execution.sorting-spill-path=/tmp/spill
      maintenance.compaction.table.committer.hardware-tier=MEDIUM
      maintenance.compaction.table.execution.equality-delete-heavy.hardware-tier=MEDIUM

worker:
  replicaCount: 2
  resources:
    requests: {cpu: 500m, memory: 5Gi}
    limits: {cpu: 2, memory: 5Gi}
  etcFiles:
    config.properties: |
      icehouse-server.type=WORKER
      node.environment=lakeops
      node.id=${ENV:ICEHOUSE_NODE_ID}
      http-server.http.port=8080
      discovery.uri=http://starburst-icehouse-coordinator:8080
      task-worker.id=${ENV:ICEHOUSE_NODE_ID}
      task-worker.worker-class-id=lakeops-worker
      task-worker.communication.shared-secret-key.base64=${ENV:ICEHOUSE_SHARED_SECRET}
      task-worker.environment=<environment-name>
      task-worker.installation=<installation-name>
      task-worker.region=<aws-region>
      task-worker.machine.class-name=lakeops-worker-class
      task-worker.task.resource.usable.cpu.milli-cores=1000
      task-worker.task.resource.usable.memory=4GB

The following settings must match:

Coordinator

Worker or Control Plane

Value in the example

node.environment

Control Plane icehouse.server.name, and the worker node.environment

lakeops

internal-communication.shared-secret

Control Plane icehouse.server.shared-secret

portal-icehouse-secret

task-coordinator.worker-class.id

Worker task-worker.worker-class-id

lakeops-worker

communication.shared-secret-key.base64

Worker task-worker.communication.shared-secret-key.base64

icehouse-shared-secret

coordinator.service.name

The host in the worker discovery.uri

starburst-icehouse-coordinator

Set all maintenance.* properties on the coordinator. They have no effect on workers.

Configure a dynamic worker pool#

With a dynamic worker pool, the coordinator creates the workers in a separate namespace. Set worker.replicaCount to 0, and set the resource limit of the pool instead of a number of workers:

worker:
  replicaCount: 0

workerNamespace:
  name: <worker-namespace>
  create: true

workerPoolRbac:
  create: true

coordinator:
  etcFiles:
    config.properties: |
      icehouse-server.type=COORDINATOR
      node.environment=lakeops
      node.id=${ENV:ICEHOUSE_NODE_ID}
      http-server.http.port=8080
      sep-control-plane.url=http://<control-plane-host>:8080
      starburst.event-bus.ingestion-base-uri=http://<control-plane-host>:8080
      internal-communication.shared-secret=${ENV:PORTAL_ICEHOUSE_SECRET}
      maintenance.compaction.table.execution.sorting-spill-path=/tmp/spill
      maintenance.compaction.table.committer.hardware-tier=MEDIUM
      maintenance.compaction.table.execution.equality-delete-heavy.hardware-tier=MEDIUM
      task-coordinator.worker-pool.type=KUBERNETES_DISCOVERY
      worker-pool.resource.total-allowed.cpu.milli-cores=4000
      worker-pool.resource.total-allowed.memory=16GB
      k8s-worker-pool.cloud-type=AWS
      k8s-worker-pool.region=<aws-region>
      k8s-worker-pool.environment=<environment-name>
      k8s-worker-pool.installation=<installation-name>
      k8s-worker-pool.kubernetes.namespace=${ENV:WORKER_NAMESPACE}
      k8s-worker-pool.worker.docker.image=${ENV:WORKER_DOCKER_IMAGE}
      k8s-worker-pool.worker.http.port=8080
      k8s-worker-pool.worker.deployable-server-classes.file=/etc/starburst/deployable-server-classes.json
      k8s-worker-pool.kubernetes.bootstrap=false
      k8s-worker-pool.kubernetes.app.service-account.name=task-worker-user
    deployable-server-classes.json: |
      [
        {
          "machineClassName": "lakeops-worker-class",
          "resourceRequest": {"cpuMilliCores": 1100, "memory": "5GB"},
          "usableCapacity": {"cpuMilliCores": 1000, "memory": "4GB"}
        }
      ]

workerNamespace.name is required, and it must differ from the namespace of the coordinator. Don’t set the static pool properties discovery.uri, communication.shared-secret-key.base64, or task-coordinator.worker-class.*. The coordinator doesn’t start if they’re set.

Sizing#

  • The memory of a worker pod must be larger than task-worker.task.resource.usable.memory. For example, with 4 GB of usable memory, use a 5Gi pod.

  • MEDIUM is the smallest hardware tier that runs maintenance tasks.

  • Size the worker pool for the number of tables that you maintain, not for the size of your SEP clusters.

Deploy the chart#

helm upgrade --install starburst-icehouse <chart> -f icehouse-values.yaml

Restart the coordinator and the workers together.

Give LakeOps access to storage#

Icehouse LakeOps accesses your tables with its own identity, separate from your SEP clusters. Give the LakeOps coordinator and workers access to the S3 bucket and the AWS Glue database of each catalog that you register. Use one of the following methods, depending on your platform:

  • Amazon EKS: IAM roles for service accounts (IRSA).

  • RKE2: The instance role of the nodes, or static access keys.

  • OpenShift: Static access keys, or a role set with the s3.iam-role catalog property.

Permissions#

The identity of Icehouse LakeOps needs the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::<bucket>"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject",
                 "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"],
      "Resource": "arn:aws:s3:::<bucket>/*"
    },
    {
      "Effect": "Allow",
      "Action": ["glue:GetDatabase", "glue:GetDatabases", "glue:GetTable",
                 "glue:GetTables", "glue:CreateTable", "glue:UpdateTable",
                 "glue:DeleteTable", "glue:GetPartition", "glue:GetPartitions",
                 "glue:BatchGetPartition", "glue:CreatePartition",
                 "glue:UpdatePartition", "glue:DeletePartition",
                 "glue:BatchCreatePartition"],
      "Resource": [
        "arn:aws:glue:<region>:<account-id>:catalog",
        "arn:aws:glue:<region>:<account-id>:database/<glue-database>",
        "arn:aws:glue:<region>:<account-id>:table/<glue-database>/*"
      ]
    }
  ]
}

If the bucket uses SSE-KMS encryption, also grant kms:GenerateDataKey and kms:Decrypt on the KMS key to both Icehouse LakeOps and your SEP clusters.

Use IRSA on Amazon EKS#

Annotate the service account of the LakeOps chart with the ARN of the IAM role:

serviceAccount:
  create: true
  name: lakeops-icehouse
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<lakeops-role>

With a dynamic worker pool, also do the following:

  1. Annotate the worker service account with the same role:

    workerPoolRbac:
      create: true
      workerServiceAccountAnnotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<lakeops-role>
    
  2. Add sts.use-default-credentials-provider=true to the coordinator configuration. Don’t set sts.access-key or sts.secret-key.

  3. Add the worker service account, system:serviceaccount:<worker-namespace>:task-worker-user, to the trust policy of the IAM role.

Access another AWS account#

To access a bucket in another AWS account, set the following properties in the catalog:

"s3.auth-type" = 'IAM_ROLE',
"s3.iam-role" = 'arn:aws:iam::<lake-account-id>:role/<lake-role>',
"s3.external-id" = '<external-id>'

Both your SEP clusters and Icehouse LakeOps assume this role. In the other account, add both of their roles to the trust policy of the role, with the sts:ExternalId condition. Give each of their roles the sts:AssumeRole permission on the role.

Use a private certificate authority#

If your S3-compatible object storage uses a certificate from a private or self-signed certificate authority (CA), give both your SEP clusters and Icehouse LakeOps a PKCS12 truststore. Build the truststore from the default Java cacerts file and your CA certificate, so that public endpoints such as AWS keep working.

With a static worker pool, mount the truststore on the LakeOps coordinator and workers, and on the coordinator and workers of your SEP clusters. Then point the JVM at it with the JAVA_TOOL_OPTIONS environment variable:

JAVA_TOOL_OPTIONS=-Djavax.net.ssl.trustStore=/etc/lakeops-truststore/truststore.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=<password>

With a dynamic worker pool, the LakeOps coordinator passes the truststore to the workers that it starts:

  1. Create a Kubernetes secret that holds the truststore, and its password if it has one, in the namespace of the LakeOps deployment.

  2. Set the workerTruststore.existingSecret, workerTruststore.truststoreKey, and workerTruststore.passwordKey values of the LakeOps chart.

  3. Add the following properties to the coordinator configuration. Leave out the password property if the truststore has no password.

    k8s-worker-pool.truststore.path=/etc/starburst-worker-truststore/<truststore-key>
    k8s-worker-pool.truststore.password=${ENV:WORKER_TRUSTSTORE_PASSWORD}
    

Your SEP clusters also need the truststore, as with a static worker pool.

Create a catalog#

Icehouse LakeOps maintains Iceberg catalogs that you create in the Control Plane. The following example creates a catalog that uses AWS Glue and IRSA:

CREATE CATALOG example USING iceberg
WITH (
  "iceberg.catalog.type" = 'glue',
  "hive.metastore.glue.region" = '<aws-region>',
  "hive.metastore.glue.use-web-identity-token-credentials-provider" = 'true',
  "hive.metastore.glue.default-warehouse-dir" = 's3://<bucket>/iceberg',
  "fs.hadoop.enabled" = 'false',
  "fs.s3.enabled" = 'true',
  "s3.region" = '<aws-region>'
);

Without IRSA, set static access keys in the catalog:

"s3.region" = '<aws-region>',
"s3.aws-access-key" = '${ENV:LAKEOPS_S3_ACCESS_KEY}',
"s3.aws-secret-key" = '${ENV:LAKEOPS_S3_SECRET_KEY}'

Follow these rules when you create a catalog for Icehouse LakeOps:

  • Write secret properties, such as s3.aws-secret-key, as ${ENV:VARIABLE} references, and set the variables on the Control Plane pod. The Control Plane rejects plain text values.

  • Use the fully qualified name, <service>.<namespace>.svc.cluster.local, for any service inside the Kubernetes cluster. Otherwise, maintenance doesn’t run.

  • Use one catalog for each Glue database. A registered catalog maintains every table in its Glue database with its own credentials, including tables that other catalogs created.