Install and configure Icehouse LakeOps#
Icehouse LakeOps maintains the Iceberg tables in catalogs that you register with
the Control Plane. To set it up, turn it on in the Control Plane, deploy its
coordinator and workers with the starburst-icehouse Helm chart, give it access
to storage, and create a catalog for it to maintain. Then
register the catalog to start maintenance.
Before you begin, make sure that you meet the requirements.
Configure the Control Plane#
To turn on Icehouse LakeOps, add the following properties to the Control Plane
config.properties file:
icehouse-tasks.enabled=true
icehouse.server.name=lakeops
icehouse.server.shared-secret=${ENV:ICEHOUSE_SHARED_SECRET}
starburst.event-bus.enabled=true
Property name |
Description |
Default |
|---|---|---|
|
Enables Icehouse LakeOps. Requires |
|
|
The name of the Icehouse LakeOps deployment. It must match the
|
None (required) |
|
The shared secret of the Icehouse LakeOps deployment. It must match the
|
None |
|
Sends Icehouse LakeOps usage events to the Control Plane. Required for Icehouse LakeOps. |
|
Install the LakeOps Helm chart#
Icehouse LakeOps runs on its own coordinator and workers, which you deploy with
the starburst-icehouse Helm chart. The workers run in one of two types of
worker pool:
Static: A fixed number of worker pods.
Dynamic: The LakeOps coordinator starts workers on demand, up to a resource limit that you set.
Create the secrets#
Create the following Kubernetes secrets in the namespace of the LakeOps deployment:
icehouse-shared-secret: The secret that the LakeOps coordinator and workers use to authenticate to each other.portal-icehouse-secret: The same value as theicehouse.server.shared-secretproperty of the Control Plane.
Configure a static worker pool#
Create an icehouse-values.yaml file. The following example configures a
coordinator and two workers:
image:
repository: <registry>/starburstdata/starburst-icehouse
sharedSecret:
existingSecret: icehouse-shared-secret
secretKey: shared-secret
coordinator:
replicas: 1
service:
name: starburst-icehouse-coordinator
resources:
requests: {cpu: 500m, memory: 3Gi}
limits: {cpu: 2, memory: 4Gi}
env:
- name: PORTAL_ICEHOUSE_SECRET
valueFrom: {secretKeyRef: {name: portal-icehouse-secret, key: secret}}
etcFiles:
config.properties: |
icehouse-server.type=COORDINATOR
node.environment=lakeops
node.id=${ENV:ICEHOUSE_NODE_ID}
http-server.http.port=8080
discovery.uri=http://localhost:8080
communication.shared-secret-key.base64=${ENV:ICEHOUSE_SHARED_SECRET}
task-coordinator.worker-class.id=lakeops-worker
task-coordinator.worker-class.cpu-milli-cores=1000
task-coordinator.worker-class.memory=4GB
worker-pool.resource.total-allowed.cpu.milli-cores=4000
worker-pool.resource.total-allowed.memory=4GB
sep-control-plane.url=http://<control-plane-host>:8080
starburst.event-bus.ingestion-base-uri=http://<control-plane-host>:8080
internal-communication.shared-secret=${ENV:PORTAL_ICEHOUSE_SECRET}
maintenance.compaction.table.execution.sorting-spill-path=/tmp/spill
maintenance.compaction.table.committer.hardware-tier=MEDIUM
maintenance.compaction.table.execution.equality-delete-heavy.hardware-tier=MEDIUM
worker:
replicaCount: 2
resources:
requests: {cpu: 500m, memory: 5Gi}
limits: {cpu: 2, memory: 5Gi}
etcFiles:
config.properties: |
icehouse-server.type=WORKER
node.environment=lakeops
node.id=${ENV:ICEHOUSE_NODE_ID}
http-server.http.port=8080
discovery.uri=http://starburst-icehouse-coordinator:8080
task-worker.id=${ENV:ICEHOUSE_NODE_ID}
task-worker.worker-class-id=lakeops-worker
task-worker.communication.shared-secret-key.base64=${ENV:ICEHOUSE_SHARED_SECRET}
task-worker.environment=<environment-name>
task-worker.installation=<installation-name>
task-worker.region=<aws-region>
task-worker.machine.class-name=lakeops-worker-class
task-worker.task.resource.usable.cpu.milli-cores=1000
task-worker.task.resource.usable.memory=4GB
The following settings must match:
Coordinator |
Worker or Control Plane |
Value in the example |
|---|---|---|
|
Control Plane |
|
|
Control Plane |
|
|
Worker |
|
|
Worker |
|
|
The host in the worker |
|
Set all maintenance.* properties on the coordinator. They have no effect on
workers.
Configure a dynamic worker pool#
With a dynamic worker pool, the coordinator creates the workers in a separate
namespace. Set worker.replicaCount to 0, and set the resource limit of the
pool instead of a number of workers:
worker:
replicaCount: 0
workerNamespace:
name: <worker-namespace>
create: true
workerPoolRbac:
create: true
coordinator:
etcFiles:
config.properties: |
icehouse-server.type=COORDINATOR
node.environment=lakeops
node.id=${ENV:ICEHOUSE_NODE_ID}
http-server.http.port=8080
sep-control-plane.url=http://<control-plane-host>:8080
starburst.event-bus.ingestion-base-uri=http://<control-plane-host>:8080
internal-communication.shared-secret=${ENV:PORTAL_ICEHOUSE_SECRET}
maintenance.compaction.table.execution.sorting-spill-path=/tmp/spill
maintenance.compaction.table.committer.hardware-tier=MEDIUM
maintenance.compaction.table.execution.equality-delete-heavy.hardware-tier=MEDIUM
task-coordinator.worker-pool.type=KUBERNETES_DISCOVERY
worker-pool.resource.total-allowed.cpu.milli-cores=4000
worker-pool.resource.total-allowed.memory=16GB
k8s-worker-pool.cloud-type=AWS
k8s-worker-pool.region=<aws-region>
k8s-worker-pool.environment=<environment-name>
k8s-worker-pool.installation=<installation-name>
k8s-worker-pool.kubernetes.namespace=${ENV:WORKER_NAMESPACE}
k8s-worker-pool.worker.docker.image=${ENV:WORKER_DOCKER_IMAGE}
k8s-worker-pool.worker.http.port=8080
k8s-worker-pool.worker.deployable-server-classes.file=/etc/starburst/deployable-server-classes.json
k8s-worker-pool.kubernetes.bootstrap=false
k8s-worker-pool.kubernetes.app.service-account.name=task-worker-user
deployable-server-classes.json: |
[
{
"machineClassName": "lakeops-worker-class",
"resourceRequest": {"cpuMilliCores": 1100, "memory": "5GB"},
"usableCapacity": {"cpuMilliCores": 1000, "memory": "4GB"}
}
]
workerNamespace.name is required, and it must differ from the namespace of
the coordinator. Don’t set the static pool properties discovery.uri,
communication.shared-secret-key.base64, or task-coordinator.worker-class.*.
The coordinator doesn’t start if they’re set.
Sizing#
The memory of a worker pod must be larger than
task-worker.task.resource.usable.memory. For example, with 4 GB of usable memory, use a5Gipod.MEDIUMis the smallest hardware tier that runs maintenance tasks.Size the worker pool for the number of tables that you maintain, not for the size of your SEP clusters.
Deploy the chart#
helm upgrade --install starburst-icehouse <chart> -f icehouse-values.yaml
Restart the coordinator and the workers together.
Give LakeOps access to storage#
Icehouse LakeOps accesses your tables with its own identity, separate from your SEP clusters. Give the LakeOps coordinator and workers access to the S3 bucket and the AWS Glue database of each catalog that you register. Use one of the following methods, depending on your platform:
Amazon EKS: IAM roles for service accounts (IRSA).
RKE2: The instance role of the nodes, or static access keys.
OpenShift: Static access keys, or a role set with the
s3.iam-rolecatalog property.
Permissions#
The identity of Icehouse LakeOps needs the following permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": "arn:aws:s3:::<bucket>"
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject",
"s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"],
"Resource": "arn:aws:s3:::<bucket>/*"
},
{
"Effect": "Allow",
"Action": ["glue:GetDatabase", "glue:GetDatabases", "glue:GetTable",
"glue:GetTables", "glue:CreateTable", "glue:UpdateTable",
"glue:DeleteTable", "glue:GetPartition", "glue:GetPartitions",
"glue:BatchGetPartition", "glue:CreatePartition",
"glue:UpdatePartition", "glue:DeletePartition",
"glue:BatchCreatePartition"],
"Resource": [
"arn:aws:glue:<region>:<account-id>:catalog",
"arn:aws:glue:<region>:<account-id>:database/<glue-database>",
"arn:aws:glue:<region>:<account-id>:table/<glue-database>/*"
]
}
]
}
If the bucket uses SSE-KMS encryption, also grant kms:GenerateDataKey and
kms:Decrypt on the KMS key to both Icehouse LakeOps and your SEP
clusters.
Use IRSA on Amazon EKS#
Annotate the service account of the LakeOps chart with the ARN of the IAM role:
serviceAccount:
create: true
name: lakeops-icehouse
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<lakeops-role>
With a dynamic worker pool, also do the following:
Annotate the worker service account with the same role:
workerPoolRbac: create: true workerServiceAccountAnnotations: eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<lakeops-role>
Add
sts.use-default-credentials-provider=trueto the coordinator configuration. Don’t setsts.access-keyorsts.secret-key.Add the worker service account,
system:serviceaccount:<worker-namespace>:task-worker-user, to the trust policy of the IAM role.
Access another AWS account#
To access a bucket in another AWS account, set the following properties in the catalog:
"s3.auth-type" = 'IAM_ROLE',
"s3.iam-role" = 'arn:aws:iam::<lake-account-id>:role/<lake-role>',
"s3.external-id" = '<external-id>'
Both your SEP clusters and Icehouse LakeOps assume this role. In the other
account, add both of their roles to the trust policy of the role, with the
sts:ExternalId condition. Give each of their roles the sts:AssumeRole
permission on the role.
Create a catalog#
Icehouse LakeOps maintains Iceberg catalogs that you create in the Control Plane. The following example creates a catalog that uses AWS Glue and IRSA:
CREATE CATALOG example USING iceberg
WITH (
"iceberg.catalog.type" = 'glue',
"hive.metastore.glue.region" = '<aws-region>',
"hive.metastore.glue.use-web-identity-token-credentials-provider" = 'true',
"hive.metastore.glue.default-warehouse-dir" = 's3://<bucket>/iceberg',
"fs.hadoop.enabled" = 'false',
"fs.s3.enabled" = 'true',
"s3.region" = '<aws-region>'
);
Without IRSA, set static access keys in the catalog:
"s3.region" = '<aws-region>',
"s3.aws-access-key" = '${ENV:LAKEOPS_S3_ACCESS_KEY}',
"s3.aws-secret-key" = '${ENV:LAKEOPS_S3_SECRET_KEY}'
Follow these rules when you create a catalog for Icehouse LakeOps:
Write secret properties, such as
s3.aws-secret-key, as${ENV:VARIABLE}references, and set the variables on the Control Plane pod. The Control Plane rejects plain text values.Use the fully qualified name,
<service>.<namespace>.svc.cluster.local, for any service inside the Kubernetes cluster. Otherwise, maintenance doesn’t run.Use one catalog for each Glue database. A registered catalog maintains every table in its Glue database with its own credentials, including tables that other catalogs created.