Google Kubernetes Engine deployments#
The Google Kubernetes Engine (GKE), is certified to work with Starburst Enterprise platform (SEP).
Introduction#
This user guide provides information on using Helm charts to deploy SEP in Google Cloud - Kubernetes Applications. For more information about deploying on Kubernetes, see our Kubernetes guide with customization guide, examples, and tips and the Kubernetes reference documentation.
We recommend that you use Google Cloud’s Marketplace UI with Click to Deploy on GKE to deploy. The command line Deploy via command line installation is not supported.
Getting started#
Setting up and deploying SEP to a Google Kubernetes Engine cluster using Google Cloud is pretty straightforward.
You can get started by simply following the UI instructions in the marketplace.
Cluster requirements#
SEP can be deployed to an existing GKE cluster, or to a newly created one. If using an existing GKE cluster, make sure it is configured with the appropriate number and type of instances needed to deploy an SEP cluster. Review the SEP requirements to ensure that you have sufficient resources. With insufficient resources available in the cluster, the deployed pods may fail to be scheduled.
You can also choose to deploy a new cluster with minimal resources and default configuration from the marketplace offering configuration page. After the default SEP chart deploys successfully in this cluster, you must review the cluster configuration before updating it and making any adjustments to ensure that the cluster is sufficiently large.
Networking#
When using GKE with a Google Cloud External Load Balancer,
expose.ingress.path:
you must have a wildcarded path ending with /\*
to
ensure that routing is not limited to a single URL. The \*
character is the
only supported wildcard character. It must follow a forward slash /
and must
be the last character in the pattern. See the Google URL Maps
documentation
for more information about path limitations and pattern matching in GKE.
Deploying#
The default values of the SEP configuration in the UI work for most of the cases.
Warning
If you are enabling global access control using Ranger, make sure you note down the values of the admin and service user passwords you provide in the UI for later use.
Once done, you can click on Deploy to install SEP in the cluster. You can track the progress of the deployment in the applications page.
Once the application is successfully deployed, SEP is deployed using the Helm charts. If you enable either or both of Ranger and Hive Metastore Service these are also deployed. The deployment uses the values you provide in the UI and some default values when deployed the first time. If you want to upgrade or change the deployment configuration, you can either use the Application info from the application page or the following SEP upgrade instructions.
Upgrading SEP#
You can upgrade SEP using the cloud shell.
You need to download the Helm charts and extract them to modify the deployment, as shown in the following:
export TAG=2.0.1
export APP_INSTANCE_NAME=<name of the application instance>
wget https://console.cloud.google.com/storage/browser/starburst-enterprise-presto/helmCharts/presto-gcp/starburst-enterprise-presto-charts-$TAG.tgz;
tar -zxvf starburst-enterprise-presto-charts-$TAG.tgz;
Now update the values.yaml
in the umbrella chart folder
(starburst-enterprise-presto-charts-$TAG
) before you apply the manifest
file. Alternatively, you can use --set
parameter in helm template
command to update the chart configuration. The values set in --set
takes
precedence over the ones set in values.yaml
.
Once you have the manifest created, apply (kubectl apply
) the file to update
the deployment.
More details about the SEP configuration are found in the reference documentation.
Make sure you have usage metrics enabled in your SEP configuration. Our Google Cloud offering relies on the metrics and if disabled, the deployment is terminated after 36 hours.
The minimum set of parameters required for the upgrade are shown below:
helm template starburst-enterprise-presto-charts-$TAG/ \
--name-template="$APP_INSTANCE_NAME" \
--set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
--set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
--set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
--set image.tag="$TAG" \
--set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
--set initImage.tag="$TAG" \
--set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
--set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
--set [PUT OTHER CHART PARAMETERS YOU WANT TO UPDATE] > presto_manifest.yaml
The following example shows how to update the number of SEP workers:
helm template starburst-enterprise-presto-charts-$TAG/ \
--name-template="$APP_INSTANCE_NAME" \
--set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
--set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
--set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
--set image.tag="$TAG" \
--set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
--set initImage.tag="$TAG" \
--set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
--set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
--set worker.count > presto_manifest.yaml
Configure kubectl
to point at your cluster and then apply the configuration:
kubectl apply -f presto_manifest.yaml
Enabling global access control with Apache Ranger#
You can deploy or upgrade Ranger for global access control by creating a manifest file with Ranger parameters set.
You can either update the values.yaml
in the umbrella chart folder
starburst-enterprise-presto-charts-$TAG
for the configuration
(starburst-ranger
) prior to generating the manifest file or or use --set
parameter while creating the manifest file. The --set
parameters must be
prefixed by starburst-ranger
.
The complete reference of the chart configuration can be found in the Ranger chart documentation.
Example with Ranger enabled:
helm template starburst-enterprise-presto-charts-$TAG/ \
--name-template="$APP_INSTANCE_NAME" \
--set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
--set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
--set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
--set image.tag="$TAG" \
--set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
--set initImage.tag="$TAG" \
--set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
--set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
--set starburst-ranger.enabled=true \
--set [PUT OTHER CHART PARAMETERS YOU WANT TO UPDATE] > presto_manifest.yaml
Enabling the Hive Metastore Service#
Similar to Ranger, to deploy or upgrade the Hive Metastore Service (HMS)
provided by Starburst, generate a manifest file with Hive chart
parameters set. You can either use values.yaml
(for starburst-hive
) or
use the --set
parameters. For Hive charts, the --set
parameters must be
prefixed by starburst-hive
.
The complete reference of the chart configuration for the metastore can be found in the HMS documentation.
Example with HMS enabled:
helm template starburst-enterprise-presto-charts-$TAG/ \
--name-template="$APP_INSTANCE_NAME" \
--set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
--set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
--set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
--set image.tag="$TAG" \
--set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
--set initImage.tag="$TAG" \
--set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
--set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
--set starburst-hive.image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/hive" \
--set starburst-hive.image.tag="$TAG" \
--set starburst-hive.enabled=true > presto_manifest.yaml
Configure kubectl
to point at your cluster and then apply the configuration:
kubectl apply -f presto_manifest.yaml
Deleting a deployment#
To delete a GKE deployment, run the following command:
kubectl delete -f presto_manifest.yaml