Secrets#
Starburst Enterprise platform (SEP) supports two secrets management methods for storing and retrieving sensitive data:
External secrets management via environment variables
External secrets management#
With external secrets management, SEP manages configuration details in static properties files. This configuration needs to include values such as usernames, passwords, and other strings, that are often required to be kept secret. Only a few selected administrators or the provisioning system has access to the actual value.
The secrets support in SEP allows you to use environment variables as values
for any configuration property. All properties files used by SEP, including
config.properties
and catalog properties files, are supported. When loading
the properties, SEP replaces the reference to the environment variable with
the value of the environment variable.
Environment variables are the most widely supported means of setting and retrieving values. Environment variables can be set in the scope of the task being performed, preventing external access. Most provisioning and configuration management systems include support for setting environment variables. This includes systems such as Ansible, often used for virtual machines, and Kubernetes for container usage. You can also manually set an environment variable on the command line.
export DB_PASSWORD=my-super-secret-pwd
To use this variable in the properties file, you reference it with the syntax
${ENV:VARIABLE}
. For example, if you want to use the password in a catalog
properties file like etc/catalog/db.properties
, add the following line:
connection-password=${ENV:DB_PASSWORD}
With this setup in place, the secret is managed by the provisioning system or by the administrators handling the machines. No secret is stored in the SEP configuration files on the filesystem or wherever they are managed.
Internal secrets management#
With internal secrets management, SEP dynamically retrieves secrets from secure configuration provider services at runtime. Because secrets are not stored in static properties files, internal secrets management provides a more secure way to retrieve sensitive data. Internal secrets management supports more complex scenarios such as automatic secret rotation, access controls, and audit logging.
SEP allows you to use the following configuration providers:
Use a secrets.toml
file to enable and configure your secrets provider. By
default, this file is located in the /etc
directory. Specify a different
location using the secretsConfig
system property.
The following example secrets.toml
file enables the Vault configuration
provider:
secrets-plugins-dir="/usr/lib/starburst-cache-service/secrets-plugins"
[vault]
secrets-provider.name = "vault"
address = "http://vault:8200"
token = "your-token"
Set the secrets-plugins-dir
property to the file path for the directory
containing secrets provider plugins. If you are using a standalone cache service
deployment, see the cache service
documentation for more information.
To configure multiple instances of the same secrets provider, use unique section names:
[vault_prod]
secrets-provider.name = "vault"
address = "http://vault-prod:8200"
token = "prod-token"
[vault_dev]
secrets-provider.name = "vault"
address = "http://vault-dev:8200"
token = "dev-token"
Reference a secrets provider instance in your configuration files using the section name:
connection-user = ${vault_prod:secret/db:user}
connection-password = ${vault_dev:secret/db:password}
Vault#
To enable Vault by Hashicorp as your
configuration provider, set the following properties in your secrets.toml
file:
[vault]
secrets-provider.name = "vault"
address = "http://vault:8200"
token = "your-token"
Reference values from Vault using the syntax ${VAULT:path:key}
, where path
is a secret path and key
is a secret key. For example:
connector.name=postgresql
connection-url=jdbc:postgresql://postgresql:15432/test
connection-user=${vault_one:secret/postgres:user}
connection-password=${vault_two:secret/postgres:password}
The following configuration properties are available:
Property |
Description |
---|---|
|
The address of the Vault server instance. This is a required property. |
|
The Vault authentication token. This is a required property. |
|
The global namespace of the Vault server instance. |
|
The KV Secrets Engine version of the Vault server instance. |
|
The path of a file containing an RSA private key, in unencrypted PEM format with UTF-8 encoding. |
|
The path of a file containing an X.509 certificate, in unencrypted PEM format with UTF-8 encoding. |
|
A JKS keystore file, containing a client certificate registered with Vault’s TLS Certificate auth backend. |
|
The password to the keystore. |
|
A JKS truststore file, containing the Vault server’s X509 certificate. |
|
A Boolean indicating whether to verify the SSL certificate used by Vault with HTTPS connections. |
|
The timeout value, in seconds, for establishing an HTTP(S) connection to the Vault server instance. |
|
The timeout value, in seconds, for downloading data after establishing an HTTP(S) connection. |
|
The depth of the prefix file path. |
|
The number of retry attempts for API operations after a failure. |
|
The wait time, in seconds, between retry attempts for API operations after a failure. |
AWS Secret Manager#
To enable AWS Secrets Manager as your
configuration provider, set the following properties in your secrets.toml
file:
[asm]
secrets-provider.name = "asm"
region = "us-east-2"
Reference values from AWS Secrets Manager using the syntax ${ASM:path}
or
${ASM:path:key}
. For example:
connector.name=postgresql
connection-url=jdbc:postgresql://postgresql:15432/test
connection-user=test
connection-password=${asm:test:password}
If your secret contains a JSON object, reference a property by using a JSON pointer expression.
The following configuration properties are available:
Property |
Description |
---|---|
|
The AWS access key used for authenticating with AWS Secrets Manager. |
|
The AWS secret key used for authenticating with AWS Secrets Manager. |
|
The endpoint URL for your AWS Secrets Manager service. |
|
The AWS region for your AWS Secrets Manager service. |
|
The ARN of the IAM role used when connecting to AWS Secrets Manager. |
|
The role session name used when connecting to AWS Secrets Manager. |
|
External ID for the IAM role trust policy used when connecting to AWS Secrets Manager. |
|
The endpoint URL for the AWS Security Token Service (STS) used for authenticating with AWS Secrets Manager. |
|
The AWS region for the AWS STS service. |
|
The maximum number of concurrent connections to AWS Secrets Manager. |
|
The URL of the HTTP proxy server used for connecting to AWS Secrets Manager. |
|
Boolean indicating whether the HTTP proxy uses TLS. |
Azure Key Vault#
To enable Azure Key Vault as your configuration provider, set the following
properties in your secrets.toml
file:
[azure-key-vault]
secrets-provider.name = "azure-key-vault"
secrets.azure.key-vault-name = "secrets"
Google Cloud Secret Manager#
To enable Google Cloud Secret Manager as your configuration provider, set the
following properties in your secrets.toml
file:
[gcp-secret-manager]
secrets-provider.name = "gcp-secret-manager"
gcp.credentials-key = "your-credentials-key"
CyberArk Conjur#
To enable CyberArk Conjur as your configuration provider, set the following
properties in your secrets.toml
file:
[conjur]
secrets-provider.name="conjur"
conjur.appliance-endpoint="https://conjur-proxy-nginx"
conjur.account="sep"
conjur.authentication-login="user@test"
conjur.api-key="your-credentials-key"
Property |
Description |
---|---|
|
The endpoint URL for your CyberArk Conjur server instance. It can be an HTTP or HTTPS endpoint. |
|
The CyberArk account that contains your secrets. |
|
The username to authenticate to CyberArk. |
|
The API key or password to authenticate to CyberArk. |
|
Set to |
|
The file path to your SSL keystore. |
|
The password for your SSL keystore. |
|
The file path to your SSL truststore. |
|
The password for your SSL truststore. |
Keystore#
To enable a Keystore as your configuration provider, set the following
properties in your secrets.toml
file:
[keystore]
secrets-provider.name = "keystore"
keystore-file-path = "/docker/presto-product-tests/conf/presto/etc/credential.jckes"
keystore-type = "pkcs12"
keystore-password = "password"
File#
To use one or more files as your configuration provider, set the following
properties in your secrets.toml
file:
secrets-provider.name="file"
allowed-file-variable-dir="/docker/presto-product-tests/conf/presto/etc"
You must specify a directory using the allowed-file-variable-dir
property.
This ensures that you only allow access to a specific directory on the host.
Note
Your files must be accessible on all nodes.
Reference values from a file using the syntax ${FILE:path}
. path
must
begin with the directory you specify in allowed-file-variable-dir
. For
example:
connector.name=postgresql
connection-url=jdbc:postgres://pg-instance-hostname:5432
connection-user=${FILE:/<path>/pg_user}
connection-password=${FILE:/<path>/pg_password}