Starburst Metastore managed credentials#

Note

Managed credentials is a public preview feature.

Starburst Metastore can issue and store credentials centrally. Instead of hardcoding static key pairs in configuration files, you create credentials through the Starburst Metastore REST API. The metastore encrypts and stores them using a master key, and they can be rotated or revoked without editing files or restarting SEP.

Use the following steps to set up managed credentials, configure SEP catalogs to use them, and manage credentials through rotation and revocation.

Prerequisites#

  • Starburst Metastore is running and accessible.

  • You have a maintenance window. Credential checking changes when managed credentials are enabled, so expect a brief disruption during cutover.

  • If migrating from file-based credentials, back up the existing credentials file and configuration before proceeding.

Generate a master key#

Managed credentials are encrypted with a master key stored in a PKCS12 keystore. Generate one using the Java keytool utility, which is included with the JDK and creates the AES-256 symmetric key used to encrypt stored credentials:

PASSWORD=<keystore-password>
keytool -genseckey -alias kek-local-v1 -keyalg AES -keysize 256 \
  -storetype PKCS12 -keystore keyring.p12 -storepass "$PASSWORD"
printf 'kek-local-v1' > keyring.active

Mount both files into the Starburst Metastore container at a path it can read, such as /etc/starburst/keyring/. The files serve the following purposes:

  • keyring.p12: A password-protected PKCS12 keystore containing the AES-256 master key. The keystore can hold multiple keys to support rotation. Each key’s alias is its key ID.

  • keyring.active: A plain-text file containing the alias of the active master key.

Use a versioned naming convention for aliases, such as kek-local-v1 and kek-local-v2, to track which key is active.

Configure Starburst Metastore#

Set the following properties in your values.yaml to enable managed credentials:

etcFiles:
  config: |
    credentials-provider.type=managed
    credentials-provider.master-key.provider=LOCAL
    credentials-provider.master-key.local.keystore.path=/etc/starburst/keyring/keyring.p12
    credentials-provider.master-key.local.keystore.key=<keystore-password>
    credentials-provider.master-key.local.active-key.path=/etc/starburst/keyring/keyring.active
    credentials-provider.master-key.local.refresh-interval=1h

Restart Starburst Metastore to apply the configuration. For a full list of available properties, see Starburst Metastore configuration properties.

Create credentials#

Use the REST API to create credentials. All credential operations require HTTP Basic authentication with an admin user.

The secret is returned only once in the creation response. Save it before continuing, as it cannot be retrieved again.

curl -u "$USER:$PASSWORD" \
  -H 'Content-Type: application/json' \
  -X POST "http://starburst-portal:8080/public/api/v1/credential" \
  -d '{"name":"sep-prod","credentialKind":"AWSKEYPAIR","expiresInDays":90,"description":"prod glue access"}'

{
  "credentialId": "<credential-id>",
  "name": "sep-prod",
  "description": "prod glue access",
  "credentialKind": "AWSKEYPAIR",
  "publicId": "<access-key>",
  "createdBy": "<admin-user>",
  "status": "ACTIVE",
  "createdAt": "<created-timestamp>",
  "expiresAt": "<expiry-timestamp>",
  "secret": "<secret>"
}

To list existing credentials:

curl -u "$USER:$PASSWORD" "http://starburst-portal:8080/public/api/v1/credential"

{
  "nextPageToken": "",
  "result": [
    {
      "credentialId": "<credential-id>",
      "name": "sep-prod",
      "description": "prod glue access",
      "credentialKind": "AWSKEYPAIR",
      "publicId": "<access-key>",
      "createdBy": "<admin-user>",
      "status": "ACTIVE",
      "createdAt": "<created-timestamp>",
      "expiresAt": "<expiry-timestamp>"
    }
  ]
}

Note

Secrets are never returned in list responses.

For the full API reference, including request fields and response schema, see the Starburst Metastore REST API documentation.

Configure SEP catalogs#

Update each SEP catalog that connects to Starburst Metastore with the publicId and secret from the creation response:

hive.metastore.glue.aws-access-key=<access-key>
hive.metastore.glue.aws-secret-key=<secret>

Reload the catalog after updating. Repeat for every catalog that uses Starburst Metastore.

After setup#

Verify

Confirm that queries and object-storage operations using the managed credential succeed, and that any previously configured static credentials are rejected.

Decommission static credentials

After you verify that managed credentials work correctly, remove the old credentials file from your configuration.

Rollback

To roll back to file-based credentials:

  1. Set credentials-provider.type=file and restore the backed-up credentials file and configuration.

  2. Restart Starburst Metastore.

Rolling back is safe as long as the static credentials file and its keys remain valid. Complete the rollback before decommissioning the old credentials file.

Manage credentials#

All credential management operations use the admin-only REST API. For the full API reference, see the Starburst Metastore REST API documentation.

Rotate a credential#

Rotating a credential generates a new secret while keeping the same publicId. The new secret is shown only once:

curl -u "$USER:$PASSWORD" -X POST \
  "http://starburst-portal:8080/public/api/v1/credential/<credentialId>:rotate"

After rotating, update every catalog using that credential with the new secret and reload each catalog.

Revoke a credential#

Revoking a credential disables it immediately. Use this when you want to cut off access or when no catalog needs the credential:

curl -u "$USER:$PASSWORD" -X POST \
  "http://starburst-portal:8080/public/api/v1/credential/<credentialId>:revoke"

Delete a credential#

Deleting permanently removes a revoked credential:

curl -u "$USER:$PASSWORD" -X DELETE \
  "http://starburst-portal:8080/public/api/v1/credential/<credentialId>"

Rotate the master key#

To rotate the master key, add a new AES-256 entry under a new alias in keyring.p12, then update keyring.active to point to the new alias:

keytool -genseckey -alias kek-local-v2 -keyalg AES -keysize 256 \
  -storetype PKCS12 -keystore keyring.p12 -storepass "$PASSWORD"
printf 'kek-local-v2' > keyring.active

Starburst Metastore picks up the change on the next poll without a restart, and re-encrypts existing credentials under the new key. Keep the previous key in the keystore until re-encryption completes, then remove it.

Note

Back up both the metastore database and the PKCS12 keystore before rotating the master key. If rotation is interrupted, you need both to restore to the pre-rotation state.