Unity catalog with Delta Lake#

The Delta Lake connector supports a subset of operations across managed and external tables, as well as Databricks Unity Catalog-managed tables with catalog-managed commits, when using the Databricks Unity Catalog as a metastore on AWS, Azure, or Google Cloud. The following table outlines which operations are supported.

Supported operations for Unity Catalog with Delta Lake#
Supported for	Operation	Notes
External tables	CREATE TABLE, INSERT, UPDATE, MERGE, DELETE, DROP TABLE, READ	To perform write operations on external tables with Unity Catalog, you must meet Databricks’ requirements for creating external Delta tables from external clients.
Unity Catalog-managed tables with catalog-managed commits	INSERT, UPDATE, MERGE, DELETE, DROP, READ	A Unity Catalog-managed table with catalog-managed commits is a managed table specific to Unity. These tables are created in Databricks and require the `delta.feature.catalogManaged` table feature.
Managed tables	READ-only	Managed tables that are not Unity Catalog-managed tables with catalog-managed commits are read-only.

Configuration#

To use Unity Catalog metastore, add the following configuration properties to your catalog configuration file:

delta.security=unity
hive.metastore.unity.host=<HOST_NAME>
hive.metastore.unity.token=<TOKEN>
hive.metastore.unity.catalog-name=<CATALOG_NAME>

The following table shows the configuration properties used to connect SEP to Unity Catalog as a metastore.

Unity configuration properties#
Property name	Description
`hive.metastore.unity.host`	Name of the host without http(s) prefix. For example: `dbc-a1b2345c-d6e7.cloud.databricks.com`
`hive.metastore.unity.token`	The personal access token used to authenticate a connection to the Unity Catalog metastore. For more information about generating access tokens, see the Databricks documentation.
`hive.metastore.unity.catalog-name`	Name of the catalog in Databricks.
`hive.metastore.unity.catalog-managed-table-enabled`	Enables support for Unity Catalog-managed tables with catalog-managed commits.
`hive.metastore.unity.vended-credentials-enabled`	Enables support for credentials provided by Unity for file system access.
`hive.metastore.unity.proxy.enabled`	Use a proxy server for communication with Unity. Default is false.
`hive.metastore.unity.proxy.host`	Hostname of the proxy server.
`hive.metastore.unity.proxy.port`	Port number of the proxy server.
`hive.metastore.unity.proxy.username`	Proxy username to use if connecting through a proxy server.
`hive.metastore.unity.proxy.password`	Proxy password to use if connecting through a proxy server.
`hive.metastore.unity.proxy.non-proxy-hosts`	Hosts list to access without going through the proxy server.
`hive.metastore.unity.connect-timeout`	Connection timeout for HTTP calls to the Unity Catalog server. Defaults to `30s`.
`hive.metastore.unity.read-timeout`	Read timeout for HTTP calls to the Unity Catalog server. Defaults to `60s`.

Unity Catalog-managed tables with catalog-managed commits#

Note

Write support for Unity Catalog-managed tables with catalog-managed commits is currently in private preview and considered experimental by Databricks. While SEP provides limited support for this functionality, it relies on specific table configurations that may change without notice.

When using Unity Catalog-managed tables with catalog-managed commits, add hive.metastore.unity.catalog-managed-table-enabled=true to your catalog configuration file. This property enables SEP to recognize and write to these tables within Unity Catalog.

Then add the following table properties on the Delta Lake table in Databricks to ensure compatibility with SEP:

CREATE TABLE catalog_name.schema_name.table_name (id int
) USING delta
TBLPROPERTIES (
  'delta.feature.catalogManaged' = 'supported',
  'delta.enableRowTracking' = 'false',
  'delta.checkpointPolicy' = 'classic'
)

Enable OAuth 2.0 token passthrough#

The Unity Catalog supports OAuth 2.0 token pass-through.

To enable OAuth 2.0 token passthrough:

Add the following configuration properties to the config.properties file on the coordinator:

http-server.authentication.type=DELEGATED-OAUTH2
web-ui.authentication.type=DELEGATED-OAUTH2
http-server.authentication.oauth2.scopes=<AZURE_DATABRICKS_APPLICATION_ID>/.default,openid
http-server.authentication.oauth2.additional-audiences=<AZURE_DATABRICKS_APPLICATION_ID>

Replace <AzureDatabricks-ApplicationID> with the Application ID for your Azure Databricks Microsoft Application which can be found in your Azure Portal under Enterprise applications.

Add only the following configuration properties to the delta.properties catalog configuration file:

delta.metastore.unity.authentication-type=OAUTH2_PASSTHROUGH
delta.security=unity
hive.metastore-cache-ttl=0s

Limitations:

Credential passthrough is only supported with Azure Databricks and when Microsoft Entra is the IdP.
When enabling credential passthrough you cannot use Hive Passthrough.

Credential vending#

This feature supports data stored in Amazon S3, ADLS Gen2, and Google Cloud Storage (GCS). Credential vending lets SEP request short-lived, temporary credentials from Unity Catalog REST API at query time. Credentials are automatically refreshed and scoped to the table ID or external location in Unity Catalog.

To enable credential vending, add the following property to your catalog configuration file:

hive.metastore.unity.vended-credentials-enabled=true

Managed tables#

The following describes operations supported for Unity-managed tables.

Supported credential vending operations#
Supported for	Description
Managed tables	Supports reading using credential vending with the table ID to request temporary credentials.
Unity Catalog-managed tables with catalog-managed commits	Supports both reading and writing using credential vending, with the table ID to request temporary credentials.

External tables#

The following describes operations supported for external tables.

Supported credential vending operations#
Supported for	Description
External tables	Supports both reading and writing using credential vending, with the table location to request temporary credentials. The location prefix must already be configured with associated credentials in Unity.

Dynamic configuration passthrough#

The Delta Lake connector supports dynamic configuration passthrough, which lets a single catalog connect to different Delta Lake Unity Catalog metastores and Amazon S3 backends on a per-connection basis. Instead of setting fixed Unity Catalog connection and storage values in the catalog configuration file, you configure the catalog to read these values from extra credentials that each client supplies at query time.

To enable dynamic configuration passthrough, add the following properties to your catalog configuration file:

connector.name=delta-lake
hive.metastore=unity
delta.dynamic-configuration-passthrough.enabled=true

If Unity Catalog provides vended credentials for storage access, configure the Unity connection properties only:

delta.dynamic-configuration-passthrough.enabled=true
dynamic.hive-metastore-unity-host.credential-name=unity_host_credential
dynamic.hive-metastore-unity-token.credential-name=unity_token_credential
dynamic.hive-metastore-unity-catalog-name.credential-name=unity_extra_credential
dynamic.hive-metastore-unity-vended-credentials-enabled.credential-name=unity_vended_credentials_extra_credential

Unity Catalog dynamic connection properties#
Property name	Description
`dynamic.hive-metastore-unity-host.credential-name`	Name of the extra credential key that supplies the Unity Catalog host.
`dynamic.hive-metastore-unity-token.credential-name`	Name of the extra credential key that supplies the Unity Catalog access token.
`dynamic.hive-metastore-unity-catalog-name.credential-name`	Name of the extra credential key that supplies the Unity Catalog catalog name.
`dynamic.hive-metastore-unity-vended-credentials-enabled.credential-name`	Name of the extra credential key that sets whether to use Unity-vended credentials for file system access. When present, this values overrides the static `hive.metastore.unity.vended-credentials-credentials-enabled` catalog setting for that request. Optional.

If clients supply Amazon S3 credentials directly, add the S3 credential name mappings:

delta.dynamic-configuration-passthrough.enabled=true
dynamic.hive-metastore-unity-host.credential-name=unity_host_credential
dynamic.hive-metastore-unity-token.credential-name=unity_token_credential
dynamic.hive-metastore-unity-catalog-name.credential-name=unity_extra_credential
dynamic.hive-metastore-unity-vended-credentials-enabled.credential-name=unity_vended_credentials_extra_credential
dynamic.s3.region.credential-name=s3_region
dynamic.s3.aws-access-key.credential-name=aws_s3_access_key_id
dynamic.s3.aws-secret-key.credential-name=aws_s3_secret_key
dynamic.s3.aws-account-id.credential-name=aws_s3_account_id
dynamic.s3.aws-session-token.credential-name=aws_s3_session_token
dynamic.s3.aws-endpoint.credential-name=s3_endpoint

S3 direct access properties#
Property name	Description
`dynamic.s3.region.credential-name`	Name of the extra credential key that supplies the S3 region.
`dynamic.s3.aws-access-key.credential-name`	Name of the extra credential key that supplies the AWS access key.
`dynamic.s3.aws-secret-key.credential-name`	Name of the extra credential key that supplies the AWS secret key.
`dynamic.s3.aws-account-id.credential-name`	Name of the extra credential key that supplies the AWS account ID. Optional.
`dynamic.s3.aws-session-token.credential-name`	Name of the extra credential key that supplies the AWS session token. Optional.
`dynamic.s3.aws-endpoint.credential-name`	Name of the extra credential key that supplies the S3 endpoint URL. Optional.

After you configure the catalog with the appropriate properties, you can run queries using the Trino CLI by supplying --extra-credential values whose key names match the credential names defined in the catalog configuration file.

Location alias mapping#

If you are using Unity catalog as a metastore when accessing external tables, the Starburst Delta Lake connector supports using a bucket-style alias for your Amazon S3 bucket access point.

To enable location alias mapping:

Create a bucket alias mapping file in JSON format:

{
  "bucket_name_1": "bucket_alias_1",
  "bucket_name_2": "bucket_alias_2"
}

Add the following properties to your catalog configuration:

location-alias.provider-type=file
location-alias.mapping.file.path=<PATH_TO_BUCKET_ALIAS_MAPPING_FILE>

Optionally, use location-alias.mapping.file.expiration-time to specify the interval at which SEP rereads the bucket alias mapping file. The default is 1m.

SEP uses the new external location path specified in the bucket alias mapping file to access the data. Only the bucket name is replaced. The URI is otherwise unchanged.