Azure Storage file system support#
Trino includes a native implementation to access Azure Data Lake Storage Gen2 with a catalog using the Delta Lake, Hive, Hudi, or Iceberg connectors.
Enable the native implementation with fs.azure.enabled=true in your
catalog properties file. Additionally, the Azure storage account must have
hierarchical namespace enabled.
SEP also supports Microsoft Fabric OneLake locations. See Microsoft Fabric OneLake support for details.
General configuration#
Use the following properties to configure general aspects of Azure Storage file system support:
Property |
Description |
|---|---|
|
Activate the native implementation for Azure Storage support. Defaults to
|
|
Authentication type to use for Azure Storage access. Defaults to |
|
Comma-separated list of hostname suffixes of the Azure storage endpoints
that locations in the catalog can reference. Defaults to |
|
Deprecated. Use |
|
Data size for blocks during read operations. Defaults
to |
|
Data size for blocks during write operations.
Defaults to |
|
Maximum number of concurrent write operations. Defaults to 8. |
|
Data size Defaults to |
|
Maximum integer number of concurrent HTTP requests to
Azure from every node. Defaults to double the number of processors on the
node. Minimum |
|
Duration that a connection can remain idle in the
connection pool before it is closed. Defaults to |
|
Duration for HTTP request timeouts, including the
time taken by the Azure SDK to retry a request. Defaults to |
|
Specify the application identifier appended to the |
|
Enable multipart writes for large files. Defaults to |
Microsoft Fabric OneLake support#
SEP can access data in Microsoft Fabric OneLake with a catalog using the Delta Lake, Hive, Hudi, or Iceberg connectors. OneLake is always hierarchical, so SEP does not check for hierarchical namespace support on OneLake locations.
OneLake is served from the fabric.microsoft.com endpoint instead of
core.windows.net. To enable access, add fabric.microsoft.com to the
azure.endpoints catalog configuration property. A single catalog can
reference locations from multiple endpoints, so you can keep core.windows.net
in the list to continue accessing Azure Data Lake Storage Gen2 locations:
fs.azure.enabled=true
azure.endpoints=core.windows.net,fabric.microsoft.com
Reference OneLake data with an abfs[s] location in the following form, where
<workspace> is the name of your Fabric workspace:
abfss://<workspace>@onelake.dfs.fabric.microsoft.com/<item>.lakehouse/Files/<path>
Regional hostnames, such as <region>-onelake.dfs.fabric.microsoft.com, and
private endpoint hostnames, such as <guid>.z54.dfs.fabric.microsoft.com, are
also supported.
User-assigned managed identity authentication#
Use the following properties to configure user-assigned managed identity authentication to Azure Storage:
Property |
Description |
|---|---|
|
Must be set to |
|
Specifies the client ID of user-assigned managed identity. |
|
Specifies the resource ID of user-assigned managed identity. |
Only one of azure.user-assigned-managed-identity.client-id or azure.user-assigned-managed-identity.resource-id can be
specified.
Access key authentication#
Use the following properties to configure access key authentication to Azure Storage:
Property |
Description |
|---|---|
|
Must be set to |
|
The decrypted access key for the Azure Storage account. Requires
authentication type |
OAuth 2.0 authentication#
Use the following properties to configure OAuth 2.0 authentication to Azure Storage:
Property |
Description |
|---|---|
|
Must be set to |
|
Tenant ID for Azure authentication. |
|
The endpoint URL for OAuth 2.0 authentication. |
|
The OAuth 2.0 service principal’s client or application ID. |
|
A OAuth 2.0 client secret for the service principal. |
Access multiple storage accounts#
To allow Trino to access multiple Azure storage accounts from a single catalog configuration, you can use OAuth 2.0 authentication with an Azure service principal. The following steps describe how to create a service principal in Azure and assign an IAM role granting access to the storage accounts:
Create a service principal in Azure Active Directory using Azure App Registrations and save the client secret.
Assign access to the storage accounts from the account’s Access Control (IAM) section. You can add Role Assignments and select appropriate roles, such as Storage Blob Data Contributor.
Assign access using the option User, group, or service principal and select the service principal created. Save to finalize the role assignment.
Once you create the service principal and configure the storage accounts use the Client ID, Secret and Tenant ID values from the application registration, to configure the catalog using properties from OAuth 2.0 authentication.
Migration from legacy Azure Storage file system#
Previous Trino releases included a legacy Azure Storage file system
implementation used by catalogs configured with fs.hadoop.enabled and
hive.azure.* properties. That legacy support has been removed. Use the
native Azure file system implementation.
To migrate a catalog to use the native file system implementation for Azure, make the following edits to your catalog configuration:
Add the
fs.azure.enabled=truecatalog configuration property.If your catalog enabled
fs.hadoop.enabledonly for legacy Azure Storage access, remove that property.Configure the
azure.auth-typecatalog configuration property.Refer to the following table to rename your existing legacy catalog configuration properties to the corresponding native configuration properties. Supported configuration values are identical unless otherwise noted.
Legacy property |
Native property |
Notes |
|---|---|---|
|
|
|
|
|
Also see |
|
|
|
|
|
|
|
|
Remove the following legacy configuration properties if they exist in your catalog configuration:
hive.azure.abfs-storage-accounthive.azure.wasb-access-keyhive.azure.wasb-storage-account
