Storage #

Starburst Galaxy handles storage by connecting to your object storage location. There, Galaxy determines the table format and file type for each table from the metadata in the location’s metastore.

This link between your object storage and Starburst Galaxy is called Great Lakes connectivity, and is supported by the following catalogs:

• Amazon S3 catalogs
• Azure Data Lake Storage catalogs
• Google Cloud Storage catalogs

Great Lakes connectivity #

In Starburst Galaxy, you do not connect to a table format’s catalog. Instead, you connect to a storage location. Galaxy then recognizes and reads from any of the supported table formats listed below that may be found in the same storage location.

The Galaxy technology that makes this possible is called Great Lakes connectivity. This feature abstracts the details about different table formats when you use certain write access statements for locations in an object storage system.

Great Lakes connectivity is built into Galaxy, and is available to all users.

Access to existing tables works transparently for all table formats. Galaxy recognizes the format of your tables by reading the metastore associated with your object storage.

This feature supports the following table formats, which are described on separate pages:

• Iceberg table format
• Delta Lake table format
• Hive table format
• Hudi table format

Default table format #

Tables in any supported format are automatically read. When creating new tables or views, you can specify the default table format when creating a new object storage catalog.

Table properties #

Each table format has its own set of properties that are supported by Great Lakes connectivity as part of CREATE TABLE statements.

Read more about the available table properties for each table format:

• Delta Lake table properties
• Hive table properties
• Iceberg table properties

Session properties #

See Shared session properties.