Release 443-e LTS (31 May 2024)

Starburst Enterprise platform (SEP) 443-e LTS is the follow up release to the 438-e STS release and the 435-e LTS release.

This release is a promotion of the original 443-e STS release in May 2024 into a long term support (LTS) release.

It contains all improvements from Starburst Enterprise releases since the 435-e LTS release:

• 438-e STS

The 443-e release includes all improvements from the following Trino releases:

• Trino 439

• Trino 440

• Trino 441

• Trino 442

• Trino 443

This release is a long term support (LTS) release.

Highlights since 435-e

• Promoted Managed statistics, SAML 2.0 authentication, AWS Lake Formation access control support, and Shared queries to general availability.

• Added support for Apache Ozone.

• Improved the Starburst Enterprise web UI with accessibility enhancements.

• Added resource group ID to query information in the Starburst Enterprise web UI.

• Added location privilege support for built-in access control.

• Added support for Vault by Hashicorp and AWS Secrets Manager as external configuration providers.

Breaking changes

• As of SEP 438-e, Starburst Warp Speed uses a new file system caching mechanism. To use Starburst Warp Speed, you must add the --enable-preview flag to your jvm.config configuration. Additionally, the following catalog configuration properties have been removed:

  • warp-speed.workerdb.db.path

  • warp-speed.file-system-reserve-percentage

  • warp-speed.call-home.enable

  You must remove these configuration properties from your cluster configuration or the cluster fails to start.

• The Elasticsearch connector no longer supports Elasticsearch version 6.x or OpenSearch 1.x. Update Elasticsearch to version 7.x or 8.x to continue using the connector. To connect to OpenSearch, use the OpenSearch connector.

• SEP now requires JDK 21 to run. See the Java runtime environment requirements for more information.

• The hive.cache.enabled configuration property has been deprecated in favor of fs.cache.enabled. Remove and replace the deprecated property from all Hive catalog configurations.

• The legacy value for hive.security has been removed, the new default value is allow-all. See Authorization for more information.

• The following Hive authorization configuration properties have been removed. These properties must be removed from all configurations or the cluster does not start:

  • hive.allow-drop-table

  • hive.allow-rename-table

  • hive.allow-add-column

  • hive.allow-drop-column

  • hive.allow-rename-column

  • hive.allow-comment-table

  • hive.allow-comment-column

• Removed the service-database.connection-pool.enabled configuration property from the cache service. You must remove this configuration property or the cluster fails to start.

• The cache service now supports the same authentication methods as the SEP backend service database. As part of this change, the following cache service configuration properties have changed and must be updated in your configuration:

  • service-database.user to insights.jdbc.user.

  • service-database.password to insights.jdbc.password.

  • service-database.jdbc-url to insights.jdbc.url.

  • service-database.connection-pool.max-size to insights.jdbc.connection-pool.max-size.

  • service-database.connection-pool.idle-timeout to insights.jdbc.connection-pool.idle-timeout.

  Read more about the requirements for the cache service storage in the documentation.

• If you are using MySQL as the externally-managed database for the cache service, you must append the parameter sessionVariables=sql_mode=ANSI to the connection string you use in the insights.jdbc.url property or the cluster fails to start.

• This release removes the snowflake_distributed connector. You must remove or migrate existing Snowflake catalogs that use the distributed connector to the parallel connector or the cluster fails to start.

• If you are using Oracle as the externally-managed database for the Backend service and upgrading to this version from SEP version 435-e or earlier, you must take additional steps prior to upgrading SEP. Contact Starburst Support for assistance.

• Removed the defunct *.http-client.max-connection configuration properties. These properties must be removed from your configuration or the cluster does not start.

443-e initial changes

General

• Added public preview support for MaxCompute connector.

• Enabled PyStarburst dataframe API by default.

• Added support for Vault by Hashicorp and AWS Secrets Manager as external configuration providers.

• Added support for updating and creating specific views or materialized views with data products in the SEP REST API.

• Added support for automatic internal transport layer security (TLS) for managed statistics.

• Added support for using a stored procedure to manually refresh tables in a table scan redirection.

• Added keyboard shortcuts to the query name links in Saved queries and the assign button in Roles and Privileges in the Starburst Enterprise web UI to improve accessibility.

• Added a toggle switch in the What can they do? dialog and in the Switch role dialog in the Starburst Enterprise web UI to improve accessibility.

• Improved subquery cache hits by removing redundant predicates on data columns from cache key.

• Changed the experimental.thread-per-driver-scheduler-enabled property to be disabled by default.

• Increased character limit from 40 to 255 in the SEP REST API.

• Fixed a bug that caused the creation of materialized views to fail when using MySQL as the cache service backend database if materialized_view_definitions is longer than 64K characters.

• Fixed issue where a dynamic row filtering fallback mechanism could cause invalid results.

Db2 connector

• Added support for variable-precision timestamps to the nanosecond.

Delta Lake connector

• Improved speed at which tables and views are listed.

DynamoDB connector

• Added limited support for partial predicate pushdown.

• Fixed unbounded VARCHAR handling.

Hive connector

• Added support for comments on partitioned columns in the File and Thrift Hive metastores.

• Improved speed at which tables and views are listed.

• Fixed bug that caused DESCRIBE materialized_view to fail.

Iceberg connector

• Improved speed at which tables and views are listed.

443-e.1 changes (31 May 2024)

• Fixed failure when translating Hive views that contain EXISTS clauses.

• Fixed under-accounting of memory usage when writing strings to Parquet files.

• Fixed potential failure when reading ORC files larger than 2GB.

• Fixed startup failure when fault-tolerant execution is enabled with Google Cloud Storage exchange.

• Fixed potential loss of a query completion event when multiple queries fail at the same time.

• Fixed potential failure when queries contain filtered aggregations.

• Fixed under-accounting of memory usage when writing strings to Parquet files.

• Fixed complex predicate handling with table scan redirection.

• Fixed last openRecordGroup not processed in FlatArrayBuilder.

• Fixed potential query hang when there is an error processing data.

• Fixed incorrect results for distinct count aggregations over a constant value.

443-e.2 was skipped

443-e.3 changes (14 Jun 2024)

• Fixed potential correctness issue on receivers refresh that could cause query hanging. Applies to the Teradata Direct connector.

• Backported IMDSv2 service metadata access.

443-e.4 changes (28 Jun 2024)

• Fixed incorrect results when specifying a value for the cassandra.partition-size-for-batch-select configuration property.

• Fixed failure when reading Parquet files without field-id on structured types.

• Fixed failure when writing to tables with Iceberg VARBINARY values.

• Fixed rare query failure for array types when the data dictionary is encoded.

• Fixed failure when partition column name contains uppercase in UNLOAD.

443-e.5 was skipped

443-e.6 changes (11 Jul 2024)

• Added flag for cleaning the storage when the system is loaded when using Starburst Warp Speed.

• Added encoding to error code in OAuth2 callback handler.

• Fixed reading empty files from S3 and GCS.

• Fixed issue syncing partition metadata which could cause data deletion.

• Fixed a bug preventing use of Starburst security in the Delta Lake connector.

443-e.7 changes (29 Jul 2024)

• Fixed error when writing a large amount of data in S3 file system.

• Fixed failure when reading tables with NULL on partition columns while the optimize_metadata_queries session property is enabled.

443-e.8 changes (14 Aug 2024)

• Fixed failure when executing vacuum procedure on tables without old transaction logs.

• Fixed potential failure for queries involving GROUP BY, UNNEST, and filters over expressions that may produce an error for certain inputs.

• Fixed optimizer timeout for certain queries involving aggregations and CASE expressions.

• Fixed failure when adding new columns with a decimal type.

• Fixed failure to read Hive tables migrated to Iceberg with Apache Spark.

• Fixed issue that caused the error ‘Multiple masks on a single column are not supported’ to occur unintentionally.

443-e.9 changes (30 Aug 2024)

• Fixed failure when a user-defined type name contains uppercase characters.

• Fixed query failure when file-based network topology is configured with the node-scheduler.network-topology.file configuration property.

• Fixed support for migration of Ranger policies in security zones.

• Fixed performance issues with partitioned tables when using Lake Formation integration in Hive connector.

• Fixed numeric overflow during managed statistics computation for large tables in Teradata connector.

• Fixed an issue that affected managed statistics collection on wide Teradata tables in specific circumstances.

443-e.10 changes (13 Sep 2024)

• Fixed a bug that caused cluster metrics to be created with incorrect intervals and subsequently led to loss of cluster metrics data.

• Fixed memory tracking issue for aggregations that could cause worker crashes with out-of-memory errors.

• Fixed Run and troubleshoot feature when insights.authorized-groups configuration property contains authorized groups.

443-e.11 and 443-e.12 were skipped

443-e.13 changes (18 Oct 2024)

• Fixed OpenX JSON decoding a JSON array line that resulted in data being written to the wrong output column.

• Fixed reading large Prometheus responses.

• Fixed failures for count(*) queries with predicates containing non-ASCII strings.

443-e.14 changes (4 Nov 2024)

• Use hive.metastore.partition-batch-size.max config property value in sync_partition_metadata procedure. The default batch size is changed to 100 from 1000.

• Updated Iceberg connector migration procedure to use nullable columns by default.

443-e.15 changes (14 Nov 2024)

• Fixed memory leak in InMemoryEventClient within cache service.

443-e.16 changes (27 Nov 2024)

• Fixed incorrect results for queries filtering on a partition columns and the NAME column mapping is used.

• Fixed server error responses printing unprocessed user input.

443-e.17 changes (13 Dec 2024)

• Updated query result caching to use session property managers to resolve session property defaults and ensure that they are consistently applied.

• Fixed issue with hanging queries in Teradata Direct connector.

• Fixed failure of S3 file listing of buckets that enforce requester pays.

• Fixed incorrect quoting of output values when the CSV_UNQUOTED or CSV_HEADER_UNQUOTED format is used.