> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Release version 0.17.0: October 2026

## Summary

Release `0.17.0` is mostly about making External Tables cheaper to run and easier to trust, and about tightening defaults in the minion tasks that keep upsert tables consistent. It also moves the runtime to JDK 25 and picks up a large Apache Pinot baseline update with a new `UUID` type, `OPEN_STRUCT` columns, and `GROUPING SETS` in the multi-stage engine.

* **External Tables:** Much lower server heap per segment, near-native index builds on Parquet-backed segments, Iceberg partition pruning at the broker, Parquet `INT96` support, working PromQL range functions, and fail-fast sync with health metrics.
* **Upsert and minion tasks:** A new `MERGE_JSON` partial-upsert merger, stricter table-config validation, and new defaults for SegmentRefreshTask. Upsert compaction tasks are now rejected on new configs in favor of SegmentRefreshTask.
* **Query:** Opt-in index-based `GROUP BY` fast paths, a preview of AI functions in SQL, and more native coverage in the StarTree Execution Engine.
* **Apache Pinot OSS:** See [Apache Pinot Highlights](#apache-pinot-highlights) for the open-source changes, including several single-stage engine behavior fixes that can change query results.

> [!NOTE]
>
> Several changes developed during this cycle also shipped in later 0.16 patch releases, including most StarTree Execution Engine work, the IAM `roleArn` option for Iceberg REST catalogs, and the system query log retention defaults. They aren't repeated below. If you're upgrading from the first 0.16 build, read the [0.16.0 notes](https://docs.startree.ai/reference/startree-release-notes/0.16.0.md) as well.

## StarTree Cloud highlights

### New Features

#### External Tables: lower memory, faster index builds, partition pruning

* **Lower server heap per segment:** The per-column state a server keeps for each loaded External Table segment dropped from about 4.3 KB to about 0.9 KB. Columns that are never queried hold almost nothing until they're read. Large External Tables with many columns and segments fit in less heap. For sizing guidance, see [Best practices and configs](https://docs.startree.ai/corecapabilities/external-table/best-practices-and-configs.md#caching-and-index-pinning).
* **Near-native index builds on Parquet-backed segments:** Star-tree, inverted, and range index builds now read Parquet through a build-optimized path. In internal benchmarks, a 300,000-row build went from about 4 minutes to about 3 seconds. See [External Table indexes](https://docs.startree.ai/corecapabilities/external-table/indexes.md).
* **Iceberg partition pruning:** Identity, bucket, and truncate partition transforms from Iceberg can be imported so the broker skips segments that can't match a query. Turn it on with `enable_partition_pruning = true` in SQL DDL, or `enablePartitionPruning` in the task config. It's off by default.
* **Parquet `INT96` columns:** Files with `INT96` timestamps, including `list<INT96>`, now load and query correctly. `INT96` footer statistics are ignored, so these columns don't get min/max pruning. If the table's time column is `INT96`, set `segmentTimeValueCheck` to `false`. See [INT96 in the data type mapping](https://docs.startree.ai/corecapabilities/external-table/data-type-mapping.md#int96-deprecated).
* **PromQL on External Tables:** Range functions (`rate`, `irate`, `increase`, `delta`) returned null for every bucket before this release, and `topk`/`bottomk` didn't work. Both are fixed, and range functions now scale linearly with the number of samples. See [PromQL functions](https://docs.startree.ai/corecapabilities/query_data/query_languages/promql.md#functions).
* **Sync fails fast:** External Table sync runs stop as soon as a segment goes to `ERROR`, and detect stalls after 5 minutes without progress instead of waiting out the full timeout. New controller metrics report unhealthy syncs and consecutive failures. See [External Table metrics and alerting](https://docs.startree.ai/corecapabilities/external-table/observability.md#metrics-and-alerting).
* **Lag snapshot size:** The External Table status and lag API now returns a raw `totalBytes` field alongside the human-readable size. See [Ingestion lag](https://docs.startree.ai/corecapabilities/external-table/observability.md#ingestion-lag).

#### `MERGE_JSON` partial-upsert merger

A new configurable partial-upsert merger merges JSON objects and `MAP` values per column instead of replacing them. Set `partialUpsertMergerClass` to `ai.startree.pinot.upsert.merger.ConfigurablePartialUpsertMerger` and mark each column with `startree.merger.columns.<column>=MERGE_JSON` in `metadataManagerConfigs`. The `startree.merger.json.mode` option picks `append` (the default, a recursive merge) or `override`. Don't combine it with `partialUpsertStrategies` on the same table. See [Merge JSON fields during partial upsert](https://docs.startree.ai/corecapabilities/manage-data/merge-json.md).

#### Additional New Features

* **AI functions in SQL (preview):** `AI_FILTER`, `AI_CLASSIFY`, and `AI_GENERATE` call a configured model from a single-stage query. They're off by default and need to be enabled on the cluster by StarTree. Contact StarTree support if you want to try them.
* **Conditional aggregation in Alter Table Task:** StarTreeAlterTableTask can aggregate a metric differently based on the value of another column (`SUM`, `MIN`, or `MAX` per value), with a fallback aggregation type for everything else. See [Alter Table Task output and rollup options](https://docs.startree.ai/corecapabilities/manage-data/alter-table-task.md#output-segment-and-rollup-options).
* **Index-based `GROUP BY` (opt-in):** For low-cardinality dictionary columns with an inverted index, `GROUP BY` with `COUNT`, `SUM`, `MIN`, `MAX`, or `AVG` can be answered from the index instead of scanning rows. Internal benchmarks showed 1.8x to 3.2x speedups. Enable per query with `SET useIndexBasedGroupByOperator=true`; the server-side switch has to be turned on as well.
* **Primary key lookup for upsert tables:** A controller debug endpoint, `GET /upsert/debug/{tableNameWithType}/metadata`, looks up a primary key across all replicas of a RocksDB-backed upsert table and reports whether the replicas agree. It needs table read permission. See [Off-heap upsert](https://docs.startree.ai/corecapabilities/manage-data/offheap-upsert.md).
* **MongoDB CDC lag:** MongoDB change-data-capture tables now report time-based ingestion lag through the standard realtime ingestion lag metrics. See [MongoDB CDC](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/mongodb-cdc.md).
* **System query log controls:** A table allowlist limits which tables are captured, and `SET skipQueryLog=true` excludes a single query. Queries against `system_query_log` itself, and requests from StarTree's health-check clients, are no longer captured by default. See [Query log](https://docs.startree.ai/corecapabilities/query_data/advanced_operations/query-logger.md).
* **Dynamic row filters:** StarTree now supports **query-time parameterized row filters**, allowing a single row-level access policy to serve multiple tenants or users without hardcoding individual filter values. Applications supply parameter values with each query, and StarTree automatically substitutes them into the policy and enforces the resulting row-level restrictions. See [Dynamic Row Filters](https://docs.startree.ai/corecapabilities/security/row-level-access-control.md#dynamic-row-filters-available-from-startree-release-0-17).

### Improvements

* **StarTree Execution Engine:** `IN` and `NOT IN` predicates now run natively instead of falling back to the Java engine. See [function support](https://docs.startree.ai/corecapabilities/query_data/query_languages/startree-engine.md#function-support).
* **External Tables:** Bounded, off-heap S3 range reads, batched bitmap posting reads, mmap'able Parquet page indexes, and fewer redundant S3 probes during segment load. Struct, list-of-struct, and map-of-struct columns now store nested schema metadata by default (see [Breaking Changes](#breaking-changes)). Binary, fixed, and UUID Iceberg columns are inferred as `BYTES`, and `required` fields carry `NOT NULL` into the Pinot schema. See [Data type mapping](https://docs.startree.ai/corecapabilities/external-table/data-type-mapping.md).
* **Tiered storage:** Time-based tiers on External Tables can now use a non-zero `segmentAge`. The server disk cache keeps cached pages across restarts, so servers come back warm. See [Set the segment age boundary](https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/setup.md#set-the-segment-age-boundary) and [Server restart optimizations](https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/setup.md#server-restart-optimizations).
* **Ingestion:** Composite JSON indexes index native `MAP` values without re-parsing them. See [Indexes](https://docs.startree.ai/corecapabilities/manage-data/indexes/overview.md).
* **Raw forward indexes default to V6 and `ZSTANDARD`:** Raw forward indexes that don't set `compressionCodec` now use `ZSTANDARD` instead of `LZ4`. New clusters also write the V6 format by default, which stores per-value sizes instead of cumulative offsets in each chunk header so the header compresses better. In internal tests on realistic data, `ZSTANDARD` with V6 made raw forward indexes about 56% smaller than `LZ4` with V4. `ZSTANDARD` alone accounted for about 38%, and V6 cut query CPU 4–8% compared with V4. `ZSTANDARD` costs more CPU to decompress than `LZ4`, most visibly on tiered storage. Existing clusters keep writing V4 until StarTree switches them; contact StarTree support to move a cluster to V6. Only newly built segments change, and columns that set `rawIndexWriterVersion` or `compressionCodec` keep their value ([#17966](https://github.com/apache/pinot/pull/17966)). See [Raw forward index format V6](https://docs.startree.ai/corecapabilities/manage-data/indexes/forward-index.md#raw-forward-index-format-v6).
* **Upsert:** Invalid partial-upsert column and function combinations are rejected with HTTP 400 at table create or update, including SQL `CREATE`. Existing tables aren't rescanned. See [Merge JSON fields during partial upsert](https://docs.startree.ai/corecapabilities/manage-data/merge-json.md).
* **Platform:** The runtime moved from JDK 21 to JDK 25.

### Bug Fixes

* **Query correctness:** Null rows are now left out of sparse and min/max index filter results when null handling is on.
* **Upsert and RocksDB:** Fixed a race between RocksDB preload finalization and store close, and RocksDB state is now recreated cleanly after a failed preload.
* **External Tables:** Fixed several Parquet reader issues, including struct leaf pages misaligned across page boundaries and a direct-memory leak in the S3 block cache. The broker now reports when it can't load the latest Iceberg snapshot instead of silently serving the previous one.
* **Minion tasks:** Fixed duplicate and unwanted [Alter Table Task](https://docs.startree.ai/corecapabilities/manage-data/alter-table-task.md) stale-segment tasks, and moved Alter Table Task deep-store cleanup off the task generation path.
* **Security:** [SegmentPurgeTask](https://docs.startree.ai/corecapabilities/manage-data/purge-task.md) now rejects selector queries that contain more than one statement, and Iceberg REST catalog endpoints are validated. Dependency CVEs were fixed in BouncyCastle, Jackson, Guava, lz4, and async-http-client.

### Breaking Changes

* **Upsert compaction tasks are rejected on new configs.** Table create and update now reject a config that adds `UpsertCompactionTask` or `UpsertCompactMergeTask` to `taskConfig`, or sets `upsertConfig.enableCommitTimeCompaction` to `true`. These can leave replicas with inconsistent metadata. Use SegmentRefreshTask instead; see [Upsert compaction with SegmentRefreshTask](https://docs.startree.ai/corecapabilities/manage-data/upsert-compaction-srt.md). Tables that already have one of these settings keep working, and removing the setting is always allowed. There's no override.
* **SegmentRefreshTask skips segments without a recognized sequence ID.** `skipInvalidSequenceId` now defaults to `true`, so segments whose names don't carry a sequence ID that SegmentRefreshTask recognizes are left out of merges. Set `skipInvalidSequenceId` to `false` in the task config to restore the old behavior. See [How to configure the SegmentRefreshTask](https://docs.startree.ai/corecapabilities/manage-data/segment-refresh-task.md#how-to-configure-the-segmentrefreshtask).
* **External Table snapshot defaults changed.** For tables without an explicit value, `iceberg.snapshotProcessing.retention.maxActiveVersions` drops from 5 to 2, so older snapshot versions are pruned on the next retention run. `iceberg.snapshotProcessing.batchSize` goes from 1,000 to 10,000 files per sync. Set either one explicitly on the table to keep the old value. See [Snapshot processing (Iceberg)](https://docs.startree.ai/corecapabilities/external-table/best-practices-and-configs.md#snapshot-processing-iceberg).
* **External Tables store nested schema metadata by default.** `storeNestedSchema.enabled` now defaults to `true` for External Table sync. On existing tables with nested columns, the first sync after upgrade may update the schema and reload segments. Set `storeNestedSchema.enabled` to `false` to keep the old behavior. See [Best practices and configs](https://docs.startree.ai/corecapabilities/external-table/best-practices-and-configs.md).
* **Minion tasks are rejected on segment-group tables.** A table with `segmentGroupConfig.enabled` can't have any minion task in `taskConfig`, except ExternalTableSyncTask running on the controller. See [Segment groups](https://docs.startree.ai/corecapabilities/external-table/segment-groups.md).
* **Deletion vector table options were removed.** The table-level keys `iceberg.dv.fetch.mode`, `iceberg.dv.puffinCache.maxBytes`, and `iceberg.dv.puffinCache.downloadTimeoutMs` are now ignored. The deletion vector cache is managed at the server level. See [Deletion vectors](https://docs.startree.ai/corecapabilities/external-table/deletion-vectors.md).
* **JDK 25 runtime.** StarTree Pinot now builds and runs on JDK 25 only. This only matters if you deploy custom plugins in a BYOC environment. Rebuild them against JDK 25 and check any dependency on internal (non-SPI) Pinot classes before upgrading.

## Apache Pinot highlights

This section describes [**Apache Pinot**](https://github.com/apache/pinot) (open source) changes in the baseline that ships with StarTree Cloud **0.17.0** compared with **0.16.0**. It doesn't describe StarTree-only extensions, and it leaves out changes that already shipped in 0.16 patch releases. Links in parentheses go to the upstream pull request, which has the full details.

### New Features

* **`UUID` data type:** A logical `UUID` column type stored as 16 bytes, with ingestion, `CAST`, literals, bloom filters, `GROUP BY`, `DISTINCT`, and multi-stage engine support. New functions include `IS_UUID`, `TO_UUID`, `UUID_TO_STRING`, `UUID_TO_BYTES`, `BYTES_TO_UUID`, `UUID_VERSION`, `UUID_TIMESTAMP`, `UUID_V4`, and `UUID_V7` ([#18870](https://github.com/apache/pinot/pull/18870), [#19091](https://github.com/apache/pinot/pull/19091)).
* **`OPEN_STRUCT` columns:** A schemaless key/value column type. Keys are discovered at ingestion; frequently present keys are stored as their own columns and the rest go into a JSON column. Query a key with `col['key']` ([#18643](https://github.com/apache/pinot/pull/18643)).
* **`GROUPING SETS`, `ROLLUP`, and `CUBE`** in the multi-stage engine, with `GROUPING()` and `GROUPING_ID()` ([#18817](https://github.com/apache/pinot/pull/18817)).
* **Window functions:** The `EXCLUDE CURRENT ROW | GROUP | TIES | NO OTHERS` frame clause ([#18482](https://github.com/apache/pinot/pull/18482)), and `IGNORE NULLS` for `LEAD` and `LAG` ([#18202](https://github.com/apache/pinot/pull/18202)). See [Window functions in MSQE](https://docs.startree.ai/corecapabilities/query_data/query_languages/msqe.md#window-functions).
* **Segment metadata virtual columns:** `$creationTime`, `$startTimeMs`, `$endTimeMs`, `$totalDocs`, and `$segmentCrc`. They're `NULL` on consuming segments ([#19179](https://github.com/apache/pinot/pull/19179)).
* **Approximate function rewrite:** An opt-in broker setting rewrites `DISTINCTCOUNT` and `COUNT(DISTINCT)` to `DISTINCTCOUNTSMARTHLL`, and `PERCENTILE` to `PERCENTILESMARTTDIGEST`. Override per query with the `useApproximateFunction` query option ([#19510](https://github.com/apache/pinot/pull/19510)).
* **Page cache warmup:** The `pageCacheWarmupConfig` table config warms the OS page cache when a server restarts or a segment is refreshed, with a time limit and QPS limit ([#16033](https://github.com/apache/pinot/pull/16033)).
* **Kafka config providers:** Realtime `streamConfigs` can use Kafka `ConfigProvider` references such as `${file:path:key}`, for example to read credentials from a file ([#19202](https://github.com/apache/pinot/pull/19202)).
* **Smaller additions:**
  * A `segmentAgeField` tier option (`endTime`, `startTime`, or `creationTime`) ([#19415](https://github.com/apache/pinot/pull/19415)). See [Set the segment age boundary](https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/setup.md#set-the-segment-age-boundary).
  * JSON and `UUID` support in `ARRAY_AGG` ([#19333](https://github.com/apache/pinot/pull/19333)), and multi-value columns in `DISTINCTCOUNTULL` ([#19302](https://github.com/apache/pinot/pull/19302)).
  * `BYTES` array literals ([#19247](https://github.com/apache/pinot/pull/19247)).
  * A `setOpOptions` hint to run `UNION`, `INTERSECT`, and `EXCEPT` without a shuffle when inputs are colocated ([#18804](https://github.com/apache/pinot/pull/18804)). See [Set operations in MSQE](https://docs.startree.ai/corecapabilities/query_data/query_languages/msqe.md#set-operations).

### Improvements

* **Multi-stage planning:** Large `IN` lists are kept out of Calcite optimization, which cuts planning time for queries with long value lists ([#19674](https://github.com/apache/pinot/pull/19674)).
* **Colocated joins:** Broker pruning now removes pruned partitions from colocated joins ([#19174](https://github.com/apache/pinot/pull/19174)), and colocated joins work when a partition has no segments instead of failing ([#19166](https://github.com/apache/pinot/pull/19166)). See [Table joins in MSQE](https://docs.startree.ai/corecapabilities/query_data/query_languages/msqe.md#table-joins).
* **Routing:** Upsert tables using strict replica-group routing now get adaptive server selection when an adaptive selector is configured ([#18947](https://github.com/apache/pinot/pull/18947)).
* **Metadata-based aggregation:** When a query mixes aggregations that can be answered from segment metadata with ones that can't, the metadata-answerable ones still skip the scan ([#18334](https://github.com/apache/pinot/pull/18334)).
* **Null handling:** With null handling enabled, every aggregation now skips `NULL` inputs, `GROUP BY` on dictionary and multi-value columns produces a proper `NULL` group, and filters follow three-valued logic. For example, `NOT (col = x)` no longer counts `NULL` rows. Results with null handling off are unchanged ([#19158](https://github.com/apache/pinot/pull/19158)).
* **Query options:** New broker settings can warn about or reject unknown SQL query options and the legacy `OPTION(...)` syntax. Both are off by default ([#19027](https://github.com/apache/pinot/pull/19027), [#19570](https://github.com/apache/pinot/pull/19570)).
* **Query log redaction:** A broker setting can redact literal values or full SQL text from query logs ([#18604](https://github.com/apache/pinot/pull/18604)).
* **Java 25:** The Apache Pinot baseline now builds and runs on JDK 25 ([#19014](https://github.com/apache/pinot/pull/19014)).

### Bug Fixes

* **Query correctness:** Fixed `GROUP BY` returning wrong groups or failing when the initial result holder capacity was optimized ([#19379](https://github.com/apache/pinot/pull/19379)), expression filters attributing matches to the wrong rows ([#19245](https://github.com/apache/pinot/pull/19245)), and the final `GROUP BY` schema depending on which server responded first ([#19543](https://github.com/apache/pinot/pull/19543)).
* **Ingestion:** Transactional Kafka topics no longer raise false stream data loss alerts ([#19161](https://github.com/apache/pinot/pull/19161)).
* **Instance assignment:** Fixed fault-domain-aware instance assignment when pool tags don't start at 0 ([#19266](https://github.com/apache/pinot/pull/19266)).
* **Arrow responses:** Fixed Arrow-encoded responses failing on non-string object values ([#19591](https://github.com/apache/pinot/pull/19591)).

### Breaking Changes

* **Single-stage engine no longer ignores `HAVING` and `QUALIFY`.** `HAVING` used to be dropped silently in some single-stage queries, and `QUALIFY` was ignored. `HAVING` is now applied, and invalid `HAVING` clauses (without `GROUP BY`, or on an ungrouped column) fail with an error like the multi-stage engine ([#19554](https://github.com/apache/pinot/pull/19554)). `QUALIFY` is rejected in the single-stage engine; run those queries on the [multi-stage engine](https://docs.startree.ai/corecapabilities/query_data/query_languages/msqe.md) ([#19553](https://github.com/apache/pinot/pull/19553)). A `FILTER (WHERE ...)` on an aggregation used only in `HAVING` or `ORDER BY` is now applied too ([#19437](https://github.com/apache/pinot/pull/19437)). Queries that relied on the old behavior may return fewer rows or fail.
* **Admin API access control is enforced when authentication is configured.** If a server access-control factory is configured, server admin routes (other than health checks and table data downloads) require admin authorization ([#19233](https://github.com/apache/pinot/pull/19233)). Controller cluster-level endpoints check the caller's permission instead of allowing any authenticated user ([#19234](https://github.com/apache/pinot/pull/19234)). `LOOKUP()` in the single-stage engine requires access to the dimension table, and is rejected if that table has a row-level security filter ([#19209](https://github.com/apache/pinot/pull/19209)). Clusters without access control configured are unaffected. See [Manage access](https://docs.startree.ai/corecapabilities/security/manage-access.md).
* **Deprecated REST endpoints removed.** Controller `GET /tables/{tableName}/livebrokers` is gone; use `GET /tables/livebrokers?tables=...`. Server `GET /segments/{table}/{segment}/validDocIds` and `GET /tables/{table}/validDocIdMetadata` are gone; use `validDocIdsBitmap` and `POST validDocIdsMetadata` ([#19141](https://github.com/apache/pinot/pull/19141), [#19142](https://github.com/apache/pinot/pull/19142)).
* **Upsert and dedup partition count is locked.** Changing `numPartitions` in `segmentPartitionConfig` on an upsert or dedup table is rejected as a backward-incompatible update ([#18868](https://github.com/apache/pinot/pull/18868)).
* **Experimental enriched join removed.** The `JoinToEnrichedJoin` planner rule is ignored and those queries run as normal joins ([#19033](https://github.com/apache/pinot/pull/19033)).
* **DDL validation errors return HTTP 400** instead of 500 ([#19544](https://github.com/apache/pinot/pull/19544)).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.