> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Release version 0.16.0: July 2026

# **Summary**

Release `0.16.0` centers on scaling **StarTree External Tables to large, frequently-updated Iceberg datasets** and maturing the **StarTree Execution Engine** (native multi-stage query runtime), alongside broad reliability and observability hardening across ingestion, upsert/dedup, and task orchestration.

* **External Tables at scale:** Segment grouping and native Iceberg deletion-vector support let external tables handle far larger file counts and row-level updates/deletes without re-ingestion.
* **StarTree Execution Engine maturity:** New join types, full `BIG_DECIMAL` support, spill-to-disk for memory-intensive queries, and transparent fallback to the legacy engine close major functional gaps.
* **Operational reliability:** The Minion Task Orchestration Framework is now on by default, audit logging is on by default, and a significant query-correctness fix eliminates silent wrong results under certain distributed execution plans.
* **Apache Pinot OSS:** The open-source baseline continues to track upstream Apache Pinot; see [Apache Pinot Highlights](#apache-pinot-highlights) below for the OSS-only feature summary.

# **StarTree Cloud highlights**

## **New Features**

### **External Tables: Segment Groups & Deletion Vectors bring Iceberg to scale**

External Tables take a major step forward this release with two foundational capabilities that make large, frequently-updated Iceberg tables production-ready:

* **Segment groups:** Many small Iceberg-derived segments are now grouped into a single logical unit end-to-end (controller, broker, and server), including for realtime tables, dramatically cutting cluster metadata overhead so external tables can scale to far larger file counts.
* **Deletion vectors:** Pinot now natively applies Iceberg's row-level delete/update markers at query time, with atomic snapshot synchronization, server-side loading, and broker-side read consistency. The underlying index moved from JSON to a scalable Parquet-backed format with lazy point lookups.
* **Broadened catalog and auth support:** Unity Catalog, Google Cloud Storage (via S3 interoperability), and OAuth2 client-credentials authentication for Iceberg REST catalogs.
* **Iceberg schema evolution:** Field-ID/alias-based column resolution keeps the Pinot table schema automatically in sync as the source Iceberg table's schema evolves.

For onboarding and usage details, see the [External Tables documentation](https://docs.startree.ai/corecapabilities/external-table/overview), including new pages on [Segment Groups](https://docs.startree.ai/corecapabilities/external-table/segment-groups) and [Deletion Vectors](https://docs.startree.ai/corecapabilities/external-table/deletion-vectors).

### **StarTree Execution Engine matures significantly**

* Added sort-merge join and native lookup-join operators, plus full `BIG_DECIMAL` support (order-preserving encoding, casts, arithmetic, and aggregation — double-precision, not arbitrary-precision), closing major functional gaps with the legacy Java engine.
* Added spill-to-disk for memory-intensive aggregations, sorts, and `GROUP BY` operations, backed by an auto-sized shared memory pool, so large queries degrade gracefully instead of failing with out-of-memory errors.
* The engine now falls back transparently to the Java engine for a wider range of previously-unsupported plans and functions, improving coverage without hard failures.
* Expanded SQL surface: `NOT IN` predicates, native array construction, automatic type widening for mixed-precision arithmetic, and multi-value column exchanges.
* Reduced per-query engine overhead through internal performance work on the native execution path.

This is an opt-in, beta capability — see [StarTree Execution Engine](https://docs.startree.ai/corecapabilities/query_data/query_languages/startree-engine) for how to enable it and its join-hint/spill-to-disk configuration.

### **Additional New Features**

* **[Multi-Stage Materialized Views](https://docs.startree.ai/corecapabilities/query_data/materialized-views/multi-stage-materialized-views):** A new StarTree-specific minion task materializes multi-stage SQL — including joins — into a regular table on a schedule, queried directly (no automatic query rewrite).
* **Kafka 4.0 connector:** A new Kafka 4.0 stream connector plugin ships alongside the existing Kafka 3.0 plugin.

## **Improvements**

* **Native / MSE query engine:** Always-on per-operator metrics, cleaner error messages (no more internal stack traces leaking to clients), configurable network runtime worker threads, independently configurable broker-as-worker behavior, and engine wait times now bounded by the query's actual timeout instead of a fixed value.
* **External Tables & segment groups:** Concurrent per-column prefetch, coalesced S3 range reads, local-disk caching of Parquet footers, parallel segment-group member loading, uniform tier/partition validation across segment-group members, and hardened lifecycle handling with failure isolation.
* **Ingestion:** Delta Lake ingestion now supports globally consistent push (atomic table swap with reconciliation); the Minion Task Orchestration Framework (MTOF) is enabled by default at the cluster level with new orchestration observability metrics; ingestion now integrates with Pinot's OOM-protection safeguards.
* **Upsert & Dedup:** Snapshot watermarks and locations are now persisted to ZooKeeper for durability across restarts; TTL cleanup is capped at the minion snapshot watermark to avoid removing data still needed for consistency.
* **Segment & Indexing:** Parquet reader performance improvements (thread-local buffer reuse, typed array stores, reduced redundant page lookups), support for `DATA_PAGE_V2`-encoded Parquet files, a bounded/shared decoded dictionary cache with orthogonal TTL eviction, and faster composite JSON index remaps.
* **Security & Observability:** Audit logging is now enabled by default on the controller and broker — see [Breaking Changes](#breaking-changes) below, since this turns on request-payload capture by default; the base runtime container image was hardened for FIPS compliance; Query Log (`system_query_log`) gained a configurable tenant, a `queryHash` column, and a default 200 GiB storage cap (up from 100 GiB) with compression on heavy string columns.
* **Table Preview:** Clearer error messages on datasource connection failures, and catalog table sampling is now parallelized at the file and column level for large catalogs.

## **Bug Fixes**

* **Query correctness:** Fixed a bug where hash-distributed exchanges in the native engine could silently return wrong results — the most significant correctness fix this release. Also fixed several native-engine crashes, panics, and query hangs under specific plan shapes.
* **External Tables & Segment Groups:** Fixed several correctness and reliability issues in Parquet/Iceberg readers, deletion-vector reads now fail closed rather than silently proceeding when snapshot state can't be read, and fixed catalog/task config drift and orphaned-fragment cleanup.
* **Upsert & Dedup:** Fixed upsert segment cleanup skipping over-replicated empty segments and a double-acquire race during reload batching.
* **Security:** Fixed a Regular-Expression-Denial-of-Service (ReDoS) vulnerability in query normalization, and removed a vulnerable embedded-Jetty dependency (CVE-2026-2332).
* **Platform:** Fixed client-facing APIs incorrectly returning HTTP 500 instead of 400 for client errors, and fixed inconsistent updates to concurrency-limit configuration.

## **Breaking Changes**

* **Audit logging is now on by default, including request-payload capture.** Both controller and broker seed `pinot.audit.{controller,broker}.enabled=true` and `capture.request.payload.enabled=true` (plus default header/exclude-pattern/JWT-claim settings) on the controller's next restart after upgrade — on both new and existing clusters. Any key you've already set explicitly is left untouched. If you need audit logging (or payload capture specifically) to stay off, set `pinot.audit.controller.enabled=false` / `pinot.audit.broker.enabled=false` via `POST /cluster/configs` **before** restarting the controller on 0.16.0. See [Pinot Audit Logging](https://docs.startree.ai/corecapabilities/security/audit/pinot-audit-logs#0160-defaults-and-how-to-opt-out) for the full default set.
* **Minion Task Orchestration Framework (MTOF) is now enabled by default** at the cluster level (`controller.startree.task.manager.enableTaskOrchestration`, previously off by default). Set it to `false` to disable orchestration cluster-wide, or set `forceLegacyTaskFlow=true` on a specific task type to route only that task type to the legacy flow. See [Minion Task Orchestration](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration).
* **`EXPLAIN` output now includes segment-level query plans by default.** `pinot.query.multistage.explain.include.segment.plan` now defaults to `true` (was `false`), which may affect tooling that parses `EXPLAIN` output. Use `EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR <query>` for the old logical-plan-only output, or set the property back to `false` explicitly.
* **Default RocksDB `max_open_files` for upsert/dedup storage was raised from 1024 to 4096.** This is the shared off-heap upsert/dedup RocksDB store's file-handle cap — clusters with tight file-descriptor `ulimit`s should verify headroom before upgrading. Any explicitly-set value is unaffected.
* **The `ls` alias for the `LsDeepStore` admin CLI subcommand (`bin/startree-pinot-admin.sh`) was removed, with no deprecation period** — only the full `LsDeepStore` command name resolves now. This affects BYOC/BYOK environments with direct CLI access; update any scripts calling `... ls <uri>` to `... LsDeepStore <uri>` before upgrading.
* **Default MSE metrics mode changed:** `pinot.metrics.mse.mode` now defaults to `MSE` on brokers and `DUAL` on servers (OSS default: `SERVER`). Brokers stop emitting `pinot.server.*`-namespaced MSE metrics and start emitting `pinot.mse.*` instead; servers emit both namespaces. Set `pinot.metrics.mse.mode=SERVER` explicitly to keep the old broker behavior.

# **Apache Pinot Highlights**

This section describes [**Apache Pinot**](https://github.com/apache/pinot) (open source) changes in the baseline that ships with StarTree Cloud **0.16.0** compared with **0.15.0**. It does not describe StarTree-only extensions.

## New Features

* **New SQL functions** — `OVERLAY`, `isPrivateIp`, `arraySliceLong`, unary `+`/`-` operators, and multi-column `CORRELATE_BY` support in `FUNNEL_COUNT`.
* **New REST/admin APIs** — Query-syntax validation without execution, table-config validation/auto-tuning, and cron-based periodic task scheduling.

## Improvements

* **Broker-side segment pruning for the multi-stage engine is now on by default** — Multi-stage queries now prune segments at the broker during routing, using the table's configured segment pruners (time, partition, etc.), before fanning out leaf stages to servers — matching single-stage engine behavior. Support was added in 0.15.0 but off by default on the default planning path; 0.16.0 flips `pinot.broker.multistage.logical.planner.use.broker.pruning` to `true`. Override per query with the `useBrokerPruning` query option, or set the broker config to `false` to restore the previous behavior.
* **MSE query optimizer** — New Calcite optimization rules, join-predicate pushdown re-enabled by default, aggregate pushdown through `UNION ALL`, and broker-side short-circuiting when all segments are pruned.
* **MSE observability** — Configurable metrics modes, adaptive-routing in-flight-request stats, per-server stats surfaced on streaming query responses, and a hook to expose the compiled MSE query plan.
* **Index and query performance** — Reduced FST regexp memory footprint with interruptible matching, avoided posting-list copying on `JSON_MATCH`, and reduced HLL dictionary sizes.
* **Ingestion** — Faster JSON-heavy ingestion transforms, intermediate transform columns no longer need to exist in the schema, and upsert `validDocIds` consensus is now enforced across task generators.
* **Reliability** — Table configs with unresolved environment-variable templates are now rejected at write time instead of failing later, and a backward-incompatibility check now protects upsert `deleteRecordColumn` changes.
* **Apache Calcite upgraded to 1.42.0**, bringing newer SQL planner/optimizer capabilities from upstream.

## Bug Fixes

* **Query correctness** — Fixed column-to-column comparisons in `WHERE` clauses, incorrect millisecond-precision truncation for `TIMESTAMP` comparisons in the multi-stage engine, and MSE null-handling with `OR` predicates.
* **Index correctness** — Fixed an out-of-bounds error in the Lucene text-index reuse path on retried segment conversion, and a doc-ID mapping corruption on unpaired UTF-16 surrogates.
* **Stability** — Fixed several null-pointer exceptions and race conditions across controller, server, and minion, plus instance-partition assignment bugs for multi-stream realtime tables.
* **Client and API fixes** — The broker query API now returns HTTP 400 (not 500) for client errors, and numerous Java client contract bugs were fixed for endpoints including schema listing, rebalance, and task management.
* **Security** — Excluded a vulnerable embedded Jetty dependency pulled in transitively via Hadoop (CVE-2026-2332).

## Backwards Incompatible Changes

* **`TimeFieldSpec` deprecated** — New schemas can no longer declare a `TimeFieldSpec`; use a regular `DateTimeFieldSpec` for the time column instead.
* **Batch message mode removed** — The already-dormant batch message mode has been removed and can no longer be enabled.