Skip to main content
This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it. It is offered as a preview capability; validate it in a non-production environment before relying on it.
Iceberg tables record row-level deletes and updates as deletion vectors (Puffin-format delete files) rather than rewriting data files in place. Historically, External Tables on Pinot only handled append-only Iceberg sources — row-level deletes and updates from the source were not reflected in query results. This release adds native support for reading and applying Iceberg v3 Puffin deletion vectors at query time, so External Tables stay correct as source data is updated or deleted.

​
How it works

  1. Server-side loading — each server decodes the Iceberg Puffin deletion files relevant to its segments and applies them when answering a query, so deleted/updated rows are excluded from results.
  2. Atomic snapshot readiness — before a new Iceberg snapshot becomes visible to queries, the controller polls every server until all of them confirm they’ve loaded the deletion vectors for that snapshot. This avoids a window where some servers see the new snapshot’s deletes and others don’t.
  3. Broker-side pinning and consistency — queries pin an explicit snapshot per table so all servers involved in a query agree on which snapshot (and therefore which deletes) to use. An invalid or not-yet-ready pinned snapshot fails the query rather than silently mixing snapshot state.

​
Enabling deletion vectors

Set on the table’s ExternalTableSyncTask config:

​
Controller: snapshot readiness

The controller also exposes a readiness endpoint:
Returns {"ready": true|false}. Returns 412 if the table has not initialized a deletion-vector manager (i.e. enableDeletionVectors=false).

​
Broker: pruning and pinning

Broker-side deletion-vector pruning only activates for OFFLINE tables with enableDeletionVectors=true — it is a zero-cost no-op otherwise.

​
Query-time snapshot pinning

Deletion-vector-enabled tables require an explicit pinned snapshot per query, set with the snapshotVersionByTable query option (this replaces the older single-table snapshotVersion option):
If the pinned snapshot is invalid or not yet confirmed ready on all relevant servers, the query fails with a QUERY_EXECUTION error rather than returning results computed from a mix of snapshot states.

​
Storage format

The deletion-vector index itself is stored as a sorted, dictionary-encoded, bloom-filtered Parquet file (dv-index.parquet) rather than JSON — cutting heap usage for large delete sets by roughly an order of magnitude compared to the original JSON-based format. There is no backward-compatibility bridge from the old JSON format; this applies to newly-synced deletion vector state.

​
Observability

The run-status endpoint’s failurePhase field (see Observability) can also report IS_EV_CONVERGENCE or SNAPSHOT_READINESS_POLL when a sync run fails during deletion-vector convergence.

​
FAQs

​
Do I need to set snapshotVersionByTable on every query once deletion vectors are enabled?

Yes — it’s required for deletion-vector-enabled tables. Without it, or with an invalid/not-yet-ready snapshot, the query fails rather than returning potentially inconsistent results.

​
What happens to existing (JSON-format) deletion-vector state after upgrading?

There is no automatic migration from the JSON format. This feature targets newly-synced deletion vector state — check with StarTree support before relying on it against a table with pre-existing deletion-vector data.

​
Does this work for raw S3 Parquet (non-Iceberg) External Tables?

No — deletion vectors are an Iceberg-specific capability (deletionVectorSource=iceberg), tied to Iceberg’s Puffin delete-file format.