Snapshot Retention

Snapshot retention cleans up old snapshot-processing artifacts and Pinot segments for StarTree Iceberg/S3 tables that use ExternalTableSyncTask. Retention keeps a bounded list of active snapshots, preserves the current in-progress snapshot, deletes Pinot segments no longer referenced by retained snapshots, and sweeps old orphan snapshot artifact directories.

What retention manages

Snapshot-processing ingestion writes artifacts under a snapshot directory. A snapshot directory can contain:

final.parquet, the merged file for a completed snapshot.
Batch Parquet files under batches/ for in-progress work.
Auxiliary files such as deletion-vector index artifacts.

Retention uses the active snapshot list and the current watermark to decide what is still live.

State	Retention behavior
Retained active snapshot	Kept. Segments referenced by its `final.parquet` are kept in Pinot.
Evicted active snapshot	Removed from the active list when it exceeds `maxActiveVersions`. Segments only referenced by evicted snapshots become deletion candidates.
In-progress snapshot	Kept. Segments referenced by in-progress batch files are protected.
Orphan snapshot directory	Deleted when it is not retained, not in progress, and older than the oldest surviving snapshot.

Configure retention

Configure retention on the external table’s ExternalTableSyncTask config.

{
  "tableName": "orders_ext_OFFLINE",
  "tableType": "OFFLINE",
  "task": {
    "taskTypeConfigsMap": {
      "ExternalTableSyncTask": {
        "executor": "controller",
        "catalogType": "iceberg-rest",
        "catalog.iceberg-rest.restUri": "https://iceberg-catalog.example.com",
        "catalog.iceberg-rest.warehouse": "s3://warehouse",
        "catalog.iceberg-rest.table.namespace": "analytics",
        "catalog.iceberg-rest.table.tableName": "orders",
        "iceberg.snapshotProcessing.retention.enabled": "true",
        "iceberg.snapshotProcessing.retention.maxActiveVersions": "5",
        "iceberg.snapshotProcessing.retention.deletedSegmentRetention": "7d"
      }
    }
  }
}

Retention properties

Property	Default	Description
`iceberg.snapshotProcessing.retention.enabled`	`true`	Enables or disables snapshot retention for the table.
`iceberg.snapshotProcessing.retention.maxActiveVersions`	`5`	Maximum active snapshots retained. Values below `2` are ignored and the default is used.
`iceberg.snapshotProcessing.retention.deletedSegmentRetention`	`7d`	Pinot deleted-segment retention period passed to segment deletion after snapshot eviction.

Snapshot artifacts live under each component’s deep-store root at snapshots/<tableNameWithType>/<snapshotId>/ — derived from controller.data.dir (controller), pinot.server.instance.segment.store.uri (server), and pinot.broker.data.dir (broker). The earlier iceberg.snapshotProcessing.snapshotDirURI override has been removed; if an old table config still carries it, it is ignored.

Controller-level scheduling is managed by StarTree:

Controller property	Default	Description
`controller.iceberg.snapshot.retention.frequencyPeriod`	`6h`	How often the leader controller runs snapshot retention.
`controller.iceberg.snapshot.retention.initialDelayInSeconds`	`120`	Delay before the first retention run after controller startup.

In StarTree Cloud, controller-level retention cadence is managed by StarTree. Per-table retention properties are the normal user-facing controls.

Cleanup flow

Each retention run does the following for eligible external tables:

Reads the current Pinot ideal-state segment set.
Reads the current ingestion watermark to find an in-progress snapshot, if any.
Reads the active snapshot list.
Evicts oldest active snapshots until the list has at most maxActiveVersions entries.
Builds a keep set from surviving snapshots’ final.parquet files and in-progress batch files.
Deletes Pinot segments not present in the keep set, using deletedSegmentRetention.
Sweeps orphan snapshot directories older than the oldest surviving snapshot.

This ordering protects snapshots that complete while retention is running and avoids deleting segments needed by an in-progress snapshot.

Verify retention behavior

Use the observability APIs to check the current watermark and snapshot artifacts. The checkpointValue field in the status endpoint response is the current watermark.

curl -X GET \
  "https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot/tables/orders_ext_OFFLINE/externalTable/status" \
  -H "Authorization: Bearer $STARTREE_TOKEN"

For a completed retained snapshot, finalParquet should be available:

curl -X GET \
  "https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot/tables/orders_ext_OFFLINE/externalTable/snapshotProcessing/463829118102938475/finalParquet" \
  -H "Authorization: Bearer $STARTREE_TOKEN"

For an evicted or swept snapshot, the same endpoint returns 404:

{
  "reason": "SNAPSHOT_NOT_FOUND",
  "tableNameWithType": "orders_ext_OFFLINE",
  "snapshotId": 463829118102938475
}

Choosing retention values

Goal	Suggested setting
Keep rollback room for several completed snapshots	Increase `maxActiveVersions`.
Minimize segment and artifact storage	Lower `maxActiveVersions`, but keep it at `2` or higher.
Give operators time to inspect deleted Pinot segments	Increase `deletedSegmentRetention`.
Clean old segment metadata faster after eviction	Lower `deletedSegmentRetention` after validating rollback needs.

Operational guidance

Do not disable retention indefinitely on high-volume external tables; snapshot-processing artifacts can grow quickly.
Keep maxActiveVersions high enough to cover your operational rollback and audit window.
Keep deletedSegmentRetention aligned with incident response. A very short value can make rollback harder after an accidental eviction.
If retention logs report missing or corrupt final.parquet for a retained snapshot, the run skips pruning for that table and retries on the next tick.
If finalParquet returns 404 for a recent snapshot, check the checkpoint state. It may still be IN_PROGRESS, FAILED, or already evicted by retention.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

What retention manages

Configure retention

Retention properties

Cleanup flow

Verify retention behavior

Choosing retention values

Operational guidance

​What retention manages

​Configure retention

​Retention properties

​Cleanup flow

​Verify retention behavior

​Choosing retention values

​Operational guidance

What retention manages

Configure retention

Retention properties

Cleanup flow

Verify retention behavior

Choosing retention values

Operational guidance