> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Snapshot Retention

Snapshot retention cleans up old snapshot-processing artifacts and Pinot segments for StarTree Iceberg/S3 tables that use `ExternalTableSyncTask`.

Retention keeps a bounded list of active snapshots, preserves the current in-progress snapshot, deletes Pinot segments no longer referenced by retained snapshots, and sweeps old orphan snapshot artifact directories.

## What retention manages

Snapshot-processing ingestion writes artifacts under a snapshot directory. A snapshot directory can contain:

* `final.parquet`, the merged file for a completed snapshot.
* Batch Parquet files under `batches/` for in-progress work.
* Auxiliary files such as deletion-vector index artifacts.

Retention uses the active snapshot list and the current watermark to decide what is still live.

| State                     | Retention behavior                                                                                                                          |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Retained active snapshot  | Kept. Segments referenced by its `final.parquet` are kept in Pinot.                                                                         |
| Evicted active snapshot   | Removed from the active list when it exceeds `maxActiveVersions`. Segments only referenced by evicted snapshots become deletion candidates. |
| In-progress snapshot      | Kept. Segments referenced by in-progress batch files are protected.                                                                         |
| Orphan snapshot directory | Deleted when it is not retained, not in progress, and older than the oldest surviving snapshot.                                             |

## Configure retention

Configure retention on the external table's `ExternalTableSyncTask` config.

```json theme={null}
{
  "tableName": "orders_ext_OFFLINE",
  "tableType": "OFFLINE",
  "task": {
    "taskTypeConfigsMap": {
      "ExternalTableSyncTask": {
        "executor": "controller",
        "catalogType": "iceberg-rest",
        "catalog.iceberg-rest.restUri": "https://iceberg-catalog.example.com",
        "catalog.iceberg-rest.warehouse": "s3://warehouse",
        "catalog.iceberg-rest.table.namespace": "analytics",
        "catalog.iceberg-rest.table.tableName": "orders",
        "iceberg.snapshotProcessing.retention.enabled": "true",
        "iceberg.snapshotProcessing.retention.maxActiveVersions": "5",
        "iceberg.snapshotProcessing.retention.deletedSegmentRetention": "7d"
      }
    }
  }
}
```

## Retention properties

| Property                                                       | Default | Description                                                                                |
| -------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------ |
| `iceberg.snapshotProcessing.retention.enabled`                 | `true`  | Enables or disables snapshot retention for the table.                                      |
| `iceberg.snapshotProcessing.retention.maxActiveVersions`       | `5`     | Maximum active snapshots retained. Values below `2` are ignored and the default is used.   |
| `iceberg.snapshotProcessing.retention.deletedSegmentRetention` | `7d`    | Pinot deleted-segment retention period passed to segment deletion after snapshot eviction. |

<Note>
  Snapshot artifacts live under each component's deep-store root at `snapshots/<tableNameWithType>/<snapshotId>/` — derived from `controller.data.dir` (controller), `pinot.server.instance.segment.store.uri` (server), and `pinot.broker.data.dir` (broker). The earlier `iceberg.snapshotProcessing.snapshotDirURI` override has been removed; if an old table config still carries it, it is ignored.
</Note>

Controller-level scheduling is managed by StarTree:

| Controller property                                           | Default | Description                                                    |
| ------------------------------------------------------------- | ------- | -------------------------------------------------------------- |
| `controller.iceberg.snapshot.retention.frequencyPeriod`       | `6h`    | How often the leader controller runs snapshot retention.       |
| `controller.iceberg.snapshot.retention.initialDelayInSeconds` | `120`   | Delay before the first retention run after controller startup. |

<Note>
  In StarTree Cloud, controller-level retention cadence is managed by StarTree. Per-table retention properties are the normal user-facing controls.
</Note>

## Cleanup flow

Each retention run does the following for eligible external tables:

1. Reads the current Pinot ideal-state segment set.
2. Reads the current ingestion watermark to find an in-progress snapshot, if any.
3. Reads the active snapshot list.
4. Evicts oldest active snapshots until the list has at most `maxActiveVersions` entries.
5. Builds a keep set from surviving snapshots' `final.parquet` files and in-progress batch files.
6. Deletes Pinot segments not present in the keep set, using `deletedSegmentRetention`.
7. Sweeps orphan snapshot directories older than the oldest surviving snapshot.

This ordering protects snapshots that complete while retention is running and avoids deleting segments needed by an in-progress snapshot.

## Verify retention behavior

Use the observability APIs to check the current watermark and snapshot artifacts. The `checkpointValue` field in the [status endpoint](./observability#run-status) response is the current watermark.

```bash theme={null}
curl -X GET \
  "https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot/tables/orders_ext_OFFLINE/externalTable/status" \
  -H "Authorization: Bearer $STARTREE_TOKEN"
```

For a completed retained snapshot, `finalParquet` should be available:

```bash theme={null}
curl -X GET \
  "https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot/tables/orders_ext_OFFLINE/externalTable/snapshotProcessing/463829118102938475/finalParquet" \
  -H "Authorization: Bearer $STARTREE_TOKEN"
```

For an evicted or swept snapshot, the same endpoint returns `404`:

```json theme={null}
{
  "reason": "SNAPSHOT_NOT_FOUND",
  "tableNameWithType": "orders_ext_OFFLINE",
  "snapshotId": 463829118102938475
}
```

## Choosing retention values

| Goal                                                  | Suggested setting                                                |
| ----------------------------------------------------- | ---------------------------------------------------------------- |
| Keep rollback room for several completed snapshots    | Increase `maxActiveVersions`.                                    |
| Minimize segment and artifact storage                 | Lower `maxActiveVersions`, but keep it at `2` or higher.         |
| Give operators time to inspect deleted Pinot segments | Increase `deletedSegmentRetention`.                              |
| Clean old segment metadata faster after eviction      | Lower `deletedSegmentRetention` after validating rollback needs. |

## Operational guidance

* Do not disable retention indefinitely on high-volume external tables; snapshot-processing artifacts can grow quickly.
* Keep `maxActiveVersions` high enough to cover your operational rollback and audit window.
* Keep `deletedSegmentRetention` aligned with incident response. A very short value can make rollback harder after an accidental eviction.
* If retention logs report missing or corrupt `final.parquet` for a retained snapshot, the run skips pruning for that table and retries on the next tick.
* If `finalParquet` returns `404` for a recent snapshot, check the checkpoint state. It may still be `IN_PROGRESS`, `FAILED`, or already evicted by retention.