> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Upsert & Dedup Operations Guide
> Supported configurations, operational procedures, and troubleshooting guidance for running upsert and dedup tables safely in StarTree Cloud.
Upsert and dedup tables maintain per-key state across replicas, which makes them more sensitive to configuration choices and operational procedures than append-only tables. This guide lists which configurations are supported, the safe way to perform common operations, and what to do when results look wrong.
## Supported configurations at a glance
| Configuration | Status | Guidance |
| --------------------------------------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `now()` or other current-time functions in ingestion transforms | **Not supported** | Evaluates differently on each replica, causing replicas to disagree. Rejected at table-creation time in an upcoming release. |
| Changing the Kafka partition count of a live upsert table | **Not supported** | Upsert state is partition-aligned; there is no recovery path after a partition-count change. Plan partition counts before go-live. |
| Multiple comparison columns | **Not supported for new tables** | Use a single comparison column. |
| `enableCommitTimeCompaction` | **Not supported** | Do not enable — risks replica inconsistency. |
| `UpsertCompactionTask` / `UpsertCompactMergeTask` | **Deprecated** | Use the [Segment Refresh Task](https://docs.startree.ai/corecapabilities/manage-data/upsert-compaction-srt) instead. These tasks will be removed in an upcoming release. |
| `dropOutOfOrderRecord` | **Use with caution** | Prefer default out-of-order handling unless you have a confirmed need. |
| Pauseless consumption + **dedup** | **Supported** | Proven at production scale. Note: a segment in error state does not auto-recover and requires support assistance. |
| Pauseless consumption + **partial or full upsert** | **Not supported** | Do not combine until support is announced. |
| `SegmentRefreshTask` on a **partial**-upsert table | **Not supported** | Known correctness limitations; it also disables in-order consumption enforcement (see below). |
| `SegmentRefreshTask` together with `FileIngestionTask` | **Supported with configuration** | Exclude file-ingested segments from refresh runs — see [the snippet below](#running-segmentrefreshtask-alongside-fileingestiontask). |
| `metadataTTL` with `SegmentRefreshTask` | **Not supported** | Do not combine. |
| `deletedKeysTTL` | **Not yet fully supported** | Check with support before enabling. |
## Operational procedures
### Reloading or force-committing a partial-upsert table
Before reloading segments or force-committing on a partial-upsert table, set the following cluster config:
```
pinot.server.consuming.segment.consistency.mode: PROTECTED
```
This setting takes effect immediately — no server restart is required. It is applied at the next segment commit. Never set this mode to `UNSAFE`.
`PROTECTED` mode is planned to become the default in an upcoming release.
### Rebalancing a partial-upsert or dedup table
Pause ingestion before rebalancing, and resume it after the rebalance completes. Rebalancing while consuming can cause per-key state to diverge across the moved replicas.
### Migrating or cloning a live upsert table
A refresh run that overlaps the copy can replace segments mid-migration, leaving the clone with missing rows.
Copy the table using your normal procedure.
Verify total row counts (and per-partition counts for large tables) match before resuming tasks and cutting traffic over.
### Running SegmentRefreshTask alongside FileIngestionTask
If a table runs both tasks, exclude file-ingested segments from refresh runs by adding a segment selector to the `SegmentRefreshTask` config:
```json theme={null}
{
"SegmentRefreshTask": {
"segmentSelector": "!(segment.download.url =~ '.*FileIngestionTask.*')"
}
}
```
Without this filter the two tasks can clash on segment names and produce data issues.
## Protections that are on by default
These behaviors are active without appearing in your table config:
### In-order consumption enforcement (`enforceConsumptionInOrder`)
Since release 0.15, partial-upsert, dedup, and `dropOutOfOrderRecord` tables enforce in-order consumption automatically, which improves data correctness. You can confirm it is active by looking for `ConsumerCoordinator` entries in server logs.
In-order enforcement is **not applied** while the table runs `SegmentRefreshTask`, `FileIngestionTask`, or `StarTreeAlterTableTask`. A partial-upsert table running the Segment Refresh Task therefore has no in-order protection — this combination is not supported (see the matrix above).
### Snapshot tasks
`UpsertSnapshotCreationTask` and `DedupSnapshotCreationTask` run automatically every 12 hours for upsert and dedup tables (since release 0.10). On releases **before 0.15**, any table-config update permanently cancelled this default schedule — if snapshots stopped after a config change on an older release, re-apply the schedule or upgrade.
## Troubleshooting
### Segment Refresh Task or compaction task keeps failing
Recent releases add safety guards that fail a task rather than risk data corruption. Mitigation:
1. Set cluster config `pinot.server.consuming.segment.consistency.mode: PROTECTED`
2. Set table config `parallelSegmentConsumptionPolicy: DISALLOW_ALWAYS`
If the task still fails after both settings, contact StarTree support.
### Wrong or flip-flopping query results
Alternating results between runs of the same query usually means replicas have diverged.
Do **not** reload the table as a first step. Reload replaces each server's local data with the deep-store copy — if the deep-store copy is the inconsistent one, the reload locks the bad data in. Contact StarTree support first so the inconsistent segments can be identified and preserved for analysis.
### Stale derived column after a partial-upsert merge
If a derived column shows a stale value after partial-upsert merges, compute the value in the query (from the source columns) as a workaround while a fix is in progress.
## Roadmap
An upcoming release converts the "not supported" items above into hard blocks at table-creation/config time, and makes `PROTECTED` consistency mode the default. Longer term, a simplified upsert data model is planned that reduces configuration surface and removes most of these exceptions.