Skip to main content
Upsert and dedup tables maintain per-key state across replicas, which makes them more sensitive to configuration choices and operational procedures than append-only tables. This guide lists which configurations are supported, the safe way to perform common operations, and what to do when results look wrong.

​
Supported configurations at a glance

​
Operational procedures

​
Reloading or force-committing a partial-upsert table

Before reloading segments or force-committing on a partial-upsert table, set the following cluster config:
This setting takes effect immediately — no server restart is required. It is applied at the next segment commit. Never set this mode to UNSAFE.
PROTECTED mode is planned to become the default in an upcoming release.

​
Rebalancing a partial-upsert or dedup table

Pause ingestion before rebalancing, and resume it after the rebalance completes. Rebalancing while consuming can cause per-key state to diverge across the moved replicas.

​
Migrating or cloning a live upsert table

1

Pause the Segment Refresh Task on the source table

A refresh run that overlaps the copy can replace segments mid-migration, leaving the clone with missing rows.
2

Perform the migration or clone

Copy the table using your normal procedure.
3

Compare row counts between source and destination

Verify total row counts (and per-partition counts for large tables) match before resuming tasks and cutting traffic over.

​
Running SegmentRefreshTask alongside FileIngestionTask

If a table runs both tasks, exclude file-ingested segments from refresh runs by adding a segment selector to the SegmentRefreshTask config:
Without this filter the two tasks can clash on segment names and produce data issues.

​
Protections that are on by default

These behaviors are active without appearing in your table config:

​
In-order consumption enforcement (enforceConsumptionInOrder)

Since release 0.15, partial-upsert, dedup, and dropOutOfOrderRecord tables enforce in-order consumption automatically, which improves data correctness. You can confirm it is active by looking for ConsumerCoordinator entries in server logs.
In-order enforcement is not applied while the table runs SegmentRefreshTask, FileIngestionTask, or StarTreeAlterTableTask. A partial-upsert table running the Segment Refresh Task therefore has no in-order protection — this combination is not supported (see the matrix above).

​
Snapshot tasks

UpsertSnapshotCreationTask and DedupSnapshotCreationTask run automatically every 12 hours for upsert and dedup tables (since release 0.10). On releases before 0.15, any table-config update permanently cancelled this default schedule — if snapshots stopped after a config change on an older release, re-apply the schedule or upgrade.

​
Troubleshooting

​
Segment Refresh Task or compaction task keeps failing

Recent releases add safety guards that fail a task rather than risk data corruption. Mitigation:
  1. Set cluster config pinot.server.consuming.segment.consistency.mode: PROTECTED
  2. Set table config parallelSegmentConsumptionPolicy: DISALLOW_ALWAYS
If the task still fails after both settings, contact StarTree support.

​
Wrong or flip-flopping query results

Alternating results between runs of the same query usually means replicas have diverged.
Do not reload the table as a first step. Reload replaces each server’s local data with the deep-store copy — if the deep-store copy is the inconsistent one, the reload locks the bad data in. Contact StarTree support first so the inconsistent segments can be identified and preserved for analysis.

​
Stale derived column after a partial-upsert merge

If a derived column shows a stale value after partial-upsert merges, compute the value in the query (from the source columns) as a workaround while a fix is in progress.

​
Roadmap

An upcoming release converts the “not supported” items above into hard blocks at table-creation/config time, and makes PROTECTED consistency mode the default. Longer term, a simplified upsert data model is planned that reduces configuration surface and removes most of these exceptions.