Skip to main content
This guide covers data that has stopped arriving, data arriving later than it should, and the tasks responsible for moving it.

​
First: lag, stopped, or never started?

These three look similar from the outside and have different causes. Establish which one you have before going further.
1

Check the ingestion health check

On the Data Portal Health Dashboard, open the table and expand TABLE_INGESTION_CHECK. It reports:Read it as:
  • partitionsNotConsuming is 0 and lag is high → you are behind, but still moving. See Ingestion is lagging.
  • partitionsNotConsuming is non-zero → some or all consumption has stopped. See Ingestion has stopped.
  • partitionsNotConsuming equals totalPartitions → nothing is consuming at all. Check first whether consumption was paused deliberately.
  • totalPartitions does not match your source → your stream topology changed. See The source changed.
2

Confirm the data really is missing

Query the table directly for the time range in question, filtering on the table’s time column rather than relying on a dashboard that may cache. On a hybrid table, query the _REALTIME and _OFFLINE halves separately — data that looks missing is sometimes present on the other side of the time boundary. See Hybrid tables.

​
Ingestion is lagging

Data is still arriving, just late. The question is whether the gap is growing or recovering.
1

Determine the direction of travel

Check maxIngestionDelayMs several times a few minutes apart.
  • Growing — consumption is slower than production. This is a capacity or throughput problem.
  • Flat and high — you fell behind once and are keeping pace without catching up. Same causes, less urgent.
  • Shrinking — you are recovering. Confirm the rate is fast enough to matter, then find what caused the original gap.
2

Check whether production rate changed

A lag that starts without any change on the StarTree side is usually a change on the producer side — a traffic peak, a backfill, or a replay. Compare your current message rate against a normal day.Lag during a known peak with recovery afterwards may not need any action beyond planning for the next one.
3

Check whether the consumers have room

If production is unchanged, the consuming side has slowed. Look for memory or CPU pressure on the servers hosting the table — see Resource pressure. Servers under pressure consume more slowly before they fail outright, so growing lag is often the first visible symptom of a capacity problem rather than a problem in its own right.
4

Check partition balance

If some partitions are far behind and others are current, the problem is skew rather than capacity — the partitioning key is concentrating traffic. SEGMENT_SKEW_HEALTH_CHECK on the Health Dashboard surfaces the segment-level version of this.

​
Ingestion has stopped

One or more partitions are no longer consuming.
Check first whether consumption was paused — deliberately, or by an operation that pauses it as a side effect. A paused table looks identical to a broken one on most surfaces. See Pause and resume consumption.
Once you have ruled out a pause, work through these:
  1. Did the upstream schema change? A producer that starts emitting a new schema version can stop consumption cleanly, with no error on the StarTree side beyond the halt itself. This is one of the most common causes and one of the least obvious. Check when your producer last deployed against when consumption stopped.
  2. Is the problem in decoding rather than consuming? If messages are arriving but failing to parse, the decoder is at fault rather than the stream. Each decoder documents its own failure modes — see Troubleshooting by feature.
  3. Did credentials or permissions change? Rotated keys, an expired token, or a changed IAM role stop consumption at the next fetch. If the stop is abrupt and total, this is a strong candidate.
  4. Is the source reachable and does the offset still exist? If retention on the source expired past the offset the table was reading, consumption cannot resume from where it left off. This is more likely the longer a table has been stopped, and it changes the remedy — you are choosing where to restart, not resuming.
  5. Did the partition count change? See The source changed.
  6. Are segments stuck in a bad state? Check TABLE_SEGMENT_AVAILABILITY_CHECK and IDEAL_STATE_EV_MISMATCH_CHECK. A consuming segment that cannot commit will block the partition behind it. Force commit is sometimes the way forward, but understand why the segment is stuck before forcing it.
For tables using pauseless consumption, the commit path differs and consumption is not interrupted by segment commit. That page’s Operational Troubleshooting section covers ordering enforcement and stalled transitions, and its Debugging APIs section covers inspecting consuming-segment state directly.

​
The source changed

Changes to your stream’s topology are not always visible from the table until they cause a symptom.
  • Partition count increased. New partitions need to be picked up. Until they are, you are silently missing whatever routes to them — the table looks healthy and the data is incomplete, which is worse than an outright failure. Compare totalPartitions on the ingestion check against your source.
  • Partition count decreased, or brokers were removed. Consumption may continue against partitions that no longer exist, or fail in ways that look like connectivity problems.
  • The topic was recreated with the same name. Offsets no longer mean what the table thinks they mean.
  • Schema version changed. See Ingestion has stopped, item 1.
A change to partition count on a partitioned table also affects query pruning, so the symptom may show up first as slower queries rather than as an ingestion problem.

​
Batch and minion task failures

Tasks that move data in batches — file ingestion, segment refresh, purge, merge, external table sync — fail differently from streaming ingestion.
1

Find the task and its state

In the Data Portal, open Tasks and locate the task plan. See Task Observability for how to read plan, task and subtask state, and for the progress logs each subtask emits.
2

Distinguish 'failed' from 'never scheduled'

These are different problems with the same symptom — no new data.A task that ran and failed has a failure message and a subtask to inspect.A task that was never generated has nothing to inspect, because nothing ran. The usual cause is that the generator evaluated its inputs and matched nothing: a path or prefix that resolves to no files, a source with no new data since the last run, or a filter that excluded everything. Verify independently that the files or rows you expect actually exist at the location the task is configured to read.
3

Read the innermost error

Task failures often surface a generic outer message with the real cause nested inside. Storage access problems in particular — wrong region, missing permission, moved object — produce inner exceptions that name the problem precisely. Always expand to the innermost cause before concluding anything.
4

Check whether the failure is resource-related

Minions that run out of memory take their tasks down with them, and a minion that restarts repeatedly produces a pattern of unrelated tasks failing together. If several different task types started failing at the same time, suspect the minion rather than the tasks. See Resource pressure.
5

Check for a stuck queue

A task queue that has stopped progressing holds up everything behind it. If tasks are pending rather than failing, and the oldest pending task is not moving, that is a queue problem rather than a task problem.

​
Data arrived but looks wrong

If rows are present but the values or counts are not what you expect:
  • Compare row counts against the source for a bounded time window, rather than comparing totals. A discrepancy in one window is far easier to trace than a discrepancy in the whole table.
  • On hybrid tables, check the time boundary. Rows can appear missing or duplicated depending on where the boundary sits relative to the data. See Hybrid tables.
  • On upsert tables, check the primary key and comparison column. Unexpected values usually mean records are being resolved against a different previous record than you assume. See the Upsert operations guide.
  • For duplicates or counts running ahead of the source, Diagnosing duplicates and count mismatches gives the concrete techniques: querying with skipUpsert to see un-masked rows, projecting the $segmentName / $docId / $hostName virtual columns to locate a duplicate, and comparing strict-replica-group results to detect cross-replica divergence.
  • Check whether retention removed the data. SEGMENT_RETENTION_CHECK on the Health Dashboard shows retention state. Data older than your configured retention is supposed to disappear.
  • Check transformations. A transformation or filter function that silently drops records will show up as a row-count shortfall with no error anywhere. If counts are short and nothing failed, review the ingestion transform config against a sample of the records you expect to be present.

​
Feature-specific guidance

For task types with their own documented failure modes — purge, backfill, the real-time to offline flow, minion orchestration — see Troubleshooting by feature. External table sync? External table troubleshooting, Sync & operations section, covers a sync that never starts, a run failing on a single unreadable file, distributed-lock contention on ExternalTableSyncTask, and segments failing to load after upstream compaction.

​
Escalating

Include: how far behind, whether the gap is growing, which partitions are affected, what changed upstream and when, and for task failures the task type, plan or task ID, and the full nested error. See what to collect.