> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Troubleshooting ingestion
> Diagnose real-time ingestion that has stopped or fallen behind, batch and minion tasks that failed, and tasks that were never scheduled.
This guide covers data that has stopped arriving, data arriving later than it should, and the tasks responsible for moving it.
## First: lag, stopped, or never started?
These three look similar from the outside and have different causes. Establish which one you have before going further.
On the Data Portal **Health Dashboard**, open the table and expand `TABLE_INGESTION_CHECK`. It reports:
| Field | What it tells you |
| ----------------------------- | ----------------------------------------------- |
| `partitionsNotConsuming` | How many stream partitions have stopped |
| `totalPartitions` | How many there should be |
| `consumingSegmentsCount` | How many segments are actively consuming |
| `maxIngestionDelayMs` | Worst-case lag between the source and the table |
| `maxEndToEndIngestionDelayMs` | Worst-case lag including the producer side |
Read it as:
* **`partitionsNotConsuming` is 0 and lag is high** → you are behind, but still moving. See [Ingestion is lagging](#ingestion-is-lagging).
* **`partitionsNotConsuming` is non-zero** → some or all consumption has stopped. See [Ingestion has stopped](#ingestion-has-stopped).
* **`partitionsNotConsuming` equals `totalPartitions`** → nothing is consuming at all. Check first whether consumption was paused deliberately.
* **`totalPartitions` does not match your source** → your stream topology changed. See [The source changed](#the-source-changed).
Query the table directly for the time range in question, filtering on the table's time column rather than relying on a dashboard that may cache. On a hybrid table, query the `_REALTIME` and `_OFFLINE` halves separately — data that looks missing is sometimes present on the other side of the time boundary. See [Hybrid tables](/corecapabilities/manage-data/hybrid-tables).
## Ingestion is lagging
Data is still arriving, just late. The question is whether the gap is growing or recovering.
Check `maxIngestionDelayMs` several times a few minutes apart.
* **Growing** — consumption is slower than production. This is a capacity or throughput problem.
* **Flat and high** — you fell behind once and are keeping pace without catching up. Same causes, less urgent.
* **Shrinking** — you are recovering. Confirm the rate is fast enough to matter, then find what caused the original gap.
A lag that starts without any change on the StarTree side is usually a change on the producer side — a traffic peak, a backfill, or a replay. Compare your current message rate against a normal day.
Lag during a known peak with recovery afterwards may not need any action beyond planning for the next one.
If production is unchanged, the consuming side has slowed. Look for memory or CPU pressure on the servers hosting the table — see [Resource pressure](/corecapabilities/cluster-operations/troubleshooting). Servers under pressure consume more slowly before they fail outright, so growing lag is often the first visible symptom of a capacity problem rather than a problem in its own right.
If some partitions are far behind and others are current, the problem is skew rather than capacity — the partitioning key is concentrating traffic. `SEGMENT_SKEW_HEALTH_CHECK` on the Health Dashboard surfaces the segment-level version of this.
## Ingestion has stopped
One or more partitions are no longer consuming.
Check first whether consumption was paused — deliberately, or by an operation that pauses it as a side effect. A paused table looks identical to a broken one on most surfaces. See [Pause and resume consumption](/recipes/pause-resume-consumption).
Once you have ruled out a pause, work through these:
1. **Did the upstream schema change?** A producer that starts emitting a new schema version can stop consumption cleanly, with no error on the StarTree side beyond the halt itself. This is one of the most common causes and one of the least obvious. Check when your producer last deployed against when consumption stopped.
2. **Is the problem in decoding rather than consuming?** If messages are arriving but failing to parse, the decoder is at fault rather than the stream. Each decoder documents its own failure modes — see [Troubleshooting by feature](/corecapabilities/observability/troubleshooting-by-feature#streaming-ingestion).
3. **Did credentials or permissions change?** Rotated keys, an expired token, or a changed IAM role stop consumption at the next fetch. If the stop is abrupt and total, this is a strong candidate.
4. **Is the source reachable and does the offset still exist?** If retention on the source expired past the offset the table was reading, consumption cannot resume from where it left off. This is more likely the longer a table has been stopped, and it changes the remedy — you are choosing where to restart, not resuming.
5. **Did the partition count change?** See [The source changed](#the-source-changed).
6. **Are segments stuck in a bad state?** Check `TABLE_SEGMENT_AVAILABILITY_CHECK` and `IDEAL_STATE_EV_MISMATCH_CHECK`. A consuming segment that cannot commit will block the partition behind it. [Force commit](/recipes/force-commit) is sometimes the way forward, but understand why the segment is stuck before forcing it.
For tables using [pauseless consumption](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption), the commit path differs and consumption is not interrupted by segment commit. That page's [Operational Troubleshooting](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#operational-troubleshooting) section covers ordering enforcement and stalled transitions, and its [Debugging APIs](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#debugging-apis) section covers inspecting consuming-segment state directly.
## The source changed
Changes to your stream's topology are not always visible from the table until they cause a symptom.
* **Partition count increased.** New partitions need to be picked up. Until they are, you are silently missing whatever routes to them — the table looks healthy and the data is incomplete, which is worse than an outright failure. Compare `totalPartitions` on the ingestion check against your source.
* **Partition count decreased, or brokers were removed.** Consumption may continue against partitions that no longer exist, or fail in ways that look like connectivity problems.
* **The topic was recreated with the same name.** Offsets no longer mean what the table thinks they mean.
* **Schema version changed.** See [Ingestion has stopped](#ingestion-has-stopped), item 1.
A change to partition count on a partitioned table also affects query pruning, so the symptom may show up first as slower queries rather than as an ingestion problem.
## Batch and minion task failures
Tasks that move data in batches — file ingestion, segment refresh, purge, merge, external table sync — fail differently from streaming ingestion.
In the Data Portal, open **Tasks** and locate the task plan. See [Task Observability](/corecapabilities/observability/task-observability) for how to read plan, task and subtask state, and for the progress logs each subtask emits.
These are different problems with the same symptom — no new data.
A task that **ran and failed** has a failure message and a subtask to inspect.
A task that was **never generated** has nothing to inspect, because nothing ran. The usual cause is that the generator evaluated its inputs and matched nothing: a path or prefix that resolves to no files, a source with no new data since the last run, or a filter that excluded everything. Verify independently that the files or rows you expect actually exist at the location the task is configured to read.
Task failures often surface a generic outer message with the real cause nested inside. Storage access problems in particular — wrong region, missing permission, moved object — produce inner exceptions that name the problem precisely. Always expand to the innermost cause before concluding anything.
Minions that run out of memory take their tasks down with them, and a minion that restarts repeatedly produces a pattern of unrelated tasks failing together. If several different task types started failing at the same time, suspect the minion rather than the tasks. See [Resource pressure](/corecapabilities/cluster-operations/troubleshooting).
A task queue that has stopped progressing holds up everything behind it. If tasks are pending rather than failing, and the oldest pending task is not moving, that is a queue problem rather than a task problem.
## Data arrived but looks wrong
If rows are present but the values or counts are not what you expect:
* **Compare row counts against the source** for a bounded time window, rather than comparing totals. A discrepancy in one window is far easier to trace than a discrepancy in the whole table.
* **On hybrid tables, check the time boundary.** Rows can appear missing or duplicated depending on where the boundary sits relative to the data. See [Hybrid tables](/corecapabilities/manage-data/hybrid-tables).
* **On upsert tables, check the primary key and comparison column.** Unexpected values usually mean records are being resolved against a different previous record than you assume. See the [Upsert operations guide](/corecapabilities/manage-data/upsert-operations-guide).
* **For duplicates or counts running ahead of the source**, [Diagnosing duplicates and count mismatches](/corecapabilities/manage-data/offheap-upsert#diagnosing-duplicates-and-count-mismatches) gives the concrete techniques: querying with `skipUpsert` to see un-masked rows, projecting the `$segmentName` / `$docId` / `$hostName` virtual columns to locate a duplicate, and comparing strict-replica-group results to detect cross-replica divergence.
* **Check whether retention removed the data.** `SEGMENT_RETENTION_CHECK` on the Health Dashboard shows retention state. Data older than your configured retention is supposed to disappear.
* **Check transformations.** A transformation or filter function that silently drops records will show up as a row-count shortfall with no error anywhere. If counts are short and nothing failed, review the ingestion transform config against a sample of the records you expect to be present.
## Feature-specific guidance
For task types with their own documented failure modes — purge, backfill, the real-time to offline flow, minion orchestration — see [Troubleshooting by feature](/corecapabilities/observability/troubleshooting-by-feature#batch-ingestion-and-minion-tasks).
**External table sync?** [External table troubleshooting](/corecapabilities/external-table/troubleshooting), *Sync & operations* section, covers a sync that never starts, a run failing on a single unreadable file, distributed-lock contention on `ExternalTableSyncTask`, and segments failing to load after upstream compaction.
## Escalating
Include: how far behind, whether the gap is growing, which partitions are affected, what changed upstream and when, and for task failures the task type, plan or task ID, and the full nested error. See [what to collect](/corecapabilities/observability/troubleshooting#before-you-open-a-support-ticket).