> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Real-time ingestion alerts

> Alerts that watch real-time ingestion from Kafka, Kinesis, and other streams: when each fires, what it means, and what you or StarTree can do.

Freshness and completeness of data coming from Kafka, Kinesis and other streams. These page first because stale data is usually what users notice.

<Info>
  **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the [overview](/corecapabilities/observability/alerts/overview) for how to read an entry.
</Info>

## Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

```mermaid placement="top-right" actions={true} theme={null}
flowchart TD
  A["New data is late or missing"] --> B{"GET /tables/{t}/pauseStatus<br/>Is consumption paused?"}
  B -- "Paused by a resource limit" --> B1["ResourceUtilizationLimitExceeded<br/>free disk or reduce keys/segments"]
  B -- "Paused by you" --> B2["Resume consumption"]
  B -- "Not paused" --> C{"Is any replica consuming<br/>the partition?"}
  C -- "No" --> D{"Consumer creation errors?"}
  D -- "Yes" --> D1["PartitionConsumerCreateExceptions<br/>credentials, TLS, ACLs, topic exists"]
  D -- "No" --> D2{"Server restarting<br/>or out of memory?"}
  D2 -- "Yes" --> D3["See Platform health"]
  D2 -- "No" --> D4["RealtimeIngestionStopped<br/>consuming segment in ERROR: reset it"]
  C -- "Yes" --> E{"Rows arriving<br/>but dropped?"}
  E -- "Decode or transform errors" --> E1["RealtimeIngestionHighRowsWithErrorsRate"]
  E -- "Filtered out" --> E2["RealtimeIngestionHighSkippedRowsRate"]
  E -- "No" --> F{"Is the backlog growing?"}
  F -- "Yes, throttled" --> F1["Raise topic.consumption.rate.limit"]
  F -- "Yes, not throttled" --> F2["HighRealtimeIngestionOffsetLag<br/>producer burst, skew, CPU"]
  F -- "One server only" --> F3["NoRealtimeIngestionPerServer<br/>often a segment commit"]
  F -- "No backlog" --> F4["Producer is sending old timestamps<br/>not a Pinot delay"]
  A --> G{"A gap, not a delay?"}
  G -- "Yes" --> G1["StreamDataLoss<br/>check stream retention"]
```

## Critical alerts

### `RealtimeIngestionStopped`

<Note>Often transient: this alert usually clears on its own.</Note>

| Field | Detail |
| - | - |
| **Fires when** | No replica has consumed a partition for 30 min, and the table isn't disabled or paused. |
| **What it means** | Data in that partition stops arriving. |
| **Check first** | • Pinot Server Ingestion → **Realtime Consumption Status by Table Partition**<br />• `GET /tables/{table}/consumingSegmentsInfo`<br />• `GET /tables/{table}/pauseStatus` |
| **What you can do** | • Confirm the topic still exists and credentials are valid<br />• If paused by a resource limit, see ResourceUtilizationLimitExceeded |
| **What StarTree does** | • Resets the consuming segment<br />• Restarts a stuck server |
| **See also** | [Pauseless consumption: debugging APIs](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#debugging-apis) |

### `HighRealtimeIngestionOffsetLag`

| Field | Detail |
| - | - |
| **Fires when** | Average lag is over 500,000 messages and still growing for 60 min. |
| **What it means** | Pinot is consuming but falling behind, so queries see older data. |
| **Check first** | • Pinot Server Ingestion → **Is Ingestion Throttled?**<br />• **Ingestion Quota Utilization** (100% means throttled)<br />• **Realtime Rows Consumed PerSecondRate** vs your producer rate |
| **What you can do** | • Raise `topic.consumption.rate.limit` if throttled<br />• Fix a skewed partition key<br />• Add partitions only on non-upsert tables; changing the partition count of a live upsert table isn't supported |
| **What StarTree does** | • Scales servers<br />• Tunes the server consumption limit |
| **See also** | [Pauseless consumption: operational troubleshooting](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#operational-troubleshooting)<br />[Upsert operations guide: supported configurations](/corecapabilities/manage-data/upsert-operations-guide#supported-configurations-at-a-glance) |

### `NoRealtimeIngestionPerServer`

<Note>Often transient: this alert usually clears on its own.</Note>

| Field | Detail |
| - | - |
| **Fires when** | On one server, a partition's delay grew 500 s+ in 10 min with no new offsets consumed. |
| **What it means** | One replica has stalled, so results can differ between replicas. |
| **Check first** | • Pinot Server Ingestion → **Last Realtime Segment Creation Duration Seconds**<br />• **Realtime Consumption Status by Table Partition** |
| **What you can do** | Consider pauseless consumption for tables with long segment commits. It isn't supported with partial or full upsert |
| **What StarTree does** | • Restarts the consumer if it's stuck<br />• Clears stale metrics |
| **See also** | [Pauseless consumption: operational troubleshooting](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#operational-troubleshooting)<br />[Upsert operations guide: supported configurations](/corecapabilities/manage-data/upsert-operations-guide#supported-configurations-at-a-glance) |

### `PartitionConsumerCreateExceptions`

<Tip>Usually your fix: the cause is most often in your stream, source system, or table config.</Tip>

| Field | Detail |
| - | - |
| **Fires when** | Any failure creating a stream consumer in the last 5 min. |
| **What it means** | A server can't connect to your stream. |
| **Check first** | • One server: a host problem. All servers on one topic: the topic. All topics: credentials or network<br />• Logging / Application Logs, filtered to the server |
| **What you can do** | Check credentials, TLS certificates, ACLs and that the topic exists |
| **What StarTree does** | Force-commits or resets the partition once the cause is fixed |

### `RealtimeConsumptionExceptions`

| Field | Detail |
| - | - |
| **Fires when** | A server keeps hitting consumption exceptions for 30 min. |
| **What it means** | Fetches from the stream fail. After repeated errors the consumer stops. |
| **Check first** | • Pinot Server Ingestion → **Realtime Consumption Exceptions PerSecondRate**<br />• Server logs for the exception class |
| **What you can do** | • Disconnects or timeouts: check stream broker health<br />• Offset out of range: retention is too short<br />• Schema registry errors: fix the schema |
| **What StarTree does** | Restarts consumption after the cause clears |

### `RealtimeIngestionHighRowsWithErrorsRate`

<Tip>Usually your fix: the cause is most often in your stream, source system, or table config.</Tip>

| Field | Detail |
| - | - |
| **Fires when** | Over 50% of fetched rows fail to decode or transform, at more than 0.1 rows/s, for 30 min. |
| **What it means** | Rows are being dropped. |
| **Check first** | Pinot Server Ingestion → **Rows filtered due to decode exception** and **transform exception** |
| **What you can do** | Fix the producer's message format or the table's transform config |
| **What StarTree does** | Confirms which column or decoder fails |
| **See also** | [Decoder troubleshooting](/corecapabilities/observability/troubleshooting-by-feature#streaming-ingestion) |

### `StreamDataLoss`

<Tip>Usually your fix: the cause is most often in your stream, source system, or table config.</Tip>

| Field | Detail |
| - | - |
| **Fires when** | A server finds a gap between the offset it expected and the first one the stream still has. Fires at once. |
| **What it means** | Some messages were never ingested. The alert clearing doesn't mean the data came back. |
| **Check first** | • Gap of 1 with no lag: a transactional topic, not real loss<br />• Large gap after a restart: stream retention ran out<br />• Large gap with high lag: ingestion was too slow |
| **What you can do** | • Set `stream.kafka.isolation.level` to `read_committed` for transactional topics<br />• Keep stream retention at least 2× the segment flush time |
| **What StarTree does** | Confirms the gap and its cause |

## Warning alerts

| Alert | Fires when | What it means | What you can do |
| - | - | - | - |
| `RealtimeIngestionHighSkippedRowsRate` | Over 90% of rows are filtered out for 15 min. | Usually a `filterConfig` that drops more than intended. | Check the table's filterConfig |
| `RealtimePartitionMismatch` | Rows land in a partition that doesn't match the table's partition config, for 30 min. | Partition pruning can return wrong results. | Match `functionName` and `numPartitions` to the producer, or stop pruning on that column |

## Metrics

The series below are available in [Grafana](/corecapabilities/observability/grafana) through the Prometheus data source. For how names, suffixes, and labels are built, see [Reading Pinot metrics](/corecapabilities/observability/alerts/overview#reading-pinot-metrics).

### Metrics worth watching

These are the ones to reach for first; they are not the complete set.

| Metric | What it tells you |
| - | - |
| `pinot_server_llcPartitionConsuming_Value` | `1` if the partition is consuming, `0` if it is not. The single most direct answer to "is ingestion running?" |
| `pinot_server_realtimeIngestionDelayMs_Value` | Age of the newest ingested record, per partition. Rising steadily means falling behind. |
| `pinot_server_realtimeIngestionOffsetLag_Value` | Offset distance behind the stream head. Compare its *direction* with the delay: a large but shrinking lag is a recovery. |
| `pinot_server_highestStreamOffsetConsumed_Value` | The last offset consumed. Flat while delay grows is the signature of a stuck consumer. |
| `pinot_server_realtime_consumptionExceptions_Count` | Consumption threw. The stream is reachable but records are failing. |
| `pinot_server_rowsWithErrors_Count` | Rows rejected during decode or transform. Almost always a schema mismatch. |
| `pinot_server_streamConsumerCreateExceptions_Count` | Could not create the consumer at all — credentials, topic, or network path. |
| `pinot_server_realtimePartitionMismatch_Count` | Pinot's partition count no longer matches the stream's. |
| `pinot_controller_tableConsumptionPaused_Value` | `1` when consumption is paused. Check this before diagnosing "stopped" ingestion. |
| `pinot_controller_tableDisabled_Value` | `1` when the table is disabled. |

### Metrics behind each alert

The Prometheus series each alert rule evaluates.

| Alert | Metrics in the rule |
| - | - |
| `RealtimeIngestionStopped` | `pinot_server_llcPartitionConsuming_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |
| `HighRealtimeIngestionOffsetLag` | `pinot_server_realtimeIngestionOffsetLag_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |
| `NoRealtimeIngestionPerServer` | `pinot_server_realtimeIngestionDelayMs_Value`, `pinot_server_highestStreamOffsetConsumed_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |
| `PartitionConsumerCreateExceptions` | `pinot_server_streamConsumerCreateExceptions_Count` |
| `RealtimeConsumptionExceptions` | `pinot_server_realtime_consumptionExceptions_Count` |
| `RealtimeIngestionHighRowsWithErrorsRate` | `pinot_server_rowsWithErrors_Count`, `pinot_server_realtimeRowsFetched_Count` |
| `StreamDataLoss` | `pinot_server_streamDataLoss_Value`, `pinot_server_documentCount_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |
| `RealtimeIngestionHighSkippedRowsRate` | `pinot_server_realtimeRowsFiltered_Count`, `pinot_server_realtimeRowsConsumed_Count` |
| `RealtimePartitionMismatch` | `pinot_server_realtimePartitionMismatch_Count`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |

## Related

* [Alerts and metrics overview](/corecapabilities/observability/alerts/overview): severity, the symptom router, the alert index, and how metric names are built.
* [Troubleshooting by feature: Streaming ingestion](/corecapabilities/observability/troubleshooting-by-feature#streaming-ingestion)
* [Ingestion troubleshooting](/corecapabilities/ingestdata/troubleshooting)
* [Pauseless consumption](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.