Real-time ingestion alerts

Freshness and completeness of data coming from Kafka, Kinesis and other streams. These page first because stale data is usually what users notice.

Critical alerts page StarTree on-call 24/7. Warning alerts reach StarTree’s alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the overview for how to read an entry.

Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

Critical alerts

`RealtimeIngestionStopped`

Often transient: this alert usually clears on its own.

Field	Detail
Fires when	No replica has consumed a partition for 30 min, and the table isn’t disabled or paused.
What it means	Data in that partition stops arriving.
Check first	• Pinot Server Ingestion → Realtime Consumption Status by Table Partition • `GET /tables/{table}/consumingSegmentsInfo` • `GET /tables/{table}/pauseStatus`
What you can do	• Confirm the topic still exists and credentials are valid • If paused by a resource limit, see ResourceUtilizationLimitExceeded
What StarTree does	• Resets the consuming segment • Restarts a stuck server
See also	Pauseless consumption: debugging APIs

`HighRealtimeIngestionOffsetLag`

Field	Detail
Fires when	Average lag is over 500,000 messages and still growing for 60 min.
What it means	Pinot is consuming but falling behind, so queries see older data.
Check first	• Pinot Server Ingestion → Is Ingestion Throttled? • Ingestion Quota Utilization (100% means throttled) • Realtime Rows Consumed PerSecondRate vs your producer rate
What you can do	• Raise `topic.consumption.rate.limit` if throttled • Fix a skewed partition key • Add partitions only on non-upsert tables; changing the partition count of a live upsert table isn’t supported
What StarTree does	• Scales servers • Tunes the server consumption limit
See also	Pauseless consumption: operational troubleshooting Upsert operations guide: supported configurations

`NoRealtimeIngestionPerServer`

Often transient: this alert usually clears on its own.

Field	Detail
Fires when	On one server, a partition’s delay grew 500 s+ in 10 min with no new offsets consumed.
What it means	One replica has stalled, so results can differ between replicas.
Check first	• Pinot Server Ingestion → Last Realtime Segment Creation Duration Seconds • Realtime Consumption Status by Table Partition
What you can do	Consider pauseless consumption for tables with long segment commits. It isn’t supported with partial or full upsert
What StarTree does	• Restarts the consumer if it’s stuck • Clears stale metrics
See also	Pauseless consumption: operational troubleshooting Upsert operations guide: supported configurations

`PartitionConsumerCreateExceptions`

Usually your fix: the cause is most often in your stream, source system, or table config.

Field	Detail
Fires when	Any failure creating a stream consumer in the last 5 min.
What it means	A server can’t connect to your stream.
Check first	• One server: a host problem. All servers on one topic: the topic. All topics: credentials or network • Logging / Application Logs, filtered to the server
What you can do	Check credentials, TLS certificates, ACLs and that the topic exists
What StarTree does	Force-commits or resets the partition once the cause is fixed

`RealtimeConsumptionExceptions`

Field	Detail
Fires when	A server keeps hitting consumption exceptions for 30 min.
What it means	Fetches from the stream fail. After repeated errors the consumer stops.
Check first	• Pinot Server Ingestion → Realtime Consumption Exceptions PerSecondRate • Server logs for the exception class
What you can do	• Disconnects or timeouts: check stream broker health • Offset out of range: retention is too short • Schema registry errors: fix the schema
What StarTree does	Restarts consumption after the cause clears

`RealtimeIngestionHighRowsWithErrorsRate`

Usually your fix: the cause is most often in your stream, source system, or table config.

Field	Detail
Fires when	Over 50% of fetched rows fail to decode or transform, at more than 0.1 rows/s, for 30 min.
What it means	Rows are being dropped.
Check first	Pinot Server Ingestion → Rows filtered due to decode exception and transform exception
What you can do	Fix the producer’s message format or the table’s transform config
What StarTree does	Confirms which column or decoder fails
See also	Decoder troubleshooting

`StreamDataLoss`

Usually your fix: the cause is most often in your stream, source system, or table config.

Field	Detail
Fires when	A server finds a gap between the offset it expected and the first one the stream still has. Fires at once.
What it means	Some messages were never ingested. The alert clearing doesn’t mean the data came back.
Check first	• Gap of 1 with no lag: a transactional topic, not real loss • Large gap after a restart: stream retention ran out • Large gap with high lag: ingestion was too slow
What you can do	• Set `stream.kafka.isolation.level` to `read_committed` for transactional topics • Keep stream retention at least 2× the segment flush time
What StarTree does	Confirms the gap and its cause

Warning alerts

Alert	Fires when	What it means	What you can do
`RealtimeIngestionHighSkippedRowsRate`	Over 90% of rows are filtered out for 15 min.	Usually a `filterConfig` that drops more than intended.	Check the table’s filterConfig
`RealtimePartitionMismatch`	Rows land in a partition that doesn’t match the table’s partition config, for 30 min.	Partition pruning can return wrong results.	Match `functionName` and `numPartitions` to the producer, or stop pruning on that column

Metrics

The series below are available in Grafana through the Prometheus data source. For how names, suffixes, and labels are built, see Reading Pinot metrics.

Metrics worth watching

These are the ones to reach for first; they are not the complete set.

Metric	What it tells you
`pinot_server_llcPartitionConsuming_Value`	`1` if the partition is consuming, `0` if it is not. The single most direct answer to “is ingestion running?”
`pinot_server_realtimeIngestionDelayMs_Value`	Age of the newest ingested record, per partition. Rising steadily means falling behind.
`pinot_server_realtimeIngestionOffsetLag_Value`	Offset distance behind the stream head. Compare its direction with the delay: a large but shrinking lag is a recovery.
`pinot_server_highestStreamOffsetConsumed_Value`	The last offset consumed. Flat while delay grows is the signature of a stuck consumer.
`pinot_server_realtime_consumptionExceptions_Count`	Consumption threw. The stream is reachable but records are failing.
`pinot_server_rowsWithErrors_Count`	Rows rejected during decode or transform. Almost always a schema mismatch.
`pinot_server_streamConsumerCreateExceptions_Count`	Could not create the consumer at all — credentials, topic, or network path.
`pinot_server_realtimePartitionMismatch_Count`	Pinot’s partition count no longer matches the stream’s.
`pinot_controller_tableConsumptionPaused_Value`	`1` when consumption is paused. Check this before diagnosing “stopped” ingestion.
`pinot_controller_tableDisabled_Value`	`1` when the table is disabled.

Metrics behind each alert

The Prometheus series each alert rule evaluates.

Alert	Metrics in the rule
`RealtimeIngestionStopped`	`pinot_server_llcPartitionConsuming_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value`
`HighRealtimeIngestionOffsetLag`	`pinot_server_realtimeIngestionOffsetLag_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value`
`NoRealtimeIngestionPerServer`	`pinot_server_realtimeIngestionDelayMs_Value`, `pinot_server_highestStreamOffsetConsumed_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value`
`PartitionConsumerCreateExceptions`	`pinot_server_streamConsumerCreateExceptions_Count`
`RealtimeConsumptionExceptions`	`pinot_server_realtime_consumptionExceptions_Count`
`RealtimeIngestionHighRowsWithErrorsRate`	`pinot_server_rowsWithErrors_Count`, `pinot_server_realtimeRowsFetched_Count`
`StreamDataLoss`	`pinot_server_streamDataLoss_Value`, `pinot_server_documentCount_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value`
`RealtimeIngestionHighSkippedRowsRate`	`pinot_server_realtimeRowsFiltered_Count`, `pinot_server_realtimeRowsConsumed_Count`
`RealtimePartitionMismatch`	`pinot_server_realtimePartitionMismatch_Count`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value`

Alerts and metrics overview: severity, the symptom router, the alert index, and how metric names are built.
Troubleshooting by feature: Streaming ingestion
Ingestion troubleshooting
Pauseless consumption

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Real-time ingestion alerts

Decision tree

Critical alerts

`RealtimeIngestionStopped`

`HighRealtimeIngestionOffsetLag`

`NoRealtimeIngestionPerServer`

`PartitionConsumerCreateExceptions`

`RealtimeConsumptionExceptions`

`RealtimeIngestionHighRowsWithErrorsRate`

`StreamDataLoss`

Warning alerts

Metrics

Metrics worth watching

Metrics behind each alert

​Decision tree

​Critical alerts

​RealtimeIngestionStopped

​HighRealtimeIngestionOffsetLag

​NoRealtimeIngestionPerServer

​PartitionConsumerCreateExceptions

​RealtimeConsumptionExceptions

​RealtimeIngestionHighRowsWithErrorsRate

​StreamDataLoss

​Warning alerts

​Metrics

​Metrics worth watching

​Metrics behind each alert

​Related

Decision tree

Critical alerts

`RealtimeIngestionStopped`

`HighRealtimeIngestionOffsetLag`

`NoRealtimeIngestionPerServer`

`PartitionConsumerCreateExceptions`

`RealtimeConsumptionExceptions`

`RealtimeIngestionHighRowsWithErrorsRate`

`StreamDataLoss`

Warning alerts

Metrics

Metrics worth watching

Metrics behind each alert

Related