Batch ingestion and minion task alerts

File, SQL, segment import and external table sync tasks, and the minion workers that run them.

Critical alerts page StarTree on-call 24/7. Warning alerts reach StarTree’s alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the overview for how to read an entry.

Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

Critical alerts

`FailedToGenerateFileIngestionTask`

Usually your fix: the cause is most often in your stream, source system, or table config.

Field	Detail
Fires when	File ingestion task generation failed in the last 30 min.
What it means	No new files are loaded for that table.
Check first	Task status message in the Data Portal
What you can do	Fix credentials, input path or non-ASCII file names

`FailedToGenerateSqlIngestionTask`

Usually your fix: the cause is most often in your stream, source system, or table config.

Field	Detail
Fires when	SQL ingestion task generation failed in the last 30 min.
What it means	No new rows are pulled from the source database.
Check first	Task status message in the Data Portal
What you can do	Fix source credentials or the query

`FailedToGenerateSegmentImportTask`

Field	Detail
Fires when	Segment import task generation failed in the last 30 min.
What it means	Real-time data isn’t moved to the offline table.
What you can do	Nothing. StarTree handles this one.
What StarTree does	Fixes the task configuration

`FailedToGenerateExternalTableSyncTask`

Usually your fix: the cause is most often in your stream, source system, or table config.

Field	Detail
Fires when	External table sync generation failed in the last 30 min.
What it means	The table stops picking up new snapshots.
Check first	External table status in the Data Portal
What you can do	Fix catalog credentials or snapshot retention
See also	External table observability: metrics and alerting External table troubleshooting

`MinionTaskGenerationFailures`

Field	Detail
Fires when	Generation failed in the last hour and hasn’t succeeded for 6 hours.
What it means	Scheduled work is stuck.
Check first	Pinot Minion Tasks → Time Since Last Successful Task Generation
What you can do	Fix the config or input named in the error
What StarTree does	Clears locks or resource limits

`MinionTaskGenerationSkipped`

Field	Detail
Fires when	Generation was skipped because a previous run is still active.
What it means	Runs are overlapping.
Check first	Pinot Minion Tasks → Cron Scheduler Job Skipped Count
What you can do	Space out the task schedule
See also	Minion task orchestration: FAQs

`MinionSubtasksInErrorForTableAndTaskType`

Field	Detail
Fires when	Over 50% of a table’s subtasks are in ERROR, for 20 min.
What it means	That table’s task is failing.
Check first	`GET /tasks/{taskType}/{tableNameWithType}/debug`, with the typed table name such as `orders_OFFLINE`
What you can do	Fix source or config errors in the subtask message
What StarTree does	Fixes minion-side failures

`FileIngestionSubtasksInErrorForTable`

Field	Detail
Fires when	Over 90% of a table’s file ingestion subtasks fail, for 20 min.
What it means	Files aren’t loading.
What you can do	Check the files and credentials

`FileIngestionSubtasksInError`

Field	Detail
Fires when	Over 90% of file ingestion subtasks fail cluster-wide, for 60 min.
What it means	File ingestion is failing broadly.
What you can do	Nothing. StarTree handles this one.
What StarTree does	Investigates a shared cause

`SqlIngestionSubtasksInError`

Field	Detail
Fires when	Over 90% of SQL ingestion subtasks fail, for 60 min.
What it means	SQL ingestion is failing broadly.
What you can do	Nothing. StarTree handles this one.
What StarTree does	Investigates a shared cause

`SegmentImportSubtasksInError`

Field	Detail
Fires when	Over 90% of segment import subtasks fail, for 60 min.
What it means	Segment import is failing broadly.
What you can do	Nothing. StarTree handles this one.
What StarTree does	Investigates a shared cause

`MinionSubtasksDroppedForTable`

Field	Detail
Fires when	More than 10 subtasks dropped, for 60 min.
What it means	A minion went away mid-task; dropped work isn’t retried.
What you can do	Re-run the task
What StarTree does	Checks minion stability

`MinionLongWaitingSubtasksForTaskType`

Field	Detail
Fires when	Waiting subtasks haven’t decreased in 20 min and minions aren’t picking any up, for 60 min.
What it means	Task execution has stalled.
Check first	Pinot Minion Tasks → Total Number of SubTasks Waiting
What you can do	Nothing. StarTree handles this one.
What StarTree does	Restarts or scales minions
See also	Minion task orchestration: FAQs

`MinionSubTaskHighWaitTime`

Often transient: this alert usually clears on its own.

Field	Detail
Fires when	A subtask has waited over 4 hours.
What it means	Not enough minion capacity, or schedules collide.
Check first	Pinot Minion Tasks → Sub Task Queueing Time P95
What you can do	Stagger task schedules
What StarTree does	Scales minions

`MinionSubTaskHighRunningTime`

Often transient: this alert usually clears on its own.

Field	Detail
Fires when	A subtask has run over 3 hours.
What it means	Usually very large inputs.
Check first	Pinot Minion Tasks → Sub Task Execution Time P95
What you can do	Split large inputs

`MinionConsistentPushFailures`

Field	Detail
Fires when	A consistent push failed in the last hour.
What it means	Usually the table’s storage quota rejected the upload.
What you can do	Raise `quota.storage`

`MinionsUnderUtilized`

Field	Detail
Fires when	Fewer than 20% of minions run tasks, for 12 hours.
What it means	Autoscaling may be stuck high.
What you can do	Nothing. StarTree handles this one.
What StarTree does	Fixes autoscaling

`OfflineSegmentDelayHoursCrossedThreshold`

Field	Detail
Fires when	The offline table is 7+ days behind while segment import is active.
What it means	Offline data is stale.
What you can do	Run segment import more often
See also	Managed offline flow: troubleshooting

Warning alerts

Alert	Fires when	What it means	What you can do
`MinionSubtaskQueueSizeHigh`	More than 20 waiting subtasks per running minion, for 6 hours.	A backlog is building.	Nothing needed
`PossibleDataLossDuringSegmentImport`	Real-time segments hold data older than the import buffer.	Late events may be skipped by import.	Raise `bufferTimePeriod`

Metrics

The series below are available in Grafana through the Prometheus data source. For how names, suffixes, and labels are built, see Reading Pinot metrics.

Metrics worth watching

These are the ones to reach for first; they are not the complete set.

Metric	What it tells you
`pinot_controller_percentMinionSubtasksInError_Value`	Share of a task type’s subtasks in error, per table.
`pinot_controller_numMinionSubtasksWaiting_Value`	Queue depth, per task type.
`pinot_controller_numMinionSubtasksDropped_Value`	Subtasks dropped rather than run.
`pinot_controller_maxSubtaskWaitTimeMs_Value`, `pinot_controller_maxSubtaskRunningTimeMs_Value`	Worst-case wait and run time, per table and task type.
`pinot_controller_taskGenerationFailureCount_Count`	Generation failed — no subtasks were even created.
`pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value`	How long since anything generated successfully. Pair it with the failure count: failures matter far more when this is also large.
`pinot_controller_taskGenerationSkippedDueToConflict_Count`	Generation skipped because a conflicting task was in flight.
`pinot_controller_consistentPushFailure_Count`	Consistent-push failures.
`pinot_controller_sqlIngestionFailure_Count`, `pinot_controller_fileIngestionFailure_Count`, `pinot_controller_segmentImportFailure_Count`	Per-ingestion-type generation failures.
`pinot_minion_taskQueueing_Count`	Minions actually dequeuing work. Zero while the queue is deep means no minion is picking anything up.
`pinot_controller_offlineSegmentDelayHours_Value`	Age of the newest offline segment — how far behind the real-time-to-offline job is.

Metrics behind each alert

The Prometheus series each alert rule evaluates.

Alert	Metrics in the rule
`FailedToGenerateFileIngestionTask`	`pinot_controller_fileIngestionFailure_Count`
`FailedToGenerateSqlIngestionTask`	`pinot_controller_sqlIngestionFailure_Count`
`FailedToGenerateSegmentImportTask`	`pinot_controller_segmentImportFailure_Count`
`FailedToGenerateExternalTableSyncTask`	`pinot_controller_externalTableSyncTaskGenerationFailure_Count`
`MinionTaskGenerationFailures`	`pinot_controller_taskGenerationFailureCount_Count`, `pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value`
`MinionTaskGenerationSkipped`	`pinot_controller_taskGenerationSkippedDueToConflict_Count`, `pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value`
`MinionSubtasksInErrorForTableAndTaskType`	`pinot_controller_percentMinionSubtasksInError_Value`
`FileIngestionSubtasksInErrorForTable`	`pinot_controller_percentMinionSubtasksInError_Value`
`FileIngestionSubtasksInError`	`pinot_controller_percentMinionSubtasksInError_Value`
`SqlIngestionSubtasksInError`	`pinot_controller_percentMinionSubtasksInError_Value`
`SegmentImportSubtasksInError`	`pinot_controller_percentMinionSubtasksInError_Value`
`MinionSubtasksDroppedForTable`	`pinot_controller_numMinionSubtasksDropped_Value`
`MinionLongWaitingSubtasksForTaskType`	`pinot_controller_numMinionSubtasksWaiting_Value`, `pinot_minion_taskQueueing_Count`
`MinionSubTaskHighWaitTime`	`pinot_controller_maxSubtaskWaitTimeMs_Value`
`MinionSubTaskHighRunningTime`	`pinot_controller_maxSubtaskRunningTimeMs_Value`
`MinionConsistentPushFailures`	`pinot_controller_consistentPushFailure_Count`
`MinionsUnderUtilized`	`pinot_minion_numberOfTasks_Value`
`OfflineSegmentDelayHoursCrossedThreshold`	`pinot_controller_offlineSegmentDelayHours_Value`, `pinot_controller_scheduledSegments_Count`
`MinionSubtaskQueueSizeHigh`	`pinot_controller_numMinionSubtasksWaiting_Value`, `kube_pod_status_phase`
`PossibleDataLossDuringSegmentImport`	`pinot_controller_segmentsWithLateData_Value`

Alerts and metrics overview: severity, the symptom router, the alert index, and how metric names are built.
Troubleshooting by feature: Batch ingestion and minion tasks
Minion task runs
Ingestion troubleshooting

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Batch ingestion and minion task alerts

Decision tree

Critical alerts

`FailedToGenerateFileIngestionTask`

`FailedToGenerateSqlIngestionTask`

`FailedToGenerateSegmentImportTask`

`FailedToGenerateExternalTableSyncTask`

`MinionTaskGenerationFailures`

`MinionTaskGenerationSkipped`

`MinionSubtasksInErrorForTableAndTaskType`

`FileIngestionSubtasksInErrorForTable`

`FileIngestionSubtasksInError`

`SqlIngestionSubtasksInError`

`SegmentImportSubtasksInError`

`MinionSubtasksDroppedForTable`

`MinionLongWaitingSubtasksForTaskType`

`MinionSubTaskHighWaitTime`

`MinionSubTaskHighRunningTime`

`MinionConsistentPushFailures`

`MinionsUnderUtilized`

`OfflineSegmentDelayHoursCrossedThreshold`

Warning alerts

Metrics

Metrics worth watching

Metrics behind each alert

​Decision tree

​Critical alerts

​FailedToGenerateFileIngestionTask

​FailedToGenerateSqlIngestionTask

​FailedToGenerateSegmentImportTask

​FailedToGenerateExternalTableSyncTask

​MinionTaskGenerationFailures

​MinionTaskGenerationSkipped

​MinionSubtasksInErrorForTableAndTaskType

​FileIngestionSubtasksInErrorForTable

​FileIngestionSubtasksInError

​SqlIngestionSubtasksInError

​SegmentImportSubtasksInError

​MinionSubtasksDroppedForTable

​MinionLongWaitingSubtasksForTaskType

​MinionSubTaskHighWaitTime

​MinionSubTaskHighRunningTime

​MinionConsistentPushFailures

​MinionsUnderUtilized

​OfflineSegmentDelayHoursCrossedThreshold

​Warning alerts

​Metrics

​Metrics worth watching

​Metrics behind each alert

​Related

Decision tree

Critical alerts

`FailedToGenerateFileIngestionTask`

`FailedToGenerateSqlIngestionTask`

`FailedToGenerateSegmentImportTask`

`FailedToGenerateExternalTableSyncTask`

`MinionTaskGenerationFailures`

`MinionTaskGenerationSkipped`

`MinionSubtasksInErrorForTableAndTaskType`

`FileIngestionSubtasksInErrorForTable`

`FileIngestionSubtasksInError`

`SqlIngestionSubtasksInError`

`SegmentImportSubtasksInError`

`MinionSubtasksDroppedForTable`

`MinionLongWaitingSubtasksForTaskType`

`MinionSubTaskHighWaitTime`

`MinionSubTaskHighRunningTime`

`MinionConsistentPushFailures`

`MinionsUnderUtilized`

`OfflineSegmentDelayHoursCrossedThreshold`

Warning alerts

Metrics

Metrics worth watching

Metrics behind each alert

Related