> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Batch ingestion and minion task alerts

> Alerts that watch file, SQL, segment import, and external table sync tasks and the minions that run them: when each fires, what it means, and what to do.

File, SQL, segment import and external table sync tasks, and the minion workers that run them.

<Info>
  **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the [overview](/corecapabilities/observability/alerts/overview) for how to read an entry.
</Info>

## Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

```mermaid placement="top-right" actions={true} theme={null}
flowchart TD
  A["Batch data didn't land"] --> B{"Was a task generated?"}
  B -- "No, generation failed" --> B1["FailedToGenerate*Task<br/>MinionTaskGenerationFailures<br/>credentials, input path, file names"]
  B -- "No, skipped" --> B2["MinionTaskGenerationSkipped<br/>previous run still active: space out the schedule"]
  B -- "Yes" --> C{"GET /tasks/{type}/{table_TYPE}/debug<br/>Subtask states?"}
  C -- "ERROR" --> C1["*SubtasksInError<br/>read the subtask error"]
  C -- "Dropped" --> C2["MinionSubtasksDroppedForTable<br/>re-run the task"]
  C -- "Waiting for hours" --> C3["MinionSubTaskHighWaitTime<br/>minion capacity, overlapping schedules"]
  C -- "Running for hours" --> C4["MinionSubTaskHighRunningTime<br/>very large inputs"]
  C -- "Push rejected" --> C5["MinionConsistentPushFailures<br/>raise quota.storage"]
```

## Critical alerts

### `FailedToGenerateFileIngestionTask`

<Tip>Usually your fix: the cause is most often in your stream, source system, or table config.</Tip>

| Field | Detail |
| - | - |
| **Fires when** | File ingestion task generation failed in the last 30 min. |
| **What it means** | No new files are loaded for that table. |
| **Check first** | Task status message in the Data Portal |
| **What you can do** | Fix credentials, input path or non-ASCII file names |

### `FailedToGenerateSqlIngestionTask`

<Tip>Usually your fix: the cause is most often in your stream, source system, or table config.</Tip>

| Field | Detail |
| - | - |
| **Fires when** | SQL ingestion task generation failed in the last 30 min. |
| **What it means** | No new rows are pulled from the source database. |
| **Check first** | Task status message in the Data Portal |
| **What you can do** | Fix source credentials or the query |

### `FailedToGenerateSegmentImportTask`

| Field | Detail |
| - | - |
| **Fires when** | Segment import task generation failed in the last 30 min. |
| **What it means** | Real-time data isn't moved to the offline table. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Fixes the task configuration |

### `FailedToGenerateExternalTableSyncTask`

<Tip>Usually your fix: the cause is most often in your stream, source system, or table config.</Tip>

| Field | Detail |
| - | - |
| **Fires when** | External table sync generation failed in the last 30 min. |
| **What it means** | The table stops picking up new snapshots. |
| **Check first** | External table status in the Data Portal |
| **What you can do** | Fix catalog credentials or snapshot retention |
| **See also** | [External table observability: metrics and alerting](/corecapabilities/external-table/observability#metrics-and-alerting)<br />[External table troubleshooting](/corecapabilities/external-table/troubleshooting) |

### `MinionTaskGenerationFailures`

| Field | Detail |
| - | - |
| **Fires when** | Generation failed in the last hour and hasn't succeeded for 6 hours. |
| **What it means** | Scheduled work is stuck. |
| **Check first** | Pinot Minion Tasks → **Time Since Last Successful Task Generation** |
| **What you can do** | Fix the config or input named in the error |
| **What StarTree does** | Clears locks or resource limits |

### `MinionTaskGenerationSkipped`

| Field | Detail |
| - | - |
| **Fires when** | Generation was skipped because a previous run is still active. |
| **What it means** | Runs are overlapping. |
| **Check first** | Pinot Minion Tasks → **Cron Scheduler Job Skipped Count** |
| **What you can do** | Space out the task schedule |
| **See also** | [Minion task orchestration: FAQs](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration#faqs) |

### `MinionSubtasksInErrorForTableAndTaskType`

| Field | Detail |
| - | - |
| **Fires when** | Over 50% of a table's subtasks are in ERROR, for 20 min. |
| **What it means** | That table's task is failing. |
| **Check first** | `GET /tasks/{taskType}/{tableNameWithType}/debug`, with the typed table name such as `orders_OFFLINE` |
| **What you can do** | Fix source or config errors in the subtask message |
| **What StarTree does** | Fixes minion-side failures |

### `FileIngestionSubtasksInErrorForTable`

| Field | Detail |
| - | - |
| **Fires when** | Over 90% of a table's file ingestion subtasks fail, for 20 min. |
| **What it means** | Files aren't loading. |
| **What you can do** | Check the files and credentials |

### `FileIngestionSubtasksInError`

| Field | Detail |
| - | - |
| **Fires when** | Over 90% of file ingestion subtasks fail cluster-wide, for 60 min. |
| **What it means** | File ingestion is failing broadly. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Investigates a shared cause |

### `SqlIngestionSubtasksInError`

| Field | Detail |
| - | - |
| **Fires when** | Over 90% of SQL ingestion subtasks fail, for 60 min. |
| **What it means** | SQL ingestion is failing broadly. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Investigates a shared cause |

### `SegmentImportSubtasksInError`

| Field | Detail |
| - | - |
| **Fires when** | Over 90% of segment import subtasks fail, for 60 min. |
| **What it means** | Segment import is failing broadly. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Investigates a shared cause |

### `MinionSubtasksDroppedForTable`

| Field | Detail |
| - | - |
| **Fires when** | More than 10 subtasks dropped, for 60 min. |
| **What it means** | A minion went away mid-task; dropped work isn't retried. |
| **What you can do** | Re-run the task |
| **What StarTree does** | Checks minion stability |

### `MinionLongWaitingSubtasksForTaskType`

| Field | Detail |
| - | - |
| **Fires when** | Waiting subtasks haven't decreased in 20 min and minions aren't picking any up, for 60 min. |
| **What it means** | Task execution has stalled. |
| **Check first** | Pinot Minion Tasks → **Total Number of SubTasks Waiting** |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Restarts or scales minions |
| **See also** | [Minion task orchestration: FAQs](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration#faqs) |

### `MinionSubTaskHighWaitTime`

<Note>Often transient: this alert usually clears on its own.</Note>

| Field | Detail |
| - | - |
| **Fires when** | A subtask has waited over 4 hours. |
| **What it means** | Not enough minion capacity, or schedules collide. |
| **Check first** | Pinot Minion Tasks → **Sub Task Queueing Time P95** |
| **What you can do** | Stagger task schedules |
| **What StarTree does** | Scales minions |

### `MinionSubTaskHighRunningTime`

<Note>Often transient: this alert usually clears on its own.</Note>

| Field | Detail |
| - | - |
| **Fires when** | A subtask has run over 3 hours. |
| **What it means** | Usually very large inputs. |
| **Check first** | Pinot Minion Tasks → **Sub Task Execution Time P95** |
| **What you can do** | Split large inputs |

### `MinionConsistentPushFailures`

| Field | Detail |
| - | - |
| **Fires when** | A consistent push failed in the last hour. |
| **What it means** | Usually the table's storage quota rejected the upload. |
| **What you can do** | Raise `quota.storage` |

### `MinionsUnderUtilized`

| Field | Detail |
| - | - |
| **Fires when** | Fewer than 20% of minions run tasks, for 12 hours. |
| **What it means** | Autoscaling may be stuck high. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Fixes autoscaling |

### `OfflineSegmentDelayHoursCrossedThreshold`

| Field | Detail |
| - | - |
| **Fires when** | The offline table is 7+ days behind while segment import is active. |
| **What it means** | Offline data is stale. |
| **What you can do** | Run segment import more often |
| **See also** | [Managed offline flow: troubleshooting](/recipes/managed-offline-flow#troubleshooting) |

## Warning alerts

| Alert | Fires when | What it means | What you can do |
| - | - | - | - |
| `MinionSubtaskQueueSizeHigh` | More than 20 waiting subtasks per running minion, for 6 hours. | A backlog is building. | Nothing needed |
| `PossibleDataLossDuringSegmentImport` | Real-time segments hold data older than the import buffer. | Late events may be skipped by import. | Raise `bufferTimePeriod` |

## Metrics

The series below are available in [Grafana](/corecapabilities/observability/grafana) through the Prometheus data source. For how names, suffixes, and labels are built, see [Reading Pinot metrics](/corecapabilities/observability/alerts/overview#reading-pinot-metrics).

### Metrics worth watching

These are the ones to reach for first; they are not the complete set.

| Metric | What it tells you |
| - | - |
| `pinot_controller_percentMinionSubtasksInError_Value` | Share of a task type's subtasks in error, per table. |
| `pinot_controller_numMinionSubtasksWaiting_Value` | Queue depth, per task type. |
| `pinot_controller_numMinionSubtasksDropped_Value` | Subtasks dropped rather than run. |
| `pinot_controller_maxSubtaskWaitTimeMs_Value`, `pinot_controller_maxSubtaskRunningTimeMs_Value` | Worst-case wait and run time, per table and task type. |
| `pinot_controller_taskGenerationFailureCount_Count` | Generation failed — no subtasks were even created. |
| `pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value` | How long since anything generated successfully. Pair it with the failure count: failures matter far more when this is also large. |
| `pinot_controller_taskGenerationSkippedDueToConflict_Count` | Generation skipped because a conflicting task was in flight. |
| `pinot_controller_consistentPushFailure_Count` | Consistent-push failures. |
| `pinot_controller_sqlIngestionFailure_Count`, `pinot_controller_fileIngestionFailure_Count`, `pinot_controller_segmentImportFailure_Count` | Per-ingestion-type generation failures. |
| `pinot_minion_taskQueueing_Count` | Minions actually dequeuing work. Zero while the queue is deep means no minion is picking anything up. |
| `pinot_controller_offlineSegmentDelayHours_Value` | Age of the newest offline segment — how far behind the real-time-to-offline job is. |

### Metrics behind each alert

The Prometheus series each alert rule evaluates.

| Alert | Metrics in the rule |
| - | - |
| `FailedToGenerateFileIngestionTask` | `pinot_controller_fileIngestionFailure_Count` |
| `FailedToGenerateSqlIngestionTask` | `pinot_controller_sqlIngestionFailure_Count` |
| `FailedToGenerateSegmentImportTask` | `pinot_controller_segmentImportFailure_Count` |
| `FailedToGenerateExternalTableSyncTask` | `pinot_controller_externalTableSyncTaskGenerationFailure_Count` |
| `MinionTaskGenerationFailures` | `pinot_controller_taskGenerationFailureCount_Count`, `pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value` |
| `MinionTaskGenerationSkipped` | `pinot_controller_taskGenerationSkippedDueToConflict_Count`, `pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value` |
| `MinionSubtasksInErrorForTableAndTaskType` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `FileIngestionSubtasksInErrorForTable` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `FileIngestionSubtasksInError` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `SqlIngestionSubtasksInError` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `SegmentImportSubtasksInError` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `MinionSubtasksDroppedForTable` | `pinot_controller_numMinionSubtasksDropped_Value` |
| `MinionLongWaitingSubtasksForTaskType` | `pinot_controller_numMinionSubtasksWaiting_Value`, `pinot_minion_taskQueueing_Count` |
| `MinionSubTaskHighWaitTime` | `pinot_controller_maxSubtaskWaitTimeMs_Value` |
| `MinionSubTaskHighRunningTime` | `pinot_controller_maxSubtaskRunningTimeMs_Value` |
| `MinionConsistentPushFailures` | `pinot_controller_consistentPushFailure_Count` |
| `MinionsUnderUtilized` | `pinot_minion_numberOfTasks_Value` |
| `OfflineSegmentDelayHoursCrossedThreshold` | `pinot_controller_offlineSegmentDelayHours_Value`, `pinot_controller_scheduledSegments_Count` |
| `MinionSubtaskQueueSizeHigh` | `pinot_controller_numMinionSubtasksWaiting_Value`, `kube_pod_status_phase` |
| `PossibleDataLossDuringSegmentImport` | `pinot_controller_segmentsWithLateData_Value` |

## Related

* [Alerts and metrics overview](/corecapabilities/observability/alerts/overview): severity, the symptom router, the alert index, and how metric names are built.
* [Troubleshooting by feature: Batch ingestion and minion tasks](/corecapabilities/observability/troubleshooting-by-feature#batch-ingestion-and-minion-tasks)
* [Minion task runs](/corecapabilities/observability/minion-task-runs)
* [Ingestion troubleshooting](/corecapabilities/ingestdata/troubleshooting)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.