> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Alerts and metrics

> How StarTree alerts on your StarTree Cloud environment and how Pinot metrics are named: severity levels, where to start from a symptom, an index of every alert, and how to find a metric.

StarTree runs 159 Prometheus alerts on every StarTree Cloud environment. They're split across five pages by what you'd notice, plus one page for the platform underneath. Use this page to find the right one, look an alert up by name in the index below, or learn how the Pinot metrics behind them are named.

<Info>
  **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds on these pages are defaults; StarTree may tune them for your environment.
</Info>

## Start from the symptom

<CardGroup cols={2}>
  <Card title="New data is late or missing" icon="arrow-down-to-line" href="/corecapabilities/observability/alerts/real-time-ingestion">
    Real-time ingestion stopped, lagging, or dropping rows. 9 alerts.
  </Card>

  <Card title="Queries fail, time out, or get rejected" icon="magnifying-glass-chart" href="/corecapabilities/observability/alerts/queries">
    Error rates, timeouts, query quota, broker availability. 12 alerts.
  </Card>

  <Card title="Batch data or a scheduled task didn't land" icon="list-check" href="/corecapabilities/observability/alerts/batch-and-minion-tasks">
    File, SQL, segment import, and external table sync tasks, and minion capacity. 20 alerts.
  </Card>

  <Card title="Results are incomplete, inconsistent, or near a limit" icon="layer-group" href="/corecapabilities/observability/alerts/data-availability-and-storage">
    Serving replicas, segments in error, disk, key, and storage limits. 11 alerts.
  </Card>

  <Card title="Upsert results or storage look wrong" icon="key" href="/corecapabilities/observability/alerts/upsert-tables">
    Primary-key growth, compaction, snapshots, replica consistency. 7 alerts.
  </Card>

  <Card title="Components restart, run hot, or infrastructure is unhealthy" icon="server" href="/corecapabilities/observability/alerts/platform-health">
    Out-of-memory kills, CPU, GC, ZooKeeper, Kubernetes, platform services. 100 alerts.
  </Card>
</CardGroup>

## How to read an alert entry

Each critical alert on the category pages has its own entry:

| Field | Meaning |
| - | - |
| **Fires when** | The condition and how long it must hold before the alert fires |
| **What it means** | The effect on your data or queries |
| **Check first** | The Grafana panel or API call that confirms the cause |
| **What you can do** | Changes on your side: stream, source, table config, queries |
| **What StarTree does** | Actions StarTree on-call takes |

Some entries carry a note. **Often transient** means the alert usually clears on its own. **Usually your fix** means the cause is most often in your stream, source system, or table config. Warning alerts are listed in a table on each page, and every page ends with the metrics each alert evaluates.

Grafana dashboards named on these pages are described in [Grafana dashboards](/corecapabilities/observability/grafana). API paths are Pinot controller endpoints, available from **Cluster Manager → Swagger API** in the Data Portal.

## Alert index

All 159 alerts, 119 critical and 40 warning, in alphabetical order.

| Alert | Severity | Page |
| - | - | - |
| `AuthServiceHighHttp5xxErrorRate` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `BrokerDataTableDeserializationExceptions` | Critical | [Queries](/corecapabilities/observability/alerts/queries#brokerdatatabledeserializationexceptions) |
| `BrokerNoServerFoundAlert` | Warning | [Queries](/corecapabilities/observability/alerts/queries#brokernoserverfoundalert) |
| `BrokerQueryExecutionLatency` | Warning | [Queries](/corecapabilities/observability/alerts/queries#brokerqueryexecutionlatency) |
| `BrokerResponseAlert` | Warning | [Queries](/corecapabilities/observability/alerts/queries#brokerresponsealert) |
| `CertificateExpiration` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `CertificateRenewal` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ConfiguredRFLessThanThreshold` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#configuredrflessthanthreshold) |
| `ContainerOOMKills` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health#containeroomkills) |
| `DetectionTaskHealth` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `FailedToGenerateExternalTableSyncTask` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#failedtogenerateexternaltablesynctask) |
| `FailedToGenerateFileIngestionTask` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#failedtogeneratefileingestiontask) |
| `FailedToGenerateSegmentImportTask` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#failedtogeneratesegmentimporttask) |
| `FailedToGenerateSqlIngestionTask` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#failedtogeneratesqlingestiontask) |
| `FileIngestionSubtasksInError` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#fileingestionsubtasksinerror) |
| `FileIngestionSubtasksInErrorForTable` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#fileingestionsubtasksinerrorfortable) |
| `HighAvailablePinotStatefulsetReplicasMismatch` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#highavailablepinotstatefulsetreplicasmismatch) |
| `HighNonServerPodsRestarts` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#highnonserverpodsrestarts) |
| `HighQueryCriticalErrorRate` | Critical | [Queries](/corecapabilities/observability/alerts/queries#highquerycriticalerrorrate) |
| `HighQueryTimeoutPercentage` | Critical | [Queries](/corecapabilities/observability/alerts/queries#highquerytimeoutpercentage) |
| `HighRealtimeIngestionOffsetLag` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#highrealtimeingestionoffsetlag) |
| `HighServerPodsRestarts` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#highserverpodsrestarts) |
| `HighSystemQueryErrorRate` | Critical | [Queries](/corecapabilities/observability/alerts/queries#highsystemqueryerrorrate) |
| `HighTotalDocsToPKCountRatio` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#hightotaldocstopkcountratio) |
| `HighTotalNumberOfPrimaryKeysPerServer` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#hightotalnumberofprimarykeysperserver) |
| `KubernetesContainerOOMKilled` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesContainerTerminated` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesMinionPodNotHealthy` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeDiskPressure` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeMemoryPressure` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeNetworkUnavailable` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeNotReady` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeOutOfPodCapacity` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesPinotContainerOOMKilled` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#kubernetespinotcontaineroomkilled) |
| `KubernetesPinotPodCrashLooping` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#kubernetespinotpodcrashlooping) |
| `KubernetesPodCrashLooping` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesPodFailed` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesPodNotHealthy` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesPodPending` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesReplicasetReplicasMismatch` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesStatefulsetUpdateNotRolledOut` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesThirdeyeContainerOOMKilled` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesThirdeyePodCrashLooping` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `LowBrokerAvailability` | Critical | [Queries](/corecapabilities/observability/alerts/queries#lowbrokeravailability) |
| `MariadbConnectionsExhausted` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbDown` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHANotAvailable` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationIOThreadStopped` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationLagHigh` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationRecvQueueSize` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationSendQueueSize` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationSQLThreadStopped` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MinionConsistentPushFailures` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionconsistentpushfailures) |
| `MinionLongWaitingSubtasksForTaskType` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionlongwaitingsubtasksfortasktype) |
| `MinionSubTaskHighRunningTime` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtaskhighrunningtime) |
| `MinionSubTaskHighWaitTime` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtaskhighwaittime) |
| `MinionSubtaskQueueSizeHigh` | Warning | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtaskqueuesizehigh) |
| `MinionSubtasksDroppedForTable` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtasksdroppedfortable) |
| `MinionSubtasksInErrorForTableAndTaskType` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtasksinerrorfortableandtasktype) |
| `MinionsUnderUtilized` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsunderutilized) |
| `MinionTaskGenerationFailures` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#miniontaskgenerationfailures) |
| `MinionTaskGenerationSkipped` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#miniontaskgenerationskipped) |
| `MySQLStorageExhaustion` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MySQLStorageUsage` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NettyResponseFetchExceptions` | Critical | [Queries](/corecapabilities/observability/alerts/queries#nettyresponsefetchexceptions) |
| `NoBrokerAvailable` | Critical | [Queries](/corecapabilities/observability/alerts/queries#nobrokeravailable) |
| `NoControllerLeaderAlert` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#nocontrollerleaderalert) |
| `NodeConntrackExhausted` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeConntrackLow` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeDataDiskUsageCritical` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeDataDiskUsageHigh` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeNetworkBandwidthInThrottled` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeNetworkBandwidthOutThrottled` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeNetworkPPSThrottled` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NoRealtimeIngestionPerServer` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#norealtimeingestionperserver) |
| `OfflineSegmentDelayHoursCrossedThreshold` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#offlinesegmentdelayhourscrossedthreshold) |
| `OperatorDisabled` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `OperatorOffline` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PartitionConsumerCreateExceptions` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#partitionconsumercreateexceptions) |
| `PinotCertificateExpiration` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PinotCertificateRenewal` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PinotComponentHighCPUUsage` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotcomponenthighcpuusage) |
| `PinotControllerHighThreadsBlocked` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotcontrollerhighthreadsblocked) |
| `PinotHighGcPercent` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinothighgcpercent) |
| `PinotPodImagePullErrors` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotpodimagepullerrors) |
| `PinotPodStuckInContainerCreating` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotpodstuckincontainercreating) |
| `PinotPodVolumeCrossed80Percent` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#pinotpodvolumecrossed80percent) |
| `PinotPodVolumeCrossed90Percent` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#pinotpodvolumecrossed90percent) |
| `PinotPodWaiting` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotpodwaiting) |
| `PinotStatefulsetReplicasMismatch` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotstatefulsetreplicasmismatch) |
| `PinotZnodeSizeTooCloseToZkBufferLimit` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#pinotznodesizetooclosetozkbufferlimit) |
| `PlatformAuthAPIP95Latency` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformAuthErrorLogs` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformAuthInfoP95Latency` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformAuthP95Latency` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformAuthZP95Latency` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformDeploymentReplicasMismatch` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformFrontendErrorLogs` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformPodWaiting` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformStatefulsetReplicasMismatch` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PodVolumeCrossed80Percent` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PodVolumeCrossed90Percent` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PossibleDataLossDuringSegmentImport` | Warning | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#possibledatalossduringsegmentimport) |
| `PrometheusContainerHighMemoryUsage` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusContainerOOMKilled` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusPodVolumeCrossed65Percent` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusPodVolumeCrossed75Percent` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbCheckpointCreationFailures` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbCheckpointDeletionFailures` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbCompactionsFailed` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbHeadTruncationsFailed` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbReloadFailures` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbWalCorruptions` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbWalTruncationsFailed` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `QueriesFailingWithUnavailableSegment` | Warning | [Queries](/corecapabilities/observability/alerts/queries#queriesfailingwithunavailablesegment) |
| `QueryQuotaAlert` | Critical | [Queries](/corecapabilities/observability/alerts/queries#queryquotaalert) |
| `RBACServiceHighHttp5xxErrorRate` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `RealtimeConsumptionExceptions` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimeconsumptionexceptions) |
| `RealtimeIngestionHighRowsWithErrorsRate` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimeingestionhighrowswitherrorsrate) |
| `RealtimeIngestionHighSkippedRowsRate` | Warning | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimeingestionhighskippedrowsrate) |
| `RealtimeIngestionStopped` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimeingestionstopped) |
| `RealtimePartitionMismatch` | Warning | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimepartitionmismatch) |
| `ResourceUtilizationLimitExceeded` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#resourceutilizationlimitexceeded) |
| `RocksDBHighCompactionRate` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#rocksdbhighcompactionrate) |
| `RocksDBHighFlushRate` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#rocksdbhighflushrate) |
| `RocksDBHighMemoryUsage` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#rocksdbhighmemoryusage) |
| `SegmentError` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#segmenterror) |
| `SegmentImportSubtasksInError` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#segmentimportsubtasksinerror) |
| `SegmentReplicasCriticallyLowForHATable` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#segmentreplicascriticallylowforhatable) |
| `SegmentReplicasUnavailable` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#segmentreplicasunavailable) |
| `SnapshotCreationFailed` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#snapshotcreationfailed) |
| `SqlIngestionSubtasksInError` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#sqlingestionsubtasksinerror) |
| `StorageQuotaUtilizationAlert` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#storagequotautilizationalert) |
| `StreamDataLoss` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#streamdataloss) |
| `SystemQueryLogStorageQuotaUtilization` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#systemquerylogstoragequotautilization) |
| `ThirdeyeCertificateExpiration` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyeCertificateRenewal` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyeMultipleNamespaceConfigurations` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyeNotificationFailure` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyePodVolumeCrossed80Percent` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyePodVolumeCrossed90Percent` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyePodWaiting` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TierStorageServerPreloadIndexReachingLimit` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#tierstorageserverpreloadindexreachinglimit) |
| `TraefikDeploymentReplicasMismatch` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikHighHttp499PinotErrorRateService` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikHighHttp4xxErrorRateService` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikHighHttp5xxErrorRateService` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikPinotBrokerHttp403BurnRateHigh` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikPinotBrokerHttp5xxBurnRateHigh` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikServiceConnectionsAbnormalGrowth` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TSDBDataInsufficient` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `UpsertInconsistentReplicas` | Warning | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#upsertinconsistentreplicas) |
| `X509ExporterReadErrors` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ZookeeperErrorCount` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeepererrorcount) |
| `ZookeeperIncorrectNumberOfLeaders` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeeperincorrectnumberofleaders) |
| `ZookeeperIncorrectQuorumSize` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeeperincorrectquorumsize) |
| `ZookeeperLeaderElectionTooLong` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeeperleaderelectiontoolong) |
| `ZookeeperLeaderUnavailable` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeeperleaderunavailable) |
| `ZookeeperMetricsMissing` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeepermetricsmissing) |

## Reading Pinot metrics

Your cluster's Pinot metrics are available in [Grafana](/corecapabilities/observability/grafana), through the Prometheus data source. Each alert page lists the metrics worth watching for its area and the series each alert evaluates. This section explains how those names are built, so you can find the one you need rather than guess.

<Warning>
  **A metric that has never been recorded does not exist.** If no value has been published — a table that has never been queried, a task type that has never run — the metric is absent rather than present with a zero. An empty chart can mean "nothing happened," not "the metric is wrong."
</Warning>

### Naming convention

Metric names follow a consistent shape:

```
pinot_<component>_<metricName>_<Suffix>
```

| Part | Values | Example |
| - | - | - |
| `component` | `controller`, `broker`, `server`, `minion` | `pinot_server_…` |
| `metricName` | The Pinot metric name, in camelCase | `…_realtimeIngestionDelayMs_…` |
| `Suffix` | The statistic — see below | `…_Value` |

So a real-time ingestion delay gauge on a server becomes `pinot_server_realtimeIngestionDelayMs_Value`, and a broker query counter becomes `pinot_broker_queries_Count`.

A few metrics do not follow the pattern. RocksDB metrics, used by off-heap upsert, appear as `pinot_server_rocksdb_ticker_<name>` and `pinot_server_rocksdb_histogram_<name>_<stat>`. JVM metrics such as `jvm_gc_collection_seconds_sum` and `jvm_threads_state` keep their standard JVM names and carry a `component` label instead of a `pinot_` prefix.

#### Suffixes

| Suffix | Meaning | Applies to |
| - | - | - |
| `_Value` | Current value | Gauges |
| `_Count` | Monotonically increasing total | Meters and timers |
| `_OneMinuteRate`, `_FiveMinuteRate` | Exponentially weighted rate | Meters |
| `_50thPercentile`, `_75thPercentile`, `_95thPercentile`, `_99thPercentile` | Latency distribution | Timers |

Not every statistic is available for every metric. If a suffix such as `_Mean` or `_FifteenMinuteRate` returns nothing, use one from the table above instead — `_50thPercentile` for a typical value, or a rate over `_Count` for throughput. Confirm what a metric actually offers by running the bare name in **Explore** and reading the series that come back.

#### Labels

Metrics are labelled so you can slice them without name explosion:

| Label | Meaning |
| - | - |
| `table` | Table name with type suffix, for example `orders_REALTIME` |
| `database` | Database the table belongs to, when the table is not in the default database |
| `tableType` | `OFFLINE` or `REALTIME` |
| `partition` | Stream partition, on ingestion metrics |
| `topic`, `topic_name` | Source stream topic, on ingestion metrics |
| `taskType` | Minion task type, for example `SegmentGenerationAndPushTask` |
| `scope` | Metric scope where a metric is emitted at more than one level |
| `column` | Column name, on the few per-column metrics |
| `throttlerSet` | Throttler pool, on server throttling metrics |

Some metrics also carry pod-level labels such as `pod` or `kubernetes_pod_name`. These identify the component instance rather than a table, so resource questions ("which server is hot?") are answered by grouping on those, and workload questions ("which table is slow?") by grouping on `table`.

<Info>
  The `table` label always carries the type suffix. `table="orders_REALTIME"` and `table="orders_OFFLINE"` are separate series for a hybrid table, so a per-table chart of a hybrid table needs both, or a `table=~"orders_.*"` matcher.
</Info>

### Finding a metric that is not listed

The lists above are curated, not exhaustive — Pinot exports far more than this. To find something specific:

<Steps>
  <Step title="Browse in Grafana Explore">
    Open Grafana, choose **Explore**, and select the Prometheus data source. Type `pinot_server_` in the metric field and the metric browser will complete against everything currently present in your cluster. This is the fastest way to check whether a metric exists at all.
  </Step>

  <Step title="Search by fragment">
    Metric names are camelCase inside a snake\_case wrapper, so search on a distinctive fragment — `Upsert`, `Ingestion`, `Segment` — rather than a whole name. Prometheus's metric browser matches on substring.
  </Step>

  <Step title="Check the labels before charting">
    Run the bare metric name first and look at the label set on the returned series. Whether a metric is per-table, per-partition, or per-pod determines whether you need `sum by (table)` or `avg by (pod)` — and getting this wrong is the most common cause of a chart that looks alarming and means nothing.
  </Step>

  <Step title="Copy from an existing panel">
    Every panel on the [built-in dashboards](/corecapabilities/observability/grafana) exposes its query. Open the panel menu, choose **Explore**, and you get a working query with the right labels and aggregation to modify.
  </Step>
</Steps>

## Related

* [Health checks](/corecapabilities/observability/health-checks) and [Table health and email alerts](/corecapabilities/observability/table-health-and-alerting): Data Portal checks and email alerts, separate from these Prometheus alerts.
* [Grafana dashboards](/corecapabilities/observability/grafana): the built-in dashboards over these metrics.
* [Accessing logs](/corecapabilities/observability/access-logs): when a metric tells you something happened and you need the log line.
* [Query Logger](/corecapabilities/query_data/advanced_operations/query-logger): per-query records rather than aggregates.
* [Troubleshooting: start here](/corecapabilities/observability/troubleshooting): step-by-step triage when something is wrong.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.