> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Metrics reference
> How Pinot metrics are named in Prometheus, which labels they carry, and the metrics worth watching for ingestion, queries, upserts, tasks, and capacity.
Your cluster's Pinot metrics are available in [Grafana](/corecapabilities/observability/grafana), through the Prometheus data source. This page explains how those metric names are constructed — so you can find the one you need rather than guess — and lists the metrics that matter most, grouped by what you would be investigating.
**A metric that has never been recorded does not exist.** If no value has been published — a table that has never been queried, a task type that has never run — the metric is absent rather than present with a zero. An empty chart can mean "nothing happened," not "the metric is wrong."
## Naming convention
Metric names follow a consistent shape:
```
pinot___
```
| Part | Values | Example |
| ------------ | ------------------------------------------ | ------------------------------ |
| `component` | `controller`, `broker`, `server`, `minion` | `pinot_server_…` |
| `metricName` | The Pinot metric name, in camelCase | `…_realtimeIngestionDelayMs_…` |
| `Suffix` | The statistic — see below | `…_Value` |
So a real-time ingestion delay gauge on a server becomes `pinot_server_realtimeIngestionDelayMs_Value`, and a broker query counter becomes `pinot_broker_queries_Count`.
A few metrics do not follow the pattern. RocksDB metrics, used by off-heap upsert, appear as `pinot_server_rocksdb_ticker_` and `pinot_server_rocksdb_histogram__`. JVM metrics such as `jvm_gc_collection_seconds_sum` and `jvm_threads_state` keep their standard JVM names and carry a `component` label instead of a `pinot_` prefix.
### Suffixes
| Suffix | Meaning | Applies to |
| -------------------------------------------------------------------------- | ------------------------------ | ----------------- |
| `_Value` | Current value | Gauges |
| `_Count` | Monotonically increasing total | Meters and timers |
| `_OneMinuteRate`, `_FiveMinuteRate` | Exponentially weighted rate | Meters |
| `_50thPercentile`, `_75thPercentile`, `_95thPercentile`, `_99thPercentile` | Latency distribution | Timers |
Not every statistic is available for every metric. If a suffix such as `_Mean` or `_FifteenMinuteRate` returns nothing, use one from the table above instead — `_50thPercentile` for a typical value, or a rate over `_Count` for throughput. Confirm what a metric actually offers by running the bare name in **Explore** and reading the series that come back.
### Labels
Metrics are labelled so you can slice them without name explosion:
| Label | Meaning |
| --------------------- | ---------------------------------------------------------------------------- |
| `table` | Table name with type suffix, for example `orders_REALTIME` |
| `database` | Database the table belongs to, when the table is not in the default database |
| `tableType` | `OFFLINE` or `REALTIME` |
| `partition` | Stream partition, on ingestion metrics |
| `topic`, `topic_name` | Source stream topic, on ingestion metrics |
| `taskType` | Minion task type, for example `SegmentGenerationAndPushTask` |
| `scope` | Metric scope where a metric is emitted at more than one level |
| `column` | Column name, on the few per-column metrics |
| `throttlerSet` | Throttler pool, on server throttling metrics |
Some metrics also carry pod-level labels such as `pod` or `kubernetes_pod_name`. These identify the component instance rather than a table, so resource questions ("which server is hot?") are answered by grouping on those, and workload questions ("which table is slow?") by grouping on `table`.
The `table` label always carries the type suffix. `table="orders_REALTIME"` and `table="orders_OFFLINE"` are separate series for a hybrid table, so a per-table chart of a hybrid table needs both, or a `table=~"orders_.*"` matcher.
## Metrics worth watching
These are the metrics behind the [Grafana dashboards](/corecapabilities/observability/grafana). They are the ones to reach for first; they are not the complete set.
### Real-time ingestion
| Metric | What it tells you |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `pinot_server_llcPartitionConsuming_Value` | `1` if the partition is consuming, `0` if it is not. The single most direct answer to "is ingestion running?" |
| `pinot_server_realtimeIngestionDelayMs_Value` | Age of the newest ingested record, per partition. Rising steadily means falling behind. |
| `pinot_server_realtimeIngestionOffsetLag_Value` | Offset distance behind the stream head. Compare its *direction* with the delay: a large but shrinking lag is a recovery. |
| `pinot_server_highestStreamOffsetConsumed_Value` | The last offset consumed. Flat while delay grows is the signature of a stuck consumer. |
| `pinot_server_realtime_consumptionExceptions_Count` | Consumption threw. The stream is reachable but records are failing. |
| `pinot_server_rowsWithErrors_Count` | Rows rejected during decode or transform. Almost always a schema mismatch. |
| `pinot_server_streamConsumerCreateExceptions_Count` | Could not create the consumer at all — credentials, topic, or network path. |
| `pinot_server_realtimePartitionMismatch_Count` | Pinot's partition count no longer matches the stream's. |
| `pinot_controller_tableConsumptionPaused_Value` | `1` when consumption is paused. Check this before diagnosing "stopped" ingestion. |
| `pinot_controller_tableDisabled_Value` | `1` when the table is disabled. |
### Queries
| Metric | What it tells you |
| ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pinot_broker_queries_Count` | Single-stage query volume. The denominator for most error-rate calculations. |
| `pinot_broker_multiStageQueries_Count`, `pinot_broker_multiStageQueriesGlobal_Count` | Multi-stage engine query volume, per table and cluster-wide. |
| `pinot_broker_queryTotalTimeMs_95thPercentile` | Broker-side P95 latency, per table. |
| `pinot_broker_queryErrorBrokerTimeout_OneMinuteRate` and siblings | Error rate split by cause — `ExecutionTimeout`, `QuerySchedulingTimeout`, `ServerNotResponding`, `ServerSegmentMissing`, `BrokerSegmentUnavailable`, `QueryPlanning`, `SqlRuntime`, `Internal`, `MergeResponse`, `QueryCancellation`, `ServerShuttingDown`, `BrokerRequestSend`. Splitting by cause is what turns "queries are failing" into a diagnosis. |
| `pinot_broker_queryCriticalError_OneMinuteRate` | Errors classed as critical. |
| `pinot_broker_brokerResponsesWithTimeouts_Count` | Responses that timed out. |
| `pinot_broker_brokerResponsesWithUnavailableSegments_Count` | Responses returned with segments missing — results may be incomplete. |
| `pinot_broker_brokerResponsesWithPartialServersResponded_Count` | Responses built without every server answering. |
| `pinot_broker_noServerFoundExceptions_Count` | No server could serve the table. Routing or availability, not query shape. |
| `pinot_broker_queryQuotaExceeded_Count` | Queries rejected by quota. |
| `pinot_server_schedulingTimeoutExceptions_Count` | Queries that timed out waiting to be scheduled rather than while executing — the distinction between "too busy" and "too slow". |
| `pinot_server_numMissingSegments_Count` | Segments a server was asked for and did not have. |
For per-query rather than aggregate analysis, use [`system_query_log`](/corecapabilities/query_data/advanced_operations/query-logger) — metrics tell you the shape of the problem, the query log tells you which query.
### Upserts and dedup
| Metric | What it tells you |
| --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `pinot_server_upsertPrimaryKeysCount_Value` | Primary keys held in memory. Grows with distinct keys, not rows — the number that outgrows a cluster silently. |
| `pinot_server_documentCount_Value` | Documents per table. Its ratio to primary-key count tells you whether compaction is keeping up. |
| `pinot_server_realtimeUpsertInconsistentRows_Count` | Rows that could not be reconciled between replicas. |
| `pinot_server_partialUpsertKeysNotReplaced_Count` | Partial-upsert keys not replaced during segment replacement. |
| `pinot_server_rocksdb_ticker_total_memtable_bytes` | Off-heap upsert metadata memory. |
| `pinot_server_rocksdb_histogram_flush_time_count`, `pinot_server_rocksdb_histogram_compaction_time_count` | Off-heap upsert write amplification. |
### Minion tasks and batch ingestion
| Metric | What it tells you |
| ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
| `pinot_controller_percentMinionSubtasksInError_Value` | Share of a task type's subtasks in error, per table. |
| `pinot_controller_numMinionSubtasksWaiting_Value` | Queue depth, per task type. |
| `pinot_controller_numMinionSubtasksDropped_Value` | Subtasks dropped rather than run. |
| `pinot_controller_subtaskWaitingTime_Max`, `pinot_controller_subtaskRunningTime_Max` | Worst-case wait and run time, per table and task type. |
| `pinot_controller_taskGenerationFailureCount_Count` | Generation failed — no subtasks were even created. |
| `pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value` | How long since anything generated successfully. Pair it with the failure count: failures matter far more when this is also large. |
| `pinot_controller_taskGenerationSkippedDueToConflict_Count` | Generation skipped because a conflicting task was in flight. |
| `pinot_controller_consistentPushFailure_Count` | Consistent-push failures. |
| `pinot_controller_sqlIngestionFailure_Count`, `pinot_controller_fileIngestionFailure_Count`, `pinot_controller_segmentImportFailure_Count` | Per-ingestion-type generation failures. |
| `pinot_minion_taskQueueing_Count` | Minions actually dequeuing work. Zero while the queue is deep means no minion is picking anything up. |
| `pinot_controller_offlineSegmentDelayHours_Value` | Age of the newest offline segment — how far behind the real-time-to-offline job is. |
### Cluster, storage, and capacity
| Metric | What it tells you |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `pinot_controller_percentOfReplicas_Value` | Share of segment replicas available, per table. Non-lead controllers emit a negative placeholder, so filter `>= 0`. |
| `pinot_controller_replicationFromConfig_Value` | Configured replication factor. |
| `pinot_controller_segmentsInErrorState_Value` | Segments stuck in error. |
| `pinot_controller_tableStorageQuotaUtilization_Value` | Percentage of the table's storage quota in use. |
| `pinot_controller_pinotControllerLeader_Value` | `1` on the leader. Summing to zero across the cluster means no leader. |
| `pinot_controller_idealstateZnodeByteSize_Value`, `pinot_controller_externalviewZnodeByteSize_Value`, `pinot_controller_propertystoreSegmentChildrenByteSize_Value` | ZooKeeper znode sizes — the constraint that eventually bounds segment count. |
| `pinot_server_preloadCacheLayerReservedSizeBytes`, `pinot_server_preloadCacheLayerMaxSizeBytes` | Tiered-storage preload index headroom. Chart the ratio. |
| `jvm_gc_collection_seconds_sum` | GC time, by `component` and `gc`. Rate of this over wall-clock time is the GC percentage. |
| `jvm_threads_state` | Thread counts by `state` and `component`. Blocked-versus-runnable is how a wedged controller shows up. |
## Finding a metric that is not listed here
The lists above are curated, not exhaustive — Pinot exports far more than this. To find something specific:
Open Grafana, choose **Explore**, and select the Prometheus data source. Type `pinot_server_` in the metric field and the metric browser will complete against everything currently present in your cluster. This is the fastest way to check whether a metric exists at all.
Metric names are camelCase inside a snake\_case wrapper, so search on a distinctive fragment — `Upsert`, `Ingestion`, `Segment` — rather than a whole name. Prometheus's metric browser matches on substring.
Run the bare metric name first and look at the label set on the returned series. Whether a metric is per-table, per-partition, or per-pod determines whether you need `sum by (table)` or `avg by (pod)` — and getting this wrong is the most common cause of a chart that looks alarming and means nothing.
Every panel on the [built-in dashboards](/corecapabilities/observability/grafana) exposes its query. Open the panel menu, choose **Explore**, and you get a working query with the right labels and aggregation to modify.
## Related
* [Grafana dashboards](/corecapabilities/observability/grafana) — the built-in dashboards over these metrics
* [Accessing logs](/corecapabilities/observability/access-logs) — when a metric tells you something happened and you need the log line
* [Query Logger](/corecapabilities/query_data/advanced_operations/query-logger) — per-query records rather than aggregates
* [Table Health Dashboard](/corecapabilities/observability/table-health-and-alerting) — evaluated checks rather than raw metrics