Metrics reference

Your cluster’s Pinot metrics are available in Grafana, through the Prometheus data source. This page explains how those metric names are constructed — so you can find the one you need rather than guess — and lists the metrics that matter most, grouped by what you would be investigating.

A metric that has never been recorded does not exist. If no value has been published — a table that has never been queried, a task type that has never run — the metric is absent rather than present with a zero. An empty chart can mean “nothing happened,” not “the metric is wrong.”

Naming convention

Metric names follow a consistent shape:

pinot_<component>_<metricName>_<Suffix>

Part	Values	Example
`component`	`controller`, `broker`, `server`, `minion`	`pinot_server_…`
`metricName`	The Pinot metric name, in camelCase	`…_realtimeIngestionDelayMs_…`
`Suffix`	The statistic — see below	`…_Value`

So a real-time ingestion delay gauge on a server becomes pinot_server_realtimeIngestionDelayMs_Value, and a broker query counter becomes pinot_broker_queries_Count. A few metrics do not follow the pattern. RocksDB metrics, used by off-heap upsert, appear as pinot_server_rocksdb_ticker_<name> and pinot_server_rocksdb_histogram_<name>_<stat>. JVM metrics such as jvm_gc_collection_seconds_sum and jvm_threads_state keep their standard JVM names and carry a component label instead of a pinot_ prefix.

Suffixes

Suffix	Meaning	Applies to
`_Value`	Current value	Gauges
`_Count`	Monotonically increasing total	Meters and timers
`_OneMinuteRate`, `_FiveMinuteRate`	Exponentially weighted rate	Meters
`_50thPercentile`, `_75thPercentile`, `_95thPercentile`, `_99thPercentile`	Latency distribution	Timers

Not every statistic is available for every metric. If a suffix such as _Mean or _FifteenMinuteRate returns nothing, use one from the table above instead — _50thPercentile for a typical value, or a rate over _Count for throughput. Confirm what a metric actually offers by running the bare name in Explore and reading the series that come back.

Labels

Metrics are labelled so you can slice them without name explosion:

Label	Meaning
`table`	Table name with type suffix, for example `orders_REALTIME`
`database`	Database the table belongs to, when the table is not in the default database
`tableType`	`OFFLINE` or `REALTIME`
`partition`	Stream partition, on ingestion metrics
`topic`, `topic_name`	Source stream topic, on ingestion metrics
`taskType`	Minion task type, for example `SegmentGenerationAndPushTask`
`scope`	Metric scope where a metric is emitted at more than one level
`column`	Column name, on the few per-column metrics
`throttlerSet`	Throttler pool, on server throttling metrics

Some metrics also carry pod-level labels such as pod or kubernetes_pod_name. These identify the component instance rather than a table, so resource questions (“which server is hot?”) are answered by grouping on those, and workload questions (“which table is slow?”) by grouping on table.

The table label always carries the type suffix. table="orders_REALTIME" and table="orders_OFFLINE" are separate series for a hybrid table, so a per-table chart of a hybrid table needs both, or a table=~"orders_.*" matcher.

Metrics worth watching

These are the metrics behind the Grafana dashboards. They are the ones to reach for first; they are not the complete set.

Real-time ingestion

Metric	What it tells you
`pinot_server_llcPartitionConsuming_Value`	`1` if the partition is consuming, `0` if it is not. The single most direct answer to “is ingestion running?”
`pinot_server_realtimeIngestionDelayMs_Value`	Age of the newest ingested record, per partition. Rising steadily means falling behind.
`pinot_server_realtimeIngestionOffsetLag_Value`	Offset distance behind the stream head. Compare its direction with the delay: a large but shrinking lag is a recovery.
`pinot_server_highestStreamOffsetConsumed_Value`	The last offset consumed. Flat while delay grows is the signature of a stuck consumer.
`pinot_server_realtime_consumptionExceptions_Count`	Consumption threw. The stream is reachable but records are failing.
`pinot_server_rowsWithErrors_Count`	Rows rejected during decode or transform. Almost always a schema mismatch.
`pinot_server_streamConsumerCreateExceptions_Count`	Could not create the consumer at all — credentials, topic, or network path.
`pinot_server_realtimePartitionMismatch_Count`	Pinot’s partition count no longer matches the stream’s.
`pinot_controller_tableConsumptionPaused_Value`	`1` when consumption is paused. Check this before diagnosing “stopped” ingestion.
`pinot_controller_tableDisabled_Value`	`1` when the table is disabled.

Queries

Metric	What it tells you
`pinot_broker_queries_Count`	Single-stage query volume. The denominator for most error-rate calculations.
`pinot_broker_multiStageQueries_Count`, `pinot_broker_multiStageQueriesGlobal_Count`	Multi-stage engine query volume, per table and cluster-wide.
`pinot_broker_queryTotalTimeMs_95thPercentile`	Broker-side P95 latency, per table.
`pinot_broker_queryErrorBrokerTimeout_OneMinuteRate` and siblings	Error rate split by cause — `ExecutionTimeout`, `QuerySchedulingTimeout`, `ServerNotResponding`, `ServerSegmentMissing`, `BrokerSegmentUnavailable`, `QueryPlanning`, `SqlRuntime`, `Internal`, `MergeResponse`, `QueryCancellation`, `ServerShuttingDown`, `BrokerRequestSend`. Splitting by cause is what turns “queries are failing” into a diagnosis.
`pinot_broker_queryCriticalError_OneMinuteRate`	Errors classed as critical.
`pinot_broker_brokerResponsesWithTimeouts_Count`	Responses that timed out.
`pinot_broker_brokerResponsesWithUnavailableSegments_Count`	Responses returned with segments missing — results may be incomplete.
`pinot_broker_brokerResponsesWithPartialServersResponded_Count`	Responses built without every server answering.
`pinot_broker_noServerFoundExceptions_Count`	No server could serve the table. Routing or availability, not query shape.
`pinot_broker_queryQuotaExceeded_Count`	Queries rejected by quota.
`pinot_server_schedulingTimeoutExceptions_Count`	Queries that timed out waiting to be scheduled rather than while executing — the distinction between “too busy” and “too slow”.
`pinot_server_numMissingSegments_Count`	Segments a server was asked for and did not have.

For per-query rather than aggregate analysis, use system_query_log — metrics tell you the shape of the problem, the query log tells you which query.

Upserts and dedup

Metric	What it tells you
`pinot_server_upsertPrimaryKeysCount_Value`	Primary keys held in memory. Grows with distinct keys, not rows — the number that outgrows a cluster silently.
`pinot_server_documentCount_Value`	Documents per table. Its ratio to primary-key count tells you whether compaction is keeping up.
`pinot_server_realtimeUpsertInconsistentRows_Count`	Rows that could not be reconciled between replicas.
`pinot_server_partialUpsertKeysNotReplaced_Count`	Partial-upsert keys not replaced during segment replacement.
`pinot_server_rocksdb_ticker_total_memtable_bytes`	Off-heap upsert metadata memory.
`pinot_server_rocksdb_histogram_flush_time_count`, `pinot_server_rocksdb_histogram_compaction_time_count`	Off-heap upsert write amplification.

Minion tasks and batch ingestion

Metric	What it tells you
`pinot_controller_percentMinionSubtasksInError_Value`	Share of a task type’s subtasks in error, per table.
`pinot_controller_numMinionSubtasksWaiting_Value`	Queue depth, per task type.
`pinot_controller_numMinionSubtasksDropped_Value`	Subtasks dropped rather than run.
`pinot_controller_subtaskWaitingTime_Max`, `pinot_controller_subtaskRunningTime_Max`	Worst-case wait and run time, per table and task type.
`pinot_controller_taskGenerationFailureCount_Count`	Generation failed — no subtasks were even created.
`pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value`	How long since anything generated successfully. Pair it with the failure count: failures matter far more when this is also large.
`pinot_controller_taskGenerationSkippedDueToConflict_Count`	Generation skipped because a conflicting task was in flight.
`pinot_controller_consistentPushFailure_Count`	Consistent-push failures.
`pinot_controller_sqlIngestionFailure_Count`, `pinot_controller_fileIngestionFailure_Count`, `pinot_controller_segmentImportFailure_Count`	Per-ingestion-type generation failures.
`pinot_minion_taskQueueing_Count`	Minions actually dequeuing work. Zero while the queue is deep means no minion is picking anything up.
`pinot_controller_offlineSegmentDelayHours_Value`	Age of the newest offline segment — how far behind the real-time-to-offline job is.

Cluster, storage, and capacity

Metric	What it tells you
`pinot_controller_percentOfReplicas_Value`	Share of segment replicas available, per table. Non-lead controllers emit a negative placeholder, so filter `>= 0`.
`pinot_controller_replicationFromConfig_Value`	Configured replication factor.
`pinot_controller_segmentsInErrorState_Value`	Segments stuck in error.
`pinot_controller_tableStorageQuotaUtilization_Value`	Percentage of the table’s storage quota in use.
`pinot_controller_pinotControllerLeader_Value`	`1` on the leader. Summing to zero across the cluster means no leader.
`pinot_controller_idealstateZnodeByteSize_Value`, `pinot_controller_externalviewZnodeByteSize_Value`, `pinot_controller_propertystoreSegmentChildrenByteSize_Value`	ZooKeeper znode sizes — the constraint that eventually bounds segment count.
`pinot_server_preloadCacheLayerReservedSizeBytes`, `pinot_server_preloadCacheLayerMaxSizeBytes`	Tiered-storage preload index headroom. Chart the ratio.
`jvm_gc_collection_seconds_sum`	GC time, by `component` and `gc`. Rate of this over wall-clock time is the GC percentage.
`jvm_threads_state`	Thread counts by `state` and `component`. Blocked-versus-runnable is how a wedged controller shows up.

Finding a metric that is not listed here

The lists above are curated, not exhaustive — Pinot exports far more than this. To find something specific:

Browse in Grafana Explore

Open Grafana, choose Explore, and select the Prometheus data source. Type pinot_server_ in the metric field and the metric browser will complete against everything currently present in your cluster. This is the fastest way to check whether a metric exists at all.

Search by fragment

Metric names are camelCase inside a snake_case wrapper, so search on a distinctive fragment — Upsert, Ingestion, Segment — rather than a whole name. Prometheus’s metric browser matches on substring.

Check the labels before charting

Run the bare metric name first and look at the label set on the returned series. Whether a metric is per-table, per-partition, or per-pod determines whether you need sum by (table) or avg by (pod) — and getting this wrong is the most common cause of a chart that looks alarming and means nothing.

Copy from an existing panel

Every panel on the built-in dashboards exposes its query. Open the panel menu, choose Explore, and you get a working query with the right labels and aggregation to modify.

Grafana dashboards — the built-in dashboards over these metrics
Accessing logs — when a metric tells you something happened and you need the log line
Query Logger — per-query records rather than aggregates
Table Health Dashboard — evaluated checks rather than raw metrics

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Naming convention

Suffixes

Labels

Metrics worth watching

Real-time ingestion

Queries

Upserts and dedup

Minion tasks and batch ingestion

Cluster, storage, and capacity

Finding a metric that is not listed here

​Naming convention

​Suffixes

​Labels

​Metrics worth watching

​Real-time ingestion

​Queries

​Upserts and dedup

​Minion tasks and batch ingestion

​Cluster, storage, and capacity

​Finding a metric that is not listed here

​Related

Naming convention

Suffixes

Labels

Metrics worth watching

Real-time ingestion

Queries

Upserts and dedup

Minion tasks and batch ingestion

Cluster, storage, and capacity

Finding a metric that is not listed here

Related