Where results appear
Two views surface these checks, and they differ in scope rather than in what they measure:- Cluster Health — log in to the Data Portal and click Cluster Health in the left navigation. This lists every check across all three entity types with a PASS/FAIL indicator and the additional details a failure returned: error messages, affected instances, or the relevant metric. You can filter by instance or by table.
- Table Health Dashboard — the per-table view, with email alerting and one-click remediation. It surfaces a subset of the table-level checks; that page covers which ones and how to change it.
How to read a check
Checks are evaluated periodically by theClusterHealthCheckTask, which runs every 20 minutes by default — the interval is set by Pinot’s controller.cluster.healthCheck.frequencyPeriod, and setting it to -1 disables periodic evaluation entirely. Results are cached in memory and overwritten on each run, so both views show the last evaluation — with a Last Checked timestamp — not a live probe. Three things follow from that:
- A failure can already be fixed. If you have just changed something, refresh or trigger a new run before acting.
- Not every failure is urgent. Several checks are advisory: they flag a configuration that will bite you during an upgrade or a node loss, not something broken now. Each entry below says which kind it is.
- Only a subset appears on the Table Health Dashboard. The others still run and are still visible on Cluster Health and through the APIs below.
pass boolean plus whatever diagnostic fields are listed below. In the Data Portal, simple fields are shown inline and nested ones open in a JSON viewer.
Running checks on demand
For immediate results, or to wire checks into a script, call the Controller APIs directly. Run the checks now, bypassing the 20-minute interval. Returns confirmation that the task was queued or executed:Table-level checks
Eighteen checks run per table, grouped below by what they are about. These are the checks the Table Health Dashboard draws from. Remediable. These two can be fixed from the Table Health Dashboard without leaving the page; see remediating failing checks for the buttons and the permissions they need.TABLE_SEGMENT_ASSIGNMENT_CHECK
Verifies: whether the table’s segments are optimally assigned across servers, by running a rebalance dry run and inspecting the result.
Fails when: the dry run reports that segments would move — the current assignment is no longer what a rebalance would produce.
What to do: rebalance the table — the Table Health Dashboard can do it in one click.
numSegmentsToMove is the size of the job: a large number means a long-running rebalance with real network and disk cost, so schedule it rather than firing it during a peak. This check commonly fails after servers are added or removed, after a tenant change, or after tier configuration changes.
Severity: advisory unless query latency is skewed. Unbalanced assignment costs you performance and headroom, not correctness.
TABLE_SEGMENTS_RELOAD_CHECK
Verifies: whether any segment needs reloading, by asking every server hosting the table whether its segments are consistent with the current table and schema config.
Fails when: at least one server reports a segment needing reload.
What to do: reload the table’s segments — again one click from the Table Health Dashboard. This check fails whenever you change a schema or index configuration — existing segments keep the old layout until reloaded, which is exactly why a newly added index appears to have no effect on query performance. Reloading is the step that applies it.
Severity: act on it. Until you reload, the table’s data does not match its configuration.
Segment health and availability.
IDEAL_STATE_EV_MISMATCH_CHECK
Verifies: that every segment’s actual state (ExternalView) matches its intended state (IdealState).
Fails when: at least one segment’s states disagree.
What to do: a mismatch means Helix wants a segment somewhere it has not arrived. Transient mismatches during a rebalance, a restart, or a fresh segment push are normal and clear on their own — re-run the check before treating it as a problem. Persistent mismatches usually mean a server cannot load the segment: check the server’s logs for that segment name (see Accessing logs) and whether the server is under resource pressure. Segments stuck in error state will also make queries return incomplete results.
Severity: act on it if it persists. This is the check that correlates with unavailable-segment query errors.
SEGMENT_COUNT_CHECK
Verifies: the table’s total segment count.
Fails when: the table has more than 50,000 segments.
What to do: high segment counts inflate the metadata every query must consider, grow ZooKeeper state, and slow every cluster operation. Usually the cause is segments that are too small — check
SEGMENT_SIZE_CHECK alongside this one, review your segment threshold configuration, and consider merging small segments (real-time, offline) for older data. See storage efficiency for the full set of levers.
Severity: advisory, but the trend matters more than the value. A table crossing this threshold while still growing needs attention before it doubles.
SEGMENT_SIZE_CHECK
Verifies: the distribution of segment sizes.
Fails when: more than a quarter of the table’s segments are smaller than 5 MB.
What to do: many small segments means more metadata, more per-segment query overhead, and worse compression than the same data in fewer segments. For real-time tables, raise the flush threshold so segments are sealed at a larger size; for offline tables, adjust the push granularity or roll up older data. See Configuring the segment threshold.
Severity: advisory. It is a performance and cost signal, not a failure.
SEGMENT_RETENTION_CHECK
Verifies: that the table has segment retention configured.
Fails when: no retention is configured.
Returns only pass.
What to do: without retention, the table grows without bound and will eventually hit its storage quota. Set a retention period appropriate to the data. If unbounded retention is genuinely intended, this check will keep failing; treat it as an acknowledged exception rather than something to fix.
Severity: advisory, but it is the leading indicator of a storage-quota incident months later.
SEGMENTS_TIME_PARTITION_CHECK
Verifies: that segments are cleanly partitioned by time — sorted by start and end time, no segment ends before its predecessor.
Fails when: segment time ranges overlap.
What to do: overlapping time ranges defeat time-based segment pruning, so time-filtered queries scan more segments than they need to. It commonly follows a backfill that wrote data into a range already covered by existing segments. Review how the overlapping segments were produced; see Backfill.
Severity: advisory. It shows up as queries slower than the time filter suggests they should be.
Configuration and schema.
REPLICATION_CHECK
Verifies: the table’s configured replication factor against the cluster’s threshold.
Fails when: replication is below the threshold — 3 unless your cluster configures otherwise.
What to do: replication below 2 means a single server loss or a rolling upgrade can make part of the table unavailable or return inconsistent results. Raising replication increases storage and memory proportionally, so it is a deliberate trade-off: development tables at 1 are reasonable, production tables generally are not.
Severity: advisory now, decisive during an incident.
TABLE_COLUMN_COUNT_CHECK
Verifies: the number of columns in the schema.
Fails when: the schema has more than 500 columns.
What to do: every column carries per-segment metadata and index structures, so wide schemas cost memory and slow segment load even for columns nobody queries. Look for columns that are never referenced, and for semi-structured data that would be better held in a JSON or map column than flattened into hundreds of physical ones. See JSON index.
Severity: advisory.
TIME_COLUMN_GRANULARITY_CHECK
Verifies: the granularity of the table’s time columns.
Fails when: a time column is defined at MILLISECONDS, MICROSECONDS, or NANOSECONDS granularity.
What to do: granularity finer than your queries need makes every distinct timestamp a distinct value, which hurts segment pruning and dictionary efficiency. If queries filter and group by minute or hour, a millisecond primary time column earns nothing. Consider a coarser time column for partitioning and retention, with the precise timestamp kept as an ordinary column, and add a timestamp index for the granularities you actually query.
Severity: advisory.
BROKER_RESOURCE_CHECK
Verifies: that the table’s broker resource includes brokers from its tenant.
Fails when: none of the tenant’s brokers are present in the table’s broker resource.
What to do: this is a routing problem — queries against the table have no broker able to serve them, which surfaces as
BrokerNoServerFoundAlert or table-not-found style errors. It usually follows a tenant rename or a broker tenant change. Confirm the table’s tenant configuration matches an existing broker tenant, and open a ticket if it does and the check still fails.
Severity: act on it. The table may be unqueryable.
INSTANCE_POOLS_N_REPLICA_GROUPS_CHECK
Verifies: that the table uses instance pools and replica groups.
Fails when: it does not.
Returns only pass.
What to do: pools and replica groups isolate failures and make rolling operations safe, and they are a prerequisite for scheduled server scaling. Adopting them on an existing table changes segment assignment, so plan for a rebalance.
Severity: advisory.
KAFKA_CHECK
Verifies: for real-time tables ingesting from Kafka, that the topic’s retention.ms is more than twice the table’s real-time segment flush threshold — enough headroom to re-consume after downtime without the data having aged out.
Fails when: topic retention is at or below twice the flush threshold. Unlimited retention (-1) always passes.
What to do: either raise topic retention or lower the flush threshold. Without headroom, a consumer outage longer than retention means the missing records are unrecoverable — the stream has already discarded them. This is one of the few checks whose fix is on the Kafka side rather than in Pinot.
Severity: advisory until you have an outage, at which point it determines whether you lose data.
Skew and distribution.
Skew checks compare one server against the average for the table. They matter because a single overloaded server sets the latency of every query that touches it, however healthy the rest of the cluster is.
SEGMENT_SKEW_HEALTH_CHECK
Verifies: segment distribution across the servers hosting the table. Only evaluated for tables with more than 50 segments.
Fails when: any server holds more than 50% above the average segment count for that table.
What to do: a rebalance is the usual answer — check
TABLE_SEGMENT_ASSIGNMENT_CHECK, which will typically be failing too. If it passes while this fails, the assignment strategy itself is producing the skew rather than drift from it, so review the table’s instance assignment configuration.
Severity: advisory; a strong hint when investigating latency that varies by query.
CONSUMING_PARTITION_SKEW_HEALTH_CHECK
Verifies: distribution of consuming segments — the real-time ingestion load — across the table’s servers. Only evaluated for tables with more than 10 consuming segments.
Fails when: any server holds more than 50% above the average consuming-segment count.
What to do: consuming segments cost far more than completed ones — memory for the in-flight segment, CPU for decode and index building, and stream connections. A server carrying a disproportionate share becomes the table’s ingestion bottleneck and will be the first to lag. Rebalance including consuming segments, and check whether stream partition count and server count are compatible.
Severity: act on it if the table is also lagging. This is a common root cause behind ingestion delay on one replica.
Upsert and dedup tables.
UPSERT_TABLE_SEGMENT_ASSIGNMENT_CHECK
Verifies: for an upsert table, that all segments of a partition within a replica group live on one server — the invariant upsert correctness depends on.
Fails when: a partition’s segments for a single replica group are spread across more than one server.
What to do: this is a correctness concern, not a performance one — upsert resolution needs every record for a primary key co-located. Do not rebalance an upsert table ad hoc to fix it; open a support ticket with the partition ID and the instance breakdown.
Severity: act on it. Query results may be wrong.
HIGH_PK_PER_PARTITION_CHECK
Verifies: primary-key count per partition on an instance.
Fails when: a partition holds more than 500 million primary keys.
What to do: upsert metadata is held per primary key and grows with distinct keys rather than with rows, so a table that was comfortable at launch can exhaust memory much later with no configuration change. Options, in rough order of preference: increase the stream’s partition count so keys spread further, move the table to off-heap upsert, or shorten the metadata TTL if your key space has a natural lifetime.
Severity: act on it. This is the check that precedes an out-of-memory server.
PK_SKEW_PER_PARTITION_CHECK
Verifies: how evenly primary keys are distributed across a table’s partitions.
Fails when: the skew falls outside the range −2 to 2.
What to do: skewed keys mean one partition — and so one server — carries a disproportionate share of upsert metadata, and it will hit memory limits while the others look fine. The cause is upstream: the partition key does not distribute evenly. Check for a dominant value (a default, a null placeholder, one high-volume tenant) in the key.
Severity: advisory, and the explanation for why one server out of many keeps running out of memory.
Instance-level checks
These evaluate a single server, broker, or controller rather than a table, so they do not appear on the Table Health Dashboard. They are reported on Cluster Health and through the APIs above.Cluster-level checks
These evaluate the cluster as a whole.Related
- Table Health Dashboard & Alerting — the per-table view of these checks, with email alerting and one-click remediation
- Troubleshooting: start here — when a check fails and you need a diagnostic path
- Scheduled Server Scaling —
INSTANCE_POOLS_CHECKandINSTANCE_POOLS_N_REPLICA_GROUPS_CHECKcover the same pool and replica-group setup that scheduled scale-down requires - Storage efficiency — for the segment-count and segment-size checks