> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Health checks

> The Cluster Health dashboard and every health check StarTree Cloud runs: what each verifies, the threshold that fails it, the diagnostic fields it returns, and what to do about a failure.

StarTree Cloud continuously checks your Apache Pinot cluster and reports each check as pass or fail. Checks run against one of three entity types — a **table**, an **instance** (a single server, broker, or controller), or the **cluster** as a whole. This page is both the dashboard guide and the full reference: how to reach the results, how to run checks on demand, and what every individual check verifies.

## Where results appear

Two views surface these checks, and they differ in scope rather than in what they measure:

* **Cluster Health** — log in to the Data Portal and click **Cluster Health** in the left navigation. This lists every check across all three entity types with a **PASS**/**FAIL** indicator and the additional details a failure returned: error messages, affected instances, or the relevant metric. You can filter by instance or by table.
* **[Table Health Dashboard](/corecapabilities/observability/table-health-and-alerting)** — the per-table view, with email alerting and one-click remediation. It surfaces a subset of the table-level checks; that page covers which ones and how to change it.

## How to read a check

Checks are evaluated periodically by the `ClusterHealthCheckTask`, which runs every **20 minutes** by default — the interval is set by Pinot's `controller.cluster.healthCheck.frequencyPeriod`, and setting it to `-1` disables periodic evaluation entirely. Results are cached in memory and overwritten on each run, so both views show the last evaluation — with a **Last Checked** timestamp — not a live probe. Three things follow from that:

* **A failure can already be fixed.** If you have just changed something, refresh or trigger a new run before acting.
* **Not every failure is urgent.** Several checks are advisory: they flag a configuration that will bite you during an upgrade or a node loss, not something broken now. Each entry below says which kind it is.
* **Only a subset appears on the Table Health Dashboard.** The others still run and are still visible on Cluster Health and through the APIs below.

Every check returns a `pass` boolean plus whatever diagnostic fields are listed below. In the Data Portal, simple fields are shown inline and nested ones open in a JSON viewer.

## Running checks on demand

For immediate results, or to wire checks into a script, call the Controller APIs directly.

**Run the checks now**, bypassing the 20-minute interval. Returns confirmation that the task was queued or executed:

```bash theme={null}
curl -X GET "https://<your-controller-host>/periodictask/run?taskName=ClusterHealthCheckTask"
```

**Fetch the latest report** — the cached pass/fail status and details of every check, the same data the dashboard renders:

```bash theme={null}
curl -X GET "https://<your-controller-host>/clusterHealth" | json_pp
```

**List the check types** the cluster monitors, with brief descriptions — useful for confirming what your version actually runs:

```bash theme={null}
curl -X GET "https://<your-controller-host>/clusterHealth/list" | json_pp
```

## Table-level checks

Eighteen checks run per table, grouped below by what they are about. These are the checks the Table Health Dashboard draws from.

**Remediable.** These two can be fixed from the Table Health Dashboard without leaving the page; see [remediating failing checks](/corecapabilities/observability/table-health-and-alerting#remediating-failing-checks) for the buttons and the permissions they need.

### `TABLE_SEGMENT_ASSIGNMENT_CHECK`

**Verifies:** whether the table's segments are optimally assigned across servers, by running a rebalance dry run and inspecting the result.

**Fails when:** the dry run reports that segments would move — the current assignment is no longer what a rebalance would produce.

| Field                 | Meaning                                      |
| --------------------- | -------------------------------------------- |
| `balanced`            | Whether the assignment is already balanced   |
| `numSegments`         | Total segments considered                    |
| `numSegmentsToMove`   | How many segments a rebalance would relocate |
| `numSegmentsToDelete` | How many replicas a rebalance would remove   |
| `message`             | The dry run's own summary                    |

**What to do:** rebalance the table — the Table Health Dashboard can [do it in one click](/corecapabilities/observability/table-health-and-alerting#remediating-failing-checks). `numSegmentsToMove` is the size of the job: a large number means a long-running rebalance with real network and disk cost, so schedule it rather than firing it during a peak. This check commonly fails after servers are added or removed, after a tenant change, or after tier configuration changes.

**Severity:** advisory unless query latency is skewed. Unbalanced assignment costs you performance and headroom, not correctness.

### `TABLE_SEGMENTS_RELOAD_CHECK`

**Verifies:** whether any segment needs reloading, by asking every server hosting the table whether its segments are consistent with the current table and schema config.

**Fails when:** at least one server reports a segment needing reload.

| Field                     | Meaning                                          |
| ------------------------- | ------------------------------------------------ |
| `tableSegmentsConsistent` | `false` when at least one segment needs a reload |

**What to do:** reload the table's segments — again [one click](/corecapabilities/observability/table-health-and-alerting#remediating-failing-checks) from the Table Health Dashboard. This check fails whenever you change a schema or index configuration — existing segments keep the old layout until reloaded, which is exactly why a newly added index appears to have no effect on query performance. Reloading is the step that applies it.

**Severity:** act on it. Until you reload, the table's data does not match its configuration.

**Segment health and availability.**

### `IDEAL_STATE_EV_MISMATCH_CHECK`

**Verifies:** that every segment's actual state (ExternalView) matches its intended state (IdealState).

**Fails when:** at least one segment's states disagree.

| Field                      | Meaning                    |
| -------------------------- | -------------------------- |
| `numberOfSegmentsMismatch` | How many segments disagree |
| `mismatchSegmentsName`     | The specific segment names |

**What to do:** a mismatch means Helix wants a segment somewhere it has not arrived. Transient mismatches during a rebalance, a restart, or a fresh segment push are normal and clear on their own — re-run the check before treating it as a problem. Persistent mismatches usually mean a server cannot load the segment: check the server's logs for that segment name (see [Accessing logs](/corecapabilities/observability/access-logs)) and whether the server is under [resource pressure](/corecapabilities/cluster-operations/troubleshooting). Segments stuck in error state will also make queries return incomplete results.

**Severity:** act on it if it persists. This is the check that correlates with unavailable-segment query errors.

### `SEGMENT_COUNT_CHECK`

**Verifies:** the table's total segment count.

**Fails when:** the table has more than 50,000 segments.

| Field              | Meaning       |
| ------------------ | ------------- |
| `numberOfSegments` | Current count |

**What to do:** high segment counts inflate the metadata every query must consider, grow ZooKeeper state, and slow every cluster operation. Usually the cause is segments that are too small — check [`SEGMENT_SIZE_CHECK`](#segment_size_check) alongside this one, review your [segment threshold configuration](/recipes/configuring-segment-threshold), and consider merging small segments ([real-time](/recipes/merge-segments-realtime), [offline](/recipes/merge-small-segments)) for older data. See [storage efficiency](/corecapabilities/manage-data/storage-efficiency) for the full set of levers.

**Severity:** advisory, but the trend matters more than the value. A table crossing this threshold while still growing needs attention before it doubles.

### `SEGMENT_SIZE_CHECK`

**Verifies:** the distribution of segment sizes.

**Fails when:** more than a quarter of the table's segments are smaller than 5 MB.

| Field                | Meaning                       |
| -------------------- | ----------------------------- |
| `smallSegmentsCount` | Segments below the size floor |
| `totalSegmentsCount` | Total segments                |
| `smallSegmentsName`  | Which segments are small      |

**What to do:** many small segments means more metadata, more per-segment query overhead, and worse compression than the same data in fewer segments. For real-time tables, raise the flush threshold so segments are sealed at a larger size; for offline tables, adjust the push granularity or roll up older data. See [Configuring the segment threshold](/recipes/configuring-segment-threshold).

**Severity:** advisory. It is a performance and cost signal, not a failure.

### `SEGMENT_RETENTION_CHECK`

**Verifies:** that the table has segment retention configured.

**Fails when:** no retention is configured.

Returns only `pass`.

**What to do:** without retention, the table grows without bound and will eventually hit its storage quota. Set a retention period appropriate to the data. If unbounded retention is genuinely intended, this check will keep failing; treat it as an acknowledged exception rather than something to fix.

**Severity:** advisory, but it is the leading indicator of a storage-quota incident months later.

### `SEGMENTS_TIME_PARTITION_CHECK`

**Verifies:** that segments are cleanly partitioned by time — sorted by start and end time, no segment ends before its predecessor.

**Fails when:** segment time ranges overlap.

| Field                | Meaning                  |
| -------------------- | ------------------------ |
| `overlappedSegments` | The overlapping segments |

**What to do:** overlapping time ranges defeat time-based segment pruning, so time-filtered queries scan more segments than they need to. It commonly follows a backfill that wrote data into a range already covered by existing segments. Review how the overlapping segments were produced; see [Backfill](/recipes/backfill).

**Severity:** advisory. It shows up as queries slower than the time filter suggests they should be.

**Configuration and schema.**

### `REPLICATION_CHECK`

**Verifies:** the table's configured replication factor against the cluster's threshold.

**Fails when:** replication is below the threshold — 3 unless your cluster configures otherwise.

| Field                       | Meaning                              |
| --------------------------- | ------------------------------------ |
| `tableReplication`          | The table's configured replication   |
| `tableReplicationThreshold` | The threshold being compared against |

**What to do:** replication below 2 means a single server loss or a rolling upgrade can make part of the table unavailable or return inconsistent results. Raising replication increases storage and memory proportionally, so it is a deliberate trade-off: development tables at 1 are reasonable, production tables generally are not.

**Severity:** advisory now, decisive during an incident.

### `TABLE_COLUMN_COUNT_CHECK`

**Verifies:** the number of columns in the schema.

**Fails when:** the schema has more than 500 columns.

| Field        | Meaning              |
| ------------ | -------------------- |
| `numColumns` | Current column count |

**What to do:** every column carries per-segment metadata and index structures, so wide schemas cost memory and slow segment load even for columns nobody queries. Look for columns that are never referenced, and for semi-structured data that would be better held in a JSON or map column than flattened into hundreds of physical ones. See [JSON index](/corecapabilities/manage-data/indexes/json-index).

**Severity:** advisory.

### `TIME_COLUMN_GRANULARITY_CHECK`

**Verifies:** the granularity of the table's time columns.

**Fails when:** a time column is defined at `MILLISECONDS`, `MICROSECONDS`, or `NANOSECONDS` granularity.

| Field                            | Meaning               |
| -------------------------------- | --------------------- |
| `timeColumnsWithHighGranularity` | The columns concerned |

**What to do:** granularity finer than your queries need makes every distinct timestamp a distinct value, which hurts segment pruning and dictionary efficiency. If queries filter and group by minute or hour, a millisecond primary time column earns nothing. Consider a coarser time column for partitioning and retention, with the precise timestamp kept as an ordinary column, and add a [timestamp index](/corecapabilities/manage-data/indexes/timestamp-index) for the granularities you actually query.

**Severity:** advisory.

### `BROKER_RESOURCE_CHECK`

**Verifies:** that the table's broker resource includes brokers from its tenant.

**Fails when:** none of the tenant's brokers are present in the table's broker resource.

| Field               | Meaning                     |
| ------------------- | --------------------------- |
| `missingBrokerList` | Brokers expected but absent |

**What to do:** this is a routing problem — queries against the table have no broker able to serve them, which surfaces as `BrokerNoServerFoundAlert` or table-not-found style errors. It usually follows a tenant rename or a broker tenant change. Confirm the table's tenant configuration matches an existing broker tenant, and open a ticket if it does and the check still fails.

**Severity:** act on it. The table may be unqueryable.

### `INSTANCE_POOLS_N_REPLICA_GROUPS_CHECK`

**Verifies:** that the table uses instance pools and replica groups.

**Fails when:** it does not.

Returns only `pass`.

**What to do:** pools and replica groups isolate failures and make rolling operations safe, and they are a prerequisite for [scheduled server scaling](/corecapabilities/cluster-operations/scheduled-server-scaling). Adopting them on an existing table changes segment assignment, so plan for a rebalance.

**Severity:** advisory.

### `KAFKA_CHECK`

**Verifies:** for real-time tables ingesting from Kafka, that the topic's `retention.ms` is more than twice the table's real-time segment flush threshold — enough headroom to re-consume after downtime without the data having aged out.

**Fails when:** topic retention is at or below twice the flush threshold. Unlimited retention (`-1`) always passes.

| Field                  | Meaning                                               |
| ---------------------- | ----------------------------------------------------- |
| `topic`                | The topic checked                                     |
| `retentionMillis`      | The topic's configured retention                      |
| `flushThresholdMillis` | The table's flush threshold                           |
| `details`              | Per-topic breakdown                                   |
| `errorMessage`         | Why the check could not be evaluated, when applicable |

**What to do:** either raise topic retention or lower the flush threshold. Without headroom, a consumer outage longer than retention means the missing records are unrecoverable — the stream has already discarded them. This is one of the few checks whose fix is on the Kafka side rather than in Pinot.

**Severity:** advisory until you have an outage, at which point it determines whether you lose data.

**Skew and distribution.**

Skew checks compare one server against the average for the table. They matter because a single overloaded server sets the latency of every query that touches it, however healthy the rest of the cluster is.

### `SEGMENT_SKEW_HEALTH_CHECK`

**Verifies:** segment distribution across the servers hosting the table. Only evaluated for tables with more than 50 segments.

**Fails when:** any server holds more than 50% above the average segment count for that table.

| Field                               | Meaning                        |
| ----------------------------------- | ------------------------------ |
| `numberOfSegments`                  | Segments in the table          |
| `numberOfSegmentsWithReplicas`      | Segments counted with replicas |
| `numberOfServers`                   | Servers hosting the table      |
| `expectedNumberOfSegmentsPerServer` | The even-distribution figure   |
| `serversWithHighNumberOfSegments`   | The servers over the line      |

**What to do:** a rebalance is the usual answer — check [`TABLE_SEGMENT_ASSIGNMENT_CHECK`](#table_segment_assignment_check), which will typically be failing too. If it passes while this fails, the assignment strategy itself is producing the skew rather than drift from it, so review the table's instance assignment configuration.

**Severity:** advisory; a strong hint when investigating latency that varies by query.

### `CONSUMING_PARTITION_SKEW_HEALTH_CHECK`

**Verifies:** distribution of *consuming* segments — the real-time ingestion load — across the table's servers. Only evaluated for tables with more than 10 consuming segments.

**Fails when:** any server holds more than 50% above the average consuming-segment count.

| Field                                        | Meaning                         |
| -------------------------------------------- | ------------------------------- |
| `numberOfConsumingSegments`                  | Consuming segments in the table |
| `numberOfConsumingSegmentsWithReplicas`      | Counted with replicas           |
| `numberOfServers`                            | Servers hosting the table       |
| `expectedNumberOfConsumingSegmentsPerServer` | The even-distribution figure    |
| `serversWithHighNumberOfConsumingSegments`   | The servers over the line       |

**What to do:** consuming segments cost far more than completed ones — memory for the in-flight segment, CPU for decode and index building, and stream connections. A server carrying a disproportionate share becomes the table's ingestion bottleneck and will be the first to lag. Rebalance including consuming segments, and check whether stream partition count and server count are compatible.

**Severity:** act on it if the table is also lagging. This is a common root cause behind ingestion delay on one replica.

**Upsert and dedup tables.**

### `UPSERT_TABLE_SEGMENT_ASSIGNMENT_CHECK`

**Verifies:** for an upsert table, that all segments of a partition within a replica group live on one server — the invariant upsert correctness depends on.

**Fails when:** a partition's segments for a single replica group are spread across more than one server.

| Field                          | Meaning                                  |
| ------------------------------ | ---------------------------------------- |
| `partitionId`                  | The offending partition                  |
| `hostedSegmentCount`           | Segments of that partition on the server |
| `totalSegmentCount`            | Segments in the partition                |
| `instanceSegmentCountMetadata` | Per-instance breakdown                   |

**What to do:** this is a correctness concern, not a performance one — upsert resolution needs every record for a primary key co-located. Do not rebalance an upsert table ad hoc to fix it; open a support ticket with the partition ID and the instance breakdown.

**Severity:** act on it. Query results may be wrong.

### `HIGH_PK_PER_PARTITION_CHECK`

**Verifies:** primary-key count per partition on an instance.

**Fails when:** a partition holds more than 500 million primary keys.

| Field                            | Meaning               |
| -------------------------------- | --------------------- |
| `permissiblePKCountPerPartition` | The threshold         |
| `partitionWithHighPKCount`       | The partition over it |

**What to do:** upsert metadata is held per primary key and grows with distinct keys rather than with rows, so a table that was comfortable at launch can exhaust memory much later with no configuration change. Options, in rough order of preference: increase the stream's partition count so keys spread further, move the table to [off-heap upsert](/corecapabilities/manage-data/offheap-upsert), or shorten the metadata TTL if your key space has a natural lifetime.

**Severity:** act on it. This is the check that precedes an out-of-memory server.

### `PK_SKEW_PER_PARTITION_CHECK`

**Verifies:** how evenly primary keys are distributed across a table's partitions.

**Fails when:** the skew falls outside the range −2 to 2.

| Field                                    | Meaning              |
| ---------------------------------------- | -------------------- |
| `skew`                                   | Measured skew        |
| `lowerBoundForSkew`, `upperBoundForSkew` | The acceptable range |

**What to do:** skewed keys mean one partition — and so one server — carries a disproportionate share of upsert metadata, and it will hit memory limits while the others look fine. The cause is upstream: the partition key does not distribute evenly. Check for a dominant value (a default, a null placeholder, one high-volume tenant) in the key.

**Severity:** advisory, and the explanation for why one server out of many keeps running out of memory.

## Instance-level checks

These evaluate a single server, broker, or controller rather than a table, so they do not appear on the Table Health Dashboard. They are reported on Cluster Health and through the APIs above.

| Check                                          | Description                                                                                                                                                                           |
| :--------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `INSTANCE_HEALTH_API_CHECK`                    | Checks if any of the instance `/health` API endpoints are not live, indicating that a Pinot instance (Server or Broker) is unresponsive.                                              |
| `HIGH_NUMBER_OF_SEGMENTS_CHECK`                | Checks whether the number of segments hosted on a server exceeds 10,000, which can lead to performance degradation.                                                                   |
| `HIGH_NUMBER_OF_CONSUMING_PARTITIONS_CHECK`    | Checks whether the number of consuming segments hosted on a server exceeds 50, indicating a potential bottleneck for real-time ingestion on that server.                              |
| `HIGH_NUMBER_OF_TABLES_CHECK`                  | Checks whether the number of tables hosted by a server is greater than 10, indicating a potential concentration of tables on a single server which might affect resource utilization. |
| `HIGH_PRIMARY_KEYS_CHECK`                      | Checks if the primary key count for all tables hosted by the instance exceeds 800 Million, indicating a very high primary key load on that instance.                                  |
| `HELIX_HOST_NAME_INSTANCE_NAME_MISMATCH_CHECK` | Checks if the Instance ID/Name does not match the expected value derived from the Instance Config. This can indicate a misconfiguration.                                              |

## Cluster-level checks

These evaluate the cluster as a whole.

| Check                                          | Description                                                                                                                                                                                                                                |
| :--------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `QUERY_HEALTH_CHECK`                           | Performs a simple `SELECT $docId FROM TABLE_NAME LIMIT 1` query for each table to check if it executes unsuccessfully, indicating fundamental queryability issues.                                                                         |
| `CLUSTER_LEVEL_SEGMENT_SKEW_CHECK`             | Checks whether the number of segments on any server exceeds 50% of the average segment count across all servers within the entire cluster, indicating overall segment distribution imbalance.                                              |
| `CLUSTER_LEVEL_CONSUMING_PARTITION_SKEW_CHECK` | Checks whether the number of consuming segments on any server exceeds 50% of the average consuming segment count across all servers within the entire cluster, highlighting overall real-time ingestion load imbalance.                    |
| `TABLE_SKEW_CHECK`                             | Checks whether the number of tables hosted by a server exceeds 50% of the average number of tables hosted across all servers, indicating an imbalance in table distribution.                                                               |
| `DATA_SIZE_SKEW_CHECK`                         | Checks whether the skew in the size of data hosted by instances is asymmetric (specifically if the skew is not in the range `[-2, 2]`), indicating an uneven distribution of data storage across your cluster.                             |
| `CONTROLLER_TABLES_SKEW_CHECK`                 | Checks whether the skew of table count hosted by lead controllers is asymmetric (specifically if the skew is not in the range `[-2, 2]`), indicating an uneven distribution of table management load among controllers.                    |
| `PRIMARY_KEY_SKEW_CHECK`                       | Checks whether the skew in the distribution of primary keys among servers is asymmetric (specifically if skew is not in the range `[-2, 2]`), indicating an uneven load of primary keys across servers.                                    |
| `PREBUILT_SNAPSHOT_CHECK`                      | Checks for any upsert or dedup table in the cluster that is missing prebuilt snapshot configurations, and identifies partitions without snapshots. Prebuilt snapshots can significantly improve preload performance for these table types. |
| `INSTANCE_POOLS_CHECK`                         | Checks if any server is not part of instance pools, or if any pool is not disjoint when the cluster has more than 8 servers. Proper instance pooling is crucial for fault isolation and resource management.                               |

## Related

* [Table Health Dashboard & Alerting](/corecapabilities/observability/table-health-and-alerting) — the per-table view of these checks, with email alerting and one-click remediation
* [Troubleshooting: start here](/corecapabilities/observability/troubleshooting) — when a check fails and you need a diagnostic path
* [Scheduled Server Scaling](/corecapabilities/cluster-operations/scheduled-server-scaling) — `INSTANCE_POOLS_CHECK` and `INSTANCE_POOLS_N_REPLICA_GROUPS_CHECK` cover the same pool and replica-group setup that scheduled scale-down requires
* [Storage efficiency](/corecapabilities/manage-data/storage-efficiency) — for the segment-count and segment-size checks