> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Segment Count Limits

> Per-table and cluster-wide segment count guardrails that gate ingestion and task generation before segment sprawl degrades the cluster.

## Overview

Ingestion tasks create segments continuously. Without a guard, a single misconfigured table or the cluster as a whole can accumulate hundreds of thousands of segments, which slows queries through per-segment fan-out overhead and puts pressure on ZooKeeper and Helix metadata.

StarTree Cloud includes a **segment limit checker**: a controller-side circuit breaker that tracks per-table and cluster-wide segment counts and automatically blocks new ingestion work when configured thresholds are reached. It runs inside the same resource-utilization framework as the disk utilization and primary-key count checkers, and it is **enabled by default**.

This makes it easier to:

* Stop a runaway table before it degrades query latency for the whole cluster
* Cap total segment metadata growth across all tables
* Recover automatically once segment counts drop back under the thresholds — there is nothing to reset

## Key concepts

### How segment counts are collected

On the controller's periodic resource-utilization cadence (`controller.resource.utilization.checker.frequency`), the checker counts every table's segments directly from Helix ideal state and caches per-table counts plus a cluster-wide total. No server calls are involved, so the collection is cheap at any cluster size.

### Check verdicts

Whenever gated work is about to happen, the cached counts are evaluated in order:

| Condition                                               | Verdict        | Effect                                                                            |
| ------------------------------------------------------- | -------------- | --------------------------------------------------------------------------------- |
| Cached counts are stale (older than one check interval) | `UNDETERMINED` | Work is **allowed** — the checker fails open rather than blocking on missing data |
| Cluster-wide total ≥ `maxSegmentsInCluster`             | `FAIL`         | Every table is blocked                                                            |
| Table count ≥ `maxSegmentsPerTable`                     | `FAIL`         | That table is blocked                                                             |
| Otherwise                                               | `PASS`         | Work proceeds                                                                     |

### What a FAIL gates

A `FAIL` verdict gates the same operations as the other utilization checkers:

* Minion ingestion task generation
* Real-time ingestion
* Offline segment uploads
* External Table sync — the watcher consults the checker before each table and skips tables that are over the limit

Queries are unaffected. The breaker only gates the creation of new segments.

***

## Configuration

| Config key                                            | Default     | Scope                                                                                                   |
| ----------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------- |
| `controller.segmentLimitChecker.enable`               | `true`      | Controller config, read at startup. Disabling requires a controller restart — contact StarTree support. |
| `controller.segmentLimitChecker.maxSegmentsPerTable`  | `350,000`   | Controller config at startup, or cluster config at runtime.                                             |
| `controller.segmentLimitChecker.maxSegmentsInCluster` | `1,000,000` | Controller config at startup, or cluster config at runtime.                                             |

Non-positive values are rejected: at startup the checker falls back to the default, and at runtime it keeps the current value.

<Callout type="info">
  The per-table default of 350,000 matches the documented [capacity guardrail for External Tables](/corecapabilities/external-table/operational-guidance), where each source Parquet file becomes one segment.
</Callout>

***

## Tune limits at runtime

Both limit keys are watched as **cluster configs**, so they can be changed without a controller restart.

**API Reference:** [`POST /cluster/configs`](/api-reference/cluster/update-cluster-configuration) · [`GET /cluster/configs`](/api-reference/cluster/list-cluster-configurations) · [`DELETE /cluster/configs/{configName}`](/api-reference/cluster/delete-cluster-configuration)

Raise the per-table limit:

`POST /cluster/configs`

```json theme={null}
{
  "controller.segmentLimitChecker.maxSegmentsPerTable": "500000"
}
```

Verify what is active with `GET /cluster/configs`, and revert to the default by deleting the key:

`DELETE /cluster/configs/controller.segmentLimitChecker.maxSegmentsPerTable`

The controller logs every applied change and revert:

```text theme={null}
Updating 'controller.segmentLimitChecker.maxSegmentsPerTable' from 350000 to 500000
Cluster config 'controller.segmentLimitChecker.maxSegmentsPerTable' removed, reverting to default: 350000
```

***

## Monitoring and troubleshooting

**Count where a table stands** with `GET /tables/{tableName}/segments` (the segment list length), or watch the controller log for the cluster-wide picture after each periodic count:

```text theme={null}
Computed segment counts for <N> tables (<F> failed), cluster total: <T>
```

**When a limit trips**, the controller log states exactly which threshold was breached:

```text theme={null}
Table my_table_OFFLINE segment count 350000 >= limit 350000, returning FAIL
Cluster segment total 1000000 >= limit 1000000, returning FAIL for table my_table_OFFLINE
```

While the verdict stays `FAIL`, new ingestion work for the affected table (or every table, for a cluster-wide breach) is skipped on each cycle and re-evaluated on the next one.

***

## Recovering

Pick one, depending on whether the segment count is legitimate:

1. **Reduce the segment count.** Tighten retention so old segments age out, or merge small segments (see [Merging Small Segments](/recipes/merge-small-segments)). For External Tables, aim for larger source Parquet files — see the [operational guidance](/corecapabilities/external-table/operational-guidance) on file sizing.
2. **Raise the limit at runtime** with the cluster config API above, if the cluster is sized for it.
3. **Disable the checker** (`controller.segmentLimitChecker.enable=false`) — a controller-config change requiring a restart; contact StarTree support. Prefer options 1 and 2.

Once the next periodic count runs and the totals are back under the thresholds, gated work resumes automatically.