> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Segment Groups

> Group many small External Table segments into one logical unit to cut per-file ZooKeeper and cluster metadata overhead, without changing how you query.

<Warning>
  This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it.
</Warning>

An External Table sync run creates one Pinot segment per source Parquet file. Iceberg tables with many small files can therefore produce a very large number of segments, each with its own ZooKeeper znode, ideal-state entry, and per-segment metadata — driving up cluster metadata overhead and controller/server memory even though the underlying data volume hasn't grown.

**Segment groups** let the controller and servers treat a batch of small member segments as one logical unit end-to-end, cutting per-file metadata overhead without changing how you query the table.

## How it works

* A **group** is a set of member segments (each still a real Parquet-backed segment) tracked and loaded together as one unit by the controller, broker, and servers.
* Groups are formed automatically by a periodic controller task, sized by row/segment count within a time bucket you configure.
* Once formed, a group's membership doesn't change — segment groups are **immutable** after creation.
* Iceberg-sourced External Tables can also be born directly into a group at ingestion time, skipping the standalone-segment step entirely (see [Grouping at ingestion time](#grouping-at-ingestion-time)).

## Enabling segment groups

Segment groups require an opt-in at both the cluster and table level.

### Cluster-level gate

| Property                                     | Default | Description                                                                                                                      |
| :------------------------------------------- | :------ | :------------------------------------------------------------------------------------------------------------------------------- |
| `pinot.cluster.segment.groups.enabled`       | `false` | Cluster-wide gate. Must be `true` before any table can enable segment groups.                                                    |
| `pinot.cluster.segmentGroup.loadParallelism` | `100`   | Max concurrent per-table member load/refresh/delete operations across the cluster. Read live — no restart required to change it. |

### Table-level config

Set under `tableConfig.tableCustomConfig.customConfigs.segmentGroupConfig` as a JSON string:

```json theme={null}
{
  "tableCustomConfig": {
    "customConfigs": {
      "segmentGroupConfig": "{\"enabled\":true,\"minSegmentsPerGroup\":25,\"maxSegmentsPerGroup\":25,\"timeBucket\":\"6h\",\"canRebalanceDuringGrouping\":true}"
    }
  }
}
```

| Field                        | Default  | Description                                                                                                                                                                                                   |
| :--------------------------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `enabled`                    | `false`  | Opt this table into segment grouping. Rejected by controller validation unless the cluster-level gate is also `true`.                                                                                         |
| `minSegmentsPerGroup`        | `25`     | Minimum member segments to form a group.                                                                                                                                                                      |
| `maxSegmentsPerGroup`        | `25`     | Maximum member segments per group.                                                                                                                                                                            |
| `timeBucket`                 | *(none)* | Time window (e.g. `"6h"`, `"1d"`) used to bound which segments can be grouped together.                                                                                                                       |
| `canRebalanceDuringGrouping` | `true`   | When `true`, the grouping task may rebalance member segments onto shared replicas before grouping. Set `false` to restrict grouping to segments that already share replicas — no rebalance or extra download. |

<Warning>
  Segment groups are **not supported for upsert, dedup, or dimension tables** in this release — the controller rejects `enabled=true` for these table types.
</Warning>

### Controller task tuning

Two periodic controller tasks drive grouping and lineage cleanup:

| Property                                                    | Default       | Description                                                                                      |
| :---------------------------------------------------------- | :------------ | :----------------------------------------------------------------------------------------------- |
| `controller.segment.grouping.task.frequency.in.seconds`     | `43200` (12h) | How often the grouping task runs.                                                                |
| `controller.segment.grouping.task.initial.delay.in.seconds` | `1800` (30m)  | Delay before the first run after controller startup.                                             |
| `controller.segment.grouping.task.max.groups.per.tick`      | `50`          | Max groups formed per run. Set `<= 0` to disable group formation while keeping the task running. |
| `controller.lineage.monitor.frequency.in.seconds`           | `3600` (1h)   | How often the lineage monitor reconciles segment/group lineage state.                            |
| `controller.lineage.monitor.initial.delay.in.seconds`       | `600` (10m)   | Delay before the first lineage-monitor run.                                                      |

## Grouping at ingestion time

For Iceberg External Tables synced with `executor=controller` (see [Best Practices & Configs](./best-practices-and-configs#catalog--mode)), member segments can be born directly into a group during ingestion instead of first landing as standalone segments and being grouped afterward. This path is snapshot-scoped and immutable, and is managed independently of the periodic `SegmentGroupingTask` — the grouping task explicitly skips tables using ingestion-time grouping.

<Note>
  Ingestion-time grouping only applies when the External Table Sync task is configured with `executor=controller`. The legacy minion-distributed sync path does not support it.
</Note>

## Deletion and lifecycle

A member segment's underlying Parquet file in deep store is only deleted once it has **no live standalone segment znode and is not referenced by any live group** — deleting an individual member early is unsafe if other members still reference shared state. Group lifecycle operations (load, refresh, delete) are isolated per member, so a failure loading one member doesn't fail the whole group, and re-registering an already-registered group is idempotent.

## FAQs

### Do I need to change how I query a table with segment groups enabled?

No. Segment groups are an internal metadata/storage optimization — queries, schemas, and results are unaffected.

### Can I disable segment groups after enabling them?

Set `segmentGroupConfig.enabled=false` and/or `pinot.cluster.segment.groups.enabled=false` to stop forming new groups. Existing groups are not automatically un-grouped.

### Why was my table rejected when I tried to enable segment groups?

Check that the cluster-level gate `pinot.cluster.segment.groups.enabled` is `true`, and that the table is not an upsert, dedup, or dimension table — these are unsupported in this release.