Segment Groups

This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it.

An External Table sync run creates one Pinot segment per source Parquet file. Iceberg tables with many small files can therefore produce a very large number of segments, each with its own ZooKeeper znode, ideal-state entry, and per-segment metadata — driving up cluster metadata overhead and controller/server memory even though the underlying data volume hasn’t grown. Segment groups let the controller and servers treat a batch of small member segments as one logical unit end-to-end, cutting per-file metadata overhead without changing how you query the table.

How it works

A group is a set of member segments (each still a real Parquet-backed segment) tracked and loaded together as one unit by the controller, broker, and servers.
Groups are formed automatically by a periodic controller task, sized by row/segment count within a time bucket you configure.
Once formed, a group’s membership doesn’t change — segment groups are immutable after creation.
Iceberg-sourced External Tables can also be born directly into a group at ingestion time, skipping the standalone-segment step entirely (see Grouping at ingestion time).

Enabling segment groups

Segment groups require an opt-in at both the cluster and table level.

Cluster-level gate

Property	Default	Description
`pinot.cluster.segment.groups.enabled`	`false`	Cluster-wide gate. Must be `true` before any table can enable segment groups.
`pinot.cluster.segmentGroup.loadParallelism`	`100`	Max concurrent per-table member load/refresh/delete operations across the cluster. Read live — no restart required to change it.

Table-level config

Set under tableConfig.tableCustomConfig.customConfigs.segmentGroupConfig as a JSON string:

{
  "tableCustomConfig": {
    "customConfigs": {
      "segmentGroupConfig": "{\"enabled\":true,\"minSegmentsPerGroup\":25,\"maxSegmentsPerGroup\":25,\"timeBucket\":\"6h\",\"canRebalanceDuringGrouping\":true}"
    }
  }
}

Field	Default	Description
`enabled`	`false`	Opt this table into segment grouping. Rejected by controller validation unless the cluster-level gate is also `true`.
`minSegmentsPerGroup`	`25`	Minimum member segments to form a group.
`maxSegmentsPerGroup`	`25`	Maximum member segments per group.
`timeBucket`	(none)	Time window (e.g. `"6h"`, `"1d"`) used to bound which segments can be grouped together.
`canRebalanceDuringGrouping`	`true`	When `true`, the grouping task may rebalance member segments onto shared replicas before grouping. Set `false` to restrict grouping to segments that already share replicas — no rebalance or extra download.

Segment groups are not supported for upsert, dedup, or dimension tables in this release — the controller rejects enabled=true for these table types.

Controller task tuning

Two periodic controller tasks drive grouping and lineage cleanup:

Property	Default	Description
`controller.segment.grouping.task.frequency.in.seconds`	`43200` (12h)	How often the grouping task runs.
`controller.segment.grouping.task.initial.delay.in.seconds`	`1800` (30m)	Delay before the first run after controller startup.
`controller.segment.grouping.task.max.groups.per.tick`	`50`	Max groups formed per run. Set `<= 0` to disable group formation while keeping the task running.
`controller.lineage.monitor.frequency.in.seconds`	`3600` (1h)	How often the lineage monitor reconciles segment/group lineage state.
`controller.lineage.monitor.initial.delay.in.seconds`	`600` (10m)	Delay before the first lineage-monitor run.

Grouping at ingestion time

For Iceberg External Tables synced with executor=controller (see Best Practices & Configs), member segments can be born directly into a group during ingestion instead of first landing as standalone segments and being grouped afterward. This path is snapshot-scoped and immutable, and is managed independently of the periodic SegmentGroupingTask — the grouping task explicitly skips tables using ingestion-time grouping.

Ingestion-time grouping only applies when the External Table Sync task is configured with executor=controller. The legacy minion-distributed sync path does not support it.

Deletion and lifecycle

A member segment’s underlying Parquet file in deep store is only deleted once it has no live standalone segment znode and is not referenced by any live group — deleting an individual member early is unsafe if other members still reference shared state. Group lifecycle operations (load, refresh, delete) are isolated per member, so a failure loading one member doesn’t fail the whole group, and re-registering an already-registered group is idempotent.

FAQs

Do I need to change how I query a table with segment groups enabled?

No. Segment groups are an internal metadata/storage optimization — queries, schemas, and results are unaffected.

Can I disable segment groups after enabling them?

Set segmentGroupConfig.enabled=false and/or pinot.cluster.segment.groups.enabled=false to stop forming new groups. Existing groups are not automatically un-grouped.

Why was my table rejected when I tried to enable segment groups?

Check that the cluster-level gate pinot.cluster.segment.groups.enabled is true, and that the table is not an upsert, dedup, or dimension table — these are unsupported in this release.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

How it works

Enabling segment groups

Cluster-level gate

Table-level config

Controller task tuning

Grouping at ingestion time

Deletion and lifecycle

FAQs

Do I need to change how I query a table with segment groups enabled?

Can I disable segment groups after enabling them?

Why was my table rejected when I tried to enable segment groups?

​How it works

​Enabling segment groups

​Cluster-level gate

​Table-level config

​Controller task tuning

​Grouping at ingestion time

​Deletion and lifecycle

​FAQs

​Do I need to change how I query a table with segment groups enabled?

​Can I disable segment groups after enabling them?

​Why was my table rejected when I tried to enable segment groups?

How it works

Enabling segment groups

Cluster-level gate

Table-level config

Controller task tuning

Grouping at ingestion time

Deletion and lifecycle

FAQs

Do I need to change how I query a table with segment groups enabled?

Can I disable segment groups after enabling them?

Why was my table rejected when I tried to enable segment groups?