Large External Tables

This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it.

A large External Table is one backed by a million source files or more. This page explains why tables of that size need a different metadata layout, and how to enable segment groups to provide it. An External Table sync run creates one Pinot segment per source data file. A table backed by millions of Parquet files therefore produces millions of segments, and every segment gets its own ZooKeeper znode plus its own entry in the table’s ideal state and external view. On tables with millions of segments, such large metadata creates problems: Helix state transitions get slower and controller and broker heap grows. Segment groups make the cluster track a batch of files as one unit. A group of member segments gets a single ideal-state entry, a single external-view entry, and a single znode that controllers and brokers read, instead of one of each per file. With the default group size of 25, a one-million-file table stores 40,000 entries instead of 1,000,000.

Query results and latency are unchanged, and servers still scan exactly the segments a query needs — a server expands each group it is routed and prunes member segments individually. See Query behavior for more details.

When segment groups help

Segment groups are built for tables with a million files or more, where per-file metadata alone can destabilize the cluster. Consider them from a few hundred thousand files upwards if the table is growing towards that size. Below that, per-file segments are fine: grouping couples member availability (see Caveats) in exchange for a metadata saving you don’t need.

What’s supported

Aspect	Support
Table type	OFFLINE External Tables only.
Sync mode	Sync must run with `executor=controller` — see Catalog & mode. The minion-distributed sync path does not support grouping.
Cluster gate	Segment groups must be activated for the cluster before any table can use them. Contact StarTree support.
Not supported	Upsert, dedup, and dimension tables — the controller rejects the config for these.
Real-time tables	Not in this release. Grouping for real-time tables is planned for a future release.
Existing data	New segments will be grouped and existing segments won’t be grouped. See Enabling on a table that already has data.

Enable segment groups

Set segmentGroupConfig as a JSON string under the table’s metadata.customConfigs:

{
  "tableName": "orders_ext_OFFLINE",
  "tableType": "OFFLINE",
  "metadata": {
    "customConfigs": {
      "segmentGroupConfig": "{\"enabled\":true,\"maxSegmentsPerGroup\":25,\"timeBucket\":\"1d\"}"
    }
  }
}

Field	Default	Description
`enabled`	`false`	Opt this table into segment grouping. Rejected by controller validation unless segment groups are activated for the cluster and the table is not an upsert, dedup, or dimension table.
`maxSegmentsPerGroup`	`25`	Maximum number of member segments per group.
`minSegmentsPerGroup`	`25`	Not used by External Table grouping, but validated: must be `>= 1` and `<= maxSegmentsPerGroup`. If you lower `maxSegmentsPerGroup` below 25, lower this too or the table config is rejected.
`timeBucket`	(none)	Period (`"1h"`, `"6h"`, `"1d"`, `"7d"`) that additionally splits members by segment start time, so one group never spans more than one bucket. Improves broker-side time pruning on time-filtered queries. Accepts `d`, `h`, `m`, `s` units and combinations such as `"4h30m"`; `"1w"` is rejected. Tables without a time column bucket by segment creation time instead.

A new table is not queryable until its first snapshot completes. Grouped tables are always queried against a pinned, fully ingested snapshot (as are all Iceberg External Tables, via enableSnapshotConsistency). Until the first sync finishes, queries fail with No active Iceberg snapshots are available for table ….

Choosing a group size

Bigger groups save more metadata; smaller groups keep the cluster more granular. The trade-offs, in the order they usually matter:

Metadata saving scales directly with group size: entries drop by roughly the group size.
Availability blast radius grows with group size. A group is queryable only once every member has loaded, so one unreadable file keeps its whole group out of query results rather than just itself.
Broker pruning gets coarser. The broker prunes using the group’s aggregate time range — the minimum start and maximum end across its members — so a group spanning a wide time range is harder to prune away. Set timeBucket to bound the span if your queries filter on time.
Per-group observability — some query statistics are reported per group rather than per member (see Observability), so a larger group makes them coarser.

Start with the default of 25 and change it only with a specific reason; StarTree support can help size it for your table.

Enabling on a table that already has data

New files form groups; segments that already exist stay standalone and are never regrouped.

Enable grouping only at a snapshot boundary — never mid-snapshot. Follow the enablement sequence below.

Check whether the table is between snapshots

A table is at a snapshot boundary when its last snapshot is fully ingested and no run is in flight. Two read-only calls tell you: GET /tables/{tableNameWithType}/externalTable/checkpoint — the ingestion watermark. Read state:

`state`	Meaning	Safe to enable?
`DONE`	The last snapshot was ingested completely; `pageCursor` is `null`.	Yes
`IN_PROGRESS`	The snapshot is partly ingested; `pageCursor` marks where the last run stopped.	No — finish the snapshot first
`FAILED`	Ingestion stopped after `iceberg.snapshotProcessing.maxRetries` consecutive failures. The response’s own `errorMessage` and `consecutiveFailures` say why; `failurePhase` on `…/externalTable/status` says where.	No — resolve the failure first
all fields `null`	The table has never synced.	Yes

GET /tables/{tableNameWithType}/externalTable/status — fileOnboardingRun.status must not be RUNNING.

A run reporting COMPLETED does not mean the snapshot is complete. A run finishes after each batch of files, and a large snapshot takes many runs. state: DONE on the checkpoint is the only signal that the snapshot itself is fully ingested.

Enabling grouping on a live table

Confirm the broker prerequisite. The broker configs in Operator reference must be set.
Pause the sync schedule. In the Data Portal, use Pause Sync on the table’s details page; through the API, remove schedule from the table’s ExternalTableSyncTask config (the Data Portal stashes the old value under schedule.paused). This stops only the automatic cadence — on-demand runs still work while paused, and already-ingested data is untouched.
Let the in-flight run finish. Poll …/externalTable/status until fileOnboardingRun.status is no longer RUNNING.

Get to a snapshot boundary. Check …/externalTable/checkpoint. If state is IN_PROGRESS, trigger a run and re-check, repeating until state is DONE:

curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=orders_ext_OFFLINE"
curl "$BASE_URL/tables/orders_ext_OFFLINE/externalTable/checkpoint"

Enable grouping. Add segmentGroupConfig to the table config as shown above.
Restore the schedule. Resume Sync in the Data Portal, or put the schedule value back. The next run starts a fresh snapshot with grouping active.

Grouping data that was already ingested

If you leave it alone. Every new file on the Iceberg table forms groups, so the grouped share of the table grows with each run. The ungrouped share drains too, but only as the source table loses files: when an upstream delete or compaction removes a file, its standalone segment stops appearing in new snapshots, and snapshot retention deletes it once the last snapshot referencing it is evicted. On a table whose data ages out or gets compacted upstream, the ungrouped remainder therefore disappears on its own over time. If you need the whole table grouped now, it has to be ingested again as a new table.

How groups form

Grouping happens during the normal External Table sync run. Within a run, new member segments are bucketed by partition, tier, and (if configured) time bucket. Any bucket that has accumulated a full maxSegmentsPerGroup worth of members is committed as a group. Members left over in a bucket are carried forward to the next page of the same snapshot; whatever remains when the snapshot finalizes is committed as one smaller group per bucket. No member ever becomes a standalone segment. Group names are content-addressed (group_<table>_<minStart>_<maxEnd>_<hash>), so a group whose file set changes in a later snapshot appears under a new name and the old one is retired by snapshot retention.

Caveats

A group is available only as a whole

Servers load a group’s members together, and the group goes ONLINE only when all of them succeed. If one member fails to load, the members already loaded in that attempt are unloaded and the group goes into ERROR. Recovery is to fix the underlying file or segment and reset the group; reset works at group granularity, not per member.

Query behavior

Your queries don’t change: same SQL, same schema, same results. What changes underneath is where pruning happens. Pruning happens at two levels. The broker prunes whole groups using their aggregate metadata, which is coarser than per-file pruning. The server then expands the groups it was routed and prunes member segments individually before scanning, so a query still only reads the files it needs.

Observability

Grouping changes what shows up where. The control plane reports groups; servers report members.

Where	What you see
Ideal state, external view, controller segment listings, the table UI	One entry per group, named with a `group_` prefix so groups are recognizable at a glance. Ungrouped segments (including everything ingested before grouping was enabled) appear as usual.
A group’s ZK metadata	Group-level aggregates — summed document count and size, the widest time range across members — plus `total.segments.count` in its custom map, which is how many members the group has.
Server metrics	Counted per member: `SEGMENT_COUNT` reflects member segments, not groups.
Query statistics	Mixed. `numSegmentsQueried` counts groups (what the broker routed to the server). `numSegmentsPrunedByServer`, `numSegmentsProcessed`, `numSegmentsMatched`, and `totalDocs` count members, since the server expands each group before pruning. Pruned and processed counts are therefore legitimately much larger than queried.
Server segment-metadata endpoints	Given a group name, the response is aggregated across the group’s members.
Member segment names	Not addressable through the controller or through the single-segment metadata endpoint — a member has no standalone metadata entry, so a lookup by member name returns not-found. Query the group instead.

External Table sync status, ingestion lag, and the file-listing endpoints are unaffected — see Observability.

The mismatch to expect: a table showing 40,000 segments in the UI and numSegmentsQueried in the low thousands, while numSegmentsProcessed on the same query is in the hundreds of thousands. All three are right — the first two count groups, the last counts member segments.

Turn it off

Setting segmentGroupConfig.enabled back to false is accepted at any time, and is harmless on a table that has not formed any groups yet — it simply reverts the table to ordinary per-file segments. On a table that already has groups, the flip is not a no-op. The flag does not un-group anything; instead, the next sync run re-ingests the whole table:

The ungrouped sync path recognizes already-ingested files by their standalone segment names in the ideal state. Member segments are never in the ideal state, so every file that was inside a group is treated as new and gets a fresh standalone segment generated and uploaded. On a million-file table that is a full re-ingest, paged across many runs.
Queries stay pinned to the last completed (grouped) snapshot while this happens, so results remain correct and complete. Once the first ungrouped snapshot completes, the broker routes to the new standalone segments only, and snapshot retention deletes the old groups after their last referencing snapshot is evicted.

Disabling grouping on a grouped table costs a full re-ingest. Plan it like a new onboarding — the same download, segment-generation, and upload volume as the initial sync — and pause the schedule at a snapshot boundary first, exactly as when enabling. If the table is large, involve StarTree support before flipping the flag.

Operator reference

These are cluster- and broker-level settings, managed by StarTree in StarTree Cloud.

Property	Scope	Default	Description
`pinot.cluster.segment.groups.enabled`	Cluster config	`false`	Cluster-wide gate. Must be `true` before any table’s `segmentGroupConfig.enabled=true` is accepted; the table config is rejected with an explanatory error otherwise.
`pinot.cluster.segmentGroup.loadParallelism`	Cluster config	`100`	Maximum concurrent member operations (load, refresh, disk delete) per table on each server. Real I/O concurrency is bounded further downstream by the segment operation throttles.
`pinot.broker.data.dir`	Broker config or cluster config	(none)	Deep-store root the broker reads snapshot manifests from. Must point at the same location as the controller’s `controller.data.dir`. Required on brokers — see the warning below.
`pinot.broker.external.table.cache.dir`	Broker config	`/home/pinot/data/external-table/`	Broker-local cache directory for downloaded snapshot manifests. Must be on a writable volume.

Brokers need a deep-store root. Enabling grouping makes broker-side snapshot pruning mandatory for the table, and pruning requires the broker to read the snapshot manifest (snapshots/<tableNameWithType>/<snapshotId>/final.parquet) from deep store. The broker resolves that root from pinot.broker.data.dir — its own configuration first, then cluster config. With neither set, queries against the table fail with brokerDataDir cannot be blank; configure pinot.broker.data.dir, even though ingestion, ZooKeeper state, and the external view all look healthy.The broker listens for cluster config changes, so setting pinot.broker.data.dir in cluster config takes effect without a broker restart (the broker re-warms its snapshot cache in the background). A value in the broker’s own config always wins over cluster config. In practice the prerequisite is usually already met: every Iceberg External Table is snapshot-pinned from its first sync, so a cluster that serves Iceberg tables already has this set.

FAQs

Do I need to change how I query the table?

No. Segment groups are a metadata-layer optimization: SQL, schema, and results are unchanged.

I enabled grouping but my existing segments are still ungrouped. Why?

Because the table had already completed a sync. Files listed in that snapshot’s manifest are recognized as already ingested and keep their existing standalone segments — only files that arrive afterwards form groups. See Enabling on a table that already has data.

My group is in ERROR. What now?

A group goes ERROR when one of its members fails to load — the group is never partially served. Fix the underlying file or segment, then reset the group by its group name. Members cannot be reset individually.

Why was my table rejected when I enabled segment groups?

Either segment groups are not activated for your cluster, or the table is an upsert, dedup, or dimension table — unsupported in this release. The validation error names the reason.

Can I use segment groups on a real-time table?

Not in this release. Support for real-time tables is planned for a future release.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

When segment groups help

What’s supported

Enable segment groups

Choosing a group size

Enabling on a table that already has data

Check whether the table is between snapshots

Enabling grouping on a live table

Grouping data that was already ingested

How groups form

Caveats

A group is available only as a whole

Query behavior

Observability

Turn it off

Operator reference

FAQs

Do I need to change how I query the table?

I enabled grouping but my existing segments are still ungrouped. Why?

My group is in ERROR. What now?

Why was my table rejected when I enabled segment groups?

Can I use segment groups on a real-time table?

​When segment groups help

​What’s supported

​Enable segment groups

​Choosing a group size

​Enabling on a table that already has data

​Check whether the table is between snapshots

​Enabling grouping on a live table

​Grouping data that was already ingested

​How groups form

​Caveats

​A group is available only as a whole

​Query behavior

​Observability

​Turn it off

​Operator reference

​FAQs

​Do I need to change how I query the table?

​I enabled grouping but my existing segments are still ungrouped. Why?

​My group is in ERROR. What now?

​Why was my table rejected when I enabled segment groups?

​Can I use segment groups on a real-time table?

When segment groups help

What’s supported

Enable segment groups

Choosing a group size

Enabling on a table that already has data

Check whether the table is between snapshots

Enabling grouping on a live table

Grouping data that was already ingested

How groups form

Caveats

A group is available only as a whole

Query behavior

Observability

Turn it off

Operator reference

FAQs

Do I need to change how I query the table?

I enabled grouping but my existing segments are still ungrouped. Why?

My group is in ERROR. What now?

Why was my table rejected when I enabled segment groups?

Can I use segment groups on a real-time table?