Schema Evolution

Schema evolution keeps an External Table’s Pinot schema in step with its Iceberg table. When a synced snapshot was written under a newer Iceberg schema, the sync adds new columns and maps renamed columns. Starting in StarTree Cloud 0.18.0, it can also widen int columns to long. Schema evolution is off by default and runs only on the periodic-watcher sync path. Added and renamed columns have been supported since StarTree Cloud 0.16.0. Type widening is new in 0.18.0. All API paths below are relative to your data-plane base URL. Set export BASE_URL=https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot.

Supported changes

How the sync handles each kind of source schema change (all rows assume schemaEvolution.enabled=true):

Change in the Iceberg table	Behavior
`int` → `long`	Applied when `schemaEvolution.typeWidening.enabled=true`. Rejected otherwise.
`float` → `double`	Not supported. Always rejected.
`decimal` precision increase	Not supported. The Pinot `BIG_DECIMAL` type doesn’t change, so the schema update doesn’t notice it. Files written before and after the change can be encoded differently, and reading across them isn’t validated.
Add a column	Applied. The new column is added to the Pinot schema.
Rename a column	Applied as an alias. The Pinot column keeps its original name, and the new Iceberg name is added to the column’s `aliases`. Files written under either name resolve to the same column. Only top-level columns are covered; nested-field renames aren’t resolved.
Drop a column	Rejected. The whole schema update is rejected, including any added or widened columns in the same update. This repeats on every later sync.
Single-value ↔ multi-value	Rejected.
Any other type change	Rejected. This includes changes that keep the stored type, such as `LONG` → `TIMESTAMP`.

When an update is rejected, the run either fails or keeps going with the old schema. schemaEvolution.failFast decides which.

Type widening (INT → LONG)

Type widening lets the sync follow an Iceberg int → long change instead of rejecting it. It’s the only supported widening, and it has its own switch, off by default. Before StarTree Cloud 0.18.0, or with type widening off: the sync tries to update the Pinot schema, and Pinot rejects the update because the column’s data type changed. The table keeps the old INT schema. If the same update also adds columns, those are rejected too. With type widening on: the sync updates the Pinot schema so the column is LONG, then reloads every segment of the table.

Pinot schema. The column’s dataType changes from INT to LONG. The column name stays the same.
Existing segments. No data is rewritten. An External Table segment reads its values from the source Parquet file, and the reader serves an old int32 file through the LONG path. On reload, each old segment only updates its own column metadata: the data type becomes LONG, and the stored null default becomes the schema’s LONG default.
New segments. Segments created after the schema update are LONG from the start.
Reload. The schema update starts a reload of all the table’s segments automatically. You only need to reload by hand if a segment didn’t pick up the new type (see Verify).
When it happens. The sync compares schemas when it ingests a snapshot. The widening is picked up by the first sync of a snapshot written under the new Iceberg schema. An ALTER TABLE on its own creates no snapshot, so the change lands after the next data commit to the Iceberg table.

Requirements

StarTree Cloud 0.18.0 or later for type widening. Added and renamed columns need 0.16.0 or later.
An Iceberg catalog: catalogType=iceberg-rest. Catalogs that don’t return the Iceberg schema history skip schema evolution.
The periodic-watcher sync path: executor=controller and enabled=true in the ExternalTableSyncTask config. The snapshot path (executor=controller with no enabled key, the default) never updates the schema, whatever the schemaEvolution.* keys say.
schemaEvolution.enabled=true. For type widening, also schemaEvolution.typeWidening.enabled=true; that key has no effect on its own.

The enable_schema_evolution option in CREATE EXTERNAL TABLE is accepted but has no effect. Set the task config keys below instead.

Enable or disable

All keys go in task.taskTypeConfigsMap.ExternalTableSyncTask, as strings.

Key	Default	What it does
`executor`	none	Must be `controller`.
`enabled`	absent	Must be `true` to select the periodic watcher. Absent selects the snapshot path. `false` pauses the sync; it doesn’t select the snapshot path.
`schemaEvolution.enabled`	`false`	Before each batch, updates the Pinot schema to match the Iceberg schema of the snapshot being synced. Required for widening.
`schemaEvolution.typeWidening.enabled`	`false`	Applies an `INT` → `LONG` change instead of rejecting it.
`schemaEvolution.failFast`	`false`	`true`: a rejected schema update fails the sync run. `false`: the rejection is logged and the run continues with the old schema.

With failFast=false, the run keeps ingesting files with the old schema. With true, the run fails before it creates any segments, and the status API reports failurePhase: UPDATING_SCHEMA. Because the sync re-checks the schema on every run, every following run fails the same way until you fix the cause. failFast also covers two other schema-evolution failures: the catalog metadata can’t be loaded, or the snapshot’s schema version can’t be found. To turn widening off, set schemaEvolution.typeWidening.enabled to false or remove it. Later widenings are then rejected. Turning it off doesn’t undo a widening that was already applied; the Pinot schema stays LONG. To turn off all schema evolution, set schemaEvolution.enabled to false. The sync then never changes the Pinot schema.

Example: widen INT to LONG

The Iceberg table lake.sales.orders has a column qty int. The External Table is orders_ext.

Turn on widening in the table config and apply it. Only the relevant keys are shown; keep the table’s other ExternalTableSyncTask keys as they are.

"task": {
  "taskTypeConfigsMap": {
    "ExternalTableSyncTask": {
      "catalogType": "iceberg-rest",
      "executor": "controller",
      "enabled": "true",
      "schemaEvolution.enabled": "true",
      "schemaEvolution.typeWidening.enabled": "true",
      "schemaEvolution.failFast": "true"
    }
  }
}

curl -X PUT "$BASE_URL/tables/orders_ext_OFFLINE" \
  -H "Content-Type: application/json" -d @orders_ext_table_config.json

Check the schema before the change:

curl -s "$BASE_URL/schemas/orders_ext" \
  | jq '[.dimensionFieldSpecs[]?, .metricFieldSpecs[]?] | map(select(.name=="qty"))'

[{ "name": "qty", "dataType": "INT", ... }]

Widen the column in Spark SQL, then write data so a snapshot exists under the new schema:

ALTER TABLE lake.sales.orders ALTER COLUMN qty TYPE bigint;
INSERT INTO lake.sales.orders VALUES (...);

The next sync logs this on the controller:

[<requestId>] Applying source type widening for orders_ext_OFFLINE on columns [qty] (snapshot <snapshotId>, schema-id <schemaId>). Segments reload onto the widened type; until every one has, queries spanning both generations of segments may fail or return partial rows.
[<requestId>] Updated Pinot schema for orders_ext_OFFLINE: unioned snapshot <snapshotId> (schema-id <schemaId>, forced=true)

Each server logs one line per old segment as it reloads:

Reconciled widened column types in segment <segmentName>: {column.qty.dataType=LONG, column.qty.defaultNullValue=...}

The schema now shows LONG:

curl -s "$BASE_URL/schemas/orders_ext" \
  | jq '[.dimensionFieldSpecs[]?, .metricFieldSpecs[]?] | map(select(.name=="qty"))'

[{ "name": "qty", "dataType": "LONG", ... }]

The reload starts on its own. To run it again, for example after a server was down:
curl -X POST "$BASE_URL/segments/orders_ext_OFFLINE/reload"

Query during a widening

During the widening. Until every segment has reloaded, old segments report INT and new or reloaded segments report LONG. Queries that read qty from both kinds of segment can go wrong:

ORDER BY qty, GROUP BY qty and DISTINCT qty fail.
A plain SELECT qty ... can drop the rows from segments on the other type and returns an exception in the response.
Aggregations such as SUM(qty) and COUNT(*) aren’t affected.
Queries that touch only old or only new segments aren’t affected.

Plan the widening for a quiet period, and expect the window to last as long as a full table reload. After the widening. All segments report LONG, and every query works as it did before. Values from old files come back as the same numbers, now typed LONG. Use LONG literals and functions in new queries. Check that all segments reloaded. Run a query that fails while types are mixed:

SELECT qty, COUNT(*) FROM orders_ext GROUP BY qty ORDER BY qty LIMIT 10

If it returns with no exceptions, every segment it touched is on LONG.

Indexes

The type update runs in the reload before any index is built. An index built during that reload reads the column as LONG. Columns that Pinot generated itself, such as derived or default-filled columns, keep their type until Pinot rebuilds them.

Verify

GET $BASE_URL/schemas/orders_ext shows "dataType": "LONG" for the column.
The controller log contains Applying source type widening for orders_ext_OFFLINE on columns [qty] followed by Updated Pinot schema for orders_ext_OFFLINE ... forced=true.
Each server log contains Reconciled widened column types in segment ... for the old segments.
The GROUP BY ... ORDER BY query in Query during a widening returns with no exceptions.
If a server logs Failed to reconcile widened column types for segment ..., reload the table again. Nothing changes if you reload a segment that already has the new type.

Limits

Only int → long. float → double and decimal precision changes aren’t supported.
Only the periodic-watcher path with an Iceberg catalog. Tables on the snapshot path never update their schema from the source.
The periodic-watcher path doesn’t import partition pruning, and it doesn’t turn on segment groups by default.
A widening is all or nothing per schema update. If the same update also carries a change that isn’t supported (a dropped column, a single-value/multi-value flip, another type change), nothing is applied.
The schema is built from the Iceberg schema’s columns. A column in the Pinot schema that isn’t in the Iceberg schema counts as dropped and blocks every schema update.
The widening applies only after a snapshot is written under the new Iceberg schema. A schema-only ALTER TABLE with no new data changes nothing.
Queries that sort, group or deduplicate on the widened column can fail until every segment reloads.
A widening can’t be undone from Pinot. Narrowing LONG back to INT is rejected.

Common errors

Message (as logged)	Cause	Fix
`Not applying the source type widening on columns [...] for <table>: schemaEvolution.typeWidening.enabled is false.`	The source widened a column and widening is off. The schema update is then rejected.	Set `schemaEvolution.typeWidening.enabled=true`.
`Not applying the source type widening on columns [...] for <table>: the diff also carries unsupported changes [...]`	The same update also drops a column, flips single-value/multi-value, or changes another type. The bracket lists each reason, for example `column x dropped from the union schema`.	Fix the other change. Widening is all or nothing per update.
`Schema update rejected for snapshot <id> (e.g. unsupported type widening): ...`	Pinot rejected the schema update: widening off, an unsupported type change (for example `float` → `double`), or a dropped column. With `failFast=false` the log line ends with `Continuing ingestion with the existing schema for <table> (schemaEvolution.failFast=false).`	Revert the source change, or recreate the External Table with the new type.
`Failed to load catalog metadata for schema union: ...`	The controller couldn’t read the Iceberg table metadata from the catalog.	Check catalog access and credentials. With `failFast=false` the run continues on the old schema.
`Could not resolve schema-id for snapshot <id> from catalog metadata snapshots`	The snapshot being synced has no schema version in the catalog metadata.	Check that the snapshot still exists in the catalog.
`Failed to reconcile widened column types for segment <segment>; it keeps its current column types and the reconcile is retried on the next reload`	A server couldn’t update one segment’s type during reload. The segment still loads, on `INT`.	Reload the table again: `POST $BASE_URL/segments/<table>_OFFLINE/reload`.
Schema never changes	The table is on the snapshot path, `schemaEvolution.enabled` is `false`, or no snapshot has been written since the `ALTER TABLE`.	Switch to the periodic watcher (`enabled=true`), turn on `schemaEvolution.enabled`, and commit data to the Iceberg table.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Supported changes

Type widening (INT → LONG)

Requirements

Enable or disable

Example: widen INT to LONG

Query during a widening

Indexes

Verify

Limits

Common errors

​Supported changes

​Type widening (INT → LONG)

​Requirements

​Enable or disable

​Example: widen INT to LONG

​Query during a widening

​Indexes

​Verify

​Limits

​Common errors

Supported changes

Type widening (INT → LONG)

Requirements

Enable or disable

Example: widen INT to LONG

Query during a widening

Indexes

Verify

Limits

Common errors