> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Schema Evolution

> Keep an External Table's Pinot schema in step with its Iceberg table: added columns, renamed columns, and INT to LONG type widening. How to turn it on, what happens to existing segments and queries, and what isn't supported yet.

Schema evolution keeps an External Table's Pinot schema in step with its Iceberg table. When a synced snapshot was written under a newer Iceberg schema, the sync adds new columns and maps renamed columns. Starting in StarTree Cloud 0.18.0, it can also widen `int` columns to `long`. Schema evolution is off by default and runs only on the periodic-watcher sync path.

Added and renamed columns have been supported since StarTree Cloud 0.16.0. Type widening is new in 0.18.0.

All API paths below are relative to your data-plane base URL. Set `export BASE_URL=https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot`.

## Supported changes

How the sync handles each kind of source schema change (all rows assume `schemaEvolution.enabled=true`):

| Change in the Iceberg table | Behavior |
| - | - |
| `int` → `long` | Applied when `schemaEvolution.typeWidening.enabled=true`. Rejected otherwise. |
| `float` → `double` | Not supported. Always rejected. |
| `decimal` precision increase | Not supported. The Pinot `BIG_DECIMAL` type doesn't change, so the schema update doesn't notice it. Files written before and after the change can be encoded differently, and reading across them isn't validated. |
| Add a column | Applied. The new column is added to the Pinot schema. |
| Rename a column | Applied as an alias. The Pinot column keeps its original name, and the new Iceberg name is added to the column's `aliases`. Files written under either name resolve to the same column. Only top-level columns are covered; nested-field renames aren't resolved. |
| Drop a column | Rejected. The whole schema update is rejected, including any added or widened columns in the same update. This repeats on every later sync. |
| Single-value ↔ multi-value | Rejected. |
| Any other type change | Rejected. This includes changes that keep the stored type, such as `LONG` → `TIMESTAMP`. |

When an update is rejected, the run either fails or keeps going with the old schema. `schemaEvolution.failFast` decides which.

## Type widening (INT → LONG)

Type widening lets the sync follow an Iceberg `int` → `long` change instead of rejecting it. It's the only supported widening, and it has its own switch, off by default.

**Before StarTree Cloud 0.18.0, or with type widening off:** the sync tries to update the Pinot schema, and Pinot rejects the update because the column's data type changed. The table keeps the old `INT` schema. If the same update also adds columns, those are rejected too.

**With type widening on:** the sync updates the Pinot schema so the column is `LONG`, then reloads every segment of the table.

* **Pinot schema.** The column's `dataType` changes from `INT` to `LONG`. The column name stays the same.
* **Existing segments.** No data is rewritten. An External Table segment reads its values from the source Parquet file, and the reader serves an old `int32` file through the `LONG` path. On reload, each old segment only updates its own column metadata: the data type becomes `LONG`, and the stored null default becomes the schema's `LONG` default.
* **New segments.** Segments created after the schema update are `LONG` from the start.
* **Reload.** The schema update starts a reload of all the table's segments automatically. You only need to reload by hand if a segment didn't pick up the new type (see [Verify](#verify)).
* **When it happens.** The sync compares schemas when it ingests a snapshot. The widening is picked up by the first sync of a snapshot written under the new Iceberg schema. An `ALTER TABLE` on its own creates no snapshot, so the change lands after the next data commit to the Iceberg table.

## Requirements

* StarTree Cloud 0.18.0 or later for type widening. Added and renamed columns need 0.16.0 or later.
* An Iceberg catalog: `catalogType=iceberg-rest`. Catalogs that don't return the Iceberg schema history skip schema evolution.
* The periodic-watcher sync path: `executor=controller` and `enabled=true` in the `ExternalTableSyncTask` config. The snapshot path (`executor=controller` with no `enabled` key, the default) never updates the schema, whatever the `schemaEvolution.*` keys say.
* `schemaEvolution.enabled=true`. For type widening, also `schemaEvolution.typeWidening.enabled=true`; that key has no effect on its own.

The `enable_schema_evolution` option in [`CREATE EXTERNAL TABLE`](./sql-ddl) is accepted but has no effect. Set the task config keys below instead.

## Enable or disable

All keys go in `task.taskTypeConfigsMap.ExternalTableSyncTask`, as strings.

| Key | Default | What it does |
| - | - | - |
| `executor` | none | Must be `controller`. |
| `enabled` | absent | Must be `true` to select the periodic watcher. Absent selects the snapshot path. `false` pauses the sync; it doesn't select the snapshot path. |
| `schemaEvolution.enabled` | `false` | Before each batch, updates the Pinot schema to match the Iceberg schema of the snapshot being synced. Required for widening. |
| `schemaEvolution.typeWidening.enabled` | `false` | Applies an `INT` → `LONG` change instead of rejecting it. |
| `schemaEvolution.failFast` | `false` | `true`: a rejected schema update fails the sync run. `false`: the rejection is logged and the run continues with the old schema. |

With `failFast=false`, the run keeps ingesting files with the old schema. With `true`, the run fails before it creates any segments, and the [status API](https://docs.startree.ai/corecapabilities/external-table/observability.md#run-status) reports `failurePhase: UPDATING_SCHEMA`. Because the sync re-checks the schema on every run, every following run fails the same way until you fix the cause. `failFast` also covers two other schema-evolution failures: the catalog metadata can't be loaded, or the snapshot's schema version can't be found.

To turn widening off, set `schemaEvolution.typeWidening.enabled` to `false` or remove it. Later widenings are then rejected. Turning it off doesn't undo a widening that was already applied; the Pinot schema stays `LONG`.

To turn off all schema evolution, set `schemaEvolution.enabled` to `false`. The sync then never changes the Pinot schema.

## Example: widen INT to LONG

The Iceberg table `lake.sales.orders` has a column `qty int`. The External Table is `orders_ext`.

1. Turn on widening in the table config and apply it. Only the relevant keys are shown; keep the table's other `ExternalTableSyncTask` keys as they are.

   ```json theme={null}
   "task": {
     "taskTypeConfigsMap": {
       "ExternalTableSyncTask": {
         "catalogType": "iceberg-rest",
         "executor": "controller",
         "enabled": "true",
         "schemaEvolution.enabled": "true",
         "schemaEvolution.typeWidening.enabled": "true",
         "schemaEvolution.failFast": "true"
       }
     }
   }
   ```

   ```bash theme={null}
   curl -X PUT "$BASE_URL/tables/orders_ext_OFFLINE" \
     -H "Content-Type: application/json" -d @orders_ext_table_config.json
   ```

2. Check the schema before the change:

   ```bash theme={null}
   curl -s "$BASE_URL/schemas/orders_ext" \
     | jq '[.dimensionFieldSpecs[]?, .metricFieldSpecs[]?] | map(select(.name=="qty"))'
   ```

   ```json theme={null}
   [{ "name": "qty", "dataType": "INT", ... }]
   ```

3. Widen the column in Spark SQL, then write data so a snapshot exists under the new schema:

   ```sql theme={null}
   ALTER TABLE lake.sales.orders ALTER COLUMN qty TYPE bigint;
   INSERT INTO lake.sales.orders VALUES (...);
   ```

4. The next sync logs this on the controller:

   ```text theme={null}
   [<requestId>] Applying source type widening for orders_ext_OFFLINE on columns [qty] (snapshot <snapshotId>, schema-id <schemaId>). Segments reload onto the widened type; until every one has, queries spanning both generations of segments may fail or return partial rows.
   [<requestId>] Updated Pinot schema for orders_ext_OFFLINE: unioned snapshot <snapshotId> (schema-id <schemaId>, forced=true)
   ```

   Each server logs one line per old segment as it reloads:

   ```text theme={null}
   Reconciled widened column types in segment <segmentName>: {column.qty.dataType=LONG, column.qty.defaultNullValue=...}
   ```

5. The schema now shows `LONG`:

   ```bash theme={null}
   curl -s "$BASE_URL/schemas/orders_ext" \
     | jq '[.dimensionFieldSpecs[]?, .metricFieldSpecs[]?] | map(select(.name=="qty"))'
   ```

   ```json theme={null}
   [{ "name": "qty", "dataType": "LONG", ... }]
   ```

6. The reload starts on its own. To run it again, for example after a server was down:

   ```bash theme={null}
   curl -X POST "$BASE_URL/segments/orders_ext_OFFLINE/reload"
   ```

## Query during a widening

**During the widening.** Until every segment has reloaded, old segments report `INT` and new or reloaded segments report `LONG`. Queries that read `qty` from both kinds of segment can go wrong:

* `ORDER BY qty`, `GROUP BY qty` and `DISTINCT qty` fail.
* A plain `SELECT qty ...` can drop the rows from segments on the other type and returns an exception in the response.
* Aggregations such as `SUM(qty)` and `COUNT(*)` aren't affected.
* Queries that touch only old or only new segments aren't affected.

Plan the widening for a quiet period, and expect the window to last as long as a full table reload.

**After the widening.** All segments report `LONG`, and every query works as it did before. Values from old files come back as the same numbers, now typed `LONG`. Use `LONG` literals and functions in new queries.

**Check that all segments reloaded.** Run a query that fails while types are mixed:

```sql theme={null}
SELECT qty, COUNT(*) FROM orders_ext GROUP BY qty ORDER BY qty LIMIT 10
```

If it returns with no exceptions, every segment it touched is on `LONG`.

## Indexes

The type update runs in the reload before any index is built. An index built during that reload reads the column as `LONG`. Columns that Pinot generated itself, such as derived or default-filled columns, keep their type until Pinot rebuilds them.

## Verify

1. `GET $BASE_URL/schemas/orders_ext` shows `"dataType": "LONG"` for the column.
2. The controller log contains `Applying source type widening for orders_ext_OFFLINE on columns [qty]` followed by `Updated Pinot schema for orders_ext_OFFLINE ... forced=true`.
3. Each server log contains `Reconciled widened column types in segment ...` for the old segments.
4. The `GROUP BY ... ORDER BY` query in [Query during a widening](#query-during-a-widening) returns with no exceptions.
5. If a server logs `Failed to reconcile widened column types for segment ...`, reload the table again. Nothing changes if you reload a segment that already has the new type.

## Limits

* Only `int` → `long`. `float` → `double` and `decimal` precision changes aren't supported.
* Only the periodic-watcher path with an Iceberg catalog. Tables on the snapshot path never update their schema from the source.
* The periodic-watcher path doesn't import partition pruning, and it doesn't turn on [segment groups](https://docs.startree.ai/corecapabilities/external-table/segment-groups.md) by default.
* A widening is all or nothing per schema update. If the same update also carries a change that isn't supported (a dropped column, a single-value/multi-value flip, another type change), nothing is applied.
* The schema is built from the Iceberg schema's columns. A column in the Pinot schema that isn't in the Iceberg schema counts as dropped and blocks every schema update.
* The widening applies only after a snapshot is written under the new Iceberg schema. A schema-only `ALTER TABLE` with no new data changes nothing.
* Queries that sort, group or deduplicate on the widened column can fail until every segment reloads.
* A widening can't be undone from Pinot. Narrowing `LONG` back to `INT` is rejected.

## Common errors

| Message (as logged) | Cause | Fix |
| - | - | - |
| `Not applying the source type widening on columns [...] for <table>: schemaEvolution.typeWidening.enabled is false.` | The source widened a column and widening is off. The schema update is then rejected. | Set `schemaEvolution.typeWidening.enabled=true`. |
| `Not applying the source type widening on columns [...] for <table>: the diff also carries unsupported changes [...]` | The same update also drops a column, flips single-value/multi-value, or changes another type. The bracket lists each reason, for example `column x dropped from the union schema`. | Fix the other change. Widening is all or nothing per update. |
| `Schema update rejected for snapshot <id> (e.g. unsupported type widening): ...` | Pinot rejected the schema update: widening off, an unsupported type change (for example `float` → `double`), or a dropped column. With `failFast=false` the log line ends with `Continuing ingestion with the existing schema for <table> (schemaEvolution.failFast=false).` | Revert the source change, or recreate the External Table with the new type. |
| `Failed to load catalog metadata for schema union: ...` | The controller couldn't read the Iceberg table metadata from the catalog. | Check catalog access and credentials. With `failFast=false` the run continues on the old schema. |
| `Could not resolve schema-id for snapshot <id> from catalog metadata snapshots` | The snapshot being synced has no schema version in the catalog metadata. | Check that the snapshot still exists in the catalog. |
| `Failed to reconcile widened column types for segment <segment>; it keeps its current column types and the reconcile is retried on the next reload` | A server couldn't update one segment's type during reload. The segment still loads, on `INT`. | Reload the table again: `POST $BASE_URL/segments/<table>_OFFLINE/reload`. |
| Schema never changes | The table is on the snapshot path, `schemaEvolution.enabled` is `false`, or no snapshot has been written since the `ALTER TABLE`. | Switch to the periodic watcher (`enabled=true`), turn on `schemaEvolution.enabled`, and commit data to the Iceberg table. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.