> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Support Matrix
> What StarTree External Tables support: catalogs and sources, table and file formats, authentication methods for every connector, features, indexes, and limits.
External Tables are available starting in **StarTree release 0.15.0** and must be enabled on demand — contact your StarTree representative to have the feature activated for your environment. Rows marked **0.16.0** require that release or later.
This page summarizes what External Tables support today, in one place. Legend:
* ✅ Supported — includes capabilities that work under documented conditions or that StarTree enables on request; conditions are noted per row
* ⚠️ Preview — early access, not yet generally available
* ❌ Not supported by StarTree
* — Not applicable: the source, service, or format does not offer this capability, so there is nothing for StarTree to support
***
## Catalogs and sources
Every catalog source is reached through a single connector, the Iceberg REST protocol (`catalogType=iceberg-rest`), with a `serviceType` selecting the catalog-specific adapter. Raw-file sources list Parquet files directly from object storage without a catalog.
| Source | `catalogType` / `serviceType` | Table format | Storage | Onboarding | Since |
| ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ------------------------------------------------------ | ------------------------------------------ | ---------------------------------------------------------------------------- | ------ |
| [AWS Glue](/corecapabilities/external-table/glue/onboarding-data-portal) | `iceberg-rest` / `glue` | Iceberg | Amazon S3 | Data Portal, [API](/corecapabilities/external-table/glue/onboarding-api) | 0.15.0 |
| [Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-data-portal) | `iceberg-rest` / `s3Tables` | Iceberg | Amazon S3 | Data Portal, [API](/corecapabilities/external-table/s3tables/onboarding-api) | 0.15.0 |
| [Unity Catalog](/corecapabilities/external-table/unity/onboarding-api) (Databricks-managed or OSS) | `iceberg-rest` / `unity` | Iceberg; Delta with [UniForm](#table-and-file-formats) | Amazon S3 (vended credentials) | API | 0.16.0 |
| [Generic Iceberg REST catalog](/corecapabilities/external-table/getting-started/connect-a-catalog) (Nessie, Polaris, Tabular, ...) | `iceberg-rest` / `rest` | Iceberg | S3-compatible | API | 0.16.0 |
| [S3 Data Lake](/corecapabilities/external-table/s3/onboarding-data-portal) (no catalog) | `s3` | Raw Parquet files under a prefix | Amazon S3 | Data Portal, [API](/corecapabilities/external-table/s3/onboarding-api) | 0.15.0 |
| [GCS Data Lake](/corecapabilities/external-table/gcs/onboarding-api) (no catalog) | `gcs-interop` | Raw Parquet files under a prefix | Google Cloud Storage (S3-interop endpoint) | API | 0.16.0 |
Notes:
* For Nessie, address a branch or ref by setting `catalog.iceberg-rest.prefix` (for example `main`); unset means the server default branch.
* Azure Blob Storage and ADLS are not supported as sources.
* `CREATE TABLE` over SQL DDL is available as a preview for Iceberg REST sources (serviceTypes `rest`, `glue`, `s3tables`); contact StarTree to enable it.
***
## Table and file formats
| Format | Status | Notes |
| ------------------------------------ | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Apache Iceberg v1 | ✅ | Snapshot-lag `snapshotsBehind` is unavailable (`null`) on v1 metadata. |
| Apache Iceberg v2 | ✅ | Full snapshot-based ingestion and lag reporting. |
| Iceberg v3 deletion vectors (Puffin) | ⚠️ Preview | Row-level deletes and updates reflected at query time; position deletes only. Contact StarTree to enable. |
| Delta Lake via UniForm | ✅ | Through [Unity Catalog](/corecapabilities/external-table/unity/onboarding-api) only. The table must have UniForm enabled (`delta.universalFormat.enabledFormats='iceberg'`). Tables with `delta.columnMapping.mode` set to `name` or `id` currently fail segment load; a fix is planned. |
| Delta Lake (native log) | ❌ | No direct Delta log reader. |
| Apache Hudi | ❌ | |
| Apache Paimon | ❌ | |
**Data files:** Parquet only, for every source. CSV, JSON, Avro, and ORC data files are not supported. Recommended file size is 128 MB to 512 MB; see [Operational Guidance](/corecapabilities/external-table/operational-guidance).
### Row-level deletes (Iceberg)
How each Iceberg delete mechanism is handled:
| Delete mechanism | Status | Notes |
| --------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Copy-on-write (deletes rewrite data files) | ✅ | Each sync materializes the new snapshot, so rewritten files are picked up. Append-only sources are fully handled; for heavily mutating or compacted sources, confirm current reconciliation behavior with StarTree. |
| Merge-on-read: Iceberg v3 deletion vectors (Puffin) | ⚠️ Preview | Position deletes encoded as deletion vectors are applied at query time. Contact StarTree to enable. |
| Merge-on-read: Iceberg v2 position delete files | ❌ | Not applied during sync; rows deleted this way remain visible in query results. |
| Merge-on-read: Iceberg v2 equality deletes | ❌ | Not applied; these are commonly produced by Flink CDC writers. Use copy-on-write or v3 deletion vectors instead. |
***
## Authentication and authorization
Iceberg REST sources authenticate on two independent surfaces: the **catalog surface** (the REST API that serves table metadata, keys under `catalog.iceberg-rest.auth.rest.*`) and the **storage surface** (reading Parquet data files from object storage, keys under `catalog.iceberg-rest.auth.storage.*`). Raw S3 and GCS sources have a single combined surface (`catalog.s3.*` / `catalog.gcs-interop.*`).
### Catalog (REST) surface
| Method | AWS Glue | S3 Tables | Unity Catalog | Generic REST |
| ---------------------------------------- | ---------- | ---------- | ---------------------- | --------------------- |
| AWS SigV4 (access key + secret + region) | ✅ Required | ✅ Required | — | — |
| Bearer token / PAT | — | — | ✅ Databricks PAT | ✅ |
| OAuth2 client credentials | — | — | ✅ Recommended (0.16.0) | ✅ |
| Unauthenticated | — | — | ✅ OSS Unity default | ✅ Self-hosted default |
* The dashes reflect the services themselves: AWS Glue and S3 Tables Iceberg REST endpoints accept only SigV4-signed requests, while Unity Catalog and spec-compliant REST catalogs use token or OAuth2 auth and do not speak SigV4.
* **Glue:** `catalog.iceberg-rest.auth.rest.service` must be set to `glue` explicitly. It defaults to `s3tables`, which makes Glue reject the SigV4 signature.
* **SigV4 sources** sign the REST call with static keys only; an assumed IAM role is not available on the catalog surface (it is on the storage surface).
* **OAuth2** uses the RFC 6749 client-credentials grant: `oauthTokenUri`, `oauthClientId`, `oauthClientSecret`, and optional `oauthScopes`. Tokens are cached and refreshed automatically.
* Raw S3 and GCS sources have no catalog surface.
### Storage (data file) surface
| Method | AWS Glue | S3 Tables | Unity Catalog | Generic REST | S3 Data Lake | GCS Data Lake |
| ------------------------------------------- | ------------- | ------------- | --------------- | ------------ | ------------- | ------------- |
| Catalog-vended credentials | ❌ | ❌ | ✅ Default | ❌ | — | — |
| Assumed IAM role (`roleArn` + `externalId`) | ✅ Recommended | ✅ Recommended | ✅ Override only | ✅ | ✅ Recommended | — |
| Cluster node role (AWS default chain) | ✅ | ✅ | ❌ | ✅ | ✅ | — |
| Static access keys | ✅ | ✅ | ✅ Override only | ✅ | ✅ | — |
| GCS HMAC keys (inline) | — | — | — | — | — | ✅ Required |
| GCS HMAC keys via GCP Secret Manager | — | — | — | — | — | ✅ |
* **Unity vends short-lived S3 credentials per table load by default.** Only `catalog.iceberg-rest.auth.storage.region` is needed, and it is required even with vended credentials. Static keys or an assumed role can be configured but are ignored whenever vending succeeds.
* **GCS accepts HMAC keys only.** There is no IAM-role or workload-identity option. With `keyType=SECRET`, the `accessKey`/`secretKey` values are Secret Manager secret names, resolved using `secretmanagertype=GCS`, `gcpprojectid`, and `gcpkeypath`. Every GCS connection must also set `disable.integrity.protections="true"` and `endpoint=https://storage.googleapis.com`.
* **Cluster node role** means omitting all credential keys so the AWS SDK default chain (instance profile, IRSA) is used.
### Configuration keys by method
| Method | Keys |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AWS SigV4 (catalog) | `catalog.iceberg-rest.auth.rest.authType=aws-sigv4` (auto-detected), `.accessKeyId`, `.secretAccessKey`, `.region`, `.service` (`glue` or `s3tables`), `.sessionToken` (optional) |
| Bearer token (catalog) | `catalog.iceberg-rest.auth.rest.token` |
| OAuth2 (catalog) | `catalog.iceberg-rest.auth.rest.authType=oauth2`, `.oauthTokenUri`, `.oauthClientId`, `.oauthClientSecret`, `.oauthScopes` (optional) |
| Assumed IAM role (storage) | `catalog.iceberg-rest.auth.storage.roleArn`, `.externalId`, `.region` |
| Static keys (storage) | `catalog.iceberg-rest.auth.storage.accessKeyId`, `.secretAccessKey`, `.region` |
| S3 Data Lake | `catalog.s3.bucketName`, `.prefix`, `.region`, then one of: `.roleArn` + `.externalId`, or `.accessKey` + `.secretKey`, or nothing (node role) |
| GCS Data Lake | `catalog.gcs-interop.bucketName`, `.prefix`, `.region="auto"`, `.endpoint`, `.accessKey`, `.secretKey`, `disable.integrity.protections="true"`; Secret Manager mode adds `.keyType=SECRET`, `.secretmanagertype=GCS`, `.gcpprojectid`, `.gcpkeypath` |
### Minimum permissions per source
| Source | Catalog permissions | Storage permissions |
| ------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------- |
| AWS Glue | `glue:GetDatabase(s)`, `glue:GetTable(s)`, `glue:GetPartitions`, `glue:GetCatalog` | `s3:GetObject`, `s3:ListBucket` |
| S3 Tables | `s3tables:GetTable`, `s3tables:GetNamespace`, `s3tables:GetTableData`, `s3tables:ListTables` | `s3:GetObject`, `s3:ListBucket` |
| Unity Catalog | `EXTERNAL USE SCHEMA` on the schema (plus catalog/schema `USE`) | vended by Unity |
| S3 Data Lake | — | `s3:GetObject`, `s3:ListBucket` |
| GCS Data Lake | — | `storage.objects.get`, `storage.objects.list` |
***
## Feature support
Each supported source has its own column. AWS Glue, S3 Tables, Unity Catalog, and generic Iceberg REST catalogs all connect through the same `iceberg-rest` connector, so they share the snapshot-driven capabilities; the Data Lake sources list files directly and have no snapshot machinery.
| Feature | AWS Glue | S3 Tables | Unity Catalog | Generic Iceberg REST | S3 Data Lake | GCS Data Lake | Notes |
| ------------------------------------------------ | ------------------- | ------------------- | ------------------- | -------------------- | ------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Scheduled incremental sync | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Quartz cron `schedule`; Data Portal default every 5 minutes. |
| Checkpointing (resume from watermark) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Watermark visible as `checkpointValue` in the [status API](/corecapabilities/external-table/observability#run-status). |
| Snapshot-based ingestion | ✅ | ✅ | ✅ | ✅ | — | — | Ingests snapshot deltas; `initialVersionSelector` = `LATEST`, `EARLIEST`, or `SNAPSHOT_ID`. Raw listings have no snapshot concept. |
| Backfill start cursor | — | — | — | — | ✅ | ✅ | `catalog.s3.startAfter` seeds the first listing (0.16.0). |
| Schema evolution (opt-in) | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | — | — | `schemaEvolution.enabled=true`; unions the source schema using Iceberg field-id history, which raw Parquet files do not carry. |
| Column renames | ✅ | ✅ | ✅ | ✅ | — | — | Top-level fields only, tracked as aliases so the Pinot column name is preserved; requires schema evolution enabled. Raw sources need a manual schema change. |
| Type widening | ❌ | ❌ | ❌ | ❌ | — | — | Rejected, not applied; `schemaEvolution.failFast` controls abort vs log-and-continue. |
| Time travel / snapshot pinning | ✅ | ✅ | ✅ | ✅ | — | — | Query option `snapshotVersionByTable`. |
| Row-level deletes and updates (deletion vectors) | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | — | — | Iceberg v3 Puffin position deletes only; equality deletes are not supported. Contact StarTree to enable. |
| Ingestion lag reporting (`includeLag`) | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | — | — | [Status API](/corecapabilities/external-table/observability#ingestion-lag); requires `executor=controller`. Raw sources return `lag: null` (snapshot lag is undefined for listings). |
| Snapshot retention (automatic cleanup) | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | Keeps the newest N snapshot versions and sweeps orphaned artifacts. Contact StarTree to enable. |
| Segment groups | ⚠️ Preview (0.16.0) | ⚠️ Preview (0.16.0) | ⚠️ Preview (0.16.0) | ⚠️ Preview (0.16.0) | ❌ | ❌ | Groups many small segments into one logical unit; ingestion-time grouping requires `executor=controller`. Not supported for upsert, dedup, or dimension tables. |
| Data and index caching | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [Parquet data cache, index cache, footer cache](/corecapabilities/external-table/data-and-index-caching). |
| Pinot indexes | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | See [Supported Indexes](/corecapabilities/external-table/indexes); every column must use `RAW` encoding, and sorted, geospatial (H3), and vector indexes are not supported. |
| Pause / resume sync | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Data Portal controls or table config; already-ingested data is untouched. |
| Manual sync trigger | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | `POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=_OFFLINE` or Data Portal **Schedule Now**. |
| Single-stage and multi-stage query engines | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| Federated queries with real-time tables | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | One query spanning Kafka-fed recent data and lake history. |
| Writes back to the source | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | External Tables are read-only. Pinot writes to the lake through the separate Data Export Task; see [Writing to the lake](#writing-to-the-lake-data-export-task) below. |
***
***
## Writing to the lake (Data Export Task)
External Tables never write to their source. The write path in the other direction is the [Data Export Task](/corecapabilities/external-table/data-export-task), a minion task that exports completed **REALTIME segments** to the lake as Parquet. It is the complement of External Table sync, used for cold-tier archival, feeding downstream Iceberg catalogs, and cross-system sharing. It requires a minion tier and is enabled on demand by StarTree.
### Export destinations
| Destination | `dataExport.target` | Status | Notes |
| ---------------------------------- | ------------------- | ------ | -------------------------------------------------------------------------------- |
| Amazon S3 | `filesystem` | ✅ | `S3PinotFS`; files written as `//.parquet`. |
| S3-compatible storage (MinIO, ...) | `filesystem` | ✅ | `output.fs.prop.endpoint` override. |
| Google Cloud Storage | `filesystem` | ✅ | Native `GcsPinotFS` with a `gs://` URI; no S3-interop needed on the export path. |
| HDFS | `filesystem` | ✅ | Via PinotFS. |
| AWS Glue (Iceberg REST) | `iceberg-rest` | ✅ | Target table must already exist, must not be partitioned, and must use Parquet. |
| Amazon S3 Tables (Iceberg REST) | `iceberg-rest` | ✅ | Same target-table conditions as Glue. |
| Generic Iceberg REST catalog | `iceberg-rest` | ✅ | Same target-table conditions. |
### Export authentication
| Destination | Methods |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| S3 filesystem | Static keys (`output.fs.prop.accessKey` / `.secretKey`) or assumed IAM role (`output.fs.prop.roleArn` + `.externalId`). |
| GCS filesystem | GCP service account JSON key (`output.fs.prop.projectId`, `.jsonKey`). Unlike ingestion, export to GCS does not use HMAC keys. |
| Iceberg REST catalog | Bearer token (`catalog.iceberg-rest.token`), OAuth2 client credential (`catalog.iceberg-rest.credential` as `clientId:clientSecret`), or AWS SigV4 (`catalog.iceberg-rest.rest.signing-region`). AWS Glue and S3 Tables vend short-lived S3 credentials for file I/O automatically; static `s3.*` keys are only for overriding them. |
Export facts worth knowing:
* **Source: REALTIME tables only.** `sourceTableName` must carry the `_REALTIME` suffix; External (OFFLINE) tables cannot be exported.
* **Output format: Parquet only**, with `SNAPPY` (default), `GZIP`, `ZSTD`, or `UNCOMPRESSED` compression.
* **Iceberg commits are batched**: staged files are committed as one Iceberg snapshot per batch (`iceberg.commitThreshold`, default 250 files), keeping snapshot churn low on high-throughput tables.
***
See [Data Type Mapping](/corecapabilities/external-table/data-type-mapping) for how source types map to Pinot types, [Operational Guidance](/corecapabilities/external-table/operational-guidance) for sizing advice and capacity guardrails, and [Troubleshooting](/corecapabilities/external-table/troubleshooting) for diagnosis guides.