Support Matrix

External Tables are available starting in StarTree release 0.15.0 and must be enabled on demand — contact your StarTree representative to have the feature activated for your environment. Rows marked 0.16.0 require that release or later.

This page summarizes what External Tables support today, in one place. Legend:

✅ Supported — includes capabilities that work under documented conditions or that StarTree enables on request; conditions are noted per row
⚠️ Preview — early access, not yet generally available
❌ Not supported by StarTree
— Not applicable: the source, service, or format does not offer this capability, so there is nothing for StarTree to support

Catalogs and sources

Every catalog source is reached through a single connector, the Iceberg REST protocol (catalogType=iceberg-rest), with a serviceType selecting the catalog-specific adapter. Raw-file sources list Parquet files directly from object storage without a catalog.

Source	`catalogType` / `serviceType`	Table format	Storage	Onboarding	Since
AWS Glue	`iceberg-rest` / `glue`	Iceberg	Amazon S3	Data Portal, API	0.15.0
Amazon S3 Tables	`iceberg-rest` / `s3Tables`	Iceberg	Amazon S3	Data Portal, API	0.15.0
Unity Catalog (Databricks-managed or OSS)	`iceberg-rest` / `unity`	Iceberg; Delta with UniForm	Amazon S3 (vended credentials)	Data Portal, API	0.16.0
Apache Polaris & Snowflake Open Catalog	`iceberg-rest` / `rest`	Iceberg	Amazon S3	Data Portal, API	0.16.0
Snowflake Horizon Catalog	`iceberg-rest` / `rest`	Iceberg (Snowflake-managed)	Amazon S3	Data Portal, API	0.16.0
Generic Iceberg REST catalog (Nessie, Tabular, …)	`iceberg-rest` / `rest`	Iceberg	S3-compatible	API	0.16.0
S3 Data Lake (no catalog)	`s3`	Raw Parquet files under a prefix	Amazon S3	Data Portal, API	0.15.0
GCS Data Lake (no catalog)	`gcs-interop`	Raw Parquet files under a prefix	Google Cloud Storage (S3-interop endpoint)	Data Portal, API	0.16.0

Notes:

For Nessie, address a branch or ref by setting catalog.iceberg-rest.prefix (for example main); unset means the server default branch.
Azure Blob Storage and ADLS are not supported as sources.
CREATE TABLE over SQL DDL is available as a preview for Iceberg REST sources (serviceTypes rest, glue, s3tables); contact StarTree to enable it.

Table and file formats

Format	Status	Notes
Apache Iceberg v1	✅	Snapshot-lag `snapshotsBehind` is unavailable (`null`) on v1 metadata.
Apache Iceberg v2	✅	Full snapshot-based ingestion and lag reporting.
Iceberg v3 deletion vectors (Puffin)	⚠️ Preview	Row-level deletes and updates reflected at query time; position deletes only. Contact StarTree to enable.
Delta Lake via UniForm	✅	Through Unity Catalog only. The table must have UniForm enabled (`delta.universalFormat.enabledFormats='iceberg'`). Tables with `delta.columnMapping.mode` set to `name` or `id` — including nested struct/map/list columns and renamed time columns — are supported in current releases: physical `col-<uuid>` Parquet names are resolved through Iceberg field IDs across preview, segment generation, and null-vector building. On older 0.16.0 builds these tables failed segment load — confirm your release with StarTree support if unsure.
Delta Lake (native log)	❌	No direct Delta log reader.
Apache Hudi	❌
Apache Paimon	❌

Data files: Parquet only, for every source. CSV, JSON, Avro, and ORC data files are not supported. Recommended file size is 128 MB to 512 MB; see Operational Guidance.

Row-level deletes (Iceberg)

How each Iceberg delete mechanism is handled:

Delete mechanism	Status	Notes
Copy-on-write (deletes rewrite data files)	✅	Each sync materializes the new snapshot, so rewritten files are picked up. Append-only sources are fully handled; for heavily mutating or compacted sources, confirm current reconciliation behavior with StarTree.
Merge-on-read: Iceberg v3 deletion vectors (Puffin)	⚠️ Preview	Position deletes encoded as deletion vectors are applied at query time. Contact StarTree to enable.
Merge-on-read: Iceberg v2 position delete files	❌	Not applied during sync; rows deleted this way remain visible in query results.
Merge-on-read: Iceberg v2 equality deletes	❌	Not applied; these are commonly produced by Flink CDC writers. Use copy-on-write or v3 deletion vectors instead.

Authentication and authorization

Iceberg REST sources authenticate on two independent surfaces: the catalog surface (the REST API that serves table metadata, keys under catalog.iceberg-rest.auth.rest.*) and the storage surface (reading Parquet data files from object storage, keys under catalog.iceberg-rest.auth.storage.*). Raw S3 and GCS sources have a single combined surface (catalog.s3.* / catalog.gcs-interop.*).

Catalog (REST) surface

Method	AWS Glue	S3 Tables	Unity Catalog	Generic REST
AWS SigV4 (access key + secret + region)	✅ Required¹	✅ Required¹	—	—
AWS SigV4 via assumed IAM role (`roleArn`)	✅	✅	—	—
Bearer token / PAT	—	—	✅ Databricks PAT	✅
OAuth2 client credentials	—	—	✅ Recommended (0.16.0)	✅
Unauthenticated	—	—	✅ OSS Unity default	✅ Self-hosted default

¹ SigV4 signing is required by the service; the signing credentials can come from static keys, an assumed IAM role, or the cluster node role.

The dashes reflect the services themselves: AWS Glue and S3 Tables Iceberg REST endpoints accept only SigV4-signed requests, while Unity Catalog and spec-compliant REST catalogs use token or OAuth2 auth and do not speak SigV4.
Glue: catalog.iceberg-rest.auth.rest.service must be set to glue explicitly. It defaults to s3tables, which makes Glue reject the SigV4 signature.
Assumed IAM role on the catalog surface: set catalog.iceberg-rest.auth.rest.roleArn (plus .externalId if the trust policy requires it). catalog.iceberg-rest.auth.rest.region is required whenever a catalog-side roleArn is set; when a role is configured, static keys on that surface are ignored. The role is assumed from the cluster’s ambient identity (IRSA / instance profile). Available on the controller-executor sync path only.
OAuth2 uses the RFC 6749 client-credentials grant: oauthTokenUri, oauthClientId, oauthClientSecret, and optional oauthScopes. Tokens are cached and refreshed automatically.
Raw S3 and GCS sources have no catalog surface.

Storage (data file) surface

Method	AWS Glue	S3 Tables	Unity Catalog	Generic REST	S3 Data Lake	GCS Data Lake
Catalog-vended credentials	❌	❌	✅ Default	❌	—	—
Assumed IAM role (`roleArn` + `externalId`)	✅ Recommended	✅ Recommended	✅ Override only	✅	✅ Recommended	—
Cluster node role (AWS default chain)	✅	✅	❌	✅	✅	—
Static access keys	✅	✅	✅ Override only	✅	✅	—
GCS HMAC keys (inline)	—	—	—	—	—	✅ Required
GCS HMAC keys via GCP Secret Manager	—	—	—	—	—	✅

Assumed IAM role (storage): auth.storage.region is required whenever a storage roleArn is set — the sync fails fast without it. Static storage keys are ignored once a role is configured, and the STS AssumeRole call authenticates with the cluster’s ambient AWS identity (node role / IRSA), which the role’s trust policy must trust.
Unity vends short-lived S3 credentials per table load by default. Only catalog.iceberg-rest.auth.storage.region is needed, and it is required even with vended credentials. Static keys or an assumed role can be configured but are ignored whenever vending succeeds.
GCS accepts HMAC keys only. There is no IAM-role or workload-identity option. With keyType=SECRET, the accessKey/secretKey values are Secret Manager secret names, resolved using secretmanagertype=GCS, gcpprojectid, and gcpkeypath. Every GCS connection must also set disable.integrity.protections="true" and endpoint=https://storage.googleapis.com.
Cluster node role means omitting all credential keys so the AWS SDK default chain (instance profile, IRSA) is used.

Configuration keys by method

Method	Keys
AWS SigV4 (catalog)	`catalog.iceberg-rest.auth.rest.authType=aws-sigv4` (auto-detected), `.accessKeyId`, `.secretAccessKey`, `.region`, `.service` (`glue` or `s3tables`), `.sessionToken` (optional)
Assumed IAM role (catalog, SigV4 sources)	`catalog.iceberg-rest.auth.rest.roleArn`, `.externalId` (optional), `.region` (required with `roleArn`)
Bearer token (catalog)	`catalog.iceberg-rest.auth.rest.token`
OAuth2 (catalog)	`catalog.iceberg-rest.auth.rest.authType=oauth2`, `.oauthTokenUri`, `.oauthClientId`, `.oauthClientSecret`, `.oauthScopes` (optional)
Assumed IAM role (storage)	`catalog.iceberg-rest.auth.storage.roleArn`, `.externalId`, `.region`
Static keys (storage)	`catalog.iceberg-rest.auth.storage.accessKeyId`, `.secretAccessKey`, `.region`
S3 Data Lake	`catalog.s3.bucketName`, `.prefix`, `.region`, then one of: `.roleArn` + `.externalId`, or `.accessKey` + `.secretKey`, or nothing (node role)
GCS Data Lake	`catalog.gcs-interop.bucketName`, `.prefix`, `.region="auto"`, `.endpoint`, `.accessKey`, `.secretKey`, `disable.integrity.protections="true"`; Secret Manager mode adds `.keyType=SECRET`, `.secretmanagertype=GCS`, `.gcpprojectid`, `.gcpkeypath`

Minimum permissions per source

Source	Catalog permissions	Storage permissions
AWS Glue	`glue:GetDatabase(s)`, `glue:GetTable(s)`, `glue:GetPartitions`, `glue:GetCatalog`	`s3:GetObject`, `s3:ListBucket`
S3 Tables	`s3tables:GetTable`, `s3tables:GetNamespace`, `s3tables:GetTableData`, `s3tables:ListTables`	`s3:GetObject`, `s3:ListBucket`
Unity Catalog	`EXTERNAL USE SCHEMA` on the schema (plus catalog/schema `USE`)	vended by Unity
S3 Data Lake	—	`s3:GetObject`, `s3:ListBucket`
GCS Data Lake	—	`storage.objects.get`, `storage.objects.list`

Feature support

Each supported source has its own column. AWS Glue, S3 Tables, Unity Catalog, and generic Iceberg REST catalogs (the column that also covers Apache Polaris, Snowflake Open Catalog, and Snowflake Horizon) all connect through the same iceberg-rest connector, so they share the snapshot-driven capabilities; the Data Lake sources list files directly and have no snapshot machinery.

Feature	AWS Glue	S3 Tables	Unity Catalog	Generic Iceberg REST	S3 Data Lake	GCS Data Lake	Notes
Scheduled incremental sync	✅	✅	✅	✅	✅	✅	Quartz cron `schedule`; Data Portal default every 5 minutes.
Checkpointing (resume from watermark)	✅	✅	✅	✅	✅	✅	Watermark visible as `checkpointValue` in the status API.
Snapshot-based ingestion	✅	✅	✅	✅	—	—	Ingests snapshot deltas; `initialVersionSelector` = `LATEST`, `EARLIEST`, or `SNAPSHOT_ID`. Raw listings have no snapshot concept.
Resilience to upstream compaction / file rewrites	✅	✅	✅	✅	❌	❌	REST catalogs track the live file set via snapshot metadata, so compaction is reconciled on the next sync. Raw Data Lake sources pin segments to physical file paths — when the source compacts and deletes files, those segments fail to load (`FileNotFoundException`). Use raw sources for append-only data only. See Troubleshooting.
Backfill start cursor	—	—	—	—	✅	✅	`catalog.s3.startAfter` seeds the first listing (0.16.0).
Schema evolution (opt-in)	✅ 0.16.0	✅ 0.16.0	✅ 0.16.0	✅ 0.16.0	—	—	`schemaEvolution.enabled=true`; unions the source schema using Iceberg field-id history, which raw Parquet files do not carry.
Nested logical names under Delta column mapping (opt-in)	✅	✅	✅	✅	—	—	`storeNestedSchema.enabled=true` resolves field names inside `struct` / `array<struct>` / `map<_,struct>` columns to logical names. Relevant only for Delta UniForm tables with `delta.columnMapping.mode` set; top-level names are always resolved. See Unity Catalog — Schema options for column mapping.
Column renames	✅	✅	✅	✅	—	—	Top-level fields only, tracked as aliases so the Pinot column name is preserved; requires schema evolution enabled. Raw sources need a manual schema change.
Type widening	❌	❌	❌	❌	—	—	Rejected, not applied; `schemaEvolution.failFast` controls abort vs log-and-continue.
Time travel / snapshot pinning	✅	✅	✅	✅	—	—	Query option `snapshotVersionByTable`.
Row-level deletes and updates (deletion vectors)	⚠️ Preview	⚠️ Preview	⚠️ Preview	⚠️ Preview	—	—	Iceberg v3 Puffin position deletes only; equality deletes are not supported. Contact StarTree to enable.
Ingestion lag reporting (`includeLag`)	✅ 0.16.0	✅ 0.16.0	✅ 0.16.0	✅ 0.16.0	—	—	Status API; requires `executor=controller`. Raw sources return `lag: null` (snapshot lag is undefined for listings).
Snapshot retention (automatic cleanup)	⚠️ Preview	⚠️ Preview	⚠️ Preview	⚠️ Preview	⚠️ Preview	⚠️ Preview	Keeps the newest N snapshot versions and sweeps orphaned artifacts. Contact StarTree to enable.
Segment groups	⚠️ Preview (0.16.0)	⚠️ Preview (0.16.0)	⚠️ Preview (0.16.0)	⚠️ Preview (0.16.0)	❌	❌	Groups many small segments into one logical unit; ingestion-time grouping requires `executor=controller`. Not supported for upsert, dedup, or dimension tables.
Data and index caching	✅	✅	✅	✅	✅	✅	Parquet data cache, index cache, footer cache.
Pinot indexes	✅	✅	✅	✅	✅	✅	See Supported Indexes; every column must use `RAW` encoding. Only sorted and geospatial (H3) indexes are rejected; vector (HNSW) is supported when the index sets `storeInSegmentFile: true`.
Pause / resume sync	✅	✅	✅	✅	✅	✅	Data Portal controls or table config; already-ingested data is untouched.
Manual sync trigger	✅	✅	✅	✅	✅	✅	`POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=<table>_OFFLINE` or Data Portal Schedule Now.
Single-stage and multi-stage query engines	✅	✅	✅	✅	✅	✅
Federated queries with real-time tables	✅	✅	✅	✅	✅	✅	One query spanning Kafka-fed recent data and lake history.
Writes back to the source	❌	❌	❌	❌	❌	❌	External Tables are read-only. Pinot writes to the lake through the separate Data Export Task; see Writing to the lake below.

Writing to the lake (Data Export Task)

External Tables never write to their source. The write path in the other direction is the Data Export Task, a minion task that exports completed REALTIME segments to the lake as Parquet. It is the complement of External Table sync, used for cold-tier archival, feeding downstream Iceberg catalogs, and cross-system sharing. It requires a minion tier and is enabled on demand by StarTree.

Export destinations

Destination	`dataExport.target`	Status	Notes
Amazon S3	`filesystem`	✅	`S3PinotFS`; files written as `<outputDirURI>/<table>/<segment>.parquet`.
S3-compatible storage (MinIO, …)	`filesystem`	✅	`output.fs.prop.endpoint` override.
Google Cloud Storage	`filesystem`	✅	Native `GcsPinotFS` with a `gs://` URI; no S3-interop needed on the export path.
HDFS	`filesystem`	✅	Via PinotFS.
AWS Glue (Iceberg REST)	`iceberg-rest`	✅	Target table must already exist, must not be partitioned, and must use Parquet.
Amazon S3 Tables (Iceberg REST)	`iceberg-rest`	✅	Same target-table conditions as Glue.
Generic Iceberg REST catalog	`iceberg-rest`	✅	Same target-table conditions.

Export authentication

Destination	Methods
S3 filesystem	Static keys (`output.fs.prop.accessKey` / `.secretKey`) or assumed IAM role (`output.fs.prop.roleArn` + `.externalId`).
GCS filesystem	GCP service account JSON key (`output.fs.prop.projectId`, `.jsonKey`). Unlike ingestion, export to GCS does not use HMAC keys.
Iceberg REST catalog	Bearer token (`catalog.iceberg-rest.token`), OAuth2 client credential (`catalog.iceberg-rest.credential` as `clientId:clientSecret`), or AWS SigV4 (`catalog.iceberg-rest.rest.signing-region`). AWS Glue and S3 Tables vend short-lived S3 credentials for file I/O automatically; static `s3.*` keys are only for overriding them.

Export facts worth knowing:

Source: REALTIME tables only. sourceTableName must carry the _REALTIME suffix; External (OFFLINE) tables cannot be exported.
Output format: Parquet only, with SNAPPY (default), GZIP, ZSTD, or UNCOMPRESSED compression.
Iceberg commits are batched: staged files are committed as one Iceberg snapshot per batch (iceberg.commitThreshold, default 250 files), keeping snapshot churn low on high-throughput tables.

See Data Type Mapping for how source types map to Pinot types, Operational Guidance for sizing advice and capacity guardrails, and Troubleshooting for diagnosis guides.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Catalogs and sources

Table and file formats

Row-level deletes (Iceberg)

Authentication and authorization

Catalog (REST) surface

Storage (data file) surface

Configuration keys by method

Minimum permissions per source

Feature support

Writing to the lake (Data Export Task)

Export destinations

Export authentication

​Catalogs and sources

​Table and file formats

​Row-level deletes (Iceberg)

​Authentication and authorization

​Catalog (REST) surface

​Storage (data file) surface

​Configuration keys by method

​Minimum permissions per source

​Feature support

​Writing to the lake (Data Export Task)

​Export destinations

​Export authentication

Catalogs and sources

Table and file formats

Row-level deletes (Iceberg)

Authentication and authorization

Catalog (REST) surface

Storage (data file) surface

Configuration keys by method

Minimum permissions per source

Feature support

Writing to the lake (Data Export Task)

Export destinations

Export authentication