External Tables are available starting in StarTree release 0.15.0 and must be enabled on demand — contact your StarTree representative to have the feature activated for your environment. Rows marked 0.16.0 require that release or later.
- ✅ Supported — includes capabilities that work under documented conditions or that StarTree enables on request; conditions are noted per row
- ⚠️ Preview — early access, not yet generally available
- ❌ Not supported by StarTree
- — Not applicable: the source, service, or format does not offer this capability, so there is nothing for StarTree to support
Catalogs and sources
Every catalog source is reached through a single connector, the Iceberg REST protocol (catalogType=iceberg-rest), with a serviceType selecting the catalog-specific adapter. Raw-file sources list Parquet files directly from object storage without a catalog.
| Source | catalogType / serviceType | Table format | Storage | Onboarding | Since |
|---|
| AWS Glue | iceberg-rest / glue | Iceberg | Amazon S3 | Data Portal, API | 0.15.0 |
| Amazon S3 Tables | iceberg-rest / s3Tables | Iceberg | Amazon S3 | Data Portal, API | 0.15.0 |
| Unity Catalog (Databricks-managed or OSS) | iceberg-rest / unity | Iceberg; Delta with UniForm | Amazon S3 (vended credentials) | API | 0.16.0 |
| Generic Iceberg REST catalog (Nessie, Polaris, Tabular, …) | iceberg-rest / rest | Iceberg | S3-compatible | API | 0.16.0 |
| S3 Data Lake (no catalog) | s3 | Raw Parquet files under a prefix | Amazon S3 | Data Portal, API | 0.15.0 |
| GCS Data Lake (no catalog) | gcs-interop | Raw Parquet files under a prefix | Google Cloud Storage (S3-interop endpoint) | API | 0.16.0 |
- For Nessie, address a branch or ref by setting
catalog.iceberg-rest.prefix (for example main); unset means the server default branch.
- Azure Blob Storage and ADLS are not supported as sources.
CREATE TABLE over SQL DDL is available as a preview for Iceberg REST sources (serviceTypes rest, glue, s3tables); contact StarTree to enable it.
| Format | Status | Notes |
|---|
| Apache Iceberg v1 | ✅ | Snapshot-lag snapshotsBehind is unavailable (null) on v1 metadata. |
| Apache Iceberg v2 | ✅ | Full snapshot-based ingestion and lag reporting. |
| Iceberg v3 deletion vectors (Puffin) | ⚠️ Preview | Row-level deletes and updates reflected at query time; position deletes only. Contact StarTree to enable. |
| Delta Lake via UniForm | ✅ | Through Unity Catalog only. The table must have UniForm enabled (delta.universalFormat.enabledFormats='iceberg'). Tables with delta.columnMapping.mode set to name or id currently fail segment load; a fix is planned. |
| Delta Lake (native log) | ❌ | No direct Delta log reader. |
| Apache Hudi | ❌ | |
| Apache Paimon | ❌ | |
Row-level deletes (Iceberg)
How each Iceberg delete mechanism is handled:
| Delete mechanism | Status | Notes |
|---|
| Copy-on-write (deletes rewrite data files) | ✅ | Each sync materializes the new snapshot, so rewritten files are picked up. Append-only sources are fully handled; for heavily mutating or compacted sources, confirm current reconciliation behavior with StarTree. |
| Merge-on-read: Iceberg v3 deletion vectors (Puffin) | ⚠️ Preview | Position deletes encoded as deletion vectors are applied at query time. Contact StarTree to enable. |
| Merge-on-read: Iceberg v2 position delete files | ❌ | Not applied during sync; rows deleted this way remain visible in query results. |
| Merge-on-read: Iceberg v2 equality deletes | ❌ | Not applied; these are commonly produced by Flink CDC writers. Use copy-on-write or v3 deletion vectors instead. |
Authentication and authorization
Iceberg REST sources authenticate on two independent surfaces: the catalog surface (the REST API that serves table metadata, keys under catalog.iceberg-rest.auth.rest.*) and the storage surface (reading Parquet data files from object storage, keys under catalog.iceberg-rest.auth.storage.*). Raw S3 and GCS sources have a single combined surface (catalog.s3.* / catalog.gcs-interop.*).
Catalog (REST) surface
| Method | AWS Glue | S3 Tables | Unity Catalog | Generic REST |
|---|
| AWS SigV4 (access key + secret + region) | ✅ Required | ✅ Required | — | — |
| Bearer token / PAT | — | — | ✅ Databricks PAT | ✅ |
| OAuth2 client credentials | — | — | ✅ Recommended (0.16.0) | ✅ |
| Unauthenticated | — | — | ✅ OSS Unity default | ✅ Self-hosted default |
- The dashes reflect the services themselves: AWS Glue and S3 Tables Iceberg REST endpoints accept only SigV4-signed requests, while Unity Catalog and spec-compliant REST catalogs use token or OAuth2 auth and do not speak SigV4.
- Glue:
catalog.iceberg-rest.auth.rest.service must be set to glue explicitly. It defaults to s3tables, which makes Glue reject the SigV4 signature.
- SigV4 sources sign the REST call with static keys only; an assumed IAM role is not available on the catalog surface (it is on the storage surface).
- OAuth2 uses the RFC 6749 client-credentials grant:
oauthTokenUri, oauthClientId, oauthClientSecret, and optional oauthScopes. Tokens are cached and refreshed automatically.
- Raw S3 and GCS sources have no catalog surface.
Storage (data file) surface
| Method | AWS Glue | S3 Tables | Unity Catalog | Generic REST | S3 Data Lake | GCS Data Lake |
|---|
| Catalog-vended credentials | ❌ | ❌ | ✅ Default | ❌ | — | — |
Assumed IAM role (roleArn + externalId) | ✅ Recommended | ✅ Recommended | ✅ Override only | ✅ | ✅ Recommended | — |
| Cluster node role (AWS default chain) | ✅ | ✅ | ❌ | ✅ | ✅ | — |
| Static access keys | ✅ | ✅ | ✅ Override only | ✅ | ✅ | — |
| GCS HMAC keys (inline) | — | — | — | — | — | ✅ Required |
| GCS HMAC keys via GCP Secret Manager | — | — | — | — | — | ✅ |
- Unity vends short-lived S3 credentials per table load by default. Only
catalog.iceberg-rest.auth.storage.region is needed, and it is required even with vended credentials. Static keys or an assumed role can be configured but are ignored whenever vending succeeds.
- GCS accepts HMAC keys only. There is no IAM-role or workload-identity option. With
keyType=SECRET, the accessKey/secretKey values are Secret Manager secret names, resolved using secretmanagertype=GCS, gcpprojectid, and gcpkeypath. Every GCS connection must also set disable.integrity.protections="true" and endpoint=https://storage.googleapis.com.
- Cluster node role means omitting all credential keys so the AWS SDK default chain (instance profile, IRSA) is used.
Configuration keys by method
| Method | Keys |
|---|
| AWS SigV4 (catalog) | catalog.iceberg-rest.auth.rest.authType=aws-sigv4 (auto-detected), .accessKeyId, .secretAccessKey, .region, .service (glue or s3tables), .sessionToken (optional) |
| Bearer token (catalog) | catalog.iceberg-rest.auth.rest.token |
| OAuth2 (catalog) | catalog.iceberg-rest.auth.rest.authType=oauth2, .oauthTokenUri, .oauthClientId, .oauthClientSecret, .oauthScopes (optional) |
| Assumed IAM role (storage) | catalog.iceberg-rest.auth.storage.roleArn, .externalId, .region |
| Static keys (storage) | catalog.iceberg-rest.auth.storage.accessKeyId, .secretAccessKey, .region |
| S3 Data Lake | catalog.s3.bucketName, .prefix, .region, then one of: .roleArn + .externalId, or .accessKey + .secretKey, or nothing (node role) |
| GCS Data Lake | catalog.gcs-interop.bucketName, .prefix, .region="auto", .endpoint, .accessKey, .secretKey, disable.integrity.protections="true"; Secret Manager mode adds .keyType=SECRET, .secretmanagertype=GCS, .gcpprojectid, .gcpkeypath |
Minimum permissions per source
| Source | Catalog permissions | Storage permissions |
|---|
| AWS Glue | glue:GetDatabase(s), glue:GetTable(s), glue:GetPartitions, glue:GetCatalog | s3:GetObject, s3:ListBucket |
| S3 Tables | s3tables:GetTable, s3tables:GetNamespace, s3tables:GetTableData, s3tables:ListTables | s3:GetObject, s3:ListBucket |
| Unity Catalog | EXTERNAL USE SCHEMA on the schema (plus catalog/schema USE) | vended by Unity |
| S3 Data Lake | — | s3:GetObject, s3:ListBucket |
| GCS Data Lake | — | storage.objects.get, storage.objects.list |
Feature support
Each supported source has its own column. AWS Glue, S3 Tables, Unity Catalog, and generic Iceberg REST catalogs all connect through the same iceberg-rest connector, so they share the snapshot-driven capabilities; the Data Lake sources list files directly and have no snapshot machinery.
| Feature | AWS Glue | S3 Tables | Unity Catalog | Generic Iceberg REST | S3 Data Lake | GCS Data Lake | Notes |
|---|
| Scheduled incremental sync | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Quartz cron schedule; Data Portal default every 5 minutes. |
| Checkpointing (resume from watermark) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Watermark visible as checkpointValue in the status API. |
| Snapshot-based ingestion | ✅ | ✅ | ✅ | ✅ | — | — | Ingests snapshot deltas; initialVersionSelector = LATEST, EARLIEST, or SNAPSHOT_ID. Raw listings have no snapshot concept. |
| Backfill start cursor | — | — | — | — | ✅ | ✅ | catalog.s3.startAfter seeds the first listing (0.16.0). |
| Schema evolution (opt-in) | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | — | — | schemaEvolution.enabled=true; unions the source schema using Iceberg field-id history, which raw Parquet files do not carry. |
| Column renames | ✅ | ✅ | ✅ | ✅ | — | — | Top-level fields only, tracked as aliases so the Pinot column name is preserved; requires schema evolution enabled. Raw sources need a manual schema change. |
| Type widening | ❌ | ❌ | ❌ | ❌ | — | — | Rejected, not applied; schemaEvolution.failFast controls abort vs log-and-continue. |
| Time travel / snapshot pinning | ✅ | ✅ | ✅ | ✅ | — | — | Query option snapshotVersionByTable. |
| Row-level deletes and updates (deletion vectors) | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | — | — | Iceberg v3 Puffin position deletes only; equality deletes are not supported. Contact StarTree to enable. |
Ingestion lag reporting (includeLag) | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | — | — | Status API; requires executor=controller. Raw sources return lag: null (snapshot lag is undefined for listings). |
| Snapshot retention (automatic cleanup) | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | Keeps the newest N snapshot versions and sweeps orphaned artifacts. Contact StarTree to enable. |
| Segment groups | ⚠️ Preview (0.16.0) | ⚠️ Preview (0.16.0) | ⚠️ Preview (0.16.0) | ⚠️ Preview (0.16.0) | ❌ | ❌ | Groups many small segments into one logical unit; ingestion-time grouping requires executor=controller. Not supported for upsert, dedup, or dimension tables. |
| Data and index caching | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Parquet data cache, index cache, footer cache. |
| Pinot indexes | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | See Supported Indexes; every column must use RAW encoding, and sorted, geospatial (H3), and vector indexes are not supported. |
| Pause / resume sync | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Data Portal controls or table config; already-ingested data is untouched. |
| Manual sync trigger | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=<table>_OFFLINE or Data Portal Schedule Now. |
| Single-stage and multi-stage query engines | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| Federated queries with real-time tables | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | One query spanning Kafka-fed recent data and lake history. |
| Writes back to the source | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | External Tables are read-only. Pinot writes to the lake through the separate Data Export Task; see Writing to the lake below. |
Writing to the lake (Data Export Task)
External Tables never write to their source. The write path in the other direction is the Data Export Task, a minion task that exports completed REALTIME segments to the lake as Parquet. It is the complement of External Table sync, used for cold-tier archival, feeding downstream Iceberg catalogs, and cross-system sharing. It requires a minion tier and is enabled on demand by StarTree.
Export destinations
| Destination | dataExport.target | Status | Notes |
|---|
| Amazon S3 | filesystem | ✅ | S3PinotFS; files written as <outputDirURI>/<table>/<segment>.parquet. |
| S3-compatible storage (MinIO, …) | filesystem | ✅ | output.fs.prop.endpoint override. |
| Google Cloud Storage | filesystem | ✅ | Native GcsPinotFS with a gs:// URI; no S3-interop needed on the export path. |
| HDFS | filesystem | ✅ | Via PinotFS. |
| AWS Glue (Iceberg REST) | iceberg-rest | ✅ | Target table must already exist, must not be partitioned, and must use Parquet. |
| Amazon S3 Tables (Iceberg REST) | iceberg-rest | ✅ | Same target-table conditions as Glue. |
| Generic Iceberg REST catalog | iceberg-rest | ✅ | Same target-table conditions. |
Export authentication
| Destination | Methods |
|---|
| S3 filesystem | Static keys (output.fs.prop.accessKey / .secretKey) or assumed IAM role (output.fs.prop.roleArn + .externalId). |
| GCS filesystem | GCP service account JSON key (output.fs.prop.projectId, .jsonKey). Unlike ingestion, export to GCS does not use HMAC keys. |
| Iceberg REST catalog | Bearer token (catalog.iceberg-rest.token), OAuth2 client credential (catalog.iceberg-rest.credential as clientId:clientSecret), or AWS SigV4 (catalog.iceberg-rest.rest.signing-region). AWS Glue and S3 Tables vend short-lived S3 credentials for file I/O automatically; static s3.* keys are only for overriding them. |
- Source: REALTIME tables only.
sourceTableName must carry the _REALTIME suffix; External (OFFLINE) tables cannot be exported.
- Output format: Parquet only, with
SNAPPY (default), GZIP, ZSTD, or UNCOMPRESSED compression.
- Iceberg commits are batched: staged files are committed as one Iceberg snapshot per batch (
iceberg.commitThreshold, default 250 files), keeping snapshot churn low on high-throughput tables.
See Data Type Mapping for how source types map to Pinot types, Operational Guidance for sizing advice and capacity guardrails, and Troubleshooting for diagnosis guides.
