Skip to main content
This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.

​
General

What is an External Table? A Pinot table whose data stays in Parquet files in your object store (an Iceberg catalog such as AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris, Snowflake Open Catalog / Horizon, or Nessie, or a raw S3/GCS data lake) instead of being copied into Pinot’s own segment format. Pinot reads the remote data at query time, with server-side caching to keep queries fast. Which sources are supported? S3 Data Lake (raw Parquet), GCS Data Lake (raw Parquet via S3 interop), AWS Glue (Iceberg REST), AWS S3 Tables (Iceberg REST), Unity Catalog (Iceberg REST, including Delta Lake tables with UniForm enabled), Apache Polaris and Snowflake Open Catalog (Iceberg REST, OAuth2 principal credentials), Snowflake Horizon Catalog (Iceberg REST, PAT), and Nessie (Iceberg REST, via the generic rest service type with a branch/ref prefix). Does StarTree support Delta Lake? Yes, indirectly: a Delta Lake table with UniForm enabled publishes Iceberg-compatible metadata alongside its Delta log, so registering it in a Unity Catalog makes it queryable as an External Table through the same Iceberg REST path used for Glue and S3 Tables. There is no direct Delta-log reader — UniForm is required. Enabling UniForm means setting 'delta.enableIcebergCompatV3' = 'true' (Databricks Runtime 18 LTS or later; 'delta.enableIcebergCompatV2' = 'true' on older runtimes) and 'delta.universalFormat.enabledFormats' = 'iceberg' on the Delta table, plus column mapping ('delta.columnMapping.mode' = 'id' recommended for new tables; existing tables can only be upgraded to 'name') — see the Databricks UniForm requirements for the full prerequisites. Three operational caveats: Databricks generates the Iceberg metadata asynchronously after each Delta commit, so freshly written data reaches StarTree only after that conversion plus the next sync; materialized views and streaming tables can only be exposed through the pipeline-managed Compatibility V3 path (Databricks Runtime 17.3 or later, plus delta.enableRowTracking and pipelines.externalMetadata.enabled); and under Compatibility V3 deletes may be written as deletion vectors rather than by rewriting data files — StarTree applies Iceberg v3 Puffin position deletes at query time, but that path is a preview feature you have to ask StarTree to enable (V2 tables keep copy-on-write deletes, which need no such setup). See Unity Catalog: Onboarding via API. Can I query Parquet on S3 or GCS without a catalog at all? Yes — the S3 Data Lake and GCS Data Lake sources scan a raw bucket prefix directly, with no Iceberg table, Glue database, or catalog service of any kind. See S3: Onboarding via API or GCS: Onboarding via API. How is it different from a regular Pinot table? A regular table ingests and stores a local copy of the data as Pinot segments. An External Table leaves the data in place and reads it remotely — so there’s no data duplication and onboarding is fast, at the cost of relying on the object store plus cache for read performance. When should I use one — and when not? Use it when you already have data in S3/Iceberg and want to query it without copying or re-ingesting — large or infrequently queried datasets, or data shared with other engines. Prefer a regular table for the lowest, most predictable query latency, or when you need an unsupported capability such as a sorted index. Does Pinot copy the Parquet into Pinot, or read it from S3 directly? It reads directly from the Parquet files in S3 — the data is never duplicated into Pinot’s segment format. Servers cache the bytes they read (data, index, and footer caches) so repeat queries are fast. See Data and Index Caching.

​
Onboarding & sync

How do I onboard one? Two ways: the Data Portal UI or the REST APIs. The Data Portal wizard covers AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris & Snowflake Open Catalog, Snowflake Horizon, S3 Data Lake, and GCS Data Lake. The API guides cover those plus Apache Polaris & Snowflake Open Catalog, Snowflake Horizon, and Nessie — Nessie is the only API-only source, with no Data Portal wizard yet. Do I have to trigger the first sync? No. After the table is created, the controller’s External Table watcher runs the first sync automatically and then re-syncs on the table’s schedule. You can trigger a run manually to start sooner — see Observability. How often does it sync? Can I change the schedule? It syncs on the schedule (Quartz cron) in the table’s ExternalTableSyncTask config. The Data Portal applies a default of every 5 minutes; via the API you set schedule yourself. Match it to your source’s commit cadence — polling faster than the source changes just adds overhead. Can I pause and resume syncing? Yes. In the Data Portal, open the table and select Pause Sync — this removes the sync schedule (stashing it so it can be restored) rather than touching the table’s data or checkpoint. In-progress runs finish normally and data is preserved; select Resume Sync to pick up where it left off. Connection validation failed — what do I check? Almost always a permission or region mismatch. Confirm the credentials can read both the catalog and the underlying S3 data, and that the region matches. For AWS Glue, the warehouse must be the numeric AWS account ID. Test Connection now checks storage credentials up front for every catalog type, and — for S3 Data Lake and S3 Tables specifically — the bucket region too; see Onboarding fails with a region error. Can multiple tables share the same credentials? Yes — reuse the same catalog connection parameters across as many External Tables as you like. Do I put s3:// in the bucket field? No. The bucket field takes the bucket name only — not an s3://... URL — and the prefix should not have a trailing slash. Make sure the AWS region is set; a missing region is a common cause of onboarding errors. See Troubleshooting. How do I grant access to the source bucket? Three options: an assumed IAM role (set roleArn, plus externalId if required), the cluster’s node IAM role, or static AWS access keys. The role/keys need s3:GetObject and s3:ListBucket on the source bucket and prefix. Verify access from the cluster with aws s3 ls s3://<bucket>/<prefix>/ before onboarding. Can I onboard many tables at once? Onboard each table through the Data Portal or the REST APIs (AWS Glue / Amazon S3 Tables / Unity Catalog). There is no single bulk table-creation API today; script the per-table API flow to automate it.

​
Schema & data types

How are source types mapped to Pinot types? See Data Type Mapping for the Parquet (raw S3) and Iceberg type tables. Complex types (struct, map, list) become JSON or multi-value columns. Can I edit the inferred schema or change a column’s data type? Avoid changing the data types that the preview/onboarding step infers. Forcing a different type (for example, switching a binary/string column to INT) breaks segment generation. Adjusting names, the time column, or null handling is fine. How do I add an inverted index (or another dictionary-backed index)? Keep the column’s forward index RAW and add an explicit dictionary block alongside the index — dictionary-backed indexes (inverted, FST, IFST) need it on External Tables:
Can I rename columns on an External Table? As of 0.16.0, conditionally yes: for catalogType=iceberg-rest sources with full Iceberg snapshot metadata access, enable schemaEvolution.enabled=true on the table (see Schema Evolution). A rename in the source is tracked as an alias against the existing Pinot column rather than creating a new column, so the Pinot column name is preserved. This only resolves top-level field renames (not nested-field renames). Raw S3/GCS Parquet sources have no Iceberg field-id history to key off, so renames there still require a manual schema change; confirm with StarTree support whether your specific catalog adapter (Glue, S3 Tables, Unity, generic REST) has the metadata access this needs. Why do JSON path queries on a struct column return NULL on my Unity Catalog / Delta UniForm table? The table was likely created with Delta column mapping (delta.columnMapping.mode = name or id), which stores every Parquet column under an opaque physical col-<uuid> name. Top-level column names are always resolved back to your logical names, but names inside complex columns (struct, array<struct>, map<_,struct>) are only resolved when storeNestedSchema.enabled is on (the default) for the table’s ExternalTableSyncTask config — if it was set to false, the nested keys are stored under their physical names and JSON_EXTRACT_SCALAR(col, '$.yourField', ...) returns NULL. See Unity Catalog — Schema options for column mapping. How is the time column chosen? Automatically, from the first suitable timestamp/date column, preserving the source granularity. See the time column section. Why are my timestamps off by ~1000×? On current releases, plain Iceberg timestamp / timestamptz columns are inferred as microseconds (matching Iceberg’s storage), timestamp_ns as nanoseconds, and timestamp(p) from its precision. Tables onboarded on older 0.16.0 builds were inferred as milliseconds regardless of the stored unit — that mismatch produces the ~1000× error. Fix it by setting the time column’s granularity explicitly to match the data. See Data Type Mapping. Why did I get a “Requires RAW encoding” error? External Tables require every column to use RAW (no-dictionary) encoding. A column was configured or defaulted to dictionary encoding. The preview/onboarding flow sets RAW automatically.

​
Indexes

Which indexes are supported? Most of them — including inverted, range, timestamp, JSON, text, sparse, star-tree, bloom, FST, IFST, and vector (HNSW, with storeInSegmentFile: true). Only sorted and geospatial/H3 are not supported. See Supported Indexes. Why isn’t the sorted index supported? A sorted index is the column’s forward data physically stored in sorted order. External Tables read the Parquet files in place and don’t own that layout, so the data can’t be reordered. Can I add an index after creating the table? Yes — update the table config to add a supported index, the same as any Pinot table. The index is built over the existing remote data.

​
Performance & caching

Where is the data cached? On each server: a Parquet data cache, an index cache, and a Parquet footer cache. See Data and Index Caching. The first query on a column is slow, then it’s fast. Why? The first read populates the cache from object storage; later reads hit the cache. Enable pre-warm / preload to populate caches at segment load instead of on first query. How do I make queries faster? Add supported indexes for your filters, enable the page cache and preload for the table, and pre-warm at segment load. See Best Practices and Configs for the relevant knobs. How do I clear a cache? Use the server page cache endpoint — see Data and Index Caching.

​
Operations

What happens when the source data changes or files are deleted? Each scheduled sync advances to the latest Iceberg snapshot (or, for raw S3/GCS, scans for new files) and ingests what’s new; the watermark is visible as checkpointValue in the status endpoint. Append-only sources are fully handled. How deletes and compaction are handled depends on the source type — see the next two questions. My source compacts or rewrites files — which source type should I use? (Iceberg REST catalog.) Use an Iceberg REST catalog (catalogType=iceberg-rest — AWS Glue, S3 Tables, Unity Catalog, Apache Polaris / Snowflake, or Nessie), not the raw S3/GCS Data Lake source. A raw Data Lake source pins each ingested segment to a physical Parquet file path; when the source compacts — rewriting small files into larger ones and deleting the originals — those paths break and the server can no longer load or serve the affected segments (you’ll see java.io.FileNotFoundException in server logs). A REST catalog instead tracks the table’s live file set through snapshot metadata, so each sync reconciles to the current post-compaction files. Raw S3/GCS is the right fit only for append-only sources whose files are never rewritten. See Segments fail to load after upstream compaction. Does continueOnFileError protect against files deleted or compacted after ingestion? No. continueOnFileError=true only lets a sync run skip a file it can’t read while it is generating segments — it has no effect once a segment is already in the table. If that segment’s backing file later disappears (e.g. compacted away upstream), the server fails to load or serve it regardless of this setting; the fix is to use an Iceberg REST catalog for compacting sources (above). What’s the difference between continueOnError and continueOnFileError? They are two separate configs, set in different places, and easy to confuse. continueOnError is an OSS ingestionConfig field that skips individual bad rows while reading a file or stream. continueOnFileError is an ExternalTableSyncTask field that skips an entire unreadable file so one bad file doesn’t fail the whole run. Both apply only during ingestion/sync — neither repairs an already-ingested segment.

​
Monitoring & observability

How do I check whether my last sync succeeded? Call the status endpoint and read fileOnboardingRun.status (COMPLETED / RUNNING / FAILED / IDLE). On failure, failurePhase and errorMessage say where and why. See Observability. Does status: COMPLETED mean my data is current? No. Run status only means the last sync run finished — the source may have committed more data since. To judge freshness, call …/externalTable/status?includeLag=true and read syncState, lag.percentDataIngested, and lag.snapshotsBehind. See Ingestion lag. How far behind the source is my data? With ?includeLag=true, subtract lag.synced.committedAt from lag.upstream.committedAt for a time delta, or read lag.snapshotsBehind / lag.percentDataIngested. There is deliberately no single “lag in seconds” field. My sync shows COMPLETED but the table is empty or stale — why? A snapshot whose segment generation keeps failing is retried up to maxRetries (default 3) runs, then abandoned; a later run with no newer snapshot then reports COMPLETED with segmentsUploaded: 0, and fileOnboardingRun — which only ever describes the run happening now — hides the earlier failures. Compare filesDiscovered vs segmentsUploaded, check syncState, and check snapshotIngestion.consecutiveFailedRuns / lastRunError for the hidden failure streak. See Sync says COMPLETED but no data lands. Can I monitor sync from dashboards instead of polling the API? The controller emits per-phase sync timers (EXTERNAL_TABLE_SYNC_*_TIME_MS) for latency dashboards, but there is no lag or per-run success/failure metric — alert on freshness and failures through the status API. See Metrics and alerting.