> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# FAQ

> Answers to common questions about StarTree External Tables — supported sources, onboarding, schema, indexes, performance, and operations.

<Warning>
  This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
</Warning>

## General

**What is an External Table?** A Pinot table whose data stays in Parquet files in your object store (an Iceberg catalog such as AWS Glue, Amazon S3 Tables, Unity Catalog, or Nessie, or a raw S3/GCS data lake) instead of being copied into Pinot's own segment format. Pinot reads the remote data at query time, with server-side caching to keep queries fast.

**Which sources are supported?** S3 Data Lake (raw Parquet), GCS Data Lake (raw Parquet via S3 interop), AWS Glue (Iceberg REST), AWS S3 Tables (Iceberg REST), Unity Catalog (Iceberg REST, including Delta Lake tables with UniForm enabled), and Nessie (Iceberg REST, via the generic `rest` service type with a branch/ref `prefix`).

**Does StarTree support Delta Lake?** Yes, indirectly: a Delta Lake table with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled publishes Iceberg-compatible metadata alongside its Delta log, so registering it in a **Unity Catalog** makes it queryable as an External Table through the same Iceberg REST path used for Glue and S3 Tables. There is no direct Delta-log reader — UniForm is required. Enabling UniForm means setting `'delta.enableIcebergCompatV2' = 'true'` and `'delta.universalFormat.enabledFormats' = 'iceberg'` on the Delta table, plus column mapping (`'delta.columnMapping.mode' = 'id'` recommended for new tables; existing tables can only be upgraded to `'name'`) — see the [Databricks UniForm requirements](https://docs.databricks.com/aws/en/delta/uniform#requirements) for the full prerequisites. Two operational caveats: Databricks generates the Iceberg metadata **asynchronously** after each Delta commit, so freshly written data reaches StarTree only after that conversion plus the next sync; and **materialized views and streaming tables can't enable UniForm**, so they can't be onboarded. See [Unity Catalog: Onboarding via API](./unity/onboarding-api).

**Can I query Parquet on S3 or GCS without a catalog at all?** Yes — the **S3 Data Lake** and **GCS Data Lake** sources scan a raw bucket prefix directly, with no Iceberg table, Glue database, or catalog service of any kind. See [S3: Onboarding via API](./s3/onboarding-api) or [GCS: Onboarding via API](./gcs/onboarding-api).

**How is it different from a regular Pinot table?** A regular table ingests and stores a local copy of the data as Pinot segments. An External Table leaves the data in place and reads it remotely — so there's no data duplication and onboarding is fast, at the cost of relying on the object store plus cache for read performance.

**When should I use one — and when not?** Use it when you already have data in S3/Iceberg and want to query it without copying or re-ingesting — large or infrequently queried datasets, or data shared with other engines. Prefer a regular table for the lowest, most predictable query latency, or when you need an unsupported capability such as a sorted index.

**Does Pinot copy the Parquet into Pinot, or read it from S3 directly?** It reads directly from the Parquet files in S3 — the data is never duplicated into Pinot's segment format. Servers cache the bytes they read (data, index, and footer caches) so repeat queries are fast. See [Data and Index Caching](./data-and-index-caching).

***

## Onboarding & sync

**How do I onboard one?** Two ways: the Data Portal UI or the REST APIs. See [AWS Glue](/corecapabilities/external-table/glue/onboarding-data-portal) or [Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-data-portal) for the wizard, and the corresponding [Glue](/corecapabilities/external-table/glue/onboarding-api) / [S3 Tables](/corecapabilities/external-table/s3tables/onboarding-api) / [Unity Catalog](/corecapabilities/external-table/unity/onboarding-api) / [Nessie](/corecapabilities/external-table/nessie/onboarding-api) API guides. Unity Catalog and Nessie are API-only for now — there's no Data Portal wizard for either yet.

**Do I have to trigger the first sync?** No. After the table is created, the controller's External Table watcher runs the first sync automatically and then re-syncs on the table's schedule. You can trigger a run manually to start sooner — see [Observability](./observability).

**How often does it sync? Can I change the schedule?** It syncs on the `schedule` (Quartz cron) in the table's `ExternalTableSyncTask` config. The Data Portal applies a default of every 5 minutes; via the API you set `schedule` yourself. Match it to your source's commit cadence — polling faster than the source changes just adds overhead.

**Can I pause and resume syncing?** Yes. In the Data Portal, open the table and select **Pause Sync** — this removes the sync schedule (stashing it so it can be restored) rather than touching the table's data or checkpoint. In-progress runs finish normally and data is preserved; select **Resume Sync** to pick up where it left off.

**Connection validation failed — what do I check?** Almost always a permission or region mismatch. Confirm the credentials can read both the catalog and the underlying S3 data, and that the region matches. For AWS Glue, the warehouse must be the numeric AWS account ID.

**Can multiple tables share the same credentials?** Yes — reuse the same catalog connection parameters across as many External Tables as you like.

**Do I put `s3://` in the bucket field?** No. The bucket field takes the bucket name only — not an `s3://...` URL — and the prefix should not have a trailing slash. Make sure the AWS region is set; a missing region is a common cause of onboarding errors. See [Troubleshooting](./troubleshooting#onboarding).

**How do I grant access to the source bucket?** Three options: an **assumed IAM role** (set `roleArn`, plus `externalId` if required), the cluster's **node IAM role**, or static **AWS access keys**. The role/keys need `s3:GetObject` and `s3:ListBucket` on the source bucket and prefix. Verify access from the cluster with `aws s3 ls s3://<bucket>/<prefix>/` before onboarding.

**Can I onboard many tables at once?** Onboard each table through the Data Portal or the REST APIs ([AWS Glue](/corecapabilities/external-table/glue/onboarding-api) / [Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-api) / [Unity Catalog](/corecapabilities/external-table/unity/onboarding-api)). There is no single bulk table-creation API today; script the per-table API flow to automate it.

***

## Schema & data types

**How are source types mapped to Pinot types?** See [Data Type Mapping](./data-type-mapping) for the Parquet (raw S3) and Iceberg type tables. Complex types (struct, map, list) become JSON or multi-value columns.

**Can I edit the inferred schema or change a column's data type?** Avoid changing the data types that the preview/onboarding step infers. Forcing a different type (for example, switching a binary/string column to `INT`) breaks segment generation. Adjusting names, the time column, or null handling is fine.

**How do I add an inverted index (or another dictionary-backed index)?** Keep the column's forward index `RAW` and add an explicit `dictionary` block alongside the index — dictionary-backed indexes (inverted, FST, IFST) need it on External Tables:

```json theme={null}
"indexes": {
  "forward":    { "encodingType": "RAW" },
  "dictionary": {},
  "inverted":   { "disabled": false }
}
```

**Can I rename columns on an External Table?** As of 0.16.0, conditionally yes: for `catalogType=iceberg-rest` sources with full Iceberg snapshot metadata access, enable `schemaEvolution.enabled=true` on the table (see [Best Practices & Configs — Checkpointing & schema evolution](./best-practices-and-configs#checkpointing--schema-evolution)). A rename in the source is tracked as an alias against the existing Pinot column rather than creating a new column, so the Pinot column name is preserved. This only resolves **top-level** field renames (not nested-field renames). Raw S3/GCS Parquet sources have no Iceberg field-id history to key off, so renames there still require a manual schema change; confirm with StarTree support whether your specific catalog adapter (Glue, S3 Tables, Unity, generic REST) has the metadata access this needs.

**How is the time column chosen?** Automatically, from the first suitable timestamp/date column, preserving the source granularity. See the [time column section](./data-type-mapping#time-column).

**Why are my timestamps off by \~1000×?** Iceberg REST catalogs report timestamps without precision, so they're inferred as milliseconds. If the underlying Parquet stores microseconds, set the time column's granularity explicitly to match.

**Why did I get a "Requires RAW encoding" error?** External Tables require every column to use RAW (no-dictionary) encoding. A column was configured or defaulted to dictionary encoding. The preview/onboarding flow sets RAW automatically.

***

## Indexes

**Which indexes are supported?** Most of them — including inverted, range, timestamp, JSON, text, sparse, star-tree, bloom, FST, and IFST. **Sorted, vector, and geospatial/H3** are not supported. See [Supported Indexes](./indexes).

**Why isn't the sorted index supported?** A sorted index is the column's forward data physically stored in sorted order. External Tables read the Parquet files in place and don't own that layout, so the data can't be reordered.

**Can I add an index after creating the table?** Yes — update the table config to add a supported index, the same as any Pinot table. The index is built over the existing remote data.

***

## Performance & caching

**Where is the data cached?** On each server: a Parquet data cache, an index cache, and a Parquet footer cache. See [Data and Index Caching](./data-and-index-caching).

**The first query on a column is slow, then it's fast. Why?** The first read populates the cache from object storage; later reads hit the cache. Enable pre-warm / preload to populate caches at segment load instead of on first query.

**How do I make queries faster?** Add supported indexes for your filters, enable the page cache and preload for the table, and pre-warm at segment load. See [Best Practices and Configs](./best-practices-and-configs) for the relevant knobs.

**How do I clear a cache?** Use the server page cache endpoint — see [Data and Index Caching](./data-and-index-caching#clearing-caches).

***

## Operations

**What happens when the source data changes or files are deleted?** Each scheduled sync advances to the latest Iceberg snapshot (or, for raw S3/GCS, scans for new files) and ingests what's new; the watermark is visible as `checkpointValue` in the [status endpoint](./observability#run-status). **Append-only sources are fully handled.** How deletes and compaction are handled depends on the source type — see the next two questions.

**My source compacts or rewrites files — which source type should I use? (Iceberg REST catalog.)** Use an **Iceberg REST catalog** (`catalogType=iceberg-rest` — AWS Glue, S3 Tables, Unity Catalog, or Nessie), not the raw S3/GCS Data Lake source. A raw Data Lake source pins each ingested segment to a **physical Parquet file path**; when the source compacts — rewriting small files into larger ones and deleting the originals — those paths break and the server can no longer load or serve the affected segments (you'll see `java.io.FileNotFoundException` in server logs). A REST catalog instead tracks the table's live file set through **snapshot metadata**, so each sync reconciles to the current post-compaction files. Raw S3/GCS is the right fit only for **append-only** sources whose files are never rewritten. See [Segments fail to load after upstream compaction](./troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction).

**Does `continueOnFileError` protect against files deleted or compacted after ingestion?** No. `continueOnFileError=true` only lets a **sync run** skip a file it can't read *while it is generating segments* — it has no effect once a segment is already in the table. If that segment's backing file later disappears (e.g. compacted away upstream), the server fails to load or serve it regardless of this setting; the fix is to use an Iceberg REST catalog for compacting sources (above).

**What's the difference between `continueOnError` and `continueOnFileError`?** They are two separate configs, set in different places, and easy to confuse. `continueOnError` is an OSS `ingestionConfig` field that skips **individual bad rows** while reading a file or stream. `continueOnFileError` is an `ExternalTableSyncTask` field that skips an **entire unreadable file** so one bad file doesn't fail the whole run. Both apply only during ingestion/sync — neither repairs an already-ingested segment.

***

## Monitoring & observability

**How do I check whether my last sync succeeded?** Call the [status endpoint](./observability#run-status) and read `fileOnboardingRun.status` (`COMPLETED` / `RUNNING` / `FAILED` / `IDLE`). On failure, `failurePhase` and `errorMessage` say where and why. See [Observability](./observability).

**Does `status: COMPLETED` mean my data is current?** No. Run status only means the last sync run *finished* — the source may have committed more data since. To judge freshness, call `…/externalTable/status?includeLag=true` and read `lag.caughtUp`, `lag.percentDataIngested`, and `lag.snapshotsBehind`. See [Ingestion lag](./observability#ingestion-lag).

**How far behind the source is my data?** With `?includeLag=true`, subtract `lag.synced.committedAt` from `lag.upstream.committedAt` for a time delta, or read `lag.snapshotsBehind` / `lag.percentDataIngested`. There is deliberately no single "lag in seconds" field.

**My sync shows `COMPLETED` but the table is empty or stale — why?** A snapshot whose segment generation keeps failing is retried up to `maxRetries` (default 3) runs, then abandoned; a later run with no newer snapshot then reports `COMPLETED` with `segmentsUploaded: 0`, and the single-slot run-status record hides the earlier failures. Compare `filesDiscovered` vs `segmentsUploaded` and check `lag.caughtUp`. See [Sync says COMPLETED but no data lands](./troubleshooting#sync-says-completed-but-no-data-lands).

**Can I monitor sync from dashboards instead of polling the API?** The controller emits per-phase sync timers (`EXTERNAL_TABLE_SYNC_*_TIME_MS`) for latency dashboards, but there is no lag or per-run success/failure metric — alert on freshness and failures through the status API. See [Metrics and alerting](./observability#metrics-and-alerting).