> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# External Tables Overview

> Query Iceberg (AWS Glue, S3 Tables, Unity Catalog, Nessie) and raw Parquet (S3, GCS) tables directly from Pinot, with no ingestion or data duplication.

<Warning>
  This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
</Warning>

An External Table is a Pinot table whose data stays in Parquet files in your object store — an Iceberg catalog (AWS Glue, Amazon S3 Tables, Unity Catalog, Nessie) or an S3/GCS data lake — instead of being copied into Pinot's own segment format. Pinot reads the remote Parquet at query time and exposes it through standard SQL. There is no ETL pipeline, no data duplication, and onboarding takes minutes instead of hours.

A watcher on the controller detects source-side changes on a schedule and builds segment index files (bloom filters, inverted indexes, range indexes, etc.) alongside the Parquet. Queries use server-side caches — a Parquet data cache, an index cache, and a footer cache — so repeat reads avoid paying S3 round-trip latency. With the right indexes configured, query times drop from minutes to milliseconds on large datasets.

<Tip>
  For a single-page view of everything External Tables support — connectors, authentication methods, table formats, and features — see the [Support Matrix](/corecapabilities/external-table/support-matrix).
</Tip>

***

## How it works

```mermaid theme={null}
flowchart LR
    UserSQL["SQL Query"] --> Broker["Pinot Broker"]
    Broker --> Server["Pinot Server"]
    Server --> CacheCheck{"Cache hit?"}
    CacheCheck -->|"Yes"| Result["Return result"]
    CacheCheck -->|"No"| S3["S3 / Object Store\nParquet files"]
    S3 --> Server
    Server --> LocalCache["Local caches\ndata · index · footer"]
    LocalCache --> Result
```

At query time, the server checks its local caches first. On a miss, it fetches the required Parquet column pages or index byte ranges from object storage and stores them for subsequent queries. Index files (built by the watcher at sync time) live in tiered storage and are also cached locally, so filters and aggregations avoid full column scans.

***

## Supported sources

| Source                  | Protocol                                | `catalogType`                                                          |
| ----------------------- | --------------------------------------- | ---------------------------------------------------------------------- |
| Iceberg (AWS Glue)      | Iceberg REST                            | `iceberg-rest` (`serviceType=glue`)                                    |
| Iceberg (AWS S3 Tables) | Iceberg REST                            | `iceberg-rest` (`serviceType=s3Tables`)                                |
| Unity Catalog           | Iceberg REST                            | `iceberg-rest` (`serviceType=unity`)                                   |
| Nessie                  | Iceberg REST                            | `iceberg-rest` (`serviceType=rest`, with `prefix` set to a branch/ref) |
| S3 Data Lake            | Raw Parquet files under an S3 prefix    | `s3`                                                                   |
| GCS Data Lake           | Raw Parquet files in GCS via S3 interop | `gcs-interop`                                                          |

`catalogType=iceberg-rest` works with any Iceberg REST–compliant catalog (built-in service adapters exist for AWS Glue, AWS S3 Tables, Unity Catalog, and generic REST — the last of which also covers Nessie, addressed via its branch/ref as the REST `prefix`). Data files must be Parquet.

<Note>
  There is no dedicated `catalogType` for Delta Lake's native log format. A **Delta Lake table with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled** publishes Iceberg-compatible metadata alongside its Delta log — registering it in a **Unity Catalog** (`serviceType=unity`) makes it queryable as an External Table through the same Iceberg REST path used for Glue and S3 Tables.
</Note>

***

## Where to start

New here? Start with a task instead of a vendor:

* **["I just have Parquet files on S3/GCS"](./getting-started/query-without-a-catalog)** — no catalog, no Iceberg, query the bucket directly.
* **["I already have a catalog"](./getting-started/connect-a-catalog)** — AWS Glue, Amazon S3 Tables, Unity Catalog, or Nessie.
* **["My queries are slow"](./getting-started/speed-up-queries)** — indexes, caching, and OOM protection.

Otherwise, choose your catalog type and pick your preferred onboarding path:

**AWS Glue**

* [Onboarding via Data Portal](./glue/onboarding-data-portal) — wizard-based setup for AWS Glue
* [Onboarding via API](./glue/onboarding-api) — 4-step REST flow with bash examples and a quickstart script

**Amazon S3 Tables**

* [Onboarding via Data Portal](./s3tables/onboarding-data-portal) — wizard-based setup for Amazon S3 Tables
* [Onboarding via API](./s3tables/onboarding-api) — 4-step REST flow with bash examples and a quickstart script

**Unity Catalog**

* [Onboarding via API](./unity/onboarding-api) — 4-step REST flow; supports Databricks-managed and OSS Unity Catalog, including Delta Lake tables with UniForm enabled
* Onboarding via Data Portal is not yet available — the API flow is the only option today

**Nessie**

* [Onboarding via API](./nessie/onboarding-api) — 4-step REST flow using the generic Iceberg REST adapter with a branch/ref `prefix`
* Onboarding via Data Portal is not yet available — the API flow is the only option today

**S3 Data Lake**

* [Onboarding via Data Portal](./s3/onboarding-data-portal) — wizard-based setup for raw Parquet on S3
* [Onboarding via API](./s3/onboarding-api) — 4-step REST flow with bash examples and a quickstart script

**GCS Data Lake**

* [Onboarding via Data Portal](./gcs/onboarding-data-portal) — wizard-based setup for Parquet files in GCS
* [Onboarding via API](./gcs/onboarding-api) — 4-step REST flow using GCS S3 interop with HMAC keys

**After onboarding**

* **Monitoring sync progress** → [Observability](./observability) — sync status, checkpoint watermark, and source file count APIs
* **Queries are slow** → [Indexes](./indexes) to add the right indexes, then [Best Practices & Configs](./best-practices-and-configs) for caching and tuning
* **Something is broken** → [Troubleshooting](./troubleshooting) for symptom-based fixes, or [FAQ](./faq) for common questions

**New in 0.16.0**

* **[SQL DDL](./sql-ddl)** — create an External Table with a single `CREATE TABLE ... WITH (...)` statement instead of the 4-step REST flow.
* **[Segment Groups](./segment-groups)** — group many small member segments into one logical unit to cut cluster metadata overhead for tables with a large number of source files.
* **[Deletion Vectors](./deletion-vectors)** (preview) — apply Iceberg row-level deletes/updates at query time instead of requiring append-only sources.

***

## Page map

| Page                                                                              | What it covers                                                                                    |
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| [Query Parquet Directly](./getting-started/query-without-a-catalog)               | Task-based entry point for raw Parquet on S3/GCS with no catalog                                  |
| [Connect a Catalog](./getting-started/connect-a-catalog)                          | Task-based entry point for Glue, S3 Tables, Unity, and Nessie                                     |
| [Speed Up Queries](./getting-started/speed-up-queries)                            | Task-based entry point for indexes, caching, and OOM protection                                   |
| [AWS Glue: Onboarding via Data Portal](./glue/onboarding-data-portal)             | Point-and-click wizard for AWS Glue                                                               |
| [AWS Glue: Onboarding via API](./glue/onboarding-api)                             | REST API 4-step flow for AWS Glue (Iceberg REST)                                                  |
| [Amazon S3 Tables: Onboarding via Data Portal](./s3tables/onboarding-data-portal) | Point-and-click wizard for Amazon S3 Tables                                                       |
| [Amazon S3 Tables: Onboarding via API](./s3tables/onboarding-api)                 | REST API 4-step flow for Amazon S3 Tables (Iceberg REST)                                          |
| [Unity Catalog: Onboarding via API](./unity/onboarding-api)                       | REST API 4-step flow for Unity Catalog (Iceberg REST), including Delta+UniForm tables             |
| [Nessie: Onboarding via API](./nessie/onboarding-api)                             | REST API 4-step flow for Nessie (Iceberg REST, generic `rest` adapter with a branch/ref `prefix`) |
| [S3: Onboarding via Data Portal](./s3/onboarding-data-portal)                     | Point-and-click wizard for raw Parquet on S3                                                      |
| [S3: Onboarding via API](./s3/onboarding-api)                                     | REST API 4-step flow for S3 Data Lake                                                             |
| [GCS Data Lake: Onboarding via Data Portal](./gcs/onboarding-data-portal)         | Point-and-click wizard for Parquet in GCS via S3 interop                                          |
| [GCS Data Lake: Onboarding via API](./gcs/onboarding-api)                         | REST API 4-step flow using GCS HMAC keys and S3 interop                                           |
| [Observability](./observability)                                                  | Sync run status, ingestion checkpoint, source file count, and manual trigger APIs                 |
| [Data Type Mapping](./data-type-mapping)                                          | Parquet → Pinot and Iceberg → Pinot type mapping tables, plus time column detection               |
| [Indexes](./indexes)                                                              | Supported indexes, why columns are RAW, and per-index config examples                             |
| [Data and Index Caching](./data-and-index-caching)                                | Three caches (data, index, footer), eviction, restart behavior, and how to clear them             |
| [Best Practices & Configs](./best-practices-and-configs)                          | Full config reference: sync task, tier backend, server/cluster, query options, and OOM protection |
| [FAQ](./faq)                                                                      | Common questions by category: general, onboarding, schema, indexes, performance, operations       |
| [DataExportTask](./data-export-task)                                              | Export completed REALTIME segments to Parquet on filesystem or Iceberg REST                       |
| [Troubleshooting](./troubleshooting)                                              | Symptom-based diagnostic playbook with exact error strings and escalation guidance                |
| [SQL DDL](./sql-ddl)                                                              | Create an External Table with `CREATE TABLE ... WITH (...)` instead of the REST flow              |
| [Segment Groups](./segment-groups)                                                | Group small member segments into one logical unit to reduce metadata overhead                     |
| [Deletion Vectors](./deletion-vectors)                                            | Apply Iceberg row-level deletes/updates at query time (preview)                                   |