Skip to main content
This feature is available starting in StarTree release 0.15.0. It must be enabled on demand — contact your StarTree representative to have it activated for your environment.
External tables support a broad set of Pinot indexes out of the box. Indexes are key to getting low-latency query performance from external tables. Without them, every query performs a full segment scan across data read from object storage. With the right indexes configured for your query patterns, Pinot can skip irrelevant segments and data blocks entirely, reducing query time from minutes/seconds to seconds/milliseconds on large datasets, while also cutting S3 I/O costs. The most impactful indexes to configure based on your workload:
  • JSON index — required for filtering or extracting nested fields from complex columns (STRUCT, LIST, MAP).
  • Text index — enables full-text search on free-text string columns.
  • Range index — speeds up range predicates on numeric or timestamp columns (e.g., WHERE ts > X AND ts < Y).
  • Inverted index — efficient equality filtering on low-cardinality columns (e.g., WHERE status = 'active').

​
Supported

Sorted and H3 are the only index types the controller rejects on an external table. Two related validations to be aware of: every non-virtual column read from Parquet must have a fieldConfig with encodingType: RAW (a column without one would default to dictionary encoding and is rejected), and a schema-level transformFunction is rejected.
Derived columns. Starting with StarTree Cloud 0.18.0, you can declare derived columns in ingestionConfig.transformConfigs; earlier releases reject them on External Tables. A server computes each one from the Parquet source columns when it loads the segment, and it can use RAW or DICTIONARY encoding. Validation also checks that:
  • the column is declared in the schema;
  • it isn’t the table’s time column (keep the Parquet timestamp as the time column and derive coarser buckets as ordinary columns);
  • it doesn’t read a column whose forward index is disabled, and derived columns don’t read each other in a cycle.
Declare derived columns only in transformConfigs. A transformFunction set on the schema field is rejected, because the External Table load path never reads it.
For complex (STRUCT/nested-JSON) columns, the open_struct settings that choose which keys get materialized — maxDenseKeys, denseKeys, denseKeyMinFillRate, maxNestedKeyDepth — are rejected at table creation on an external table. Those settings are applied by the dense/sparse key splitter while a segment’s column data is written; an external table never writes column data, so the split is whatever the source Parquet file’s shredding already chose, and the setting can only be ignored.Indexes are not rejected, in either of the two places an OPEN_STRUCT (VARIANT) column can carry them: per-key valueFieldConfigs over a materialized key, and sparseFieldConfig (or its older spelling sparseJsonIndex) over the shared sparse blob. Both name indexes over columns that already exist rather than a split, and a reload builds them by reading back through the Parquet reader — the same path a per-key index already took. See VARIANT columns for the full config and which filters each index serves.

​
Why are columns RAW (no dictionary)?

In a standard Pinot table, every column is dictionary-encoded by default. Pinot builds a lookup table that maps each unique value to a compact integer ID and stores those IDs in the forward index. This dictionary is created during ingestion, when data is converted into Pinot’s native segment format. External tables never go through that ingestion step — they read Parquet files directly from object storage at query time. Because the data is never materialized into Pinot segments, there is no opportunity to build a dictionary, so every column is encoded as RAW (no dictionary). You can explicitly see this in the column config generated by the onboarding flow — every column gets RAW encoding and a RAW forward index, and nothing else:
This means the forward index stores the actual values from Parquet, not dictionary IDs. Indexes that need a dictionary — inverted, FST, IFST, star-tree — still work because Pinot builds a per-segment dictionary sidecar when it writes the index files alongside the Parquet data. The main forward index stays RAW; only the index sidecar carries the dictionary mapping. You opt in by adding a "dictionary": {} block to the column’s indexes config (see examples below). Sorted index is ruled out entirely because it requires data to be physically sorted during the ingestion write path. External tables have no write path — they read whatever order the Parquet files are in — so a sorted index cannot be built or maintained.

​
Adding an index

​
Choosing the right index

If you’re unsure, start with a range index on time and numeric filters and an inverted index on the low-cardinality string columns you filter on most. Add others only when a query needs them.

​
Inverted index (low-cardinality equality)

Dictionary-backed, so keep the forward index RAW and add a dictionary block:
Example query:

​
Range index (numeric / time ranges)

No dictionary needed:
Example query:

​
Bloom filter (high-cardinality equality)

Example query:

​
JSON index

Example query:

​
Text index (full-text search)

Example query:
The onboarding/preview flow generates the base fieldConfigList with RAW encoding for every column. To add an index, merge the indexes block above into the matching column entry — don’t change encodingType.