Data and Index caching - StarTree Docs

1. Overview

This document describes how Parquet data cache and Pinot index cache work in case of External tables as well as regular tables configured with remote tiered storage. External tables read their data from object storage on every query. The Parquet Data Cache keeps recently-read Parquet pages on the server so subsequent queries do not pay the round-trip. Indexes for both external tables and Pinot tiered storage tables also live on object storage. The Index Cache keeps recently-read index byte ranges on the server.

Cache	What it stores	Used by	On-disk location	When is it populated
Parquet Data Cache	Compressed Parquet column data and dictionary pages	External tables (Iceberg/Delta Lake)	`{dataDir}/remote_data_cache`	On every Parquet column read (always on)
Index Cache	Raw byte ranges of segment index files (`columns.psf`, `inverted_index`)	External tables and Pinot tiered storage tables	`{dataDir}/index/index_cache`	On query opt-in

Both caches share the same underlying machinery, but populate different tiers (memory and disk):

Parquet Data Cache populates both tiers — an in-memory prefetch tier (decompressed, decoded values on heap) sits in front of an mmap-backed disk tier.
Index Cache populates only the disk tier. Index byte ranges are stored as raw bytes; there is no decoded form to keep in memory.

Chat GPT Image May 12, 2026, 04 41 09 PM

The two caches do not share quota, directory, or eviction. They are sized and tuned independently. Following image depicts the high level query flow and how it uses the different caches.

Chat GPT Image May 13, 2026, 08 39 05 PM

2. What’s enabled out of the box

2.1 Parquet Data Cache — always on

This cache is always on and does not need to be explicitly enabled. The cache is created at server startup and serves every read against a remote-Parquet column.

Tier	Default size	Backing store
Prefetch (in-memory)	1 GB	Java heap (deserialized values)
Disk	30% of total disk	mmap’d files at `{dataDir}/remote_data_cache`

2.2 Index Cache — opt-in per query

The Index Cache is populated only when a query is preceded by

SET `enable.prefetch.page.cache` = 'true'

Byte ranges of segment index files fetched from object storage during that query are persisted on disk, so subsequent queries (with or without the option) get cache hits. This applies to both external tables and Pinot tiered storage tables — any query that needs to read an index buffer from object storage benefits from this cache.

Tier	Default size	Backing store
Disk	30% of total disk	mmap’d files at `{dataDir}/index_cache`

Without the query option and without StarTree bitmap reads, the Index Cache stays empty and the directory remains untouched.

3. Advanced operations: Enable, disable, skip

3.1 Parquet Data Cache

Always on at the server level and cannot be globally disabled. However, you can use a per-query bypass as shown below:

SET `skip.remote.table.cache` = 'true';
SELECT ... FROM remoteTable WHERE ...;

When set, both tiers are bypassed for that query. Pages are read directly from object storage and not stored.

3.2 Index Cache

In this case, there is no global enable flag; enablement is a per-query decision. You can enable index cache for a specified query as shown below:

Action	How
Enable for one query	`SET` enable.prefetch.page.cache `= 'true'`

As mentioned previously, once set, subsequent queries can benefit from this index cache as applicable.

4. Behavior on restart

The on-disk tier is wiped on server startup by default. To keep cached data across restarts, use the following server JVM option:

-DPINOT_PARQUET_DISK_CACHE_SNAPSHOT_ENABLED=true

This applies to both caches identically. With the flag set, the disk tier reloads its key index from a snapshot file written during shutdown and resumes serving immediately. The prefetch tier is in-memory — it always starts empty and is not persisted across restarts.

5. Eviction policies

Tier	Cache	Policy	Defaults
Prefetch (in-memory)	Parquet Data Cache only	Circular buffer on heap; oldest slot is overwritten on insert	1 GB capacity, no TTL
Disk	Both	LRU over fixed-size mmap fragment files. Time based eviction coming soon.	Parquet Data Cache: 32 MB fragments. Index Cache: 512 MB fragments. Background eviction sweep every 60s; immediate non-blocking pass when fragments are 90% full.

When the disk tier is full, the oldest fragment file is unmapped and deleted; new pages are written to a fresh fragment.

6. Clearing caches

HTTP endpoints

There are certain scenarios in which user may want to clear the caches. StarTree provides Cluster-wide APIs for this (controller fans out to every server):

Endpoint	Effect
`DELETE /pageCache?consumerName=PARQUET_INDEX`	Clear Parquet Data Cache (all tiers)
`DELETE /pageCache?consumerName=SEGMENT_INDEX`	Clear Index Cache (all tiers)
`DELETE /pageCache`	Clear both

This can also be done via a per-server API (skip the controller, hit one server directly):

DELETE http://<server-host>:<adminPort>/pageCache?consumerName=PARQUET_INDEX

Response is JSON listing cleared cache.

​1. Overview

​2. What’s enabled out of the box

​2.1 Parquet Data Cache — always on

​2.2 Index Cache — opt-in per query

​3. Advanced operations: Enable, disable, skip

​3.1 Parquet Data Cache

​3.2 Index Cache

​4. Behavior on restart

​5. Eviction policies

​6. Clearing caches

​HTTP endpoints