> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage efficiency

> Reduce a table's footprint without hurting query performance — choose encodings and compression codecs, size segments, prune indexes, and measure the effect of each change on your own data.

[Table Storage Usage](/corecapabilities/manage-data/table-storage-usage) tells you how much a table is consuming and where. This page is about the next question: what to change, and how to know whether the change helped.

There is no universally correct configuration. Compression ratio depends on your data's cardinality, sortedness, and value distribution, so the honest answer to "which codec is smallest" is always "measure it on this column." What this page gives you is the set of levers, in the order that usually pays best, and a method for measuring each one.

## Where the bytes actually go

Before changing anything, find out what is large. A table's on-disk footprint is the sum of:

| Component             | Typically dominated by                                                                                        |
| --------------------- | ------------------------------------------------------------------------------------------------------------- |
| **Forward indexes**   | The raw column values. Usually the largest single contributor.                                                |
| **Dictionaries**      | High-cardinality string columns, where the dictionary approaches the size of the data                         |
| **Auxiliary indexes** | Inverted, range, text, JSON, and star-tree indexes — each one is additional bytes on top of the forward index |
| **Metadata**          | Per-segment and per-column metadata. Small individually, significant across tens of thousands of segments     |

Open the table's **Storage** tab and note three figures before you start: **Used**, **Orphaned**, and **Avg Segment Size** alongside the segment-size percentiles.

<Warning>
  **Check `Orphaned` first.** If a meaningful share of the footprint is orphaned — unknown files, stale versions, or deleted segments past retention — no amount of codec tuning is the right fix. That storage is not holding your data. See [What "Orphaned" includes](/corecapabilities/manage-data/table-storage-usage#what-orphaned-includes).
</Warning>

## Lever 1: retention and tiering

The cheapest byte is the one you are not storing.

* **Retention.** Confirm the table has a retention period and that it matches what the data is actually used for. A table with no retention configured fails the [`SEGMENT_RETENTION_CHECK`](/corecapabilities/observability/health-checks#segment_retention_check) health check and grows until it hits its quota.
* **Tiered storage.** Move cold data to object storage rather than keeping it on server disks. This changes where the bytes live and what they cost, without deleting anything. See [Tiered storage](/corecapabilities/manage-data/set-up-tiered-storage/motivation).

Both of these usually move more bytes than any encoding change, and neither requires touching the schema. Do them first.

## Lever 2: segment sizing

Segment size affects both storage and query cost. Many small segments compress worse than the same rows in fewer segments — compression works on blocks, and small blocks give it less to work with — while also multiplying per-segment metadata and per-query overhead.

Signals that this is your problem:

* [`SEGMENT_SIZE_CHECK`](/corecapabilities/observability/health-checks#segment_size_check) failing (more than a quarter of segments under 5 MB)
* [`SEGMENT_COUNT_CHECK`](/corecapabilities/observability/health-checks#segment_count_check) failing (more than 50,000 segments)
* A large gap between **Avg Segment Size** and the p90/p99 figures on the Storage tab

What to do:

* For real-time tables, raise the flush threshold so segments seal larger. See [Configuring the segment threshold](/recipes/configuring-segment-threshold).
* For existing small segments, merge them — [real-time](/recipes/merge-segments-realtime) or [offline](/recipes/merge-small-segments).
* For offline tables, review push granularity: a daily push of a small table produces small segments by construction.

## Lever 3: encoding — dictionary or raw

Every column is stored either dictionary-encoded or as raw values, and this choice matters more than the codec applied on top of it.

| Encoding       | Stores                                                       | Best for                                                                                                                                                         |
| -------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Dictionary** | A dictionary of distinct values, plus per-row dictionary IDs | Low to medium cardinality. Repeated values collapse to a small ID, so the saving grows as cardinality falls relative to row count.                               |
| **Raw**        | The values themselves, in compressed chunks                  | High cardinality. When almost every value is distinct, the dictionary is nearly as large as the data and buys nothing while adding an indirection on every read. |

The failure mode to look for is a **high-cardinality string column left dictionary-encoded** — request IDs, URLs, free-text fields, UUIDs. The dictionary is large, poorly compressible, and held in memory. Switching such a column to raw encoding with a codec is frequently the single largest saving available on a wide table.

Dictionary encoding is also a prerequisite for some index types, so check what else a column is used for before switching it. See [Dictionary index](/corecapabilities/manage-data/indexes/dictionary-index) and [Forward index](/corecapabilities/manage-data/indexes/forward-index).

## Lever 4: compression codec

Codecs are set per column through `fieldConfigList`. See [Forward index](/corecapabilities/manage-data/indexes/forward-index) for the configuration syntax; this section is about which one to pick.

### Codecs for raw-encoded columns

| Codec          | Character                                                                                                                                    |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `PASS_THROUGH` | No compression. Smallest CPU cost, largest footprint. Reasonable only when the column is already incompressible and latency-critical.        |
| `LZ4`          | Fast to compress and decompress, moderate ratio. The usual default for raw columns on a query-latency-sensitive table.                       |
| `SNAPPY`       | Similar profile to LZ4 — balanced ratio and speed.                                                                                           |
| `ZSTANDARD`    | Noticeably better ratio, more CPU on decompress. The usual choice when the column is large and not on the hot path of your tightest queries. |
| `GZIP`         | High ratio, slowest. Rarely the right answer for query-serving data; occasionally right for very cold columns.                               |

The trade-off is always the same shape: better ratio costs decompress CPU on every query that reads the column. That is why "which codec" cannot be answered without knowing whether the column is read by your latency-critical queries or only by occasional ad-hoc ones — the same table can justify `ZSTANDARD` on one column and `LZ4` on another.

### Specialized codecs

| Codec                                     | Applies to                                                                                                                                                                                                                                                                                                                                                                           |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `MV_ENTRY_DICT`                           | Multi-value **dictionary-encoded** columns. Adds a second level of dictionary encoding over the multi-value entries themselves, which pays when the same combinations of values recur across rows. The only codec applicable to dictionary-encoded columns.                                                                                                                          |
| `DELTA`, `DELTADELTA`                     | Numeric `INT` and `LONG` columns whose consecutive values differ by small amounts — monotonic timestamps, sequence numbers, counters. Stores differences rather than values, then compresses those with LZ4. `DELTADELTA` takes the difference twice, which suits values increasing at a near-constant rate. Ineffective, and possibly worse than `LZ4`, on values that jump around. |
| `CLP`, `CLPV2`, `CLPV2_ZSTD`, `CLPV2_LZ4` | Log-line text. These exploit the repeated structure of log messages — a static template with variable fields — and are not general-purpose string codecs. Use them for columns holding log messages and nowhere else.                                                                                                                                                                |

<Note>
  The [forward block index](/corecapabilities/manage-data/indexes/forward-block-index) has its own compression configuration with its own codec names. Do not assume a name from one applies to the other.
</Note>

## Lever 5: prune indexes you are not using

Each auxiliary index is additional storage. It is worth periodically asking, per index, which query needs it — an inverted index added during a past investigation and never removed is pure cost.

Two things help you answer that:

* [`system_query_log`](/corecapabilities/query_data/advanced_operations/query-logger) shows the predicates your queries actually use. Columns nobody filters on do not need filter indexes.
* `numEntriesScannedInFilter` on those queries tells you whether an index is being used effectively. A filter index that is not reducing scanned entries is not earning its bytes.

The same logic applies in reverse: a column that is scanned heavily might be *worth* more index bytes. Storage efficiency is not the only objective — the goal is knowing what each byte buys.

## Measuring a change

The only credible answer about compression is one measured on your data. The method:

<Steps>
  <Step title="Record the baseline">
    From the table's **Storage** tab, note **Used**, **Avg Segment Size**, and the segment-size percentiles. For a per-column view, pull segment metadata for a representative segment through the [segment APIs](/api-reference/segment/reload-a-segment) — per-column index sizes are what tell you whether the column you changed is the one that moved.
  </Step>

  <Step title="Change one thing">
    One column, or one codec, at a time. Changing an encoding and a codec together leaves you unable to attribute the result — and the two interact, so the combined effect is not the sum of the separate ones.
  </Step>

  <Step title="Reload, and wait for it to finish">
    Configuration changes apply to **new** segments immediately and to existing segments only on reload. Until you reload, the measurement is of the old layout. Use [Reload all segments](/api-reference/segment/reload-all-segments) — or the **Reload segments** remediation on the [`TABLE_SEGMENTS_RELOAD_CHECK`](/corecapabilities/observability/health-checks#table_segments_reload_check) health check — and confirm completion before measuring. For larger structural changes, see the [Alter Table task](/corecapabilities/manage-data/alter-table-task).
  </Step>

  <Step title="Measure both sides">
    Compare storage *and* query latency. A codec change that saves 30% of a column's bytes and adds 15 ms to your p99 may or may not be a win — that depends on your workload, and you cannot tell from the storage number alone. Check p95 and p99 for the queries that read the column before and after.
  </Step>

  <Step title="Test on a copy where the stakes are high">
    For a large production table, validate on a table built from a representative subset of the same data rather than iterating on the production table. Reloads on a large table are not free, and reverting is another reload.
  </Step>
</Steps>

<Info>
  Compression ratios reported elsewhere — in benchmarks, in other companies' blog posts, in Pinot's own documentation — are properties of the data they were measured on, not of the codec. A codec that halves one team's log column may gain a few percent on your numeric one. Treat published ratios as an ordering hint, never as a projection for your table.
</Info>

## Related

* [Table Storage Usage](/corecapabilities/manage-data/table-storage-usage) — measuring the footprint and understanding orphaned storage
* [Forward index](/corecapabilities/manage-data/indexes/forward-index) — encodings and the `compressionCodec` configuration syntax
* [Dictionary index](/corecapabilities/manage-data/indexes/dictionary-index) — when dictionary encoding pays
* [Tiered storage](/corecapabilities/manage-data/set-up-tiered-storage/motivation) — moving cold data to object storage
* [Health check reference](/corecapabilities/observability/health-checks) — the segment-size, segment-count, and retention checks
* [Configuring the segment threshold](/recipes/configuring-segment-threshold) — sizing real-time segments