Skip to main content
Table Storage Usage tells you how much a table is consuming and where. This page is about the next question: what to change, and how to know whether the change helped. There is no universally correct configuration. Compression ratio depends on your data’s cardinality, sortedness, and value distribution, so the honest answer to “which codec is smallest” is always “measure it on this column.” What this page gives you is the set of levers, in the order that usually pays best, and a method for measuring each one.

​
Where the bytes actually go

Before changing anything, find out what is large. A table’s on-disk footprint is the sum of: Open the table’s Storage tab and note three figures before you start: Used, Orphaned, and Avg Segment Size alongside the segment-size percentiles.
Check Orphaned first. If a meaningful share of the footprint is orphaned — unknown files, stale versions, or deleted segments past retention — no amount of codec tuning is the right fix. That storage is not holding your data. See What “Orphaned” includes.

​
Lever 1: retention and tiering

The cheapest byte is the one you are not storing.
  • Retention. Confirm the table has a retention period and that it matches what the data is actually used for. A table with no retention configured fails the SEGMENT_RETENTION_CHECK health check and grows until it hits its quota.
  • Tiered storage. Move cold data to object storage rather than keeping it on server disks. This changes where the bytes live and what they cost, without deleting anything. See Tiered storage.
Both of these usually move more bytes than any encoding change, and neither requires touching the schema. Do them first.

​
Lever 2: segment sizing

Segment size affects both storage and query cost. Many small segments compress worse than the same rows in fewer segments — compression works on blocks, and small blocks give it less to work with — while also multiplying per-segment metadata and per-query overhead. Signals that this is your problem:
  • SEGMENT_SIZE_CHECK failing (more than a quarter of segments under 5 MB)
  • SEGMENT_COUNT_CHECK failing (more than 50,000 segments)
  • A large gap between Avg Segment Size and the p90/p99 figures on the Storage tab
What to do:
  • For real-time tables, raise the flush threshold so segments seal larger. See Configuring the segment threshold.
  • For existing small segments, merge them — real-time or offline.
  • For offline tables, review push granularity: a daily push of a small table produces small segments by construction.

​
Lever 3: encoding — dictionary or raw

Every column is stored either dictionary-encoded or as raw values, and this choice matters more than the codec applied on top of it. The failure mode to look for is a high-cardinality string column left dictionary-encoded — request IDs, URLs, free-text fields, UUIDs. The dictionary is large, poorly compressible, and held in memory. Switching such a column to raw encoding with a codec is frequently the single largest saving available on a wide table. Dictionary encoding is also a prerequisite for some index types, so check what else a column is used for before switching it. See Dictionary index and Forward index.

​
Lever 4: compression codec

Codecs are set per column through fieldConfigList. See Forward index for the configuration syntax; this section is about which one to pick.

​
Codecs for raw-encoded columns

The trade-off is always the same shape: better ratio costs decompress CPU on every query that reads the column. That is why “which codec” cannot be answered without knowing whether the column is read by your latency-critical queries or only by occasional ad-hoc ones — the same table can justify ZSTANDARD on one column and LZ4 on another.

​
Specialized codecs

The forward block index has its own compression configuration with its own codec names. Do not assume a name from one applies to the other.

​
Lever 5: prune indexes you are not using

Each auxiliary index is additional storage. It is worth periodically asking, per index, which query needs it — an inverted index added during a past investigation and never removed is pure cost. Two things help you answer that:
  • system_query_log shows the predicates your queries actually use. Columns nobody filters on do not need filter indexes.
  • numEntriesScannedInFilter on those queries tells you whether an index is being used effectively. A filter index that is not reducing scanned entries is not earning its bytes.
The same logic applies in reverse: a column that is scanned heavily might be worth more index bytes. Storage efficiency is not the only objective — the goal is knowing what each byte buys.

​
Measuring a change

The only credible answer about compression is one measured on your data. The method:
1

Record the baseline

From the table’s Storage tab, note Used, Avg Segment Size, and the segment-size percentiles. For a per-column view, pull segment metadata for a representative segment through the segment APIs — per-column index sizes are what tell you whether the column you changed is the one that moved.
2

Change one thing

One column, or one codec, at a time. Changing an encoding and a codec together leaves you unable to attribute the result — and the two interact, so the combined effect is not the sum of the separate ones.
3

Reload, and wait for it to finish

Configuration changes apply to new segments immediately and to existing segments only on reload. Until you reload, the measurement is of the old layout. Use Reload all segments — or the Reload segments remediation on the TABLE_SEGMENTS_RELOAD_CHECK health check — and confirm completion before measuring. For larger structural changes, see the Alter Table task.
4

Measure both sides

Compare storage and query latency. A codec change that saves 30% of a column’s bytes and adds 15 ms to your p99 may or may not be a win — that depends on your workload, and you cannot tell from the storage number alone. Check p95 and p99 for the queries that read the column before and after.
5

Test on a copy where the stakes are high

For a large production table, validate on a table built from a representative subset of the same data rather than iterating on the production table. Reloads on a large table are not free, and reverting is another reload.
Compression ratios reported elsewhere — in benchmarks, in other companies’ blog posts, in Pinot’s own documentation — are properties of the data they were measured on, not of the codec. A codec that halves one team’s log column may gain a few percent on your numeric one. Treat published ratios as an ordering hint, never as a projection for your table.