How a column gets smaller
Each column’s values live in its forward index. The column’s encoding decides how that index is built.
A sorted column that’s dictionary-encoded is stored as runs of the same value instead of one ID per row, which is smaller still. See Sorted forward index.
Dictionary encoding pays off when values repeat. When nearly every value is distinct, the dictionary is as large as the data and the IDs come on top, so the column can end up bigger than its values. The worked example shows both cases.
What’s supported
Codecs
Set a codec per column withcompressionCodec.
Codec pipelines for numbers
Starting with StarTree Cloud 0.18, a raw single-valueINT or LONG column can use codecSpec instead of compressionCodec. A codec spec is a comma-separated list of stages, applied in order:
- Transforms:
DELTA(stores differences between consecutive values),DELTADELTA(differences of differences),T64,GORILLA. Use at most one ofT64andGORILLA, as the last transform. - Compression (optional):
ZSTD(also writtenZSTANDARD, optionally with a level such asZSTD(5)),LZ4,SNAPPY,GZIP
"DELTA,ZSTD". DELTA and DELTADELTA suit monotonic timestamps, sequence numbers and counters. On values that jump around they do little or make things worse. StarTree Cloud 0.17 accepts a codecSpec in the table config but doesn’t use it, so set one only once the cluster runs 0.18.
Not supported
- A general codec on a dictionary-encoded column. Table config validation rejects it:
Compression codec: SNAPPY is not applicable to dictionary encoded column: <column>. Switch the column to RAW, and remove any index that needs a dictionary, such as an inverted or FST index. A RAW column keeps its dictionary while such an index is configured, and the codec is still rejected. DELTAorDELTADELTAas acompressionCodecvalue. They’re rejected on every column. UsecodecSpecon StarTree Cloud 0.18 and later.codecSpecon anything other than a raw single-valueINTorLONGcolumn, or together withcompressionCodecon the same column.- Codecs in consuming segments. A realtime table’s consuming segment keeps raw columns in plain memory buffers, and only the CLP codecs apply there. The configured codec takes effect when the segment commits. Until then, a raw column that’s multi-value
STRINGorBYTES, or that has an inverted index, is held with a dictionary. The committed segment is built raw, as configured. - The forward index codec on other indexes.
compressionCodecandcodecSpecapply to the forward index only. The star-tree index sets its own codec for each aggregation inaggregationConfigs. Inverted, range, JSON and text indexes have no codec setting. - External Tables. Their data stays in object storage in the files’ own format, so Pinot codecs don’t apply, and the Data Portal has no Compression sub-tab for them.
Defaults
The cluster default applies only to segments built after it changed. Existing segments keep the codec they were written with, and reloading doesn’t rewrite them.
Tables created in the Data Portal start with a config that sets these explicitly.
JSON columns and columns with a text index are RAW with ZSTANDARD. Other dimension columns are DICTIONARY with an inverted index, using a variable-length dictionary for STRING and BYTES. Metric and primary key columns are DICTIONARY.
The raw index’s on-disk format version also affects size. See Raw forward index format V6.
Configure compression
Set encoding and codec per column infieldConfigList. Put encodingType on the field itself, not inside indexes.forward, or validation fails with a conflicting encoding error.
- Raw with a codec
- Dictionary
- Codec pipeline (0.18+)
- Log messages
"compressionCodec": "ZSTANDARD" directly on the field, still works. Use one form per column, not both.tableIndexConfig.noDictionaryColumns is the older way to mark columns RAW. Don’t list a column there if it also has a fieldConfigList entry, because that entry’s encoding defaults to DICTIONARY and the two conflict.
In the Data Portal
- Encoding: on the table’s Indexes tab, select Edit and pick the column’s Encoding. None means RAW. The other options (Dictionary, On heap dictionary, Variable length dictionary, and On heap and variable length dictionary) are dictionary-encoded. When you edit several columns at once, the bulk-edit bar points out combinations that won’t work, such as a codec that can’t be used with a dictionary. The table config is validated when you save.
- Codec: edit the column’s entry in the table config from the Configuration tab.
Apply a change to existing segments
New segments use the new config straight away. Existing segments change only when you reload the table. After you save, the Data Portal offers to reload, or you can use Reload all segments. A reload rewrites a segment’s forward index for:- a change of
compressionCodecfrom one codec to another - a change to or removal of
codecSpec - a switch between DICTIONARY and RAW
- removing
compressionCodecso the column falls back to the default - a change of the cluster’s default codec
- a change of
rawIndexWriterVersion
Measure compression in the Data Portal
Open the table, go to Storage, and select the Compression sub-tab. It’s available for Pinot tables on StarTree Cloud 0.16 and later, and for tables with tiered storage on 0.17 and later. Compression stats describes each element of the tab. This section covers how the numbers are calculated.Turn stats on
Stats are off by default. Record Compression Stats on the sub-tab turns them on by settingtableIndexConfig.compressionStatsEnabled to true. Pinot records stats when it builds a segment, so:
- segments built from then on are measured
- existing segments aren’t backfilled
- a reload that rewrites a column’s forward index, such as one after a codec change, measures that column on the older segments
What each figure counts
All sizes are per replica. When a segment has several replicas, one is read. On a hybrid table, every figure is for the part picked in the Offline / Realtime toggle.
Uncompressed is measured per row, so a value that repeats in a million rows counts a million times:
Bytes are added up before dividing. The table’s ratio isn’t the average of the column ratios, so large columns dominate it.
Three things keep segments or columns out of the figures:
- Consuming segments are counted in the total but are never measured, so a realtime table always shows partial coverage while it ingests.
- A segment that lacks stats for any column is left out of the table summary. Its columns still appear in the per-column figures, which is why a column’s Measured count can be higher than Segments measured.
- Columns without a forward index are left out.
Worked example
Take one segment of 10 million rows with five columns. We generated the rows with a script and worked out every size from them:- Dictionary columns: the sizes follow from the rules above.
ZSTANDARDandPASS_THROUGHcolumns: we compressed the values the way the raw forward index does. That’s chunks of 1,024 values for fixed-width types and about 36 KB for the strings, with ZSTD at its default level, in the V6 raw format that new clusters use. On the older V4 format,event_idtakes about 233 MB instead of 188 MB, because V4’s chunk headers compress worse.
LONG column that’s 10 million rows × 8 bytes = 80 MB.
Reading the rows:
countryneeds 8 bits per row to number 200 values, so 10 million rows take 10 MB of IDs. The dictionary adds about 2.4 KB. 81 MB of values become 10 MB.user_hashhas 10 million distinct values, so each ID needs 24 bits (30 MB), and the dictionary holds every value again (80 MB). The column takes more space than its values, which the Data Portal shows as a ratio below 1×.event_idcomes to 1.9×, and no codec will do much better. A random UUID carries about 15 bytes of information spread over 36 characters.event_timecompresses 7× withZSTANDARDalone, because neighboring timestamps share their high bytes.latency_msisn’t compressed: it’s set to RAW here, and raw metric columns default toPASS_THROUGH. Left on the default dictionary, its 1,466 distinct values would need 11 bits per row, about 13.8 MB.- The table ratio is 641.1 ÷ 359.8, about 1.8×. The average of the five column ratios would be about 3.75×, which overstates what the table saves.
user_hash to RAW with ZSTANDARD and reload. Random 64-bit values don’t compress, so the column stores about its own 80 MB:
The table drops about 30 MB and the ratio goes from 1.8× to 1.9×. The column loses its dictionary, though, and with it any index that needs one, such as the inverted index. Check what your queries do with the column before you switch. Storage efficiency covers that trade-off.
The same rows as source files
Uncompressed doesn’t depend on how the data arrived. The table below shows the same 10 million rows written as Parquet by pyarrow 24 with its default settings, once with each of the two common page codecs, next to Pinot’s figures:
As JSON lines, the same rows take 1,486.6 MB.
- Parquet files are already encoded and compressed, so most Parquet columns come in under Pinot’s uncompressed figure.
latency_mshas about 1,500 distinct values, so Parquet dictionary-encodes it to 12.6 MB, a third of its 40 MB of values. - Parquet drops a column’s dictionary once it grows past a size limit (1 MB per column chunk by default in pyarrow) and writes plain values instead. That’s why
user_hashis about 83 MB in Parquet but 110 MB in Pinot, which keeps the full dictionary. - The page codec matters as much as the format. Snappy barely shrinks random hex, so
event_idstays at 357.6 MB. ZSTD takes it to 198.4 MB, close to Pinot’s 188.3 MB on V6. - Against the source files, Pinot’s 359.8 MB is 4.1× smaller than JSON lines and 1.4× smaller than Snappy Parquet, but about 14% larger than ZSTD Parquet. None of those is the 1.8× the Data Portal reports, because the Data Portal divides by Pinot’s uncompressed figure, never by the source.
- Two settings in this example account for most of the gap to ZSTD Parquet.
user_hashkeeps its dictionary, andlatency_msis set to RAW, where metrics get no compression. Withlatency_mson its default dictionary, Pinot’s total would be about 333 MB, about 5% larger than ZSTD Parquet. On the V4 raw format the total would be about 405 MB instead.
Uncompressed size isn’t source data size
Uncompressed measures values after Pinot has read them into columns, not the data you ingested. The worked example shows the gap for whole columns. Value by value, the two can differ in either direction:
JSON input is usually larger than the Pinot figure, because it repeats field names and writes numbers as text. Avro and Protobuf input can be smaller, because they store numbers in as few bytes as they need. Parquet input is already encoded and compressed. Source fields that aren’t in the schema, derived columns, and rows dropped by an ingestion filter widen the gap further.
The same applies when you compare with another database. Some measure their uncompressed size from their own on-disk format, after their own dictionary encoding, so the same data can report a different ratio there. To compare two systems fairly, divide the same input size, such as the raw input files, by each system’s bytes on disk, and count the same structures on both sides.
Read the stats through the API
The Data Portal reads the table size endpoint with per-column stats turned on:offlineSegments and a realtimeSegments part. Each carries a compressionStats summary and a columnCompressionStats list. If the worked example’s segment belonged to a realtime table, the summary would look like this:
totalSegments is 2 because the table also has a consuming segment, which is never measured. Each columnCompressionStats entry has the column’s uncompressedValueSizeInBytes and forwardIndexAndDictionaryStorageSizeInBytes, plus numSegments and an encodingBreakdown list with one entry per encoding and chunk codec found on disk. A column whose compressionCodec changed, on a table where older segments weren’t rebuilt, has more than one entry. Apache Pinot’s compression stats page documents the response fields.
Related
- Table Storage Usage: the Storage tab, including the Compression sub-tab
- Storage efficiency: which changes save the most, and how to measure them
- Forward index: raw index settings and format versions
- Dictionary index: when dictionary encoding pays

