> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Compression

> How StarTree stores each column compactly, which encodings and codecs are supported, how to configure them, and how the Data Portal calculates a table's compression ratio.

StarTree stores every column of a segment separately, and each column shrinks in one of two ways: a dictionary, or a compression codec applied to the raw values. Which way you pick, and the data itself, decide how small the column gets. The data matters as much as the setting: a column with 200 distinct values shrinks a lot, and a column of random IDs barely shrinks at all.

This page covers what's supported, how to configure it, and how the Data Portal measures the result. For deciding which lever to pull on a large table, see [Storage efficiency](https://docs.startree.ai/corecapabilities/manage-data/storage-efficiency.md).

## How a column gets smaller

Each column's values live in its [forward index](https://docs.startree.ai/corecapabilities/manage-data/indexes/forward-index.md). The column's encoding decides how that index is built.

| Encoding | What's stored | How it saves space |
| :- | :- | :- |
| **DICTIONARY** (default) | Each distinct value once, in a dictionary, plus one dictionary ID per row | IDs are bit-packed to the fewest bits that can count the distinct values: 200 distinct values need 8 bits per row, and 10 million need 24. No codec runs on top. |
| **RAW** | The values themselves, in chunks | Each chunk is compressed with the column's codec, such as `ZSTANDARD` or `LZ4`. |

A sorted column that's dictionary-encoded is stored as runs of the same value instead of one ID per row, which is smaller still. See [Sorted forward index](https://docs.startree.ai/corecapabilities/manage-data/indexes/forward-index.md#sorted-forward-index).

Dictionary encoding pays off when values repeat. When nearly every value is distinct, the dictionary is as large as the data and the IDs come on top, so the column can end up bigger than its values. The [worked example](#worked-example) shows both cases.

## What's supported

### Codecs

Set a codec per column with `compressionCodec`.

| Codec | Encoding | Column types | Use it for |
| :- | :- | :- | :- |
| `ZSTANDARD` | RAW | All | Best ratio of the general codecs. Costs more CPU to decompress than `LZ4`. |
| `LZ4` | RAW | All | Fast to decompress, moderate ratio. |
| `SNAPPY` | RAW | All | Similar profile to `LZ4`. |
| `GZIP` | RAW | All | High ratio, slowest to read. Not recommended: its decompressor isn't safe for concurrent queries on the same column, which is also why it can't be a cluster default. |
| `PASS_THROUGH` | RAW | All | No compression. |
| `MV_ENTRY_DICT` | DICTIONARY | Multi-value (accepted on single-value columns, but has no effect) | Adds a second dictionary over whole multi-value entries, which helps when the same combinations of values repeat across rows. The only codec that works on a dictionary-encoded column. |
| `CLP`, `CLPV2`, `CLPV2_ZSTD`, `CLPV2_LZ4` | RAW | `STRING`, and `JSON` (stored as `STRING`) | Log messages. These split each line into a template and its variables, and suit nothing else. |

### Codec pipelines for numbers

Starting with StarTree Cloud 0.18, a raw single-value `INT` or `LONG` column can use `codecSpec` instead of `compressionCodec`. A codec spec is a comma-separated list of stages, applied in order:

* **Transforms:** `DELTA` (stores differences between consecutive values), `DELTADELTA` (differences of differences), `T64`, `GORILLA`. Use at most one of `T64` and `GORILLA`, as the last transform.
* **Compression (optional):** `ZSTD` (also written `ZSTANDARD`, optionally with a level such as `ZSTD(5)`), `LZ4`, `SNAPPY`, `GZIP`

Transforms come before compression, for example `"DELTA,ZSTD"`. `DELTA` and `DELTADELTA` suit monotonic timestamps, sequence numbers and counters. On values that jump around they do little or make things worse. StarTree Cloud 0.17 accepts a `codecSpec` in the table config but doesn't use it, so set one only once the cluster runs 0.18.

### Not supported

* **A general codec on a dictionary-encoded column.** Table config validation rejects it: `Compression codec: SNAPPY is not applicable to dictionary encoded column: <column>`. Switch the column to RAW, and remove any index that needs a dictionary, such as an inverted or FST index. A RAW column keeps its dictionary while such an index is configured, and the codec is still rejected.
* **`DELTA` or `DELTADELTA` as a `compressionCodec` value.** They're rejected on every column. Use `codecSpec` on StarTree Cloud 0.18 and later.
* **`codecSpec` on anything other than a raw single-value `INT` or `LONG` column**, or together with `compressionCodec` on the same column.
* **Codecs in consuming segments.** A realtime table's consuming segment keeps raw columns in plain memory buffers, and only the CLP codecs apply there. The configured codec takes effect when the segment commits. Until then, a raw column that's multi-value `STRING` or `BYTES`, or that has an inverted index, is held with a dictionary. The committed segment is built raw, as configured.
* **The forward index codec on other indexes.** `compressionCodec` and `codecSpec` apply to the forward index only. The [star-tree index](https://docs.startree.ai/corecapabilities/manage-data/indexes/startree-index.md) sets its own codec for each aggregation in `aggregationConfigs`. Inverted, range, JSON and text indexes have no codec setting.
* **External Tables.** Their data stays in object storage in the files' own format, so Pinot codecs don't apply, and the Data Portal has no Compression sub-tab for them.

## Defaults

| Situation | What StarTree uses |
| :- | :- |
| Column with no `encodingType` set | DICTIONARY |
| RAW column with no codec set | `ZSTANDARD` on StarTree Cloud 0.18 and on up-to-date 0.17 clusters (`LZ4` before) |
| RAW metric column with no codec set | `PASS_THROUGH` |
| RAW `MAP` column with no codec set | `LZ4` |
| Segments built outside the cluster, for example by a Spark or Hadoop job, with no codec set | `LZ4` |

The cluster default applies only to segments built after it changed. Existing segments keep the codec they were written with, and reloading doesn't rewrite them.

Tables created in the Data Portal start with a config that sets these explicitly. `JSON` columns and columns with a text index are RAW with `ZSTANDARD`. Other dimension columns are DICTIONARY with an inverted index, using a variable-length dictionary for `STRING` and `BYTES`. Metric and primary key columns are DICTIONARY.

The raw index's on-disk format version also affects size. See [Raw forward index format V6](https://docs.startree.ai/corecapabilities/manage-data/indexes/forward-index.md#raw-forward-index-format-v6).

## Configure compression

Set encoding and codec per column in `fieldConfigList`. Put `encodingType` on the field itself, not inside `indexes.forward`, or validation fails with a conflicting encoding error.

> [tab] Raw with a codec
>
> ```json theme={null}
> {
>   "fieldConfigList": [
>     {
>       "name": "event_id",
>       "encodingType": "RAW",
>       "indexes": {
>         "forward": {
>           "compressionCodec": "ZSTANDARD"
>         }
>       }
>     }
>   ]
> }
> ```
>
> The older form, with `"compressionCodec": "ZSTANDARD"` directly on the field, still works. Use one form per column, not both.

> [tab] Dictionary
>
> ```json theme={null}
> {
>   "fieldConfigList": [
>     {
>       "name": "country",
>       "encodingType": "DICTIONARY"
>     },
>     {
>       "name": "tags",
>       "encodingType": "DICTIONARY",
>       "indexes": {
>         "forward": {
>           "compressionCodec": "MV_ENTRY_DICT"
>         }
>       }
>     }
>   ]
> }
> ```
>
> `tags` is a multi-value column. `MV_ENTRY_DICT` is optional, and it only takes effect on multi-value columns.

> [tab] Codec pipeline (0.18+)
>
> ```json theme={null}
> {
>   "fieldConfigList": [
>     {
>       "name": "event_time",
>       "encodingType": "RAW",
>       "indexes": {
>         "forward": {
>           "codecSpec": "DELTA,ZSTD"
>         }
>       }
>     }
>   ]
> }
> ```
>
> The column must be a single-value `INT` or `LONG` (a `TIMESTAMP` is stored as `LONG`).

> [tab] Log messages
>
> ```json theme={null}
> {
>   "fieldConfigList": [
>     {
>       "name": "message",
>       "encodingType": "RAW",
>       "indexes": {
>         "forward": {
>           "compressionCodec": "CLPV2_ZSTD"
>         }
>       }
>     }
>   ]
> }
> ```

`tableIndexConfig.noDictionaryColumns` is the older way to mark columns RAW. Don't list a column there if it also has a `fieldConfigList` entry, because that entry's encoding defaults to DICTIONARY and the two conflict.

### In the Data Portal

* **Encoding:** on the table's **Indexes** tab, select **Edit** and pick the column's **Encoding**. **None** means RAW. The other options (**Dictionary**, **On heap dictionary**, **Variable length dictionary**, and **On heap and variable length dictionary**) are dictionary-encoded. When you edit several columns at once, the bulk-edit bar points out combinations that won't work, such as a codec that can't be used with a dictionary. The table config is validated when you save.
* **Codec:** edit the column's entry in the table config from the **Configuration** tab.

### Apply a change to existing segments

New segments use the new config straight away. Existing segments change only when you reload the table. After you save, the Data Portal offers to reload, or you can use [Reload all segments](https://docs.startree.ai/api-reference/segment/reload-all-segments.md).

A reload rewrites a segment's forward index for:

* a change of `compressionCodec` from one codec to another
* a change to or removal of `codecSpec`
* a switch between DICTIONARY and RAW

A reload doesn't apply:

* removing `compressionCodec` so the column falls back to the default
* a change of the cluster's default codec
* a change of `rawIndexWriterVersion`

Those reach existing segments only when they're rebuilt, for example by a minion task or a re-push. For larger rewrites, see the [Alter Table task](https://docs.startree.ai/corecapabilities/manage-data/alter-table-task.md).

## Measure compression in the Data Portal

Open the table, go to **Storage**, and select the **Compression** sub-tab. It's available for Pinot tables on StarTree Cloud 0.16 and later, and for tables with tiered storage on 0.17 and later. [Compression stats](https://docs.startree.ai/corecapabilities/manage-data/table-storage-usage.md#compression-stats) describes each element of the tab. This section covers how the numbers are calculated.

### Turn stats on

Stats are off by default. **Record Compression Stats** on the sub-tab turns them on by setting `tableIndexConfig.compressionStatsEnabled` to `true`. Pinot records stats when it builds a segment, so:

* segments built from then on are measured
* existing segments aren't backfilled
* a reload that rewrites a column's forward index, such as one after a codec change, measures that column on the older segments

Turn stats on before you change anything, so you have a baseline to compare against.

### What each figure counts

| Data Portal figure | Calculation |
| :- | :- |
| **Uncompressed** | The column values at Pinot's own width for their type. See the next table. |
| **Stored** | The forward index file plus, for a dictionary-encoded column, the dictionary file. Nothing else. |
| **Compression ratio** | Total **Uncompressed** ÷ total **Stored**, over the measured segments. |
| **Share of stored** | A column's **Stored** ÷ the total **Stored** of all the columns in the grid. |
| **Segments measured** | Segments with stats for every column ÷ all segments in the table. |

All sizes are per replica. When a segment has several replicas, one is read. On a hybrid table, every figure is for the part picked in the **Offline** / **Realtime** toggle.

**Uncompressed** is measured per row, so a value that repeats in a million rows counts a million times:

| Column type | Bytes per value |
| :- | :- |
| `INT`, `FLOAT` | 4 |
| `LONG`, `DOUBLE`, `TIMESTAMP` | 8 |
| `BOOLEAN` | 4 (stored as `INT`) |
| `STRING` | The value's length in UTF-8 bytes |
| `BYTES` | The value's length |
| Multi-value | The sum over every entry in the row. Raw multi-value `STRING` and `BYTES` columns also count 4 bytes per row and 4 bytes per entry for lengths. Raw fixed-width multi-value columns on the V4 format count 4 bytes per row. |

Bytes are added up before dividing. The table's ratio isn't the average of the column ratios, so large columns dominate it.

Three things keep segments or columns out of the figures:

* **Consuming segments** are counted in the total but are never measured, so a realtime table always shows partial coverage while it ingests.
* **A segment that lacks stats for any column** is left out of the table summary. Its columns still appear in the per-column figures, which is why a column's **Measured** count can be higher than **Segments measured**.
* **Columns without a forward index** are left out.

On an upsert table the figures count every row stored in the segments, including rows a later upsert replaced.

### Worked example

Take one segment of 10 million rows with five columns. We generated the rows with a script and worked out every size from them:

* **Dictionary columns:** the sizes follow from the rules above.
* **`ZSTANDARD` and `PASS_THROUGH` columns:** we compressed the values the way the raw forward index does. That's chunks of 1,024 values for fixed-width types and about 36 KB for the strings, with ZSTD at its default level, in the V6 raw format that new clusters use. On the older V4 format, `event_id` takes about 233 MB instead of 188 MB, because V4's chunk headers compress worse.

The figures are reproduced from Pinot's segment format rather than read from a running cluster, so expect small differences on real data. Sizes are in MB (millions of bytes), and totals are added up before rounding.

**Uncompressed** is the figure the Data Portal shows: each column's values at Pinot's own width, from the per-type table above. For a `LONG` column that's 10 million rows × 8 bytes = 80 MB.

| Column | Data | Encoding | Uncompressed | Stored | Ratio |
| :- | :- | :- | -: | -: | -: |
| `country` | `STRING`, 200 distinct values, 8 bytes on average | DICTIONARY | 81.1 MB | 10.0 MB | 8.1× |
| `user_hash` | `LONG`, random, every value distinct | DICTIONARY | 80.0 MB | 110.0 MB | 0.7× |
| `event_id` | `STRING`, a random 36-character UUID | RAW · ZSTANDARD | 360.0 MB | 188.3 MB | 1.9× |
| `event_time` | `TIMESTAMP`, rising 0 to 20 ms per row | RAW · ZSTANDARD | 80.0 MB | 11.4 MB | 7.0× |
| `latency_ms` | `INT` metric, set to RAW, about 1,500 distinct values | RAW · PASS\_THROUGH | 40.0 MB | 40.1 MB | 1.0× |
| **Table** | | | **641.1 MB** | **359.8 MB** | **1.8×** |

Reading the rows:

* **`country`** needs 8 bits per row to number 200 values, so 10 million rows take 10 MB of IDs. The dictionary adds about 2.4 KB. 81 MB of values become 10 MB.
* **`user_hash`** has 10 million distinct values, so each ID needs 24 bits (30 MB), and the dictionary holds every value again (80 MB). The column takes more space than its values, which the Data Portal shows as a ratio below 1×.
* **`event_id`** comes to 1.9×, and no codec will do much better. A random UUID carries about 15 bytes of information spread over 36 characters.
* **`event_time`** compresses 7× with `ZSTANDARD` alone, because neighboring timestamps share their high bytes.
* **`latency_ms`** isn't compressed: it's set to RAW here, and raw metric columns default to `PASS_THROUGH`. Left on the default dictionary, its 1,466 distinct values would need 11 bits per row, about 13.8 MB.
* **The table ratio** is 641.1 ÷ 359.8, about 1.8×. The average of the five column ratios would be about 3.75×, which overstates what the table saves.

Now switch `user_hash` to RAW with `ZSTANDARD` and reload. Random 64-bit values don't compress, so the column stores about its own 80 MB:

| Column | Encoding | Uncompressed | Stored | Ratio |
| :- | :- | -: | -: | -: |
| `user_hash` | RAW · ZSTANDARD | 80.0 MB | 80.2 MB | 1.0× |
| **Table** | | **641.1 MB** | **329.9 MB** | **1.9×** |

The table drops about 30 MB and the ratio goes from 1.8× to 1.9×. The column loses its dictionary, though, and with it any index that needs one, such as the inverted index. Check what your queries do with the column before you switch. [Storage efficiency](https://docs.startree.ai/corecapabilities/manage-data/storage-efficiency.md) covers that trade-off.

#### The same rows as source files

**Uncompressed** doesn't depend on how the data arrived. The table below shows the same 10 million rows written as Parquet by pyarrow 24 with its default settings, once with each of the two common page codecs, next to Pinot's figures:

| Column | Pinot uncompressed | Parquet, Snappy | Parquet, ZSTD | Pinot stored |
| :- | -: | -: | -: | -: |
| `country` | 81.1 MB | 10.1 MB | 10.1 MB | 10.0 MB |
| `user_hash` | 80.0 MB | 82.8 MB | 82.6 MB | 110.0 MB |
| `event_id` | 360.0 MB | 357.6 MB | 198.4 MB | 188.3 MB |
| `event_time` | 80.0 MB | 41.8 MB | 13.7 MB | 11.4 MB |
| `latency_ms` | 40.0 MB | 12.6 MB | 11.4 MB | 40.1 MB |
| **Total** | **641.1 MB** | **504.8 MB** | **316.3 MB** | **359.8 MB** |

As JSON lines, the same rows take 1,486.6 MB.

* **Parquet files are already encoded and compressed,** so most Parquet columns come in under Pinot's uncompressed figure. `latency_ms` has about 1,500 distinct values, so Parquet dictionary-encodes it to 12.6 MB, a third of its 40 MB of values.
* **Parquet drops a column's dictionary once it grows past a size limit** (1 MB per column chunk by default in pyarrow) and writes plain values instead. That's why `user_hash` is about 83 MB in Parquet but 110 MB in Pinot, which keeps the full dictionary.
* **The page codec matters as much as the format.** Snappy barely shrinks random hex, so `event_id` stays at 357.6 MB. ZSTD takes it to 198.4 MB, close to Pinot's 188.3 MB on V6.
* **Against the source files, Pinot's 359.8 MB is 4.1× smaller than JSON lines and 1.4× smaller than Snappy Parquet, but about 14% larger than ZSTD Parquet.** None of those is the 1.8× the Data Portal reports, because the Data Portal divides by Pinot's uncompressed figure, never by the source.
* **Two settings in this example account for most of the gap to ZSTD Parquet.** `user_hash` keeps its dictionary, and `latency_ms` is set to RAW, where metrics get no compression. With `latency_ms` on its default dictionary, Pinot's total would be about 333 MB, about 5% larger than ZSTD Parquet. On the V4 raw format the total would be about 405 MB instead.

### Uncompressed size isn't source data size

**Uncompressed** measures values after Pinot has read them into columns, not the data you ingested. The worked example shows the gap for whole columns. Value by value, the two can differ in either direction:

| Value | Pinot uncompressed | In JSON | In Avro |
| :- | -: | -: | -: |
| `1731542400000` as `LONG` | 8 B | 13 B, plus the field name | 6 B |
| `7` as `INT` | 4 B | 1 B, plus the field name | 1 B |
| `true` as `BOOLEAN` | 4 B | 4 B, plus the field name | 1 B |
| A 36-character UUID as `STRING` | 36 B | 38 B, plus the field name | 37 B |
| `"2026-11-17T10:00:00Z"` parsed into a `TIMESTAMP` | 8 B | 22 B, plus the field name | 21 B |

JSON input is usually larger than the Pinot figure, because it repeats field names and writes numbers as text. Avro and Protobuf input can be smaller, because they store numbers in as few bytes as they need. Parquet input is already encoded and compressed. Source fields that aren't in the schema, derived columns, and rows dropped by an ingestion filter widen the gap further.

The same applies when you compare with another database. Some measure their uncompressed size from their own on-disk format, after their own dictionary encoding, so the same data can report a different ratio there. To compare two systems fairly, divide the same input size, such as the raw input files, by each system's bytes on disk, and count the same structures on both sides.

### Read the stats through the API

The Data Portal reads the table size endpoint with per-column stats turned on:

```bash theme={null}
GET /tables/{tableName}/size?includeColumnCompressionStats=true
```

The response has an `offlineSegments` and a `realtimeSegments` part. Each carries a `compressionStats` summary and a `columnCompressionStats` list. If the worked example's segment belonged to a realtime table, the summary would look like this:

```json theme={null}
{
  "compressionStats": {
    "uncompressedValueSizePerReplicaInBytes": 641059515,
    "forwardIndexAndDictionaryStorageSizePerReplicaInBytes": 359752342,
    "compressionRatio": 1.7819467454641338,
    "segmentsWithCompleteStats": 1,
    "totalSegments": 2,
    "partialCoverage": true
  }
}
```

`totalSegments` is 2 because the table also has a consuming segment, which is never measured. Each `columnCompressionStats` entry has the column's `uncompressedValueSizeInBytes` and `forwardIndexAndDictionaryStorageSizeInBytes`, plus `numSegments` and an `encodingBreakdown` list with one entry per encoding and chunk codec found on disk. A column whose `compressionCodec` changed, on a table where older segments weren't rebuilt, has more than one entry. Apache Pinot's [compression stats](https://docs.pinot.apache.org/build-with-pinot/indexing/compression-stats) page documents the response fields.

## Related

* [Table Storage Usage](https://docs.startree.ai/corecapabilities/manage-data/table-storage-usage.md): the Storage tab, including the Compression sub-tab
* [Storage efficiency](https://docs.startree.ai/corecapabilities/manage-data/storage-efficiency.md): which changes save the most, and how to measure them
* [Forward index](https://docs.startree.ai/corecapabilities/manage-data/indexes/forward-index.md): raw index settings and format versions
* [Dictionary index](https://docs.startree.ai/corecapabilities/manage-data/indexes/dictionary-index.md): when dictionary encoding pays


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.