Table Storage Usage - StarTree Docs

Overview

Unexpected growth in table storage increases both infrastructure and object-storage costs. Pinot keeps a table’s data in several places — the Pinot servers’ local disks, the deep store, and the remote object stores used by tiered storage — and the Storage tab in the StarTree Data Portal reports the size at each location and highlights the gap between what should be there and what is actually stored. Use it to:

Track the total storage footprint across local disk, deep store, and tiered storage
Spot untracked segments and stale versions that no longer belong to the table
Find deleted segments that are past retention but still consuming deep store space
Debug discrepancies during rebalances, segment replacement, and tiered-storage cleanup

Most of what you need is on the Storage tab. If you want to script checks or drill into the specific items behind the numbers, the same data is available from the storage usage API.

The Storage tab and its API are available starting StarTree Cloud version 0.12.0.

Reading the Storage tab

Open a table in the Data Portal and select the Storage tab. Storage is grouped into up to three cards, one per physical location:

Card	What it holds
Local Storage	Segment data on the Pinot servers’ local disks — the copy that serves queries with the lowest latency.
Deep Storage	The durable backup copy of every segment in external object storage (S3, GCS, HDFS), used for recovery and replication.
Tiered Storage	Cost-optimized tiers holding colder data in remote object storage. Shown only when the table uses tiered storage.

What each number means

Every card shows the same set of figures:

Number	What it tells you	API field
Used	Total storage physically occupied at this location, across all replicas.	`actualSizeInBytes`
Orphaned	How much of Used is not backed by the table’s current data, and may be reclaimable. It bundles three things — unknown files, stale versions, and beyond-retention deleted segments (see What “Orphaned” includes).	`unexpectedSizeInBytes`
Segments	Number of segments at this location. On Local Storage this is the table’s expected segment count; on Deep and Tiered Storage it is the number of items found.	`segmentCount` / `actualItemsCount`
Unexpected Segments	How many individual segments or files are unexpected — unknown items plus beyond-retention. Stale versions add to Orphaned size but are not counted here.	`unexpectedItemsCount`
Avg Segment Size	Average size of the table’s segments, shown per server. A large gap between this and your p90/p99 sizes points to segment-size skew.	`avgSegmentSizeInBytes`

Deep Storage breakdown

The Deep Storage card expands into per-area cards so you can see which part of the deep store is using space:

Card	What it is
Online Segments	Live segments that belong to the table.
Deleted Segments	Segments in the deep store’s retention queue after deletion. Within retention they are expected; past retention they count as orphaned.
Upsert Snapshot	Snapshot files for upsert-enabled tables. Never counted as orphaned.
Dedupe Snapshot	Snapshot files for dedup-enabled tables. Never counted as orphaned.

The Deleted, Upsert, and Dedupe cards appear only when they hold data.

Understanding the numbers

Used, orphaned, and expected

Two ideas explain every figure on the tab:

Used is what is physically stored right now.
Orphaned is the part of Used that is not backed by the table’s current metadata.

What remains is what should be there — the expected size:

expected = Used − Orphaned

Storage is expected when it is a live segment recorded in the table’s metadata, or a deleted segment still inside its retention window. Everything else is orphaned.

What “Orphaned” includes

The Orphaned figure is a single number, but it bundles three different situations, each with a different cause and a different fix:

Category	What it is
Unknown items	Files on storage with no record in the table at all — also called untracked segments. Usually left by interrupted operations, failed uploads, or manual copies.
Stale versions	Older copies of a segment that is still current, kept alongside the latest version when version cleanup lags.
Beyond-retention segments	Deleted segments still in the deep store past their retention deadline.

“Orphaned” means the combined total, not one category. The Storage tab shows a single number labeled Orphaned for all three of the above added together. This trips people up because “orphan segment” is also used loosely to mean just the first category — files with no metadata. Throughout this page, Orphaned always refers to the combined Data Portal total; the no-metadata files are called unknown items (the API’s unknownItems). A large Orphaned figure can therefore be entirely stale versions or beyond-retention segments, with zero unknown items. The API calls this same combined total unexpected; to see the split, call it at verbose=2 (see Using the storage usage API).

Stale versions count toward the Orphaned size but not toward the Unexpected Segments count. This is why a table can show a large Orphaned size with zero Unexpected Segments — the excess is entirely stale versions.

Why these numbers differ from the old Pinot table size

If you are used to the OSS Pinot table size API (GET /tables/{tableName}/size, shown as Reported Size and Estimated Size in the Pinot console), the numbers here will look different, and usually larger. That is expected: the two answer different questions.

	Pinot `getTableSize`	Table Storage Usage (this page)
Question	How big is the table as served on Pinot servers?	How many bytes is the table physically storing, everywhere?
Where it looks	Server local disk only — it polls each server for the on-disk size of the segments it hosts	Server local disk and deep store and every tiered/remote object store, each reported separately
What counts	Only segments in the table’s ideal state (currently served)	Every object found at each location, including orphaned, stale, deleted-but-within-retention, and snapshot data
Replicas	`reportedSizeInBytes` sums all replicas; `reportedSizePerReplicaInBytes` is a single copy	Local Storage Used sums all replicas’ on-disk bytes
Unreachable servers	Estimates the missing size from other replicas (`estimatedSizeInBytes` ≥ `reportedSizeInBytes`)	Lists the actual objects present at each location; no estimation

How getTableSize computes Reported Size. For each served segment it polls every server that hosts a replica and reads that replica’s on-disk size:

size(s, r) = on-disk bytes of replica r of segment s, as reported by its server

Reported Size              =  Σ segments s  Σ reporting replicas r   size(s, r)
Reported Size Per Replica  =  Σ segments s   max over replicas r   size(s, r)      (≈ one full copy)
Estimated Size             =  Reported Size  +  Σ segments s ( unreachable replicas of s  ×  largest replica of s )

In words: Reported Size adds up every replica that actually responded — roughly one copy’s size × replication factor. Replicas on an unreachable server are left out of Reported Size and only approximated in Estimated Size (so Estimated ≥ Reported), a segment whose replicas all fail to report is flagged missing and contributes nothing, and nothing outside the Pinot servers is ever counted — no deep store, no tiered storage. Mapping the old numbers:

The old Reported Size maps most closely to Local Storage → Used: both sum the on-disk size of served segments across all replicas. Local Storage can be slightly higher because it also includes the tiered-storage preload and metadata caches and any orphaned local files.
The old Reported Size Per Replica ≈ Local Storage → Used ÷ replication factor.
Deep Storage and Tiered Storage have no counterpart in getTableSize. This is usually why the totals look much larger than before: the deep store keeps a durable copy of every segment, and tiered storage holds cold data that servers no longer keep fully on local disk. The old API never counted either.

The two views are complementary. Use getTableSize for “how much data is my query tier serving,” and the Storage tab for “what is my total footprint, and how much of it is reclaimable.” A higher total does not mean the table grew — it means you can now see storage that was always there but previously invisible to the table size API (deep store, tiered storage, and orphaned data).

What to do when Orphaned is high

The Storage tab and the API report the Orphaned total but never delete it. What to do depends on which of the three categories it is made of, so start by identifying the category rather than acting on the headline number.

The Orphaned number bundles all three categories — unknown items, stale versions, and beyond-retention segments. The Storage tab does not break it down; call the API with verbose=2 and inspect the unexpected object to see which one dominates before taking any action.

Recommended workflow

Rule out transient growth first. During and shortly after a rebalance, segment replacement, or heavy ingestion, storage legitimately exceeds the expected size because of overlapping placement and intermediate artifacts. Re-check after the operation settles, and across a couple of the controller’s periodic cycles, before treating anything as a real orphan.
Identify the exact items and their category with verbose=2. The remedy is driven entirely by category.
Apply the matching remedy:

Category (`unexpected.*`)	Typical cause	Remedy
`beyondRetentionSegments`	Deleted-segment cleanup is not keeping up	The controller’s retention job purges these automatically. A persistent non-zero count means retention is not running or is lagging — verify the deleted-segment retention configuration and that the controller retention task is active.
`staleVersions`	Re-indexed or reconfigured segments leaving older versions behind	Run Deep Store Sync (the Alter Table Task in `reloadOnly` mode) and enable tiered-storage version cleanup so superseded versions age out.
`unknownItems`	Interrupted ingestion or rebalance, failed uploads, `.tmp` / `.tar.compressed` leftovers, or manually copied files	Transient artifacts on server local disk usually clear on the next segment reload or server restart. For confirmed, persistent items in the deep store or remote object store, contact StarTree support to reconcile them.

Do not manually delete objects from the deep store or remote object store based on this list alone — a wrong deletion removes live data. Confirm an item is genuinely unreferenced, and for deep store or object-store orphans prefer StarTree support over manual deletion.

Preventing recurrence

Most orphaned growth traces back to interrupted work. To keep it from returning:

Avoid stopping ingestion, rebalance, or segment-refresh operations mid-run.
Right-size batch ingestion so failed uploads are rare; for External Tables, aim for larger source files.
Monitor the Orphaned figure so growth is caught while it is small, rather than after it accumulates.

Glossary

Unknown items

Segments or files physically present on storage with no matching entry in Pinot (ZooKeeper) metadata — also called untracked or orphan segments. In the API these are unexpected.unknownItems. In the Data Portal they contribute to both Orphaned (size) and Unexpected Segments (count). Common causes: interrupted ingestion or rebalance, failed deep store uploads, and manually copied files.

Stale versions

Older copies of a segment that is still part of the table, kept alongside the current version in tiered/remote storage or in the server’s preload/metadata caches. A version string encodes the segment CRC and a hash of the table config and schema, so changing the schema or config, or re-indexing a segment, produces a new version; any version on storage that does not match the current one is stale. Reported in totalStaleVersions and unexpected.staleVersions; their bytes count toward Orphaned but not toward Unexpected Segments.

Beyond-retention segments

Deleted segments that remain in the deep store Deleted_Segments area past their retention deadline. The deadline comes from the __RETENTION_UNTIL__<yyyyMMddHHmm> suffix on the file name, or, if absent, from the last-modified time plus the configured deleted-segment retention. Reported in unexpected.beyondRetentionSegments. A persistent non-zero count usually means the retention/cleanup job is not running.

Online segments

Live segments that belong to the table, stored in the deep store under <data.dir>/<tableNameWithType>. These make up the bulk of expected deep store size.

Deleted segments

Segments moved to the deep store retention queue at <data.dir>/Deleted_Segments/<rawTableName>. Those within retention are expected; those past retention are counted as beyond-retention (orphaned).

Upsert / dedup snapshots

Snapshot files written by upsert- or dedup-enabled tables, stored in the deep store. Always reported as expected (never counted as orphaned).

Preload index & metadata cache

Server-local caches used by tiered storage. The preload index stages remote-tier segment indexes for low-latency serving; the metadata cache holds cached metadata for remote-tier segments. Both are keyed by segment version, so both are checked for stale versions.

Segment size stats (p50 / p90 / p99 / avg)

Percentile and average segment sizes computed over the table’s expected segments. Returned per server (segmentSizeStats) and aggregated across all servers (serverAggregateSegmentSizeStats). A large gap between p50 and p90/p99 indicates size skew.

Using the storage usage API

The Storage tab is powered by a controller endpoint. Most users never need to call it directly. Reach for the API when you want to script storage checks, feed a dashboard, or drill into the specific orphaned or stale items behind the aggregate numbers — the Data Portal shows the totals but not the individual item names. The endpoint provides a consolidated view for a table across server instances, deep store, and the remote object store, aggregating results collected from each server.

GET /tables/{tableName}/storageUsage?verbose={0|1|2}

tableName — table name with type, for example myTable_OFFLINE or myTable_REALTIME.
verbose — verbosity level (integer, default 0):

Level	Contents	Use for
`0`	Summary only: per-location totals and segment size stats. No `breakdown`.	Dashboards, alerting, routine monitoring.
`1`	Adds the per-sub-directory `breakdown`, with counts and sizes, but without the lists of individual unexpected items.	Locating which area is growing. This is what the Data Portal Storage tab uses.
`2`	Full breakdown including the `unexpected` item lists (specific segment/version names).	Investigating and cleaning up specific orphaned or stale items.

Listing large deep store or remote object-store prefixes can be slow and may incur object-store request charges. Use verbose=0 for routine monitoring and reserve verbose=1/2 for investigation.

The size and count invariants below always hold (and are enforced by integration tests):

unexpectedSizeInBytes  =  unknownItems.sizeInBytes
                       +  staleVersions.sizeInBytes
                       +  beyondRetentionSegments.sizeInBytes

unexpectedItemsCount   =  unknownItems.count
                       +  beyondRetentionSegments.count        (staleVersions are NOT counted)

unexpectedSizeInBytes  ≤  actualSizeInBytes

Note that staleVersions appears in the size total but not in the item count — this is exactly why a location can report non-zero Orphaned bytes with zero Unexpected Segments. Stale versions are counted separately in totalStaleVersions.

Response: summary (`verbose=0`)

Per-location scalar totals plus segment size statistics. breakdown is omitted.

Example response

{
  "tableName": "myTable_OFFLINE",
  "serverInstances": {
    "Server_pinot-server-0": {
      "actualItemsCount": 1305,
      "unexpectedItemsCount": 38,
      "totalStaleVersions": 0,
      "actualSizeInBytes": 1119418454,
      "unexpectedSizeInBytes": 253932,
      "segmentSizeStats": {
        "segmentCount": 999,
        "p50SegmentSizeInBytes": 1100000,
        "p90SegmentSizeInBytes": 1200000,
        "p99SegmentSizeInBytes": 1300000,
        "avgSegmentSizeInBytes": 1150000
      }
    }
  },
  "serverAggregateSegmentSizeStats": {
    "segmentCount": 999,
    "p50SegmentSizeInBytes": 1100000,
    "p90SegmentSizeInBytes": 1200000,
    "p99SegmentSizeInBytes": 1300000,
    "avgSegmentSizeInBytes": 1150000
  },
  "deepStore": {
    "actualItemsCount": 2890,
    "unexpectedItemsCount": 3,
    "actualSizeInBytes": 21805648,
    "unexpectedSizeInBytes": 14193
  },
  "remoteS3": {
    "s3Tier": {
      "actualItemsCount": 5,
      "unexpectedItemsCount": 0,
      "totalStaleVersions": 3,
      "actualSizeInBytes": 1331507185,
      "unexpectedSizeInBytes": 1006228697
    }
  }
}

Response: full breakdown (`verbose=2`)

verbose=1 adds a breakdown for each area but omits the unexpected item lists. verbose=2 additionally lists the individual unexpected contributors — unknown items, stale versions, and beyond-retention segments.

Example response

{
  "tableName": "fake_table_SRT_Compaction_REALTIME",
  "serverInstances": {
    "Server_pinot-server-server-0-1": {
      "actualItemsCount": 1305,
      "unexpectedItemsCount": 38,
      "totalStaleVersions": 0,
      "actualSizeInBytes": 1119418454,
      "unexpectedSizeInBytes": 253932,
      "breakdown": {
        "localSegmentsDirectory": {
          "actualItemsCount": 1305,
          "unexpectedItemsCount": 38,
          "actualSizeInBytes": 1119418454,
          "unexpectedSizeInBytes": 253932,
          "unexpected": {
            "unknownItems": {
              "count": 38,
              "sizeInBytes": 253932,
              "items": [
                "fake_table_SRT_Compaction__0__14199__20251222T2339Z.tar.compressed",
                "..."
              ]
            }
          }
        }
      }
    }
  },
  "deepStore": {
    "actualItemsCount": 2890,
    "unexpectedItemsCount": 3,
    "actualSizeInBytes": 21805648,
    "unexpectedSizeInBytes": 14193,
    "breakdown": {
      "onlineSegments": {
        "actualSizeInBytes": 19301473,
        "unexpectedItemsCount": 2,
        "unexpectedSizeInBytes": 14193,
        "unexpected": {
          "unknownItems": {
            "count": 2,
            "sizeInBytes": 14193,
            "items": [
              "fake_table_SRT_Compaction__1__162105__20260107T1819Z.tmp",
              "fake_table_SRT_Compaction__0__144799__20260107T1819Z.tmp"
            ]
          }
        }
      },
      "deletedSegments": {
        "actualItemsCount": 1,
        "unexpectedItemsCount": 1,
        "actualSizeInBytes": 1201,
        "unexpectedSizeInBytes": 1201,
        "unexpected": {
          "beyondRetentionSegments": {
            "count": 1,
            "sizeInBytes": 1201,
            "items": [
              "fake_table_SRT_Compaction__1__10702__20251222T1506Z"
            ]
          }
        }
      }
    }
  },
  "remoteS3": {
    "s3Tier": {
      "actualItemsCount": 5,
      "unexpectedItemsCount": 0,
      "totalStaleVersions": 3,
      "actualSizeInBytes": 1331507185,
      "unexpectedSizeInBytes": 1006228697,
      "breakdown": {
        "segmentDirectory": {
          "actualItemsCount": 5,
          "unexpectedItemsCount": 0,
          "totalStaleVersions": 3,
          "actualSizeInBytes": 1331507185,
          "unexpectedSizeInBytes": 1006228697,
          "unexpected": {
            "staleVersions": {
              "count": 3,
              "sizeInBytes": 1006228697,
              "items": [
                "3769746207_-1890732847",
                "3811220169_-1890732847",
                "3118351592_-1890732847"
              ]
            }
          }
        }
      }
    }
  }
}

Response schema

The response is hierarchical. It is defined below in three parts:

Root response fields
The StorageStats object (reused recursively)
Breakdown definitions (per-location keys)

1. Root response fields

Field	Type	Description
`tableName`	`string`	Table name with type, for example `myTable_OFFLINE`.
`serverInstances`	`map`	Map of server instance ID (for example `Server_pinot-server-0`) to a StorageStats object. Storage on the actual server nodes; powers the Local Storage card.
`serverAggregateSegmentSizeStats`	`object`	Segment size stats computed over all expected segments across all servers. Present in every verbosity level, including `verbose=0`.
`deepStore`	`object`	A StorageStats object for the deep store (PinotFS/S3); powers the Deep Storage card.
`remoteS3`	`map`	Map of tier name to a StorageStats object for each tiered-storage object store; powers the Tiered Storage card. Absent when the table has no S3-backed tier.

2. The `StorageStats` object

This structure is used for serverInstances, deepStore, remoteS3, and every nested breakdown object.

Field	Type	Description
`actualItemsCount`	`integer`	Total count of unique items (segments/files) physically found at the location.
`actualSizeInBytes`	`long`	Total physical size, in bytes, of all items found. Shown as Used.
`unexpectedItemsCount`	`integer`	Count of items found that are not expected — unknown items plus beyond-retention segments. Excludes stale versions. Shown as Unexpected Segments.
`unexpectedSizeInBytes`	`long`	Total size, in bytes, of all unexpected storage — unknown items plus stale versions plus beyond-retention segments. Shown as Orphaned.
`totalStaleVersions`	`integer`	(Tiered/remote and server caches only) Number of unique segment versions present that differ from the version expected from table config and segment CRC.
`segmentSizeStats`	`object`	(Server instances only) Segment size stats for the expected segments on that server.
`breakdown`	`map`	Per-sub-directory breakdown. See Breakdown definitions. Present only when `verbose ≥ 1`.
`unexpected`	`object`	Inside a breakdown entry: itemized unexpected contributors. Contains `unknownItems`, `staleVersions`, and/or `beyondRetentionSegments`. Item lists are present only when `verbose=2`.

Segment size stats

Returned as serverAggregateSegmentSizeStats at the root and as segmentSizeStats per server instance. Computed over the table’s expected segments.

Field	Type	Description
`segmentCount`	`integer`	Number of segments included in the statistics.
`p50SegmentSizeInBytes`	`long`	50th percentile (median) segment size.
`p90SegmentSizeInBytes`	`long`	90th percentile segment size.
`p99SegmentSizeInBytes`	`long`	99th percentile segment size.
`avgSegmentSizeInBytes`	`long`	Average segment size. Shown as Avg Segment Size.

The `unexpected` object

Key	Description
`unknownItems`	Items on storage with no entry in Pinot metadata (untracked segments, temp files).
`staleVersions`	Older versions of known segments in tiered/remote storage or local caches.
`beyondRetentionSegments`	Deleted segments past their retention deadline in the deep store.

Each entry is an object with count (integer), sizeInBytes (long), and items (array of segment/version names). The items array is populated only at verbose=2.

3. Breakdown definitions

The keys inside a breakdown object depend on the location. Server instance (serverInstances):

Key	Description
`localSegmentsDirectory`	Segment index directories for locally stored segments, on the server’s local data directory.
`remoteSegmentsPreloadIndex`	Local preload-index cache for a tiered table’s remote segments, when preload is configured.
`remoteSegmentsMetadataCache`	Local metadata cache for a tiered table’s remote segments, when configured.

Deep store (deepStore):

Key	Description
`onlineSegments`	Live segments under `<data.dir>/<tableNameWithType>`.
`deletedSegments`	Segments in the retention queue at `<data.dir>/Deleted_Segments/<rawTableName>`.
`upsertSnapshots`	Snapshot files for upsert-enabled tables.
`dedupSnapshots`	Snapshot files for dedup-enabled tables.

Remote / tiered store (remoteS3):

Key	Description
`segmentDirectory`	Objects under the configured prefix for the tier, `<pathPrefix>/<tableNameWithType>/`.

Replication considerations

Replication increases Local Storage usage, because segment directories exist per replica. Deep store and remote object-store sizes are reported independently of replica placement.

Limitations

During cluster transitions such as rebalances or segment replacement, Used can temporarily exceed the expected size because of overlapping placement and intermediate artifacts. Re-check after the operation completes before treating the excess as reclaimable.

​Overview

​Reading the Storage tab

​What each number means

​Deep Storage breakdown

​Understanding the numbers

​Used, orphaned, and expected

​What “Orphaned” includes

​Why these numbers differ from the old Pinot table size

​What to do when Orphaned is high

​Recommended workflow

​Preventing recurrence

​Glossary

​Using the storage usage API

​Response: summary (verbose=0)

​Response: full breakdown (verbose=2)

​Response schema

​1. Root response fields

​2. The StorageStats object

Segment size stats

The unexpected object

​3. Breakdown definitions

​Replication considerations

​Limitations

Overview

Reading the Storage tab

What each number means

Deep Storage breakdown

Understanding the numbers

Used, orphaned, and expected

What “Orphaned” includes

Why these numbers differ from the old Pinot table size

What to do when Orphaned is high

Recommended workflow

Preventing recurrence

Glossary

Using the storage usage API

Response: summary (`verbose=0`)

Response: full breakdown (`verbose=2`)

Response schema

1. Root response fields

2. The `StorageStats` object

The `unexpected` object

3. Breakdown definitions

Replication considerations

Limitations