> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting resource pressure

> Diagnose out-of-memory conditions, high CPU, disk pressure, and degradation after an upgrade or a traffic peak.

This guide covers the cluster running out of something: memory, CPU, or disk. These problems announce themselves indirectly — as query timeouts, ingestion lag, or tasks failing in unrelated places — so the first job is recognising that resource pressure is what you are looking at.

## Recognising it

Resource pressure is the likely cause when:

* **Several unrelated things degrade at once.** Queries slow, ingestion falls behind, and tasks start failing, all within the same window. A single bad query does not do this.
* **Symptoms correlate with load rather than with a specific query or table.** The same query is fast at 03:00 and slow at 14:00.
* **Failures cluster in time.** Garbage-collection pauses and restarts produce bursts of timeouts separated by normal periods, rather than a steady failure rate.
* **A component restarted.** Anything that was killed and came back was almost certainly killed for consuming too much of something.

## Identify which component and which resource

Both halves matter. "The cluster is slow" is not diagnosable; "brokers are fine, servers on one tenant are at 90% heap" is.

<Steps>
  <Step title="Narrow to a component">
    | Component      | Symptom when under pressure                                                               |
    | -------------- | ----------------------------------------------------------------------------------------- |
    | **Server**     | Query timeouts, slow scans, ingestion lag on tables it hosts, segments going unavailable  |
    | **Broker**     | Timeouts affecting all tables equally, planning-phase failures, connection errors         |
    | **Controller** | Table operations hang or fail, segment state stops converging, tasks stop being scheduled |
    | **Minion**     | Batch and ingestion tasks fail in unrelated groups, tasks restart repeatedly              |

    Controllers and minions are easy to overlook because their symptoms surface elsewhere. If tables are healthy but nothing is *changing* — no new segments, no task progress — look at the controller.
  </Step>

  <Step title="Narrow to a resource">
    * **Memory** — restarts, out-of-memory failures, pauses that look like timeouts, degradation that worsens gradually and resets on restart.
    * **CPU** — latency that scales with load, no restarts, scheduler wait time rising while execution time stays flat.
    * **Disk** — segment operations failing, ingestion stalling on commit, errors mentioning storage rather than compute.

    Memory and CPU are frequently confused because both produce slow queries. The distinguishing signal is restarts: a component that keeps restarting is running out of memory.
  </Step>
</Steps>

## Out-of-memory conditions

<Steps>
  <Step title="Establish when it started and what changed">
    Out-of-memory conditions almost always follow a change. Check for:

    * A version upgrade — memory characteristics change between versions
    * A new table, especially an upsert or dedup table
    * Growth in an existing upsert table's primary-key count
    * A change in query shape — new aggregations, larger result sets, joins that were not there before
    * A change in data volume or retention
  </Step>

  <Step title="Check upsert and dedup memory">
    Upsert and dedup tables hold per-primary-key state in memory, and that state grows with distinct keys rather than with rows. A table that was comfortable at launch can exhaust memory months later with no change to its config.

    Pinot exposes an API to estimate memory usage for an upsert table — use it to size before creating one, and to check what an existing one should be consuming against what it is. Consider off-heap options where they suit your workload: see [Off-heap upsert](/corecapabilities/manage-data/offheap-upsert) and [Off-heap dedup](/corecapabilities/manage-data/offheap-dedup).
  </Step>

  <Step title="Check whether queries are driving it">
    Queries with large intermediate results — high-cardinality group-bys, large joins, big `DISTINCT` sets — allocate heavily. In `system_query_log`:

    ```sql theme={null}
    SELECT logTimestampMs, tableName, timeUsedMs,
           offlineTotalMemAllocatedBytes + realtimeTotalMemAllocatedBytes AS memAllocated,
           numGroupsLimitReached, maxRowsInJoinReached, maxRowsInDistinctReached,
           query
    FROM system_query_log
    WHERE logTimestampMs > now() - 3600000
    ORDER BY memAllocated DESC
    LIMIT 20
    ```

    Queries hitting the limit flags are being trimmed — they are consuming as much as they are allowed to, which suggests they would consume more if permitted.
  </Step>

  <Step title="Consider workload isolation">
    If analytical queries and latency-sensitive queries share the same servers, the analytical ones will periodically starve the others. StarTree supports isolating workloads so one class cannot exhaust the resources the other depends on — see [Replica group based workload isolation](/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation) and the [memory-throttled scheduler](/corecapabilities/query_data/advanced_operations/workload-isolation/memory-throttled-scheduler).
  </Step>
</Steps>

## High CPU

<Steps>
  <Step title="Establish whether it is query-driven">
    Rank recent queries by CPU:

    ```sql theme={null}
    SELECT queryHash,
           COUNT(*) AS executions,
           AVG(timeUsedMs) AS avgLatencyMs,
           SUM(offlineTotalCpuTimeNs + realtimeTotalCpuTimeNs) AS totalCpuNs,
           ANY_VALUE(query) AS sampleQuery
    FROM system_query_log
    WHERE logTimestampMs > now() - 3600000
    GROUP BY queryHash
    ORDER BY totalCpuNs DESC
    LIMIT 20
    ```

    Aggregate CPU matters more than per-query CPU. A cheap query running constantly costs more than an expensive one running hourly, and only the grouped view shows that.
  </Step>

  <Step title="Check whether the work is necessary">
    High CPU with high `numEntriesScannedInFilter` means the cluster is scanning data it should be skipping. That is an indexing problem wearing a capacity problem's clothes, and adding servers will not fix it. See [Queries failing or slow](/corecapabilities/query_data/troubleshooting#timeouts) and run [Query Analyzer](/corecapabilities/query_data/query-analyzer) on the worst offenders.
  </Step>

  <Step title="Check for background work">
    Rebalances, segment reloads, merges and purge tasks all consume CPU on the same servers that answer queries. If CPU rose without a change in query volume, check whether a maintenance operation is running. Check `TABLE_SEGMENT_ASSIGNMENT_CHECK` and `TABLE_SEGMENTS_RELOAD_CHECK` on the Health Dashboard.
  </Step>
</Steps>

## Disk and storage

* **Check what is actually stored.** The table Storage view in the Data Portal breaks down deep, local and tiered storage. See also [Table storage usage](/corecapabilities/manage-data/recipes/table-storage-usage).

* **Check retention is running.** `SEGMENT_RETENTION_CHECK` on the Health Dashboard. Data past retention that has not been removed consumes space it should not.

* **Check for segments that should not be there.** Failed or interrupted operations can leave segments behind that are no longer referenced but still occupy storage. If your segment count is much higher than your data volume suggests, this is worth raising.

* **Check whether you are near a segment-count limit.** [Segment count limits](/corecapabilities/manage-data/recipes/segment-count-limits#monitoring-and-troubleshooting) shows how to count where a table stands and what the cluster-wide picture looks like.

* **Check segment sizing.** `SEGMENT_SIZE_CHECK` and `SEGMENT_COUNT_CHECK` flag tables whose segments are too small or too numerous. Many small segments cost more than the same data in fewer, larger ones — in memory, in query fan-out, and in metadata overhead. See [Merge small segments](/recipes/merge-small-segments).

* **Consider tiered storage.** If most of your data is queried rarely, moving it to a cheaper tier reduces local pressure. See [Cloud tiered storage](/corecapabilities/manage-data/set-up-tiered-storage/motivation).

## Degradation after an upgrade

Several resource problems appear immediately after a version upgrade. When that is the sequence:

<Steps>
  <Step title="State the correlation explicitly">
    "This started after the upgrade" is a diagnosis-shortening fact. Note the previous version, the new version, and the exact time the symptom began.
  </Step>

  <Step title="Check the release notes for the versions you crossed">
    See [StarTree release notes](/reference/startree-release-notes). If you skipped versions, read all of them — behaviour changes accumulate.
  </Step>

  <Step title="Check segments that need attention">
    Upgrades can leave segments requiring reload before they behave as expected, and occasionally leave segments in an error state. Check `TABLE_SEGMENTS_RELOAD_CHECK` and `TABLE_SEGMENT_AVAILABILITY_CHECK` on the Health Dashboard immediately after any upgrade.
  </Step>

  <Step title="Compare query plans, not just latencies">
    A query that got slower after an upgrade may be planning differently. Compare scan counts before and after — if `numEntriesScannedInFilter` changed materially for the same query, the plan changed, and that is a more specific thing to report than "it got slower."
  </Step>
</Steps>

<Tip>
  Capture a baseline before your next upgrade: p95 latency and scan counts for a handful of representative queries, current segment and document counts per table, and steady-state resource utilisation. Ten minutes beforehand turns "something feels slower" into a measurement.
</Tip>

## Planning capacity

To avoid the reactive version of all of the above:

* **Size before you create.** Estimate storage, and for upsert tables estimate primary-key memory, before adding a table to a tenant rather than after.
* **Know your headroom before a peak.** If you have a known traffic event, check utilisation against it in advance — scaling ahead of a peak is routine, scaling during one is not.
* **Watch upsert key growth.** It grows with distinct keys, not rows, and it does not shrink on its own.
* **Revisit sizing after workload changes.** New dashboards, new consumers and new query patterns change resource profiles without changing any config you own.

See [Scheduled server scaling](/corecapabilities/cluster-operations/scheduled-server-scaling) for scaling around predictable load, and [Query quotas](/corecapabilities/query_data/advanced_operations/query-quotas) for capping what any single workload can consume.

## Feature-specific guidance

For scaling operations that did not take effect, see [Scheduled server scaling → Troubleshooting](/corecapabilities/cluster-operations/scheduled-server-scaling#troubleshooting). External tables have their own operational failure modes: [External table troubleshooting](/corecapabilities/external-table/troubleshooting#queries) covers server OOM and pod restarts under query load, and [Operational guidance](/corecapabilities/external-table/operational-guidance#additional-troubleshooting) covers ZooKeeper packet-size limits as segment count grows. More at [Troubleshooting by feature](/corecapabilities/observability/troubleshooting-by-feature).

## Escalating

Include: which component, which resource, when it began, what changed, whether anything restarted, and your current tenant sizing. If queries are implicated, attach the CPU or memory ranking from `system_query_log`. See [what to collect](/corecapabilities/observability/troubleshooting#before-you-open-a-support-ticket).