Troubleshooting resource pressure

This guide covers the cluster running out of something: memory, CPU, or disk. These problems announce themselves indirectly — as query timeouts, ingestion lag, or tasks failing in unrelated places — so the first job is recognising that resource pressure is what you are looking at.

Recognising it

Resource pressure is the likely cause when:

Several unrelated things degrade at once. Queries slow, ingestion falls behind, and tasks start failing, all within the same window. A single bad query does not do this.
Symptoms correlate with load rather than with a specific query or table. The same query is fast at 03:00 and slow at 14:00.
Failures cluster in time. Garbage-collection pauses and restarts produce bursts of timeouts separated by normal periods, rather than a steady failure rate.
A component restarted. Anything that was killed and came back was almost certainly killed for consuming too much of something.

Identify which component and which resource

Both halves matter. “The cluster is slow” is not diagnosable; “brokers are fine, servers on one tenant are at 90% heap” is.

Narrow to a component

Component	Symptom when under pressure
Server	Query timeouts, slow scans, ingestion lag on tables it hosts, segments going unavailable
Broker	Timeouts affecting all tables equally, planning-phase failures, connection errors
Controller	Table operations hang or fail, segment state stops converging, tasks stop being scheduled
Minion	Batch and ingestion tasks fail in unrelated groups, tasks restart repeatedly

Controllers and minions are easy to overlook because their symptoms surface elsewhere. If tables are healthy but nothing is changing — no new segments, no task progress — look at the controller.

Narrow to a resource

Memory — restarts, out-of-memory failures, pauses that look like timeouts, degradation that worsens gradually and resets on restart.
CPU — latency that scales with load, no restarts, scheduler wait time rising while execution time stays flat.
Disk — segment operations failing, ingestion stalling on commit, errors mentioning storage rather than compute.

Memory and CPU are frequently confused because both produce slow queries. The distinguishing signal is restarts: a component that keeps restarting is running out of memory.

Out-of-memory conditions

Establish when it started and what changed

Out-of-memory conditions almost always follow a change. Check for:

A version upgrade — memory characteristics change between versions
A new table, especially an upsert or dedup table
Growth in an existing upsert table’s primary-key count
A change in query shape — new aggregations, larger result sets, joins that were not there before
A change in data volume or retention

Check upsert and dedup memory

Upsert and dedup tables hold per-primary-key state in memory, and that state grows with distinct keys rather than with rows. A table that was comfortable at launch can exhaust memory months later with no change to its config.Pinot exposes an API to estimate memory usage for an upsert table — use it to size before creating one, and to check what an existing one should be consuming against what it is. Consider off-heap options where they suit your workload: see Off-heap upsert and Off-heap dedup.

Check whether queries are driving it

Queries with large intermediate results — high-cardinality group-bys, large joins, big DISTINCT sets — allocate heavily. In system_query_log:

SELECT logTimestampMs, tableName, timeUsedMs,
       offlineTotalMemAllocatedBytes + realtimeTotalMemAllocatedBytes AS memAllocated,
       numGroupsLimitReached, maxRowsInJoinReached, maxRowsInDistinctReached,
       query
FROM system_query_log
WHERE logTimestampMs > now() - 3600000
ORDER BY memAllocated DESC
LIMIT 20

Queries hitting the limit flags are being trimmed — they are consuming as much as they are allowed to, which suggests they would consume more if permitted.

Consider workload isolation

If analytical queries and latency-sensitive queries share the same servers, the analytical ones will periodically starve the others. StarTree supports isolating workloads so one class cannot exhaust the resources the other depends on — see Replica group based workload isolation and the memory-throttled scheduler.

High CPU

Establish whether it is query-driven

Rank recent queries by CPU:

SELECT queryHash,
       COUNT(*) AS executions,
       AVG(timeUsedMs) AS avgLatencyMs,
       SUM(offlineTotalCpuTimeNs + realtimeTotalCpuTimeNs) AS totalCpuNs,
       ANY_VALUE(query) AS sampleQuery
FROM system_query_log
WHERE logTimestampMs > now() - 3600000
GROUP BY queryHash
ORDER BY totalCpuNs DESC
LIMIT 20

Aggregate CPU matters more than per-query CPU. A cheap query running constantly costs more than an expensive one running hourly, and only the grouped view shows that.

Check whether the work is necessary

High CPU with high numEntriesScannedInFilter means the cluster is scanning data it should be skipping. That is an indexing problem wearing a capacity problem’s clothes, and adding servers will not fix it. See Queries failing or slow and run Query Analyzer on the worst offenders.

Check for background work

Rebalances, segment reloads, merges and purge tasks all consume CPU on the same servers that answer queries. If CPU rose without a change in query volume, check whether a maintenance operation is running. Check TABLE_SEGMENT_ASSIGNMENT_CHECK and TABLE_SEGMENTS_RELOAD_CHECK on the Health Dashboard.

Disk and storage

Check what is actually stored. The table Storage view in the Data Portal breaks down deep, local and tiered storage. See also Table storage usage.
Check retention is running. SEGMENT_RETENTION_CHECK on the Health Dashboard. Data past retention that has not been removed consumes space it should not.
Check for segments that should not be there. Failed or interrupted operations can leave segments behind that are no longer referenced but still occupy storage. If your segment count is much higher than your data volume suggests, this is worth raising.
Check whether you are near a segment-count limit. Segment count limits shows how to count where a table stands and what the cluster-wide picture looks like.
Check segment sizing. SEGMENT_SIZE_CHECK and SEGMENT_COUNT_CHECK flag tables whose segments are too small or too numerous. Many small segments cost more than the same data in fewer, larger ones — in memory, in query fan-out, and in metadata overhead. See Merge small segments.
Consider tiered storage. If most of your data is queried rarely, moving it to a cheaper tier reduces local pressure. See Cloud tiered storage.

Degradation after an upgrade

Several resource problems appear immediately after a version upgrade. When that is the sequence:

State the correlation explicitly

“This started after the upgrade” is a diagnosis-shortening fact. Note the previous version, the new version, and the exact time the symptom began.

Check the release notes for the versions you crossed

See StarTree release notes. If you skipped versions, read all of them — behaviour changes accumulate.

Check segments that need attention

Upgrades can leave segments requiring reload before they behave as expected, and occasionally leave segments in an error state. Check TABLE_SEGMENTS_RELOAD_CHECK and TABLE_SEGMENT_AVAILABILITY_CHECK on the Health Dashboard immediately after any upgrade.

Compare query plans, not just latencies

A query that got slower after an upgrade may be planning differently. Compare scan counts before and after — if numEntriesScannedInFilter changed materially for the same query, the plan changed, and that is a more specific thing to report than “it got slower.”

Capture a baseline before your next upgrade: p95 latency and scan counts for a handful of representative queries, current segment and document counts per table, and steady-state resource utilisation. Ten minutes beforehand turns “something feels slower” into a measurement.

Planning capacity

To avoid the reactive version of all of the above:

Size before you create. Estimate storage, and for upsert tables estimate primary-key memory, before adding a table to a tenant rather than after.
Know your headroom before a peak. If you have a known traffic event, check utilisation against it in advance — scaling ahead of a peak is routine, scaling during one is not.
Watch upsert key growth. It grows with distinct keys, not rows, and it does not shrink on its own.
Revisit sizing after workload changes. New dashboards, new consumers and new query patterns change resource profiles without changing any config you own.

See Scheduled server scaling for scaling around predictable load, and Query quotas for capping what any single workload can consume.

Feature-specific guidance

For scaling operations that did not take effect, see Scheduled server scaling → Troubleshooting. External tables have their own operational failure modes: External table troubleshooting covers server OOM and pod restarts under query load, and Operational guidance covers ZooKeeper packet-size limits as segment count grows. More at Troubleshooting by feature.

Escalating

Include: which component, which resource, when it began, what changed, whether anything restarted, and your current tenant sizing. If queries are implicated, attach the CPU or memory ranking from system_query_log. See what to collect.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Troubleshooting resource pressure

Recognising it

Identify which component and which resource

Out-of-memory conditions

High CPU

Disk and storage

Degradation after an upgrade

Planning capacity

Feature-specific guidance

Escalating

​Recognising it

​Identify which component and which resource

​Out-of-memory conditions

​High CPU

​Disk and storage

​Degradation after an upgrade

​Planning capacity

​Feature-specific guidance

​Escalating

Recognising it

Identify which component and which resource

Out-of-memory conditions

High CPU

Disk and storage

Degradation after an upgrade

Planning capacity

Feature-specific guidance

Escalating