> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Memory-Throttled FCFS Query Scheduler

> Throttle server query concurrency when JVM memory usage crosses a configured threshold.

The Memory-Throttled FCFS query scheduler is a first-come, first-served server query scheduler that reduces query concurrency when JVM memory usage crosses a configured threshold.

Use it when a server can become memory constrained under concurrent query load and you prefer queueing new work over letting the server continue scheduling queries until the JVM is at risk.

<Warning>
  Enable this only after validating it in the target workload. It changes server-side query scheduling behavior and can increase query latency while protecting memory headroom.
</Warning>

## Class and Scheduler Name

| Setting         | Value                                                           |
| --------------- | --------------------------------------------------------------- |
| Scheduler name  | `MemoryThrottledFCFS`                                           |
| Scheduler class | `ai.startree.query.scheduler.MemoryThrottledFCFSQueryScheduler` |

## How It Works

The scheduler:

1. Queues incoming server query requests in first-come, first-served order.
2. Schedules up to the current maximum allowed running queries.
3. Checks JVM memory usage with `Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()`.
4. Enters throttling when used memory is above the configured percentage of `Runtime.getRuntime().maxMemory()`.
5. Caps the maximum allowed running queries to the number already running, with a minimum of one.
6. Slowly increases the maximum allowed running queries after the memory condition clears.
7. Requests JVM garbage collection if throttling is sustained.

The scheduler starts with no throttling and allows all query runner threads to be used. Throttling only starts after the memory threshold is crossed.

## Enable the Scheduler

Set the scheduler name in the Pinot server configuration:

```properties theme={null}
pinot.query.scheduler.name=MemoryThrottledFCFS
```

## Configuration

| Server Config                                                       | Default                | Description                                                                                     |
| ------------------------------------------------------------------- | ---------------------- | ----------------------------------------------------------------------------------------------- |
| `pinot.query.scheduler.name`                                        | Not enabled by default | Set to `MemoryThrottledFCFS` to enable this scheduler.                                          |
| `pinot.query.scheduler.throttle.memoryUsagePercentageThreshold`     | `95`                   | Memory usage percentage that triggers throttling. Must be greater than `0` and less than `100`. |
| `pinot.query.scheduler.throttle.maxRunningQueriesUpdateFrequencyMs` | `1000`                 | Minimum interval for increasing the allowed running-query count after throttling clears.        |

Example:

```properties theme={null}
pinot.query.scheduler.name=MemoryThrottledFCFS
pinot.query.scheduler.throttle.memoryUsagePercentageThreshold=90
pinot.query.scheduler.throttle.maxRunningQueriesUpdateFrequencyMs=1000
```

Use a lower threshold to preserve more memory headroom. Use a higher threshold to allow more concurrency before throttling.

## Tuning Example

For a server with a 16 GB JVM heap and a threshold of `90`, throttling begins when used heap is above roughly 14.4 GB:

```text theme={null}
usedMemoryThreshold = Runtime.maxMemory * 90 / 100
```

When throttling starts:

* Already-running queries continue.
* New queries wait in the scheduler queue.
* The scheduler lowers the maximum allowed running queries to the current running query count, with a minimum of one.
* After memory usage drops below the threshold, the scheduler increases the allowed running-query count by one per update interval until it reaches the query runner thread count.

## Full Server Config Example

```properties theme={null}
# Query runner pool sizing still controls the upper bound for concurrent query execution.
pinot.server.query.executor.num.groups.limit=100
pinot.server.query.executor.num.groups.per.query=10

# Enable memory-aware scheduling.
pinot.query.scheduler.name=MemoryThrottledFCFS
pinot.query.scheduler.throttle.memoryUsagePercentageThreshold=90
pinot.query.scheduler.throttle.maxRunningQueriesUpdateFrequencyMs=1000
```

The exact query executor thread settings depend on your server profile. Keep the scheduler threshold aligned with JVM heap size, query memory profile, and expected concurrency.

## Metrics to Watch

The scheduler updates StarTree server gauges for query scheduling and throttling:

| Gauge                           | What to Watch                                                           |
| ------------------------------- | ----------------------------------------------------------------------- |
| `QUERY_NUM_RUNNER_THREADS`      | Upper bound available from the query runner pool.                       |
| `QUERY_RUNNING_TOTAL`           | Current running query count.                                            |
| `QUERY_MAX_ALLOWED_QUERIES`     | Dynamic concurrency cap set by the scheduler.                           |
| `QUERY_THROTTLING_SINCE_MS`     | Non-zero when throttling is active.                                     |
| `QUERY_THROTTLING_TOTAL_MS`     | Accumulated time spent throttling.                                      |
| `QUERY_THROTTLING_GC_REQUESTED` | Indicates that sustained throttling caused the scheduler to request GC. |

Healthy behavior after a short spike is:

1. `QUERY_THROTTLING_SINCE_MS` becomes non-zero.
2. `QUERY_MAX_ALLOWED_QUERIES` drops.
3. Memory usage falls.
4. `QUERY_THROTTLING_SINCE_MS` returns to zero.
5. `QUERY_MAX_ALLOWED_QUERIES` ramps back toward `QUERY_NUM_RUNNER_THREADS`.

## Operational Guidance

* Start with a threshold between `85` and `95`, then tune from observed query latency and heap pressure.
* Prefer fixing query memory regressions or oversized result sets before relying on scheduler throttling.
* Keep enough broker and client timeout budget for queued queries.
* Avoid setting the threshold too low on latency-sensitive clusters; it can queue queries even when the JVM has usable headroom.
* If throttling is continuous, add capacity, reduce per-query memory, lower concurrency, or investigate memory leaks.

## Troubleshooting

| Symptom                                                           | Cause                                                                                  | Fix                                                                                                                        |
| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Server fails to start with an illegal threshold error             | `pinot.query.scheduler.throttle.memoryUsagePercentageThreshold` is `<= 0` or `>= 100`. | Set a value between `0` and `100`, such as `90`.                                                                           |
| Query latency increases during memory spikes                      | Queries are queueing while the scheduler throttles.                                    | Confirm memory pressure and either raise capacity, reduce query memory, or adjust the threshold.                           |
| Throttling never clears                                           | Used memory remains above the threshold.                                               | Inspect heap usage, query mix, result sizes, segment memory, and possible leaks.                                           |
| `QUERY_MAX_ALLOWED_QUERIES` ramps up too quickly after throttling | Update frequency is too low for your workload.                                         | Increase `pinot.query.scheduler.throttle.maxRunningQueriesUpdateFrequencyMs`.                                              |
| Queries still consume all runner threads                          | Scheduler not enabled on the server, or config was not applied.                        | Confirm `pinot.query.scheduler.name=MemoryThrottledFCFS` on each server and restart/reload the server process as required. |