> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scheduled Broker Scaling

> [!WARNING]
>
> This feature requires StarTree release 0.17.0 or later.

Scheduled Broker Scaling lets you automatically change the number of Apache Pinot brokers that serve a tenant on a fixed schedule. This is useful when your query load is predictable — for example, running fewer brokers overnight or on weekends to save cost, and restoring full capacity before peak hours.

Scaling is driven by **cron schedules** defined per Pinot **broker tenant**. At each scheduled time, the operator compares the tenant's current broker count with your target. It then creates the missing brokers (scale up) or deletes the extra brokers (scale down). A scale-down can remove brokers gradually, and it can wait until broker CPU and memory usage are low enough.

> [!NOTE]
>
> Broker scaling operates on **individual brokers**. You specify the absolute number of brokers that the tenant must have after each action (`targetReplicas`). This is different from [Scheduled Server Scaling](https://docs.startree.ai/corecapabilities/cluster-operations/scheduled-server-scaling.md), which operates on server replica groups.

> [!WARNING]
>
> Scaling down reduces query capacity and redundancy for the tenant until the matching scale-up restores it. A schedule with `targetReplicas: 1` leaves the tenant with **no broker failover** for the duration of the scale-down window — if that single broker has an issue, queries to the tenant's tables fail. Choose a target that keeps the minimum capacity and redundancy that your off-peak traffic needs.

## Prerequisites

* Your cluster is on **StarTree release 0.17.0 or later**.
* Your cluster is on Vintner, the StarTree Kubernetes operator. If your cluster is not on Vintner, the StarTree Ops team can migrate it. The StarTree Ops team configures this feature for you — see [Enabling it](#enabling-it).
* All brokers of the tenant are in one broker instance group, and that group serves only this tenant. See [Requirements for scaling](#requirements-for-scaling) for the exact conditions that the operator checks.
* The metric gate is optional. The StarTree Ops team enables it when you ask for it.

## Enabling it

Scheduled broker scaling is not available in the Data Portal or in a self-serve UI at this time. To enable it for your cluster, contact the StarTree Ops team.

> [!TIP]
>
> Self-serve configuration of scheduled scaling from the Cloud Console is coming soon.

Give StarTree Ops the following information for each tenant that you want to schedule:

| Input | Description |
| :- | :- |
| Tenant name | The Pinot broker tenant to schedule (for example, `DefaultTenant`). |
| Schedules | One or more combinations of `name`, `action`, `cron` and `targetReplicas` — see the [configuration reference](#configuration-reference) for the meaning of each setting. |
| Desired broker counts | The `targetReplicas` to scale down to (off-peak) and to scale back up to (peak). |
| Cron times | When each scale-down and scale-up must fire, in **UTC**. |
| Scale-down policy (optional) | How many brokers to remove in each step, how long to wait between steps, and the CPU and memory limits for the metric gate. |

To change, pause or remove the schedules of a tenant, contact StarTree Ops with the updated details. When scheduled scaling is paused or removed for a tenant, the operator returns the tenant's brokers to the normal broker count of the cluster.

## Configuration reference

Each schedule has these settings:

| Setting | Type | Required | Description |
| :- | :- | :- | :- |
| `name` | string | yes | A label for the schedule. |
| `action` | enum | yes | Scale down or scale up. |
| `cron` | string | yes | 5-field Unix cron expression (`minute hour day-of-month month day-of-week`), evaluated in **UTC**. See the examples below. |
| `targetReplicas` | integer | yes | The number of brokers that the tenant must have after this action completes. Must be 1 or more. |
| `missedExecutionWindow` | string | no | Grace period after the scheduled time. During this period, a missed occurrence still runs if the operator was down (for example, `30m`, `1h`). Defaults to `15m`. |
| `scaleDown` | object | no | Policy for a scale-down. The operator ignores it for a scale-up. See below. |

`scaleDown`:

| Setting | Type | Description |
| :- | :- | :- |
| `stepSize` | integer | The maximum number of brokers that the operator removes in one step. Defaults to `1`. |
| `stepInterval` | string | How long the operator waits between steps (for example, `3m`, `90s`). Defaults to `3m`. Must be positive. The operator uses a maximum of `1h`. |
| `metricsGate` | object | CPU and memory limits that the operator checks before each step. When this setting is not set, the operator does not check utilization. |

`metricsGate`:

| Setting | Type | Description |
| :- | :- | :- |
| `cpuThreshold` | number | Maximum broker CPU utilization for a scale-down step, as a fraction of the broker's CPU request (0 to 1). |
| `memoryThreshold` | number | Maximum broker memory utilization for a scale-down step, as a fraction of the broker's memory request (0 to 1). |
| `buffer` | number | Safety margin that the operator subtracts from each threshold before the comparison (for example, `0.1`). Defaults to `0`. |
| `maxDeferrals` | integer | How many times the operator postpones a step while utilization is too high, before it stops the scale-down. Defaults to `3` (about 90 seconds). Range 1 to 120. |

> [!NOTE]
>
> The operator does not drain a broker itself. When the operator deletes a broker, the broker gets a shutdown signal and drains its in-flight queries during its termination grace period. Then the broker stops. There is no separate drain timeout for brokers.

### Rules for a schedule set

The operator checks the schedules of a tenant as a set. It does not run any schedule of the tenant if the set breaks one of these rules:

* The set has at least one scale-down and at least one scale-up. A set with only one direction can remove brokers with nothing to bring them back.
* At least one valid scale-up has a `targetReplicas` that is greater than the lowest `targetReplicas` of the scale-down schedules. Otherwise no scale-up can undo the scale-down.

When the schedules are corrected, the operator starts to run them again.

### Cron format

Schedules use standard **5-field Unix cron** (`minute hour day-of-month month day-of-week`). The operator evaluates all times in **UTC**. Convert your local schedule to UTC before you give it to StarTree Ops.

| Expression | Meaning (UTC) |
| :- | :- |
| `0 22 * * *` | Every day at 22:00 UTC |
| `0 6 * * 1-5` | Weekdays at 06:00 UTC |
| `0 0 * * 0` | Every Sunday at midnight UTC |
| `30 18 * * 5` | Every Friday at 18:30 UTC |

## Gradual scale-down and the metric gate

By default, a scale-down removes one broker at a time and waits 3 minutes between brokers. Use `stepSize` and `stepInterval` to change this. For example, a scale-down from 6 to 2 brokers with `stepSize: 2` and `stepInterval: 5m` removes two brokers, waits 5 minutes, and then removes two more.

The metric gate stops a scale-down from removing capacity while the tenant's brokers are busy. Before each step, the operator reads the CPU and memory usage of each tenant broker. It divides the usage by the broker's request. The step continues only when every broker of the tenant is below `threshold - buffer` for each configured threshold. Brokers that serve other tenants do not affect the result.

When the gate blocks a step, the result depends on the cause:

| Cause | What the operator does |
| :- | :- |
| A broker's utilization is too high | The operator checks again every 30 seconds, up to `maxDeferrals` times. If the utilization is still too high, the operator stops the scale-down at the broker count that it reached. |
| No utilization reading, before the first step | The operator checks again every 30 seconds until the `missedExecutionWindow` closes. This does not use `maxDeferrals`. |
| No utilization reading, after the first step | The operator counts it against `maxDeferrals`, as for high utilization. |

The operator does not use the gate on a step that it already started. If the operator restarts in the middle of a step, it completes the deletion of the brokers that it selected for that step.

A stopped scale-down does not retry. The tenant stays at the broker count that it reached until the next scheduled occurrence.

## Typical pattern: nightly scale-down, morning scale-up

Pair a scale-down with a scale-up to reduce capacity during off-hours and restore it before peak load:

* **Scale down** at night to a low `targetReplicas` (for example, `2`).
* **Scale up** in the morning back to your full `targetReplicas` (for example, `4`).

Use the same tenant for both schedules. A scale-up brings back the same broker names that the scale-down removed.

For example, you can give StarTree Ops these two schedules for `DefaultTenant`:

| Setting | Nightly scale-down | Morning scale-up |
| :- | :- | :- |
| `name` | `nightly-scale-down` | `morning-scale-up` |
| `action` | Scale down | Scale up |
| `cron` | `0 22 * * *` | `0 6 * * *` |
| `targetReplicas` | `2` | `4` |
| `missedExecutionWindow` | `30m` | Default (`15m`) |
| `stepSize` | `1` | Not used |
| `stepInterval` | `3m` | Not used |
| `metricsGate` | `cpuThreshold: 0.6, memoryThreshold: 0.8, buffer: 0.1` | Not used |

## Checking status

Use the Vintner Operator dashboard in Grafana to see the scaling state of each tenant. Go to the **Broker scheduled scaling** row. It has these panels:

| Panel | What it shows |
| :- | :- |
| Status | The state of each tenant, the target and live broker counts, the next scale-down and scale-up times, and if the tenant needs attention. |
| State over time | When each scale-down and scale-up ran, and how long it took. |
| Scale-down gate | The highest CPU and memory usage of the tenant's brokers that the metric gate read, and how many times the gate postponed a step. |
| Why a schedule did not run | Each reason that stopped or delayed a schedule in the selected time range, for example a rejected schedule set, a shared broker instance group, or a metric gate that stopped a scale-down. |

If a schedule does not run as expected, open a ticket with StarTree.

## Notes and limitations

* `targetReplicas` must be 1 or more. A schedule cannot remove all brokers of a tenant.
* If the tenant already has `targetReplicas` brokers, the action does nothing.
* The operator does not set a maximum for `targetReplicas`. A scale-up needs node capacity for the brokers that it creates.
* Cron times are evaluated in **UTC**. Convert your local time before you give a cron to StarTree Ops.
* A deleted broker stays registered in Pinot as an offline instance, with its tenant tag. This is expected. A later scale-up starts a broker with the same name, and it comes back online in the same tenant.
* While a scale-down or scale-up is in progress, the operator does not start other updates of the cluster, for example a version upgrade or a configuration change. These updates start when the scale operation completes. Because of this, the operator uses a maximum `stepInterval` of 1 hour and a maximum `maxDeferrals` of 120.
* If the operator restarts during an operation, it completes that operation, even after the `missedExecutionWindow` closes. A scale-down resumes with the same brokers that it selected before the restart.
* If a scale-up does not reach its target within 1 hour after its `missedExecutionWindow` closes, the operator stops it.
* Scheduled broker scaling only affects the brokers of the named tenant. It does not change Zookeeper, Controller, Server or Minion components, or the brokers of other tenants.

## Requirements for scaling

Before each scale-up and each scale-down, the operator checks the following conditions. If a condition is not met, the operator does not add or remove any broker.

**Cluster state**

* No cluster update is in progress.
* Every online broker of the tenant is ready.

If one of these checks fails, the operator tries again every 5 minutes until the `missedExecutionWindow` closes.

**Tenant topology**

* All brokers of the tenant are in one broker instance group.
* The broker instance group serves only this tenant. No broker of the group, online or offline, has a tag for another tenant.

If the group is shared with another tenant, the operator skips the occurrence. The operator checks again at the next scheduled occurrence.

## How it works

```mermaid theme={null}
flowchart TD
    Cron["Cron time fires\nfor a schedule"] --> Checks{"No cluster update in progress\nand all tenant brokers ready?"}
    Checks -->|no| Retry["Retry every 5 minutes\nuntil missedExecutionWindow closes"]
    Checks -->|yes| Action{"action"}
    Action -->|Scale up| Create["Create the missing brokers\nin the tenant's instance group"]
    Create --> Join["New brokers join the tenant\non startup"]
    Join --> Restored["Done when the tenant\nhas targetReplicas brokers"]
    Action -->|Scale down| Gate{"Metric gate\n(if enabled)"}
    Gate -->|usage too high| Defer["Re-check every 30s,\nup to maxDeferrals times"]
    Gate -->|usage low| Step["Delete up to stepSize brokers"]
    Step --> Drain["Each broker drains its queries\nduring graceful shutdown"]
    Drain --> More{"At target?"}
    More -->|no| Wait["Wait stepInterval"] --> Gate
    More -->|yes| Deleted["Done"]
```

* The StarTree Ops team enables scheduled scaling for each tenant that you ask for, with the schedules that you give — see [Enabling it](#enabling-it).
* At each schedule's cron time, the operator first checks that no cluster update is in progress and that every current broker of the tenant is ready. If a check fails, the operator tries again every 5 minutes until the `missedExecutionWindow` closes.
* Then the operator reads the tenant's brokers from the Pinot controller and does the action:
  * **Scale up.** The operator creates the missing brokers in the tenant's broker instance group. Each new broker joins the tenant when it starts. The operation is complete when the tenant has `targetReplicas` online brokers.
  * **Scale down.** The operator deletes `stepSize` brokers at a time. Each broker drains its in-flight queries during its graceful shutdown. Between steps, the operator waits for `stepInterval` so that the load moves to the remaining brokers. If the metric gate is enabled, the operator checks it again before each step.
* If the operator was down when a schedule fired, it still runs the missed occurrence if it restarts within the `missedExecutionWindow`. After that window closes, the operator skips the occurrence.

While a scale operation is in progress, the operator does not start other updates of the cluster, so a cluster update does not fight the schedule. A cluster update that comes after a scale operation also keeps the scaled broker count — it does not delete the brokers that a scale-up added, or bring back the brokers that a scale-down removed.

## Related

* [Scheduled Server Scaling](https://docs.startree.ai/corecapabilities/cluster-operations/scheduled-server-scaling.md) — the equivalent feature for Pinot server replica groups.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.