Skip to main content
This feature requires StarTree release 0.17.0 or later.
Scheduled Broker Scaling lets you automatically change the number of Apache Pinot brokers that serve a tenant on a fixed schedule. This is useful when your query load is predictable — for example, running fewer brokers overnight or on weekends to save cost, and restoring full capacity before peak hours. Scaling is driven by cron schedules defined per Pinot broker tenant. At each scheduled time, the operator compares the tenant’s current broker count with your target. It then creates the missing brokers (scale up) or deletes the extra brokers (scale down). A scale-down can remove brokers gradually, and it can wait until broker CPU and memory usage are low enough.
Broker scaling operates on individual brokers. You specify the absolute number of brokers that the tenant must have after each action (targetReplicas). This is different from Scheduled Server Scaling, which operates on server replica groups.
Scaling down reduces query capacity and redundancy for the tenant until the matching scale-up restores it. A schedule with targetReplicas: 1 leaves the tenant with no broker failover for the duration of the scale-down window — if that single broker has an issue, queries to the tenant’s tables fail. Choose a target that keeps the minimum capacity and redundancy that your off-peak traffic needs.

​
Prerequisites

  • Your cluster is on StarTree release 0.17.0 or later.
  • Your cluster is on Vintner, the StarTree Kubernetes operator. If your cluster is not on Vintner, the StarTree Ops team can migrate it. The StarTree Ops team configures this feature for you — see Enabling it.
  • All brokers of the tenant are in one broker instance group, and that group serves only this tenant. See Requirements for scaling for the exact conditions that the operator checks.
  • The metric gate is optional. The StarTree Ops team enables it when you ask for it.

​
Enabling it

Scheduled broker scaling is not available in the Data Portal or in a self-serve UI at this time. To enable it for your cluster, contact the StarTree Ops team.
Self-serve configuration of scheduled scaling from the Cloud Console is coming soon.
Give StarTree Ops the following information for each tenant that you want to schedule: To change, pause or remove the schedules of a tenant, contact StarTree Ops with the updated details. When scheduled scaling is paused or removed for a tenant, the operator returns the tenant’s brokers to the normal broker count of the cluster.

​
Configuration reference

Each schedule has these settings: scaleDown: metricsGate:
The operator does not drain a broker itself. When the operator deletes a broker, the broker gets a shutdown signal and drains its in-flight queries during its termination grace period. Then the broker stops. There is no separate drain timeout for brokers.

​
Rules for a schedule set

The operator checks the schedules of a tenant as a set. It does not run any schedule of the tenant if the set breaks one of these rules:
  • The set has at least one scale-down and at least one scale-up. A set with only one direction can remove brokers with nothing to bring them back.
  • At least one valid scale-up has a targetReplicas that is greater than the lowest targetReplicas of the scale-down schedules. Otherwise no scale-up can undo the scale-down.
When the schedules are corrected, the operator starts to run them again.

​
Cron format

Schedules use standard 5-field Unix cron (minute hour day-of-month month day-of-week). The operator evaluates all times in UTC. Convert your local schedule to UTC before you give it to StarTree Ops.

​
Gradual scale-down and the metric gate

By default, a scale-down removes one broker at a time and waits 3 minutes between brokers. Use stepSize and stepInterval to change this. For example, a scale-down from 6 to 2 brokers with stepSize: 2 and stepInterval: 5m removes two brokers, waits 5 minutes, and then removes two more. The metric gate stops a scale-down from removing capacity while the tenant’s brokers are busy. Before each step, the operator reads the CPU and memory usage of each tenant broker. It divides the usage by the broker’s request. The step continues only when every broker of the tenant is below threshold - buffer for each configured threshold. Brokers that serve other tenants do not affect the result. When the gate blocks a step, the result depends on the cause: The operator does not use the gate on a step that it already started. If the operator restarts in the middle of a step, it completes the deletion of the brokers that it selected for that step. A stopped scale-down does not retry. The tenant stays at the broker count that it reached until the next scheduled occurrence.

​
Typical pattern: nightly scale-down, morning scale-up

Pair a scale-down with a scale-up to reduce capacity during off-hours and restore it before peak load:
  • Scale down at night to a low targetReplicas (for example, 2).
  • Scale up in the morning back to your full targetReplicas (for example, 4).
Use the same tenant for both schedules. A scale-up brings back the same broker names that the scale-down removed. For example, you can give StarTree Ops these two schedules for DefaultTenant:

​
Checking status

Use the Vintner Operator dashboard in Grafana to see the scaling state of each tenant. Go to the Broker scheduled scaling row. It has these panels: If a schedule does not run as expected, open a ticket with StarTree.

​
Notes and limitations

  • targetReplicas must be 1 or more. A schedule cannot remove all brokers of a tenant.
  • If the tenant already has targetReplicas brokers, the action does nothing.
  • The operator does not set a maximum for targetReplicas. A scale-up needs node capacity for the brokers that it creates.
  • Cron times are evaluated in UTC. Convert your local time before you give a cron to StarTree Ops.
  • A deleted broker stays registered in Pinot as an offline instance, with its tenant tag. This is expected. A later scale-up starts a broker with the same name, and it comes back online in the same tenant.
  • While a scale-down or scale-up is in progress, the operator does not start other updates of the cluster, for example a version upgrade or a configuration change. These updates start when the scale operation completes. Because of this, the operator uses a maximum stepInterval of 1 hour and a maximum maxDeferrals of 120.
  • If the operator restarts during an operation, it completes that operation, even after the missedExecutionWindow closes. A scale-down resumes with the same brokers that it selected before the restart.
  • If a scale-up does not reach its target within 1 hour after its missedExecutionWindow closes, the operator stops it.
  • Scheduled broker scaling only affects the brokers of the named tenant. It does not change Zookeeper, Controller, Server or Minion components, or the brokers of other tenants.

​
Requirements for scaling

Before each scale-up and each scale-down, the operator checks the following conditions. If a condition is not met, the operator does not add or remove any broker. Cluster state
  • No cluster update is in progress.
  • Every online broker of the tenant is ready.
If one of these checks fails, the operator tries again every 5 minutes until the missedExecutionWindow closes. Tenant topology
  • All brokers of the tenant are in one broker instance group.
  • The broker instance group serves only this tenant. No broker of the group, online or offline, has a tag for another tenant.
If the group is shared with another tenant, the operator skips the occurrence. The operator checks again at the next scheduled occurrence.

​
How it works

  • The StarTree Ops team enables scheduled scaling for each tenant that you ask for, with the schedules that you give — see Enabling it.
  • At each schedule’s cron time, the operator first checks that no cluster update is in progress and that every current broker of the tenant is ready. If a check fails, the operator tries again every 5 minutes until the missedExecutionWindow closes.
  • Then the operator reads the tenant’s brokers from the Pinot controller and does the action:
    • Scale up. The operator creates the missing brokers in the tenant’s broker instance group. Each new broker joins the tenant when it starts. The operation is complete when the tenant has targetReplicas online brokers.
    • Scale down. The operator deletes stepSize brokers at a time. Each broker drains its in-flight queries during its graceful shutdown. Between steps, the operator waits for stepInterval so that the load moves to the remaining brokers. If the metric gate is enabled, the operator checks it again before each step.
  • If the operator was down when a schedule fired, it still runs the missed occurrence if it restarts within the missedExecutionWindow. After that window closes, the operator skips the occurrence.
While a scale operation is in progress, the operator does not start other updates of the cluster, so a cluster update does not fight the schedule. A cluster update that comes after a scale operation also keeps the scaled broker count — it does not delete the brokers that a scale-up added, or bring back the brokers that a scale-down removed.