Skip to main content
This feature requires StarTree release 0.15.0 or later.
Scheduled Server Scaling lets you automatically scale Apache Pinot server replica groups up or down on a fixed schedule. This is useful when your query load is predictable — for example, running fewer server replica groups overnight or on weekends to save cost, and restoring full capacity before peak hours. Scaling is driven by cron schedules defined per Pinot tenant. At each scheduled time, the operator asks the Pinot controller which servers to remove (or restore) to reach your target replica-group count, drains queries off the affected servers, and scales their StatefulSets accordingly.
Scaling operates at the granularity of replica groups, not individual servers. You specify a target number of replica groups; the Pinot controller computes the exact set of servers to remove or restore to achieve that target. See Replica Group based Workload Isolation for background on replica groups and pools.
Scaling down reduces redundancy for the tenant until the matching scale-up restores it. A schedule that targets targetReplicaGroups: 1 leaves the tenant with no replica group failover for the duration of the scale-down window — if that single replica group has an issue, queries to the tenant’s tables will fail. Choose a target that keeps the minimum redundancy your workload needs.

​
Prerequisites

  • Your cluster is on StarTree release 0.15.0 or later.
  • Your cluster is managed by the StarTree Kubernetes operator via the PinotCluster custom resource. This feature is configured at the operator level by the StarTree Ops team — see Enabling it.
  • The tenant’s tables already use pool-based, replica-group-aware instance assignment — see Controller requirements for scale-down for the exact conditions the controller checks before it will scale down a tenant.

​
How it works

  1. Scheduled scaling is enabled in the PinotCluster spec under the server component, with one or more schedules defined per tenant (see Enabling it).
  2. The operator creates and maintains a ScheduledServerScaling custom resource for each tenant you configure.
  3. At each schedule’s cron time:
    • Scale down — the operator calls the Pinot controller to determine which servers can be removed to reach the target replica-group count. Each affected server marks itself unavailable and drains its in-flight queries (up to 10 minutes) before its StatefulSet is scaled to zero.
    • Scale up — the operator determines which previously-removed servers must come back to reach the target, and restores their StatefulSets.
  4. If the operator was down when a schedule fired, it still executes the missed run as long as it restarts within the missedExecutionWindow. After that window passes, the missed run is skipped until the next occurrence.
While a scale operation is in progress, the operator suppresses normal replica reconciliation for the affected servers, so a manually-running cluster reconcile will not fight the schedule.

​
Enabling it

Scheduled scaling is configured at the Kubernetes operator level, via the PinotCluster custom resource. This isn’t currently exposed in the Data Portal or any self-serve UI, so contact the StarTree Ops team to have it enabled for your cluster.
Self-serve configuration of scheduled scaling from the Cloud Console is coming soon.
To enable it, share the following with StarTree Ops for each tenant you want to schedule: To change or disable a schedule, or to remove scheduled scaling from a tenant altogether, reach out to StarTree Ops with the updated details. When a tenant is removed from the spec, the operator deletes its ScheduledServerScaling resource and restores any servers that were left scaled down.

​
Configuration reference

scheduledScaling (under spec.components.server): Each entry in tenants: Each entry in schedules:
Before a server is scaled down, it marks itself unavailable and drains its in-flight queries, for up to 10 minutes, before its StatefulSet is scaled to zero. This drain timeout is fixed and isn’t currently configurable.

​
Cron format

Schedules use standard 5-field Unix cron (minute hour day-of-month month day-of-week). All times are evaluated in UTC — convert your local schedule to UTC before setting cron.

​
Typical pattern: nightly scale-down, morning scale-up

Pair a SCHEDULED_SCALE_DOWN with a SCHEDULED_SCALE_UP to shrink capacity during off-hours and restore it before peak load:
  • Scale down at night to a low targetReplicaGroups (e.g. 1).
  • Scale up in the morning back to your full targetReplicaGroups (e.g. 3).
Use the same tenant for both schedules. The scale-up restores exactly the servers that the matching scale-down removed.

​
Checking status

The operator tracks each tenant’s scaling state in the ScheduledServerScaling resource:
Key status fields:

​
Notes and limitations

  • targetReplicaGroups must be greater than 0 — a schedule cannot remove every replica group.
  • If targetReplicaGroups equals the tenant’s current total replica groups, a scale-down is a no-op (nothing to remove).
  • Cron times are evaluated in UTC — convert your local time before setting cron.
  • Scheduled scaling only affects server StatefulSets for the named tenant. It does not change Zookeeper, Controller, Broker, or Minion components.

​
Controller requirements for scale-down

At each scale-down, the operator asks the Pinot controller (GET /serverReplicaGroupScaleDown) which servers can be removed for the tenant. The controller only returns a server list when all of the following hold. If any fails, the scale-down is rejected and no servers are removed — fix the underlying condition and the next scheduled run (or a manual retry) will proceed. Tenant and topology
  • The tenant exists and has servers tagged for it (<tenant>_OFFLINE / <tenant>_REALTIME).
  • Servers are exclusive to the tenant — no server tagged for this tenant may also carry tags for another tenant.
  • Every tenant server has a pool assignment, and each pool matches at least one of the tenant’s tags.
  • If a server has both OFFLINE and REALTIME tags, both must point to the same pool number.
Target value
  • targetReplicaGroups must be ≥ 1 and ≤ the current number of replica groups (pools).
  • If targetReplicaGroups equals the current replica-group count, the call succeeds but returns no servers (nothing to remove).
Table configuration (when the tenant has tables)
  • No table may use instancePartitionsMap (which bypasses pool-based assignment).
  • All non-dimension tables must use pool-based, replica-group-aware instance assignment.
  • Each table’s configured numReplicaGroups (when non-zero) must equal the current pool count.
No rebalance in progress
  • No table in the tenant may have an active or failed table rebalance job.
  • The tenant may not have an active, aborted, cancelled, or unscheduled tenant rebalance job.
The controller evaluates these against a point-in-time snapshot of cluster state. The highest numbered pools are always selected for removal, so a given (tenant, targetReplicaGroups) request is deterministic.

​
Troubleshooting

A scale-down is rejected, with no servers removed, whenever one of the controller requirements above isn’t met. Fix the underlying condition — the next scheduled run (or a manual retry) will proceed once it’s resolved.