Binary scheduler feature has not been certified for production use in Startree Cloud. This is an experimental feature.
How Does Binary Scheduler Work?
The Binary Workload Scheduler is designed to handle two distinct types of workloads: primary workloads (regular application queries) and secondary workloads (adhoc queries, testing, etc.)Primary Workload Processing:
Primary workload queries are executed with priority and submitted to runner threads immediately upon arrival
Resources used by primary workload queries are not capped
Secondary Workload Processing:
Secondary workload queries are identified using the query optionSET isSecondaryWorkload=true
These queries are processed through a constrained system with restrictions on runner threads, worker threads per query, and total worker threads for all in-progress secondary queries
The key benefit is to provide workload isolation: this scheduler can help to prevent expensive adhoc queries from impacting production workloads. Secondary workload can be used for adhoc triaging, testing and other non critical use cases on the same datasets.
How to Enable
Configure the Scheduler: Set the query scheduler algorithm to binary_workload in your Pinot cluster configuration:
pinot.query.scheduler.name=binary_workload
Mark Secondary Workload Queries: Use the query option to identify secondary workload queries
SET isSecondaryWorkload=true
Optional Configuration: Configure the maximum number of pending secondary queries