StarTree Execution Engine

Overview

The StarTree Execution Engine is a beta capability. Both its opt-in mechanism and the controls described on this page change between releases — a configuration key, query option, or statistic documented here may not exist, or may behave differently, on an older deployment. Check behavior against your cluster’s version, and contact StarTree support if something described here is not available in your environment.

The StarTree Execution Engine is an alternative, high-performance execution runtime for the Multi-Stage Query Engine (MSQE). It keeps everything you already know about MSQE — the SQL dialect, the query planner, and Pinot’s segment/index-powered table scans — but executes most of the query’s operators (joins, aggregations, sorts, window functions, set operations) in native code instead of Java. Under the hood it is built on Apache DataFusion, a vectorized query engine written in Rust, extended with a native Arrow-based data exchange layer for moving data between query stages.

What changes — and what doesn’t

A multi-stage query runs in three kinds of stages:

Stage	Java MSQE	StarTree Execution Engine
Leaf stages (segment scans, filter/index pushdown)	Pinot Java execution	Pinot Java execution (unchanged)
Intermediate stages (joins, aggregations, sorts, windows)	Java operators	Native, vectorized operators
Root stage (final merge on the broker)	Java operators	Native operators
Data exchange between stages	Java mailbox	Native Arrow-based exchange

Query planning is unchanged: the broker still parses, plans, and optimizes the query the same way, so EXPLAIN plans, query options, and the general query lifecycle look the same. Because leaf stages stay in Java, all of Pinot’s indexing (inverted, sorted, star-tree, JSON, etc.) keeps working exactly as before.

Enabling the StarTree Execution Engine

The native runtime is available on every broker and server. Queries still run on the Java MSQE runtime until something routes them to it — either the per-query option, or the broker-wide default, which routes every multi-stage query without a per-query choice.

Use v2.196.47 or later on the 0.16 line. Earlier 0.16 patches lack several controls on this page, and on those patches a setting from this page can be rejected or have no effect. Contact StarTree support to confirm the version that your cluster runs.

The native runtime

The runtime starts by default — pinot.startree.rust.mailbox.port defaults to 0, which picks an ephemeral free port. No configuration and no restart are needed to make it available. To stop a broker or server from starting it, set the port to a negative value:

# Negative: do not start the native runtime on this process.
pinot.startree.rust.mailbox.port=-1

A query that sets workerRuntime = 'startree' against a process that has opted out, or whose runtime failed to start, fails with a runtimeKey is not registered error.

In StarTree Cloud these settings are managed as part of the deployment. Contact StarTree support if the StarTree Execution Engine is not available in your environment.

Optional tuning (defaults shown):

Key	Default	Purpose
`pinot.startree.rust.metrics.port`	`8079`	Prometheus `/metrics` endpoint for the native runtime (`<= 0` disables)
`pinot.startree.datafusion.batch.size`	`8192`	Rows per vectorized batch

Memory and spill-to-disk behavior has its own set of settings — see Spill to disk for large queries.

Per-query switch

Route an individual multi-stage query to the StarTree Execution Engine with the workerRuntime query option. Using the SET command:

SET useMultistageEngine = true;
SET workerRuntime = 'startree';
SELECT ...

Or through the query options field of the REST APIs:

curl -X POST http://localhost:8000/query/sql -d '
{
  "sql": "select * from baseballStats limit 10",
  "queryOptions": "useMultistageEngine=true;workerRuntime=startree"
}'

The StarTree Execution Engine only applies to multi-stage queries. Single-stage queries are unaffected by the workerRuntime option — workerRuntime on its own does nothing unless useMultistageEngine = true is also set.

Broker-wide default

To route every multi-stage query to the StarTree Execution Engine without setting the option on each one, set the following in the cluster configuration:

pinot.startree.datafusion.default.enabled=true

This takes effect live and is reversible the same way, with no restart. Routing per query keeps the change scoped to the queries you picked, so prefer the query option while you are still evaluating the engine.

Returning a query to Java MSQE

DEFAULT is the explicit identifier for the Java runtime:

SET workerRuntime = 'DEFAULT';

An explicit workerRuntime always beats the broker-wide default, so this works even on a cluster where pinot.startree.datafusion.default.enabled is true. Use it to A/B a single query, and to unblock one that misbehaves without changing anything for other queries.

Confirming which runtime ran

Set the option and the query still looks unchanged? Check before concluding anything about performance:

EXPLAIN showing a BROKER_REDUCE node means the query never reached the multi-stage engine at all — useMultistageEngine is missing.
A queryEngineDowngraded field in the response metadata means the query reached MSQE but was routed back to Java — see Monitoring for how to read it.

Leaf-stage work and the columnar pull-up

The LeafOperator keeps running in Java, and it normally carries more than a scan: the filters, the projections, and the partial aggregate are all folded into it. Pushing work down into the leaf is the right choice on Java MSQE, where every row is a Java object either way. On the StarTree Execution Engine it can backfire — work done in the leaf is done on Java objects, and the result then has to be converted into Arrow to cross into the native runtime, which is exactly the allocation the engine exists to avoid. So the engine does the opposite where it pays: it takes the projection, and sometimes the partial aggregate, out of the LeafOperator and runs it natively above. The scan then emits Arrow columns directly and nothing is ever built as a Java row.

What moves, and what stays

	Pulled up, runs above the leaf	Stays in the `LeafOperator`
Projections	On a non-aggregate leaf, when every expression is a cast, arithmetic, a comparison, `AND`/`OR`, or `CASE`	Projections that use any other function (string, date, JSON)
Aggregates	`COUNT`; `SUM`, `MIN`, `MAX`, `AVG` and variance over `FLOAT`/`DOUBLE`; `MIN`, `MAX` over `INT`; `DISTINCTCOUNT` over a column without a dictionary, except `BIG_DECIMAL`	`SUM`, `AVG` and variance over `INT`/`LONG`; `SUM`, `AVG` over `BIG_DECIMAL`; `MIN`, `MAX` over `LONG`/`STRING`; percentiles, sketches, funnel functions; `DISTINCTCOUNT` over a dictionary-encoded or `BIG_DECIMAL` column
Query shapes	—	Aggregates with no `GROUP BY`; group-trim aggregates (their own `ORDER BY` + `LIMIT`); multi-value group keys; `is_leaf_return_final_result` combined with `DISTINCTCOUNT`; anything a star-tree index can already serve
Group keys	Numeric group keys. With `enableNullHandling = true`, dictionary-encoded string keys too	Dictionary-encoded string group keys — grouping on dictionary ids in Java beats interning UTF-8 in the native runtime

The type-specific entries above are not arbitrary. With the engine on, the planner renames SUM over INT/LONG to SUMINT/SUMLONG, and MIN/MAX over LONG/STRING to type-specific names; those names are not on the pull-up list, so that work stays in the leaf. AVG and variance are rewritten into SUM and COUNT first, which is why they pull up over FLOAT/DOUBLE.

The pull-up decides where an aggregate runs. Every aggregate it can take has a native implementation. An aggregate that needs the Java bridge (see Function support) is never pulled up.

Apart from the cost check on group keys, these are structural gates: the decision reads the column’s type and encoding — never its cardinality.

Overriding it

SET columnarLeaf = 'on';   -- pull up whenever eligible (no cost check)
SET columnarLeaf = 'off';  -- never; leave the work in the LeafOperator

'on' only skips the cost check — the structural gates in the table above still apply, so an aggregate with no GROUP BY, a group-trim aggregate, a multi-value key, a star-tree-servable query, or an aggregate that is not on the list stays in the leaf regardless. Only 'on' forces the pull-up. 'true' keeps the default, and 'false', 'disabled' or 'none' act like 'off'.

The case that needs 'off'. The cost check never keeps a numeric group key in Java, whatever its cardinality. SELECT status_code, COUNT(*) FROM t GROUP BY status_code over a billion rows therefore ships every row into the native runtime in order to produce a handful of groups. A low-cardinality numeric GROUP BY over a large table is the one shape where the pull-up reliably costs more than it saves — set columnarLeaf = 'off' for it.

When to expect benefits

The StarTree Execution Engine accelerates the parts of the query that run above the table scan. It shines when intermediate stages do significant work:

Join-heavy queries, especially large hash joins between fact tables.
Large aggregations and group-bys over data shuffled between servers.
Sorts and window functions over sizeable intermediate results.
Queries that move a lot of data between stages — the native exchange layer transfers Arrow batches with far less serialization and garbage-collection overhead than the Java mailbox.
Memory-intensive queries: the native runtime uses a bounded off-heap memory pool with spill-to-disk for sorts and aggregations, so large queries degrade gracefully instead of pressuring the JVM heap.

Single-query latency vs. behavior under load

These are two different results, and conflating them is the most common way to misread a benchmark. Single-query latency is mixed and shape-dependent. In internal benchmarks, join- and aggregation-heavy queries — where the work above the leaf dominates — run meaningfully faster than Java MSQE, while queries whose cost sits in the leaf, or which lean on shapes listed under When it doesn’t help, can be slower. Benchmark the queries you actually care about rather than assuming a uniform speedup. Behavior under concurrency is the dependable win. Because intermediate rows stop being Java objects and the exchange moves Arrow buffers instead of serialized Java rows, the JVM stops allocating and collecting that working set. In internal concurrency runs — a synthetic workload on a single machine, so directional rather than a guarantee — this showed up as a large reduction in both allocation and total GC pause time, and in queries completing that had been killed under memory pressure on Java MSQE. Both effects scale with load, not with query size, which is precisely why a single-query benchmark hides them. Note that the leaf stage is not covered by this: see The leaf stage is not covered by any of this.

Do not evaluate on a small dataset. Crossing into the native runtime has a fixed per-query cost. On a small enough dataset that fixed cost is the measurement, and queries can look several times slower than Java; the same queries at realistic scale converge or come out ahead. Size the evaluation dataset to the workload you actually run.

When it doesn’t help

Leaf-dominated queries. If most of the time is spent scanning segments (highly selective filters, simple aggregations pushed down to the leaf), the leaf stage is identical in both runtimes and the StarTree Execution Engine has little to accelerate.
Very fast, small queries. Crossing between the JVM and the native runtime has a fixed cost per query and per batch. Below roughly 100 ms on Java MSQE, that crossing cost dominates and the query can come out slower.
Bulk casts out of BIG_DECIMAL. There is no native decimal type, so BIG_DECIMAL is carried as order-preserving binary. That is why comparison, sort, join, and GROUP BY on it are native and lossless — and why converting out of it pays per value. A query that casts a large number of BIG_DECIMAL values, for example CAST(price AS FLOAT) over millions of distinct decimals, pays that cost on every one.
A few specific query shapes currently regress and are being actively closed (for example, some ASOF joins and certain UNION ALL patterns).

Switching runtimes is a one-line query option, so A/B comparison on your own workload is cheap. If a query matters to you, benchmark it on both runtimes.

Join strategies

Beyond the default hash join, two join strategies are available natively through the standard Calcite join hint:

SELECT /*+ joinOptions(join_strategy='sort_merge') */ a.*, b.*
FROM a JOIN b ON a.k = b.k

Strategy	Hint value	Notes
Sort-merge join	`sort_merge`	INNER/LEFT joins with at least one equality key (including `BIG_DECIMAL` keys). Can spill to disk, so prefer it for very large joins. Falls back to hash join (same results) for unsupported shapes.
Native lookup join	`lookup`	Dimension-table joins (INNER/LEFT/SEMI/ANTI; non-equality conditions are allowed for INNER/LEFT only) run natively; the dimension-table lookup itself still happens in Java. Unsupported shapes are query errors, not silent fallbacks.

Spill to disk for large queries

Aggregations (GROUP BY/DISTINCT), sorts, and sort-merge joins can spill to local disk under memory pressure instead of failing with an out-of-memory error — the query gets slower, not killed. Spill is backed by a shared native memory pool on each broker and server, sized from the available off-heap memory (container memory minus the JVM max heap). Native operators reserve memory from the pool before consuming it. If the pool cannot grant a reservation the engine knows before the allocation happens and can act on it — spilling, or cancelling that one query. This is the structural difference from the Java heap, where a query finds out it is out of memory only after the fact.

Two things about the pool are easy to get wrong:

It is per worker process, not per query. One pool is created at startup and shared by every query and every stage that worker runs. A query that succeeds when run alone can therefore be cancelled under concurrency, because its neighbours take a share of the pool.
The pool is divided evenly between op-chains, not between queries. The engine splits a multi-stage query into stages, and each stage into op-chains. Each op-chain registers with the pool as a separate consumer. If a server runs one op-chain of query Q1 and three op-chains of query Q2, each op-chain gets one quarter of the pool, so Q2 holds three quarters. The share of a query thus depends on how many op-chains its neighbours run on that server, not on how much memory they use. The same query can also get a different share on each server.
It does not bound the process’s memory. It bounds native intermediate execution only. The JVM heap — including everything in the leaf stage — sits outside it.

Config	Default	Description
`pinot.startree.datafusion.spill.enabled`	`true`	Master on/off switch. When off, the pool is unbounded.
`pinot.startree.datafusion.spill.memory.size`	`50%`	Percent of off-heap headroom, or an absolute size (`4G`, `512M`).
`pinot.startree.datafusion.spill.compression`	`uncompressed`	Spill file compression: `uncompressed`, `lz4_frame`, or `zstd`.
`pinot.startree.datafusion.spill.max.disk.size`	`100G`	Max spill data size before the query fails, or `unlimited`.
`pinot.startree.datafusion.spill.dir`	OS temp dir	Absolute path override for spill files.
`pinot.startree.datafusion.udaf.size.accounting.enabled`	`false`	Charge Java-heap-backed UDAF state (e.g. `DISTINCTCOUNT`, percentiles, sketches) toward the spill trigger. Off by default since the estimate is heap-based, not off-heap.
`pinot.startree.datafusion.udaf.groupby.max.capacity`	`1,000,000`	Max simultaneously-live groups for a `GROUP BY` over a Java-bridged aggregate. The limit always applies, whether UDAF size accounting is enabled or not.

Malformed or out-of-range values for these settings fail broker/server startup rather than silently falling back to a default. Under memory pressure a query therefore meets one of three outcomes:

Outcome	Which work	What you see
Spills — slower, still finishes	Final `GROUP BY` and `DISTINCT`, sorts, sort-merge joins	Added latency; spill counters on the operator
Cancelled	Hash joins, nested-loop joins, cross joins, window aggregates	A resources-exhausted error. The query dies; the process and its neighbours keep serving
Never charged to the pool at all	Everything inside the `LeafOperator`, and Java-bridged aggregate state — sketches, percentiles, `DISTINCTCOUNT` variants	No pool pressure. This state lives on the JVM heap, outside the pool, unless UDAF size accounting is enabled. A `GROUP BY` over it still stops at 1,000,000 live groups (`udaf.groupby.max.capacity`)

Hash joins cannot spill. The build side (the right input, as in Java MSQE) has to fit in the pool; above that the query is cancelled. If a large join hits this, use the sort_merge join hint instead of the default hash join, or raise the memory pool size. Note that sort-merge buys completion, not speed — reach for it deliberately, on the queries that would otherwise die.

The leaf stage is not covered by any of this

Everything above describes the native pool. The LeafOperator does not use it: it runs the original model, on the JVM heap, unbounded and without spilling. A large partial aggregate in the leaf can still cause GC pressure and still kill the query — in exactly the situations where Java MSQE kills it today. Moving a query to this engine does not, by itself, fix a leaf-side memory problem. Three things help:

Streaming leaf aggregates, which flush partial groups instead of holding them all, are on by default — this is why the problem is rarer than it used to be.
Let the pull-up take the aggregate. If it qualifies, the work moves into the native runtime and onto the bounded pool — see Leaf-stage work and the columnar pull-up.
Skip the leaf aggregate entirely with the standard Pinot hint, so grouping happens above the leaf:
SELECT /*+ aggOptions(is_skip_leaf_stage_group_by='true') */ ...
is_skip_leaf_stage_group_by is an existing Pinot aggregate hint and is not specific to this engine.

Unsupported cases

Not every query pattern runs natively yet. There are two very different behaviors, depending on when the engine detects the unsupported pattern.

Transparent fallback to Java MSQE (query still succeeds)

At planning time the broker checks the query plan for patterns the StarTree Execution Engine does not support. If it finds one, it silently ignores the workerRuntime option and runs the whole query on the Java MSQE runtime. The query succeeds and returns the same results as if you had never set the option. The response metadata reports the fallback with queryEngineDowngraded and its reason, and the broker writes an INFO log line (Disabling DataFusion for this query: <reason> ...). See Monitoring. Patterns that currently fall back at planning time:

INTERSECT ALL and EXCEPT ALL / MINUS ALL set operations.
Aggregations that return arrays or collections (e.g. ARRAY_AGG).
Certain optimizer-hint-driven aggregation paths that have no native equivalent, such as is_leaf_return_final_result combined with SUMPRECISION, funnel functions, or raw sketch aggregations.
UNNEST when the column-pruning optimization is enabled for it.
GROUPING SETS, ROLLUP, and CUBE.
Window frames using EXCLUDE, and LEAD/LAG ... IGNORE NULLS alongside another call.
Unsupported column types, when they are visible in the logical plan (see below).

If the planner finds any of these patterns, the whole query falls back to the classic Java engine — not just the affected stage.

Query fails with an error (no fallback)

Some patterns can only be detected after planning, while the plan is being prepared for native execution on the servers. At that point there is no fallback: the query fails and the client receives an error. To run such a query, set workerRuntime = 'DEFAULT' on it (see Returning a query to Java MSQE), or rewrite the query. Removing the option is not enough when the broker-wide default is on: the broker sets it again, and the query runs on the StarTree Execution Engine. Cases that fail at runtime:

Unsupported column types reaching an intermediate stage when not detectable at planning time: MAP, OBJECT, and exotic array shapes. Supported types are INT, LONG, FLOAT, DOUBLE, BOOLEAN, TIMESTAMP, STRING, JSON, BIG_DECIMAL, BYTES, and their common array variants.
Unsupported join sub-cases: LOOKUP joins outside INNER/LEFT/SEMI/ANTI, SEMI/ANTI lookup joins with non-equality conditions, and certain ASOF join match conditions.
Functions outside the supported envelope — see the function support section below for the specific lists of native, bridged, and unsupported functions.

Function support

Aggregation functions. The common aggregations run fully natively: COUNT, SUM, MIN, MAX, AVG, DISTINCTCOUNT (except over BIG_DECIMAL), BOOL_AND, BOOL_OR, and the statistics VAR_POP, STDDEV_POP, COVAR_POP, COVAR_SAMP, VAR_SAMP, and STDDEV_SAMP. Every other Pinot aggregation function — percentiles, the DISTINCTCOUNT sketch variants, SUMPRECISION, funnel functions, and so on — executes through a built-in Java bridge: results are identical to Java MSQE, at some extra cost when aggregation state crosses the native/Java boundary, and with a cap of 1,000,000 simultaneously-live groups (udaf.groupby.max.capacity). Aggregations that return arrays or collections (e.g. ARRAY_AGG) fall back to Java MSQE at planning time, as noted above. Scalar functions. Comparison, arithmetic, and logical operators, CASE, IN/NOT IN, and array construction run natively. All other registered Pinot scalar functions are automatically bridged to their Java implementation as long as their argument and return types are among INT, LONG, FLOAT, DOUBLE, BOOLEAN, STRING, BYTES, and TIMESTAMP (BIG_DECIMAL and JSON results are carried as strings). A scalar function outside that envelope — for example one that takes or returns arrays or Java objects (theta-sketch set operations being a supported exception) — fails at runtime with an Unsupported function error. Window functions. The standard window functions run natively: ROW_NUMBER, RANK, DENSE_RANK, PERCENT_RANK, CUME_DIST, NTILE, LAG, LEAD, FIRST_VALUE, LAST_VALUE, and NTH_VALUE, plus the native aggregations listed above used with an OVER clause. There is no Java bridge for window functions: a Pinot-specific window function outside this set fails at runtime.

BIG_DECIMAL precision. BIG_DECIMAL columns are fully supported in comparisons, sorting, joins, GROUP BY, DISTINCT, and MIN/MAX/SUM/AVG/COUNT, using an order-preserving binary encoding. However, arithmetic (+ - * / %) and aggregation over BIG_DECIMAL are computed in double-precision floating point — matching Java MSQE behavior, not arbitrary-precision decimal arithmetic. If your workload depends on exact decimal arithmetic beyond double precision, verify results before relying on it. Custom Java UDFs/UDAFs over BIG_DECIMAL are not yet supported on the StarTree Execution Engine.

Behavioral differences to be aware of

Results are identical to Java MSQE apart from a short list of cases. Two of them are worth flagging to users before you enable the engine for their queries, because they change what comes back rather than only how it is computed.

Difference	What happens	What to do
Integer division	`1 / int_col` truncates to an integer, following standard SQL. Java MSQE historically evaluated it as floating-point division.	Add an explicit `CAST(... AS DOUBLE)` where the legacy behavior is wanted.
`BIG_DECIMAL` scale	A `BIG_DECIMAL` value that crosses a native stage comes back scale-stripped — `1.00` renders as `1`. The numeric value is preserved; only the trailing scale is lost.	Flag before enabling, if output formatting matters.
`numGroupsLimit`	Java MSQE silently drops groups to stay inside a memory bound. Above the leaf, this engine applies no group limit: it returns groups that Java MSQE dropped, and pays for them in memory. An aggregate that stays in the Java leaf still applies `numGroupsLimit` per segment.	Nothing — but expect a query that was quietly truncating to start returning more groups, and more memory pressure with it.
Group-trim hints	The hint runs but its limit is ignored. That limit was also acting as a memory bound, so a query that completes on Java MSQE can need much more memory here, and spill or be cancelled.	Flag before enabling.
`ASOF` with extra predicates	Rejected outright rather than executed. Java MSQE silently drops the extra predicate and returns a wrong result.	Rewrite the query. The rejection is the correct behavior; do not report it as a defect.

Detailed stats require the streaming stats channel. Per-stage and per-operator response stats for natively executed stages — including leaf counters such as numSegmentsQueried — are only delivered when MSQE’s streaming stats channel is active. Enable it per query with SET streamStats = true; or cluster-wide with the broker configuration pinot.broker.mse.stream.stats=true (all servers must run a version that supports it). Without it, StarTree Execution Engine responses carry only broker-side stats. In addition, a few plan shapes (e.g. semi-join and dynamic-broadcast sub-plans) do not yet surface their leaf-stage stats. All of this affects observability, not results.

Reading per-operator statistics

Every native operator reports into the query’s stage-stats tree, so a slow query can be read operator by operator instead of guessed at. These stats travel over MSQE’s streaming stats channel — SET streamStats = true; if your build does not have it on by default. The tree uses thirteen native node types: NATIVE_JOIN, NATIVE_AGGREGATE, NATIVE_SORT, NATIVE_WINDOW, NATIVE_FILTER, NATIVE_TRANSFORM, NATIVE_SET, NATIVE_LIMIT, NATIVE_LITERAL, NATIVE_MAILBOX_SEND, NATIVE_MAILBOX_RECEIVE, NATIVE_LEAF_BRIDGE, and NATIVE_ROOT.

Finding the operator that is actually slow

Compare each node’s wall time against its self time. Do not use EXECUTION_TIME_MS for this: it is cumulative, covering every operator below the node as well as the Java leaf. The broker emits self time as selfExecutionTimeMs.

Reading	Meaning
`WALL_TIME_MS` ≈ self time (`selfExecutionTimeMs`)	The operator is doing the work. If it is slow, it is the bottleneck.
`WALL_TIME_MS` >> self time (`selfExecutionTimeMs`)	The operator mostly waited rather than worked. The wait counters below say in which direction.

Java MSQE has no per-operator wall time, so a starved operator and a busy one are harder to tell apart there. Its mailbox receive does report wait time.

Node types are written in upper snake case, but the keys in the response JSON are camelCase — WALL_TIME_MS appears as wallTimeMs.

Supporting counters:

UPSTREAM_WAIT_MS / DOWNSTREAM_WAIT_MS — these turn a wall-vs-self gap into a direction. A high UPSTREAM_WAIT_MS means the operator was starved by its input, so the bottleneck is below it in the tree; a high DOWNSTREAM_WAIT_MS means it was back-pressured by its consumer, so the bottleneck is above it. Reading the gap on its own and always walking downwards can lead away from the real cause.
TIME_BUILDING_HASH_TABLE_MS / TIME_PROBING_HASH_TABLE_MS — a join split into its two halves. Build-heavy means a large right input, which is the build side (as in Java MSQE). BUILD_MEM_BYTES shows its size.

Two gaps to know about: NUM_GROUPS is always 0, because the native runtime emits no such metric; and pipeline-breaker sub-plans (semi-join, dynamic broadcast) do not surface their leaf-stage stats.

Did this query spill to disk?

Per query, the clearest signal is on the operator itself. With SET streamStats = true;, NATIVE_SORT and NATIVE_AGGREGATE nodes carry spillCount, spilledBytes, and spilledRows in stageStats. Zero values are omitted from the JSON, so the key being present means that operator spilled, and its absence means it did not. Across the process, the native metrics endpoint carries the same story: rust_operator_spills_total and rust_operator_spilled_bytes_total for spill activity, and rust_spill_pool_reserved_bytes against rust_spill_pool_size_bytes for how full the pool is. Both failures surface as ResourcesExhausted, so the error type does not tell them apart — the message text does:

...exceeded the allowable limit... max_temp_directory_size — the query spilled and then hit the disk cap (pinot.startree.datafusion.spill.max.disk.size, default 100G). All queries on a worker share that cap.
Failed to allocate additional ... — the pool had no memory left, which usually means an operator that cannot spill.

Two blind spots. A sort-merge join spills through its inner sorts, but the join node itself carries no spill keys — that spill does not show up. And with streamStats off there are no per-operator statistics at all.

Monitoring

Did this query run natively?

When the planner routes a query back to Java MSQE, the response metadata says so:

"queryEngineDowngraded": true,
"queryEngineDowngradeReason": "INTERSECT ALL"

Two broker meters count the two outcomes of that same planning decision:

Metric	What it counts
`pinot_broker_datafusionQueriesExecuted_Count`	Queries sent to the StarTree Execution Engine, including queries that then fail
`pinot_broker_datafusionQueriesDegradedToMse_Count`	Queries that were routed back to Java MSQE at planning time

Executed counts the planner’s choice, not the outcome: a query the engine accepted and then failed on still increments it. Read it next to your error rate rather than as a success count. Because both are read at the same decision point:

coverage = executed / (executed + degraded)

What this ratio does and does not measure. It is the share of queries that asked for the engine and got it — not the share of your workload running natively. When you route per query, every multi-stage query you did not select runs on Java MSQE and appears in neither meter, so this can read 100% while most traffic is still on Java. Compare against pinot_broker_multiStageQueriesGlobal_Count for a workload-wide denominator. Queries that fail before planning — on parse, compile, or authorization — increment neither meter and sit outside the ratio entirely.

Other signals

The native runtime exposes its own Prometheus metrics endpoint (pinot.startree.rust.metrics.port, default 8079). All metrics are prefixed rust_:

Metric	Type	What it tells you
`rust_spill_pool_reserved_bytes` / `rust_spill_pool_size_bytes`	gauge	Native memory pool usage vs. configured size (labelled by runtime)
`rust_operator_spills_total` / `rust_operator_spilled_bytes_total`	counter	Spill events and bytes written by sorts/aggregations under memory pressure
`rust_operator_hash_build_seconds_total`	counter	Cumulative time building hash-join tables
`rust_mailbox_compress_` / `rust_mailbox_decompress_`	counter	Inter-stage exchange traffic: batches per codec, bytes before/after compression, and time spent
`rust_mailbox_fanout_payload_reuse_*`	counter	Broadcast exchange payload reuse
`rust_mailbox_grpc_recv_tasks_active` / `_total`	gauge / counter	Active and total cross-server exchange receive tasks
`rust_tokio_workers`	gauge	Network runtime worker threads
In StarTree Cloud these metrics are also stored in Prometheus and can be read using Grafana as usual.

In addition, brokers and servers export three engine-related meters through the standard Pinot metrics pipeline. datafusionOpchainAllocatorLeakCloseCount and datafusionOpchainAllocatorLeakedBytes count native memory reclaimed when a query’s allocator is closed while it still holds buffers. datafusionOpchainResourceLeakReclaimedCount counts op-chains whose native resources a backstop reclaimed because the normal close never ran. All three are safety-net counters and must stay at zero; a non-zero rate means a query path leaks.

To watch for a leak while it develops, the live-count gauges are the earlier signal. They follow the standard Pinot naming scheme (pinot_<component>_<metric>_Value) and are exported by both brokers and servers:

Gauge	What it counts
`pinot_broker_datafusionArrowLiveQueries_Value` / `pinot_server_datafusionArrowLiveQueries_Value`	Queries currently holding Arrow allocations
`pinot_broker_datafusionOpchainLiveResources_Value` / `pinot_server_datafusionOpchainLiveResources_Value`	Live operator-chain resources
`pinot_broker_datafusionNativeStreamHandlesLive_Value` / `pinot_server_datafusionNativeStreamHandlesLive_Value`	Open native stream handles

Any of these failing to return to zero when the cluster goes idle is the thing to investigate — the allocator-leak meters above only confirm a leak afterwards.

There is no dedicated Grafana panel for these yet; add them to a dashboard by hand. Watching three questions covers most of an evaluation: is the engine actually being used (the executed/degraded ratio), is memory getting tight (rust_spill_pool_reserved_bytes climbing toward rust_spill_pool_size_bytes, and rust_operator_spills_total rising), and is anything leaking (the live gauges above).

Standard Pinot query response metadata is returned for all queries, with the stats-coverage caveat noted above.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

StarTree Execution Engine

Overview

What changes — and what doesn’t

Enabling the StarTree Execution Engine

The native runtime

Per-query switch

Broker-wide default

Returning a query to Java MSQE

Confirming which runtime ran

Leaf-stage work and the columnar pull-up

What moves, and what stays

Overriding it

When to expect benefits

Single-query latency vs. behavior under load

When it doesn’t help

Join strategies

Spill to disk for large queries

The leaf stage is not covered by any of this

Unsupported cases

Transparent fallback to Java MSQE (query still succeeds)

Query fails with an error (no fallback)

Function support

Behavioral differences to be aware of

Reading per-operator statistics

Finding the operator that is actually slow

Did this query spill to disk?

Monitoring

Did this query run natively?

Other signals

​Overview

​What changes — and what doesn’t

​Enabling the StarTree Execution Engine

​The native runtime

​Per-query switch

​Broker-wide default

​Returning a query to Java MSQE

​Confirming which runtime ran

​Leaf-stage work and the columnar pull-up

​What moves, and what stays

​Overriding it

​When to expect benefits

​Single-query latency vs. behavior under load

​When it doesn’t help

​Join strategies

​Spill to disk for large queries

​The leaf stage is not covered by any of this

​Unsupported cases

​Transparent fallback to Java MSQE (query still succeeds)

​Query fails with an error (no fallback)

​Function support

​Behavioral differences to be aware of

​Reading per-operator statistics

​Finding the operator that is actually slow

​Did this query spill to disk?

​Monitoring

​Did this query run natively?

​Other signals

Overview

What changes — and what doesn’t

Enabling the StarTree Execution Engine

The native runtime

Per-query switch

Broker-wide default

Returning a query to Java MSQE

Confirming which runtime ran

Leaf-stage work and the columnar pull-up

What moves, and what stays

Overriding it

When to expect benefits

Single-query latency vs. behavior under load

When it doesn’t help

Join strategies

Spill to disk for large queries

The leaf stage is not covered by any of this

Unsupported cases

Transparent fallback to Java MSQE (query still succeeds)

Query fails with an error (no fallback)

Function support

Behavioral differences to be aware of

Reading per-operator statistics

Finding the operator that is actually slow

Did this query spill to disk?

Monitoring

Did this query run natively?

Other signals