> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Custom Partition Function
> Derive segment partition IDs from a Pinot scalar expression so StarTree Cloud can prune segments on transformed column values.
## Overview and Purpose
The `Custom` partition function computes a column's partition ID by evaluating a Pinot scalar expression against the column value, instead of applying one of the built-in partition functions such as `Murmur` or `Modulo`.
Partitioning writes a partition ID into each segment's metadata. At query time, StarTree Cloud evaluates the same partition function on the value in an equality predicate and skips every segment whose metadata says it cannot hold that partition. Built-in partition functions hash the raw column value, which works only when the value itself is what you partitioned on. The `Custom` function removes that restriction: the partition ID can come from any deterministic scalar expression over the value, such as decoding a hex string or hashing a string ID after converting it to bytes.
Use the `Custom` partition function when:
* The partition ID is derived from a transformed column value rather than the raw value.
* An upstream system already partitions data with a specific hash, and StarTree Cloud needs to reproduce that same partition ID to keep segment metadata aligned.
* You want segment pruning on a string ID column, hashed through a scalar function chain such as `fnv1a_hash_32_utf8(md5(toUtf8(v)))`.
* A built-in partition function exists but uses the wrong normalization for your partition count.
The `Custom` partition function drives two different things depending on how segments are built.
It always writes partition metadata into each segment, which is what enables pruning. In addition, ingestion flows that run the `SegmentProcessorFramework` — `FileIngestionTask`, merge/rollup, and realtime-to-offline tasks — read `segmentPartitionConfig` and bucket rows by the configured partition function, so the same expression also decides which output segment each row lands in. See [Partitioning upstream data with FileIngestionTask](/corecapabilities/ingestdata/adv-concepts/batch/offline-upserts#3-partitioning-upstream-data-with-fileingestiontask).
In those repartitioning flows, every value that evaluates to `-1` is grouped into a single output bucket. Validate the expression against sample values before pointing one of these tasks at a `Custom`-partitioned column.
## Availability
The `Custom` partition function is available in StarTree release 0.15 and later. The corresponding STP release branch is `release/1.6.0-STP-2.164.x`.
## How It Works
The function is configured per column under `indexingConfig.segmentPartitionConfig.columnPartitionMap`. For each value it processes:
1. The column value is bound to the reserved identifier `v` in the configured `partitionExpression`.
2. The expression is evaluated as a Pinot scalar expression.
3. The numeric result is mapped into `[0, numPartitions)` by the configured `partitionIdNormalizer`.
4. If the normalized value falls inside `[0, numPartitions)`, it becomes the partition ID.
If any of those steps cannot produce a valid partition ID, the function returns `-1`.
Pruning is a set-membership check: segment build records both the partition IDs a segment contains and the partition function that produced them. At query time StarTree Cloud evaluates that recorded function on the literal in an equality predicate and skips segments whose recorded set does not contain the resulting ID. Because the same expression runs on both sides, a `-1` is consistent and results stay correct. What you lose is selectivity — every value that fails to evaluate collapses into the same `-1` bucket, which most segments will contain, so those queries prune nothing.
The function returns `-1` when:
* No `partitionExpression` is configured, so the raw string value is used and is not numeric.
* Expression evaluation throws at runtime, for example `hex_decimal_to_long(v)` on the value `not-a-hex-value`.
* The expression returns a non-numeric result, for example `md5(v)`, which returns bytes.
* The normalized value falls outside `[0, numPartitions)`, which is possible with the `NO_OP` normalizer.
## Requirements
* The partition column must exist in the table schema.
* The partition column must be single-value.
* `numPartitions` must be greater than `0`.
* Every scalar function used in the expression must be registered in the Pinot function registry.
## Configuration
Configure the function through the standard `ColumnPartitionConfig` entry for the column.
```json theme={null}
{
"tableName": "events_OFFLINE",
"tableType": "OFFLINE",
"segmentsConfig": {
"schemaName": "events"
},
"indexingConfig": {
"segmentPartitionConfig": {
"columnPartitionMap": {
"correlationId": {
"functionName": "Custom",
"numPartitions": 16,
"functionConfig": {
"partitionExpression": "fnv1a_hash_32_utf8(md5(toUtf8(v)))",
"partitionIdNormalizer": "ABS"
}
}
}
}
},
"routing": {
"segmentPrunerTypes": [
"partition"
]
}
}
```
### Enabling Segment Pruning
Partition metadata is consumed by two independent pruning layers, and they need different setup:
* **Server-side pruning works with no extra configuration.** `ColumnValueSegmentPruner` is part of the default server pruner list, so each server skips its own non-matching segments as soon as the request lands.
* **Broker-side pruning requires opting in.** Add `partition` to `routing.segmentPrunerTypes`, as in the example above. Without it, the broker fans the query out to every server hosting the table and pruning happens only after dispatch. With it, the broker drops non-matching segments during routing, which also reduces the number of servers contacted.
Both layers evaluate the partition function recorded in each segment's own metadata, not the function currently in the table config.
### Configuration Parameters
| Parameter | Required | Default | Description |
| -------------------------------------- | -------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `functionName` | Yes | — | Must be `Custom`. |
| `numPartitions` | Yes | — | Number of logical partitions for the column. Must be greater than `0`. |
| `functionConfig.partitionExpression` | No | None | Pinot scalar-function expression evaluated for the column value. Without it, every value resolves to partition `-1`. |
| `functionConfig.partitionIdNormalizer` | No | `POSITIVE_MODULO` | Name of the `PartitionIdNormalizer` used to map the raw expression result into `[0, numPartitions)`. Blank values fall back to the default. |
### Expression Rules
The partition column value is exposed to the expression as the reserved identifier `v`.
* `v` is the **only** allowed argument name. `hex_decimal_to_long(partitionValue)` and `concat(v, otherColumn)` are both rejected.
* The expression must reference `v` at least once. A constant expression such as `plus(1, 0)` is rejected.
* `v` may appear more than once, and every occurrence is bound to the same column value. For example, `fnv1a_hash_32_utf8(md5(toUtf8(concat(v, v))))` is valid.
* The expression should return a number. `Integer` and `Long` results are normalized directly; any other `Number` is converted with its long value first, so `floor(v)` on `"11.9"` produces `11`.
* The expression must be deterministic. A non-deterministic scalar function produces segment metadata that does not match the partition ID computed at query time, which silently breaks pruning.
Use the key `partitionExpression`. Other spellings, including `functionExpr`, are ignored, which leaves the function without an expression and sends every value to partition `-1`.
### Partition ID Normalizers
The normalizer maps the raw expression result to a partition ID. The default is `POSITIVE_MODULO`.
| Normalizer | Behavior | When to use |
| ----------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `POSITIVE_MODULO` | `value % numPartitions`, then adds `numPartitions` if the remainder is negative. | The default. Safe for any signed result. |
| `ABS` | Absolute value of `value % numPartitions`. | Hash functions that can return negative integers, when you want to match this normalization. |
| `MASK` | Clears the sign bit, then applies modulo. | Partition counts that are compatible with bit masking. |
| `PRE_MODULO_ABS` | `abs(value) % numPartitions`, mapping `Integer.MIN_VALUE` and `Long.MIN_VALUE` to `0`. | Matching the legacy semantics of `HashCodePartitionFunction` and `ByteArrayPartitionFunction`. |
| `NO_OP` | Uses the expression result as the partition ID unchanged. | The expression already returns a valid ID in `[0, numPartitions)`. Out-of-range results become `-1`. |
Normalizer names are case-insensitive.
## Examples
### Hash a String ID
Convert a string column to UTF-8 bytes, hash it with `md5`, then fold the hash into an integer. This is the common shape for pruning on a high-cardinality string ID.
```json theme={null}
{
"correlationId": {
"functionName": "Custom",
"numPartitions": 16,
"functionConfig": {
"partitionExpression": "fnv1a_hash_32_utf8(md5(toUtf8(v)))",
"partitionIdNormalizer": "ABS"
}
}
}
```
Queries with an equality predicate on the column can then use the partition metadata for pruning:
```sql theme={null}
SELECT count(*)
FROM events
WHERE correlationId = 'correlation-123';
```
### Convert a Hex String to a Partition ID
Decode a hexadecimal string into a long and let the default `POSITIVE_MODULO` normalizer map it into the partition count.
```json theme={null}
{
"hexUserId": {
"functionName": "Custom",
"numPartitions": 128,
"functionConfig": {
"partitionExpression": "hex_decimal_to_long(v)"
}
}
}
```
With `numPartitions` set to `128`:
| Value | Raw result | Partition |
| ----------------- | ------------------ | --------- |
| `ff` | `255` | `127` |
| `80` | `128` | `0` |
| `11` | `17` | `17` |
| `not-a-hex-value` | Evaluation failure | `-1` |
### Use an Expression Result Directly
`NO_OP` skips normalization. Use it only when the expression already returns a valid partition ID.
```json theme={null}
{
"partitionIdText": {
"functionName": "Custom",
"numPartitions": 4,
"functionConfig": {
"partitionExpression": "hex_decimal_to_long(v)",
"partitionIdNormalizer": "NO_OP"
}
}
}
```
With `numPartitions` set to `4`:
| Value | Raw result | Partition |
| ----- | ---------- | ------------------- |
| `0` | `0` | `0` |
| `3` | `3` | `3` |
| `63` | `99` | `-1` (out of range) |
### Reference the Value More Than Once
Repeat `v` when the expression needs the same value in several positions. No column name is configured anywhere in the expression.
```json theme={null}
{
"userId": {
"functionName": "Custom",
"numPartitions": 32,
"functionConfig": {
"partitionExpression": "fnv1a_hash_32_utf8(md5(toUtf8(concat(v, v))))",
"partitionIdNormalizer": "ABS"
}
}
}
```
## Invalid Configurations
| Expression | Result | Reason |
| ------------------------------------- | -------------- | ------------------------------------------------------------------ |
| `hex_decimal_to_long(partitionValue)` | Rejected | `partitionValue` is not the reserved identifier `v`. |
| `concat(v, otherColumn)` | Rejected | `otherColumn` is a second argument name. |
| `plus(1, 0)` | Rejected | The expression never references `v`. |
| `md5(v)` | Partition `-1` | Accepted by the evaluator, but returns bytes rather than a number. |
| `v` | Partition `-1` | Returns the raw string value, which is not numeric. |
Rejected expressions throw when the partition function is first constructed. Table config validation does not build the partition function, so the failure surfaces during segment build or query-time pruning rather than when you apply the config. Validate the expression on sample values before pushing it to a production table.
## Operational Considerations
1. **Changing the config does not rewrite existing segments.** Partition metadata is written at segment build, and both pruners evaluate the function recorded in each segment's own metadata rather than the current table config. Old segments therefore keep pruning correctly against the expression they were built with, and a config change does not produce wrong results. What you are left with is a table whose segments are partitioned inconsistently, which matters wherever partitioning also drives segment-to-server assignment — upsert, dedup, and `strictReplicaGroup` routing. StarTree Cloud rejects a `numPartitions` change on an already-partitioned column of an upsert or dedup table for exactly this reason. Treat these fields as fixed for the life of the table, and rebuild segments if you have to change them.
2. **Pruning applies to equality predicates.** Both layers prune on `=` against the partition column, and the broker pruner also handles `IN`. Range predicates, `LIKE`, and expressions wrapped around the column in the `WHERE` clause do not benefit.
3. **Watch for `-1` in segment metadata.** If a column's recorded partitions include `-1`, the expression is missing, failing at runtime, or returning a non-numeric result for some values. Check the segment metadata after the first segments are built rather than waiting to notice that pruning never engaged.
4. **Prefer a built-in function when one fits.** `Murmur`, `Murmur3`, `Modulo`, `HashCode`, `Fnv`, and `ByteArray` avoid per-value expression evaluation. Reach for `Custom` only when none of them produce the partition ID you need.