Custom Partition Function

Overview and Purpose

The Custom partition function computes a column’s partition ID by evaluating a Pinot scalar expression against the column value, instead of applying one of the built-in partition functions such as Murmur or Modulo. Partitioning writes a partition ID into each segment’s metadata. At query time, StarTree Cloud evaluates the same partition function on the value in an equality predicate and skips every segment whose metadata says it cannot hold that partition. Built-in partition functions hash the raw column value, which works only when the value itself is what you partitioned on. The Custom function removes that restriction: the partition ID can come from any deterministic scalar expression over the value, such as decoding a hex string or hashing a string ID after converting it to bytes. Use the Custom partition function when:

The partition ID is derived from a transformed column value rather than the raw value.
An upstream system already partitions data with a specific hash, and StarTree Cloud needs to reproduce that same partition ID to keep segment metadata aligned.
You want segment pruning on a string ID column, hashed through a scalar function chain such as fnv1a_hash_32_utf8(md5(toUtf8(v))).
A built-in partition function exists but uses the wrong normalization for your partition count.

The Custom partition function drives two different things depending on how segments are built.It always writes partition metadata into each segment, which is what enables pruning. In addition, ingestion flows that run the SegmentProcessorFramework — FileIngestionTask, merge/rollup, and realtime-to-offline tasks — read segmentPartitionConfig and bucket rows by the configured partition function, so the same expression also decides which output segment each row lands in. See Partitioning upstream data with FileIngestionTask.In those repartitioning flows, every value that evaluates to -1 is grouped into a single output bucket. Validate the expression against sample values before pointing one of these tasks at a Custom-partitioned column.

Availability

The Custom partition function is available in StarTree release 0.15 and later. The corresponding STP release branch is release/1.6.0-STP-2.164.x.

How It Works

The function is configured per column under indexingConfig.segmentPartitionConfig.columnPartitionMap. For each value it processes:

The column value is bound to the reserved identifier v in the configured partitionExpression.
The expression is evaluated as a Pinot scalar expression.
The numeric result is mapped into [0, numPartitions) by the configured partitionIdNormalizer.
If the normalized value falls inside [0, numPartitions), it becomes the partition ID.

If any of those steps cannot produce a valid partition ID, the function returns -1. Pruning is a set-membership check: segment build records both the partition IDs a segment contains and the partition function that produced them. At query time StarTree Cloud evaluates that recorded function on the literal in an equality predicate and skips segments whose recorded set does not contain the resulting ID. Because the same expression runs on both sides, a -1 is consistent and results stay correct. What you lose is selectivity — every value that fails to evaluate collapses into the same -1 bucket, which most segments will contain, so those queries prune nothing. The function returns -1 when:

No partitionExpression is configured, so the raw string value is used and is not numeric.
Expression evaluation throws at runtime, for example hex_decimal_to_long(v) on the value not-a-hex-value.
The expression returns a non-numeric result, for example md5(v), which returns bytes.
The normalized value falls outside [0, numPartitions), which is possible with the NO_OP normalizer.

Requirements

The partition column must exist in the table schema.
The partition column must be single-value.
numPartitions must be greater than 0.
Every scalar function used in the expression must be registered in the Pinot function registry.

Configuration

Configure the function through the standard ColumnPartitionConfig entry for the column.

{
  "tableName": "events_OFFLINE",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "schemaName": "events"
  },
  "indexingConfig": {
    "segmentPartitionConfig": {
      "columnPartitionMap": {
        "correlationId": {
          "functionName": "Custom",
          "numPartitions": 16,
          "functionConfig": {
            "partitionExpression": "fnv1a_hash_32_utf8(md5(toUtf8(v)))",
            "partitionIdNormalizer": "ABS"
          }
        }
      }
    }
  },
  "routing": {
    "segmentPrunerTypes": [
      "partition"
    ]
  }
}

Enabling Segment Pruning

Partition metadata is consumed by two independent pruning layers, and they need different setup:

Server-side pruning works with no extra configuration. ColumnValueSegmentPruner is part of the default server pruner list, so each server skips its own non-matching segments as soon as the request lands.
Broker-side pruning requires opting in. Add partition to routing.segmentPrunerTypes, as in the example above. Without it, the broker fans the query out to every server hosting the table and pruning happens only after dispatch. With it, the broker drops non-matching segments during routing, which also reduces the number of servers contacted.

Both layers evaluate the partition function recorded in each segment’s own metadata, not the function currently in the table config.

Configuration Parameters

Parameter	Required	Default	Description
`functionName`	Yes	—	Must be `Custom`.
`numPartitions`	Yes	—	Number of logical partitions for the column. Must be greater than `0`.
`functionConfig.partitionExpression`	No	None	Pinot scalar-function expression evaluated for the column value. Without it, every value resolves to partition `-1`.
`functionConfig.partitionIdNormalizer`	No	`POSITIVE_MODULO`	Name of the `PartitionIdNormalizer` used to map the raw expression result into `[0, numPartitions)`. Blank values fall back to the default.

Expression Rules

The partition column value is exposed to the expression as the reserved identifier v.

v is the only allowed argument name. hex_decimal_to_long(partitionValue) and concat(v, otherColumn) are both rejected.
The expression must reference v at least once. A constant expression such as plus(1, 0) is rejected.
v may appear more than once, and every occurrence is bound to the same column value. For example, fnv1a_hash_32_utf8(md5(toUtf8(concat(v, v)))) is valid.
The expression should return a number. Integer and Long results are normalized directly; any other Number is converted with its long value first, so floor(v) on "11.9" produces 11.
The expression must be deterministic. A non-deterministic scalar function produces segment metadata that does not match the partition ID computed at query time, which silently breaks pruning.

Use the key partitionExpression. Other spellings, including functionExpr, are ignored, which leaves the function without an expression and sends every value to partition -1.

Partition ID Normalizers

The normalizer maps the raw expression result to a partition ID. The default is POSITIVE_MODULO.

Normalizer	Behavior	When to use
`POSITIVE_MODULO`	`value % numPartitions`, then adds `numPartitions` if the remainder is negative.	The default. Safe for any signed result.
`ABS`	Absolute value of `value % numPartitions`.	Hash functions that can return negative integers, when you want to match this normalization.
`MASK`	Clears the sign bit, then applies modulo.	Partition counts that are compatible with bit masking.
`PRE_MODULO_ABS`	`abs(value) % numPartitions`, mapping `Integer.MIN_VALUE` and `Long.MIN_VALUE` to `0`.	Matching the legacy semantics of `HashCodePartitionFunction` and `ByteArrayPartitionFunction`.
`NO_OP`	Uses the expression result as the partition ID unchanged.	The expression already returns a valid ID in `[0, numPartitions)`. Out-of-range results become `-1`.

Normalizer names are case-insensitive.

Examples

Hash a String ID

Convert a string column to UTF-8 bytes, hash it with md5, then fold the hash into an integer. This is the common shape for pruning on a high-cardinality string ID.

{
  "correlationId": {
    "functionName": "Custom",
    "numPartitions": 16,
    "functionConfig": {
      "partitionExpression": "fnv1a_hash_32_utf8(md5(toUtf8(v)))",
      "partitionIdNormalizer": "ABS"
    }
  }
}

Queries with an equality predicate on the column can then use the partition metadata for pruning:

SELECT count(*)
FROM events
WHERE correlationId = 'correlation-123';

Convert a Hex String to a Partition ID

Decode a hexadecimal string into a long and let the default POSITIVE_MODULO normalizer map it into the partition count.

{
  "hexUserId": {
    "functionName": "Custom",
    "numPartitions": 128,
    "functionConfig": {
      "partitionExpression": "hex_decimal_to_long(v)"
    }
  }
}

With numPartitions set to 128:

Value	Raw result	Partition
`ff`	`255`	`127`
`80`	`128`	`0`
`11`	`17`	`17`
`not-a-hex-value`	Evaluation failure	`-1`

Use an Expression Result Directly

NO_OP skips normalization. Use it only when the expression already returns a valid partition ID.

{
  "partitionIdText": {
    "functionName": "Custom",
    "numPartitions": 4,
    "functionConfig": {
      "partitionExpression": "hex_decimal_to_long(v)",
      "partitionIdNormalizer": "NO_OP"
    }
  }
}

With numPartitions set to 4:

Value	Raw result	Partition
`0`	`0`	`0`
`3`	`3`	`3`
`63`	`99`	`-1` (out of range)

Reference the Value More Than Once

Repeat v when the expression needs the same value in several positions. No column name is configured anywhere in the expression.

{
  "userId": {
    "functionName": "Custom",
    "numPartitions": 32,
    "functionConfig": {
      "partitionExpression": "fnv1a_hash_32_utf8(md5(toUtf8(concat(v, v))))",
      "partitionIdNormalizer": "ABS"
    }
  }
}

Invalid Configurations

Expression	Result	Reason
`hex_decimal_to_long(partitionValue)`	Rejected	`partitionValue` is not the reserved identifier `v`.
`concat(v, otherColumn)`	Rejected	`otherColumn` is a second argument name.
`plus(1, 0)`	Rejected	The expression never references `v`.
`md5(v)`	Partition `-1`	Accepted by the evaluator, but returns bytes rather than a number.
`v`	Partition `-1`	Returns the raw string value, which is not numeric.

Rejected expressions throw when the partition function is first constructed. Table config validation does not build the partition function, so the failure surfaces during segment build or query-time pruning rather than when you apply the config. Validate the expression on sample values before pushing it to a production table.

Operational Considerations

Changing the config does not rewrite existing segments. Partition metadata is written at segment build, and both pruners evaluate the function recorded in each segment’s own metadata rather than the current table config. Old segments therefore keep pruning correctly against the expression they were built with, and a config change does not produce wrong results. What you are left with is a table whose segments are partitioned inconsistently, which matters wherever partitioning also drives segment-to-server assignment — upsert, dedup, and strictReplicaGroup routing. StarTree Cloud rejects a numPartitions change on an already-partitioned column of an upsert or dedup table for exactly this reason. Treat these fields as fixed for the life of the table, and rebuild segments if you have to change them.
Pruning applies to equality predicates. Both layers prune on = against the partition column, and the broker pruner also handles IN. Range predicates, LIKE, and expressions wrapped around the column in the WHERE clause do not benefit.
Watch for -1 in segment metadata. If a column’s recorded partitions include -1, the expression is missing, failing at runtime, or returning a non-numeric result for some values. Check the segment metadata after the first segments are built rather than waiting to notice that pruning never engaged.
Prefer a built-in function when one fits. Murmur, Murmur3, Modulo, HashCode, Fnv, and ByteArray avoid per-value expression evaluation. Reach for Custom only when none of them produce the partition ID you need.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Custom Partition Function

Overview and Purpose

Availability

How It Works

Requirements

Configuration

Enabling Segment Pruning

Configuration Parameters

Expression Rules

Partition ID Normalizers

Examples

Hash a String ID

Convert a Hex String to a Partition ID

Use an Expression Result Directly

Reference the Value More Than Once

Invalid Configurations

Operational Considerations

​Overview and Purpose

​Availability

​How It Works

​Requirements

​Configuration

​Enabling Segment Pruning

​Configuration Parameters

​Expression Rules

​Partition ID Normalizers

​Examples

​Hash a String ID

​Convert a Hex String to a Partition ID

​Use an Expression Result Directly

​Reference the Value More Than Once

​Invalid Configurations

​Operational Considerations

Overview and Purpose

Availability

How It Works

Requirements

Configuration

Enabling Segment Pruning

Configuration Parameters

Expression Rules

Partition ID Normalizers

Examples

Hash a String ID

Convert a Hex String to a Partition ID

Use an Expression Result Directly

Reference the Value More Than Once

Invalid Configurations

Operational Considerations