Columnar Segment Processing

Columnar segment processing builds Pinot segments column by column instead of converting every input record into rows first. It is designed for columnar input formats such as Parquet and for workflows that rebuild existing Pinot segments. For large Parquet ingestion jobs, columnar processing can reduce intermediate storage and improve segment generation throughput because the minion reads source columns directly into Pinot indexes.

Columnar segment processing is an opt-in task configuration. Keep row-based processing as the default until compare mode succeeds for your table.

When to use it

Use case	Recommendation
Parquet batch ingestion	Best fit. Enable compare mode first, then switch to columnar processing.
Existing Pinot segment rewrite	Useful when a supported task rewrites Pinot segments without row-oriented transforms.
CSV or JSON ingestion	Keep row-based processing. Columnar processing is optimized for columnar sources.
Tables with complex transforms	Validate carefully with compare mode. Some transform patterns may fall back or produce different precision behavior.

Supported task types

Task type	Support
`FileIngestionTask`	Supported for Parquet input.

Additional segment rewrite tasks may use the same task properties as support is added in StarTree releases.

Configuration properties

Set these properties in the task config for the segment-building task.

Property	Default	Description
`useColumnarProcessing`	`false`	Enables the columnar processing path.
`compareRowAndColumnProcessing`	`false`	Runs both row-based and columnar processing on the same input and compares the generated segments. Use this before switching production tasks.
`arrowMemoryLimitMB`	`100`	Arrow direct-memory limit for Parquet reads. Increase for very wide or large Parquet files.

Enable compare mode for FileIngestionTask

Compare mode is the safest migration path. It builds row-based and columnar output for the same input and fails the task if the generated segments differ.

{
  "tableName": "orders_OFFLINE",
  "tableType": "OFFLINE",
  "task": {
    "taskTypeConfigsMap": {
      "FileIngestionTask": {
        "schedule": "0 0 * * * ?",
        "inputDirURI": "s3://example-bucket/orders/date=${yyyyMMdd}",
        "inputFormat": "parquet",
        "push.mode": "tar",
        "compareRowAndColumnProcessing": "true",
        "tableMaxNumTasks": "1",
        "arrowMemoryLimitMB": "512"
      }
    }
  }
}

Use tableMaxNumTasks=1 during compare mode so the validation run does not consume unnecessary minion capacity. Look for minion log messages that report row-based and columnar timings and whether segment comparison succeeded. If comparison fails, keep row-based processing enabled and inspect the columns reported in the comparison error.

Enable columnar processing

After compare mode succeeds on representative input, turn compare mode off and enable columnar processing.

{
  "tableName": "orders_OFFLINE",
  "tableType": "OFFLINE",
  "task": {
    "taskTypeConfigsMap": {
      "FileIngestionTask": {
        "schedule": "0 0 * * * ?",
        "inputDirURI": "s3://example-bucket/orders/date=${yyyyMMdd}",
        "inputFormat": "parquet",
        "push.mode": "tar",
        "useColumnarProcessing": "true",
        "compareRowAndColumnProcessing": "false",
        "tableMaxNumTasks": "12",
        "arrowMemoryLimitMB": "512"
      }
    }
  }
}

You can also trigger an ad hoc ingestion with the same task configs:

curl -X POST "https://<data-plane-host>/api/pinot/tasks/schedule" \
  -H "Authorization: Bearer $STARTREE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "taskType": "FileIngestionTask",
    "tableName": "orders_OFFLINE",
    "taskConfigs": {
      "inputDirURI": "s3://example-bucket/orders/backfill/2025-01-01",
      "inputFormat": "parquet",
      "push.mode": "tar",
      "useColumnarProcessing": "true",
      "arrowMemoryLimitMB": "512"
    }
  }'

Memory tuning

Columnar processing keeps active column data, partition mapping, stats collectors, and segment index creators in memory. Memory use grows with:

Number of active partitions in one task.
Number of concurrently built segments.
Column cardinality and configured indexes.
Parquet file width and nested column size.

If a task runs out of memory:

Adjustment	Why it helps
Reduce files per task	Lowers the number of active input rows and partitions.
Lower `maxNumRecordsPerSegment`	Builds smaller segment indexes at a time.
Use no-dictionary columns for high-cardinality fields	Reduces dictionary build pressure.
Increase `arrowMemoryLimitMB`	Gives Arrow more direct memory for Parquet reads.
Enable `optimizeNoDictStatsCollection` when appropriate	Reduces no-dictionary stats collection cost.

Behavioral differences to check

Columnar output is often closer to the Parquet source representation, but it can differ from row-based processing in edge cases.

Dependent transforms

Row-based processing evaluates all transforms for a row before final type conversion. Columnar processing can type-convert a derived column before a later derived column reads it. For example, if transformA is declared as INT and transformB uses transformA, columnar processing may use the converted integer value while row-based processing uses the intermediate expression value. Validate dependent transforms in compare mode before enabling columnar processing.

Nested Parquet values

Parquet struct and nested values can be represented differently from row-based conversion. For example, row-based conversion may stringify booleans or wrap nested values, while columnar processing preserves the Parquet value shape more directly.

Unsupported features

If the task detects unsupported features, it can fall back to row-based processing. Watch minion logs when using:

Sorted columns.
Complex type transformers.
Filter functions.
Partition transform functions.
Rollup or aggregation merge types.

Troubleshooting

Symptom	What to check
Compare mode fails	Inspect the reported column differences, dependent transforms, and nested Parquet fields. Keep row-based processing until the difference is understood.
Direct-memory errors	Increase `arrowMemoryLimitMB` and reduce files per task.
Lower than expected speedup	Check whether the task fell back to row-based processing because of unsupported features.
Segment size skew	Lower `maxNumRecordsPerSegment` or reduce the number of input files per task.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Columnar Segment Processing

When to use it

Supported task types

Configuration properties

Enable compare mode for FileIngestionTask

Enable columnar processing

Memory tuning

Behavioral differences to check

Dependent transforms

Nested Parquet values

Unsupported features

Troubleshooting

​When to use it

​Supported task types

​Configuration properties

​Enable compare mode for FileIngestionTask

​Enable columnar processing

​Memory tuning

​Behavioral differences to check

​Dependent transforms

​Nested Parquet values

​Unsupported features

​Troubleshooting

When to use it

Supported task types

Configuration properties

Enable compare mode for FileIngestionTask

Enable columnar processing

Memory tuning

Behavioral differences to check

Dependent transforms

Nested Parquet values

Unsupported features

Troubleshooting