> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Columnar Segment Processing

Columnar segment processing builds Pinot segments column by column instead of converting every input record into rows first. It is designed for columnar input formats such as Parquet and for workflows that rebuild existing Pinot segments.

For large Parquet ingestion jobs, columnar processing can reduce intermediate storage and improve segment generation throughput because the minion reads source columns directly into Pinot indexes.

<Note>
  Columnar segment processing is an opt-in task configuration. Keep row-based processing as the default until compare mode succeeds for your table.
</Note>

## When to use it

| Use case                       | Recommendation                                                                                                       |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| Parquet batch ingestion        | Best fit. Enable compare mode first, then switch to columnar processing.                                             |
| Existing Pinot segment rewrite | Useful when a supported task rewrites Pinot segments without row-oriented transforms.                                |
| CSV or JSON ingestion          | Keep row-based processing. Columnar processing is optimized for columnar sources.                                    |
| Tables with complex transforms | Validate carefully with compare mode. Some transform patterns may fall back or produce different precision behavior. |

## Supported task types

| Task type           | Support                      |
| ------------------- | ---------------------------- |
| `FileIngestionTask` | Supported for Parquet input. |

Additional segment rewrite tasks may use the same task properties as support is added in StarTree releases.

## Configuration properties

Set these properties in the task config for the segment-building task.

| Property                        | Default | Description                                                                                                                                    |
| ------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `useColumnarProcessing`         | `false` | Enables the columnar processing path.                                                                                                          |
| `compareRowAndColumnProcessing` | `false` | Runs both row-based and columnar processing on the same input and compares the generated segments. Use this before switching production tasks. |
| `arrowMemoryLimitMB`            | `100`   | Arrow direct-memory limit for Parquet reads. Increase for very wide or large Parquet files.                                                    |

## Enable compare mode for FileIngestionTask

Compare mode is the safest migration path. It builds row-based and columnar output for the same input and fails the task if the generated segments differ.

```json theme={null}
{
  "tableName": "orders_OFFLINE",
  "tableType": "OFFLINE",
  "task": {
    "taskTypeConfigsMap": {
      "FileIngestionTask": {
        "schedule": "0 0 * * * ?",
        "inputDirURI": "s3://example-bucket/orders/date=${yyyyMMdd}",
        "inputFormat": "parquet",
        "push.mode": "tar",
        "compareRowAndColumnProcessing": "true",
        "tableMaxNumTasks": "1",
        "arrowMemoryLimitMB": "512"
      }
    }
  }
}
```

Use `tableMaxNumTasks=1` during compare mode so the validation run does not consume unnecessary minion capacity.

Look for minion log messages that report row-based and columnar timings and whether segment comparison succeeded. If comparison fails, keep row-based processing enabled and inspect the columns reported in the comparison error.

## Enable columnar processing

After compare mode succeeds on representative input, turn compare mode off and enable columnar processing.

```json theme={null}
{
  "tableName": "orders_OFFLINE",
  "tableType": "OFFLINE",
  "task": {
    "taskTypeConfigsMap": {
      "FileIngestionTask": {
        "schedule": "0 0 * * * ?",
        "inputDirURI": "s3://example-bucket/orders/date=${yyyyMMdd}",
        "inputFormat": "parquet",
        "push.mode": "tar",
        "useColumnarProcessing": "true",
        "compareRowAndColumnProcessing": "false",
        "tableMaxNumTasks": "12",
        "arrowMemoryLimitMB": "512"
      }
    }
  }
}
```

You can also trigger an ad hoc ingestion with the same task configs:

```bash theme={null}
curl -X POST "https://<data-plane-host>/api/pinot/tasks/schedule" \
  -H "Authorization: Bearer $STARTREE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "taskType": "FileIngestionTask",
    "tableName": "orders_OFFLINE",
    "taskConfigs": {
      "inputDirURI": "s3://example-bucket/orders/backfill/2025-01-01",
      "inputFormat": "parquet",
      "push.mode": "tar",
      "useColumnarProcessing": "true",
      "arrowMemoryLimitMB": "512"
    }
  }'
```

## Memory tuning

Columnar processing keeps active column data, partition mapping, stats collectors, and segment index creators in memory. Memory use grows with:

* Number of active partitions in one task.
* Number of concurrently built segments.
* Column cardinality and configured indexes.
* Parquet file width and nested column size.

If a task runs out of memory:

| Adjustment                                              | Why it helps                                           |
| ------------------------------------------------------- | ------------------------------------------------------ |
| Reduce files per task                                   | Lowers the number of active input rows and partitions. |
| Lower `maxNumRecordsPerSegment`                         | Builds smaller segment indexes at a time.              |
| Use no-dictionary columns for high-cardinality fields   | Reduces dictionary build pressure.                     |
| Increase `arrowMemoryLimitMB`                           | Gives Arrow more direct memory for Parquet reads.      |
| Enable `optimizeNoDictStatsCollection` when appropriate | Reduces no-dictionary stats collection cost.           |

## Behavioral differences to check

Columnar output is often closer to the Parquet source representation, but it can differ from row-based processing in edge cases.

### Dependent transforms

Row-based processing evaluates all transforms for a row before final type conversion. Columnar processing can type-convert a derived column before a later derived column reads it.

For example, if `transformA` is declared as `INT` and `transformB` uses `transformA`, columnar processing may use the converted integer value while row-based processing uses the intermediate expression value. Validate dependent transforms in compare mode before enabling columnar processing.

### Nested Parquet values

Parquet struct and nested values can be represented differently from row-based conversion. For example, row-based conversion may stringify booleans or wrap nested values, while columnar processing preserves the Parquet value shape more directly.

### Unsupported features

If the task detects unsupported features, it can fall back to row-based processing. Watch minion logs when using:

* Sorted columns.
* Complex type transformers.
* Filter functions.
* Partition transform functions.
* Rollup or aggregation merge types.

## Troubleshooting

| Symptom                     | What to check                                                                                                                                           |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Compare mode fails          | Inspect the reported column differences, dependent transforms, and nested Parquet fields. Keep row-based processing until the difference is understood. |
| Direct-memory errors        | Increase `arrowMemoryLimitMB` and reduce files per task.                                                                                                |
| Lower than expected speedup | Check whether the task fell back to row-based processing because of unsupported features.                                                               |
| Segment size skew           | Lower `maxNumRecordsPerSegment` or reduce the number of input files per task.                                                                           |