Alter Table Task

The StarTreeAlterTableTask keeps segments in sync with the latest Pinot table configuration by detecting inconsistencies, rebuilding affected segments, and replacing outdated segments atomically. It is used for changes such as re-sorting, re-partitioning, adding indexes, resizing segments, and syncing deep store segments with the table configuration.

How is the Alter Table Task different from the Segment Refresh Task?

The StarTree Alter Table Task is an improved alternative to the Segment Refresh Task, and is designed for more efficient, batch-wise segment refreshes. The Segment Refresh Task performs the map and segment generation phases locally, which can create redundant segment files and extra merge work. The StarTree Alter Table Task writes mapper output files to shared storage, then generates replacement segments directly from that output. The Alter Table Task follows a structured Map, Reduce, and Upload workflow. This gives better resource utilization and fault tolerance for large table refreshes. Built-in retries adjust memory and segment-generation parameters when subtasks fail.

When is the Alter Table Task used?

Re-partitioning in case of a change in the partition columns.
Re-sorting by changing the sorted column.
Merge or split segments to keep segment sizes healthy.
Change a column data type.
Change a column encoding.
Adding new indexes.
Other table config changes that cannot be applied on the server side with segment reload.

How does the Alter Table Task work?

The Alter Table Task is a Minion task in Apache Pinot, designed to run asynchronously without affecting real-time or offline ingestion. The Alter Table Task refreshes table segments in batches. In each batch, the task selects a subset of segments, processes them, and replaces them in the original table. The process repeats until all segments are updated.

If new segments requiring a refresh arrive during an ongoing batch, they are deferred until the current batch completes.

Example workflow

Suppose we need to refresh a 10TB table, with the following table configuration:

tableMaxNumTasks: 100 (the default for non-upsert Alter Table Task runs)
desiredSegmentSize: 500MB

The Alter Table Task workflow splits the table into batches based on this formula:

Each map task processes data up to taskMaxDataSize, defaulting to 5GB, or maxNumRecordsPerTask, defaulting to 3 times the computed maxNumRecordsPerSegment.
With tableMaxNumTasks set to 100 and taskMaxDataSize at 5GB, each batch can process up to about 500GB.
Therefore, it will take approximately 20 batches (10TB / 500GB = 20) to process a 10TB table.

Batch Processing Stages

Each batch has these stages:

The Map Phase reads input segments and generates intermediate files. Each subtask partitions the data (partition_timeBucket) and uploads the files to deep storage.
The Reduce Phase downloads the necessary intermediate files, processes them to generate new segments and metadata, and then uploads both to deep storage.
The Upload Phase retrieves the list of fromSegments from metadata, determines toSegments from deep storage, and performs batch segment replacement. By default, batch upload is used as it is the most efficient mode.

All stages are associated with the same session ID, ensuring continuity and data integrity.

Enabling the Alter Table Task

To enable the Alter Table Task, simply configure it in your table configuration as follows:

In task.taskTypeConfigsMap, add an object called StarTreeAlterTableTask.
Add the following configuration:

{
  "task": {
    "taskTypeConfigsMap": {
      "StarTreeAlterTableTask": {
        "bucketTimePeriod": "1d",
        "desiredSegmentSize": "500MB",
        "schedule": "0 */10 * ? * * *"
      }
    }
  }
}

Run the task immediately using the POST /tasks/schedule?taskType=StarTreeAlterTableTask&tableName=<TABLE_NAME_WITH_TYPE> API, or wait for the cron job to trigger it. In the example above, the task is scheduled every 10 minutes.

Configuration Parameters

The following parameters are configured under task.taskTypeConfigsMap.StarTreeAlterTableTask.

Scheduling and mode

Parameter	Required	Default	Applies to	Description
`schedule`	No	None	Scheduled runs	Quartz cron expression for recurring task generation. If omitted, trigger the task manually with the task scheduling API.
`bucketTimePeriod`	Yes, unless `reloadOnly=true`	None	Non-upsert Map/Reduce/Upload runs	Time bucket for output segments. A generated segment should not span more than one bucket.
`reloadOnly`	No	`false`	Deep store sync	When `true`, ATT runs only the reload phase and refreshes stale deep-store segments in place. This is the mode used by Deep Store Segment Sync.
`forceReload`	No	`false`	Deep store sync	Used only when `reloadOnly=true`. Reloads all selected segments without relying on the table-hash freshness check.
`minionInstanceTag`	No	Untagged Minion instance tag	All runs	Restricts generated ATT subtasks to Minions with the configured Helix instance tag.

Upsert tables use the upsert refresh path. In that path, validDocIds*, invalidRecordsThresholdPercent, and maxSegmentsPerTask apply; reloadOnly and forceReload are deep-store-sync controls and do not change upsert compaction selection.

Segment selection and task sizing

Parameter	Required	Default	Applies to	Description
`tableMaxNumTasks`	No	`100` for non-upsert ATT; `1000` for upsert tables	All runs	Maximum subtasks generated per table per scheduling round. Lower this on large tables to reduce upload, replacement, and disk pressure.
`desiredSegmentSize`	No	`500MB`	All runs	Desired output segment size. When set, ATT computes `maxNumRecordsPerSegment` from table statistics. This takes precedence over `maxNumRecordsPerSegment`.
`maxNumRecordsPerSegment`	No	Computed from `desiredSegmentSize`; fallback code default is `5000000`	All runs	Maximum rows per output segment when `desiredSegmentSize` is not set or cannot be computed.
`maxNumRecordsPerTask`	No	`3 * maxNumRecordsPerSegment`	Non-upsert Map/Reduce/Upload runs	Maximum records processed by a map task.
`taskMaxDataSize`	No	`5GB`	Non-upsert Map/Reduce/Upload runs	Maximum input bytes processed by a map task.
`maxNumRowsSegmentDiffPercentThreshold`	No	`20` when `desiredSegmentSize` is used; otherwise `0`	Non-upsert Map/Reduce/Upload runs	Skips resizing when an existing segment’s row count is within this percentage of the target.
`segmentSelector`	No	None	All runs	Expression used to filter selected segments by segment metadata, for example `segment.creation.time < 1780617600000`.
`segmentExpirationBufferPeriod`	No	`0`	All runs	Skips segments that are already expired or will expire within this extra retention buffer. Set a negative value to disable expiration filtering.
`bufferTimePeriod`	No	`3d`	Realtime tables	Skips recently completed realtime segments and the latest committed segment, so ATT does not interfere with ingestion offsets.

Output segment and rollup options

Parameter	Required	Default	Applies to	Description
`generateSegmentOnlyAtPartitionTimeBucketBoundary`	No	`true`	Non-upsert Reduce phase	Keeps generated output segments from crossing the configured partition time bucket boundary.
`mergeType`	No	Pinot segment-generation default	Non-upsert Reduce phase	Advanced merge/rollup mode forwarded to Pinot segment generation. Omit this for ordinary ATT rebuilds.
`roundBucketTimePeriod`	No	None	Non-upsert rollup runs	Optional time rounding period forwarded to Pinot segment generation when rollup-style output is required.
`<column>.aggregationType`	No	None	Non-upsert rollup runs	Per-column aggregation function forwarded to Pinot segment generation, for example `views.aggregationType`.

Upsert table options

Parameter	Required	Default	Description
`validDocIdsType`	No	`SNAPSHOT`, or `SNAPSHOT_WITH_DELETE` when `upsertConfig.deleteRecordColumn` is set	Source used to fetch valid-doc-ids from servers. Supported values are `SNAPSHOT`, `IN_MEMORY`, `SNAPSHOT_WITH_DELETE`, and `IN_MEMORY_WITH_DELETE`. Snapshot modes require snapshots to be enabled in `upsertConfig`.
`validDocIdsConsensusMode`	No	`EQUAL`	Replica consensus policy when fetching valid-doc-ids for the deep-store segment CRC. Supported values are `UNSAFE`, `EQUAL`, and `MOST_VALID_DOCS`. Leave the default unless directed by StarTree support.
`validDocIdsFetchTimeoutMs`	No	`5000`	Timeout for fetching valid-doc-ids metadata from servers. Increase this for large tables or slow servers.
`validDocIdsFetchBatchSize`	No	`500`	Number of segments requested per server call when fetching valid-doc-ids metadata. Lower this if server requests are too heavy.
`invalidRecordsThresholdPercent`	No	`20`	A segment is eligible for refresh only when invalid records exceed this percentage. Use `0` to refresh any segment with invalid records.
`minPrematureBatchSize`	No	`0`	Minimum segment count required before flushing a partial batch when a segment is skipped during upsert batching.
`maxSegmentsPerTask`	No	`100`	Maximum input segments assigned to a single upsert refresh subtask. Lower this to reduce per-task memory, valid-doc-ids fetch size, and temporary disk growth.
`mergeMode`	No	`carryOverLineage`	Upsert segment lineage mode. Supported internal values include `carryOverLineage`, `noCarryOverLineage`, and `reProcessSegments`; use non-default modes only with StarTree support.
`segmentMapperFileSizeThresholdInBytes`	No	Pinot segment processor default	Upsert executor only. Optional intermediate mapper file size threshold passed to the upsert segment processor.

Large upsert table guidance

On large upsert tables, high ATT parallelism can temporarily increase table storage because new segments are uploaded before old segments are fully replaced and deleted. If validDocIdsType uses snapshot mode and snapshot generation lags behind ATT, old segments can remain visible to cleanup logic for longer than expected.

For large upsert maintenance runs:

Prefer conservative tableMaxNumTasks and maxSegmentsPerTask values.
Use segmentSelector to process older or narrower segment ranges first.
Use bufferTimePeriod to avoid segments close to active ingestion.
Use validDocIdsType: "IN_MEMORY" or IN_MEMORY_WITH_DELETE for one-off maintenance when valid-doc-ids snapshots are stale or snapshot generation takes too long. Snapshot mode is still preferred for routine runs when snapshots are fresh.
Increase validDocIdsFetchTimeoutMs and reduce validDocIdsFetchBatchSize when valid-doc-ids requests time out or overload servers.

Example large upsert run:

{
  "task": {
    "taskTypeConfigsMap": {
      "StarTreeAlterTableTask": {
        "bufferTimePeriod": "1d",
        "maxSegmentsPerTask": "3",
        "tableMaxNumTasks": "100",
        "validDocIdsFetchTimeoutMs": "60000",
        "validDocIdsFetchBatchSize": "100",
        "invalidRecordsThresholdPercent": "0",
        "segmentSelector": "segment.creation.time < 1780617600000",
        "validDocIdsType": "IN_MEMORY"
      }
    }
  }
}

Segment freshness checks

Parameter	Required	Default	Applies to	Description
`skipSegmentIndexCheck`	No	`false`	Non-upsert Map/Reduce/Upload runs	Skips the metadata compatibility check that detects segments whose indexes, schema, sorting, partitioning, or column metadata do not match the current table config.
`useServerApiForSegmentStaleCheck`	No	`true`	Non-upsert Map/Reduce/Upload runs	Uses the server stale-segment API instead of controller-side batched metadata reads. Set to `false` to use the older batched metadata path.
`segmentMetadataFetchBatchSize`	No	`10000`	Non-upsert Map/Reduce/Upload runs with `useServerApiForSegmentStaleCheck=false`	Number of segment metadata records fetched per batch during compatibility checks.

Deep store, push, tiered storage, and cleanup

Parameter	Required	Default	Description
`output.segment.dir.uri`	No	Table deep-store directory	Shared storage location for ATT intermediate files and generated segment artifacts. Override only for advanced deployments or tests.
`overwriteOutput`	No	`false`; `true` for `reloadOnly` metadata writes when absent	Allows ATT to overwrite existing files in the output directory.
`push.mode`	No	`METADATA` in StarTree Minion ATT push config	Segment push mode. ATT normally uses metadata push; use other modes only for advanced deployments.
`batchSegmentUpload`	No	`true`	Enables batch segment upload when `push.mode` is `METADATA`.
`push.segmentUriPrefix`	No	None	Optional URI prefix passed through to Pinot segment push.
`push.segmentUriSuffix`	No	None	Optional URI suffix passed through to Pinot segment push.
`push.attempts`	No	Pinot batch push default	Number of segment push attempts.
`push.retry.interval.millis`	No	Pinot batch push default	Delay between segment push retries.
`directUploadToTier`	No	Minion config `pinot.minion.direct.tier.upload.enable`, default `true`	For tiered storage tables, upload generated segments directly to the target tier when applicable.
`cleanupDanglingIntermediateFiles`	No	`true`	Cleans historical dangling ATT intermediate files after a session cleanup.
`bufferDaysToPurgeOutputSegments`	No	`3`	Extra retention buffer used when deleting old segment-list files from deep store.
`skipDeepStoreCleanup`	No	`false`	Skips deletion of ATT intermediate files and session cleanup artifacts. Use only for troubleshooting.
`endReplaceSegmentsTimeoutMs`	No	`3600000`	Timeout for the final `endReplaceSegments` controller call in the Upload phase.
`numMetadataFileDownloadThreads`	No	`4`	Parallelism for Reduce-phase metadata file downloads. Increase if metadata file download times out.

Execution and memory tuning

Parameter	Required	Default	Description
`intermediateFileFormat`	No	`ARROW`	Intermediate mapper output format. Supported values are `ARROW`, `PARQUET`, and `GENERIC_ROW`.
`numTaskExecutorServiceThreads`	No	Available processors	Thread count for Map and Upload phase executor services.
`numSegmentGenerationThreads`	No	`2`	Thread count for Reduce phase segment generation. Retries can lower this value automatically.
`maxSubTaskRetryCount`	No	`5`	Maximum retries per failed subtask before ATT cleans up the session.
`offHeapMemoryAllocationForArrow`	No	Minimum of half MaxDirectMemory and 4GB; fallback 2GB	Total Arrow off-heap memory allocation before division by concurrent task count.
`upperCapForOffHeapMemoryAllocationForArrow`	No	`4294967296`	Upper cap, in bytes, used when deriving the default Arrow off-heap allocation.
`offHeapMemoryToMaxBatchBytesFactor`	No	`2`	Divides Arrow off-heap allocation to derive `maxBatchBytes` when `maxBatchBytes` is absent.
`maxBatchBytes`	No	Derived from Arrow off-heap memory	Maximum Arrow batch bytes before flushing mapper output. Retries lower this value down to a 200MB floor.
`maxBatchRows`	No	`1000000`	Maximum Arrow batch rows before flushing mapper output.
`rowsPerLoadForSortedCase`	No	`5000`	Rows loaded at a time for sorted-column reduce processing.
`fileWriterBufferSizeInBytes`	No	`1048576`	File writer buffer size for intermediate output.
`useFixedWindowScan`	No	`false`	Uses fixed-window reads for Arrow sorted-case processing. Use only when directed by StarTree support.
`use.InMemorySegmentProcessorFramework`	No	`false`	Experimental in-memory segment processor path. Use only when directed by StarTree support.

Ingestion transform controls

By default, ATT rebuild subtasks skip user-defined ingestion filters and transforms to avoid reapplying non-idempotent ingestion logic to already-built segments. Timestamp-index-generated transform configs and safety transformers still apply. Enable these only when you know the transform should be re-evaluated during rebuild.

Parameter	Required	Default	Description
`skipIngestionFilters`	No	`true`	Skips inherited `ingestionConfig.filterConfig`.
`usePreComplexTypeTransformer`	No	`false`	Reuses inherited pre-complex enrichment configs.
`useComplexTypeTransformer`	No	`false`	Reuses inherited `complexTypeConfig`.
`usePostComplexTypeTransformer`	No	`false`	Reuses inherited post-complex enrichment configs, user transform configs, field transform functions, and schema-conforming transformer config unless overridden by the more specific flags below.
`useIngestionTransformFunctions`	No	Same as `usePostComplexTypeTransformer`	Reuses user-authored `transformConfigs` and non-virtual schema `transformFunction` values.
`useSchemaConformingTransformer`	No	Same as `usePostComplexTypeTransformer`	Reuses inherited `schemaConformingTransformerConfig`.
`ingestionTransformPipelineConfig`	No	None	JSON object that replaces the whole user-defined transform pipeline for the task. When set, it takes precedence over the shortcut flags below.
`ingestionFilterConfig`	No	None	JSON task-level replacement for `ingestionConfig.filterConfig`.
`preComplexTypeTransformerConfigs`	No	None	JSON list of pre-complex `EnrichmentConfig` replacements.
`complexTypeTransformerConfig`	No	None	JSON task-level replacement for `complexTypeConfig`.
`postComplexTypeTransformerConfigs`	No	None	JSON list of post-complex `EnrichmentConfig` replacements.
`ingestionTransformConfigs`	No	None	JSON list of task-level `TransformConfig` replacements.
`fieldTransformFunctions`	No	None	JSON object mapping schema column names to task-level field transform functions.
`schemaConformingTransformerConfig`	No	None	JSON task-level replacement for `schemaConformingTransformerConfig`.

Internal phase properties

ATT generates additional subtask properties such as taskSessionId, isReloadMode, isMapMode, isReduceMode, isUploadMode, segmentsToRefresh, toSegmentsList, subTaskIdentifier, retryCount, numConcurrentTasks, partitionId, sequenceId, segmentName, downloadURL, uploadURL, enableReplaceSegments, tableName, dataChunksRowCount, sortColumnUrls, nonSortColumnUrls, partitionBucketTimePeriod, segmentNamePrefix, taskNamePrefix, taskNameSuffix, deepStoreSegmentsDir, mapperDir, dataDir, metadata, and bytesPerRow. These are generated and managed by ATT. Do not set them in the table config unless StarTree support specifically asks you to recover or debug a task session.

Monitoring

You can monitor the StarTree Alter Table Task in two levels of granularity:

Batch-Level: Monitoring individual task batches.
Cycle-Level: Tracking overall task cycles.

Batch-Level Monitoring

Batch-level monitoring can be done using the Minion Task Manager or Minion Metadata.

Name-Based Task Monitoring: Each task name contains two key details:
- Session ID: Identifies a specific batch within a cycle.
- Task Phase: Indicates whether the task is in the Map, Reduce, or Upload phase.
- Since multiple batches can run within a cycle, they are linked together using the session ID.
Minion Metadata-Based Monitoring: Minion Metadata provides real-time batch tracking through Zookeeper:
- Path: Zookeeper Browser -> MINION_TASK_METADATA -> TableName -> StarTreeAlterTableTask.
- Each batch has a fixed list of input segments, which remains unchanged during processing.

Cycle-Level Monitoring

Cycle-level monitoring helps track the overall progress of the Alter Table Task across multiple batches. Progress Tracking & Estimation API:

Endpoint: /alterTable/tableNameWithType/estimatedTime
Method: GET

Sample Response:

{
  "estimatedCompletionTimeMs": "123456789",
  "percentProgress": "50.37"
}

New Timer Metrics for Observability

Additional metrics have been introduced for better tracking of each task phase:

ALTER_TABLE_MAP_PHASE_DURATION_MS: Time taken for the Map phase.
ALTER_TABLE_REDUCE_PHASE_DURATION_MS: Time taken for the Reduce phase.
ALTER_TABLE_UPLOAD_PHASE_DURATION_MS: Time taken for the Upload phase.

These metrics help monitor task performance and identifying potential bottlenecks during execution.

Migrating from the Segment Refresh Task

To migrate, simply change the task name from SegmentRefreshTask to StarTreeAlterTableTask.

The Alter Table Task configuration is backward compatible with the Segment Refresh Task. Only one of these tasks is needed, so there is no need to run both.

The Alter Table Task will eventually replace the Segment Refresh Task as the default.

Handling Task Failures and Retries

Minion subtasks can fail due to various reasons. When this happens, tasks are automatically retried with adjusted configurations to improve success rates:

Map Phase: maxBatchBytes = maxBatchBytes/(retryCount + 1).
Reduce Phase: numSegmentGenerationThreads = numSegmentGenerationThreads/2, with a minimum value of 1.

By handling failures efficiently, the system ensures seamless segment refreshes with minimal disruptions.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

How is the Alter Table Task different from the Segment Refresh Task?

When is the Alter Table Task used?

How does the Alter Table Task work?

Example workflow

Batch Processing Stages

Enabling the Alter Table Task

Configuration Parameters

Scheduling and mode

Segment selection and task sizing

Output segment and rollup options

Upsert table options

Large upsert table guidance

Segment freshness checks

Deep store, push, tiered storage, and cleanup

Execution and memory tuning

Ingestion transform controls

Internal phase properties

Monitoring

Batch-Level Monitoring

Cycle-Level Monitoring

New Timer Metrics for Observability

Migrating from the Segment Refresh Task

Handling Task Failures and Retries

​How is the Alter Table Task different from the Segment Refresh Task?

​When is the Alter Table Task used?

​How does the Alter Table Task work?

​Example workflow

​Batch Processing Stages

​Enabling the Alter Table Task

​Configuration Parameters

​Scheduling and mode

​Segment selection and task sizing

​Output segment and rollup options

​Upsert table options

​Large upsert table guidance

​Segment freshness checks

​Deep store, push, tiered storage, and cleanup

​Execution and memory tuning

​Ingestion transform controls

​Internal phase properties

​Monitoring

​Batch-Level Monitoring

​Cycle-Level Monitoring

​New Timer Metrics for Observability

​Migrating from the Segment Refresh Task

​Handling Task Failures and Retries

How is the Alter Table Task different from the Segment Refresh Task?

When is the Alter Table Task used?

How does the Alter Table Task work?

Example workflow

Batch Processing Stages

Enabling the Alter Table Task

Configuration Parameters

Scheduling and mode

Segment selection and task sizing

Output segment and rollup options

Upsert table options

Large upsert table guidance

Segment freshness checks

Deep store, push, tiered storage, and cleanup

Execution and memory tuning

Ingestion transform controls

Internal phase properties

Monitoring

Batch-Level Monitoring

Cycle-Level Monitoring

New Timer Metrics for Observability

Migrating from the Segment Refresh Task

Handling Task Failures and Retries