Introduction

Pauseless consumption enhances real-time analytics by minimizing ingestion delays. By allowing data ingestion to continue while the previous segment is being built and uploaded, it significantly reduces the latency gap between data arrival and query availability.

How to Enable Pauseless Consumption

Enabling this feature involves two major phases: preparing the cluster/table and updating the configurations.

Step 1: Preparation (Existing Tables Only)

If you are creating a new table from scratch, you may skip to Step 2. This step is mandatory for existing tables that currently consume data from a stream.

Pause Ingestion

Stop data ingestion to ensure all current segments are committed across all servers/replicas. API: POST /tables/{tableName}/pauseConsumption

Verify Inactive State

Confirm that no consuming segments are active before proceeding.

Ideal State: Verify that all segments are marked as ONLINE.
Consuming Info: Use the API below to check for active segments.

API: GET /tables/{tableName}/consumingSegmentsInfo
Success Criteria: The field _segmentToConsumingInfoMap in the response must be empty.

Step 2: Cluster & Controller Configuration

These changes are applied at the system level and require a restart of the Servers and Controllers. a. Enable Peer Download: Ensure peer segment download is enabled on your cluster. Follow this documentation to do so.
b. Update Cluster Configuration: Add the following parameters to your cluster configuration. API: POST /cluster/configs
Use the payload below to the API:

{
    "pinot.server.instance.segment.upload.to.deep.store": "true",
    "controller.segment.level.validation.intervalPeriod" : "7200s"
}

Parameter	Value	Reason
pinot.server.instance.segment.upload.to.deep.store	true	Servers upload completed segments directly to the deep store on commit.
controller.segment.level.validation.intervalPeriod	7200s	Controls how often the segment-level validation runs, which retries deep store uploads for committed segments that are missing from the deep store. The default is 24 hours; reduce it to 2 hours for pauseless.

The following configs were previously required but are now applied automatically and no longer need to be set:

controller.realtime.segment.deepStoreUploadRetryEnabled: deep store upload retry always runs for tables with pauseless consumption enabled, regardless of this flag.
pinot.controller.segment.completion.fsm.scheme.pauseless: the pauseless segment completion FSM is registered by default and selected automatically when pauselessConsumptionEnabled is true.

Step 3: Table Configuration

Update the table configuration to enable the pauseless flag and set the peer segment download scheme. Important: The pauselessConsumptionEnabled flag must be placed outside of the streamConfigMaps array, directly under streamIngestionConfig.

"segmentsConfig": {
  "peerSegmentDownloadScheme": "https"
},
"ingestionConfig": {
  "streamIngestionConfig": {
    "pauselessConsumptionEnabled": true,
    "streamConfigMaps": [
      {
        "streamType": "kafka",
        "stream.kafka.topic.name": "githubEvents"
      }
    ]
  }
}

peerSegmentDownloadScheme is mandatory for tables with replication greater than 1 — table creation/update fails without it when pauseless consumption is enabled. It allows replicas to download a committed segment from a peer server if the segment is not yet available in the deep store.

Step 4: Resume Ingestion

Once the configurations are applied and the system is restarted (if applicable), activate pauseless ingestion by resuming the table. Use API: POST /tables/{tableName}/resumeConsumption

How to Enable Pauseless Consumption for Dedup-enabled Table

Deduplication (Dedup) ensures data correctness by removing duplicate records based on primary keys during ingestion. When combined with Pauseless Consumption, users achieve both low-latency ingestion and data integrity. Dedup is available since the StarTree plugin release, 1.4.0-ST.39. Pauseless mode normally runs background tasks (build/commit) parallel to ingestion. However, because Dedup relies on knowing the exact state of previous data to identify duplicates, strict enforcement of ordering is required across replicas. No additional cluster configuration is required beyond Step 2 above.

Table Configuration

To enable dedup with pauseless, you need to add the following configs in addition to the above mentioned configuration:

{
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  },
  "dedupConfig": {
    "dedupEnabled": true
  },
  "task": {
    "taskTypeConfigsMap": {
      "DedupSnapshotCreationTask": {
        "schedule": "0 0 12 * * ?"
      }
    }
  },
  "ingestionConfig": {
    "streamIngestionConfig": {
      "pauselessConsumptionEnabled": true,
      "enforceConsumptionInOrder": true
    }
  }
}

Configuration Parameters

Parameter	Value	Reason
instanceSelectorType	strictReplicaGroup	Required for all dedup tables. Routes queries to a single replica group so each query sees a consistent dedup state.
dedupEnabled	true	Activates record-level deduplication.
enforceConsumptionInOrder	true	Ensures Segment N+1 cannot start until Segment N finishes. Prevents data gaps if a server lags.

The following settings are applied automatically for pauseless dedup tables and no longer need to be set explicitly:

controller.segment.error.autoReset (controller config): defaults to true, and error-segment repair always runs for pauseless tables regardless of this flag.
parallelSegmentConsumptionPolicy: when unset, pauseless dedup tables automatically resolve to ALLOW_DURING_BUILD_ONLY, which releases the consumption semaphore only during the build phase (while the server still holds its exact local copy of the data) to ensure correctness.
allowDedupConsumptionDuringCommit: deprecated in favor of parallelSegmentConsumptionPolicy; the default false is the required value.
useIdealStateToCalculatePreviousSegment: defaults to false, which is the recommended value (avoids extra Zookeeper load).

Observability

Metric Name	Type	Unit	Description	Usage
Controller Metrics
`PAUSELESS_CONSUMPTION_ENABLED`	Gauge (per-table)	Boolean (0/1)	Indicates whether pauseless consumption is enabled for the table.	Quick verification if pauseless mode is active for a table.
`PAUSELESS_SEGMENTS_IN_ERROR_COUNT`	Gauge (per-table)	Count	Number of segments where ALL replicas are in ERROR state.	Critical metric to detect when segments are completely failed across all replicas. Non-zero value indicates immediate attention needed.
`PAUSELESS_SEGMENTS_IN_UNRECOVERABLE_ERROR_COUNT`	Gauge (per-table)	Count	Number of segments in COMMITTING state with all replicas in ERROR that cannot be auto-repaired.	Tracks segments that need manual recovery, typically for tables with dedup/partial upsert where auto-repair is disabled.
Server Metrics
`PAUSELESS_CONSUMPTION_ENABLED`	Gauge (per-table)	Boolean (0/1)	Server-side indicator of pauseless consumption status.	Verify server and controller are in sync about pauseless mode.

Measuring Ingestion Lag

To assess the impact of pauseless consumption, monitor the offset lag during ingestion using Grafana. Update the query for the metric, “Realtime Ingestion Delay in Ms (Avg)”, as follows:

Ingestion Lag Query (Grafana)

    avg by (${GroupBy:csv}) (pinot_server_realtimeIngestionOffsetLag_Value{kubernetes_pod_name=~"$Server$" , table=~"$Tables$"})

Debugging APIs

1. Get Pauseless Debug Information

Description: Retrieves diagnostic information to identify segments that are stuck in COMMITTING or are in ERROR state. Usage:

Error detection: Identify which servers have segments in ERROR state.
Commit tracking: Monitor segments that are stuck in the COMMITTING state.
Debugging hangs: If commits appear stalled, check whether segments remain in the committingSegments list for extended periods.

API: GET /tables/{tableName}/pauselessDebugInfo Response

{
  "instanceToErrorSegmentsMap": {
    "Server_host1_8098": ["segment1", "segment2"],
    "Server_host2_8098": ["segment3"]
  },
  "committingSegments": [
    "tableName__0__123__20231201T1000Z",
    "tableName__1__456__20231201T1005Z"
  ]
}

2. Get Re-ingestion Jobs

Description: Returns list of all currently running re-ingestion jobs.
Usage: Useful for monitoring auto-repair progress.
API: GET /reingestSegment/jobs

3. Pauseless Debug Metadata Path

Description: Tracks all segments currently in COMMITTING state for debugging purposes. Usage:

Maintains a persisted list of segments that are currently in the commit process.
Used by the controller to track the lifecycle of segment commits.
Entries are automatically cleaned up once segments complete the commit.
Useful for understanding which segments were in flight during failures and can be accessed using the API listed above.

API: GET /PAUSELESS_DEBUG_METADATA/{tableNameWithType} Response:

{
  "id": "tableName_REALTIME",
  "listFields": {
    "committingSegments": [
      "tableName__0__123__20231201T1000Z",
      "tableName__1__456__20231201T1005Z"
    ]
  }
}

Operational Troubleshooting

Ordering Enforcement

Pinot enforces segment ordering using a ConsumerCoordinator that serializes Helix OFFLINE → CONSUMING state transitions. Consumers wait for prior segments to either finish consumption or be fully loaded before starting to consume new segments. This guarantees strict ordering and prevents gaps in data ingestion. The enforcement strategy depends on the useIdealStateToCalculatePreviousSegment flag. It is strongly recommended to keep this flag set to false to avoid additional load on Zookeeper. Pinot includes a built-in safety mechanism that handles ordering without relying on IdealState.

Segment Build Failures (Rollback)

In rare scenarios, such as a server crash before a segment is uploaded, it may be necessary to roll back ingestion to the last successfully committed segment. To perform a rollback:

Pause ingestion for the table.
Identify the last valid committed segment.
Execute the delete operation using the Delete from Sequence Number API.

API: DELETE /deleteSegmentsFromSequenceNum/{tableNameWithType}?segments=segmentA,segmentB This operation deletes all segments with a sequence ID greater than or equal to the specified segment. 4. Resume ingestion for the table.

Server Restart Hangs

If a server restarts and continues consuming data but does not transition to the READY state, follow these steps:

Pause ingestion for the table.
Wait for the server to transition to READY. This typically takes a few minutes.
Resume ingestion. Once resumed, the server will catch up quickly.

Ingestion Lag Spikes

When enabling pauseless consumption, ingestion must be paused first. Resuming ingestion forces a commit of all active segments simultaneously. During this transition, expect a temporary spike in controller load and ingestion lag. This behavior is expected and limited to the enablement phase.

COMMITTING vs ERROR State

If a segment is in the COMMITTING state in Zookeeper but appears as ERROR or OFFLINE in the External View, corrective validation is required. Run the Realtime Segment Validation job manually, or wait for the scheduled execution. The schedule is controlled by controller.realtime.segment.validation.frequencyPeriod, which defaults to 1 hour.

Key Logging Patterns for Debugging

Success Path Logs

“Starting to commit changes to ZK and ideal state for the segment: during pauseless ingestion”
“Successfully updated segment metadata for segment: ”
“Removing segment: from committing segment list”

Warning Logs

“Committing segment: was not uploaded to deep store”
“Found segments with at least one replica in ERROR state”
“Segment: in table: is COMMITTING with all replicas in ERROR state”

Error Logs

“Failed to update committing segments list for table: ”
“No alive server found to re-ingest segment: in table: ”

​Introduction

​How to Enable Pauseless Consumption

​Step 1: Preparation (Existing Tables Only)

​Pause Ingestion

​Verify Inactive State

​Step 2: Cluster & Controller Configuration

​Step 3: Table Configuration

​Step 4: Resume Ingestion

​How to Enable Pauseless Consumption for Dedup-enabled Table

​Table Configuration

​Configuration Parameters

​Observability

​Measuring Ingestion Lag

​Ingestion Lag Query (Grafana)

​Debugging APIs

​1. Get Pauseless Debug Information

​2. Get Re-ingestion Jobs

​3. Pauseless Debug Metadata Path

​Operational Troubleshooting

​Ordering Enforcement

​Segment Build Failures (Rollback)

​Server Restart Hangs

​Ingestion Lag Spikes

​COMMITTING vs ERROR State

​Key Logging Patterns for Debugging

​Success Path Logs

​Warning Logs

​Error Logs

Introduction

How to Enable Pauseless Consumption

Step 1: Preparation (Existing Tables Only)

Pause Ingestion

Verify Inactive State

Step 2: Cluster & Controller Configuration

Step 3: Table Configuration

Step 4: Resume Ingestion

How to Enable Pauseless Consumption for Dedup-enabled Table

Table Configuration

Configuration Parameters

Observability

Measuring Ingestion Lag

Ingestion Lag Query (Grafana)

Debugging APIs

1. Get Pauseless Debug Information

2. Get Re-ingestion Jobs

3. Pauseless Debug Metadata Path

Operational Troubleshooting

Ordering Enforcement

Segment Build Failures (Rollback)

Server Restart Hangs

Ingestion Lag Spikes

COMMITTING vs ERROR State

Key Logging Patterns for Debugging

Success Path Logs

Warning Logs

Error Logs