> ## Documentation Index > Fetch the complete documentation index at: https://docs.startree.ai/llms.txt > Use this file to discover all available pages before exploring further. # Segment Backfill Task ## Overview As data governance, compliance, and quality expectations continue to evolve, organizations require more precise tools to manage and maintain data in real-time analytics platforms. The Minion-Based Segment Backfill Task in StarTree Cloud addresses this need by enabling selective deletion and backfill of data in a way that is both powerful and easy to control, without resorting to complex, manual workarounds. ## Key Capabilities ### Selective Data Deletion The Backfill Task supports multiple modes to select data for deletion: * **Time range** (e.g., a specific day or hour) * **Dimension filters** (e.g., currency = EUR, region = APAC) * **Combination** of time range and dimension filters It is recommended to configure one of these criteria for data deletion. If deletion is not required, use the File Ingestion Task instead. ### Selective Data Backfill Users can re-ingest targeted subsets of data, such as corrected records or missing slices, without reprocessing entire partitions or tables. ### Atomic and Consistent Updates Backfill operations are atomic by design. When segment replacement is involved, either all related segments are updated or none are, maintaining table integrity and avoiding partial updates or broken transitions. ## How Does it Work? The Backfill Task is initiated via an ad-hoc execution API and operates on time-partitioned data with all-or-none consistency. It consists of four phases: ### 1. Segment Selector Filters segments based on deletion or purge criteria. It identifies full deletions and partial purges, generating tasks with types: `Purge`, `Delete`, or `Ingest`. ### 2. Segment Purger Removes specific records from segments using defined filters, without deleting entire segments. ### 3. Segment Deletor Completely deletes segments that the Segment Selector marked for removal. ### 4. Data Ingestor Uses the FileIngestionTask to bring in new data by creating segments that replace or append to existing ones. ## Limitations | Constraint | Description | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | No Native Scheduling Support | Triggered on demand via API; users must manage automation. | | No Support for Upsert-Enabled Tables | Not supported currently. | | Task Interference | Disable other tasks (FileIngestion, SegmentRefresh, AlterTableTask) during backfill execution. | | No Support for Sync Mode enabled Tables | The Backfill Task should not be executed on the Pinot tables that were originally ingested using [sync-mode in File Ingestion Task](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/file-ingestion-task#sync-mode) | | No Support for Delta Ingested Tables | The Backfill Task should not be executed on Pinot tables that were originally ingested from a Delta table. | ## Configuration Parameters If Backfill also needs to ingest new data, it is important to configure the Segment Backfill Task using the [File Ingestion Task configuration parameters](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/file-ingestion-task). A reference table of key Backfill-related configuration parameters is provided below. | Parameter | Description | Accepted Values / Format | Example | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- | ------------------------------- | | `backfill.start.time.ms` | Start timestamp for purge/replace | Epoch ms | `1727748000000` | | `backfill.end.time.ms` | End timestamp | Epoch ms | `1727755200000` | | `backfill.field.names` | Multi-dimensional filters | Comma-separated | `industry,country` | | `backfill.field.values` | Values for field names | Comma-separated | `ENERGY,Germany` | | `backfill.comparison.operator` | Field comparison logic | `==`(default), `!=`, `<`, `<=`, `>`, `>=` | `"!="` | | `backfill.input.dir` | Data source path for ingestion. Make sure backfill input directory is different to the actual ingestion directory. | File path | `/data/clean/stocks/2024-10-01` | | `backfill.logical.operator` | Combines multiple filters | `&&`(default), `\|\|` | `\|\|` | | `backfill.input.format` | Input file format | `CSV`, `JSON`, etc. | `CSV` | ## Preview with Dry Run Use dry run before scheduling a Backfill Task. Dry run validates task configs, selects matching segments, estimates generated purge and ingestion tasks, and returns warnings without mutating table data. The dry-run endpoint supports `SegmentBackfillTask` and `SegmentPurgeTask`. ```bash theme={null} curl -X POST \ "https:///api/pinot/tasks/SegmentBackfillTask/orders_OFFLINE/dryRun?verbose=true" \ -H "Authorization: Bearer $STARTREE_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "segmentSelector": "TIME_DIMENSION", "backfill.start.time.ms": "1735689600000", "backfill.end.time.ms": "1735776000000", "backfill.field.names": "country", "backfill.field.values": "DE", "backfill.comparison.operator": "==", "backfill.input.dir": "s3://example-bucket/orders/corrections/dt=2025-01-01", "backfill.input.format": "parquet", "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "us-west-2" }' ``` Use `verbose=false` or omit the query parameter when you only need summary counts. ### Dry-run response ```json theme={null} { "mode": "dry-run", "table": "orders_OFFLINE", "summary": { "deleteCount": 2, "purgeCount": 4, "ingestFileCount": 12, "purgeTaskCount": 4, "ingestTaskCount": 3, "totalTaskCount": 7, "skippedRealtimeNotCompleted": 0, "deletePercent": 1.25, "purgePercent": 2.5, "warnings": [ "[MOSTLY_PARTIAL_OVERLAP] 4/6 segments partially overlap and require record-level purge. This is CPU/IO intensive on minions." ] }, "segments": [ { "action": "DELETE", "segment": "orders_OFFLINE_2025-01-01_0", "reason": "segment window [2025-01-01T00:00:00Z, 2025-01-01T06:00:00Z) fully inside selection window [2025-01-01T00:00:00Z, 2025-01-02T00:00:00Z)" }, { "action": "PURGE", "segment": "orders_OFFLINE_2025-01-01_1", "reason": "time predicate match `eventTime in [2025-01-01T00:00:00Z, 2025-01-02T00:00:00Z)`, segment window [2025-01-01T06:00:00Z, 2025-01-01T12:00:00Z). Dimension predicate will be matched during task execution;" }, { "action": "INGEST", "files": [ "s3://example-bucket/orders/corrections/dt=2025-01-01/part-000.parquet", "s3://example-bucket/orders/corrections/dt=2025-01-01/part-001.parquet" ], "totalBytes": "512.00MB" } ] } ``` ### How to read the output | Field | Meaning | | -------------------------------- | ----------------------------------------------------------------------------------------------------- | | `deleteCount` | Segments fully covered by the selection and planned for deletion. | | `purgeCount` | Segments partially covered by the selection and planned for record-level purge. | | `ingestFileCount` | Input files discovered under `backfill.input.dir`. | | `purgeTaskCount` | Estimated purge subtasks. | | `ingestTaskCount` | Estimated ingestion subtasks based on input file grouping. | | `totalTaskCount` | Estimated purge plus ingestion subtasks. | | `deletePercent` / `purgePercent` | Percentage of active segments affected by each action. | | `warnings` | Validation, scope, active-task, missing-input, and high-impact warnings to resolve before scheduling. | | `segments` | Verbose per-segment and per-ingestion-group details. Returned when `verbose=true`. | Treat warnings as blockers unless you intentionally want that outcome. For example, `[DELETE_ONLY]` means the plan deletes segments without replacement data, and `[DIMENSION_SCANS_ALL]` means a dimension-only selector must scan all active segments. ## Sample Task Configuration ```json theme={null} { "task": { "taskTypeConfigsMap": { "SegmentBackfillTask": { "backfill.start.time.ms": 1735689601000, "backfill.end.time.ms": 1735775999000, "backfill.input.dir": "/path/to/data", "backfill.input.format": "json", "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.accessKey": "MY_ACCESS_KEY", "input.fs.prop.secretKey": "MY_SECRET_KEY", "input.fs.prop.region": "us-west-2" } } } } ``` ## Key Use Case Scenarios ### 1. Delete a Subset of Table Remove data/segments for a specific time range. ```json theme={null} { "backfill.start.time.ms": "", "backfill.end.time.ms": "" } ``` ### 2. Replace a Subset of the Table Delete existing data and replace with clean data for a specific time range. ```json theme={null} { "backfill.start.time.ms": "", "backfill.end.time.ms": "", "backfill.input.dir": "/path/to/clean/data", "backfill.input.format": "CSV" } ``` ### 3. Mixed Operation: Delete Some, Replace Others, and Add New Segments A table has inconsistent data - some segments are invalid and should be removed, some are outdated and need replacing, and new records need to be appended. Perform a combination of: * Deleting some segments outright, * Replacing a set of segments, * Ingesting entirely new segments. ```json theme={null} { "backfill.segment.list": "segment_old1, segment_old2", "backfill.start.time.ms": "", "backfill.end.time.ms": "", "backfill.input.dir": "/path/to/new/data", "backfill.input.format": "CSV" } ``` ### 4. Replace Data Matching Multiple Values **Example: currency is EUR or USD** ```json theme={null} { "backfill.field.names": "currency, currency", "backfill.field.values": "EUR,USD", "backfill.logical.operator": "||", "backfill.input.dir": "/path/to/data", "backfill.input.format": "CSV" } ``` **Example: industry is ENERGY or country is Germany** ```json theme={null} { "backfill.field.names": "industry,country", "backfill.field.values": "ENERGY,Germany", "backfill.logical.operator": "||", "backfill.input.dir": "/path/to/data", "backfill.input.format": "CSV" } ``` **Example: currency is not EUR** ```json theme={null} { "backfill.field.names": "currency", "backfill.field.values": "EUR", "backfill.comparison.operator": "!=", "backfill.input.dir": "/path/to/data", "backfill.input.format": "CSV" } ``` ### 5. Replace with Time and Multi-dimension Filter ```json theme={null} { "backfill.start.time.ms": "", "backfill.end.time.ms": "", "backfill.field.names": "currency, currency", "backfill.field.values": "EUR,USD", "backfill.logical.operator": "||", "backfill.input.dir": "/path/to/data", "backfill.input.format": "CSV" } ``` ### 6. Add New Data or Segments Only **Use the File Ingestion Task when no deletion is needed.** ## FAQs & Recommendations No, segment purger performs record-level scans. No, it’s not supported at the moment. Yes, Backfill Task can be used for deletion only. A time based or dimension based filter can also be applied for selective deletion. No, use the File Ingestion Task for that. No, disable concurrent tasks. Cluster Health Check dashboard can give insights into Segment sizes and skewness if there is any. Use Alter Table Task after Backfill to optimize the Segment size.