Purge
, Delete
, or Ingest
.
Constraint | Description |
---|---|
No Native Scheduling Support | Triggered on demand via API; users must manage automation. |
No Preview or Dry Run Mode | No simulation capability; filters should be validated thoroughly. |
No Support for Upsert-Enabled Tables | Not supported currently. |
Task Interference | Disable other tasks (FileIngestion, SegmentRefresh, AlterTableTask) during Backfill execution. |
Parameter | Description | Accepted Values / Format | Example |
---|---|---|---|
backfill.start.time.ms | Start timestamp for purge/replace | Epoch ms | 1727748000000 |
backfill.end.time.ms | End timestamp | Epoch ms | 1727755200000 |
backfill.field.names | Multi-dimensional filters | Comma-separated | industry,country |
backfill.field.values | Values for field names | Comma-separated | ENERGY,Germany |
backfill.comparison.operator | Field comparison logic | == (default), != , < , <= , > , >= | "!=" |
backfill.input.dir | Data source path | File path | /data/clean/stocks/2024-10-01 |
backfill.logical.operator | Combines multiple filters | && (default), || | || |
backfill.input.format | Input file format | CSV , JSON , etc. | CSV |
Is custom dimension partitioning required?
Can I use Backfill Task on Upsert tables?
Can Backfill Task be used for deletion only?
Should I use Backfill to only ingest new data?
Can it run with other tasks?
What if new data is small and segments become uneven?