Use the SegmentPurgeTask to purge records from a Pinot table, for example, to ensure GDPR (General Data Protection Regulation) compliance. TheSegmentPurgeTask reads input files from a remote location, like AWS S3, to build hash keys used to uniquely identify records to purge in a Pinot table.

Pinot Version: 0.13.0-ST.58+

TheSegmentBackfillTaskoffers extended functionality and performance when compared to the SegmentPurgeTask and will eventually replace this task as the default.

​
Configure SegmentPurgeTask

Configure SegmentPurgeTask under taskConfig in the table configuration.

Property NameRequiredDescription
input.fs.classNameYesThe class name used to read the files from source location.
inputFormatYesThe input file format.
inputDirURIYesThe input dir containing the purge input files.
input.fs.prop.accessKeyNoRequired if reading from AWS S3.
input.fs.prop.secretKeyNoRequired if reading from AWS S3.
input.fs.prop.regionNoRequired if reading from AWS S3.
recordReader.prop.delimiterNoDefaults to ”,”. Supported values are ”,” and ”;“.
max.num.purge.input.filesNoMaximum number of input files processed at a time. Default value is 10.
max.total.purge.input.file.sizeNoMaximum total size of all input files combined. Default value is 100_000_000 bytes. A single file cannot exceed this value.
table.max.num.tasksNoMaximum number of minion sub tasks generated per task invocation. Default value is 10.

​
Example Table Configuration

"task": {
  "taskTypeConfigsMap": 
  {
    "SegmentPurgeTask": 
    {
      "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
      "inputFormat": "CSV",
      "inputDirURI": "s3://myBucket/myTable/input/",
      "input.fs.prop.accessKey": "MY_ACCESS_KEY",
      "input.fs.prop.secretKey": "MY_SECRET_KEY",
      "input.fs.prop.region": "us-west-2",
      "table.max.num.tasks": "100",
      "schedule": "0 */5 * * * ?"
    }
  }
}

​
Example Input Files

​
File#1

File Name: purgeRecords1.csv

userId
1000
1001
1002
1003

If the above input file was processed against a table named users, all records that match the userId field in the Pinot table would be deleted.

​
File#2

FileName: purgeRecords2.csv

firstName,lastName
john,doe
jane,doe

If the above input file was processed against a table named users, all records that match both the firstName and lastName fields in the Pinot table would be deleted.

​
Guidance on input files

  • An input file must not be appended or overwritten.
  • New data should not be generated for the records which are to be purged. If this does happen, new input files must be dropped.
  • We recommend periodically cleaning up the input directory to purge files that have been processed.

​
Limits

Note the following limits.

ItemLimitDescription
Input file formatCSVContents within the input file should conform to CSV format.
Input files sourceAWS S3Only S3 is supported as of now.
Data Typesint, long, boolean, stringThe input files must contain fields that conform to one of these types.
Field ValuesSingle ValuesAll fields in the input file must be of single value dimension. Multi values are not supported.
Comparison Type==Fields from the input record would be matched against the Pinot record using the equals operator.
Null ValueNot SupportedMatching against null values are not supported.
Empty ValueNot SupportedMatching against emtpy values are not supported.
Input file Name ExtensionNot RequiredFile extension is not required.
Input file field delimiter”,” and ”;“Other delimeters like space, tab are not supported. Default value is ”,”.

​
Limitations

  • SegmentPurgeTask works with both OFFLINE and REALTIME tables. The task is not tested against upsert enabled REALTIME tables.
  • Running other tasks such as SegmentRefresh or MergeRollup along with the SegmentPurgeTask can lead to issues. The SegmentRefresh and MergeRollup tasks needs to be disabled while the SegmentPurgeTask is running to avoid race conditions. These tasks can be re-enabled after the SegmentPurgeTask completes.

​
Field Values

The string literal null is not supported as a field value. Records with this value are not purged.

As an example, consider the input file purgeRecords.csv.

firstName,lastName
john,null

This input record would be skipped, and no record in the corresponding Pinot table would be deleted. You can make use of alternate fields in the table to delete these records.

​
Frequently Asked Questions