JFR Record Reader

Use the JFR record reader to convert Java Flight Recorder recordings into Pinot rows for offline analysis. Each JFR event becomes a row, with common event metadata and flattened event fields available as Pinot columns.

Classes

Purpose	Class
Record reader	`org.apache.pinot.plugin.inputformat.jfr.JFRRecordReader`
Record reader config	`org.apache.pinot.plugin.inputformat.jfr.JFRRecordReaderConfig`

When to Use

Use this reader when you want to:

Load .jfr recordings into an offline Pinot table.
Analyze JVM, GC, thread, allocation, and profiling events with SQL.
Ingest StarTree query tracing recordings and join operational JFR fields with trace context.
Process LZ4-compressed JFR files whose file name ends in .lz4.

Input Files

The reader supports:

Input	Behavior
`.jfr`	Opens the recording directly with `jdk.jfr.consumer.RecordingFile`.
`.lz4`	Decompresses the file into a temporary `.jfr` file, reads it, and deletes the temporary file when the reader closes.

Use file-name filters to make sure the ingestion job only selects JFR files:

includeFileNamePattern: 'glob:**/*.jfr'

For compressed recordings:

includeFileNamePattern: 'glob:**/*.lz4'

Record Reader Config

The JFR config has one option:

Config	Default	Description
`denormalize`	`false`	When `true`, enriches non-StarTree events with StarTree trace context from overlapping StarTree JFR events on the same thread.

Standalone Batch Job Example

Use dataFormat: 'jfr' with the JFR record reader and config class:

executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'

jobType: SegmentCreationAndTarPush
inputDirURI: 's3://<BUCKET_NAME>/jfr/'
includeFileNamePattern: 'glob:**/*.jfr'
outputDirURI: 's3://<BUCKET_NAME>/profileEvents/'
overwriteOutput: true

recordReaderSpec:
  dataFormat: 'jfr'
  className: 'org.apache.pinot.plugin.inputformat.jfr.JFRRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.jfr.JFRRecordReaderConfig'
  configs:
    denormalize: true

tableSpec:
  tableName: 'profileEvents'
  schemaURI: 'http://localhost:9000/tables/profileEvents/schema'
  tableConfigURI: 'http://localhost:9000/tables/profileEvents'

pinotClusterSpecs:
  - controllerURI: 'http://localhost:9000'

FileIngestionTask Example

For minion-based file ingestion, use the JFR input format and keep the file filter narrow:

{
  "task": {
    "taskTypeConfigsMap": {
      "FileIngestionTask": {
        "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
        "input.fs.prop.region": "<REGION>",
        "inputDirURI": "s3://<BUCKET_NAME>/jfr/",
        "includeFileNamePattern": "glob:**/*.jfr",
        "inputFormat": "jfr",
        "recordReader.prop.denormalize": "true",
        "tableMaxNumTasks": "<MAX_NUM_TASKS>",
        "taskMaxDataSize": "1G",
        "push.mode": "metadata",
        "schedule": "<QUARTZ_SCHEDULE>"
      }
    }
  }
}

Use the standalone job spec when you need to explicitly set className and configClassName. Use FileIngestionTask when your cluster has the jfr format registered and available to minions.

Output Fields

Every output row includes:

Column	Type	Description
`eventType`	`STRING`	JFR event type name, such as `jdk.ExecutionSample` or a StarTree event type.
`startTime`	`LONG` or `TIMESTAMP`	Event start time in epoch milliseconds.
`duration`	`LONG`	Event duration in milliseconds.

The reader also flattens JFR event fields:

JFR Field Type	Output Behavior
Simple scalar values	Written to a column with the same field name.
`RecordedThread`	Produces `threadJVMName`, `threadOSName`, `threadOSID`, `threadJVMID`, and `threadGroup`.
`RecordedStackTrace`	Produces an array column using the original field name, and `frame` containing the first frame.
`RecordedClassLoader`	Writes the class loader name.
`RecordedClass`	Writes the class name.
`RecordedMethod`	Writes `ClassName.methodName`.
Other nested `RecordedObject` values	Flattens child fields by appending capitalized child names to the parent field name.

Because JFR event types differ, the set of populated columns varies by row. Use default null values or Pinot null handling for sparse event fields.

Minimal Schema Example

This schema captures common fields that are useful for query profiling:

{
  "schemaName": "profileEvents",
  "dimensionFieldSpecs": [
    {"name": "eventType", "dataType": "STRING"},
    {"name": "frame", "dataType": "STRING"},
    {"name": "threadJVMName", "dataType": "STRING"},
    {"name": "threadOSName", "dataType": "STRING"},
    {"name": "threadGroup", "dataType": "STRING"},
    {"name": "spanName", "dataType": "STRING"}
  ],
  "metricFieldSpecs": [
    {"name": "duration", "dataType": "LONG", "defaultNullValue": 0},
    {"name": "threadJVMID", "dataType": "LONG", "defaultNullValue": 0},
    {"name": "threadOSID", "dataType": "LONG", "defaultNullValue": 0},
    {"name": "traceId", "dataType": "LONG", "defaultNullValue": 0},
    {"name": "spanId", "dataType": "LONG", "defaultNullValue": 0}
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "startTime",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]
}

Table Config Example

Use indexes on event type, stack frame, trace fields, and time ranges to make profile exploration faster:

{
  "tableName": "profileEvents",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "segmentPushType": "REFRESH",
    "replication": "1",
    "timeColumnName": "startTime"
  },
  "tableIndexConfig": {
    "sortedColumn": ["threadJVMID"],
    "loadMode": "MMAP",
    "invertedIndexColumns": [
      "frame",
      "eventType",
      "traceId",
      "spanName"
    ],
    "rangeIndexColumns": [
      "duration",
      "startTime"
    ]
  }
}

Denormalized StarTree Trace Context

When denormalize is true, the reader performs a pre-pass over the recording:

Finds StarTree JFR events whose event type starts with startree and that contain traceId.
Records each event’s threadId, startTime, duration, traceId, spanId, and spanName.
Rewinds the recording.
Enriches non-StarTree events on the same thread when the event start time falls inside a recorded StarTree trace frame.

The enriched row gets:

{
  "traceId": 123456789,
  "spanId": 987654321,
  "spanName": "ScanFilterOperator"
}

Enable this when the recording contains StarTree tracing events and you want JVM/profile events to be queryable by trace/span context.

Example Queries

Find the busiest event types:

SELECT eventType, COUNT(*) AS events, SUM(duration) AS totalDurationMs
FROM profileEvents
GROUP BY eventType
ORDER BY events DESC
LIMIT 20;

Find hot stack frames:

SELECT frame, COUNT(*) AS samples
FROM profileEvents
WHERE eventType = 'jdk.ExecutionSample'
GROUP BY frame
ORDER BY samples DESC
LIMIT 50;

Inspect events for a trace:

SELECT startTime, eventType, spanName, frame, duration
FROM profileEvents
WHERE traceId = 123456789
ORDER BY startTime
LIMIT 1000;

Troubleshooting

Symptom	Cause	Fix
No rows are ingested	File filter does not match JFR files, or the input directory is wrong.	Check `inputDirURI` and `includeFileNamePattern`.
Reader initialization fails	File is not a valid JFR recording, or a `.lz4` file cannot be decompressed.	Validate the recording with JDK tools and confirm the file extension matches the content.
`traceId`, `spanId`, and `spanName` are empty	`denormalize` is disabled or the recording does not contain StarTree trace events.	Set `denormalize: true` and confirm the recording has `startree` events with trace fields.
Many columns are null	Different JFR event types populate different fields.	Keep only useful fields in the Pinot schema and set default null values where appropriate.
Stack traces are hard to query	Stack traces are arrays and `frame` only stores the first frame.	Use `frame` for top-frame analysis and add schema/index support for stack-trace arrays only when needed.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Classes

When to Use

Input Files

Record Reader Config

Standalone Batch Job Example

FileIngestionTask Example

Output Fields

Minimal Schema Example

Table Config Example

Denormalized StarTree Trace Context

Example Queries

Troubleshooting

​Classes

​When to Use

​Input Files

​Record Reader Config

​Standalone Batch Job Example

​FileIngestionTask Example

​Output Fields

​Minimal Schema Example

​Table Config Example

​Denormalized StarTree Trace Context

​Example Queries

​Troubleshooting

Classes

When to Use

Input Files

Record Reader Config

Standalone Batch Job Example

FileIngestionTask Example

Output Fields

Minimal Schema Example

Table Config Example

Denormalized StarTree Trace Context

Example Queries

Troubleshooting