> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# JFR Record Reader

> Ingest Java Flight Recorder files into Pinot using the StarTree JFR record reader.

Use the JFR record reader to convert Java Flight Recorder recordings into Pinot rows for offline analysis. Each JFR event becomes a row, with common event metadata and flattened event fields available as Pinot columns.

## Classes

| Purpose              | Class                                                           |
| -------------------- | --------------------------------------------------------------- |
| Record reader        | `org.apache.pinot.plugin.inputformat.jfr.JFRRecordReader`       |
| Record reader config | `org.apache.pinot.plugin.inputformat.jfr.JFRRecordReaderConfig` |

## When to Use

Use this reader when you want to:

* Load `.jfr` recordings into an offline Pinot table.
* Analyze JVM, GC, thread, allocation, and profiling events with SQL.
* Ingest StarTree query tracing recordings and join operational JFR fields with trace context.
* Process LZ4-compressed JFR files whose file name ends in `.lz4`.

## Input Files

The reader supports:

| Input  | Behavior                                                                                                             |
| ------ | -------------------------------------------------------------------------------------------------------------------- |
| `.jfr` | Opens the recording directly with `jdk.jfr.consumer.RecordingFile`.                                                  |
| `.lz4` | Decompresses the file into a temporary `.jfr` file, reads it, and deletes the temporary file when the reader closes. |

Use file-name filters to make sure the ingestion job only selects JFR files:

```yaml theme={null}
includeFileNamePattern: 'glob:**/*.jfr'
```

For compressed recordings:

```yaml theme={null}
includeFileNamePattern: 'glob:**/*.lz4'
```

## Record Reader Config

The JFR config has one option:

| Config        | Default | Description                                                                                                                    |
| ------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `denormalize` | `false` | When `true`, enriches non-StarTree events with StarTree trace context from overlapping StarTree JFR events on the same thread. |

## Standalone Batch Job Example

Use `dataFormat: 'jfr'` with the JFR record reader and config class:

```yaml theme={null}
executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'

jobType: SegmentCreationAndTarPush
inputDirURI: 's3://<BUCKET_NAME>/jfr/'
includeFileNamePattern: 'glob:**/*.jfr'
outputDirURI: 's3://<BUCKET_NAME>/profileEvents/'
overwriteOutput: true

recordReaderSpec:
  dataFormat: 'jfr'
  className: 'org.apache.pinot.plugin.inputformat.jfr.JFRRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.jfr.JFRRecordReaderConfig'
  configs:
    denormalize: true

tableSpec:
  tableName: 'profileEvents'
  schemaURI: 'http://localhost:9000/tables/profileEvents/schema'
  tableConfigURI: 'http://localhost:9000/tables/profileEvents'

pinotClusterSpecs:
  - controllerURI: 'http://localhost:9000'
```

## FileIngestionTask Example

For minion-based file ingestion, use the JFR input format and keep the file filter narrow:

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "FileIngestionTask": {
        "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
        "input.fs.prop.region": "<REGION>",
        "inputDirURI": "s3://<BUCKET_NAME>/jfr/",
        "includeFileNamePattern": "glob:**/*.jfr",
        "inputFormat": "jfr",
        "recordReader.prop.denormalize": "true",
        "tableMaxNumTasks": "<MAX_NUM_TASKS>",
        "taskMaxDataSize": "1G",
        "push.mode": "metadata",
        "schedule": "<QUARTZ_SCHEDULE>"
      }
    }
  }
}
```

Use the standalone job spec when you need to explicitly set `className` and `configClassName`. Use `FileIngestionTask` when your cluster has the `jfr` format registered and available to minions.

## Output Fields

Every output row includes:

| Column      | Type                  | Description                                                                  |
| ----------- | --------------------- | ---------------------------------------------------------------------------- |
| `eventType` | `STRING`              | JFR event type name, such as `jdk.ExecutionSample` or a StarTree event type. |
| `startTime` | `LONG` or `TIMESTAMP` | Event start time in epoch milliseconds.                                      |
| `duration`  | `LONG`                | Event duration in milliseconds.                                              |

The reader also flattens JFR event fields:

| JFR Field Type                       | Output Behavior                                                                                 |
| ------------------------------------ | ----------------------------------------------------------------------------------------------- |
| Simple scalar values                 | Written to a column with the same field name.                                                   |
| `RecordedThread`                     | Produces `threadJVMName`, `threadOSName`, `threadOSID`, `threadJVMID`, and `threadGroup`.       |
| `RecordedStackTrace`                 | Produces an array column using the original field name, and `frame` containing the first frame. |
| `RecordedClassLoader`                | Writes the class loader name.                                                                   |
| `RecordedClass`                      | Writes the class name.                                                                          |
| `RecordedMethod`                     | Writes `ClassName.methodName`.                                                                  |
| Other nested `RecordedObject` values | Flattens child fields by appending capitalized child names to the parent field name.            |

Because JFR event types differ, the set of populated columns varies by row. Use default null values or Pinot null handling for sparse event fields.

## Minimal Schema Example

This schema captures common fields that are useful for query profiling:

```json theme={null}
{
  "schemaName": "profileEvents",
  "dimensionFieldSpecs": [
    {"name": "eventType", "dataType": "STRING"},
    {"name": "frame", "dataType": "STRING"},
    {"name": "threadJVMName", "dataType": "STRING"},
    {"name": "threadOSName", "dataType": "STRING"},
    {"name": "threadGroup", "dataType": "STRING"},
    {"name": "spanName", "dataType": "STRING"}
  ],
  "metricFieldSpecs": [
    {"name": "duration", "dataType": "LONG", "defaultNullValue": 0},
    {"name": "threadJVMID", "dataType": "LONG", "defaultNullValue": 0},
    {"name": "threadOSID", "dataType": "LONG", "defaultNullValue": 0},
    {"name": "traceId", "dataType": "LONG", "defaultNullValue": 0},
    {"name": "spanId", "dataType": "LONG", "defaultNullValue": 0}
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "startTime",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]
}
```

## Table Config Example

Use indexes on event type, stack frame, trace fields, and time ranges to make profile exploration faster:

```json theme={null}
{
  "tableName": "profileEvents",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "segmentPushType": "REFRESH",
    "replication": "1",
    "timeColumnName": "startTime"
  },
  "tableIndexConfig": {
    "sortedColumn": ["threadJVMID"],
    "loadMode": "MMAP",
    "invertedIndexColumns": [
      "frame",
      "eventType",
      "traceId",
      "spanName"
    ],
    "rangeIndexColumns": [
      "duration",
      "startTime"
    ]
  }
}
```

## Denormalized StarTree Trace Context

When `denormalize` is `true`, the reader performs a pre-pass over the recording:

1. Finds StarTree JFR events whose event type starts with `startree` and that contain `traceId`.
2. Records each event's `threadId`, `startTime`, `duration`, `traceId`, `spanId`, and `spanName`.
3. Rewinds the recording.
4. Enriches non-StarTree events on the same thread when the event start time falls inside a recorded StarTree trace frame.

The enriched row gets:

```json theme={null}
{
  "traceId": 123456789,
  "spanId": 987654321,
  "spanName": "ScanFilterOperator"
}
```

Enable this when the recording contains StarTree tracing events and you want JVM/profile events to be queryable by trace/span context.

## Example Queries

Find the busiest event types:

```sql theme={null}
SELECT eventType, COUNT(*) AS events, SUM(duration) AS totalDurationMs
FROM profileEvents
GROUP BY eventType
ORDER BY events DESC
LIMIT 20;
```

Find hot stack frames:

```sql theme={null}
SELECT frame, COUNT(*) AS samples
FROM profileEvents
WHERE eventType = 'jdk.ExecutionSample'
GROUP BY frame
ORDER BY samples DESC
LIMIT 50;
```

Inspect events for a trace:

```sql theme={null}
SELECT startTime, eventType, spanName, frame, duration
FROM profileEvents
WHERE traceId = 123456789
ORDER BY startTime
LIMIT 1000;
```

## Troubleshooting

| Symptom                                       | Cause                                                                              | Fix                                                                                                      |
| --------------------------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| No rows are ingested                          | File filter does not match JFR files, or the input directory is wrong.             | Check `inputDirURI` and `includeFileNamePattern`.                                                        |
| Reader initialization fails                   | File is not a valid JFR recording, or a `.lz4` file cannot be decompressed.        | Validate the recording with JDK tools and confirm the file extension matches the content.                |
| `traceId`, `spanId`, and `spanName` are empty | `denormalize` is disabled or the recording does not contain StarTree trace events. | Set `denormalize: true` and confirm the recording has `startree` events with trace fields.               |
| Many columns are null                         | Different JFR event types populate different fields.                               | Keep only useful fields in the Pinot schema and set default null values where appropriate.               |
| Stack traces are hard to query                | Stack traces are arrays and `frame` only stores the first frame.                   | Use `frame` for top-frame analysis and add schema/index support for stack-trace arrays only when needed. |