Avro Message Decoder

​

Class

​

When to Use

• Kafka messages are serialized using Avro format.
• Kafka messages must be Avro-encoded binary payloads and the schema must be registered in Confluent Schema Registry. Pinot uses the registry to deserialize the messages correctly.
• A Confluent Schema Registry is managing your Avro schemas.

​

Configuration Example

"streamConfig": {
  "streamType": "kafka",
  "stream.kafka.topic.name": "orders_topic",
  "stream.kafka.consumer.type": "lowlevel",
  "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
  "stream.kafka.broker.list": "broker:9092",
  "stream.kafka.decoder.prop.schema.registry.url": "<SCHEMA_REGISTRY_URL>"
}

​

Decoder Behavior

• Automatically fetches and caches Avro schemas from the registry.
• Deserializes messages based on the schema ID embedded in the message.
• Converts Avro records into Pinot-compatible rows.
• Drops fields not present in the Pinot schema.
• Avoid nested records - flatten them before ingestion if needed.
• Use nullHandlingEnabled if fields might be missing or optional.