Avro message decoder is used for real-time ingestion in Pinot when consuming Avro-encoded messages from Kafka topics that use Confluent Schema Registry.

​
Class

org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder

​
When to Use

Use this decoder when:
  • Kafka messages are serialized using Avro format.
  • Kafka messages must be Avro-encoded binary payloads and the schema must be registered in Confluent Schema Registry. Pinot uses the registry to deserialize the messages correctly.
  • A Confluent Schema Registry is managing your Avro schemas.

​
Configuration Example

Include it in your streamConfig section: 
"streamConfig": {
  "streamType": "kafka",
  "stream.kafka.topic.name": "orders_topic",
  "stream.kafka.consumer.type": "lowlevel",
  "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
  "stream.kafka.broker.list": "broker:9092",
  "stream.kafka.decoder.prop.schema.registry.url": "http://schema-registry:8081"
}

​
Decoder Behavior

  • Automatically fetches and caches Avro schemas from the registry.
  • Deserializes messages based on the schema ID embedded in the message.
  • Converts Avro records into Pinot-compatible rows.
  • Drops fields not present in the Pinot schema.
  • Avoid nested records - flatten them before ingestion if needed.
  • Use nullHandlingEnabled if fields might be missing or optional.