Protobuf message decoder is used in real-time ingestion to handle Protobuf-encoded messages from Kafka topics using the Confluent Schema Registry for schema management.

​
Class

org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder

​
When to Use

Use this decoder when:
  • The Kafka messages are serialized using Protocol Buffers (Protobuf).
  • The schema must be registered in the Confluent Schema Registry.
  • You need to ingest structured Protobuf data into a Pinot real-time table.

​
Configuration Example

Specify this decoder in the streamConfig of your table config: 
"streamConfig": {
  "streamType": "kafka",
  "stream.kafka.topic.name": "metrics_topic",
  "stream.kafka.consumer.type": "lowlevel",
  "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder",
  "stream.kafka.broker.list": "broker:9092",
  "stream.kafka.decoder.prop.schema.registry.url": "http://schema-registry:8081"
}

​
Decoder Behavior

  • Retrieves and caches Protobuf schemas from the Schema Registry.
  • Uses the schema to parse Protobuf messages into column-value pairs.
  • The decoder fetches the schema dynamically using the schema ID in the message payload.
  • Ignores fields not defined in the Pinot schema.
  • Supports nested fields if flattened before ingestion.
  • Enable nullHandlingEnabled for optional fields.