Protobuf Message Decoder

​

Class

​

When to Use

• The Kafka messages are serialized using Protocol Buffers (Protobuf).
• The schema must be registered in the Confluent Schema Registry.
• You need to ingest structured Protobuf data into a Pinot real-time table.

​

Configuration Example

"streamConfig": {
  "streamType": "kafka",
  "stream.kafka.topic.name": "metrics_topic",
  "stream.kafka.consumer.type": "lowlevel",
  "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder",
  "stream.kafka.broker.list": "broker:9092",
  "stream.kafka.decoder.prop.schema.registry.url": "<SCHEMA_REGISTRY_URL>"
}

​

Decoder Behavior

• Retrieves and caches Protobuf schemas from the Schema Registry.
• Uses the schema to parse Protobuf messages into column-value pairs.
• The decoder fetches the schema dynamically using the schema ID in the message payload.
• Ignores fields not defined in the Pinot schema.
• Supports nested fields if flattened before ingestion.
• Enable nullHandlingEnabled for optional fields.