> ## Documentation Index > Fetch the complete documentation index at: https://docs.startree.ai/llms.txt > Use this file to discover all available pages before exploring further. # Prometheus Remote Write Decoders > Decode Prometheus remote-write protobuf samples and native histograms into Pinot rows. Prometheus remote-write decoders convert Prometheus remote-write protobuf payloads into Pinot rows during realtime ingestion. Use these decoders when your stream contains Prometheus `WriteRequest` messages rather than JSON-like Prometheus records. This page is different from the [Prometheus Message Decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/prometheus), which documents the JSON-style `PrometheusMessageDecoder`. Remote-write payloads use Prometheus protobuf wire format. ## Decoder Classes | Decoder | Class | Output | | ----------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------- | | Sample decoder | `ai.startree.pinot.plugin.inputformat.prometheus.PrometheusRemoteWriteSampleDecoder` | One row per counter/gauge sample. | | Histogram decoder | `ai.startree.pinot.plugin.inputformat.prometheus.PrometheusRemoteWriteHistogramDecoder` | One row per native histogram or summary histogram point. | Both decoders parse a Prometheus remote-write `WriteRequest`, iterate through each time series, and emit multiple Pinot rows from one stream message. ## When to Use Use the sample decoder for: * Counters. * Gauges. * Prometheus time series where each data point is a timestamp/value sample. Use the histogram decoder for: * Prometheus native histograms. * Histogram or summary-like remote-write payloads that need bucket spans and counts preserved. Use separate Pinot tables for samples and histograms unless your upstream stream already separates those payloads. ## Sample Decoder Schema The sample decoder emits: | Field | Type | Description | | -------- | --------------------- | -------------------------------------------- | | `time` | `TIMESTAMP` or `LONG` | Prometheus sample timestamp in milliseconds. | | `name` | `STRING` | Metric name from the `__name__` label. | | `value` | `DOUBLE` | Sample value. | | `labels` | `STRING[]` or `JSON` | Labels encoded as `name:value` strings. | Example schema: ```json theme={null} { "schemaName": "prom_remote_samples", "dimensionFieldSpecs": [ {"name": "name", "dataType": "STRING"}, {"name": "labels", "dataType": "STRING", "singleValueField": false} ], "metricFieldSpecs": [ {"name": "value", "dataType": "DOUBLE"} ], "dateTimeFieldSpecs": [ { "name": "time", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Realtime table stream config: ```json theme={null} { "streamType": "kafka", "stream.kafka.topic.name": "prometheus-remote-samples", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "broker-1:9092,broker-2:9092", "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.prometheus.PrometheusRemoteWriteSampleDecoder", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } ``` Example query: ```sql theme={null} SELECT DATETRUNC('minute', time) AS minute, AVG(value) FROM prom_remote_samples WHERE name = 'http_requests_total' AND ARRAY_CONTAINS(labels, 'status:500') GROUP BY minute ORDER BY minute ``` ## Histogram Decoder Schema The histogram decoder preserves positive and negative bucket spans, bucket deltas, bucket counts, and zero-bucket metadata. | Field | Type | Description | | ------------------ | --------------------- | --------------------------------------- | | `time` | `TIMESTAMP` or `LONG` | Histogram timestamp in milliseconds. | | `name` | `STRING` | Metric name from the `__name__` label. | | `labels` | `STRING[]` or `JSON` | Labels encoded as `name:value` strings. | | `neg_span_lengths` | `INT[]` | Negative bucket span lengths. | | `neg_span_offsets` | `INT[]` | Negative bucket span offsets. | | `neg_span_deltas` | `LONG[]` | Negative bucket deltas. | | `neg_span_counts` | `DOUBLE[]` | Negative bucket counts. | | `pos_span_lengths` | `INT[]` | Positive bucket span lengths. | | `pos_span_offsets` | `INT[]` | Positive bucket span offsets. | | `pos_span_deltas` | `LONG[]` | Positive bucket deltas. | | `pos_span_counts` | `DOUBLE[]` | Positive bucket counts. | | `zero_threshold` | `DOUBLE` | Zero bucket threshold. | | `zero_count_int` | `LONG` | Integer zero bucket count. | | `zero_count_float` | `DOUBLE` | Floating-point zero bucket count. | Example schema: ```json theme={null} { "schemaName": "prom_remote_histograms", "dimensionFieldSpecs": [ {"name": "name", "dataType": "STRING"}, {"name": "labels", "dataType": "STRING", "singleValueField": false}, {"name": "neg_span_lengths", "dataType": "INT", "singleValueField": false}, {"name": "neg_span_offsets", "dataType": "INT", "singleValueField": false}, {"name": "neg_span_deltas", "dataType": "LONG", "singleValueField": false}, {"name": "neg_span_counts", "dataType": "DOUBLE", "singleValueField": false}, {"name": "pos_span_lengths", "dataType": "INT", "singleValueField": false}, {"name": "pos_span_offsets", "dataType": "INT", "singleValueField": false}, {"name": "pos_span_deltas", "dataType": "LONG", "singleValueField": false}, {"name": "pos_span_counts", "dataType": "DOUBLE", "singleValueField": false} ], "metricFieldSpecs": [ {"name": "zero_threshold", "dataType": "DOUBLE"}, {"name": "zero_count_int", "dataType": "LONG"}, {"name": "zero_count_float", "dataType": "DOUBLE"} ], "dateTimeFieldSpecs": [ { "name": "time", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Realtime table stream config: ```json theme={null} { "streamType": "kafka", "stream.kafka.topic.name": "prometheus-remote-histograms", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "broker-1:9092,broker-2:9092", "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.prometheus.PrometheusRemoteWriteHistogramDecoder", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } ``` ## Output Behavior Each remote-write message can contain many time series, and each time series can contain many samples or histograms. The decoders emit multiple rows from one input payload. Boundary safeguards: | Limit | Value | Behavior | | -------------------------------- | ----- | ----------------------------- | | Time series per message | `500` | Logs a warning when exceeded. | | Samples or histograms per series | `500` | Logs a warning when exceeded. | | Labels per series | `50` | Logs a warning when exceeded. | The decoder extracts the metric name from the `__name__` label. If no `__name__` label exists, or more than one exists, the row is emitted with a missing metric name and a warning is logged. Labels are emitted as concatenated strings: ```text theme={null} __name__:http_requests_total status:500 method:GET instance:server-01 ``` ## Modeling Tips * Keep samples and histograms in separate tables unless you have a downstream reason to co-locate them. * Use a sorted or timestamp index on `time` for common time-window queries. * Add an inverted index on `name` because almost every query filters by metric name. * If you frequently filter by a small set of labels, consider extracting those labels upstream into dedicated Pinot columns. * Keep the raw multi-value `labels` column for exploratory queries and long-tail labels. Example optimized sample table field config: ```json theme={null} { "fieldConfigList": [ { "name": "name", "encodingType": "DICTIONARY", "indexTypes": ["INVERTED"] }, { "name": "labels", "encodingType": "DICTIONARY", "indexTypes": ["INVERTED"] } ] } ``` ## Troubleshooting | Symptom | Cause | Fix | | ---------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------- | | No rows emitted | Stream payload is not a Prometheus remote-write `WriteRequest`. | Confirm the producer writes Prometheus protobuf remote-write payloads. | | `name` is null | Missing or duplicate `__name__` label. | Fix upstream labels so each time series has exactly one metric name label. | | Queries by label are slow | Labels are stored only in a multi-value column. | Promote hot labels to dedicated columns or add an inverted index to `labels`. | | Histogram fields are empty | The payload contains samples, not histograms. | Use `PrometheusRemoteWriteSampleDecoder` for sample payloads. | | Sample table receives histogram payloads | Mixed payloads or wrong topic routing. | Split upstream topics or configure separate tables/decoders. |