> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect to Google Cloud Managed Service for Apache Kafka

> Ingest from Google Cloud Managed Service for Apache Kafka with SASL/OAUTHBEARER and IAM, without service account keys.

> [!NOTE]
>
> This feature is available starting in **StarTree release 0.17.0**. It requires a StarTree data plane on Google Cloud.

StarTree connects to Google Cloud Managed Service for Apache Kafka with SASL/OAUTHBEARER. Pinot authenticates with the Google identity of your StarTree data plane, so the connection needs no service account key, password or other secret. You control access with IAM.

## Prerequisites

### Grant access to the StarTree principal

Pinot authenticates as the IAM principal of its Kubernetes service account. StarTree Support provides the principal for your data plane. It has this form:

```text theme={null}
principal://iam.googleapis.com/projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<PROJECT_ID>.svc.id.goog/subject/ns/<NAMESPACE>/sa/<SERVICE_ACCOUNT>
```

Grant the principal the Managed Kafka Client role (`roles/managedkafka.client`) on the project that contains the cluster:

```bash theme={null}
gcloud projects add-iam-policy-binding <KAFKA_PROJECT_ID> \
  --role=roles/managedkafka.client \
  --member="<PRINCIPAL>"
```

IAM changes can take a few minutes to apply.

If the topic has Kafka ACLs, also allow the principal to `READ` the topic. Topics without ACLs are open to every authenticated principal.

### Connect the network

Pinot must reach the cluster's bootstrap address. Connect the subnet of your StarTree data plane to the cluster; StarTree Support provides the subnet. Managed Kafka then creates DNS entries for the cluster in that network.

* The subnet must be in the cluster's region. A cluster accepts up to 10 subnets.
* If the data plane runs in a different project from the cluster, the data plane's project must grant the Managed Kafka Service Agent role (`roles/managedkafka.serviceAgent`) to `service-<CLUSTER_PROJECT_NUMBER>@gcp-sa-managedkafka.iam.gserviceaccount.com`. Coordinate this step with StarTree Support.

For details, see [Configure networking for Managed Service for Apache Kafka](https://docs.cloud.google.com/managed-service-for-apache-kafka/docs/networking-kafka).

### Get the bootstrap address

```bash theme={null}
gcloud managed-kafka clusters describe <CLUSTER_ID> \
  --location=<REGION> \
  --format="value(bootstrapAddress)"
```

Use the address exactly as returned, for example `bootstrap.<CLUSTER_ID>.<REGION>.managedkafka.<KAFKA_PROJECT_ID>.cloud.goog:9092`.

## Create the table

**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select Custom as the Data Source.**

**3. Create a New Connection.**

Select **Streaming** as the Data Source Category, and use this connection JSON:

```json theme={null}
{
  "stream.kafka.broker.list": "<BOOTSTRAP_ADDRESS>",
  "security.protocol": "SASL_SSL",
  "sasl.mechanism": "OAUTHBEARER",
  "sasl.login.callback.handler.class": "ai.startree.pinot.kafka.gcp.GcpManagedKafkaLoginCallbackHandler",
  "sasl.client.callback.handler.class": "ai.startree.pinot.kafka.gcp.GcpManagedKafkaClientCallbackHandler",
  "sasl.jaas.config": "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.broker.list` | Yes | The cluster's bootstrap address. |
| `security.protocol` | Yes | `SASL_SSL`. |
| `sasl.mechanism` | Yes | `OAUTHBEARER`. |
| `sasl.login.callback.handler.class` | Yes | `ai.startree.pinot.kafka.gcp.GcpManagedKafkaLoginCallbackHandler`. Gets a Google access token for the data plane's principal. |
| `sasl.client.callback.handler.class` | Yes | `ai.startree.pinot.kafka.gcp.GcpManagedKafkaClientCallbackHandler`. Checks the settings of every Kafka client before it sends the token. |
| `sasl.jaas.config` | Yes | `org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;`. To override the principal Pinot derives, add `principal="<PRINCIPAL>"` before the semicolon. Set it only when StarTree Support asks you to. |

**4. Test the Connection and Configure Data Ingestion**

Click **Test Connection**. When the test succeeds, configure ingestion:

```json theme={null}
{
  "stream.kafka.topic.name": "<TOPIC_NAME>",
  "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory",
  "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
  "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
  "realtime.segment.flush.threshold.rows": "0",
  "realtime.segment.flush.threshold.segment.size": "200M",
  "realtime.segment.flush.threshold.time": "24h"
}
```

These are the standard Kafka ingestion properties. For descriptions, see **Configure Data Ingestion** in [Connect to Apache Kafka](https://docs.startree.ai/corecapabilities/ingestdata/dataportal/streaming/kafka.md).

**5. Preview the Data**

Click **Show Sample Data** to preview the source data.

> [accordion] Table configuration example
>
> To create the table without the Data Portal, put the connection and ingestion properties in `tableIndexConfig.streamConfigs`:
>
> ```json theme={null}
> {
>   "tableName": "<TABLE_NAME>",
>   "tableType": "REALTIME",
>   "segmentsConfig": {
>     "timeColumnName": "<TIME_COLUMN>",
>     "retentionTimeUnit": "DAYS",
>     "retentionTimeValue": "7",
>     "replication": "1"
>   },
>   "tenants": {
>     "broker": "DefaultTenant",
>     "server": "DefaultTenant"
>   },
>   "tableIndexConfig": {
>     "streamConfigs": {
>       "streamType": "kafka",
>       "stream.kafka.topic.name": "<TOPIC_NAME>",
>       "stream.kafka.broker.list": "<BOOTSTRAP_ADDRESS>",
>       "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory",
>       "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
>       "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
>       "realtime.segment.flush.threshold.rows": "0",
>       "realtime.segment.flush.threshold.segment.size": "200M",
>       "realtime.segment.flush.threshold.time": "24h",
>       "security.protocol": "SASL_SSL",
>       "sasl.mechanism": "OAUTHBEARER",
>       "sasl.login.callback.handler.class": "ai.startree.pinot.kafka.gcp.GcpManagedKafkaLoginCallbackHandler",
>       "sasl.client.callback.handler.class": "ai.startree.pinot.kafka.gcp.GcpManagedKafkaClientCallbackHandler",
>       "sasl.jaas.config": "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;"
>     }
>   },
>   "metadata": {}
> }
> ```

## Connection requirements

Pinot sends the data plane's Google access token only to Managed Kafka brokers over verified TLS. Before it requests a token, Pinot checks these settings and rejects the connection with an error that names the setting:

| Setting | Requirement |
| - | - |
| `stream.kafka.broker.list` | Every address is a Managed Kafka hostname: `<name>.managedkafka.<project>.cloud.goog`. IP addresses, load balancers and custom DNS names are rejected. |
| `security.protocol` | `SASL_SSL`. |
| `sasl.client.callback.handler.class` | `ai.startree.pinot.kafka.gcp.GcpManagedKafkaClientCallbackHandler`. |
| `ssl.endpoint.identification.algorithm` | `https`, the default. Don't disable hostname verification. |
| `client.dns.lookup` | `use_all_dns_ips`, the default. |
| `ssl.truststore.location`, `ssl.truststore.certificates`, `ssl.engine.factory.class` | Not set. Pinot verifies the brokers against the platform's trusted certificate authorities, so don't add truststore settings. |

## Troubleshooting

| Error | Cause and fix |
| - | - |
| `GCP Managed Kafka authentication ...` | A setting doesn't meet a [connection requirement](#connection-requirements). The message names the setting. |
| `Set sasl.client.callback.handler.class=...` | The client callback handler is missing. Add `sasl.client.callback.handler.class`. |
| `SaslAuthenticationException` | The principal doesn't have `roles/managedkafka.client` on the project that contains the cluster, or the grant hasn't applied yet. If you set `principal` in `sasl.jaas.config`, it must match the data plane's identity. |
| `No resolvable bootstrap urls given in bootstrap.servers` | The data plane can't resolve the bootstrap address. Connect its subnet to the cluster. |

> [card] [Next Step](https://docs.startree.ai/corecapabilities/ingestdata/dataportal/data-modeling/overview.md)
>
> Proceed with [Data Modeling](https://docs.startree.ai/corecapabilities/ingestdata/dataportal/data-modeling/overview.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.