Skip to main content
This feature is available starting in StarTree release 0.17.0. It requires a StarTree data plane on Google Cloud.
StarTree connects to Google Cloud Managed Service for Apache Kafka with SASL/OAUTHBEARER. Pinot authenticates with the Google identity of your StarTree data plane, so the connection needs no service account key, password or other secret. You control access with IAM.

Prerequisites

Grant access to the StarTree principal

Pinot authenticates as the IAM principal of its Kubernetes service account. StarTree Support provides the principal for your data plane. It has this form:
Grant the principal the Managed Kafka Client role (roles/managedkafka.client) on the project that contains the cluster:
IAM changes can take a few minutes to apply. If the topic has Kafka ACLs, also allow the principal to READ the topic. Topics without ACLs are open to every authenticated principal.

Connect the network

Pinot must reach the cluster’s bootstrap address. Connect the subnet of your StarTree data plane to the cluster; StarTree Support provides the subnet. Managed Kafka then creates DNS entries for the cluster in that network.
  • The subnet must be in the cluster’s region. A cluster accepts up to 10 subnets.
  • If the data plane runs in a different project from the cluster, the data plane’s project must grant the Managed Kafka Service Agent role (roles/managedkafka.serviceAgent) to service-<CLUSTER_PROJECT_NUMBER>@gcp-sa-managedkafka.iam.gserviceaccount.com. Coordinate this step with StarTree Support.
For details, see Configure networking for Managed Service for Apache Kafka.

Get the bootstrap address

Use the address exactly as returned, for example bootstrap.<CLUSTER_ID>.<REGION>.managedkafka.<KAFKA_PROJECT_ID>.cloud.goog:9092.

Create the table

1

In the Data Portal, click Tables and then click Create Table.

2

Select Custom as the Data Source.

3

Create a New Connection.

Select Streaming as the Data Source Category, and use this connection JSON:

Property Descriptions

4

Test the Connection and Configure Data Ingestion

Click Test Connection. When the test succeeds, configure ingestion:
These are the standard Kafka ingestion properties. For descriptions, see Configure Data Ingestion in Connect to Apache Kafka.
5

Preview the Data

Click Show Sample Data to preview the source data.
To create the table without the Data Portal, put the connection and ingestion properties in tableIndexConfig.streamConfigs:

Connection requirements

Pinot sends the data plane’s Google access token only to Managed Kafka brokers over verified TLS. Before it requests a token, Pinot checks these settings and rejects the connection with an error that names the setting:

Troubleshooting

Next Step

Proceed with Data Modeling.