​
Step 1: In the Data Portal, click Tables and then click Create Table.

​
Step 2: Select Kinesis as the Data Source.

​
Step 3: Create a New Connection.

Click New Connection. If you want to use an existing connection, select the connection from the list and proceed to Step 5.

Enter a Source Name for the new connection and specify the Broker URL.

​
Step 4: Configure Connection Parameters

​
Connect Kinesis Using Basic Authentication

Use the following JSON configuration when Kinesis is set up with basic authentication using an access key and secret key.

{
  "region": "us-west-1",
  "accessKey": "AKIAEXAMPLEACCESSKEY",
  "secretKey": "SECRETKEYEXAMPLE12345"
}

​
Property Descriptions

PropertyRequiredDescription
regionYesThe AWS region (e.g., us-east-1) where the Kinesis stream is located.
accessKeyYesThe AWS access key used for authentication.
secretKeyYesThe AWS secret key used for authentication (paired with the access key).

​
Step 5: Test the Connection and Configure Data Ingestion

After configuring the connection properties, test the connection to ensure it functions properly. Once the connection is successful, configure the additional data settings using the following JSON format:

{
  "stream.kinesis.topic.name": "topic-name",
  "stream.kinesis.decoder.prop.format": "Message-Format-Type",
  "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
  "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory"
}

​
Property Descriptions

PropertyRequiredDescription
stream.kinesis.topic.nameYesThe name of the Kinesis topic from which Pinot will consume data.
stream.kinesis.decoder.prop.formatYesThe format of the Kinesis messages. Supported values include csv, json, avro, protobuf, parquet, etc.
stream.kinesis.decoder.class.nameYesThe class name of the decoder used for Kinesis message parsing. It is set based on the message format and schema.
Examples:
* For JSON Messages: org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder
* For AVRO Messages: org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder
* For PROTO Messages: org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder

See Message Decoders for more information.
stream.kinesis.consumer.factory.class.nameYesThe Kinesis consumer factory class to use. The default class is org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory for AWS Kinesis.

​
Configure Record Reader

​
Step 6: Preview the Data

Click Show Sample Data to preview the source data before finalizing the configuration.