​
Step 1: In the Data Portal, click Tables and then click Create Table.

​
Step 2: Select S3 as the Data Source.

​
Step 3: Create a New Connection.

Click New Connection. If you want to use an existing connection, select the connection from the list and proceed to Step 5.

Enter a Source Name for the new connection.

Select the Authentication Type from the drop-down list.

​
Step 4: Configure Connection Parameters.

​
Connecting to S3 Using Basic Authentication

Use the following JSON configuration when S3 is set up with basic authentication using an access key and secret key.

{
  "inputDirURI": "s3://my-bucket/my-data-directory/",
  "input.fs.prop.region": "us-east-1",
  "input.fs.prop.accessKey": "AKIAEXAMPLEACCESSKEY",
  "input.fs.prop.secretKey": "SECRETKEYEXAMPLE12345"
}

​
Property Descriptions

PropertyRequiredDescription
inputDirURIYesURI of the input directory/files for ingestion. It tells Pinot where the data resides.
input.fs.prop.regionYesRegion of the file system (e.g., us-east-1).
input.fs.prop.accessKeyYesAccess key for authentication to the file system.
input.fs.prop.secretKeyYesSecret key for authentication (paired with access key).

​
Connecting to S3 Using IAM-Based Authentication

Use the following JSON configuration when S3 is set up with IAM-based authentication by assuming an IAM Role for secure access.

{
  "inputDirURI": "s3://my-bucket/my-data-directory/",
  "aws.iamRoleArnToAssume": "arn:aws:iam::123456789012:role/PinotS3AccessRole",
  "input.fs.prop.region": "us-west-2",
  "input.fs.prop.externalId": "env-12abc234xyz",
  "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS"
}

​
Property Descriptions

PropertyRequiredDescription
inputDirURIYesURI of the input directory/files for ingestion. It tells Pinot where the data resides.
input.fs.prop.regionYesRegion of the file system (e.g., us-east-1).
input.fs.prop.externalIdYesExternal ID of the AWS Account of your StarTree Cloud.
input.fs.prop.roleArnYesThe Amazon Resource Name (ARN) of an AWS IAM role to assume for accessing AWS S3. Allows Pinot to securely access resources in different AWS accounts. Example Use Case: If Pinot is running in Account A but the S3 bucket is in Account B, you can assume a role in Account B that grants access to the bucket. The role must have permissions like “s3:List*” and “s3:GetObject” for proper access.
input.fs.classNameYesThe file system implementation class to access the input directory. Examples: - org.apache.pinot.plugin.filesystem.S3PinotFS (for Amazon S3) - org.apache.pinot.plugin.filesystem.LocalPinotFS (for local file systems) - org.apache.pinot.plugin.filesystem.HadoopPinotFS (for HDFS)

​
Step 5: Test the Connection and Configure Data Ingestion.

After you have configured the connection properties, test the connection to ensure it is working.

When the connection is successful, use the following JSON to configure additional data settings:

{
  "inputFormat": "",
  "includeFileNamePattern": "",
  "excludeFileNamePattern": ""
}

​
Property Descriptions

PropertyRequiredDescription
inputFormatYesThe format of the input files. Supported values include csv, json, avro, parquet, etc.
includeFileNamePatternYesThe glob pattern to filter which files to include for ingestion. Used when the input directory contains mixed files and only specific files should be ingested.
excludeFileNamePatternNoThe glob pattern to filter which files to exclude from ingestion. Used when the input directory contains mixed files and only specific files should not be ingested.

​
Configure Record Reader

Configure the record reader to customize how the file format is read during ingestion.

​
Step 6: Preview the Data

Click Show Sample Data to preview the source data before finalizing the configuration.