> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect to Amazon S3

> Create a connection to ingest files stored in Amazon S3 buckets.

> [!WARNING]
>
> **Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly.
>
> If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](https://docs.startree.ai/corecapabilities/external-table/best-practices-and-configs.md#networking-and-data-transfer-costs) for details.

**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select S3 as the Data Source.**

**3. Create a New Connection.**

Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**.

Enter a **Source Name** for the new connection.

Select the **Authentication Type** from the drop-down list.

**4. Configure Connection Parameters.**

## Connecting to S3 Using Basic Authentication

Use the following JSON configuration when S3 is set up with **basic authentication** using an **access key** and **secret key**.

```json theme={null}
{
  "inputDirURI": "s3://<BUCKET_NAME>/<PATH>/",
  "input.fs.prop.region": "<REGION>",
  "input.fs.prop.accessKey": "<ACCESS_KEY>",
  "input.fs.prop.secretKey": "<SECRET_KEY>"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `inputDirURI` | Yes | URI of the input directory/files for ingestion. It tells Pinot where the data resides. |
| `input.fs.prop.region` | Yes | Region of the file system (e.g., `us-east-1`). |
| `input.fs.prop.accessKey` | Yes | Access key for authentication to the file system. |
| `input.fs.prop.secretKey` | Yes | Secret key for authentication (paired with access key). |

## Connecting to S3 Using IAM-Based Authentication

Please follow [these steps](https://docs.startree.ai/corecapabilities/ingestdata/recipes/iam-role-s3.md) to create IAM role for S3 setup.

Use the following JSON configuration when S3 is set up with IAM-based authentication by assuming an IAM Role for secure access.

```json theme={null}
{
  "inputDirURI": "s3://<BUCKET_NAME>/<PATH>/",
  "input.fs.prop.roleArn": "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME>",
  "input.fs.prop.region": "<REGION>",
  "input.fs.prop.externalId": "<EXTERNAL_ID>",
  "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `inputDirURI` | Yes | URI of the input directory/files for ingestion. It tells Pinot where the data resides. |
| `input.fs.prop.region` | Yes | Region of the file system (e.g., `us-east-1`). |
| `input.fs.prop.externalId` | Yes | External ID of the AWS Account of your StarTree Cloud. |
| `input.fs.prop.roleArn` | Yes | The Amazon Resource Name (ARN) of an AWS IAM role to assume for accessing AWS S3. Allows Pinot to securely access resources in different AWS accounts.    **Example Use Case**: If Pinot is running in Account A but the S3 bucket is in Account B, you can assume a role in Account B that grants access to the bucket. The role must have permissions like "s3:List\*" and "s3:GetObject" for proper access. |
| `input.fs.className` | Yes | The file system implementation class to access the input directory.    **Examples:**    - `org.apache.pinot.plugin.filesystem.S3PinotFS` (for Amazon S3)    - `org.apache.pinot.plugin.filesystem.LocalPinotFS` (for local file systems)    - `org.apache.pinot.plugin.filesystem.HadoopPinotFS` (for HDFS) |

**5. Test the Connection and Configure Data Ingestion.**

After you have configured the connection properties, test the connection to ensure it is working.

## Configure Data Ingestion

When the connection is successful, use the following JSON to configure additional data settings:

```json theme={null}
{
  "inputFormat": "",
  "includeFileNamePattern": "",
  "excludeFileNamePattern": ""
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `inputFormat` | Yes | The format of the input files. Supported values include **csv, json, avro, parquet**, etc. |
| `includeFileNamePattern` | Yes | The glob pattern to filter which files to include for ingestion. Used when the input directory contains mixed files and only specific files should be ingested. |
| `excludeFileNamePattern` | No | The glob pattern to filter which files to exclude from ingestion. Used when the input directory contains mixed files and only specific files should not be ingested. |

## Configure Record Reader

Configure the record reader to customize how the file format is read during ingestion.

> [accordion] CSV
>
> The `CSVRecordReaderConfig` is used for handling CSV files with the following customizable options:
>
> * *header*: Provide a header when the input file has no headers.
> * *skipHeader*: Provide an alternate header when the input file has a corrupt header.
> * *delimiter*: Use an alternate delimiter when fields are not separated by the default delimiter comma.
> * *skipUnParseableLines*: Skip records that are not parseable instead of failing ingestion.
>
> **Example:** Provide a header when the input file has no headers.
>
> ```bash theme={null}
> {
>   "header": "colA,colB,colC"
> }
> ```
>
> **Example:** Provide an alternate header when the input file has a corrupt header.
>
> ```bash theme={null}
> {
>  "header": "colA,colB,colC",
>  "skipHeader": "true"
> }
> ```
>
> **Example**: Use an alternate delimiter when fields are not separated by the default delimiter comma.
>
> ```bash theme={null}
> {
>  "delimiter": ";"
> }
> ```
>
> OR
>
> ```bash theme={null}
> {
>   "delimiter": "\\t"
> }
> ```
>
> **Example:** Skipping records that are not parseable instead of failing ingestion. This option to be used with caution as it can lead to data loss.
>
> ```bash theme={null}
> {
>   skipUnParseableLines: "true"
> }
> ```
>
> **Example:** Handling CSV files with no header, tab-separated fields, empty lines, and unparsable records.
>
> ```bash theme={null}
> {
>   "header": "colA,colB,colC",
>   "delimiter": "\\t",
>   "ignoreEmptyLines": "true",
>   "skipUnParseableLines": "true"
> }
> ```
>
> For a comprehensive list of available CSV record reader configurations, see the Pinot [CSV documentation](https://docs.pinot.apache.org/basics/data-import/pinot-input-formats#csv).

> [accordion] AVRO
>
> One configuration option `AvroRecordReaderConfig` is supported.
>
> * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS.
>
> ```bash theme={null}
> {
>   "enableLogicalTypes": "true"
> }
> ```
>
> For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start.

> [accordion] Parquet
>
> For Parquet files, Data Manager provides the ParquetRecordReaderConfig with customizable configurations in Data Manager.
>
> Use Parquet Avro Record Reader:
>
> ```bash theme={null}
> {
>     "useParquetAvroRecordReader" : "true"
> }
> ```
>
> When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetAvroRecordReader`
>
> Use Parquet Native Record Reader:
>
> ```bash theme={null}
> {
>     "useParquetNativeRecordReader" : "true"
> }
> ```
>
> When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetNativeRecordReader`

**6. Preview the Data**

Click Show Sample Data to preview the source data before finalizing the configuration.

> [card] [Next Step](https://docs.startree.ai/corecapabilities/ingestdata/dataportal/data-modeling/overview.md)
>
> Proceed with [Data Modeling](https://docs.startree.ai/corecapabilities/ingestdata/dataportal/data-modeling/overview.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.