> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Connect to Azure Data Lake Storage (ADLS)

> Create a connection to ingest files from Azure Data Lake Storage.

<Steps>
  <Step title="In the Data Portal, click Tables and then click Create Table." />

  <Step title="Select ADLS as the Data Source." />

  <Step title="Create a New Connection.">
    Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**.

    Enter a **Source Name** for the new connection.

    Select the **Authentication Type** from the drop-down list.
  </Step>

  <Step title="Configure Connection Parameters">
    ## Connecting to ADLS Using the Access Key

    Use the following JSON configuration when ADLS is set up with **basic authentication** using an access key.

    ```json theme={null}
    {
      "input.fs.className": "org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS",
      "input.fs.prop.accountName": "<STORAGE_ACCOUNT>",
      "authenticationType": "ACCESS_KEY",
      "input.fs.prop.accessKey": "<ACCESS_KEY>",
      "input.fs.prop.fileSystemName": "<CONTAINER_NAME>",
      "inputDirURI": "abfs://<CONTAINER_NAME>@<STORAGE_ACCOUNT>.dfs.core.windows.net/<PATH>/"
    }
    ```

    ### Property Descriptions

    | Property | Required | Description |
    | - | - | - |
    | `input.fs.className` | Yes | The class name of the ADLS Gen2 Pinot file system. |
    | `input.fs.prop.accountName` | Yes | The Azure Storage account name used for ADLS Gen2 integration. |
    | `authenticationType` | Optional | Set as `ACCESS_KEY` to use an access key for authentication. |
    | `input.fs.prop.accessKey` | Yes | The access key used for authentication when `authenticationType` is `ACCESS_KEY`. |
    | `input.fs.prop.fileSystemName` | Yes | The name of the file system (container) in the Azure Storage account. |
    | `inputDirURI` | Yes | The URI of the input directory in ADLS Gen2 where Pinot reads data. |

    ## Connecting to ADLS Using Azure Active Directory

    Use the following JSON configuration when ADLS is set up with **Azure Active Directory authentication**.

    ```json theme={null}
    {
      "input.fs.className": "org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS",
      "input.fs.prop.accountName": "<STORAGE_ACCOUNT>",
      "authenticationType": "AZURE_AD",
      "input.fs.prop.clientId": "<CLIENT_ID>",
      "input.fs.prop.clientSecret": "<CLIENT_SECRET>",
      "input.fs.prop.tenantId": "<TENANT_ID>",
      "input.fs.prop.fileSystemName": "<CONTAINER_NAME>",
      "inputDirURI": "abfs://<CONTAINER_NAME>@<STORAGE_ACCOUNT>.dfs.core.windows.net/<PATH>/"
    }
    ```

    ### Property Descriptions

    | Property | Required | Description |
    | - | - | - |
    | `input.fs.className` | Yes | The class name of the ADLS Gen2 Pinot file system. Must be set to `org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS` for ADLS Gen2 integration. |
    | `input.fs.prop.accountName` | Yes | The name of the Azure Storage account to be used for ADLS Gen2 integration. |
    | `authenticationType` | Optional | The type of authentication used to access the ADLS storage. Set as `AZURE_AD` to use Azure Active Directory (AAD) Service Principal for authentication. |
    | `input.fs.prop.clientId` | Yes | The Client ID of the Azure Service Principal used for authentication. |
    | `input.fs.prop.clientSecret` | Yes | The Client Secret key for the Azure Service Principal. |
    | `input.fs.prop.tenantId` | Yes | The Tenant ID associated with the Azure Active Directory (AAD). |
    | `input.fs.prop.fileSystemName` | Yes | The name of the file system (container) in the Azure Storage account. |
    | `inputDirURI` | Yes | The URI for the input directory in ADLS Gen2 where Pinot reads data. It is generally in the format: `abfs://my-container-name@myazure-storage-account.dfs.core.windows.net/data-directory/`. |

    ## Connecting to ADLS Using Azure Active Directory with Proxy

    Use the following JSON configuration when ADLS is set up with **Azure Active Directory authentication via a proxy server**.

    ```json theme={null}
    {
      "input.fs.className": "org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS",
      "input.fs.prop.accountName": "<STORAGE_ACCOUNT>",
      "authenticationType": "AZURE_AD_WITH_PROXY",
      "input.fs.prop.clientId": "<CLIENT_ID>",
      "input.fs.prop.clientSecret": "<CLIENT_SECRET>",
      "input.fs.prop.tenantId": "<TENANT_ID>",
      "input.fs.prop.proxyHost": "<HOSTNAME>",
      "input.fs.prop.proxyPort": "<PORT>",
      "input.fs.prop.proxyUsername": "<USERNAME>",
      "input.fs.prop.proxyPassword": "<PASSWORD>",
      "input.fs.prop.fileSystemName": "<CONTAINER_NAME>",
      "inputDirURI": "abfs://<CONTAINER_NAME>@<STORAGE_ACCOUNT>.dfs.core.windows.net/<PATH>/"
    }
    ```

    ### Property Descriptions

    | Property | Required | Description |
    | - | - | - |
    | `input.fs.className` | Yes | The class name of the ADLS Gen2 Pinot file system. Must be set to `org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS` for ADLS Gen2 integration. |
    | `input.fs.prop.accountName` | Yes | The name of the Azure Storage account to be used for ADLS Gen2 integration. |
    | `authenticationType` | Optional | The type of authentication used to access the ADLS storage. Set as `AZURE_AD_WITH_PROXY` to use Azure Active Directory (AAD) Service Principal for authentication via a proxy server. |
    | `input.fs.prop.clientId` | Yes | The Client ID of the Azure Service Principal used for authentication. |
    | `input.fs.prop.clientSecret` | Yes | The Client Secret key for the Azure Service Principal. |
    | `input.fs.prop.tenantId` | Yes | The Tenant ID associated with the Azure Active Directory (AAD). |
    | `input.fs.prop.proxyHost` | Yes | The hostname of the proxy server used to connect to ADLS. |
    | `input.fs.prop.proxyPort` | Yes | The port number of the proxy server. |
    | `input.fs.prop.proxyUsername` | Yes | The username for proxy authentication. |
    | `input.fs.prop.proxyPassword` | Yes | The password for proxy authentication. |
    | `input.fs.prop.fileSystemName` | Yes | The name of the file system (container) in the Azure Storage account. |
    | `inputDirURI` | Yes | The URI for the input directory in ADLS Gen2 where Pinot reads data. It is generally in the format: `abfs://my-container-name@myazure-storage-account.dfs.core.windows.net/data-directory/`. |
  </Step>

  <Step title="Test the Connection and Configure Data Ingestion">
    After you have configured the connection properties, test the connection to ensure it is working.

    ## Configure Data Ingestion

    When the connection is successful, use the following JSON to configure additional data settings:

    ```json theme={null}
    {
      "inputFormat": "",
      "includeFileNamePattern": ""
    }
    ```

    ### Property Descriptions

    | Property | Required | Description |
    | - | - | - |
    | `inputFormat` | Yes | The format of the input files. Supported values include **csv, json, avro, parquet**, etc. |
    | `includeFileNamePattern` | Yes | The glob pattern to identify which files to include for ingestion. This is useful when the input directory contains a mix of files, and only specific files should be ingested. |
    | `excludeFileNamePattern` | Optional | The glob pattern to identify which files to exclude from ingestion. This is useful when the input directory contains a mix of files, and only specific files should not be ingested. |

    ## Configure Record Reader

    Configure the record reader to customize how the file format is read during ingestion.

    <AccordionGroup>
      <Accordion title="CSV">
        The `CSVRecordReaderConfig` is used for handling CSV files with the following customizable options:

        * *header*: Provide a header when the input file has no headers.
        * *skipHeader*: Provide an alternate header when the input file has a corrupt header.
        * *delimiter*: Use an alternate delimiter when fields are not separated by the default delimiter comma.
        * *skipUnParseableLines*: Skip records that are not parseable instead of failing ingestion.

        **Example:** Provide a header when the input file has no headers.

        ```bash theme={null}
        {
          "header": "colA,colB,colC"
        }
        ```

        **Example:** Provide an alternate header when the input file has a corrupt header.

        ```bash theme={null}
        {
         "header": "colA,colB,colC",
         "skipHeader": "true"
        }
        ```

        **Example:** Use an alternate delimiter when fields are not separated by the default delimiter comma.

        ```bash theme={null}
        {
         "delimiter": ";"
        }
        ```

        OR

        ```bash theme={null}
        {
          "delimiter": "\\t"
        }
        ```

        **Example:** Skipping records that are not parseable instead of failing ingestion. This option to be used with caution as it can lead to data loss.

        ```bash theme={null}
        {
          skipUnParseableLines: "true"
        }
        ```

        **Example:** Handling CSV files with no header, tab-separated fields, empty lines, and unparsable records.

        ```bash theme={null}
        {
          "header": "colA,colB,colC",
          "delimiter": "\\t",
          "ignoreEmptyLines": "true",
          "skipUnParseableLines": "true"
        }
        ```

        For a comprehensive list of available CSV record reader configurations, see the Pinot [CSV documentation](https://docs.pinot.apache.org/basics/data-import/pinot-input-formats#csv).
      </Accordion>

      <Accordion title="AVRO">
        One configuration option `AvroRecordReaderConfig` is supported.

        * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS.

        ```bash theme={null}
        {
          "enableLogicalTypes": "true"
        }
        ```

        For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start.
      </Accordion>

      <Accordion title="Parquet">
        For Parquet files, Data Manager provides the ParquetRecordReaderConfig with customizable configurations in Data Manager.

        Use Parquet Avro Record Reader:

        ```bash theme={null}
        {
            "useParquetAvroRecordReader" : "true"
        }
        ```

        When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetAvroRecordReader`

        Use Parquet Native Record Reader:

        ```bash theme={null}
        {
            "useParquetNativeRecordReader" : "true"
        }
        ```

        When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetNativeRecordReader`
      </Accordion>
    </AccordionGroup>
  </Step>

  <Step title="Preview the Data">
    Click Show Sample Data to preview the source data before finalizing the configuration.
  </Step>
</Steps>

<Card title="Next Step" icon="forward-step" iconType="solid" href="/corecapabilities/ingestdata/dataportal/data-modeling/overview">
  Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview).
</Card>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.