> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Unity Catalog + ADLS: Onboarding via API

> Onboard a Unity Catalog table stored in Azure Data Lake Storage Gen2 as an External Table with the REST API, using Unity-issued credentials or a SAS token.

This page onboards a Databricks **Unity Catalog** table stored in **Azure Data Lake Storage (ADLS) Gen2** with the StarTree controller REST APIs. Tables are read over the **Iceberg REST protocol** (`catalogType=iceberg-rest`, `serviceType=unity`), which covers Iceberg tables and Delta tables with [UniForm](https://docs.delta.io/latest/delta-uniform.html) turned on, including ones that use column mapping. Data files must be **Parquet**.

The flow is the same as [Unity Catalog + S3](https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api.md). What changes is the storage block: two keys pick ADLS and how Pinot gets access to it.

## How it works

Onboarding is four calls. Each one feeds the next:

```text theme={null}
1. Validate & browse   POST /connections/browse   →  validate; list namespaces/tables
2. Preview             POST /tables/preview       →  infer schema + enriched table config
3. Create schema       POST /schemas              →  register the schema
4. Create table        POST /tables               →  register the External Table
```

Pinot reads the table's metadata from Unity's Iceberg REST endpoint, then reads the Parquet files straight from ADLS. Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule.

> [!NOTE]
>
> All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot`, the StarTree Cloud data-plane proxy that fronts the controller.
>
> If your controller requires authentication, add an `Authorization` header to every request, e.g. `-H "Authorization: Bearer <token>"`. The examples below omit it for brevity.

## Prerequisites

* **StarTree Cloud 0.18.0 or later** with the External Table feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure.
* Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth.
* A Unity Catalog set up as in [Prepare Databricks](https://docs.startree.ai/corecapabilities/external-table/unity/prepare-databricks.md): the table has Iceberg reads (UniForm) turned on if it's a Delta table, you have OAuth service-principal credentials (or a PAT, with a SAS token only), and the principal has the grants it needs, including `EXTERNAL USE SCHEMA`.
* The Unity Catalog **catalog name** (used as `warehouse` below) and the schema and table to onboard.

## Storage access: choose one

Set `sourceStorageProvider` to `azure`, then pick how Pinot reads the files with `externalTableAccessPolicy`:

| `externalTableAccessPolicy` | How Pinot gets storage credentials | Works with |
| - | - | - |
| `unity` (recommended) | Unity issues a short-lived SAS token scoped to each table, and Pinot renews it before it expires. You manage no storage secrets. | ADLS Gen2 at `abfss://…dfs.core.windows.net` locations. Azure Databricks or Databricks on AWS workspaces. OAuth catalog auth only. |
| `azure-sas` | A SAS token you provide, with the location it's scoped to. Pinot doesn't renew it. | Azure Blob Storage and ADLS Gen2. Any catalog auth method. |

No region is needed for either.

### Unity-issued credentials

```json theme={null}
{
  "sourceStorageProvider": "azure",
  "externalTableAccessPolicy": "unity"
}
```

Before you use it:

* Turn on **external data access** for the metastore and grant the service principal `EXTERNAL USE SCHEMA`, as in [Set up a service principal and grants](https://docs.startree.ai/corecapabilities/external-table/unity/prepare-databricks.md#set-up-a-service-principal-and-grants). Without them, browsing works but Unity returns no SAS token.
* Give Unity its own access to the storage account: a storage credential and an external location covering the container. On Databricks on AWS, the credential must be an Azure service principal with **Storage Blob Data Contributor** on the account.
* The catalog connection must use OAuth. A PAT is rejected in this mode, and workspaces hosted on Google Cloud aren't supported.

### A SAS token you provide

```json theme={null}
{
  "sourceStorageProvider": "azure",
  "externalTableAccessPolicy": "azure-sas",
  "catalog.iceberg-rest.auth.storage.sasToken": "<sas-token>",
  "catalog.iceberg-rest.auth.storage.credentialPrefix": "abfss://<container>@<account>.dfs.core.windows.net/<path>"
}
```

Both `sasToken` and `credentialPrefix` are required, and every data file in the table must sit under the prefix. You can pass the token as an `srn2:secret#...` reference instead of the raw value.

**The SAS token must:**

* grant read (`r`) and list (`l`)
* have an expiry (`se`) and not be expired yet
* have a start time (`st`), if set, that isn't in the future
* include `ske` if it's a user-delegation SAS

**Locations.** `abfs://`, `abfss://`, `wasb://`, `wasbs://` and `https://<account>.blob.core.windows.net/...` or `.dfs.core.windows.net/...` are all accepted. Only public Azure endpoints work. Sovereign clouds and custom endpoints fail with `Azure source requires a public Azure storage endpoint`.

**Rotating the token.** Pinot pins the token it first read. If you change the secret behind the same `srn2:secret#...` reference, reads fail with `Azure source SAS credential changed; reload the source configuration before reading`. To rotate, store the new token under a new secret reference and update the table config to point at it.

## Catalog authentication

Catalog keys go under `catalog.iceberg-rest.auth.rest.*`, the same as for [Unity Catalog + S3](https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api.md#catalog-rest-authentication-choose-one). Use OAuth client credentials:

```json theme={null}
{
  "catalog.iceberg-rest.auth.rest.authType": "oauth2",
  "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://<workspace-host>/oidc/v1/token",
  "catalog.iceberg-rest.auth.rest.oauthClientId": "<service-principal-client-id>",
  "catalog.iceberg-rest.auth.rest.oauthClientSecret": "<service-principal-secret>",
  "catalog.iceberg-rest.auth.rest.oauthScopes": "all-apis"
}
```

* `<workspace-host>` is `adb-<workspace-id>.<n>.azuredatabricks.net` on Azure Databricks, or `<deployment-name>.cloud.databricks.com` on Databricks on AWS.
* The token URI must be on the same host as `restUri`, at `/oidc/v1/token`, and the scope must be `all-apis`.
* With `azure-sas` only, a PAT works too: set `catalog.iceberg-rest.auth.rest.token` instead of the OAuth keys.

You don't need the `X-Iceberg-Access-Delegation` header or `token-exchange-enabled`. Pinot sets what Unity needs.

***

## Step 1: Validate and browse the connection

**`POST /connections/browse`**

Browsing the catalog is the validation step. A `200` with an `items` list (even an empty one) confirms the catalog credentials and connectivity. It doesn't touch storage, so a storage problem shows up at preview.

```bash theme={null}
curl -X POST "$BASE_URL/connections/browse" \
  -H "Content-Type: application/json" \
  -d '{
    "connection": {
      "type": "CATALOG",
      "params": {
        "catalogType": "iceberg-rest",
        "catalog.iceberg-rest.serviceType": "unity",
        "catalog.iceberg-rest.restUri": "https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest",
        "catalog.iceberg-rest.warehouse": "<unity-catalog-name>",
        "catalog.iceberg-rest.auth.rest.authType": "oauth2",
        "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://<workspace-host>/oidc/v1/token",
        "catalog.iceberg-rest.auth.rest.oauthClientId": "<service-principal-client-id>",
        "catalog.iceberg-rest.auth.rest.oauthClientSecret": "<service-principal-secret>",
        "catalog.iceberg-rest.auth.rest.oauthScopes": "all-apis"
      }
    },
    "path": ""
  }'
```

Set `path` to `""` to list namespaces, or to a namespace name to list its tables. The response format is described in [Unity Catalog + S3](https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api.md#response).

***

## Step 2: Preview the schema

**`POST /tables/preview`**

Samples the source, infers a Pinot schema, and returns an enriched table config plus sample rows. This is the first call that reads from ADLS, so it also confirms storage access.

```bash theme={null}
curl -X POST "$BASE_URL/tables/preview" \
  -H "Content-Type: application/json" \
  -d '{
    "tableConfig": {
      "tableName": "events_OFFLINE",
      "tableType": "OFFLINE",
      "task": {
        "taskTypeConfigsMap": {
          "ExternalTableSyncTask": {
            "catalogType": "iceberg-rest",
            "executor": "controller",
            "inputFormat": "parquet",
            "catalog.iceberg-rest.serviceType": "unity",
            "catalog.iceberg-rest.restUri": "https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest",
            "catalog.iceberg-rest.warehouse": "<unity-catalog-name>",
            "catalog.iceberg-rest.table.namespace": "<schema>",
            "catalog.iceberg-rest.table.tableName": "<table>",
            "catalog.iceberg-rest.auth.rest.authType": "oauth2",
            "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://<workspace-host>/oidc/v1/token",
            "catalog.iceberg-rest.auth.rest.oauthClientId": "<service-principal-client-id>",
            "catalog.iceberg-rest.auth.rest.oauthClientSecret": "<service-principal-secret>",
            "catalog.iceberg-rest.auth.rest.oauthScopes": "all-apis",
            "sourceStorageProvider": "azure",
            "externalTableAccessPolicy": "unity"
          }
        }
      }
    },
    "config": { "inference": { "embeddedSchemaEnabled": true } }
  }'
```

For a SAS token, replace the last two keys with the four from [A SAS token you provide](#a-sas-token-you-provide).

* `table.namespace` and `table.tableName` are the Unity schema and table names. Each is a single name with no dots.
* Use `executor: controller`. It's required for the controller-watcher flow and the [observability endpoints](https://docs.startree.ai/corecapabilities/external-table/observability.md).
* Leave out `catalog.iceberg-rest.auth.storage.region` and the other S3 storage keys.

The response carries `schema` (use it in Step 3) and `tableConfigs.offline` (use it in Step 4), with secrets masked as `*****`. The full request and response fields, including the column-mapping options, are in [Unity Catalog + S3](https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api.md#step-2-preview-the-schema).

***

## Step 3: Create the schema

**`POST /schemas`**

Send the `schema` object from the preview response, with `schemaName` set to match your table.

```bash theme={null}
curl -X POST "$BASE_URL/schemas" \
  -H "Content-Type: application/json" \
  -d @schema.json
```

***

## Step 4: Create the table

**`POST /tables`**

Send the enriched `tableConfigs.offline` object from the preview response.

```bash theme={null}
curl -X POST "$BASE_URL/tables" \
  -H "Content-Type: application/json" \
  -d @tableConfig.json
```

The controller's External Table watcher discovers the table, runs the first sync, and re-syncs on the `schedule` (cron) interval. To start the first sync right away:

```bash theme={null}
curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=events_OFFLINE"
```

***

## Quickstart: onboard a table end-to-end

The whole flow as one script, with Unity-issued credentials. Fill in the variables at the top and run it.

```bash theme={null}
#!/usr/bin/env bash
set -euo pipefail

BASE_URL="https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot"
AUTH="Authorization: Bearer <token>"
WORKSPACE_HOST="adb-<workspace-id>.<n>.azuredatabricks.net"
WAREHOUSE="<unity-catalog-name>"
NAMESPACE="<schema>"
TABLE="<table>"
CLIENT_ID="<service-principal-client-id>"
CLIENT_SECRET="<service-principal-secret>"

SOURCE=$(jq -n \
  --arg uri "https://$WORKSPACE_HOST/api/2.1/unity-catalog/iceberg-rest" \
  --arg tok "https://$WORKSPACE_HOST/oidc/v1/token" \
  --arg wh "$WAREHOUSE" --arg id "$CLIENT_ID" --arg secret "$CLIENT_SECRET" '{
    "catalogType": "iceberg-rest",
    "catalog.iceberg-rest.serviceType": "unity",
    "catalog.iceberg-rest.restUri": $uri,
    "catalog.iceberg-rest.warehouse": $wh,
    "catalog.iceberg-rest.auth.rest.authType": "oauth2",
    "catalog.iceberg-rest.auth.rest.oauthTokenUri": $tok,
    "catalog.iceberg-rest.auth.rest.oauthClientId": $id,
    "catalog.iceberg-rest.auth.rest.oauthClientSecret": $secret,
    "catalog.iceberg-rest.auth.rest.oauthScopes": "all-apis"
  }')

# 1. Validate & browse root.
jq -n --argjson p "$SOURCE" '{connection: {type: "CATALOG", params: $p}, path: ""}' \
  | curl -sf -X POST "$BASE_URL/connections/browse" -H "$AUTH" -H 'Content-Type: application/json' -d @- | jq .

# 2. Preview: infer schema + enriched table config.
jq -n --argjson p "$SOURCE" --arg ns "$NAMESPACE" --arg t "$TABLE" '{
  tableConfig: {
    tableName: ($t + "_OFFLINE"), tableType: "OFFLINE",
    task: {taskTypeConfigsMap: {ExternalTableSyncTask: ($p + {
      "executor": "controller",
      "inputFormat": "parquet",
      "catalog.iceberg-rest.table.namespace": $ns,
      "catalog.iceberg-rest.table.tableName": $t,
      "sourceStorageProvider": "azure",
      "externalTableAccessPolicy": "unity"
    })}}
  },
  config: {inference: {embeddedSchemaEnabled: true}}
}' | curl -sf -X POST "$BASE_URL/tables/preview" -H "$AUTH" -H 'Content-Type: application/json' -d @- > preview.json

# 3. Extract schema and table config.
jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json
jq '.tableConfigs.offline'                        preview.json > tableConfig.json

# 4. Create the schema, then the table. The watcher starts the first sync.
curl -sf -X POST "$BASE_URL/schemas" -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json
curl -sf -X POST "$BASE_URL/tables"  -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json

# 5. (optional) Check whether the first sync has finished.
curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \
  | jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered'
```

When step 5 exits `0`, run `SELECT count(*) FROM <table>` to confirm the data is live. For run status, checkpoints and file counts, see [Observability](https://docs.startree.ai/corecapabilities/external-table/observability.md).

***

## How credentials behave

With `unity`:

* Pinot renews the SAS shortly before it expires. If a renewal fails, Pinot keeps using the current SAS until it expires and retries every few seconds. It never extends a SAS past its signed expiry.
* New queries need a valid SAS, even when the data is already cached.
* Revoking a grant in Unity doesn't invalidate a SAS that's already been issued, and queries already running aren't cancelled. The revocation takes effect when that SAS expires.

With `azure-sas`, Pinot checks the token against the storage account for each new query. Once the token expires, syncs and queries fail until you rotate it.

[Cursor pagination](https://docs.startree.ai/corecapabilities/query_data/advanced_operations/cursor-pagination.md) isn't supported on these tables, with either policy.

## SQL DDL

[SQL DDL](https://docs.startree.ai/corecapabilities/external-table/sql-ddl.md#azure-source-storage) can create the same table with a SAS token you provide. It can't set up Unity-issued credentials: `storage.provider = 'azure'` always uses a configured SAS token.

```sql theme={null}
CREATE TABLE unity_azure_events WITH (
  type = 'iceberg',
  catalog_type = 'unity',
  catalog_uri = 'https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest',
  catalog_name = '<unity-catalog-name>',
  schema_name = '<schema>',
  table_name = '<table>',
  auth_type = 'oauth',
  client_id = 'srn2:secret#<client-id-reference>',
  client_secret = 'srn2:secret#<client-secret-reference>',
  oauth_token_uri = 'https://<workspace-host>/oidc/v1/token',
  oauth_scope = 'all-apis',
  storage.provider = 'azure',
  storage.sas_token = 'srn2:secret#<sas-reference>',
  storage.credential_prefix = 'abfss://<container>@<account>.dfs.core.windows.net/<path>',
  refresh_interval = '5m'
);
```

`storage.region`, `storage.access_key`, `storage.secret_key`, `storage.session_token` and `storage.endpoint` are rejected with `storage.provider = 'azure'`. Leave out `storage.access_policy` too. If Pinot can't list files under the prefix when the table is created, `CREATE TABLE` fails.

## Common errors

Catalog and OAuth errors are listed in [Prepare Databricks](https://docs.startree.ai/corecapabilities/external-table/unity/prepare-databricks.md#common-errors). These are specific to ADLS:

| Error | Cause |
| - | - |
| `Invalid Azure storage location` | The location isn't a valid Azure storage URI. With `unity`, it must be an `abfss://<container>@<account>.dfs.core.windows.net/...` location |
| `Unable to obtain scoped Azure storage credentials from the catalog` | Unity didn't return a SAS that covers the file. Check the grants and external data access, and that every data file sits under the table's location |
| `Azure SAS storage requires a SAS token and credential prefix` | `azure-sas` is set but `sasToken` or `credentialPrefix` is missing or empty |
| `Azure SAS has expired` | The SAS token you provided is past its `se` expiry. Rotate it |
| `Azure source requires a public Azure storage endpoint` | The location points at a sovereign cloud or a custom endpoint |
| `Azure source SAS credential changed; reload the source configuration before reading` | The secret behind the token's `srn2:secret#...` reference changed. Store the new token under a new reference and update the table |

## Not supported

* Delta tables without Iceberg reads (UniForm) turned on.
* Azure storage for Pinot's own segment tier. Pinot's index files still go to its configured tier storage, while the source Parquet stays in ADLS.

***

## What's next

1. **Add indexes for your query patterns.** Without indexes, every query scans the remote Parquet data. → [Indexes](https://docs.startree.ai/corecapabilities/external-table/indexes.md)
2. **Turn on caching and pin hot indexes.** → [Data and Index Caching](https://docs.startree.ai/corecapabilities/external-table/data-and-index-caching.md)
3. **Watch ongoing syncs** with the observability endpoints. → [Observability](https://docs.startree.ai/corecapabilities/external-table/observability.md)

For common questions and failures, see the [FAQ](https://docs.startree.ai/corecapabilities/external-table/faq.md) and [Troubleshooting](https://docs.startree.ai/corecapabilities/external-table/troubleshooting.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.