Skip to main content
This page onboards a Databricks Unity Catalog table stored in Azure Data Lake Storage (ADLS) Gen2 with the StarTree controller REST APIs. Tables are read over the Iceberg REST protocol (catalogType=iceberg-rest, serviceType=unity), which covers Iceberg tables and Delta tables with UniForm turned on, including ones that use column mapping. Data files must be Parquet. The flow is the same as Unity Catalog + S3. What changes is the storage block: two keys pick ADLS and how Pinot gets access to it.

How it works

Onboarding is four calls. Each one feeds the next:
Pinot reads the table’s metadata from Unity’s Iceberg REST endpoint, then reads the Parquet files straight from ADLS. Once the table exists, the controller’s watcher runs the first sync and re-syncs on schedule.
All paths are relative to your data-plane base URL. Set export BASE_URL=https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot, the StarTree Cloud data-plane proxy that fronts the controller.If your controller requires authentication, add an Authorization header to every request, e.g. -H "Authorization: Bearer <token>". The examples below omit it for brevity.

Prerequisites

  • StarTree Cloud 0.18.0 or later with the External Table feature enabled, and tiered storage configured on the cluster. Contact StarTree support if unsure.
  • Network access to the controller REST endpoint, plus an Authorization token if your cluster requires auth.
  • A Unity Catalog set up as in Prepare Databricks: the table has Iceberg reads (UniForm) turned on if it’s a Delta table, you have OAuth service-principal credentials (or a PAT, with a SAS token only), and the principal has the grants it needs, including EXTERNAL USE SCHEMA.
  • The Unity Catalog catalog name (used as warehouse below) and the schema and table to onboard.

Storage access: choose one

Set sourceStorageProvider to azure, then pick how Pinot reads the files with externalTableAccessPolicy: No region is needed for either.

Unity-issued credentials

Before you use it:
  • Turn on external data access for the metastore and grant the service principal EXTERNAL USE SCHEMA, as in Set up a service principal and grants. Without them, browsing works but Unity returns no SAS token.
  • Give Unity its own access to the storage account: a storage credential and an external location covering the container. On Databricks on AWS, the credential must be an Azure service principal with Storage Blob Data Contributor on the account.
  • The catalog connection must use OAuth. A PAT is rejected in this mode, and workspaces hosted on Google Cloud aren’t supported.

A SAS token you provide

Both sasToken and credentialPrefix are required, and every data file in the table must sit under the prefix. You can pass the token as an srn2:secret#... reference instead of the raw value. The SAS token must:
  • grant read (r) and list (l)
  • have an expiry (se) and not be expired yet
  • have a start time (st), if set, that isn’t in the future
  • include ske if it’s a user-delegation SAS
Locations. abfs://, abfss://, wasb://, wasbs:// and https://<account>.blob.core.windows.net/... or .dfs.core.windows.net/... are all accepted. Only public Azure endpoints work. Sovereign clouds and custom endpoints fail with Azure source requires a public Azure storage endpoint. Rotating the token. Pinot pins the token it first read. If you change the secret behind the same srn2:secret#... reference, reads fail with Azure source SAS credential changed; reload the source configuration before reading. To rotate, store the new token under a new secret reference and update the table config to point at it.

Catalog authentication

Catalog keys go under catalog.iceberg-rest.auth.rest.*, the same as for Unity Catalog + S3. Use OAuth client credentials:
  • <workspace-host> is adb-<workspace-id>.<n>.azuredatabricks.net on Azure Databricks, or <deployment-name>.cloud.databricks.com on Databricks on AWS.
  • The token URI must be on the same host as restUri, at /oidc/v1/token, and the scope must be all-apis.
  • With azure-sas only, a PAT works too: set catalog.iceberg-rest.auth.rest.token instead of the OAuth keys.
You don’t need the X-Iceberg-Access-Delegation header or token-exchange-enabled. Pinot sets what Unity needs.

Step 1: Validate and browse the connection

POST /connections/browse Browsing the catalog is the validation step. A 200 with an items list (even an empty one) confirms the catalog credentials and connectivity. It doesn’t touch storage, so a storage problem shows up at preview.
Set path to "" to list namespaces, or to a namespace name to list its tables. The response format is described in Unity Catalog + S3.

Step 2: Preview the schema

POST /tables/preview Samples the source, infers a Pinot schema, and returns an enriched table config plus sample rows. This is the first call that reads from ADLS, so it also confirms storage access.
For a SAS token, replace the last two keys with the four from A SAS token you provide.
  • table.namespace and table.tableName are the Unity schema and table names. Each is a single name with no dots.
  • Use executor: controller. It’s required for the controller-watcher flow and the observability endpoints.
  • Leave out catalog.iceberg-rest.auth.storage.region and the other S3 storage keys.
The response carries schema (use it in Step 3) and tableConfigs.offline (use it in Step 4), with secrets masked as *****. The full request and response fields, including the column-mapping options, are in Unity Catalog + S3.

Step 3: Create the schema

POST /schemas Send the schema object from the preview response, with schemaName set to match your table.

Step 4: Create the table

POST /tables Send the enriched tableConfigs.offline object from the preview response.
The controller’s External Table watcher discovers the table, runs the first sync, and re-syncs on the schedule (cron) interval. To start the first sync right away:

Quickstart: onboard a table end-to-end

The whole flow as one script, with Unity-issued credentials. Fill in the variables at the top and run it.
When step 5 exits 0, run SELECT count(*) FROM <table> to confirm the data is live. For run status, checkpoints and file counts, see Observability.

How credentials behave

With unity:
  • Pinot renews the SAS shortly before it expires. If a renewal fails, Pinot keeps using the current SAS until it expires and retries every few seconds. It never extends a SAS past its signed expiry.
  • New queries need a valid SAS, even when the data is already cached.
  • Revoking a grant in Unity doesn’t invalidate a SAS that’s already been issued, and queries already running aren’t cancelled. The revocation takes effect when that SAS expires.
With azure-sas, Pinot checks the token against the storage account for each new query. Once the token expires, syncs and queries fail until you rotate it. Cursor pagination isn’t supported on these tables, with either policy.

SQL DDL

SQL DDL can create the same table with a SAS token you provide. It can’t set up Unity-issued credentials: storage.provider = 'azure' always uses a configured SAS token.
storage.region, storage.access_key, storage.secret_key, storage.session_token and storage.endpoint are rejected with storage.provider = 'azure'. Leave out storage.access_policy too. If Pinot can’t list files under the prefix when the table is created, CREATE TABLE fails.

Common errors

Catalog and OAuth errors are listed in Prepare Databricks. These are specific to ADLS:

Not supported

  • Delta tables without Iceberg reads (UniForm) turned on.
  • Azure storage for Pinot’s own segment tier. Pinot’s index files still go to its configured tier storage, while the source Parquet stays in ADLS.

What’s next

  1. Add indexes for your query patterns. Without indexes, every query scans the remote Parquet data. → Indexes
  2. Turn on caching and pin hot indexes. → Data and Index Caching
  3. Watch ongoing syncs with the observability endpoints. → Observability
For common questions and failures, see the FAQ and Troubleshooting.