Skip to main content
Use the Data Portal wizard to connect a Databricks Unity Catalog whose tables live in Azure Data Lake Storage (ADLS) Gen2 and onboard a table. It works for Iceberg tables and Delta tables with UniForm turned on, including ones that use column mapping. Data files must be Parquet. The steps match Unity Catalog + S3 except for the Storage section of the connection form.

Prerequisites

  • StarTree Cloud 0.18.0 or later with the External Table feature enabled and tiered storage configured for your environment. On older environments, ADLS isn’t offered in the Storage Provider list. Contact StarTree support if unsure.
  • A Unity Catalog set up as in Prepare Databricks: the table has Iceberg reads (UniForm) turned on if it’s a Delta table, you have OAuth service-principal credentials (or a PAT, if you’ll use a SAS token) with the all-apis scope, and the principal has the grants it needs, including EXTERNAL USE SCHEMA.
  • The Unity Catalog catalog name and the schema and table you want to onboard.
  • Read access to the data, in one of two ways:
    • Unity Catalog Credentials (recommended). Unity issues a short-lived SAS token for each table. The workspace runs on Azure Databricks or Databricks on AWS, the catalog connection uses OAuth, and external data access is on for the metastore. Unity itself needs a storage credential and an external location covering the container.
    • SAS Token. You provide a SAS token with read and list permissions on the container.

Step 1: Open the External Tables

  1. Log in to Data Portal.
  2. In the left navigation, go to Tables.
  3. Click + Connect External Table.
The wizard opens with a connection configuration screen.

Step 2: Configure the Unity Catalog Connection

Choose Unity Catalog as the catalog type. Details Metastore: how Data Portal authenticates with the Unity Catalog Iceberg REST API. For OAuth (Databricks service principal): For Personal Access Token, enter a Databricks token issued with the all-apis scope. A PAT works only with the SAS Token access policy. Unity won’t vend Azure credentials for it. Storage: how Pinot reads the Parquet data files.

Access policy: Unity Catalog Credentials

Unity issues a short-lived, table-scoped SAS token, and Pinot renews it before it expires. There are no storage fields to fill in, and no region.
  • The table must be stored in ADLS Gen2 at an abfss://<container>@<account>.dfs.core.windows.net/... location. Blob-only (.blob.core.windows.net) locations aren’t supported with this policy.
  • Workspaces hosted on Google Cloud aren’t supported with this policy.
  • On Databricks on AWS, Unity’s storage credential for the container must be an Azure service principal with Storage Blob Data Contributor on the storage account.

Access policy: SAS Token

The token must grant read (r) and list (l), carry an expiry (se) that hasn’t passed, and, if it’s a user-delegation SAS, include ske. Only public Azure endpoints work.
Data Portal doesn’t renew a SAS token you provide. Once it expires, syncs and queries on the table fail. Use a token whose expiry covers how long you’ll keep the table, or use Unity Catalog Credentials.
Click Validate Connection. Data Portal browses the catalog to confirm the catalog credentials and connectivity. Storage access is checked later, when you pick a table.

Step 3: Browse and Select a Table

Once the connection is validated:
  1. Data Portal lists the available schemas (the field is labeled Database) in the Unity Catalog.
  2. Select a schema to expand its tables.
  3. Click the table you want to onboard.
Data Portal reads the Iceberg schema, reads a sample of the data from ADLS, and derives a Pinot schema automatically. A storage permission problem shows up here, not at Validate Connection.

Step 4: Review the Schema

The inferred Pinot schema is displayed for review. For External Tables this step is mostly read-only: columns can’t be added, removed, or renamed, and the inferred data types should be kept, because overriding them can break segment generation. The Pinot Field Type column is editable inline, so you can switch a column between Dimension, Metric, and Date-Time where its data type allows. Selecting Date-Time opens a dialog to set the value format and granularity. Multi-value columns are flagged in a dedicated column. Click Next when the schema looks correct.

Step 5: Configure the Table

The sync schedule is fixed at every 5 minutes at creation, and null handling is enabled automatically on the generated table config. Click Create Table. Data Portal registers the schema and table with Pinot and starts the first onboarding run right away.

Step 6: Monitor Onboarding

The table detail view shows the onboarding status in real time:
  • Running: the task is reading Iceberg snapshots and building Pinot segments.
  • Completed: onboarding finished. The last ingested snapshot ID is shown.
  • Failed: onboarding hit an error. The error message and the number of files discovered vs. segments generated help diagnose it.
For watcher status, checkpoint values, and per-snapshot file counts, see Observability. To pause and resume syncs, use Pause Sync and Resume Sync, as described in Unity Catalog + S3.

Frequently Asked Questions

ADLS isn’t in the Storage Provider list. Why? The option appears only on StarTree Cloud 0.18.0 or later. Contact StarTree support to check your environment’s version.
The table preview or sync fails even though browsing worked. What should I check?
  • With Unity Catalog Credentials: confirm external data access is on for the metastore, and the service principal has EXTERNAL USE SCHEMA and SELECT. Without them, browsing works but Unity returns no SAS token. Also check that the table sits on an abfss://…dfs.core.windows.net location and that the connection uses OAuth, not a PAT. See Set up a service principal and grants.
  • With SAS Token: confirm the token grants read and list, hasn’t expired, and that every data file sits under the Credential Prefix.
The exact error messages are listed in Common errors.
Can I onboard multiple tables from the same catalog? Yes. Start the wizard again and reuse the same connection. Each table is an independent Pinot table with its own onboarding schedule. With a SAS token, every table’s files must sit under the connection’s Credential Prefix.
Can I query these tables with cursor pagination? No. Cursor pagination isn’t supported on tables that read ADLS, with either access policy.