Unity Catalog: Onboarding via API

This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it.

This page shows how to onboard a Unity Catalog External Table with the StarTree controller REST APIs. There is no Data Portal wizard for Unity Catalog yet — this API flow is the only way to onboard one today. Unity Catalog tables are accessed over the Iceberg REST protocol (catalogType=iceberg-rest, serviceType=unity), the same mechanism used for AWS Glue and Amazon S3 Tables — only the endpoint and auth differ. This works for both OSS Unity Catalog and Databricks-managed Unity Catalog, and for Delta Lake tables with UniForm enabled (UniForm publishes Iceberg-compatible metadata alongside the Delta log, so a Delta+UniForm table registered in Unity Catalog is queryable the same way as a native Iceberg table). Data files must be Parquet.

Known limitation: Delta tables with column mapping. Delta’s columnMapping.mode set to name or id — mandatory for Iceberg Compatibility V2, which covers most Databricks-managed UniForm tables — syncs successfully but currently fails segment load with org.apache.parquet.io.InvalidRecordException: <logical_name> not found in message spark_schema. A fix is planned but not yet available. Before onboarding a Databricks UniForm table, check its delta.columnMapping.mode table property; if it’s name or id, contact StarTree support to confirm current status before proceeding.

Using AWS Glue instead? See AWS Glue: Onboarding via API. Using Amazon S3 Tables instead? See Amazon S3 Tables: Onboarding via API. Using Nessie instead? See Nessie: Onboarding via API.

How it works

Onboarding is four calls. Each one feeds the next:

Validate & browse   POST /connections/browse   →  validate; list namespaces/tables
Preview             POST /tables/preview       →  infer schema + enriched table config
Create schema       POST /schemas              →  register the schema
Create table        POST /tables               →  register the External Table

The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output. Once the table exists, the controller’s watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the observability endpoints.

All paths are relative to your controller base URL. The examples assume export CONTROLLER=https://<your-controller>. On StarTree Cloud, the controller is reached through the data-plane proxy, so use export CONTROLLER=https://<data-plane-host>/api/pinot.If your controller requires authentication, add an Authorization header to every request — e.g. -H "Authorization: Bearer <token>". The examples below omit it for brevity.

Prerequisites

StarTree 0.16.0 or later with the External Table Beta feature enabled, and tiered storage configured on the cluster. Contact StarTree support if unsure.
Network access to the controller REST endpoint, plus an Authorization token if your cluster requires auth.
A running Unity Catalog:
- Databricks-managed — a workspace with Unity Catalog enabled, plus a personal access token (PAT) generated from the workspace (User Settings → Developer → Access tokens).
- OSS Unity Catalog — a reachable server. The OSS server is unauthenticated by default; no token is needed.
The Unity Catalog catalog name (used as warehouse below) and the namespace/table to onboard.
For Delta Lake tables: UniForm must be enabled. A Delta table is only exposed over the Iceberg REST endpoint once UniForm’s Iceberg support is turned on. For a new table, set the properties at creation time — Databricks recommends id column mapping for new tables:
CREATE TABLE <catalog>.<schema>.<table> (...) TBLPROPERTIES ( 'delta.columnMapping.mode' = 'id', 'delta.enableIcebergCompatV2' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg' );
For an existing table, use ALTER TABLE — note that name is the only column-mapping mode an existing table can be upgraded to (id cannot be applied after creation):
ALTER TABLE <catalog>.<schema>.<table> SET TBLPROPERTIES ( 'delta.columnMapping.mode' = 'name', 'delta.enableIcebergCompatV2' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg' );
To check whether an existing table already has these set, run SHOW TBLPROPERTIES <catalog>.<schema>.<table>; and look for the three properties above in the output. UniForm requires all writers to the table to run Databricks Runtime 14.3 LTS or later, and enabling it on an existing table via ALTER TABLE requires 15.4 LTS or later — see the Databricks UniForm requirements for the full list. Note that delta.columnMapping.mode is also the property involved in the known segment-load limitation — check Limitations before onboarding.
Read access to the underlying object storage. Unity Catalog vends short-lived storage credentials automatically for tables it manages, so static keys are usually unnecessary — see Authentication.

Limitations

Delta tables with column mapping fail segment load. delta.columnMapping.mode = name or id — mandatory for Iceberg Compatibility V2, which covers most Databricks-managed UniForm tables — syncs successfully but currently fails with org.apache.parquet.io.InvalidRecordException. Check this table property before onboarding a UniForm table; see the Warning below.
auth.storage.region is required even when relying on vended credentials — Unity’s short-lived credentials don’t carry a region, so the S3 client needs it set explicitly.

Databricks UniForm caveats

These are Databricks-side behaviors of UniForm that affect a UniForm-backed External Table:

New data appears only after Iceberg metadata generation. Databricks converts Delta commits to Iceberg metadata asynchronously, batches frequent commits into fewer Iceberg commits, and skips generation while another run is already in progress — so the Iceberg view (and therefore StarTree’s sync) can lag the Delta table. End-to-end freshness is the conversion delay plus the External Table sync schedule. Run MSCK REPAIR TABLE <table> SYNC METADATA in Databricks to force a refresh if the Iceberg view is stale.
Keep VACUUM retention longer than your sync interval. VACUUM physically deletes Parquet files that older Iceberg snapshots still reference; a sync run reading a snapshot older than the retention window fails when it fetches those files. Set retention to comfortably exceed the sync interval, plus any period the sync might be paused.
Materialized views and streaming tables can’t be onboarded. Databricks doesn’t support Iceberg Compatibility V2 on them, so they’re never exposed over the Iceberg REST endpoint. Tables with VOID columns are also unsupported.
Enabling UniForm is a one-way door for the Delta table. It permanently upgrades the table’s protocol (minReaderVersion ≥ 2, minWriterVersion ≥ 7) and switches new files from Snappy to Zstandard compression. StarTree reads both fine, but older external Delta clients reading the same table may break — check the table’s other consumers before enabling.
Deletes are copy-on-write. Deletion vectors are disallowed with Iceberg Compatibility V2, so updates and deletes rewrite data files and every Iceberg snapshot is self-contained — no Iceberg delete files are produced for StarTree to apply.

Authentication

An External Table authenticates in two places, configured independently:

Catalog (REST) — keys under catalog.iceberg-rest.auth.rest.*. Choose one of three methods below, depending on your Unity Catalog deployment.
Storage (S3 data files) — keys under catalog.iceberg-rest.auth.storage.*. Unity Catalog vends short-lived S3 credentials per table load, and those take priority over anything set here — see Storage authentication below.

Catalog (REST) authentication: choose one

Databricks: personal access token (PAT)

authType doesn’t need to be set explicitly — it’s auto-detected as token auth when .token is present.

{
  "catalog.iceberg-rest.auth.rest.token": "<databricks-pat>"
}

Databricks: OAuth2 client-credentials (recommended, new in 0.16.0)

For service-principal / machine-to-machine access — the preferred method over a long-lived PAT for automated pipelines. authType is optional; it’s auto-detected as oauth2 once the three required keys below are all present.

{
  "catalog.iceberg-rest.auth.rest.authType": "oauth2",
  "catalog.iceberg-rest.auth.rest.oauthTokenUri": "<oauth2-token-endpoint>",
  "catalog.iceberg-rest.auth.rest.oauthClientId": "<service-principal-client-id>",
  "catalog.iceberg-rest.auth.rest.oauthClientSecret": "<service-principal-secret>",
  "catalog.iceberg-rest.auth.rest.oauthScopes": "<space-separated-scopes-if-required>"
}

Key	Required	Description
`oauthTokenUri`	Yes	OAuth2 token endpoint.
`oauthClientId`	Yes	Service-principal client ID.
`oauthClientSecret`	Yes	Service-principal client secret.
`oauthScopes`	No	Space-separated OAuth2 scopes, if your Unity Catalog deployment requires them.

The client-credentials grant (RFC 6749) is exchanged for a short-lived token that’s cached and refreshed automatically ahead of expiry — you don’t need to rotate or refresh anything yourself. Note the SQL DDL shortcut for this same setup uses auth_type = 'oauth', which normalizes to this OAuth2 configuration.

OSS Unity Catalog (unauthenticated)

The OSS server is unauthenticated by default — omit auth.rest.* entirely.

{}

Storage authentication

Vended credentials (default — no keys needed)

Unity Catalog vends short-lived S3 credentials scoped to the table’s location on every load; StarTree uses those automatically. On Databricks, credential vending has two workspace-side prerequisites: external data access must be enabled on the metastore, and the principal behind the PAT or OAuth2 credentials needs the EXTERNAL USE SCHEMA privilege on the schema — see Enable external data access to Unity Catalog. Without them, catalog browsing works but table loads fail. Only region is required on the StarTree side, so the S3 client knows which region to sign requests for:

{
  "catalog.iceberg-rest.auth.storage.region": "us-west-1"
}

Override with static access keys

Set these only if you need to bypass the vended credentials — for example, a fixed IAM identity for the tier’s deep-store access, separate from the source read path. If present, these values are ignored whenever Unity successfully vends credentials for the load.

{
  "catalog.iceberg-rest.auth.storage.accessKeyId": "<key>",
  "catalog.iceberg-rest.auth.storage.secretAccessKey": "<secret>",
  "catalog.iceberg-rest.auth.storage.region": "us-west-1"
}

Override with an assumed IAM role

Same override caveat as above — same STS assumed-role mechanism used for Glue and S3 Tables.

{
  "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader",
  "catalog.iceberg-rest.auth.storage.externalId": "<external-id-if-your-trust-policy-requires-one>",
  "catalog.iceberg-rest.auth.storage.region": "us-west-1"
}

`restUri` formats

Deployment	Example `restUri`
Databricks-managed Unity Catalog	`https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest`
OSS Unity Catalog	`http://<host>:8080/api/2.1/unity-catalog/iceberg`

Both suffixes are exactly what you provide — StarTree doesn’t rewrite the host or path, it only appends the standard Iceberg REST /v1/... routes.

Step 1: Validate and browse the connection

POST /connections/browse There is no separate “validate” endpoint. Browsing the catalog is the validation step: a 200 with an items list (even an empty one) confirms your credentials and connectivity.

Set path to "" to browse the root — this both validates the connection and lists namespaces.
Set path to a namespace name to list the tables inside it.

Request

Field	Type	Required	Description
`connection.type`	string	Yes	Use `CATALOG` for all External Table sources.
`connection.params`	object	Yes	Catalog-specific connection settings (see example below).
`path`	string	No	Where to browse. `""` or omitted = root. A namespace name = that namespace’s tables.

Databricks-managed
OSS Unity Catalog

curl -X POST "$CONTROLLER/connections/browse" \
  -H "Content-Type: application/json" \
  -d '{
    "connection": {
      "type": "CATALOG",
      "params": {
        "catalogType": "iceberg-rest",
        "catalog.iceberg-rest.serviceType": "unity",
        "catalog.iceberg-rest.restUri": "https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest",
        "catalog.iceberg-rest.warehouse": "<unity-catalog-name>",
        "catalog.iceberg-rest.auth.rest.token": "<databricks-pat>"
      }
    },
    "path": ""
  }'

curl -X POST "$CONTROLLER/connections/browse" \
  -H "Content-Type: application/json" \
  -d '{
    "connection": {
      "type": "CATALOG",
      "params": {
        "catalogType": "iceberg-rest",
        "catalog.iceberg-rest.serviceType": "unity",
        "catalog.iceberg-rest.restUri": "http://<host>:8080/api/2.1/unity-catalog/iceberg",
        "catalog.iceberg-rest.warehouse": "<unity-catalog-name>"
      }
    },
    "path": ""
  }'

Response

{
  "items": [
    { "name": "default", "type": "NAMESPACE" },
    { "name": "countries", "type": "TABLE" }
  ]
}

`items[].type`	Meaning
`NAMESPACE`	A namespace you can drill into — pass its `name` as `path`.
`TABLE`	A selectable table.

Step 2: Preview the schema

POST /tables/preview Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4.

The request and response share the same JSON shape. You send a tableConfig describing the source; the response fills in schema, the enriched tableConfigs.offline, sampled rows, and a summary.

Request

Top-level fields:

Field	Type	Required	Description
`tableConfig`	object	Yes	A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source.
`config`	object	No	Sampling and inference controls (see below). Defaults are applied if omitted.
`schema`	object	No	An explicit schema to use instead of inferring one.

config.inference — how the schema is derived:

Field	Type	Default	Description
`embeddedSchemaEnabled`	boolean	`false`	Read the schema embedded in the Iceberg metadata. Set this to `true` for External Tables.
`schemaInferenceEnabled`	boolean	`false`	Infer the schema from sampled rows. Fallback when no embedded schema exists.

Resolution order: explicit schema → embedded schema → inferred schema. config.sampling — how rows are sampled:

Field	Type	Default	Description
`min`	integer	`1`	Minimum records to sample.
`max`	integer	`100`	Maximum records to sample.

To list source files without sampling any data, set config.previewFiles.previewFilesOnly = true. The response returns matching file URIs in sourceFiles and skips schema inference. (OFFLINE only.)

curl -X POST "$CONTROLLER/tables/preview" \
  -H "Content-Type: application/json" \
  -d '{
    "tableConfig": {
      "tableName": "countries_OFFLINE",
      "tableType": "OFFLINE",
      "task": {
        "taskTypeConfigsMap": {
          "ExternalTableSyncTask": {
            "catalogType": "iceberg-rest",
            "executor": "controller",
            "inputFormat": "parquet",
            "catalog.iceberg-rest.serviceType": "unity",
            "catalog.iceberg-rest.restUri": "https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest",
            "catalog.iceberg-rest.warehouse": "<unity-catalog-name>",
            "catalog.iceberg-rest.table.namespace": "default",
            "catalog.iceberg-rest.table.tableName": "countries",
            "catalog.iceberg-rest.auth.rest.token": "<databricks-pat>",
            "catalog.iceberg-rest.auth.storage.region": "us-west-1"
          }
        }
      }
    },
    "config": { "inference": { "embeddedSchemaEnabled": true } }
  }'

Setting the namespace and table. Take them from the browse response — set catalog.iceberg-rest.table.namespace / .tableName to the NAMESPACE and TABLE you selected.auth.storage.region is required even when relying on Unity’s vended storage credentials (see Authentication) — it tells the S3 client which region to sign requests for.Use executor: controller — it’s required for the controller-watcher flow and the observability endpoints. The input tableConfig is intentionally minimal; /tables/preview returns the complete tableConfigs.offline you persist in Step 4.

Response

Field	Type	Description
`schema`	object	The resolved Pinot schema. Use this in step 3.
`tableConfigs.offline`	object	The enriched OFFLINE table config, ready to persist. Use this in step 4. Secrets are masked as `*****`.
`rows`	array	Sample records after schema + ingestion transforms.
`sourceRows`	array	Raw records before transformation.
`summary`	object	Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found).

Adjust the schema (time column, column names, null handling) before moving on.

Step 3: Create the schema

POST /schemas Send the schema object from the preview response. Rename its schemaName to match your table.

curl -X POST "$CONTROLLER/schemas" \
  -H "Content-Type: application/json" \
  -d @schema.json

{ "status": "countries successfully added" }

Step 4: Create the table

POST /tables Send the enriched tableConfigs.offline object from the preview response. Its ExternalTableSyncTask block is what marks the table as external.

curl -X POST "$CONTROLLER/tables" \
  -H "Content-Type: application/json" \
  -d @tableConfig.json

{ "status": "Table countries_OFFLINE successfully added" }

The controller’s External Table watcher then discovers the table, runs the first sync, and re-syncs at the schedule (cron) interval. There is no separate start call.

The first sync runs on the watcher’s next tick. To kick it off immediately instead of waiting, you can manually trigger a run:

curl -X POST "$CONTROLLER/tasks/schedule?taskType=ExternalTableSyncTask&tableName=countries_OFFLINE"

Quickstart: onboard a table end-to-end

The whole flow as one script (Databricks-managed Unity Catalog). Fill in the variables at the top, run the script, and the first sync starts automatically. schema.json and tableConfig.json — referenced in steps 3 and 4 above — are extracted here from the preview response with jq.

#!/usr/bin/env bash
set -euo pipefail

CONTROLLER="https://<your-controller>"
AUTH="Authorization: Bearer <token>"
REST_URI="https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest"
WAREHOUSE="<unity-catalog-name>"
NAMESPACE="default"
TABLE="countries"
PAT="<databricks-pat>"
STORAGE_REGION="us-west-1"

# 1. Validate & browse root — lists Unity Catalog namespaces.
curl -sf -X POST "$CONTROLLER/connections/browse" \
  -H "$AUTH" -H 'Content-Type: application/json' \
  --data-raw "{
    \"connection\": {
      \"type\": \"CATALOG\",
      \"params\": {
        \"catalogType\": \"iceberg-rest\",
        \"catalog.iceberg-rest.serviceType\": \"unity\",
        \"catalog.iceberg-rest.restUri\": \"$REST_URI\",
        \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\",
        \"catalog.iceberg-rest.auth.rest.token\": \"$PAT\"
      }
    },
    \"path\": \"\"
  }" | jq .

# 2. Preview — infer schema + enriched table config.
curl -sf -X POST "$CONTROLLER/tables/preview" \
  -H "$AUTH" -H 'Content-Type: application/json' \
  --data-raw "{
    \"tableConfig\": {
      \"tableName\": \"${TABLE}_OFFLINE\",
      \"tableType\": \"OFFLINE\",
      \"task\": {
        \"taskTypeConfigsMap\": {
          \"ExternalTableSyncTask\": {
            \"catalogType\": \"iceberg-rest\",
            \"executor\": \"controller\",
            \"inputFormat\": \"parquet\",
            \"catalog.iceberg-rest.serviceType\": \"unity\",
            \"catalog.iceberg-rest.restUri\": \"$REST_URI\",
            \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\",
            \"catalog.iceberg-rest.table.namespace\": \"$NAMESPACE\",
            \"catalog.iceberg-rest.table.tableName\": \"$TABLE\",
            \"catalog.iceberg-rest.auth.rest.token\": \"$PAT\",
            \"catalog.iceberg-rest.auth.storage.region\": \"$STORAGE_REGION\"
          }
        }
      }
    },
    \"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } }
  }" > preview.json

# 3. Extract schema and table config from preview response.
jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json
jq '.tableConfigs.offline'                        preview.json > tableConfig.json

# 4. Create the schema.
curl -sf -X POST "$CONTROLLER/schemas" \
  -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json

# 5. Create the table — watcher starts the first sync automatically.
curl -sf -X POST "$CONTROLLER/tables" \
  -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json

# 6. (optional) Poll until sync completes.
curl -sf "$CONTROLLER/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \
  | jq -e '.status == "COMPLETED" and .segmentsUploaded == .filesDiscovered'

For OSS Unity Catalog, drop the PAT variable and the auth.rest.token key from both requests, and point REST_URI at your server (e.g. http://<host>:8080/api/2.1/unity-catalog/iceberg). Once step 6 exits 0, verify it’s queryable.

Monitor onboarding

Three read-only endpoints report ingestion progress — run status, ingestion checkpoint, and source file count — and require executor=controller (set automatically). See Observability for full request and response details.

Verify it’s queryable

When status is COMPLETED and segmentsUploaded matches filesDiscovered, run a query against the broker (or the Data Portal query console) to confirm the data is live:

SELECT count(*) FROM countries;

If status is COMPLETED but the count is 0, give segments a moment to load on the servers, then recheck; if it persists, see Troubleshooting.

What’s next

Now that the table is created and data is loading, these are the highest-impact follow-up steps:

Add indexes for your query patterns. Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → Indexes
Enable caching and preload. Set enable.prefetch.page.cache=true and preload.enable=true on the S3 tier so index data is served from local disk on repeated queries instead of re-fetched from S3. → Data and Index Caching
Protect large-scan queries from OOM. For tables that receive heavy aggregations or wide scans, enable the query OOM killer so a runaway query is killed instead of crashing the server. → Best Practices & Configs — Query OOM protection
Monitor ongoing syncs. Use the observability endpoints to check run status, ingestion checkpoint, and source file count after each scheduled sync. → Observability

For common questions and failures, see the FAQ and Troubleshooting.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Unity Catalog: Onboarding via API

How it works

Prerequisites

Limitations

Databricks UniForm caveats

Authentication

Catalog (REST) authentication: choose one

Databricks: personal access token (PAT)

Databricks: OAuth2 client-credentials (recommended, new in 0.16.0)

OSS Unity Catalog (unauthenticated)

Storage authentication

Vended credentials (default — no keys needed)

Override with static access keys

Override with an assumed IAM role

`restUri` formats

Step 1: Validate and browse the connection

Request

Response

Step 2: Preview the schema

Request

Response

Step 3: Create the schema

Step 4: Create the table

Quickstart: onboard a table end-to-end

Monitor onboarding

Verify it’s queryable

What’s next

​How it works

​Prerequisites

​Limitations

​Databricks UniForm caveats

​Authentication

​Catalog (REST) authentication: choose one

​Databricks: personal access token (PAT)

​Databricks: OAuth2 client-credentials (recommended, new in 0.16.0)

​OSS Unity Catalog (unauthenticated)

​Storage authentication

​Vended credentials (default — no keys needed)

​Override with static access keys

​Override with an assumed IAM role

​restUri formats

​Step 1: Validate and browse the connection

​Request

​Response

​Step 2: Preview the schema

​Request

​Response

​Step 3: Create the schema

​Step 4: Create the table

​Quickstart: onboard a table end-to-end

​Monitor onboarding

​Verify it’s queryable

​What’s next

How it works

Prerequisites

Limitations

Databricks UniForm caveats

Authentication

Catalog (REST) authentication: choose one

Databricks: personal access token (PAT)

Databricks: OAuth2 client-credentials (recommended, new in 0.16.0)

OSS Unity Catalog (unauthenticated)

Storage authentication

Vended credentials (default — no keys needed)

Override with static access keys

Override with an assumed IAM role

`restUri` formats

Step 1: Validate and browse the connection

Request

Response

Step 2: Preview the schema

Request

Response

Step 3: Create the schema

Step 4: Create the table

Quickstart: onboard a table end-to-end

Monitor onboarding

Verify it’s queryable

What’s next