catalogType=iceberg-rest, serviceType=unity), which covers Iceberg tables and Delta tables with UniForm turned on, including ones that use column mapping (see Schema options for column mapping). Data files must be Parquet.
How it works
Onboarding is four calls. Each one feeds the next:All paths are relative to your data-plane base URL. Set
export BASE_URL=https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot — the StarTree Cloud data-plane proxy that fronts the controller.If your controller requires authentication, add an Authorization header to every request — e.g. -H "Authorization: Bearer <token>". The examples below omit it for brevity.Prerequisites
- StarTree 0.17.0 or later with the External Table feature enabled, and tiered storage configured on the cluster. Contact StarTree support if unsure.
- Network access to the controller REST endpoint, plus an
Authorizationtoken if your cluster requires auth. - A Unity Catalog set up as in Prepare Databricks: the table has Iceberg reads (UniForm) turned on if it’s a Delta table, you have a PAT or OAuth service-principal credentials, and the principal has the grants it needs. OSS Unity Catalog needs only a reachable server.
- The Unity Catalog catalog name (used as
warehousebelow) and the namespace/table to onboard. - Read access to the underlying S3 storage. Unity Catalog vends short-lived storage credentials automatically for tables it manages, so static keys are usually unnecessary, but the S3 region is always required. See Storage authentication.
Limitations
auth.storage.regionis required even when relying on vended credentials — Unity’s short-lived credentials don’t carry a region, so the S3 client needs it set explicitly.- Databricks-side UniForm behaviour (metadata lag,
VACUUMretention, deletion vectors) also affects these tables. See Databricks UniForm caveats.
Schema options for column mapping
UniForm tables created with column mapping (delta.columnMapping.mode = name or id) don’t store your column names in the Parquet data files — each physical column is an opaque col-<uuid>; the logical name and a stable field-id live only in table metadata. StarTree always resolves top-level column names back to your logical names, through Iceberg field IDs, across preview, segment generation, null-vector building and renamed time columns. Two ExternalTableSyncTask settings control what happens beyond that: storeNestedSchema.enabled defaults to true (opt out with false); schemaEvolution.enabled defaults to false (opt in).
storeNestedSchema.enabled — resolves field names inside complex JSON columns (struct, array<struct>, map<_,struct>) to your logical names. It’s on by default; set it to false to opt out. When it’s off, nested keys are stored under their physical col-<uuid> names, so JSON path queries on logical names return NULL:
add_files tables where physical names already equal logical names.
schemaEvolution.enabled (plus schemaEvolution.failFast) — before each snapshot’s segments, unions the registered Pinot schema with the source schema by field-id: new source columns are added automatically, and renamed columns keep resolving (the existing Pinot column name is preserved, with the source’s names tracked as aliases). failFast=true aborts the run on an incompatible change (reported as failurePhase: SCHEMA_UPDATE in the status API); false logs it and continues on the previous schema. Leave off to pin the Pinot schema and manage changes manually. See Best Practices & Configs for the resolution rules and limitations.
storeNestedSchema.enabled is on by default, so onboarding needs no setting for it. To opt out, or to set schema evolution, use the ExternalTableSyncTask block of the preview request (Step 2):
Changing
storeNestedSchema.enabled on an already-onboarded table takes effect on the next schema write (re-preview/re-register, or the next sync when schema evolution is on), followed by a segment reload — nested-name resolution happens at read time, so segments don’t need regenerating. A table onboarded before this setting defaulted to on gets that schema update and reload automatically, once, on its first sync after upgrade, if it has complex columns and no explicit setting of its own. To opt out cleanly, set storeNestedSchema.enabled=false before onboarding.Authentication
An External Table authenticates in two places, configured independently:- Catalog (REST) — keys under
catalog.iceberg-rest.auth.rest.*. Choose one of three methods below, depending on your Unity Catalog deployment. Which method to pick, theall-apisscope and the catalog endpoint formats are covered in Prepare Databricks. - Storage (S3 data files) — keys under
catalog.iceberg-rest.auth.storage.*. Unity Catalog vends short-lived S3 credentials per table load, and those take priority over anything set here — see Storage authentication below.
Catalog (REST) authentication: choose one
Databricks: personal access token (PAT)
authType doesn’t need to be set explicitly — it’s auto-detected as token auth when .token is present.
all-apis scope; a narrower token fails on POST /connections/browse.
Databricks: OAuth2 client-credentials (recommended)
For service-principal / machine-to-machine access — the preferred method over a long-lived PAT for automated pipelines.authType is optional; it’s auto-detected as oauth2 once the three required keys below are all present.
The client-credentials grant (RFC 6749) is exchanged for a short-lived token that’s cached and refreshed automatically ahead of expiry — you don’t need to rotate or refresh anything yourself. Note the SQL DDL shortcut for this same setup uses
auth_type = 'oauth', which normalizes to this OAuth2 configuration.
OSS Unity Catalog (unauthenticated)
The OSS server is unauthenticated by default — omitauth.rest.* entirely.
Storage authentication
Vended credentials (default — no keys needed)
Unity Catalog vends short-lived S3 credentials scoped to the table’s location on every load; StarTree uses those automatically. On Databricks, vending needs external data access on the metastore andEXTERNAL USE SCHEMA for your principal; see Set up a service principal and grants. Without them, catalog browsing works but table loads fail. Only region is required on the StarTree side, so the S3 client knows which region to sign requests for:
Override with static access keys
Set these only if you need to bypass the vended credentials — for example, a fixed IAM identity for the tier’s deep-store access, separate from the source read path. If present, these values are ignored whenever Unity successfully vends credentials for the load.Override with an assumed IAM role
Same override caveat as above — same STS assumed-role mechanism used for Glue and S3 Tables.Step 1: Validate and browse the connection
POST /connections/browse
There is no separate “validate” endpoint. Browsing the catalog is the validation step: a 200 with an items list (even an empty one) confirms your credentials and connectivity.
- Set
pathto""to browse the root — this both validates the connection and lists namespaces. - Set
pathto a namespace name to list the tables inside it.
Request
- Databricks-managed
- OSS Unity Catalog
Response
Step 2: Preview the schema
POST /tables/preview
Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4.
The request and response share the same JSON shape. You send a
tableConfig describing the source; the response fills in schema, the enriched tableConfigs.offline, sampled rows, and a summary.Request
Top-level fields:config.inference — how the schema is derived:
Resolution order: explicit
schema → embedded schema → inferred schema.
config.sampling — how rows are sampled:
Setting the namespace and table. Take them from the browse response — set
catalog.iceberg-rest.table.namespace / .tableName to the NAMESPACE and TABLE you selected.auth.storage.region is required even when relying on Unity’s vended storage credentials (see Authentication) — it tells the S3 client which region to sign requests for.Use executor: controller — it’s required for the controller-watcher flow and the observability endpoints. The input tableConfig is intentionally minimal; /tables/preview returns the complete tableConfigs.offline you persist in Step 4.Response
Adjust the schema (time column, column names, null handling) before moving on.
Step 3: Create the schema
POST /schemas
Send the schema object from the preview response. Rename its schemaName to match your table.
Step 4: Create the table
POST /tables
Send the enriched tableConfigs.offline object from the preview response. Its ExternalTableSyncTask block is what marks the table as external.
schedule (cron) interval. There is no separate start call.
Quickstart: onboard a table end-to-end
The whole flow as one script (Databricks-managed Unity Catalog). Fill in the variables at the top, run the script, and the first sync starts automatically.schema.json and tableConfig.json — referenced in steps 3 and 4 above — are extracted here from the preview response with jq.
PAT variable and the auth.rest.token key from both requests, and point REST_URI at your server (e.g. http://<host>:8080/api/2.1/unity-catalog/iceberg).
Once step 6 exits 0, verify it’s queryable.
Monitor onboarding
Three read-only endpoints report ingestion progress — run status, ingestion checkpoint, and source file count — and requireexecutor=controller (set automatically). See Observability for full request and response details.
Verify it’s queryable
WhenfileOnboardingRun.status is COMPLETED and segmentsUploaded matches filesDiscovered, run a query against the broker (or the Data Portal query console) to confirm the data is live:
COMPLETED but the count is 0, give segments a moment to load on the servers, then recheck; if it persists, see Troubleshooting.
What’s next
Now that the table is created and data is loading, these are the highest-impact follow-up steps:- Add indexes for your query patterns. Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → Indexes
-
Enable caching and pin hot indexes. Set
enable.prefetch.page.cache=trueon the S3 tier. An enabled bloom filter is pinned topreload.index.keys.overridefor you automatically; list any other small hot index (dictionary, range index) there yourself so it’s served from local disk on repeated queries instead of re-fetched from S3. → Data and Index Caching - Monitor ongoing syncs. Use the observability endpoints to check run status, ingestion checkpoint, and source file count after each scheduled sync. → Observability
For common questions and failures, see the FAQ and Troubleshooting.

