Unity Catalog: Onboarding via Data Portal

This guide walks through connecting a Unity Catalog to StarTree using the Data Portal UI. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface. Unity Catalog tables are accessed over the Iceberg REST protocol, and this path also covers Delta Lake tables with UniForm enabled registered in the catalog. This works for both Databricks-managed and OSS Unity Catalog. Data files must be Parquet.

Looking for the API-based approach? See Unity Catalog: Onboarding via API. Using AWS Glue instead? See AWS Glue: Onboarding via Data Portal. Using Amazon S3 Tables instead? See Amazon S3 Tables: Onboarding via Data Portal.

---

​

Prerequisites

Before starting, ensure you have:

• StarTree 0.16.0 or later with the external table Beta feature enabled and tiered storage configured for your environment. Contact StarTree support if unsure.
• A running Unity Catalog:
  • Databricks-managed — a workspace with Unity Catalog enabled, plus either a personal access token (PAT) or OAuth2 service-principal credentials (client ID + secret) to authenticate with.
  • OSS Unity Catalog — a reachable server. The OSS server is unauthenticated by default, so no token is needed.
• The Unity Catalog catalog name (used as the warehouse identifier) and the namespace/table you want to onboard.
• For Delta Lake tables: UniForm must be enabled. A Delta table is only exposed over the Iceberg REST endpoint once UniForm’s Iceberg support is turned on. For a new table, set the properties at creation time — Databricks recommends id column mapping for new tables:
  CREATE TABLE <catalog>.<schema>.<table> (...) TBLPROPERTIES (
    'delta.columnMapping.mode' = 'id',
    'delta.enableIcebergCompatV2' = 'true',
    'delta.universalFormat.enabledFormats' = 'iceberg'
  );
  
  For an existing table, use ALTER TABLE — note that name is the only column-mapping mode an existing table can be upgraded to (id cannot be applied after creation):
  ALTER TABLE <catalog>.<schema>.<table> SET TBLPROPERTIES (
    'delta.columnMapping.mode' = 'name',
    'delta.enableIcebergCompatV2' = 'true',
    'delta.universalFormat.enabledFormats' = 'iceberg'
  );
  
  To check whether an existing table already has these set, run SHOW TBLPROPERTIES <catalog>.<schema>.<table>; and look for the three properties above in the output. UniForm requires all writers to the table to run Databricks Runtime 14.3 LTS or later, and enabling it on an existing table via ALTER TABLE requires 15.4 LTS or later — see the Databricks UniForm requirements for the full list.
• Read access to the underlying S3 object storage. Unity Catalog vends short-lived storage credentials automatically for tables it manages, so static keys are usually unnecessary — but the storage region is always required.

---

​

Step 1: Open the External Tables

1. Log in to Data Portal.
2. In the left navigation, go to Tables.
3. Click + Connect External Table.

The wizard opens with a connection configuration screen.

---

​

Step 2: Configure the Unity Catalog Connection

Choose Unity Catalog as the catalog type. Fill in the credentials and connection details: Details Metastore — how Data Portal authenticates with the Unity Catalog Iceberg REST API. For Personal Access Token: For OAuth (Databricks service principal — recommended over a long-lived PAT for automated pipelines, new in 0.16.0): Storage — credentials for reading the underlying Parquet data files. Unity Catalog vends short-lived storage credentials automatically for most tables — whatever you configure here acts as the fallback/override for tables where vending doesn’t apply. Region is always required, even with vended credentials, because Unity’s short-lived credentials don’t carry one. Click Validate Connection. Data Portal calls the catalog’s validate endpoint and confirms credentials and connectivity before proceeding.

---

​

Step 3: Browse and Select a Table

Once the connection is validated:

1. Data Portal lists the available schemas (the field is labeled Schema) in the Unity Catalog.
2. Select a schema to expand its tables.
3. Click the table you want to onboard.

Data Portal reads the Iceberg schema and derives a Pinot schema automatically.

---

​

Step 4: Review the Schema

The inferred Pinot schema is displayed for review. For External Tables this step is mostly read-only — columns can’t be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing is editable inline: the Pinot Field Type column lets you switch a column between Dimension, Metric, and Date-Time where its data type allows (for example, a LONG can be any of the three; a TIMESTAMP can only be Date-Time). Selecting Date-Time opens a dialog to set the value format and granularity. Multi-value columns are flagged in a dedicated column. Click Next when the schema looks correct.

---

​

Step 5: Configure the Table

Set the final table options: The sync schedule is fixed at every 5 minutes at creation (change it afterwards via the table config, or use Pause Sync / Resume Sync), and null handling is enabled automatically on the generated table config. Click Create Table to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required.

---

​

Step 6: Monitor Onboarding

Once the table is created, onboarding starts automatically. The table detail view shows the status in real time:

• Running — the task is actively reading Iceberg snapshots and building Pinot segments.
• Completed — onboarding finished successfully. The last ingested snapshot ID is shown.
• Failed — onboarding encountered an error. The error message and the number of files discovered vs. segments generated are surfaced to help diagnose the issue.

For deeper observability — watcher status, checkpoint values, and per-snapshot file counts — see the Observability page.

---

​

Pausing Onboarding

To pause scheduled onboarding from Data Portal:

1. Open the table in the Tables view.
2. Click Pause Sync. This removes the schedule (cron trigger) from the table’s ExternalTableSyncTask config and stashes its value under schedule.paused, so the watcher stops firing new runs. Any run currently in progress completes normally, and existing segments and the last checkpoint are untouched. Click Resume Sync to restore the schedule from schedule.paused — onboarding resumes from where it left off; it does not restart from scratch.

---

​

Frequently Asked Questions

The Validate step fails — what should I check?

• For Databricks with PAT: confirm the token is valid and hasn’t expired, and that the workspace has Unity Catalog enabled.
• For Databricks with OAuth2: confirm the token URI, client ID, and client secret are correct, and that the service principal has access to the catalog.
• For OSS Unity Catalog: confirm the server is reachable from the cluster and the REST URI includes the correct path suffix (/api/2.1/unity-catalog/iceberg).
• Verify the warehouse value matches the Unity Catalog catalog name exactly.

---

Table load or sync fails even though browsing worked — what should I check?

• Confirm the storage Region is set — it’s required even when relying on Unity’s vended credentials, which don’t carry a region.
• On Databricks, credential vending requires external data access enabled on the metastore and the EXTERNAL USE SCHEMA privilege on the schema for the principal behind your PAT or OAuth2 credentials. Without them, catalog browsing works but table loads fail. See Onboarding via API — Storage authentication.

---

Can I onboard multiple tables from the same catalog? Yes. After creating the first table, start the wizard again and reuse the same connection credentials. Each table is registered as an independent Pinot table with its own onboarding schedule.

---

The table was created but onboarding hasn’t started — what should I check? Data Portal triggers the first onboarding run automatically after table creation. If onboarding hasn’t started, check the table’s detail page for an error status and review the error message. You can also trigger a run manually via the trigger API.