> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Unity Catalog: Prepare Databricks

> Get a Databricks or OSS Unity Catalog ready for StarTree External Tables: table requirements, catalog authentication, grants and external data access.

Do these steps once in Databricks before you onboard a Unity Catalog table, whichever storage your data lives in.

StarTree reads Unity Catalog over the **Iceberg REST protocol**. That covers native Iceberg tables and **Delta Lake tables with [UniForm](https://docs.delta.io/latest/delta-uniform.html) turned on**. Data files must be **Parquet**.

## Table requirements

The table must be an Iceberg table, or a Delta table with UniForm's Iceberg support turned on. A Delta table isn't exposed over the Iceberg REST endpoint until UniForm is on.

For a **new** Delta table, set the properties when you create it. Databricks recommends `id` column mapping for new tables:

```sql theme={null}
CREATE TABLE <catalog>.<schema>.<table> (...) TBLPROPERTIES (
  'delta.columnMapping.mode' = 'id',
  'delta.enableIcebergCompatV3' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);
```

For an **existing** table, use `ALTER TABLE`. `name` is the only column-mapping mode an existing table can be upgraded to; `id` can't be applied after creation:

```sql theme={null}
ALTER TABLE <catalog>.<schema>.<table> SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV3' = 'true',
  'delta.enableIcebergCompatV2' = 'false',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);
```

To check an existing table, run `SHOW TBLPROPERTIES <catalog>.<schema>.<table>;` and look for the properties above.

Iceberg Compatibility V3 needs **Databricks Runtime 18 LTS or later** and applies to Unity Catalog **managed** tables. If the table already has V2 on, set `'delta.enableIcebergCompatV2' = 'false'` in the same `ALTER TABLE`, as shown. See [Use Apache Iceberg v3 features](https://docs.databricks.com/aws/en/iceberg/iceberg-v3).

> [!NOTE]
>
> **On an older runtime?** Use Iceberg Compatibility V2 instead: set `'delta.enableIcebergCompatV2' = 'true'` in place of the V3 property in either recipe. V2 needs all writers on **Databricks Runtime 14.3 LTS or later**, and turning it on for an existing table with `ALTER TABLE` needs **15.4 LTS or later**. See the [Databricks UniForm requirements](https://docs.databricks.com/aws/en/delta/uniform#requirements). StarTree onboards either level. The difference that matters downstream is how deletes are stored; see [Databricks UniForm caveats](#databricks-uniform-caveats).

**Column mapping.** Delta tables with `delta.columnMapping.mode` set to `name` or `id` are supported. That's mandatory for UniForm's Iceberg reads, so it covers most Databricks-managed UniForm tables. Field names inside complex columns (`struct`, `array<struct>`, `map<_,struct>`) are resolved by the `storeNestedSchema.enabled` task setting, which is on by default. See [Schema options for column mapping](https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api.md#schema-options-for-column-mapping).

### Databricks UniForm caveats

These are Databricks-side behaviours of [UniForm](https://docs.databricks.com/aws/en/delta/uniform) that affect a UniForm-backed External Table:

* **New data appears only after Iceberg metadata generation.** Databricks converts Delta commits to Iceberg metadata **asynchronously**, batches frequent commits into fewer Iceberg commits, and skips generation while another run is in progress. So the Iceberg view, and therefore StarTree's sync, can lag the Delta table. End-to-end freshness is the conversion delay plus the sync schedule. Run `MSCK REPAIR TABLE <table> SYNC METADATA` in Databricks to force a refresh if the Iceberg view is stale.
* **Keep `VACUUM` retention longer than your sync interval.** `VACUUM` deletes Parquet files that older Iceberg snapshots still reference, and a sync reading a snapshot older than the retention window fails when it fetches those files. Set retention well above the sync interval, plus any time the sync might be paused.
* **Materialized views and streaming tables need the pipeline-managed path.** Iceberg Compatibility V2 doesn't support them at all. With V3, pipeline-managed materialized views and streaming tables can be exposed over the Iceberg REST endpoint on **Databricks Runtime 17.3 or later**, in default publishing mode, and only with `'delta.enableRowTracking' = 'true'` and `'pipelines.externalMetadata.enabled' = 'true'` set alongside the UniForm properties. See [Enable external data access to streaming tables and materialized views](https://docs.databricks.com/aws/en/external-access/external-for-pipelines). Tables with `VOID` columns stay unsupported.
* **Turning on UniForm can't be undone for the Delta table.** It permanently upgrades the table protocol (`minReaderVersion` ≥ 2, `minWriterVersion` ≥ 7) and switches new files from Snappy to Zstandard compression. StarTree reads both, but older external Delta clients reading the same table may break, so check the table's other consumers first.
* **Deletes can produce Iceberg deletion vectors under Compatibility V3.** V3 permits deletion vectors and Databricks turns them on by default for new Iceberg v3 tables, so updates and deletes may be written as Puffin delete files instead of rewritten data files. StarTree applies Iceberg v3 Puffin **position** deletes at query time, but that's a **preview** feature you have to turn on: see [Deletion Vectors](https://docs.startree.ai/corecapabilities/external-table/deletion-vectors.md) and contact StarTree to enable it. Equality deletes aren't applied. A table left at V2 disallows deletion vectors, so deletes stay copy-on-write and every snapshot is self-contained.

## Catalog authentication

Pick one way for StarTree to authenticate to the catalog:

| Method | Use when | What you need |
| - | - | - |
| **OAuth service principal** (recommended) | Databricks, especially for automated pipelines | A service principal with an OAuth secret. The token URI is `https://<workspace-host>/oidc/v1/token`, on the same host as the catalog endpoint, with scope `all-apis`. |
| **Personal access token (PAT)** | Databricks, for quick tests | A token from **User Settings → Developer → Access tokens** with the **`all-apis`** scope. |
| **None** | OSS Unity Catalog | Nothing. The OSS server is unauthenticated by default. |

**The `all-apis` scope matters.** Browsing, preview and every sync run go through the Unity Catalog Iceberg REST API, so a token with a narrower scope is rejected (usually a `401` or `403` when the connection is validated) even though it's valid and unexpired. If your workspace lets you pick a scope when generating a token, pick **all-apis**. Tokens created without an explicit scope get it by default.

Store the client secret or PAT as a StarTree secret and reference it as `srn2:secret#...` in SQL and table configs. Don't paste raw values.

### Catalog endpoint formats

| Deployment | Catalog endpoint (`restUri`) |
| - | - |
| Databricks-managed Unity Catalog | `https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest` |
| OSS Unity Catalog | `http://<host>:8080/api/2.1/unity-catalog/iceberg` |

StarTree uses the endpoint exactly as you give it. It doesn't rewrite the host or path; it only appends the standard Iceberg REST `/v1/...` routes. The catalog name you onboard from is used as the `warehouse`.

## Set up a service principal and grants

1. **Create a service principal** (skip if you use a PAT). Go to **Settings → Identity and access → Service principals → Add**. Open it, go to **Secrets → Generate secret**, and copy the client ID and secret. The secret is shown once.
2. **Grant access.** Run this in a SQL editor, using the service principal's client ID or the PAT owner:

   ```sql theme={null}
   GRANT USE CATALOG ON CATALOG <catalog> TO `<principal>`;
   GRANT USE SCHEMA, EXTERNAL USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<principal>`;
   GRANT SELECT ON TABLE <catalog>.<schema>.<table> TO `<principal>`;
   ```

   Grant `SELECT` on the schema instead to cover every table in it. `EXTERNAL USE SCHEMA` is what lets Unity hand short-lived storage credentials to StarTree.
3. **Turn on external data access** if Unity will vend storage credentials. In the workspace, open **Catalog**, click the gear icon, choose **Metastore**, open the **Details** tab and turn on **External data access**. Only a metastore admin sees this switch. See [Enable external data access to Unity Catalog](https://docs.databricks.com/aws/en/external-access/admin).

Without step 3 or `EXTERNAL USE SCHEMA`, browsing the catalog works but table loads fail.

## Common errors

These come from the catalog connection, whichever storage the table uses:

| Error | Cause |
| - | - |
| `Unity credential vending requires a Databricks Iceberg REST endpoint` | The catalog endpoint isn't a Databricks workspace URL ending in `/api/2.1/unity-catalog/iceberg-rest` |
| `OAuth endpoint must belong to the configured Databricks workspace` | The token URI is on a different host from the catalog endpoint, or isn't `/oidc/v1/token` |
| `Databricks OAuth scope must be all-apis` | The OAuth scope is set to something other than `all-apis` |
| `Unable to obtain scoped storage credentials from Unity Catalog` | Unity refused to vend credentials. Check `EXTERNAL USE SCHEMA` and `SELECT`, and that external data access is on for the metastore |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.