> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Unity Catalog + GCS: Onboarding via API

> Onboard a Unity Catalog table stored in Google Cloud Storage as an External Table with the REST API, using Unity-issued credentials or StarTree's Google identity.

This page onboards a **Unity Catalog** table on **Databricks on Google Cloud**, stored in **Google Cloud Storage (GCS)**, with the StarTree controller REST APIs. Tables are read over the **Iceberg REST protocol** (`catalogType=iceberg-rest`, `serviceType=unity`), which covers Iceberg tables and Delta tables with [UniForm](https://docs.delta.io/latest/delta-uniform.html) turned on, including ones that use column mapping. Data files must be **Parquet**.

The flow is the same as [Unity Catalog + S3](https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api.md). What changes is the storage block: two keys pick GCS and how Pinot gets access to the bucket.

## How it works

Onboarding is four calls. Each one feeds the next:

```text theme={null}
1. Validate & browse   POST /connections/browse   →  validate; list namespaces/tables
2. Preview             POST /tables/preview       →  infer schema + enriched table config
3. Create schema       POST /schemas              →  register the schema
4. Create table        POST /tables               →  register the External Table
```

Pinot reads the table's metadata from Unity's Iceberg REST endpoint, then reads the Parquet files straight from GCS. Once the table exists, the controller syncs new snapshots on schedule.

> [!NOTE]
>
> All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot`, the StarTree Cloud data-plane proxy that fronts the controller.
>
> If your controller requires authentication, add an `Authorization` header to every request, e.g. `-H "Authorization: Bearer <token>"`. The examples below omit it for brevity.

## Prerequisites

* **StarTree Cloud 0.19.0 or later** with the External Table feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure.
* Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth.
* A Unity Catalog on a Databricks workspace hosted on Google Cloud, set up as in [Prepare Databricks](https://docs.startree.ai/corecapabilities/external-table/unity/prepare-databricks.md): the table has Iceberg reads (UniForm) turned on if it's a Delta table, you have OAuth service-principal credentials or a PAT with the `all-apis` scope, and the principal has the grants it needs, including `EXTERNAL USE SCHEMA`.
* The Unity Catalog **catalog name** (used as `warehouse` below) and the schema and table to onboard.

> [!NOTE]
>
> **Domain restricted sharing.** Unity Catalog on Google Cloud reaches your bucket through a service account that Databricks creates in its own project. If your organization enforces the `iam.allowedPolicyMemberDomains` policy, GCS rejects that grant with `One or more users named in the policy do not belong to a permitted customer`. Your GCP admin needs to allow Databricks' customer ID `C01p0oudw` on the project that holds the bucket. See [Databricks' Google Cloud requirements](https://docs.databricks.com/gcp/en/admin/account-settings-gcp/create-subscription). Granting StarTree's identity for `gcs-adc` may need a similar exception.

## Storage access: choose one

Set `sourceStorageProvider` to `gcs`, then pick how Pinot reads the bucket with `externalTableAccessPolicy`. Always set it: a table config without it is rejected.

| `externalTableAccessPolicy` | How Pinot reads GCS | Use for |
| - | - | - |
| `unity` (recommended) | Unity issues a short-lived GCS token scoped to each table, and Pinot renews it before it expires. You manage no storage credentials. | Unity-managed Iceberg tables, and Delta tables with UniForm turned on |
| `gcs-adc` | Pinot uses StarTree's own Google identity, with read-only access. You grant that identity read access on the bucket. | Tables Unity won't issue credentials for, such as foreign Iceberg tables |

```json theme={null}
{
  "sourceStorageProvider": "gcs",
  "externalTableAccessPolicy": "unity"
}
```

* **For `unity`**, turn on **external data access** for the metastore and grant the principal `EXTERNAL USE SCHEMA`, as in [Set up a service principal and grants](https://docs.startree.ai/corecapabilities/external-table/unity/prepare-databricks.md#set-up-a-service-principal-and-grants). Without them, Unity returns the table's metadata but no storage token, so reads fail.
* **For `gcs-adc`**, grant StarTree's Google identity `storage.objects.get` and `storage.objects.list` on the bucket. The **Storage Object Viewer** role covers both. Contact StarTree support for the identity to grant.

GCS takes no region and no storage keys. Leave out every `catalog.iceberg-rest.auth.storage.*` key.

## Catalog authentication

Catalog keys go under `catalog.iceberg-rest.auth.rest.*`, the same as for [Unity Catalog + S3](https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api.md#catalog-rest-authentication-choose-one). Use the workspace host, `<workspace-id>.<n>.gcp.databricks.com`, for both the catalog endpoint and the token URI. Account-console hosts (`accounts.gcp.databricks.com`) and regional hosts are rejected with `unity`.

**OAuth client credentials** (recommended):

```json theme={null}
{
  "catalog.iceberg-rest.auth.rest.authType": "oauth2",
  "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://<workspace-id>.<n>.gcp.databricks.com/oidc/v1/token",
  "catalog.iceberg-rest.auth.rest.oauthClientId": "<service-principal-client-id>",
  "catalog.iceberg-rest.auth.rest.oauthClientSecret": "<service-principal-secret>",
  "catalog.iceberg-rest.auth.rest.oauthScopes": "all-apis"
}
```

Always set `oauthTokenUri` to the workspace's `/oidc/v1/token`. The `unity` policy rejects any other token URI, and the scope must be `all-apis`.

**Personal access token.** On Google Cloud workspaces, a PAT works with both policies:

```json theme={null}
{
  "catalog.iceberg-rest.auth.rest.token": "<databricks-pat>"
}
```

You don't need the `X-Iceberg-Access-Delegation` header or `token-exchange-enabled`. With `unity`, Pinot sets them itself.

***

## Step 1: Validate and browse the connection

**`POST /connections/browse`**

Browsing the catalog is the validation step. A `200` with an `items` list (even an empty one) confirms the catalog credentials and connectivity. It doesn't touch storage, so a storage problem shows up at preview.

```bash theme={null}
curl -X POST "$BASE_URL/connections/browse" \
  -H "Content-Type: application/json" \
  -d '{
    "connection": {
      "type": "CATALOG",
      "params": {
        "catalogType": "iceberg-rest",
        "catalog.iceberg-rest.serviceType": "unity",
        "catalog.iceberg-rest.restUri": "https://<workspace-id>.<n>.gcp.databricks.com/api/2.1/unity-catalog/iceberg-rest",
        "catalog.iceberg-rest.warehouse": "<unity-catalog-name>",
        "catalog.iceberg-rest.auth.rest.authType": "oauth2",
        "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://<workspace-id>.<n>.gcp.databricks.com/oidc/v1/token",
        "catalog.iceberg-rest.auth.rest.oauthClientId": "<service-principal-client-id>",
        "catalog.iceberg-rest.auth.rest.oauthClientSecret": "<service-principal-secret>",
        "catalog.iceberg-rest.auth.rest.oauthScopes": "all-apis"
      }
    },
    "path": ""
  }'
```

Set `path` to `""` to list namespaces, or to a namespace name to list its tables. The response format is described in [Unity Catalog + S3](https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api.md#response).

***

## Step 2: Preview the schema

**`POST /tables/preview`**

Samples the source, infers a Pinot schema, and returns an enriched table config plus sample rows. This is the first call that reads from GCS, so it also confirms storage access.

```bash theme={null}
curl -X POST "$BASE_URL/tables/preview" \
  -H "Content-Type: application/json" \
  -d '{
    "tableConfig": {
      "tableName": "events_OFFLINE",
      "tableType": "OFFLINE",
      "task": {
        "taskTypeConfigsMap": {
          "ExternalTableSyncTask": {
            "catalogType": "iceberg-rest",
            "executor": "controller",
            "inputFormat": "parquet",
            "catalog.iceberg-rest.serviceType": "unity",
            "catalog.iceberg-rest.restUri": "https://<workspace-id>.<n>.gcp.databricks.com/api/2.1/unity-catalog/iceberg-rest",
            "catalog.iceberg-rest.warehouse": "<unity-catalog-name>",
            "catalog.iceberg-rest.table.namespace": "<schema>",
            "catalog.iceberg-rest.table.tableName": "<table>",
            "catalog.iceberg-rest.auth.rest.authType": "oauth2",
            "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://<workspace-id>.<n>.gcp.databricks.com/oidc/v1/token",
            "catalog.iceberg-rest.auth.rest.oauthClientId": "<service-principal-client-id>",
            "catalog.iceberg-rest.auth.rest.oauthClientSecret": "<service-principal-secret>",
            "catalog.iceberg-rest.auth.rest.oauthScopes": "all-apis",
            "sourceStorageProvider": "gcs",
            "externalTableAccessPolicy": "unity"
          }
        }
      }
    },
    "config": { "inference": { "embeddedSchemaEnabled": true } }
  }'
```

* `table.namespace` and `table.tableName` are the Unity schema and table names. Each is a single name with no dots.
* `executor` must be `controller`. GCS tables sync snapshots on the controller, and file watching must stay off.

The response carries `schema` (use it in Step 3) and `tableConfigs.offline` (use it in Step 4), with secrets masked as `*****`. The full request and response fields, including the column-mapping options, are in [Unity Catalog + S3](https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api.md#step-2-preview-the-schema).

***

## Step 3: Create the schema

**`POST /schemas`**

Send the `schema` object from the preview response, with `schemaName` set to match your table.

```bash theme={null}
curl -X POST "$BASE_URL/schemas" \
  -H "Content-Type: application/json" \
  -d @schema.json
```

***

## Step 4: Create the table

**`POST /tables`**

Send the enriched `tableConfigs.offline` object from the preview response.

```bash theme={null}
curl -X POST "$BASE_URL/tables" \
  -H "Content-Type: application/json" \
  -d @tableConfig.json
```

The controller runs the first sync and then re-syncs on the `schedule` (cron) interval. To start the first sync right away:

```bash theme={null}
curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=events_OFFLINE"
```

***

## Quickstart: onboard a table end-to-end

The whole flow as one script, with OAuth and Unity-issued credentials. Fill in the variables at the top and run it.

```bash theme={null}
#!/usr/bin/env bash
set -euo pipefail

BASE_URL="https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot"
AUTH="Authorization: Bearer <token>"
WORKSPACE_HOST="<workspace-id>.<n>.gcp.databricks.com"
WAREHOUSE="<unity-catalog-name>"
NAMESPACE="<schema>"
TABLE="<table>"
CLIENT_ID="<service-principal-client-id>"
CLIENT_SECRET="<service-principal-secret>"

SOURCE=$(jq -n \
  --arg uri "https://$WORKSPACE_HOST/api/2.1/unity-catalog/iceberg-rest" \
  --arg tok "https://$WORKSPACE_HOST/oidc/v1/token" \
  --arg wh "$WAREHOUSE" --arg id "$CLIENT_ID" --arg secret "$CLIENT_SECRET" '{
    "catalogType": "iceberg-rest",
    "catalog.iceberg-rest.serviceType": "unity",
    "catalog.iceberg-rest.restUri": $uri,
    "catalog.iceberg-rest.warehouse": $wh,
    "catalog.iceberg-rest.auth.rest.authType": "oauth2",
    "catalog.iceberg-rest.auth.rest.oauthTokenUri": $tok,
    "catalog.iceberg-rest.auth.rest.oauthClientId": $id,
    "catalog.iceberg-rest.auth.rest.oauthClientSecret": $secret,
    "catalog.iceberg-rest.auth.rest.oauthScopes": "all-apis"
  }')

# 1. Validate & browse root.
jq -n --argjson p "$SOURCE" '{connection: {type: "CATALOG", params: $p}, path: ""}' \
  | curl -sf -X POST "$BASE_URL/connections/browse" -H "$AUTH" -H 'Content-Type: application/json' -d @- | jq .

# 2. Preview: infer schema + enriched table config.
jq -n --argjson p "$SOURCE" --arg ns "$NAMESPACE" --arg t "$TABLE" '{
  tableConfig: {
    tableName: ($t + "_OFFLINE"), tableType: "OFFLINE",
    task: {taskTypeConfigsMap: {ExternalTableSyncTask: ($p + {
      "executor": "controller",
      "inputFormat": "parquet",
      "catalog.iceberg-rest.table.namespace": $ns,
      "catalog.iceberg-rest.table.tableName": $t,
      "sourceStorageProvider": "gcs",
      "externalTableAccessPolicy": "unity"
    })}}
  },
  config: {inference: {embeddedSchemaEnabled: true}}
}' | curl -sf -X POST "$BASE_URL/tables/preview" -H "$AUTH" -H 'Content-Type: application/json' -d @- > preview.json

# 3. Extract schema and table config.
jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json
jq '.tableConfigs.offline'                        preview.json > tableConfig.json

# 4. Create the schema, then the table. The controller starts the first sync.
curl -sf -X POST "$BASE_URL/schemas" -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json
curl -sf -X POST "$BASE_URL/tables"  -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json

# 5. (optional) Check whether the first sync has finished.
curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \
  | jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered'
```

When step 5 exits `0`, run `SELECT count(*) FROM <table>` to confirm the data is live. For run status, checkpoints and file counts, see [Observability](https://docs.startree.ai/corecapabilities/external-table/observability.md).

***

## How credentials behave

These apply to the `unity` policy.

* Pinot fetches a GCS token for each table location and renews it shortly before it expires.
* Every query checks the table's access with Unity first. If you revoke a grant, new queries on the table fail. Queries already running aren't cancelled.
* [Cursor pagination](https://docs.startree.ai/corecapabilities/query_data/advanced_operations/cursor-pagination.md) isn't supported on these tables. Cursor requests fail with `Credential-backed external tables do not support stored result cursors`.

## SQL DDL

[SQL DDL](https://docs.startree.ai/corecapabilities/external-table/sql-ddl.md) can create the same table. With OAuth and the `unity` policy:

```sql theme={null}
CREATE TABLE unity_gcs_events WITH (
  type = 'iceberg',
  catalog_type = 'unity',
  catalog_uri = 'https://<workspace-id>.<n>.gcp.databricks.com/api/2.1/unity-catalog/iceberg-rest',
  catalog_name = '<catalog>',
  schema_name = '<schema>',
  table_name = '<table>',
  auth_type = 'oauth',
  client_id = 'srn2:secret#<client-id-reference>',
  client_secret = 'srn2:secret#<client-secret-reference>',
  oauth_token_uri = 'https://<workspace-id>.<n>.gcp.databricks.com/oidc/v1/token',
  oauth_scope = 'all-apis',
  storage.provider = 'gcs',
  storage.access_policy = 'unity',
  refresh_interval = '5m'
);
```

* Always set `oauth_token_uri`. If you leave it out, the token URI defaults to `<catalog_uri>/v1/oauth/tokens`, which the `unity` policy rejects.
* For a PAT, replace the five OAuth lines with `auth_type = 'token'` and `token = 'srn2:secret#<pat-reference>'`.
* In SQL DDL only, `storage.access_policy` defaults to `gcs-adc` when you leave it out. The API has no default.
* `storage.region`, `storage.access_key`, `storage.secret_key`, `storage.session_token`, `storage.endpoint`, `storage.sas_token` and `storage.credential_prefix` are rejected with `Option '<key>' is incompatible with storage.provider = 'gcs'.`

## Common errors

Catalog and OAuth errors are listed in [Prepare Databricks](https://docs.startree.ai/corecapabilities/external-table/unity/prepare-databricks.md#common-errors). These are specific to GCS:

| Error | Cause |
| - | - |
| `No Unity storage credentials returned` | Unity loaded the table but didn't issue a token. Usually external data access is off for the metastore, or the principal lacks `EXTERNAL USE SCHEMA` on the schema. A workspace admin without that grant gets no token either. If the table is a type Unity doesn't issue tokens for, use `gcs-adc` |
| `GCS source storage requires a compatible catalog authorization policy` | `externalTableAccessPolicy` is missing or set to `azure-sas`. A misspelled policy fails with `Unsupported external table authorization policy` instead |
| `Native GCS source storage requires an Iceberg REST catalog and controller snapshot sync` | `executor` isn't `controller`, or file watching is turned on |
| `Cannot read the selected GCS data file. Check the source storage authorization and object location.` | The check Pinot runs at create time couldn't read a data file. With `gcs-adc`, the bucket grant is missing |
| `No GCS credential covers the requested source path` | A data file sits outside the location Unity issued the token for |
| `Invalid GCS source location` | The table location isn't a valid `gs://<bucket>/<path>` URI |
| `GCS source location is outside its configured bucket` | A data file points to a different bucket from the table's |
| `GCS source object was not found` | A file listed in the table's metadata is gone from the bucket |

## Not supported

* Delta tables without Iceberg reads (UniForm) turned on.
* `type = 'delta'` tables read through Unity's Delta APIs.

***

## What's next

1. **Add indexes for your query patterns.** Without indexes, every query scans the remote Parquet data. → [Indexes](https://docs.startree.ai/corecapabilities/external-table/indexes.md)
2. **Turn on caching and pin hot indexes.** → [Data and Index Caching](https://docs.startree.ai/corecapabilities/external-table/data-and-index-caching.md)
3. **Watch ongoing syncs** with the observability endpoints. → [Observability](https://docs.startree.ai/corecapabilities/external-table/observability.md)

For common questions and failures, see the [FAQ](https://docs.startree.ai/corecapabilities/external-table/faq.md) and [Troubleshooting](https://docs.startree.ai/corecapabilities/external-table/troubleshooting.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.