Skip to main content
This page onboards a Unity Catalog table on Databricks on Google Cloud, stored in Google Cloud Storage (GCS), with the StarTree controller REST APIs. Tables are read over the Iceberg REST protocol (catalogType=iceberg-rest, serviceType=unity), which covers Iceberg tables and Delta tables with UniForm turned on, including ones that use column mapping. Data files must be Parquet. The flow is the same as Unity Catalog + S3. What changes is the storage block: two keys pick GCS and how Pinot gets access to the bucket.

How it works

Onboarding is four calls. Each one feeds the next:
Pinot reads the table’s metadata from Unity’s Iceberg REST endpoint, then reads the Parquet files straight from GCS. Once the table exists, the controller syncs new snapshots on schedule.
All paths are relative to your data-plane base URL. Set export BASE_URL=https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot, the StarTree Cloud data-plane proxy that fronts the controller.If your controller requires authentication, add an Authorization header to every request, e.g. -H "Authorization: Bearer <token>". The examples below omit it for brevity.

Prerequisites

  • StarTree Cloud 0.19.0 or later with the External Table feature enabled, and tiered storage configured on the cluster. Contact StarTree support if unsure.
  • Network access to the controller REST endpoint, plus an Authorization token if your cluster requires auth.
  • A Unity Catalog on a Databricks workspace hosted on Google Cloud, set up as in Prepare Databricks: the table has Iceberg reads (UniForm) turned on if it’s a Delta table, you have OAuth service-principal credentials or a PAT with the all-apis scope, and the principal has the grants it needs, including EXTERNAL USE SCHEMA.
  • The Unity Catalog catalog name (used as warehouse below) and the schema and table to onboard.
Domain restricted sharing. Unity Catalog on Google Cloud reaches your bucket through a service account that Databricks creates in its own project. If your organization enforces the iam.allowedPolicyMemberDomains policy, GCS rejects that grant with One or more users named in the policy do not belong to a permitted customer. Your GCP admin needs to allow Databricks’ customer ID C01p0oudw on the project that holds the bucket. See Databricks’ Google Cloud requirements. Granting StarTree’s identity for gcs-adc may need a similar exception.

Storage access: choose one

Set sourceStorageProvider to gcs, then pick how Pinot reads the bucket with externalTableAccessPolicy. Always set it: a table config without it is rejected.
  • For unity, turn on external data access for the metastore and grant the principal EXTERNAL USE SCHEMA, as in Set up a service principal and grants. Without them, Unity returns the table’s metadata but no storage token, so reads fail.
  • For gcs-adc, grant StarTree’s Google identity storage.objects.get and storage.objects.list on the bucket. The Storage Object Viewer role covers both. Contact StarTree support for the identity to grant.
GCS takes no region and no storage keys. Leave out every catalog.iceberg-rest.auth.storage.* key.

Catalog authentication

Catalog keys go under catalog.iceberg-rest.auth.rest.*, the same as for Unity Catalog + S3. Use the workspace host, <workspace-id>.<n>.gcp.databricks.com, for both the catalog endpoint and the token URI. Account-console hosts (accounts.gcp.databricks.com) and regional hosts are rejected with unity. OAuth client credentials (recommended):
Always set oauthTokenUri to the workspace’s /oidc/v1/token. The unity policy rejects any other token URI, and the scope must be all-apis. Personal access token. On Google Cloud workspaces, a PAT works with both policies:
You don’t need the X-Iceberg-Access-Delegation header or token-exchange-enabled. With unity, Pinot sets them itself.

Step 1: Validate and browse the connection

POST /connections/browse Browsing the catalog is the validation step. A 200 with an items list (even an empty one) confirms the catalog credentials and connectivity. It doesn’t touch storage, so a storage problem shows up at preview.
Set path to "" to list namespaces, or to a namespace name to list its tables. The response format is described in Unity Catalog + S3.

Step 2: Preview the schema

POST /tables/preview Samples the source, infers a Pinot schema, and returns an enriched table config plus sample rows. This is the first call that reads from GCS, so it also confirms storage access.
  • table.namespace and table.tableName are the Unity schema and table names. Each is a single name with no dots.
  • executor must be controller. GCS tables sync snapshots on the controller, and file watching must stay off.
The response carries schema (use it in Step 3) and tableConfigs.offline (use it in Step 4), with secrets masked as *****. The full request and response fields, including the column-mapping options, are in Unity Catalog + S3.

Step 3: Create the schema

POST /schemas Send the schema object from the preview response, with schemaName set to match your table.

Step 4: Create the table

POST /tables Send the enriched tableConfigs.offline object from the preview response.
The controller runs the first sync and then re-syncs on the schedule (cron) interval. To start the first sync right away:

Quickstart: onboard a table end-to-end

The whole flow as one script, with OAuth and Unity-issued credentials. Fill in the variables at the top and run it.
When step 5 exits 0, run SELECT count(*) FROM <table> to confirm the data is live. For run status, checkpoints and file counts, see Observability.

How credentials behave

These apply to the unity policy.
  • Pinot fetches a GCS token for each table location and renews it shortly before it expires.
  • Every query checks the table’s access with Unity first. If you revoke a grant, new queries on the table fail. Queries already running aren’t cancelled.
  • Cursor pagination isn’t supported on these tables. Cursor requests fail with Credential-backed external tables do not support stored result cursors.

SQL DDL

SQL DDL can create the same table. With OAuth and the unity policy:
  • Always set oauth_token_uri. If you leave it out, the token URI defaults to <catalog_uri>/v1/oauth/tokens, which the unity policy rejects.
  • For a PAT, replace the five OAuth lines with auth_type = 'token' and token = 'srn2:secret#<pat-reference>'.
  • In SQL DDL only, storage.access_policy defaults to gcs-adc when you leave it out. The API has no default.
  • storage.region, storage.access_key, storage.secret_key, storage.session_token, storage.endpoint, storage.sas_token and storage.credential_prefix are rejected with Option '<key>' is incompatible with storage.provider = 'gcs'.

Common errors

Catalog and OAuth errors are listed in Prepare Databricks. These are specific to GCS:

Not supported

  • Delta tables without Iceberg reads (UniForm) turned on.
  • type = 'delta' tables read through Unity’s Delta APIs.

What’s next

  1. Add indexes for your query patterns. Without indexes, every query scans the remote Parquet data. → Indexes
  2. Turn on caching and pin hot indexes. → Data and Index Caching
  3. Watch ongoing syncs with the observability endpoints. → Observability
For common questions and failures, see the FAQ and Troubleshooting.