> ## Documentation Index > Fetch the complete documentation index at: https://docs.startree.ai/llms.txt > Use this file to discover all available pages before exploring further. # Snowflake Horizon Catalog: Onboarding via API > Query Snowflake-managed Iceberg tables in StarTree — onboard a Snowflake Horizon Catalog table as an External Table over the Iceberg REST API, authenticating with a Snowflake programmatic access token (PAT). This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it. This page shows how to onboard a **Snowflake Horizon Catalog** External Table with the StarTree controller REST APIs. There is no Data Portal wizard for Horizon yet — this API flow is the only way to onboard one today. Snowflake Horizon exposes Snowflake-managed Iceberg tables over the same **Iceberg REST protocol** as Apache Polaris, served on the Snowflake account host under `/polaris/api/catalog`. StarTree reaches it through the generic spec-compliant adapter — `catalogType=iceberg-rest`, `serviceType=rest`. Data files must be **Parquet**. > Running **self-managed Apache Polaris** or **Snowflake Open Catalog** instead? Those use principal `client-id`/`client-secret` credentials — see [Apache Polaris & Snowflake Open Catalog: Onboarding via API](../polaris/onboarding-api). ## How Horizon differs from Apache Polaris Horizon speaks the Polaris REST protocol, but authentication and identifier mapping follow Snowflake's model, not Polaris's: | | Snowflake Horizon Catalog | | ------------------ | ------------------------------------------------------------------------------------------------- | | REST base URI | `https://-.snowflakecomputing.com/polaris/api/catalog` | | Identity | Snowflake **service user** + a **programmatic access token (PAT)** (or key-pair JWT) | | Catalog credential | the **PAT**, passed via `catalog.iceberg-rest.credential` (see [Authentication](#authentication)) | | `oauthScopes` | `session:role:` — Polaris-style `PRINCIPAL_ROLE:ALL` returns `401` | | `warehouse` | the Snowflake **database** name | | Namespace | the Snowflake **schema** (single level; the database is the prefix) | | `prefix` | **required** — set it to the database name (same value as `warehouse`) | | Access control | native Snowflake RBAC (`GRANT SELECT` / `USAGE`); the PAT also needs a **network policy** | **Why the PAT goes in `catalog.iceberg-rest.credential`.** Horizon's token endpoint expects a `client_credentials` exchange that sends the PAT as `client_secret` with **no** `client_id`. Passing the PAT through `catalog.iceberg-rest.credential` produces exactly that request. (Setting `oauthClientSecret` with an empty `oauthClientId` does **not** currently work — the empty client id causes the credential to be dropped and the catalog call goes out unauthenticated. Use `catalog.iceberg-rest.credential` until first-class secret-only support lands.) ## How it works Onboarding is four calls. Each one feeds the next: ```text theme={null} 1. Validate & browse POST /connections/browse → validate; list namespaces/tables 2. Preview POST /tables/preview → infer schema + enriched table config 3. Create schema POST /schemas → register the schema 4. Create table POST /tables → register the External Table ``` The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output. Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding). All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` (the StarTree Cloud data-plane proxy). A browse call then posts to `$BASE_URL/connections/browse`. If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity. ## Prerequisites * StarTree 0.16.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure. * Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth. * A Snowflake account with **Horizon Catalog** and the target Iceberg table backed by an **external volume** (S3). * A Snowflake **service user** with a **programmatic access token (PAT)**, a role that can read the table, and a **network policy** governing the user (see [Snowflake setup](#snowflake-setup) below). * The Snowflake **database** (used as both `warehouse` and `prefix`), the **schema** (the namespace), and the **table** to onboard. * Read access to the underlying object storage. This connector supports **S3-compatible storage** for the data files. ## Snowflake setup Run these once in Snowflake (or have an ACCOUNTADMIN do so). Replace the placeholders with your own names. ```sql theme={null} -- 1. A role that can read the Iceberg table's database, schema, and external volume. GRANT USAGE ON DATABASE TO ROLE ; GRANT USAGE ON SCHEMA . TO ROLE ; GRANT SELECT ON ALL TABLES IN SCHEMA . TO ROLE ; GRANT USAGE ON EXTERNAL VOLUME TO ROLE ; -- 2. A dedicated service user, and the role granted to it. CREATE USER IF NOT EXISTS DEFAULT_ROLE = ; GRANT ROLE TO USER ; -- 3. A network policy is REQUIRED before a PAT can be created or used. ALTER USER SET NETWORK_POLICY = ; -- 4. The programmatic access token. Copy the secret it prints — it is shown only once. ALTER USER IF EXISTS ADD PAT startree_external_table DAYS_TO_EXPIRY = 90 ROLE_RESTRICTION = '' COMMENT = 'StarTree External Table catalog access'; ``` The `ROLE_RESTRICTION` role must match the role named in the `session:role:` scope you configure below, and it must hold the grants from step 1. Without a **network policy** in effect, `ADD PAT` fails and the token endpoint later returns `401 unauthorized_client`. ## Authentication An External Table authenticates in two places, configured independently: * **Catalog (REST)** — the Snowflake PAT, supplied via `catalog.iceberg-rest.credential` plus the token URI and scope. * **Storage (S3 data files)** — keys under `catalog.iceberg-rest.auth.storage.*`. Choose one of the methods below. ### Catalog (REST) authentication Pass the **PAT** as the catalog credential. StarTree exchanges it at the Snowflake token endpoint for a short-lived token, which it caches and refreshes automatically. ```json theme={null} { "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens", "catalog.iceberg-rest.auth.rest.oauthScopes": "session:role:", "catalog.iceberg-rest.credential": "" } ``` | Key | Value | Description | | ---------------------------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `catalog.iceberg-rest.credential` | `` | The Snowflake **programmatic access token**. Sent to the token endpoint as `client_secret` with no client id. | | `catalog.iceberg-rest.auth.rest.oauthTokenUri` | `https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens` | Snowflake's OAuth token endpoint (the account host, `/polaris/api/catalog/v1/oauth/tokens`). | | `catalog.iceberg-rest.auth.rest.oauthScopes` | `session:role:` | Activates the Snowflake role for the session. Must match the PAT's `ROLE_RESTRICTION`. | | `catalog.iceberg-rest.auth.rest.authType` | `oauth2` | Optional; auto-detected as `oauth2` when a credential is present. | The PAT is a **secret** — treat the table config as sensitive and rotate the token on the `DAYS_TO_EXPIRY` cadence. A static PAT is not auto-rotated; when it expires, syncs fail until you update the table config with a new one. Key-pair JWT auth (a signed JWT in place of the PAT) works the same way — put the JWT in `catalog.iceberg-rest.credential`. ### Storage authentication: choose one #### Static access keys What most Snowflake external-volume setups use. Accepts either `accessKeyId`/`secretAccessKey` or the shorter `accessKey`/`secretKey` aliases. ```json theme={null} { "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Assumed IAM role (recommended for production) No static secrets in the table config — every S3 read assumes `roleArn` via STS. ```json theme={null} { "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Cluster node role Omit every `auth.storage.*` credential key — only `region` is required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have S3 access. ```json theme={null} { "catalog.iceberg-rest.auth.storage.region": "" } ``` Horizon can vend short-lived storage credentials, but the generic `rest` adapter does not consume vended credentials — configure static keys, an assumed role, or the cluster's node role for storage explicitly. ### Selecting the database Set both `catalog.iceberg-rest.warehouse` and `catalog.iceberg-rest.prefix` to the Snowflake **database** name. `prefix` is **required** for Horizon: the generic adapter doesn't send the `warehouse` query parameter that Snowflake's `/v1/config` gates on, so without an explicit prefix, browse falls back to the un-prefixed `/v1/namespaces` path and fails with `404 Unable to find matching target resource method`. ```json theme={null} { "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.prefix": "" } ``` The Snowflake **schema** is the Iceberg namespace — set `catalog.iceberg-rest.table.namespace` to the schema name and `catalog.iceberg-rest.table.tableName` to the table. *** ## Step 1: Validate and browse the connection **`POST /connections/browse`** There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your PAT, scope, and connectivity. * Set `path` to `""` to browse the **root** — this both validates the connection and lists schemas (namespaces). * Set `path` to a schema name to list the **tables** inside it. ### Request | Field | Type | Required | Description | | ------------------- | ------ | -------- | ------------------------------------------------------------------------------ | | `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. | | `connection.params` | object | Yes | Catalog-specific connection settings (see example below). | | `path` | string | No | Where to browse. `""` or omitted = root. A schema name = that schema's tables. | ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "iceberg-rest", "catalog.iceberg-rest.serviceType": "rest", "catalog.iceberg-rest.restUri": "https://-.snowflakecomputing.com/polaris/api/catalog", "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.prefix": "", "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens", "catalog.iceberg-rest.auth.rest.oauthScopes": "session:role:", "catalog.iceberg-rest.credential": "", "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } }, "path": "" }' ``` | Key | Value | Description | | ---------------------------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `catalog.iceberg-rest.restUri` | `https://-.snowflakecomputing.com/polaris/api/catalog` | The Snowflake Iceberg REST base URI. | | `catalog.iceberg-rest.serviceType` | `rest` | Selects the generic spec-compliant adapter — there's no Snowflake-specific `serviceType`. | | `catalog.iceberg-rest.warehouse` | `` | The Snowflake database to browse. | | `catalog.iceberg-rest.prefix` | `` | Pins the catalog prefix (the database). Required for Horizon. | ### Response ```json theme={null} { "items": [ { "name": "SCHEMA_ONE", "type": "NAMESPACE" }, { "name": "nyc_taxi_trips", "type": "TABLE" } ] } ``` | `items[].type` | Meaning | | -------------- | -------------------------------------------------------- | | `NAMESPACE` | A schema you can drill into — pass its `name` as `path`. | | `TABLE` | A selectable table. | *** ## Step 2: Preview the schema **`POST /tables/preview`** Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4. The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`. ### Request **Top-level fields:** | Field | Type | Required | Description | | ------------- | ------ | -------- | --------------------------------------------------------------------------------------- | | `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. | | `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. | | `schema` | object | No | An explicit schema to use instead of inferring one. | **`config.inference` — how the schema is derived:** | Field | Type | Default | Description | | ------------------------ | ------- | ------- | --------------------------------------------------------------------------------------------- | | `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Iceberg metadata. **Set this to `true` for External Tables.** | | `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. | Resolution order: explicit `schema` → embedded schema → inferred schema. **`config.sampling` — how rows are sampled:** | Field | Type | Default | Description | | ----- | ------- | ------- | -------------------------- | | `min` | integer | `1` | Minimum records to sample. | | `max` | integer | `100` | Maximum records to sample. | ```bash theme={null} curl -X POST "$BASE_URL/tables/preview" \ -H "Content-Type: application/json" \ -d '{ "tableConfig": { "tableName": "nyc_taxi_trips_OFFLINE", "tableType": "OFFLINE", "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "catalogType": "iceberg-rest", "executor": "controller", "inputFormat": "parquet", "catalog.iceberg-rest.serviceType": "rest", "catalog.iceberg-rest.restUri": "https://-.snowflakecomputing.com/polaris/api/catalog", "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.prefix": "", "catalog.iceberg-rest.table.namespace": "", "catalog.iceberg-rest.table.tableName": "nyc_taxi_trips", "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens", "catalog.iceberg-rest.auth.rest.oauthScopes": "session:role:", "catalog.iceberg-rest.credential": "", "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } } } }, "config": { "inference": { "embeddedSchemaEnabled": true } } }' ``` **Setting the namespace and table.** Take them from the [browse](#step-1-validate-and-browse-the-connection) response — set `catalog.iceberg-rest.table.namespace` (the Snowflake schema) and `.tableName` to the `NAMESPACE` and `TABLE` you selected. **Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4. ### Response | Field | Type | Description | | ---------------------- | ------ | ----------------------------------------------------------------------------------------------------------- | | `schema` | object | The resolved Pinot schema. **Use this in step 3.** | | `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. | | `rows` | array | Sample records after schema + ingestion transforms. | | `sourceRows` | array | Raw records before transformation. | | `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). | Adjust the schema (time column, column names, null handling) before moving on. *** ## Step 3: Create the schema **`POST /schemas`** Send the `schema` object from the preview response. Rename its `schemaName` to match your table. ```bash theme={null} curl -X POST "$BASE_URL/schemas" \ -H "Content-Type: application/json" \ -d @schema.json ``` ```json theme={null} { "status": "nyc_taxi_trips successfully added" } ``` *** ## Step 4: Create the table **`POST /tables`** Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external. ```bash theme={null} curl -X POST "$BASE_URL/tables" \ -H "Content-Type: application/json" \ -d @tableConfig.json ``` ```json theme={null} { "status": "Table nyc_taxi_trips_OFFLINE successfully added" } ``` The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call. The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run: ```bash theme={null} curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE" ``` *** ## Quickstart: onboard a table end-to-end The whole flow as one script. Fill in the variables at the top, run the script, and the first sync starts automatically. ```bash theme={null} #!/usr/bin/env bash set -euo pipefail BASE_URL="https://dp..cp..startree.cloud/api/pinot" AUTH="Authorization: Bearer " ACCOUNT_HOST="-.snowflakecomputing.com" REST_URI="https://$ACCOUNT_HOST/polaris/api/catalog" TOKEN_URI="https://$ACCOUNT_HOST/polaris/api/catalog/v1/oauth/tokens" DATABASE="" # used as both warehouse and prefix SCHEMA="" # the Iceberg namespace TABLE="nyc_taxi_trips" ROLE="" SCOPES="session:role:$ROLE" PAT="" STORAGE_REGION="" STORAGE_ROLE_ARN="arn:aws:iam::123456789012:role/pinot-external-table-reader" STORAGE_EXTERNAL_ID="" # 1. Validate & browse root — lists schemas in the Snowflake database. curl -sf -X POST "$BASE_URL/connections/browse" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"connection\": { \"type\": \"CATALOG\", \"params\": { \"catalogType\": \"iceberg-rest\", \"catalog.iceberg-rest.serviceType\": \"rest\", \"catalog.iceberg-rest.restUri\": \"$REST_URI\", \"catalog.iceberg-rest.warehouse\": \"$DATABASE\", \"catalog.iceberg-rest.prefix\": \"$DATABASE\", \"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\", \"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\", \"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\", \"catalog.iceberg-rest.credential\": \"$PAT\" } }, \"path\": \"\" }" | jq . # 2. Preview — infer schema + enriched table config. curl -sf -X POST "$BASE_URL/tables/preview" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"tableConfig\": { \"tableName\": \"${TABLE}_OFFLINE\", \"tableType\": \"OFFLINE\", \"task\": { \"taskTypeConfigsMap\": { \"ExternalTableSyncTask\": { \"catalogType\": \"iceberg-rest\", \"executor\": \"controller\", \"inputFormat\": \"parquet\", \"catalog.iceberg-rest.serviceType\": \"rest\", \"catalog.iceberg-rest.restUri\": \"$REST_URI\", \"catalog.iceberg-rest.warehouse\": \"$DATABASE\", \"catalog.iceberg-rest.prefix\": \"$DATABASE\", \"catalog.iceberg-rest.table.namespace\": \"$SCHEMA\", \"catalog.iceberg-rest.table.tableName\": \"$TABLE\", \"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\", \"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\", \"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\", \"catalog.iceberg-rest.credential\": \"$PAT\", \"catalog.iceberg-rest.auth.storage.roleArn\": \"$STORAGE_ROLE_ARN\", \"catalog.iceberg-rest.auth.storage.externalId\": \"$STORAGE_EXTERNAL_ID\", \"catalog.iceberg-rest.auth.storage.region\": \"$STORAGE_REGION\" } } } }, \"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } } }" > preview.json # 3. Extract schema and table config from preview response. jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json jq '.tableConfigs.offline' preview.json > tableConfig.json # 4. Create the schema. curl -sf -X POST "$BASE_URL/schemas" \ -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json # 5. Create the table — watcher starts the first sync automatically. curl -sf -X POST "$BASE_URL/tables" \ -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json # 6. (optional) Poll until sync completes. curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \ | jq -e '.status == "COMPLETED" and .segmentsUploaded == .filesDiscovered' ``` Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable). ## Monitor onboarding The [status endpoint](../observability#run-status) reports each sync run's progress, and `?includeLag=true` adds how far the table trails the upstream catalog. Both require `executor=controller` (set automatically by this flow). See [Observability](../observability) for full request and response details. ## Verify it's queryable When `status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips; ``` The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If `status` is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting). *** ## What's next Now that the table is created and data is loading, these are the highest-impact follow-up steps: 1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes) 2. **Enable caching and preload.** Set `enable.prefetch.page.cache=true` and `preload.enable=true` on the S3 tier so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching) 3. **Protect large-scan queries from OOM.** For tables that receive heavy aggregations or wide scans, enable the query OOM killer so a runaway query is killed instead of crashing the server. → [Best Practices & Configs — Query OOM protection](../best-practices-and-configs#query-oom-protection-large-scans) 4. **Monitor ongoing syncs.** Use the status endpoint (with `includeLag=true` for ingestion lag) after each scheduled sync. → [Observability](../observability) *** For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting).