# Get progress and estimated completion time for the task
Source: https://docs.startree.ai/api-reference/altertable/get-progress-and-estimated-completion-time-for-the-task
get /alterTable/{tableNameWithType}/estimatedTime
# Get appconfigs
Source: https://docs.startree.ai/api-reference/appconfigs/get-appconfigs
get /appconfigs
# Get all application qps quotas
Source: https://docs.startree.ai/api-reference/application/get-all-application-qps-quotas
get /applicationQuotas
Get all application qps quotas
# Get application qps quota
Source: https://docs.startree.ai/api-reference/application/get-application-qps-quota
get /applicationQuotas/{appName}
Get application qps quota
# Update application quota
Source: https://docs.startree.ai/api-reference/application/update-application-quota
post /applicationQuotas/{appName}
Update application quota
# Delete the minion task metadata
Source: https://docs.startree.ai/api-reference/atomicingestion/delete-the-minion-task-metadata
post /segments/{tableName}/deleteMinionTaskMetadata
# Get Segment to rows factor from customMap of progress metadata
Source: https://docs.startree.ai/api-reference/atomicingestion/get-segment-to-rows-factor-from-custommap-of-progress-metadata
post /segments/{tableName}/getSegmentToRowsFactor
# Mark the end of data ingestion to upload multiple segments
Source: https://docs.startree.ai/api-reference/atomicingestion/mark-the-end-of-data-ingestion-to-upload-multiple-segments
post /segments/{tableName}/endDataIngestRequest
# Mark the start of data ingestion to upload multiple segments
Source: https://docs.startree.ai/api-reference/atomicingestion/mark-the-start-of-data-ingestion-to-upload-multiple-segments
post /segments/{tableName}/startDataIngestRequest
# Check whether authentication is enabled
Source: https://docs.startree.ai/api-reference/auth/check-whether-authentication-is-enabled
get /auth/verify
# Retrieve auth workflow info
Source: https://docs.startree.ai/api-reference/auth/retrieve-auth-workflow-info
get /auth/info
# Get the list of servers to restart in sequence
Source: https://docs.startree.ai/api-reference/batchrestart/get-the-list-of-servers-to-restart-in-sequence
get /serverBatchRestartSequence
# Enable/disable the query rate limiting for a broker instance
Source: https://docs.startree.ai/api-reference/broker/enabledisable-the-query-rate-limiting-for-a-broker-instance
post /brokers/instances/{instanceName}/qps
Enable/disable the query rate limiting for a broker instance
# List brokers for a given table
Source: https://docs.startree.ai/api-reference/broker/list-brokers-for-a-given-table
get /brokers/tables/{tableName}
List brokers for a given table
# List brokers for a given table
Source: https://docs.startree.ai/api-reference/broker/list-brokers-for-a-given-table-1
get /v2/brokers/tables/{tableName}
List brokers for a given table
# List brokers for a given tenant
Source: https://docs.startree.ai/api-reference/broker/list-brokers-for-a-given-tenant
get /brokers/tenants/{tenantName}
List brokers for a given tenant
# List brokers for a given tenant
Source: https://docs.startree.ai/api-reference/broker/list-brokers-for-a-given-tenant-1
get /v2/brokers/tenants/{tenantName}
List brokers for a given tenant
# List tables to brokers mappings
Source: https://docs.startree.ai/api-reference/broker/list-tables-to-brokers-mappings
get /brokers/tables
List tables to brokers mappings
# List tables to brokers mappings
Source: https://docs.startree.ai/api-reference/broker/list-tables-to-brokers-mappings-1
get /v2/brokers/tables
List tables to brokers mappings
# List tenants and tables to brokers mappings
Source: https://docs.startree.ai/api-reference/broker/list-tenants-and-tables-to-brokers-mappings
get /brokers
List tenants and tables to brokers mappings
# List tenants and tables to brokers mappings
Source: https://docs.startree.ai/api-reference/broker/list-tenants-and-tables-to-brokers-mappings-1
get /v2/brokers
List tenants and tables to brokers mappings
# List tenants to brokers mappings
Source: https://docs.startree.ai/api-reference/broker/list-tenants-to-brokers-mappings
get /brokers/tenants
List tenants to brokers mappings
# List tenants to brokers mappings
Source: https://docs.startree.ai/api-reference/broker/list-tenants-to-brokers-mappings-1
get /v2/brokers/tenants
List tenants to brokers mappings
# Delete cluster configuration
Source: https://docs.startree.ai/api-reference/cluster/delete-cluster-configuration
delete /cluster/configs/{configName}
# Get cluster Info
Source: https://docs.startree.ai/api-reference/cluster/get-cluster-info
get /cluster/info
Get cluster Info
# Get debug information for segment.
Source: https://docs.startree.ai/api-reference/cluster/get-debug-information-for-segment
get /debug/segments/{tableName}/{segmentName}
Debug information for segment.
# Get debug information for table.
Source: https://docs.startree.ai/api-reference/cluster/get-debug-information-for-table
get /debug/tables/{tableName}
Debug information for table.
# Get the configuration for Groovy Static analysis
Source: https://docs.startree.ai/api-reference/cluster/get-the-configuration-for-groovy-static-analysis
get /cluster/configs/groovy/staticAnalyzerConfig
Get the configuration for Groovy static analysis
# Get the default configuration for Groovy Static analysis
Source: https://docs.startree.ai/api-reference/cluster/get-the-default-configuration-for-groovy-static-analysis
get /cluster/configs/groovy/staticAnalyzerConfig/default
Get the default configuration for Groovy static analysis
# List cluster configurations
Source: https://docs.startree.ai/api-reference/cluster/list-cluster-configurations
get /cluster/configs
List cluster level configurations
# Update cluster configuration
Source: https://docs.startree.ai/api-reference/cluster/update-cluster-configuration
post /cluster/configs
# Update Groovy static analysis configuration
Source: https://docs.startree.ai/api-reference/cluster/update-groovy-static-analysis-configuration
post /cluster/configs/groovy/staticAnalyzerConfig
# Enable / disable the periodic cluster health check task. Note that this setting isn't persisted across controller restarts and /cluster/configs should be used to disable it permanently
Source: https://docs.startree.ai/api-reference/clusterhealth/enable-disable-the-periodic-cluster-health-check-task-note-that-this-setting-isnt-persisted-across-controller-restarts-and-clusterconfigs-should-be-used-to-disable-it-permanently
put /clusterHealth
# Get all available cluster health checks and their details
Source: https://docs.startree.ai/api-reference/clusterhealth/get-all-available-cluster-health-checks-and-their-details
get /clusterHealth/list
# Get cached cluster health details for all pinot entities
Source: https://docs.startree.ai/api-reference/clusterhealth/get-cached-cluster-health-details-for-all-pinot-entities
get /clusterHealth
# Get cluster properties if deployed in AZ aware mode
Source: https://docs.startree.ai/api-reference/clusterhealth/get-cluster-properties-if-deployed-in-az-aware-mode
get /azAwareCluster
# browse
Source: https://docs.startree.ai/api-reference/connection/browse
post /connections/browse
Browse resources in a connection. Eg list directories/files in a s3 bucket, list tables in a database, list topics in a Kafka cluster etc.
# Update consistentPush state
Source: https://docs.startree.ai/api-reference/consistentpush/update-consistentpush-state
post /segments/{tableNameWithType}/consistentPushRequest
# Delete all tables in given database name
Source: https://docs.startree.ai/api-reference/database/delete-all-tables-in-given-database-name
delete /databases/{databaseName}
Delete all tables in given database name
# Get database quota configs
Source: https://docs.startree.ai/api-reference/database/get-database-quota-configs
get /databases/{databaseName}/quotas
Get database quota configs
# List all database names
Source: https://docs.startree.ai/api-reference/database/list-all-database-names
get /databases
Lists all database names
# Update database quotas
Source: https://docs.startree.ai/api-reference/database/update-database-quotas
post /databases/{databaseName}/quotas
Update database quotas
# Query Data
Source: https://docs.startree.ai/api-reference/dataplane/query
POST /query/sql
Run SQL queries on StarTree Cloud's real-time analytics engine.
# Add metadata location of a new dedup snapshot
Source: https://docs.startree.ai/api-reference/dedupsnapshot/add-metadata-location-of-a-new-dedup-snapshot
post /dedupSnapshots/{tableNameWithType}/{snapshotName}/location
# Add metadata of a new dedup snapshot
Source: https://docs.startree.ai/api-reference/dedupsnapshot/add-metadata-of-a-new-dedup-snapshot
post /dedupSnapshots/{tableNameWithType}
# Delete dedup snapshot and its metadata
Source: https://docs.startree.ai/api-reference/dedupsnapshot/delete-dedup-snapshot-and-its-metadata
delete /dedupSnapshots/{tableNameWithType}/snapshotAndMetadata
# Get all dedup snapshot names for the table
Source: https://docs.startree.ai/api-reference/dedupsnapshot/get-all-dedup-snapshot-names-for-the-table
get /dedupSnapshots/{tableNameWithType}/names
# Get dedup snapshot metadata by the snapshot name
Source: https://docs.startree.ai/api-reference/dedupsnapshot/get-dedup-snapshot-metadata-by-the-snapshot-name
get /dedupSnapshots/{tableNameWithType}/{snapshotName}/metadata
# Get latest dedup snapshot names for all partitions
Source: https://docs.startree.ai/api-reference/dedupsnapshot/get-latest-dedup-snapshot-names-for-all-partitions
get /dedupSnapshots/{tableNameWithType}/latest
# Check controller health
Source: https://docs.startree.ai/api-reference/health/check-controller-health
get /pinot-controller/admin
# Check controller health
Source: https://docs.startree.ai/api-reference/health/check-controller-health-1
get /health
# Get controller start time
Source: https://docs.startree.ai/api-reference/health/get-controller-start-time
get /start-time
# Get controller uptime
Source: https://docs.startree.ai/api-reference/health/get-controller-uptime
get /uptime
# Check if it's safe to drop the given instances. If not list all the reasons why its not safe.
Source: https://docs.startree.ai/api-reference/instance/check-if-its-safe-to-drop-the-given-instances-if-not-list-all-the-reasons-why-its-not-safe
get /instances/dropInstance/validate
# Check if it's safe to update the tags of the given instances. If not list all the reasons.
Source: https://docs.startree.ai/api-reference/instance/check-if-its-safe-to-update-the-tags-of-the-given-instances-if-not-list-all-the-reasons
post /instances/updateTags/validate
# Create a new instance
Source: https://docs.startree.ai/api-reference/instance/create-a-new-instance
post /instances
Creates a new instance with given instance config
# Drop an instance
Source: https://docs.startree.ai/api-reference/instance/drop-an-instance
delete /instances/{instanceName}
Drop an instance
# Enable/disable an instance
Source: https://docs.startree.ai/api-reference/instance/enabledisable-an-instance
put /instances/{instanceName}/state
Enable/disable an instance
# Enable/disable/drop an instance
Source: https://docs.startree.ai/api-reference/instance/enabledisabledrop-an-instance
post /instances/{instanceName}/state
Enable/disable/drop an instance
# Get instance information
Source: https://docs.startree.ai/api-reference/instance/get-instance-information
get /instances/{instanceName}
# List all instances
Source: https://docs.startree.ai/api-reference/instance/list-all-instances
get /instances
# List all live instances
Source: https://docs.startree.ai/api-reference/instance/list-all-live-instances
get /liveinstances
# Update the specified instance
Source: https://docs.startree.ai/api-reference/instance/update-the-specified-instance
put /instances/{instanceName}
Update specified instance with given instance config
# Update the tables served by the specified broker instance in the broker resource
Source: https://docs.startree.ai/api-reference/instance/update-the-tables-served-by-the-specified-broker-instance-in-the-broker-resource
post /instances/{instanceName}/updateBrokerResource
Broker resource should be updated when a new broker instance is added, or the tags for an existing broker are changed. Updating broker resource requires reading all the table configs, which can be costly for large cluster. Consider updating broker resource for each table individually.
# Update the tags of the specified instance
Source: https://docs.startree.ai/api-reference/instance/update-the-tags-of-the-specified-instance
put /instances/{instanceName}/updateTags
Update the tags of the specified instance
# Introduction
Source: https://docs.startree.ai/api-reference/introduction
Learn about StarTree Cloud APIs for managing tables and querying data at scale.
# StarTree Cloud APIs
StarTree Cloud provides **powerful APIs**, enabling users to **manage tables, schemas, and query data efficiently at scale**.
Create and manage tables, schemas, and indexes with programmatic control.
Query real-time data at scale with low latency and high concurrency.
***
## **Authentication & Prerequisites**
To interact with StarTree Cloud APIs, follow these steps:
Access the [StarTree Cloud Console](https://startree.ai/saas-signup) and log in to your account.
Navigate to the **API Access** section in the console and generate your **Bearer Token**.\
This token must be included in every API request for authentication.
The **Workspace ID** can be found in the **top navigation bar** of the console.\
This ID must be passed in the `database` HTTP header for all API calls.
Workspace ID is required **only for multi-tenant clusters** (e.g., Free Tier).\
It is **not needed for dedicated clusters** (Standard or Premium plans).
Explore the API reference in the following pages to learn how to use the Bearer Token and Workspace ID in API requests.
# Gets leader for a given table
Source: https://docs.startree.ai/api-reference/leader/gets-leader-for-a-given-table
get /leader/tables/{tableName}
Gets leader for a given table
# Gets leaders for all tables
Source: https://docs.startree.ai/api-reference/leader/gets-leaders-for-all-tables
get /leader/tables
Gets leaders for all tables
# Collect log files from a given instance
Source: https://docs.startree.ai/api-reference/logger/collect-log-files-from-a-given-instance
get /loggers/instances/{instanceName}
# Collect log files from all the instances
Source: https://docs.startree.ai/api-reference/logger/collect-log-files-from-all-the-instances
get /loggers/instances
# Download a log file
Source: https://docs.startree.ai/api-reference/logger/download-a-log-file
get /loggers/download
# Download a log file from a given instance
Source: https://docs.startree.ai/api-reference/logger/download-a-log-file-from-a-given-instance
get /loggers/instances/{instanceName}/download
# Get all local log files
Source: https://docs.startree.ai/api-reference/logger/get-all-local-log-files
get /loggers/files
# Get all the loggers
Source: https://docs.startree.ai/api-reference/logger/get-all-the-loggers
get /loggers
Return all the logger names
# Get logger configs
Source: https://docs.startree.ai/api-reference/logger/get-logger-configs
get /loggers/{loggerName}
Return logger info
# Set logger level
Source: https://docs.startree.ai/api-reference/logger/set-logger-level
put /loggers/{loggerName}
Set logger level for a given logger
# Get perf advisor recommendations given a query
Source: https://docs.startree.ai/api-reference/perfadvisor/get-perf-advisor-recommendations-given-a-query
get /perfadvisor/advise
# Get comma-delimited list of all available periodic task names.
Source: https://docs.startree.ai/api-reference/periodictask/get-comma-delimited-list-of-all-available-periodic-task-names
get /periodictask/names
# Run periodic task against table. If table name is missing, task will run against all tables.
Source: https://docs.startree.ai/api-reference/periodictask/run-periodic-task-against-table-if-table-name-is-missing-task-will-run-against-all-tables
get /periodictask/run
# Cancel a query as identified by the clientQueryId
Source: https://docs.startree.ai/api-reference/query/cancel-a-query-as-identified-by-the-clientqueryid
delete /clientQuery/{brokerId}/{clientQueryId}
No effect if no query exists for the given clientQueryId on the requested broker. Query may continue to run for a short while after callingcancel as it's done in a non-blocking manner. The cancel method can be called multiple times.
# Cancel a query as identified by the clientQueryId
Source: https://docs.startree.ai/api-reference/query/cancel-a-query-as-identified-by-the-clientqueryid-1
delete /clientQuery/{clientQueryId}
No effect if no query exists forthe given clientQueryId on any broker. Query may continue to run for a short while after callingcancel as it's done in a non-blocking manner. The cancel method can be called multiple times.
# Cancel a query as identified by the queryId
Source: https://docs.startree.ai/api-reference/query/cancel-a-query-as-identified-by-the-queryid
delete /query/{brokerId}/{queryId}
No effect if no query exists for the given queryId on the requested broker. Query may continue to run for a short while after calling cancel as it's done in a non-blocking manner. The cancel method can be called multiple times.
# Get running queries from all brokers
Source: https://docs.startree.ai/api-reference/query/get-running-queries-from-all-brokers
get /queries
The queries are returned with brokers running them
# Update rate limiter configs
Source: https://docs.startree.ai/api-reference/ratelimiter/update-rate-limiter-configs
post /upsertRateLimitConfigs
Update all the rate limiter configs
# Cancel a restream operation
Source: https://docs.startree.ai/api-reference/restream/cancel-a-restream-operation
delete /tables/{tableNameWithType}/restream
Cancel a restream operation
# DEBUG ONLY: Process the next stage of the restream workflow.
Source: https://docs.startree.ai/api-reference/restream/debug-only:-process-the-next-stage-of-the-restream-workflow
put /tables/{tableNameWithType}/restream
DEBUG ONLY: Process the next stage of the restream workflow.
# Gets detailed stats of a restream operation
Source: https://docs.startree.ai/api-reference/restream/gets-detailed-stats-of-a-restream-operation
get /tables/{tableNameWithType}/restream
Gets detailed stats of a restream operation
# Restream data from a stream to a new table. Then replace the table
Source: https://docs.startree.ai/api-reference/restream/restream-data-from-a-stream-to-a-new-table-then-replace-the-table
post /tables/{tableName}/restream
Restream a table
# Add a new schema
Source: https://docs.startree.ai/api-reference/schema/add-a-new-schema
post /schemas
Adds a new schema
# Delete a schema
Source: https://docs.startree.ai/api-reference/schema/delete-a-schema
delete /schemas/{schemaName}
Deletes a schema by name
# Get a schema
Source: https://docs.startree.ai/api-reference/schema/get-a-schema
get /schemas/{schemaName}
Gets a schema by name
# Get fieldSpec metadata
Source: https://docs.startree.ai/api-reference/schema/get-fieldspec-metadata
get /schemas/fieldSpec
Get fieldSpec metadata
# Get table schema
Source: https://docs.startree.ai/api-reference/schema/get-table-schema
get /tables/{tableName}/schema
Read table schema
# List all schema names
Source: https://docs.startree.ai/api-reference/schema/list-all-schema-names
get /schemas
Lists all schema names
# List all schemas info with count of field specs
Source: https://docs.startree.ai/api-reference/schema/list-all-schemas-info-with-count-of-field-specs
get /schemas/info
Lists all schemas with field count details
# Update a schema
Source: https://docs.startree.ai/api-reference/schema/update-a-schema
put /schemas/{schemaName}
Updates a schema
# Validate schema
Source: https://docs.startree.ai/api-reference/schema/validate-schema
post /schemas/validate
This API returns the schema that matches the one you get from 'GET /schema/{schemaName}'. This allows us to validate schema before apply.
# Delete a segment
Source: https://docs.startree.ai/api-reference/segment/delete-a-segment
delete /segments/{tableName}/{segmentName}
Delete a segment
# Delete selected segments. An optional 'excludeReplacedSegments' parameter is used to get the list of segments which has not yet been replaced (determined by segment lineage entries) and can be queried from the table. The value is false by default.
Source: https://docs.startree.ai/api-reference/segment/delete-selected-segments-an-optional-excludereplacedsegments-parameter-is-used-to-get-the-list-of-segments-which-has-not-yet-been-replaced-determined-by-segment-lineage-entries-and-can-be-queried-from-the-table-the-value-is-false-by-default
delete /segments/{tableName}/choose
List all segments
# Delete the list of segments provided in the queryParam else all segments
Source: https://docs.startree.ai/api-reference/segment/delete-the-list-of-segments-provided-in-the-queryparam-else-all-segments
delete /segments/{tableName}
Delete the list of segments provided in the queryParam else all segments
# Delete the segments in the JSON array payload
Source: https://docs.startree.ai/api-reference/segment/delete-the-segments-in-the-json-array-payload
post /segments/{tableName}/delete
Delete the segments in the JSON array payload
# Download a segment
Source: https://docs.startree.ai/api-reference/segment/download-a-segment
get /segments/{tableName}/{segmentName}
Download a segment
# End to replace segments
Source: https://docs.startree.ai/api-reference/segment/end-to-replace-segments
post /segments/{tableName}/endReplaceSegments
End to replace segments
# Get a map from segment to CRC of the segment (only apply to OFFLINE table)
Source: https://docs.startree.ai/api-reference/segment/get-a-map-from-segment-to-crc-of-the-segment-only-apply-to-offline-table
get /segments/{tableName}/crc
Get a map from segment to CRC of the segment (only apply to OFFLINE table)
# Get a map from server to segments hosted by the server
Source: https://docs.startree.ai/api-reference/segment/get-a-map-from-server-to-segments-hosted-by-the-server
get /segments/{tableName}/servers
Get a map from server to segments hosted by the server
# Get selected segments timestamps
Source: https://docs.startree.ai/api-reference/segment/get-selected-segments-timestamps
get /segments/{tableName}/select
Get the selected segments given the (inclusive) start and (exclusive) end timestamps in milliseconds. These timestamps will be compared against minmax values of the time column in each segment. If the table is a refresh use case, the value of start and end timestamp is voided, since there is no time column for refresh use case; instead, the whole qualified segments will be returned. If no timestamps are provided, all the qualified segments will be returned. For the segments that partially belong to the time range, the boolean flag 'excludeOverlapping' is introduced in order for user to determine whether to exclude this kind of segments in the response.
# Get status for a submitted reload operation
Source: https://docs.startree.ai/api-reference/segment/get-status-for-a-submitted-reload-operation
get /segments/segmentReloadStatus/{jobId}
Get status for a submitted reload operation
# Get storage tier for all segments in the given table
Source: https://docs.startree.ai/api-reference/segment/get-storage-tier-for-all-segments-in-the-given-table
get /segments/{tableName}/tiers
Get storage tier for all segments in the given table
# Get storage tiers for the given segment
Source: https://docs.startree.ai/api-reference/segment/get-storage-tiers-for-the-given-segment
get /segments/{tableName}/{segmentName}/tiers
Get storage tiers for the given segment
# Get the metadata for a segment
Source: https://docs.startree.ai/api-reference/segment/get-the-metadata-for-a-segment
get /segments/{tableName}/{segmentName}/metadata
Get the metadata for a segment
# Get the server metadata for all table segments
Source: https://docs.startree.ai/api-reference/segment/get-the-server-metadata-for-all-table-segments
get /segments/{tableName}/metadata
Get the server metadata for all table segments
# Get the zookeeper metadata for all table segments
Source: https://docs.startree.ai/api-reference/segment/get-the-zookeeper-metadata-for-all-table-segments
get /segments/{tableName}/zkmetadata
Get the zookeeper metadata for all table segments
# Gets a list of segments that are stale from servers hosting the table
Source: https://docs.startree.ai/api-reference/segment/gets-a-list-of-segments-that-are-stale-from-servers-hosting-the-table
get /segments/{tableNameWithType}/isStale
Gets a list of segments that are stale from servers hosting the table
# Gets the metadata of reload segments check from servers hosting the table
Source: https://docs.startree.ai/api-reference/segment/gets-the-metadata-of-reload-segments-check-from-servers-hosting-the-table
get /segments/{tableNameWithType}/needReload
Returns true if reload is needed on the table from any one of the servers
# List all segments. An optional 'excludeReplacedSegments' parameter is used to get the list of segments which has not yet been replaced (determined by segment lineage entries) and can be queried from the table. The value is false by default.
Source: https://docs.startree.ai/api-reference/segment/list-all-segments-an-optional-excludereplacedsegments-parameter-is-used-to-get-the-list-of-segments-which-has-not-yet-been-replaced-determined-by-segment-lineage-entries-and-can-be-queried-from-the-table-the-value-is-false-by-default
get /segments/{tableName}
List all segments
# List segment lineage
Source: https://docs.startree.ai/api-reference/segment/list-segment-lineage
get /segments/{tableName}/lineage
List segment lineage in chronologically sorted order
# Reingest a realtime segment
Source: https://docs.startree.ai/api-reference/segment/reingest-a-realtime-segment
post /segments/reingested
Reingest a segment as multipart file
# Reload a segment
Source: https://docs.startree.ai/api-reference/segment/reload-a-segment
post /segments/{tableName}/{segmentName}/reload
Reload a segment
# Reload all segments
Source: https://docs.startree.ai/api-reference/segment/reload-all-segments
post /segments/{tableName}/reload
Reload all segments
# Resets a segment by first disabling it, waiting for external view to stabilize, and finally enabling it again
Source: https://docs.startree.ai/api-reference/segment/resets-a-segment-by-first-disabling-it-waiting-for-external-view-to-stabilize-and-finally-enabling-it-again
post /segments/{tableNameWithType}/{segmentName}/reset
Resets a segment by disabling and then enabling it
# Resets all segments (when errorSegmentsOnly = false) or segments with Error state (when errorSegmentsOnly = true) of the table, by first disabling them, waiting for external view to stabilize, and finally enabling them
Source: https://docs.startree.ai/api-reference/segment/resets-all-segments-when-errorsegmentsonly-=-false-or-segments-with-error-state-when-errorsegmentsonly-=-true-of-the-table-by-first-disabling-them-waiting-for-external-view-to-stabilize-and-finally-enabling-them
post /segments/{tableNameWithType}/reset
Resets segments by disabling and then enabling them
# Revert segments replacement
Source: https://docs.startree.ai/api-reference/segment/revert-segments-replacement
post /segments/{tableName}/revertReplaceSegments
Revert segments replacement
# Start to replace segments
Source: https://docs.startree.ai/api-reference/segment/start-to-replace-segments
post /segments/{tableName}/startReplaceSegments
Start to replace segments
# Update the start and end time of the segments based on latest schema
Source: https://docs.startree.ai/api-reference/segment/update-the-start-and-end-time-of-the-segments-based-on-latest-schema
post /segments/{tableNameWithType}/updateZKTimeInterval
Update the start and end time of the segments based on latest schema
# Upload a batch of segments
Source: https://docs.startree.ai/api-reference/segment/upload-a-batch-of-segments
post /segments/batchUpload
Upload a batch of segments with METADATA upload type
# Upload a segment
Source: https://docs.startree.ai/api-reference/segment/upload-a-segment
post /segments
Upload a segment as binary
# Upload a segment
Source: https://docs.startree.ai/api-reference/segment/upload-a-segment-1
post /v2/segments
Upload a segment as binary
# Add the TableConfigs using the tableConfigsStr json
Source: https://docs.startree.ai/api-reference/table/add-the-tableconfigs-using-the-tableconfigsstr-json
post /tableConfigs
Add the TableConfigs using the tableConfigsStr json
# Adds a table
Source: https://docs.startree.ai/api-reference/table/adds-a-table
post /tables
Adds a table
# Assign server instances to a table
Source: https://docs.startree.ai/api-reference/table/assign-server-instances-to-a-table
post /tables/{tableName}/assignInstances
# Cancel all rebalance jobs for the given table, and noop if no rebalance is running
Source: https://docs.startree.ai/api-reference/table/cancel-all-rebalance-jobs-for-the-given-table-and-noop-if-no-rebalance-is-running
delete /tables/{tableName}/rebalance
Cancel all rebalance jobs for the given table, and noop if no rebalance is running
# Create/update the instance partitions
Source: https://docs.startree.ai/api-reference/table/createupdate-the-instance-partitions
put /tables/{tableName}/instancePartitions
# Delete hybrid table query time boundary
Source: https://docs.startree.ai/api-reference/table/delete-hybrid-table-query-time-boundary
delete /tables/{tableName}/timeBoundary
Delete hybrid table query time boundary
# Delete tables partitions
Source: https://docs.startree.ai/api-reference/table/delete-tables-partitions
delete /tables/{tableNameWithType}/partitions
# Delete the TableConfigs
Source: https://docs.startree.ai/api-reference/table/delete-the-tableconfigs
delete /tableConfigs/{tableName}
Delete the TableConfigs
# Deletes a table
Source: https://docs.startree.ai/api-reference/table/deletes-a-table
delete /tables/{tableName}
Deletes a table
# Enable/disable a table
Source: https://docs.startree.ai/api-reference/table/enabledisable-a-table
put /tables/{tableName}/state
Enable/disable a table
# Force commit the current consuming segments
Source: https://docs.startree.ai/api-reference/table/force-commit-the-current-consuming-segments
post /tables/{tableName}/forceCommit
Force commit the current segments in consuming state and restart consumption. This should be used after schema/table config changes. Please note that this is an asynchronous operation, and 200 response does not mean it has actually been done already.If specific partitions or consuming segments are provided, only those partitions or consuming segments will be force committed.
# Get current table state
Source: https://docs.startree.ai/api-reference/table/get-current-table-state
get /tables/{tableName}/state
Get current table state
# Get list of controller jobs for this table
Source: https://docs.startree.ai/api-reference/table/get-list-of-controller-jobs-for-this-table
get /table/{tableName}/jobs
Get list of controller jobs for this table
# Get segment names to segment status map
Source: https://docs.startree.ai/api-reference/table/get-segment-names-to-segment-status-map
get /tables/{tableName}/segmentsStatus
Get segment statuses of each segment
# Get status for a submitted force commit operation
Source: https://docs.startree.ai/api-reference/table/get-status-for-a-submitted-force-commit-operation
get /tables/forceCommitStatus/{jobId}
Get status for a submitted force commit operation
# Get table external view
Source: https://docs.startree.ai/api-reference/table/get-table-external-view
get /tables/{tableName}/externalview
Get table external view
# Get table ideal state
Source: https://docs.startree.ai/api-reference/table/get-table-ideal-state
get /tables/{tableName}/idealstate
Get table ideal state
# Get the aggregate index details of all segments for a table
Source: https://docs.startree.ai/api-reference/table/get-the-aggregate-index-details-of-all-segments-for-a-table
get /tables/{tableName}/indexes
Get the aggregate index details of all segments for a table
# Get the aggregate metadata of all segments for a table
Source: https://docs.startree.ai/api-reference/table/get-the-aggregate-metadata-of-all-segments-for-a-table
get /tables/{tableName}/metadata
Get the aggregate metadata of all segments for a table
# Get the aggregate validDocIds metadata of all segments for a table
Source: https://docs.startree.ai/api-reference/table/get-the-aggregate-validdocids-metadata-of-all-segments-for-a-table
get /tables/{tableName}/validDocIdsMetadata
Get the aggregate validDocIds metadata of all segments for a table
# Get the instance partitions
Source: https://docs.startree.ai/api-reference/table/get-the-instance-partitions
get /tables/{tableName}/instancePartitions
# Get the TableConfigs for a given raw tableName
Source: https://docs.startree.ai/api-reference/table/get-the-tableconfigs-for-a-given-raw-tablename
get /tableConfigs/{tableName}
Get the TableConfigs for a given raw tableName
# Gets detailed stats of a rebalance operation
Source: https://docs.startree.ai/api-reference/table/gets-detailed-stats-of-a-rebalance-operation
get /rebalanceStatus/{jobId}
Gets detailed stats of a rebalance operation
# Ingest a file
Source: https://docs.startree.ai/api-reference/table/ingest-a-file
post /ingestFromFile
Creates a segment using given file and pushes it to Pinot.
All steps happen on the controller. This API is NOT meant for production environments/large input files.
Example usage (query params need encoding):
```
curl -X POST -F file=@data.json -H "Content-Type: multipart/form-data" "http://localhost:9000/ingestFromFile?tableNameWithType=foo_OFFLINE&
batchConfigMapStr={
"inputFormat":"csv",
"recordReader.prop.delimiter":"|"
}"
```
# Ingest from the given URI
Source: https://docs.startree.ai/api-reference/table/ingest-from-the-given-uri
post /ingestFromURI
Creates a segment using file at the given URI and pushes it to Pinot.
All steps happen on the controller. This API is NOT meant for production environments/large input files.
Example usage (query params need encoding):
```
curl -X POST "http://localhost:9000/ingestFromURI?tableNameWithType=foo_OFFLINE
&batchConfigMapStr={
"inputFormat":"json",
"input.fs.className":"org.apache.pinot.plugin.filesystem.S3PinotFS",
"input.fs.prop.region":"us-central",
"input.fs.prop.accessKey":"foo",
"input.fs.prop.secretKey":"bar"
}
&sourceURIStr=s3://test.bucket/path/to/json/data/data.json"
```
# List table instances
Source: https://docs.startree.ai/api-reference/table/list-table-instances
get /tables/{tableName}/instances
List instances of the given table
# List tables to live brokers mappings
Source: https://docs.startree.ai/api-reference/table/list-tables-to-live-brokers-mappings
get /tables/livebrokers
List tables to live brokers mappings based on EV
# List the brokers serving a table
Source: https://docs.startree.ai/api-reference/table/list-the-brokers-serving-a-table
get /tables/{tableName}/livebrokers
List live brokers of the given table based on EV
# Lists all TableConfigs in cluster
Source: https://docs.startree.ai/api-reference/table/lists-all-tableconfigs-in-cluster
get /tableConfigs
Lists all TableConfigs in cluster
# Lists all tables in cluster
Source: https://docs.startree.ai/api-reference/table/lists-all-tables-in-cluster
get /mytables
Lists all tables in cluster
# Lists all tables in cluster
Source: https://docs.startree.ai/api-reference/table/lists-all-tables-in-cluster-1
get /tables
Lists all tables in cluster
# Lists the table configs
Source: https://docs.startree.ai/api-reference/table/lists-the-table-configs
get /tables/{tableName}
# Pause consumption of a realtime table
Source: https://docs.startree.ai/api-reference/table/pause-consumption-of-a-realtime-table
post /tables/{tableName}/pauseConsumption
Pause the consumption of a realtime table
# preview
Source: https://docs.startree.ai/api-reference/table/preview
post /tables/preview
Simulates a preview of the table given the table config and schema
# Read table sizes
Source: https://docs.startree.ai/api-reference/table/read-table-sizes
get /tables/{tableName}/size
Get table size details. Table size is the size of untarred segments including replication
# Rebalances a table (reassign instances and segments for a table)
Source: https://docs.startree.ai/api-reference/table/rebalances-a-table-reassign-instances-and-segments-for-a-table
post /tables/{tableName}/rebalance
Rebalances a table (reassign instances and segments for a table)
# Rebuild broker resource for table
Source: https://docs.startree.ai/api-reference/table/rebuild-broker-resource-for-table
post /tables/{tableName}/rebuildBrokerResourceFromHelixTags
when new brokers are added
# Recommend config
Source: https://docs.startree.ai/api-reference/table/recommend-config
put /tables/recommender
Recommend a config with input json
# Remove the instance partitions
Source: https://docs.startree.ai/api-reference/table/remove-the-instance-partitions
delete /tables/{tableName}/instancePartitions
# Replace an instance in the instance partitions
Source: https://docs.startree.ai/api-reference/table/replace-an-instance-in-the-instance-partitions
post /tables/{tableName}/replaceInstance
# Resume consumption of a realtime table
Source: https://docs.startree.ai/api-reference/table/resume-consumption-of-a-realtime-table
post /tables/{tableName}/resumeConsumption
Resume the consumption for a realtime table. ConsumeFrom parameter indicates from which offsets consumption should resume. Recommended value is 'lastConsumed', which indicates consumption should continue based on the offsets in segment ZK metadata, and in case the offsets are already gone, the first available offsets are picked to minimize the data loss.
# Return pause status of a realtime table
Source: https://docs.startree.ai/api-reference/table/return-pause-status-of-a-realtime-table
get /tables/{tableName}/pauseStatus
Return pause status of a realtime table along with list of consuming segments.
# Returns state of consuming segments
Source: https://docs.startree.ai/api-reference/table/returns-state-of-consuming-segments
get /tables/{tableName}/consumingSegmentsInfo
Gets the status of consumers from all servers.Note that the partitionToOffsetMap has been deprecated and will be removed in the next release. The info is now embedded within each partition's state as currentOffsetsMap.
# Returns state of pauseless table
Source: https://docs.startree.ai/api-reference/table/returns-state-of-pauseless-table
get /tables/{tableName}/pauselessDebugInfo
Gets the segments that are in error state and segments with COMMITTING status in ZK metadata
# Set hybrid table query time boundary based on offline segments' metadata
Source: https://docs.startree.ai/api-reference/table/set-hybrid-table-query-time-boundary-based-on-offline-segments-metadata
post /tables/{tableName}/timeBoundary
Set hybrid table query time boundary based on offline segments' metadata
# table stats
Source: https://docs.startree.ai/api-reference/table/table-stats
get /tables/{tableName}/stats
Provides metadata info/stats about the table.
# table status
Source: https://docs.startree.ai/api-reference/table/table-status
get /tables/{tableName}/status
Provides status of the table including ingestion status
# Update the TableConfigs provided by the tableConfigsStr json
Source: https://docs.startree.ai/api-reference/table/update-the-tableconfigs-provided-by-the-tableconfigsstr-json
put /tableConfigs/{tableName}
Update the TableConfigs provided by the tableConfigsStr json
# Updates table config for a table
Source: https://docs.startree.ai/api-reference/table/updates-table-config-for-a-table
put /tables/{tableName}
Updates table config for a table
# Validate table config for a table
Source: https://docs.startree.ai/api-reference/table/validate-table-config-for-a-table
post /tables/validate
This API returns the table config that matches the one you get from 'GET /tables/{tableName}'. This allows us to validate table config before apply.
# Validate the TableConfigs
Source: https://docs.startree.ai/api-reference/table/validate-the-tableconfigs
post /tableConfigs/validate
Validate the TableConfigs
# Clean up finished tasks (COMPLETED, FAILED) for the given task type
Source: https://docs.startree.ai/api-reference/task/clean-up-finished-tasks-completed-failed-for-the-given-task-type
put /tasks/{taskType}/cleanup
# Count of all tasks for the given task type
Source: https://docs.startree.ai/api-reference/task/count-of-all-tasks-for-the-given-task-type
get /tasks/{taskType}/tasks/count
# Delete a single task given its task name
Source: https://docs.startree.ai/api-reference/task/delete-a-single-task-given-its-task-name
delete /tasks/task/{taskName}
# Delete all tasks (as well as the task queue) for the given task type
Source: https://docs.startree.ai/api-reference/task/delete-all-tasks-as-well-as-the-task-queue-for-the-given-task-type
delete /tasks/{taskType}
# Delete task metadata for the given task type and table
Source: https://docs.startree.ai/api-reference/task/delete-task-metadata-for-the-given-task-type-and-table
delete /tasks/{taskType}/{tableNameWithType}/metadata
# Execute a task on minion
Source: https://docs.startree.ai/api-reference/task/execute-a-task-on-minion
post /tasks/execute
# Fetch count of sub-tasks for each of the tasks for the given task type
Source: https://docs.startree.ai/api-reference/task/fetch-count-of-sub-tasks-for-each-of-the-tasks-for-the-given-task-type
get /tasks/{taskType}/taskcounts
# Fetch cron scheduler information
Source: https://docs.startree.ai/api-reference/task/fetch-cron-scheduler-information
get /tasks/scheduler/information
# Fetch cron scheduler job keys
Source: https://docs.startree.ai/api-reference/task/fetch-cron-scheduler-job-keys
get /tasks/scheduler/jobKeys
# Fetch information for all the tasks for the given task type
Source: https://docs.startree.ai/api-reference/task/fetch-information-for-all-the-tasks-for-the-given-task-type
get /tasks/{taskType}/debug
# Fetch information for all the tasks for the given task type and table
Source: https://docs.startree.ai/api-reference/task/fetch-information-for-all-the-tasks-for-the-given-task-type-and-table
get /tasks/{taskType}/{tableNameWithType}/debug
# Fetch information for the given task name
Source: https://docs.startree.ai/api-reference/task/fetch-information-for-the-given-task-name
get /tasks/task/{taskName}/debug
# Fetch job details for table tasks
Source: https://docs.startree.ai/api-reference/task/fetch-job-details-for-table-tasks
get /tasks/scheduler/jobDetails
# Fetch task generation information for the recent runs of the given task for the given table
Source: https://docs.startree.ai/api-reference/task/fetch-task-generation-information-for-the-recent-runs-of-the-given-task-for-the-given-table
get /tasks/generator/{tableNameWithType}/{taskType}/debug
# Get a map from task to task state for the given task type
Source: https://docs.startree.ai/api-reference/task/get-a-map-from-task-to-task-state-for-the-given-task-type
get /tasks/{taskType}/taskstates
# Get progress of all subtasks with specified state tracked by minion worker in memory
Source: https://docs.startree.ai/api-reference/task/get-progress-of-all-subtasks-with-specified-state-tracked-by-minion-worker-in-memory
get /tasks/subtask/workers/progress
# Get progress of specified sub tasks for the given task tracked by minion worker in memory
Source: https://docs.startree.ai/api-reference/task/get-progress-of-specified-sub-tasks-for-the-given-task-tracked-by-minion-worker-in-memory
get /tasks/subtask/{taskName}/progress
# Get task metadata for the given task type and table
Source: https://docs.startree.ai/api-reference/task/get-task-metadata-for-the-given-task-type-and-table
get /tasks/{taskType}/{tableNameWithType}/metadata
# Get the configs of specified sub tasks for the given task
Source: https://docs.startree.ai/api-reference/task/get-the-configs-of-specified-sub-tasks-for-the-given-task
get /tasks/subtask/{taskName}/config
# Get the state (task queue state) for the given task type
Source: https://docs.startree.ai/api-reference/task/get-the-state-task-queue-state-for-the-given-task-type
get /tasks/{taskType}/state
# Get the states of all the sub tasks for the given task
Source: https://docs.startree.ai/api-reference/task/get-the-states-of-all-the-sub-tasks-for-the-given-task
get /tasks/subtask/{taskName}/state
# Get the task config (a list of child task configs) for the given task
Source: https://docs.startree.ai/api-reference/task/get-the-task-config-a-list-of-child-task-configs-for-the-given-task
get /tasks/task/{taskName}/config
# Get the task runtime config for the given task
Source: https://docs.startree.ai/api-reference/task/get-the-task-runtime-config-for-the-given-task
get /tasks/task/{taskName}/runtime/config
# Get the task state for the given task
Source: https://docs.startree.ai/api-reference/task/get-the-task-state-for-the-given-task
get /tasks/task/{taskName}/state
# List all task types
Source: https://docs.startree.ai/api-reference/task/list-all-task-types
get /tasks/tasktypes
# List all tasks for the given task type
Source: https://docs.startree.ai/api-reference/task/list-all-tasks-for-the-given-task-type
get /tasks/{taskType}/tasks
# List all tasks for the given task type
Source: https://docs.startree.ai/api-reference/task/list-all-tasks-for-the-given-task-type-1
get /tasks/{taskType}/{tableNameWithType}/state
# Resume all stopped tasks (as well as the task queue) for the given task type
Source: https://docs.startree.ai/api-reference/task/resume-all-stopped-tasks-as-well-as-the-task-queue-for-the-given-task-type
put /tasks/{taskType}/resume
# Schedule tasks and return a map from task type to task name scheduled. If task type is missing, schedules all tasks. If table name is missing, schedules tasks for all tables in the database. If database is missing in headers, uses default.
Source: https://docs.startree.ai/api-reference/task/schedule-tasks-and-return-a-map-from-task-type-to-task-name-scheduled-if-task-type-is-missing-schedules-all-tasks-if-table-name-is-missing-schedules-tasks-for-all-tables-in-the-database-if-database-is-missing-in-headers-uses-default
post /tasks/schedule
# Stop all running/pending tasks (as well as the task queue) for the given task type
Source: https://docs.startree.ai/api-reference/task/stop-all-runningpending-tasks-as-well-as-the-task-queue-for-the-given-task-type
put /tasks/{taskType}/stop
# Change tenant state
Source: https://docs.startree.ai/api-reference/tenant/change-tenant-state
post /tenants/{tenantName}/metadata
# Create a tenant
Source: https://docs.startree.ai/api-reference/tenant/create-a-tenant
post /tenants
# Delete a tenant
Source: https://docs.startree.ai/api-reference/tenant/delete-a-tenant
delete /tenants/{tenantName}
# enable/disable a tenant
Source: https://docs.startree.ai/api-reference/tenant/enabledisable-a-tenant
post /tenants/{tenantName}
# Get tenant information
Source: https://docs.startree.ai/api-reference/tenant/get-tenant-information
get /tenants/{tenantName}/metadata
# Get the instance partitions of a tenant
Source: https://docs.startree.ai/api-reference/tenant/get-the-instance-partitions-of-a-tenant
get /tenants/{tenantName}/instancePartitions
# Gets detailed stats of a tenant rebalance operation
Source: https://docs.startree.ai/api-reference/tenant/gets-detailed-stats-of-a-tenant-rebalance-operation
get /tenants/rebalanceStatus/{jobId}
Gets detailed stats of a tenant rebalance operation
# List all tenants
Source: https://docs.startree.ai/api-reference/tenant/list-all-tenants
get /tenants
# List instance for a tenant
Source: https://docs.startree.ai/api-reference/tenant/list-instance-for-a-tenant
get /tenants/{tenantName}
# List tables on a server or broker tenant
Source: https://docs.startree.ai/api-reference/tenant/list-tables-on-a-server-or-broker-tenant
get /tenants/{tenantName}/tables
# Rebalances all the tables that are part of the tenant
Source: https://docs.startree.ai/api-reference/tenant/rebalances-all-the-tables-that-are-part-of-the-tenant
post /tenants/{tenantName}/rebalance
# Update a tenant
Source: https://docs.startree.ai/api-reference/tenant/update-a-tenant
put /tenants
# Update an instance partition for a server type in a tenant
Source: https://docs.startree.ai/api-reference/tenant/update-an-instance-partition-for-a-server-type-in-a-tenant
put /tenants/{tenantName}/instancePartitions
# Apply specific tuner to a table
Source: https://docs.startree.ai/api-reference/tuner/apply-specific-tuner-to-a-table
post /tuner/{tableName}
# Apply tuner(s) to a table
Source: https://docs.startree.ai/api-reference/tuner/apply-tuners-to-a-table
get /tuner/{tableName}
# Estimate memory usage for an upsert table
Source: https://docs.startree.ai/api-reference/upsert/estimate-memory-usage-for-an-upsert-table
post /upsert/estimateHeapUsage
This API returns the estimated heap usage based on primary key column stats. This allows us to estimate table size before onboarding.
# Add metadata location of a new upsert snapshot
Source: https://docs.startree.ai/api-reference/upsertsnapshot/add-metadata-location-of-a-new-upsert-snapshot
post /upsertSnapshots/{tableNameWithType}/{snapshotName}/location
# Add metadata of a new upsert snapshot
Source: https://docs.startree.ai/api-reference/upsertsnapshot/add-metadata-of-a-new-upsert-snapshot
post /upsertSnapshots/{tableNameWithType}
# Delete upsert snapshot and its metadata
Source: https://docs.startree.ai/api-reference/upsertsnapshot/delete-upsert-snapshot-and-its-metadata
delete /upsertSnapshots/{tableNameWithType}/snapshotAndMetadata
# Delete upsert snapshot metadata
Source: https://docs.startree.ai/api-reference/upsertsnapshot/delete-upsert-snapshot-metadata
delete /upsertSnapshots/{tableNameWithType}/metadata
# Get all upsert snapshot names for the table
Source: https://docs.startree.ai/api-reference/upsertsnapshot/get-all-upsert-snapshot-names-for-the-table
get /upsertSnapshots/{tableNameWithType}/names
# Get latest upsert snapshot names for all partitions
Source: https://docs.startree.ai/api-reference/upsertsnapshot/get-latest-upsert-snapshot-names-for-all-partitions
get /upsertSnapshots/{tableNameWithType}/latest
# Get upsert snapshot metadata by the snapshot name
Source: https://docs.startree.ai/api-reference/upsertsnapshot/get-upsert-snapshot-metadata-by-the-snapshot-name
get /upsertSnapshots/{tableNameWithType}/{snapshotName}/metadata
# Add a user
Source: https://docs.startree.ai/api-reference/user/add-a-user
post /users
Add a user
# Delete a user
Source: https://docs.startree.ai/api-reference/user/delete-a-user
delete /users/{username}
Delete a user
# Get an user in cluster
Source: https://docs.startree.ai/api-reference/user/get-an-user-in-cluster
get /users/{username}
Get an user in cluster
# List all uses in cluster
Source: https://docs.startree.ai/api-reference/user/list-all-uses-in-cluster
get /users
List all users in cluster
# Update user config for a user
Source: https://docs.startree.ai/api-reference/user/update-user-config-for-a-user
put /users/{username}
Update user config for user
# Get version number of Pinot components
Source: https://docs.startree.ai/api-reference/version/get-version-number-of-pinot-components
get /version
# Create a node at a given path
Source: https://docs.startree.ai/api-reference/zookeeper/create-a-node-at-a-given-path
post /zk/create
# Delete the znode at this path
Source: https://docs.startree.ai/api-reference/zookeeper/delete-the-znode-at-this-path
delete /zk/delete
# Get all child znodes
Source: https://docs.startree.ai/api-reference/zookeeper/get-all-child-znodes
get /zk/getChildren
# Get content of the znode
Source: https://docs.startree.ai/api-reference/zookeeper/get-content-of-the-znode
get /zk/get
# Get the stat
Source: https://docs.startree.ai/api-reference/zookeeper/get-the-stat
get /zk/stat
Use this api to fetch additional details of a znode such as creation time, modified time, numChildren etc
# List the child znodes
Source: https://docs.startree.ai/api-reference/zookeeper/list-the-child-znodes
get /zk/ls
# List the child znodes along with Stats
Source: https://docs.startree.ai/api-reference/zookeeper/list-the-child-znodes-along-with-stats
get /zk/lsl
# Update the content of multiple znRecord node under the same path
Source: https://docs.startree.ai/api-reference/zookeeper/update-the-content-of-multiple-znrecord-node-under-the-same-path
put /zk/putChildren
# Update the content of the node
Source: https://docs.startree.ai/api-reference/zookeeper/update-the-content-of-the-node
put /zk/put
# Overview
Source: https://docs.startree.ai/concepts/about-startree-cloud
StarTree Cloud delivers real-time analytics at scale and is supported on all major cloud platforms.
StarTree Cloud is a fully managed analytics-as-a-service platform built on Apache Pinot™, designed to deliver real-time, high-concurrency analytics at scale. Supported on on all major cloud platforms, including AWS, GCP, and Azure, it empowers organizations to unlock the value of their data by enabling ultra-fast queries on fresh data with minimal infrastructure overhead. With built-in connectors, intuitive UI, and enterprise-grade features, StarTree Cloud makes it easy for teams to ingest, manage, and visualize operational insights — whether it’s powering user-facing analytics, anomaly detection, or internal dashboards. Businesses choose StarTree Cloud to turn their data into action, instantly.
StarTree Cloud offers public SaaS (software-as-a-service) and private BYOC (bring your own cloud) deployment options. For more information about our public or private SaaS offering, [schedule a demo](https://startree.ai/demo) with our StarTree solutions architect team.
## Why use StarTree Cloud?
We recommend using StarTree Cloud for the following use cases:
* User facing analytics
* Business metrics
* Anomaly detection
* Root cause analysis
* Dashboards
* Log analytics
* Cohort analytics
* Ad hoc exploration
| | **Query throughput (queries/sec)** | **Query latency (p95th)** | **Consistency/accuracy** | **Query flexibility** |
| - | - | - | - | - |
| **User facing analytics** | Very high: 10k-100k | 10ms-100ms | Best effort | Low |
| **Personalization** | Very high: 10k-100k | 10ms-100ms | Best effort | Low |
| **Metrics** | High: 100s-10k | 10ms-100ms | Accurate | Low |
| **Anomaly detection** | Moderate: 100s-1k | 10ms-100ms | Best effort | Low |
| **Root cause analysis** | High: 100s-10k | 10ms-100ms | Best effort | Low |
| **Visualization/dashboarding** | Moderate: 100s-1k | 100ms | Best effort | Medium |
| **Ad hoc analytics** | Low | Single-digit seconds | Best effort | High |
| **Log analytics/text search** | Moderate: 100s-1k | Sub-second | Best effort | Medium |
# What is Real-Time Analytics?
Source: https://docs.startree.ai/concepts/what-is-real-time-analytics
Real-time analytics enable users to make better and more timely decisions. Real-time analytics products pull data in as soon as it happens, pull data out as soon as it gets pulled in, and do so at scale.
Many modern software or web applications offer analytics capabilities to their users. But what does it mean for those analytics to be real-time?
*Tim Berglund explains Real-Time Analytics*
## Types of analytics
There are, broadly speaking, three types of analytics:
#### Dashboards and BI Tools
Normally used for internal purposes. BI Analysts are often the ones that are assessing this data.
#### User-Facing Analytics
Analytics that you provide to the end users of your software or web applications.
#### Machine Learning
Also referred to as machine-powered, or machine-fed analytics. These are when you feed analytics or events directly into your systems and then have your systems do the processing, automatically. Anomaly detection and fraud detection fit into this category, or any time a machine is generating insights without human interaction.
## Why is fresh data important?
When you're working with very large data sets that are constantly changing, you need to make sure the analytics that you have are based on the most recent data. Data gets outdated quickly and not having the freshest data can set you back in each of the type of analytics defined above.
* Dashboards need to be able to monitor the constant changes.
* You can build more interactive applications on top of this fresh data with actionable triggers for your end users. Organizations like LinkedIn provide their end users with real-time data through features like \_Who viewed my profile \_where a user can message other users upon seeing that they viewed their profile. Another example is in LinkedIn's news feed where a user can react, comment, or like a post as soon as they see it.
* For machine learning analytics you want your systems to know right away when something is happening, when there are changes in patterns or trends, so that you or your systems can take action.
## Building a real-time analytics system
What does it take to build a real-time analytics system? There are three requirements that these systems must satisfy:
### Speed of ingestion
Your application has data coming from a bunch of different sources, including batch and streaming sources. Streaming sources are built to pull new data as soon as it comes in. The data ingestion process needs to be really really fast. As soon as any changes happen, you should see this reflected in the real-time analytics system.
### Speed of queries
After you pull this data in, you need to make sure that the data can be queried as quickly as possible. If you're building dashboards, providing your end users with their own analytics, or feeding your data into some kind of machine learning algorithm, you need to make sure that all the freshly ingested data can be pulled out as quickly as it was pulled in.
### Scalability
Your system need to be able to handle a large number of queries run at the same time. It's not uncommon for real-time analytics systems to process 100s of thousands of queries per second.
# Claude Desktop
Source: https://docs.startree.ai/corecapabilities/ai/mcp/claude
# Claude Desktop Configuration
Configure Claude Desktop to connect with your StarTree MCP Server for natural language analytics.
## Prerequisites
* Claude Desktop installed ([download here](https://claude.ai/download))
* StarTree MCP Server [installed and running](./installation)
## Configuration Steps
### 1. Locate Claude's Configuration File
**macOS**
```bash theme={null}
~/Library/Application Support/Claude/claude_desktop_config.json
```
**Windows**
```bash theme={null}
%APPDATA%/Claude/claude_desktop_config.json
```
### 2. Add MCP Server Configuration
Open the configuration file and add your MCP server entry:
```json theme={null}
{
"mcpServers": {
"pinot_mcp_claude": {
"command": "/path/to/uv",
"args": [
"--directory",
"/path/to/mcp-pinot",
"run",
"mcp_pinot/server.py"
],
"env": {}
}
}
}
```
Replace the paths with your actual system paths:
```bash theme={null}
# Find your uv path
which uv
# Example: /Users/username/.local/bin/uv
# Use absolute path to your mcp-pinot directory
# Example: /Users/username/projects/mcp-pinot
```
### 3. Environment Configuration (Optional)
You can include environment variables directly in the configuration instead of using the `.env` file:
```
{
"mcpServers": {
"pinot_mcp_claude": {
"command": "/Users/username/.local/bin/uv",
"args": [
"--directory",
"/Users/username/projects/mcp-pinot",
"run",
"mcp_pinot/server.py"
],
"env": {
"PINOT_CONTROLLER_URL": "https://pinot.example.startree.cloud",
"PINOT_BROKER_HOST": "broker.pinot.example.startree.cloud",
"PINOT_BROKER_PORT": "443",
"PINOT_BROKER_SCHEME": "https",
"PINOT_TOKEN": "Bearer your_token_here",
"PINOT_DATABASE": "your_workspace_id"
}
}
}
}
```
### 4. Restart Claude Desktop
Completely restart Claude Desktop for the configuration to take effect.
### 5. Verify Connection
1. Look for the **hammer icon** 🛠️ in the Claude chat interface
2. Click the hammer to see available tools
3. Verify **Pinot tools** are listed (list-tables, read-query, etc.)
## Start Querying
You're ready to analyze your data! Try these sample prompts:
```
Can you help me analyze my data in Pinot? What tables are available?
```
```
Can you do a histogram plot on the GitHub events against time?
```
```
How many orders were placed in 24 hours on average? Can you help me
```
# Cursor IDE
Source: https://docs.startree.ai/corecapabilities/ai/mcp/cursor
# Cursor IDE Integration
Configure Cursor IDE to connect with your StarTree MCP Server for natural language analytics directly within your development environment.
## Prerequisites
* Cursor IDE installed ([download here](https://cursor.sh/))
* **StarTree MCP Server already installed** ([see installation guide](https://docs.startree.ai/corecapabilities/ai/mcp/installation))
## Configuration Steps
Since you already have a working `.env` file in your mcp-pinot directory, Cursor will automatically use those settings. No need to duplicate configuration!
Create or edit `~/.cursor/mcp.json`
```json theme={null}
{
"mcpServers": {
"pinot": {
"command": "uv",
"args": [
"--directory",
"/path/to/mcp-pinot",
"run",
"mcp_pinot/server.py"
]
}
}
}
```
Replace `/path/to/mcp-pinot` with the absolute path to your mcp-pinot directory
Add this config in Cursor MCP global settings
### Restart Cursor IDE
Completely restart Cursor IDE for the configuration to take effect.
## Verify Connection
Try asking questions related to Pinot:
# Installation
Source: https://docs.startree.ai/corecapabilities/ai/mcp/installation
# MCP Server Installation
This guide covers installing and configuring the StarTree MCP Server for Apache Pinot & StarTree Cloud
## Prerequisites
### System Requirements
* **Python 3.9+** installed on your system
* **Apache Pinot cluster & StarTree Cloud**
* **Git** for cloning the repository
### Install uv Package Manager
[uv](https://github.com/astral-sh/uv) is a fast Python package installer and resolver, written in Rust. It's designed to be a drop-in replacement for pip with significantly better performance.
```bash theme={null}
curl -LsSf https://astral.sh/uv/install.sh | sh
# Reload your shell configuration
source ~/.bashrc # or ~/.zshrc
# Alternatively, restart your terminal
# Verify installation
uv --version
```
## Installation Steps
### 1. Clone the Repository
```
git clone https://github.com/startreedata/mcp-pinot.git
cd mcp-pinot
```
### 2. Install Dependencies
```bash theme={null}
# Install the MCP server
uv pip install -e .
# For development (includes testing tools)
uv pip install -e .[dev]
```
### 3. Configure Pinot Connection
The MCP server requires a `.env` file to configure the Pinot cluster connection.
```bash theme={null}
# Copy the example configuration
cp .env.example .env
```
Edit the `.env` file with your Pinot cluster details:
#### Local Pinot Quickstart
```
# .env file for local Pinot
PINOT_CONTROLLER_URL=http://localhost:9000
PINOT_BROKER_HOST=localhost
PINOT_BROKER_PORT=8000
PINOT_BROKER_SCHEME=http
PINOT_USE_MSQE=true
```
#### StarTree Cloud
```
# .env file for StarTree Cloud
PINOT_CONTROLLER_URL=https://pinot.celpxu.cp.s7e.startree.cloud
PINOT_BROKER_HOST=broker.pinot.celpxu.cp.s7e.startree.cloud
PINOT_BROKER_PORT=443
PINOT_BROKER_SCHEME=https
PINOT_TOKEN=Bearer
PINOT_USE_MSQE=true
PINOT_DATABASE=ws_2kc8eddzzb0 (startree workspace id)
```
### 4. Run the MCP Server
```
uv --directory . run mcp_pinot/server.py
```
You should see logs indicating that the server is running and listening on STDIO:
```
INFO: MCP server started successfully
INFO: Listening on STDIO for MCP connections
```
### 5. Verify
With your MCP server running, test the connection in another terminal:
```bash theme={null}
uv --directory . run tests/test_service/test_pinot_quickstart.py
```
Expected output:
```
✓ MCP server connection successful
✓ Available tools discovered: ['list-tables', 'read-query', ...]
✓ Sample query executed successfully
✓ All tests passed
```
## Configure MCP Clients
With your MCP server installed and running, the final step is to configure your MCP client of choice to connect and start querying your data with natural language.
Claude Desktop is a desktop application that interacts with LLMs and connects to the StarTree MCP server.
LibreChat is a self-hosted chat interface that interacts with private LLMs (including AWS Bedrock models) and connects to the StarTree MCP server for secure, on-premises AI analytics.
Cursor IDE is an AI-powered code editor that integrates MCP pinot servers with your broader development workflow
# Librechat
Source: https://docs.startree.ai/corecapabilities/ai/mcp/librechat
# LibreChat Integration
Configure LibreChat to connect with your StarTree MCP Server and AWS Bedrock for natural language analytics with powerful foundation models.
## Prerequisites
* Docker and Docker Compose installed
* AWS Account with Bedrock access and appropriate IAM permissions
* StarTree MCP Server [installed and running](/corecapabilities/ai/mcp/installation)
* Node.js 18+ (for MCP Supergateway)
* LibreChat repository cloned locally
## Configuration Steps
### 1. Clone and Setup LibreChat
```
# Clone the repository
git clone https://github.com/danny-avila/LibreChat.git
cd LibreChat
# Copy environment file
cp .env.example .env
```
### 2. Configure Environment Variables
Edit the `.env` file with your AWS Bedrock credentials:
```yaml theme={null}
# Basic Settings
HOST=localhost
PORT=3080
MONGO_URI=mongodb://mongodb:27017/LibreChat
# User/Group IDs (to avoid Docker warnings)
UID=1000
GID=1000
# AWS Bedrock Configuration
BEDROCK_AWS_DEFAULT_REGION=us-east-1
BEDROCK_AWS_ACCESS_KEY_ID=your_access_key_id
BEDROCK_AWS_SECRET_ACCESS_KEY=your_secret_access_key
# Specify available Bedrock models
BEDROCK_AWS_MODELS=anthropic.claude-3-5-haiku-20241022-v1:0,anthropic.claude-3-sonnet-20240229-v1:0,meta.llama3-70b-instruct-v1:0
# Application Settings
APP_TITLE="LibreChat with Pinot Analytics"
ALLOW_REGISTRATION=true
```
### 3. Setup MCP Supergateway
Since mcp-pinot is Python-based, we use MCP Supergateway to bridge it to LibreChat:
```yaml theme={null}
# Install supergateway globally
npm install -g supergateway
# Start the gateway (from your mcp-pinot directory)
cd /path/to/mcp-pinot
supergateway --stdio "uv --directory . run mcp_pinot/server.py" --port 8110
# Alternative: Run in background using screen
screen -S pinot-gateway
supergateway --stdio "uv --directory . run mcp_pinot/server.py" --port 8110
# Press Ctrl+A, then D to detach
```
### 4. Create LibreChat Configuration
Create `librechat.yaml` in your LibreChat root directory:
```yaml theme={null}
# LibreChat Configuration
version: 1.1.5
cache: true
# Interface settings
interface:
privacyPolicy:
externalUrl: 'https://librechat.ai/privacy-policy'
openNewTab: true
termsOfService:
externalUrl: 'https://librechat.ai/tos'
openNewTab: true
# Enable registration and social logins
registration:
socialLogins: ['github', 'google']
# AWS Bedrock endpoint configuration
endpoints:
bedrock:
titleModel: 'anthropic.claude-3-5-haiku-20241022-v1:0'
streamRate: 35
availableRegions:
- "us-east-1"
- "us-west-2"
modelDisplayLabel: "AWS Bedrock"
# MCP Server configurations
mcpServers:
# StarTree Pinot MCP Server via Supergateway
pinot:
type: sse
url: "http://host.docker.internal:8110/sse"
timeout: 30000
initTimeout: 10000
iconPath: "/app/client/public/assets/logo.svg"
# Optional: Additional MCP servers
memory:
command: npx
args:
- -y
- "@modelcontextprotocol/server-memory"
timeout: 30000
filesystem:
command: npx
args:
- -y
- "@modelcontextprotocol/server-filesystem"
- "/app/uploads"
timeout: 30000
```
### 5. Create Docker Override Configuration
Create `docker-compose.override.yml`:
```yaml theme={null}
version: '3.4'
services:
api:
volumes:
- type: bind
source: ./librechat.yaml
target: /app/librechat.yaml
extra_hosts:
# Allow container to access host services
- "host.docker.internal:host-gateway"
environment:
# Pass AWS credentials to container
- BEDROCK_AWS_DEFAULT_REGION=${BEDROCK_AWS_DEFAULT_REGION}
- BEDROCK_AWS_ACCESS_KEY_ID=${BEDROCK_AWS_ACCESS_KEY_ID}
- BEDROCK_AWS_SECRET_ACCESS_KEY=${BEDROCK_AWS_SECRET_ACCESS_KEY}
- BEDROCK_AWS_MODELS=${BEDROCK_AWS_MODELS}
```
### 6. Start LibreChat
```bash theme={null}
# Launch the application
docker compose down # Stop any existing containers
docker compose up -d # Start in detached mode
# Verify startup
docker compose logs -f api
```
Look for these success messages in the logs:
```bash theme={null}
LibreChat | 2025-06-04 13:58:17 info: Server listening on all interfaces at port 3080. Use http://localhost:3080 to access it
LibreChat | 2025-06-04 13:58:36 info: [MCP][User: 68404ef09963a7b64176c39b][pinot] Establishing new connection
LibreChat | 2025-06-04 13:58:36 info: [MCP][User: 68404ef09963a7b64176c39b][pinot] Creating SSE transport: http://host.docker.internal:8110/sse
LibreChat | 2025-06-04 13:58:36 info: [MCP][User: 68404ef09963a7b64176c39b][pinot] Connection successfully established
```
## Verify Connection
### Access LibreChat Interface
* Open your browser and go to [http://localhost:3080](http://localhost:3080)
* Register a new account or login with existing credentials
* Look for "AWS Bedrock" in the model provider dropdown
* Look for pinot mcp in the chat dialog
## Start Querying
You're ready to analyze your data! Try these sample prompts:
```bash theme={null}
"What tables are available in my Pinot cluster?"
"Show me the schema for the airlineStats table"
"How many records are in the GitHub events table?"
```
# Overview
Source: https://docs.startree.ai/corecapabilities/ai/mcp/overview
# StarTree MCP Server for Apache Pinot
StarTree MCP Server represents a profound transformation in how businesses engage with their data. Move beyond traditional query consoles and complex SQL—MCP Server is the database client reinvented for the AI age.
Built on the Model Context Protocol (MCP) standard pioneered by Anthropic, this server creates a standardized bridge between large language models and real-time analytical data, serving everyone—AI agents, data analysts, and business users who've never written SQL.
### Your New Query Console
Traditional query consoles require complex SQL knowledge and manual data exploration. Whether you're a business user needing quick insights or an AI agent requiring contextual data access, StarTree MCP Server eliminates this friction by enabling natural language interactions with your analytical data at machine speed.
## Key Features
| **Feature** | **Description** |
| :- | :- |
| **Real-time Data Freshness** | Access the most current data available with Pinot's real-time ingestion capabilities, ensuring AI agents and analysts work with up-to-the-second information. |
| **Millisecond Query Performance** | Leverage Pinot's industry-leading query speed for instantaneous responses, critical for maintaining AI agent context and fluid human interactions. |
| **Natural Language Interface** | Transform plain English questions directly into precise database results without writing SQL—perfect for business users and AI agents alike. |
| **Schema Discovery** | Automatically identify table relationships, columns, and data types |
| **Multi-language Support** | Query data in your preferred language with automatic translation |
| **Metadata Integration** | Access table information, segment details, and performance metrics via API |
| **Universal Client Support** | Compatible with most MCP clients including Claude Desktop, LibreChat, Cursor, Windsurf & Agents. |
## Supported Operations
| Tool | Purpose |
| :- | :- |
| `list-tables` | Enumerate available tables |
| `read-query` | Execute SELECT queries |
| `table-details` | Get table metadata |
| `segment-list` | List table segments |
| `index-column-details` | Examine segment indexes |
| `segment-metadata-details` | Get segment metadata |
| `tableconfig-schema-details` | Get table configuration |
# Overview
Source: https://docs.startree.ai/corecapabilities/ai/overview
# AI-Powered Analytics at StarTree
The future of analytics is agent-facing. As artificial intelligence transforms how we interact with data, StarTree is pioneering the next generation of AI-native analytical experiences that bridge the gap between intelligent agents and real-time business insights.
## The Analytics SuperBrain Vision
Traditional analytics systems were built for human queries and dashboards. The next generation of intelligent systems—autonomous agents, conversational AI, and adaptive applications—requires fundamentally different data infrastructure: the ability to access thousands of contextual data points per conversation at machine speed, with real-time freshness and millisecond responsiveness.
StarTree's AI strategy centers on building a comprehensive **Analytics SuperBrain** architecture that transforms how intelligent systems interact with enterprise data. This isn't just about faster queries—it's about creating the data foundation that powers the next wave of AI-native applications, where systems continuously adapt, reason, and act based on live business context.
### Key Capabilities
**Next-Generation Query Console** Move beyond traditional SQL interfaces with a reimagined database client that enables natural language interactions for both human analysts and AI agents.
**Real-time AI Data Access** Seamlessly connect AI agents and business users to live business data with millisecond query performance, ensuring responses are based on the most current information available.
**Enterprise-Grade AI Integration** Deploy AI-powered analytics with the security, governance, and scale required for mission-critical business applications, serving both internal users and external AI systems.
Dive deeper into StarTree's AI strategy with our founder and CEO Kishore Gopalakrishna's keynote presentation on building the Analytics SuperBrain.
[**Watch the Keynote: "Building the Analytics SuperBrain"**](https://www.youtube.com/watch?v=your-keynote-video)
*In this presentation, Kishore outlines how Apache Pinot's proven user-facing analytics capabilities are evolving to meet the demands of agent-facing analytics, where AI systems require continuous interaction with live business data.*
## AI Solutions
Conversational analytics agent for Apache Pinot. Ask questions in plain English and get answers backed by real Pinot SQL, grounded in curated Spaces.
Your new query console reimagined for the AI age. Natural language analytics for business users and AI agents.
AI-powered query optimization that analyzes your Pinot multi-stage queries and produces prioritized, evidence-backed recommendations.
Automatic vector embeddings for semantic search combined with real-time analytics on structured data
# Query Analyzer API Reference
Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/api-reference
REST API reference for the Query Analyzer static and runtime analysis endpoints.
Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment.
Query Analyzer exposes two REST endpoints on the StarTree Cloud data-manager service. Both return the same unified response structure.
## Endpoints
| Method | Path | Purpose |
| - | - | - |
| `POST` | `/api/data/manager/api/query-analyzer/static-analysis` | Analyze a query using table metadata and explain plan — no execution required |
| `POST` | `/api/data/manager/api/query-analyzer/runtime-analysis` | Execute the query, collect stage statistics, then analyze with evidence-backed findings |
## Authentication and Headers
All requests require a valid StarTree Cloud Bearer token. The `workspace` header scopes the request to a specific workspace.
| Header | Required | Description |
| - | - | - |
| `Authorization` | Yes | `Bearer ` |
| `Content-Type` | Yes | `application/json` |
| `workspace` | No | Workspace ID. When omitted, the default workspace for the principal is used. |
***
## POST /api/data/manager/api/query-analyzer/static-analysis
Analyzes a SQL query using only table metadata and explain plan. Does not execute the query. Returns results within seconds.
### Request Body
```json theme={null}
{
"sql": "SELECT COUNT(*) FROM orders WHERE status = 'shipped' AND ts > 1700000000",
"timeout": "PT60S"
}
```
| Field | Type | Required | Description |
| - | - | - | - |
| `sql` | string | Yes | The SQL query to analyze |
| `timeout` | string | No | ISO-8601 duration override for the LLM call (e.g. `"PT60S"`, `"PT5M"`). Min: 10s. Max: 10 minutes. |
### Example Request
```bash theme={null}
curl -X POST "https:///api/data/manager/api/query-analyzer/static-analysis" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-H "workspace: " \
-d '{
"sql": "SELECT COUNT(*) FROM orders WHERE status = '\''shipped'\'' AND ts > 1700000000"
}'
```
### Example Response
```json theme={null}
{
"querySummary": "Counts all orders with status 'shipped' after a given Unix timestamp.",
"analysisType": "STATIC",
"findings": [
{
"type": "MISSING_INVERTED_INDEX",
"severity": "HIGH",
"description": "Column 'status' is used in an equality predicate but has no inverted index. Every segment will be fully scanned.",
"evidence": [
{
"type": "MISSING_INDEX",
"detail": "status: no inverted index found"
}
],
"confidence": "HIGH"
}
],
"recommendations": [
{
"rank": 1,
"title": "Add inverted index on 'status'",
"type": "ADD_INVERTED_INDEX",
"description": "An inverted index on 'status' will allow Pinot to skip segments that do not contain the predicate value, eliminating the full scan.",
"suggestedChange": {
"before": "\"invertedIndexColumns\": []",
"after": "\"invertedIndexColumns\": [\"status\"]"
},
"expectedImpact": "HIGH",
"confidence": "HIGH",
"tradeoffs": "Adds a small amount of index storage per segment. Requires a segment reload to take effect on existing data."
}
]
}
```
***
## POST /api/data/manager/api/query-analyzer/runtime-analysis
Executes the query against your Pinot cluster and collects execution statistics, using stage statistics wherever they're available. It then produces evidence-backed findings tied to the measured metrics.
If the query can't be executed or doesn't return execution statistics, the service falls back to static analysis. The response `analysisType` field shows which mode was used, and `mseIncompatibleReason` explains the fallback.
### Request Body
```json theme={null}
{
"sql": "SELECT status, COUNT(*) FROM orders WHERE ts BETWEEN 1700000000 AND 1710000000 GROUP BY status",
"timeout": "PT120S"
}
```
| Field | Type | Required | Description |
| - | - | - | - |
| `sql` | string | Yes | The SQL query to execute and analyze |
| `timeout` | string | No | ISO-8601 duration override for the LLM call. Min: 10s. Max: 10 minutes. |
Runtime analysis executes your query against the cluster. For very large or expensive queries, run static analysis first to catch structural issues before spending compute.
### Example Request
```bash theme={null}
curl -X POST "https:///api/data/manager/api/query-analyzer/runtime-analysis" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-H "workspace: " \
-d '{
"sql": "SELECT status, COUNT(*) FROM orders WHERE ts BETWEEN 1700000000 AND 1710000000 GROUP BY status"
}'
```
### Example Response
```json theme={null}
{
"querySummary": "Counts orders grouped by status within a time window.",
"analysisType": "RUNTIME",
"findings": [
{
"type": "MISSING_RANGE_INDEX",
"severity": "HIGH",
"description": "Column 'ts' is used in a BETWEEN predicate but has no range index. All segments are being scanned.",
"evidence": [
{
"type": "MISSING_INDEX",
"detail": "ts: no range index found"
},
{
"type": "HIGH_POST_FILTER_SCAN",
"detail": "numEntriesScannedPostFilter: 48,200,000"
}
],
"confidence": "HIGH",
"operator": "TABLE_SCAN",
"metrics": {
"executionTimeMs": 3420,
"memoryAllocatedBytes": 0,
"rowsProduced": 12400
}
}
],
"recommendations": [
{
"rank": 1,
"title": "Add range index on 'ts'",
"type": "ADD_RANGE_INDEX",
"description": "A range index on 'ts' allows Pinot to prune segments outside the time window, reducing the scan to only matching segments.",
"suggestedChange": {
"before": "\"rangeIndexColumns\": []",
"after": "\"rangeIndexColumns\": [\"ts\"]"
},
"expectedImpact": "HIGH",
"confidence": "HIGH",
"tradeoffs": "Adds index storage per segment. Requires a segment reload to take effect on existing data."
}
]
}
```
***
## Response Schema
Both endpoints return a `QueryAnalysisResponse` object.
| Field | Type | Description |
| - | - | - |
| `querySummary` | string | Plain-English description of what the query does |
| `analysisType` | string | `RUNTIME` when query execution succeeded; `STATIC` when the service fell back to static analysis |
| `mseIncompatibleReason` | string | Reason runtime analysis fell back to static analysis. Null when `analysisType` is `RUNTIME`. |
| `findings` | array | Risk flags detected, each with type, severity, description, evidence, confidence, and (runtime only) operator and metrics |
| `recommendations` | array | Ranked, actionable suggestions with title, type, description, suggested change, expected impact, confidence, evidence references, tradeoffs, and correctness notes |
| `error` | object | Structured error. Null on success. See [Error Response](#error-response). |
***
## Error Response
When a request fails, the response body contains an `error` object and the HTTP status reflects the failure category.
```json theme={null}
{
"error": {
"errorType": "BAD_REQUEST",
"description": "Invalid timeout value 'PT1S'. Expected ISO-8601 duration (e.g. PT60S, PT5M)."
}
}
```
| Field | Type | Description |
| - | - | - |
| `errorType` | string | Machine-readable error category (e.g. `BAD_REQUEST`, `PINOT_UNAVAILABLE`, `LLM_ERROR`, `UNKNOWN`) |
| `description` | string | Human-readable explanation of the failure |
| `cause` | object | Optional nested cause chain for debugging |
Common HTTP status codes:
| Status | When |
| - | - |
| `200` | Analysis completed successfully |
| `400` | Malformed request (e.g. missing `sql`, invalid `timeout` format, timeout out of bounds) |
| `401` | Missing or invalid authentication token |
| `403` | The authenticated principal does not have access to the requested tables |
| `500` | Internal error — contact support if it persists |
***
## Related Documentation
Step-by-step guide to running analysis from the Query Console and via API.
Full reference for every finding type, recommendation category, operator type, and evidence field in the response.
What data is sent to the LLM, access controls, and platform compliance posture.
# Query Analyzer
Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/overview
Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment.
Query Analyzer is an AI-powered tool built into StarTree Cloud that helps you understand and optimize Apache Pinot queries. It analyzes your SQL query alongside table metadata, explain plans, and execution statistics to produce prioritized, evidence-backed recommendations — all without requiring deep Pinot expertise.
## The Problem
Optimizing Pinot queries is hard because the debugging surface is **fragmented and expert-only**:
* **Explain plans** are detailed but difficult to interpret for most users
* **Stage statistics** show where time was spent, but not what to change
* **Table metadata** (indexes, cardinality, data types) is spread across multiple pages
* Users end up guessing, iterating blindly, or escalating to support
This leads to slow time-to-resolution, wasted compute, and repetitive support loops.
## How to Access
Query Analyzer is available directly in the **StarTree Data Portal Query Console**. Once enabled for your environment, no additional configuration is required — write your SQL query and use the built-in Analyze options to get instant, AI-generated optimization guidance.
Navigate to **Data Portal → Query Console**, write your query, and select **Analyze** to get started.
## What You Get Back
Query Analyzer returns three top-level sections for every analysis:
| Output | Description |
| - | - |
| **Query Summary** | Plain-English description of what the query does |
| **Risk Flags** | Issues found, ranked by severity. Each flag carries its own supporting evidence, bottleneck metrics (runtime), and any correctness alerts — so context and cause are always together |
| **Recommendations** | Ranked, actionable suggestions with expected impact, a before/after change, and tradeoffs |
Every recommendation is grounded in measured evidence, not generic advice.
## Two Analysis Modes
Run **before** executing your query. Flags likely problems using table metadata, index configuration, and the explain plan — so you can fix issues before spending compute.
Run **after** executing your query. Uses real execution statistics to pinpoint actual bottlenecks and provide evidence-based fixes.
**Proactive Analyzer:** Query Analyzer also supports a proactive mode that runs on a schedule, ranks your most expensive real queries from the query log, and analyzes them automatically — no query submission required. Disabled by default. See [Proactive Analyzer](/corecapabilities/ai/query-analyzer/proactive-analyzer/overview).
## What Gets Analyzed
**Index gaps**
Missing inverted, range, sorted, bloom filter, text, FST, JSON, star-tree, timestamp, geospatial, sparse, composite JSON, and vector indexes, as well as full scan risk.
**Query patterns**
Wrong join order, filters not pushed below joins, broadcast join risk, cross-join risk, OFFSET-based pagination, `SELECT *`, leading-wildcard `LIKE`, high result LIMIT, unnecessary sort, high-cardinality `GROUP BY`, group limit risk, heavy aggregation, suboptimal column encoding, missing cross-table filter, large table without pre-filter, and missing date range propagation.
**Operator-level execution** *(runtime only)*
Each risk flag produced during runtime analysis includes measured metrics from the relevant operator — execution time, memory allocation, rows produced, bytes shuffled, and correctness signals such as whether the aggregation group limit was hit.
## Security and Data Privacy
Query Analyzer uses an AI model to generate recommendations. Your SQL and table metadata are processed as part of every analysis — review what is sent, how it is protected, and how it maps to your security and compliance requirements **before** enabling or using the feature.
**Essential reading for security, compliance, and data governance teams.**
* AI model and cloud provider by deployment
* What data is sent to the LLM — and what is explicitly excluded
* Credential masking, data flows, and retention
* Access controls, encryption, and platform certifications (SOC 2, ISO 27001, HIPAA readiness)
## Guardrails
Query Analyzer is designed to produce high-signal, actionable output:
* Every recommendation includes specific evidence from the query plan or execution statistics
* Low-confidence and low-impact findings are suppressed — only high-signal results are shown
* Correctness alerts (such as results being potentially incomplete due to group limits) are surfaced prominently
* Query rewrite suggestions are validated against Pinot SQL syntax rules
* All suggestions include tradeoffs so you can make informed decisions
## Next Steps
Step-by-step guide to running static and runtime analysis from the Query Console, and how to act on the results.
Full reference for the static and runtime analysis REST endpoints — request schemas, response fields, curl examples, and error handling.
Scheduled background mode that finds and analyzes your most expensive real queries automatically. Disabled by default.
Reference guide to every output field, risk flag type, recommendation category, and what each means for your query.
AI model, data privacy, access controls, and platform compliance.
# Proactive Analyzer Configuration and Limitations
Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/proactive-analyzer/configuration
Configuration keys, the API endpoints, and current known limitations for Proactive Analyzer.
**Disabled by default.** Proactive Analyzer is off by default and requires the reactive [Query Analyzer](/corecapabilities/ai/query-analyzer/overview) to be enabled first. Contact your StarTree account team to have it enabled for your environment.
## Enablement
There is no single on/off flag for Proactive Analyzer. It activates when **both** of these are true:
1. The reactive analyzer is on (`query-analyzer.enabled=true`) — Proactive Analyzer reuses the same analysis engine.
2. At least one **delivery target** is configured: a Slack webhook URL or an email recipients list (see below).
If either condition is missing, the scheduled job never loads and the proactive API endpoints are not available.
## Configuration
All keys live under the `query-analyzer.proactive.` prefix:
| Setting | Default | Controls |
| - | - | - |
| `cron` | `0 0 9 * * *` | When the scan runs — daily at 09:00 **UTC** by default |
| `interval` | `PT24H` | The lookback window used to rank queries from the log |
| `query-engines` | `MSE,SSE` | Which query engines are scanned — one ranking pass per engine (multi-stage and single-stage by default) |
| `min-time-used-ms` | `100` | Minimum latency for a query shape to qualify |
| `max-analyses-per-run` | `5` | Cap on analyses **per engine** per run — up to 10 total with the default two engines |
| `system-query-log-table` | `system_query_log` | The query log table the scan reads |
| `notify.slack.webhook-url` | *(empty)* | Slack digest delivery — setting a webhook URL turns Slack delivery on |
| `notify.email.recipients` | *(empty)* | Accepted as a delivery target for activation, but email delivery is not implemented yet |
The skip-when-unchanged behavior (re-analyzing a query shape only when the table configuration has changed since the last analysis) is built in and not configurable.
## API
| Endpoint | Behavior |
| - | - |
| `POST /api/query-analyzer/proactive/run` | Starts one scan immediately on a background thread, without waiting for the schedule. Returns `202 Accepted`, or `409 Conflict` if a scan is already in progress. |
| `GET /api/query-analyzer/proactive/recommendations` | Lists the stored recommendations for the workspace, most expensive first. |
| `GET /api/query-analyzer/proactive/status` | Reports the effective configuration and a summary of the last run — candidates ranked, analyzed, skipped, failed, and the trigger (scheduled or manual). |
## Current Limitations
* **Recommendations are stored in memory** and reset when the service restarts.
* **Scans a single workspace on a single cluster per run.** Multi-cluster and multi-workspace scanning are not yet supported.
* **No automatic ticketing and no dedicated UI.** The digest points a human at the work; it does not file tickets or change any cluster setting on its own.
* **Email delivery is not yet available.** Slack and the log are the supported channels today.
## Related
What Proactive Analyzer does, how it works, and how it delivers results.
# Proactive Analyzer
Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/proactive-analyzer/overview
A scheduled job that finds your most expensive queries from real traffic and analyzes them automatically — no query submission required.
**Disabled by default.** Proactive Analyzer is off by default and requires the reactive [Query Analyzer](/corecapabilities/ai/query-analyzer/overview) to be enabled first. Contact your StarTree account team to have it enabled for your environment.
## What Is Proactive Analyzer?
Static and Runtime Analysis both require a deliberate trigger — you submit a query and ask for it to be analyzed. Proactive Analyzer removes that requirement.
Proactive Analyzer runs on a **schedule** (once a day by default), scans the queries that actually ran on your cluster, ranks them by cost, and runs the same analysis engine against the worst offenders automatically. It does this **without re-running any query** — it reads execution statistics that Pinot already recorded.
The goal is to catch expensive query patterns as they occur in production — the queries that quietly burn cluster resources without anyone flagging them — rather than waiting for a user to notice a slowdown and submit it for analysis.
## How It Works
Pinot records every query it runs in a system table, `system_query_log`, including the SQL, a stable hash identifying the query shape, execution time, CPU counters, scan volume, and per-stage execution statistics. Proactive Analyzer's scheduled job reads that log and, on each run:
1. **Checks the log is available.** If `system_query_log` is not queryable on the cluster, the run skips cleanly.
2. **Ranks query shapes by cost, per query engine.** The scan runs one ranking pass for each configured query engine (multi-stage and single-stage by default). Within each pass, shapes are grouped by query hash and ranked by **mean execution time** over the lookback window (24 hours by default) — a shape that is consistently slow ranks above one that is merely popular. Failed runs are excluded, and a minimum latency floor filters out trivially fast queries.
3. **Selects the worst run of each shape.** For each of the top-ranked shapes (up to 5 per engine by default), the single slowest successful run in the window is pulled, carrying its SQL and recorded execution statistics.
4. **Filters.** Queries against the log table itself are excluded, as are shapes with no matching successful run in the window and queries that consist entirely of SQL comments.
5. **Analyzes each one**, reusing the existing analysis engine (see below), up to a per-engine cap (5 by default — up to 10 analyses per run with the default two engines).
6. **Stores the results**, keyed by workspace, query shape, and query engine.
7. **Delivers a digest** summarizing what it found.
The schedule, lookback window, ranking thresholds, and per-run caps are all configurable — see [Configuration and Limitations](/corecapabilities/ai/query-analyzer/proactive-analyzer/configuration).
### It never re-runs a slow query
Re-running the exact queries just flagged as expensive would place the heaviest possible load back on the cluster — the opposite of what a tuning tool should do. Proactive Analyzer avoids this by reusing the execution statistics already captured in the query log. The only live data it fetches during analysis is cheap and current: table config, schema, table stats, and the explain plan — so recommendations reflect how the table is set up today, without replaying any heavy query. (When the analyzer proposes a rewritten query, the *rewrite* is validated against the cluster as usual; the original slow query is still never re-executed.)
### It reuses the same analysis engine
Proactive Analyzer does not introduce a second recommendation engine. It selects which queries to analyze, then hands each one to the same engine used by [Runtime Analysis](/corecapabilities/ai/query-analyzer/usage#running-runtime-analysis) — same prompt, same evidence gathering, same validation and guardrail checks — with one difference: instead of executing the query to capture statistics, it injects the statistics already recorded in the query log.
This matters for quality: the analyzer's most useful checks (for example, "this query is already fast enough" or "this ran out of memory") depend on seeing real runtime numbers. Because Proactive Analyzer feeds in the real numbers from the logged slow run, those checks behave exactly as they do for a manually submitted query — the output isn't a lower-quality approximation.
### Skips work that hasn't changed
Before analyzing a query shape, the job checks whether it has already analyzed it. It fingerprints the current table configuration and compares it to the fingerprint from the last analysis. If nothing has changed, there is nothing new to recommend — the job refreshes the cost numbers and skips the analysis call for that shape entirely, keeping run cost predictable.
## Delivery
Every run stores its recommendations and, on top of that, sends a digest:
| Channel | Behavior |
| - | - |
| **Log** | Always on. A formatted digest is written to the service log. |
| **Slack** | Optional. Configuring an incoming webhook URL turns it on; the run posts a summary of newly analyzed recommendations. |
| **Email** | Not yet available. A recipients setting is accepted (and counts as a delivery target for activation), but email delivery is not implemented yet. |
The digest also identifies where it came from — organization, environment, and a link back to the Query Console — so a shared Slack channel receiving digests from several environments stays readable.
If a run finds nothing to report (no log table, or zero new recommendations), the digest is skipped rather than sent empty. A failure in one delivery channel doesn't affect the others or the run itself.
## Available Today
While you evaluate Proactive Analyzer, both on-demand analysis modes are fully available:
Run before executing your query. Catches structural issues using table metadata, index configuration, and the explain plan.
Run after executing your query. Uses real execution statistics to pinpoint actual bottlenecks with measured evidence.
Both modes are available from the Query Console and via the [Query Analyzer API](/corecapabilities/ai/query-analyzer/api-reference).
Config keys, the manual trigger endpoint, and current known limitations.
# Understanding Recommendations
Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/reading-results
Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment.
This page is a reference for everything Query Analyzer returns. Use it to understand what each field means, what triggered a specific finding, and what the different recommendation and risk flag categories represent.
## Output Sections
### Risk Flags
The main output section. Each risk flag represents one detected issue and bundles everything related to that issue in one place: what was found, why it matters, the measured evidence behind it, and — for runtime analysis — the operator metrics and any correctness concerns.
| Field | Description |
| - | - |
| **Type** | The category of issue detected (see [Risk Flag Types](#risk-flag-types) below) |
| **Severity** | How serious the issue is — `HIGH` or `MEDIUM` |
| **Description** | A plain-English explanation of what was found and why it matters |
| **Evidence** | The specific data that triggered the flag — for example, a column name, cardinality value, index configuration, or measured execution metric |
| **Confidence** | How confident the analyzer is in the finding — `HIGH` or `MEDIUM` |
| **Operator** *(runtime)* | The execution operator this flag relates to — for example, Hash Join or Table Scan (see [Operator Types](#operator-types)) |
| **Metrics** *(runtime)* | Measured execution values for the operator — time, memory, rows, bytes shuffled |
Only `HIGH` and `MEDIUM` confidence findings are shown. Low-confidence results are suppressed to keep output focused.
When a risk flag carries a correctness alert, address it before acting on performance recommendations. A correctness issue — such as the group limit being reached — means query results may already be incomplete, which affects how you should interpret the rest of the analysis.
### Recommendations
Ranked, actionable suggestions. Each recommendation includes:
| Field | Description |
| - | - |
| **Rank** | Priority order — rank 1 has the highest expected impact |
| **Title** | Short description of the action to take |
| **Type** | The category of change recommended (see [Recommendation Types](#recommendation-types) below) |
| **Description** | Detailed explanation of why this change helps |
| **Suggested Change** | A before/after view of the specific change — a query snippet, index config, or hint |
| **Expected Impact** | How much improvement is expected — `HIGH` or `MEDIUM` |
| **Confidence** | How confident the analyzer is in this recommendation — `HIGH` or `MEDIUM` |
| **Evidence** | References to the supporting evidence items that justify this recommendation |
| **Tradeoffs** | Side effects or caveats to consider before applying the change |
| **Correctness Notes** | Semantic considerations — for example, whether a join reorder affects NULL handling |
Recommendations are ranked by expected impact first, then confidence. A query rewrite that eliminates an expensive operation ranks above an index addition at the same impact level.
***
## Risk Flag Types
Risk flags are detected during both **static and runtime analysis**. There are two categories.
### Index Gaps
These flags indicate that a column used in a filter, join, or aggregation is missing an index that would significantly speed up the query.
| Risk Flag | What It Means |
| - | - |
| **Missing Inverted Index** | A column used in an equality or IN predicate has no inverted index, requiring a full scan of each segment |
| **Missing Range Index** | A column used in a range comparison (`>`, `<`, `BETWEEN`) has no range index |
| **Missing Sorted Index** | A high-selectivity column could reduce segment pruning with a sorted index |
| **Missing Bloom Filter** | A low-cardinality column with equality predicates could benefit from a bloom filter to skip segments quickly |
| **Missing Text Index** | A column used with `LIKE` or text search has no full-text index |
| **Missing FST Index** | A column used with regex predicates has no FST (Finite State Transducer) index |
| **Missing JSON Index** | A column with JSON path access has no JSON index |
| **Missing Star-Tree Index** | An aggregation pattern could be pre-computed with a star-tree index, potentially eliminating the scan entirely |
| **Missing Timestamp Index** | A time column used in time-bucketing functions has no timestamp index |
| **Missing Geospatial Index** | A column used in spatial predicates has no geospatial index |
| **Missing Sparse Index** | A column stored in cold/tiered storage could benefit from a sparse index to reduce remote reads |
| **Missing Composite JSON Index** | Multiple JSON path accesses could be accelerated with a composite JSON index |
| **Missing Vector Index** | A column used in vector similarity search has no vector index |
| **Full Scan Risk** | The query structure is likely to cause a full segment scan regardless of indexes |
### Query Pattern Issues
These flags indicate structural patterns in the SQL or query plan that are likely to cause poor performance.
| Risk Flag | What It Means |
| - | - |
| **Wrong Join Order** | The larger table appears on the right (hash build) side of a join. Pinot builds the in-memory hash table from the right input — putting the larger table there wastes memory and slows the build |
| **Filter Not Pushed Down** | A filter appears above a join in the execution plan instead of below it. Pushing filters down reduces the number of rows that reach the join |
| **Broadcast Join** | The right-side input of a join is broadcast to every worker, which is expensive for large tables |
| **Self Join** | A table is joined with itself, which may indicate a query design issue |
| **Cross Join Risk** | A join condition is non-equality, which can produce a Cartesian product |
| **Offset Pagination** | `OFFSET` is used for pagination. For large offsets this requires scanning and discarding thousands of rows — use keyset (cursor) pagination instead |
| **Select Star** | `SELECT *` fetches all columns including potentially large or unused ones |
| **Leading Wildcard** | A `LIKE '%...'` pattern cannot use index prefix optimization and requires scanning every value |
| **High Result Limit** | A very large `LIMIT` value may produce excessive result sets |
| **Unnecessary Sort** | An `ORDER BY` clause has no corresponding `LIMIT`, meaning all rows are sorted |
| **High Cardinality Group By** | `GROUP BY` on a high-cardinality column produces a large intermediate state that requires significant memory |
| **Group Limit Risk** | The estimated number of groups may approach or exceed the default limit of 100,000 groups |
| **Heavy Aggregation** | A memory-intensive aggregate function such as `DISTINCTCOUNT` is used |
| **Suboptimal Encoding** | A column uses raw encoding where dictionary encoding would improve index effectiveness |
### Filter Propagation Issues
These flags identify missed opportunities to reduce data early by pushing filters across the query plan.
| Risk Flag | What It Means |
| - | - |
| **Missing Cross-Table Filter** | A filter applied to one side of a join could also be applied to the other side, but isn't — the engine ends up joining more rows than necessary |
| **Large Table Without Pre-filter** | A large table is scanned with no pre-filter, causing a full scan before the join or aggregation that follows |
| **Missing Date Range Propagation** | A date or time range filter on one side of a join is not propagated to the other side, missing an opportunity to prune data before the join executes |
### Diagnostic Values
These values may appear in the `type` field of an API response but are not user-facing risk flags. They are used internally to drive post-processing logic and short-circuit gates.
| Value | What It Means |
| - | - |
| `QUERY_FAILED` | The query did not execute successfully. When present, the analyzer may suppress or adjust downstream recommendations that depend on execution data. |
| `STAR_TREE_FAST_PATH_FIRED` | A positive signal — an existing star-tree index's metadata fast-path served the query without a full scan. Used to trigger the "already optimal" short-circuit. |
***
## Operator Types
Operator types appear within risk flags during runtime analysis. The **API value** column shows the exact string returned in the `operator` field of a `Finding` object.
| Operator | API Value | What It Does | Key Metrics to Watch |
| - | - | - | - |
| **Table Scan** | `LEAF` | Scans segments and applies filters | Entries scanned in-filter (high = indexes not being used); entries scanned post-filter (high = star-tree opportunity) |
| **Hash Join** | `HASH_JOIN` | Builds a hash table from the right input and probes it with the left | Hash table build time; memory allocated; rows emitted (high rows = join explosion) |
| **Data Send** | `MAILBOX_SEND` | Sends rows to downstream workers across stages | Bytes serialized; fan-out (number of workers data is sent to) |
| **Data Receive** | `MAILBOX_RECEIVE` | Receives rows from upstream workers | Bytes deserialized; upstream wait time (high = upstream stage is slow) |
| **Aggregation** | `AGGREGATE` | Groups rows and computes aggregate functions | Memory allocated; groups emitted; whether the group limit was hit |
| **Window Function** | `WINDOW` | Computes window functions over a partition | Execution time; memory allocated (high = large partitions or wide frames) |
| **Sort** | `SORT` | Sorts rows, typically in combination with a LIMIT | Execution time and memory for sort buffer |
| **Filter** | `FILTER` | Applies a predicate to filter rows mid-plan | Execution time; rows emitted (high = filter is not selective enough) |
| **Transform** | `TRANSFORM` | Applies column transformations or expression evaluation | Execution time; memory allocated (high = expensive expressions or many columns) |
| **Union** | `UNION` | Combines row sets from multiple inputs (`UNION ALL`) | Execution time; total rows emitted across all inputs |
| **Intersect** | `INTERSECT` | Returns rows present in all inputs (`INTERSECT`) | Execution time; memory allocated for set tracking |
| **Minus** | `MINUS` | Returns rows from the first input not present in subsequent inputs (`EXCEPT`) | Execution time; memory allocated for set tracking |
***
## Evidence Types
Evidence is embedded within each risk flag to show what triggered it. The type tells you where the finding came from.
### Found in Both Modes
| Evidence Type | What It Means |
| - | - |
| **Full Scan** | A full segment scan was detected in the explain plan |
| **Index Used** | A specific index was used — for example, an inverted index for an equality predicate |
| **Join Order** | Join order analysis based on table row counts |
| **Missing Index** | A column is missing an index that would help — covers inverted, range, text, FST, JSON, bloom filter, star-tree, timestamp, geospatial, sparse, composite JSON, and vector index types |
| **Filter-Only Join** | The right side of a join is used only to filter rows, making it a candidate for rewriting as `WHERE col IN (SELECT ...)` |
| **Star-Tree Blocked** | A star-tree index cannot be used because the aggregation depends on a cross-table join |
| **Column Cardinality** | Cardinality data from table statistics, used to assess index recommendations |
| **Predicate Pattern** | Analysis of the query's filter structure |
| **Suboptimal Encoding** | Column encoding is suboptimal for the query's access pattern |
### Found During Static Analysis Only
| Evidence Type | What It Means |
| - | - |
| **Plan Structure** | A structural observation from the explain plan |
| **Exchange Type** | The data distribution pattern between stages — hash, broadcast, or random |
### Found During Runtime Analysis Only
| Evidence Type | What It Means |
| - | - |
| **High Post-Filter Scan** | Many entries were scanned after the filter, suggesting a star-tree index opportunity |
| **Join Build Time** | High hash table build time indicates the right-side table is too large |
| **Join Explosion** | Large number of output rows from the join, indicating a many-to-many relationship |
| **Group Limit Hit** | The 100,000 group limit was reached — results may be incomplete |
| **High Memory** | An operator allocated significant heap memory |
| **Heavy Shuffle** | Large volumes of data were sent between workers |
| **Upstream Wait** | A stage spent significant time waiting for its upstream stage to produce data |
| **Downstream Wait** | A stage was blocked waiting for the downstream consumer to process rows |
| **Broadcast Detected** | A broadcast exchange was detected in the execution tree |
***
## Recommendation Types
Recommendations appear in both static and runtime analysis.
### Query Changes
| Type | What Gets Changed |
| - | - |
| **Query Rewrite** | The SQL itself — for example, reordering joins, replacing OFFSET with keyset pagination, or rewriting a join as a subquery |
| **Hint Addition** | A query hint added inline to control join strategy, aggregation behavior, or partition handling |
### Index Additions
| Type | Index Added |
| - | - |
| **Add Inverted Index** | Speeds up equality and IN predicates on the target column |
| **Add Range Index** | Speeds up range comparisons on the target column |
| **Add Bloom Filter** | Allows fast segment skipping for equality predicates on the target column |
| **Add Text Index** | Enables full-text search and speeds up LIKE predicates |
| **Add FST Index** | Enables fast regex matching on the target column |
| **Add JSON Index** | Speeds up JSON path access on the target column |
| **Add Star-Tree Index** | Pre-computes aggregations to potentially eliminate scanning entirely |
| **Add Timestamp Index** | Speeds up time-bucketing functions on datetime columns |
| **Add Geospatial Index** | Speeds up spatial predicate evaluation |
| **Add Sparse Index** | Reduces remote reads for columns in cold/tiered storage |
| **Add Composite JSON Index** | Optimizes multiple JSON path accesses together |
| **Add Vector Index** | Enables efficient vector similarity search |
### Configuration Changes
| Type | What Gets Changed |
| - | - |
| **Encoding Change** | Column encoding type — for example, switching from raw to dictionary encoding |
| **Join Strategy** | How the join is executed — for example, using a lookup join for dimension tables or a colocated join to eliminate data shuffling |
| **Aggregation Hint** | Aggregation behavior — for example, enabling group trimming for ORDER BY + LIMIT patterns, or increasing the group count limit |
| **Repartition Data** | Set or align `segmentPartitionConfig` and repartition existing data. The config change takes effect on new segments immediately; existing segments require a segment refresh or re-ingest to physically repartition. |
| **Table Rebalance** | Trigger a controller-level table rebalance to redistribute segments across servers. No config diff is required — the rebalance operation alone addresses the imbalance. |
| **Cluster Scale** | Scale the cluster up or out — add server nodes, increase memory, or add replicas. Recommended only as a last resort for sustained resource saturation (OOM, repeated timeouts) that no query rewrite or configuration change can resolve. |
| **Scope Reduction** | Reduce what the query asks for — for example, narrow the time window, add a selective pre-filter, drop high-cardinality GROUP BY columns, or accept approximate aggregates. **This changes the result set by design.** Always a secondary recommendation on resource-exhaustion cases (group-limit hit, OOM, timeout) where the volume of data demanded is itself the problem. Never ranked above an equivalence-preserving fix, and always flagged as result-altering in the suggested change. |
### Sentinel
| Type | What It Means |
| - | - |
| **No Optimization Needed** | The query is already optimal — for example, a trivial `SELECT * FROM t LIMIT n` preview. No suggested change is produced; the recommendation carries only a title and description confirming the query needs no changes. |
# Security and Compliance
Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/security-and-compliance
Security controls, data privacy, and compliance posture for Query Analyzer in StarTree Cloud.
Query Analyzer is an opt-in, AI-powered feature in StarTree Cloud. This page describes how your data is handled, which security controls apply, and how Query Analyzer inherits StarTree Cloud's platform compliance posture.
Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment.
## Overview
Query Analyzer is **request-scoped**: it processes one analysis per request and does **not** persist query text, metadata, or LLM responses to any database or customer-accessible storage.
## Data Flows
```mermaid theme={null}
flowchart LR
Portal[Data Portal]
Cloud[StarTree Cloud]
Pinot[Pinot Cluster]
LLM[LLM Provider]
Portal -->|"SQL + auth headers"| Cloud
Cloud -->|"Metadata, explain, execution stats"| Pinot
Cloud -->|"Minimized prompt"| LLM
LLM -->|"Analysis response"| Cloud
Cloud -->|"Recommendations"| Portal
```
### Collected from Pinot
For normal (non-direct) analysis, Query Analyzer uses your authenticated identity to call **Pinot Controller** admin APIs on your cluster. The following operations run per table referenced in your SQL (except explain and query execution, which run once per analysis request).
| Operation | Pinot API | Static Analysis | Runtime Analysis | API reference |
| - | - | - | - | - |
| Fetch table config and schema | `GET /tableConfigs/{tableName}` | Yes | Yes | [Get table configs](/api-reference/table/get-the-tableconfigs-for-a-given-raw-tablename) |
| Infer index configuration | `POST /tables/inferIndexes` | Yes | Yes | StarTree extension API (normalizes index config for analysis) |
| Fetch table statistics | `GET /tables/{tableName}/metadata` | Yes | Yes | [Get aggregate table metadata](/api-reference/table/get-the-aggregate-metadata-of-all-segments-for-a-table) |
| Run explain plan | `POST /sql` with `EXPLAIN PLAN FOR` and `explainAskingServers=true` | Yes | Yes | Pinot Controller SQL query API |
| Execute the SQL query | `POST /sql` with `collectGcStats=true` | No | Yes | Pinot Controller SQL query API |
The `POST /sql` endpoint is the Pinot Controller SQL query API. It is used for explain plans and, in runtime analysis, to execute your query and collect `stageStats`. This is separate from the broker query API (`POST /query/sql`) used for interactive querying in some deployments.
Pinot's own RBAC policies govern which tables and operations your identity can access. Runtime analysis **executes your query** against the cluster. Use it only when the query is safe to run in your environment.
### Sent to the LLM
#### What is never included in the LLM prompt
| Never included | Detail |
| - | - |
| **Row data and query result values** | Your actual analytics data remains in Pinot |
| **Credentials and secret references** | Masked in table configuration before transmission |
| **Personally identifiable information (PII)** | No user-identifying information is included |
| **Model training** | Prompt data is **not used to train or improve** the underlying LLM |
Only structural metadata and execution statistics are sent to the LLM provider. Your actual data values are not included.
#### What is included in the LLM prompt
| Data | Source | Notes |
| - | - | - |
| SQL query text | User input | The query you submit for analysis |
| Table config (reduced) | Pinot | Stripped and masked (see note below) |
| Schema | Pinot | Field types, encoding, and index configuration |
| Column cardinality estimates | Pinot | Approximate distinct value counts per column |
| Explain plan | Pinot | Execution structure only |
| Execution statistics | Pinot | Operator timing and memory metrics (runtime analysis only) |
Before transmission, StarTree Cloud **masks** credentials and secret references in table configuration JSON and **strips** non-essential sections (ingestion config, task definitions, tenants, metadata, and routing).
## Data Lifecycle and Retention
| Aspect | Behavior |
| - | - |
| **Persistence** | No query text, metadata, or LLM responses are written to durable storage |
| **In-memory scope** | Analysis data exists only for the duration of the HTTP request |
| **Application logs** | StarTree Cloud application logs may record SQL text at INFO level; subject to platform log retention and access controls |
## AI Model and Provider
| Deployment | AI Provider | Model |
| - | - | - |
| AWS | Amazon Bedrock | Claude (Anthropic), `us.anthropic.claude-sonnet-4-20250514-v1:0` |
| Google Cloud (Coming Soon) | Vertex AI | Not yet available |
| Azure (Coming Soon) | Azure AI Foundry | Not yet available |
On AWS, prompts are processed by Amazon Bedrock under AWS's data privacy commitments.
## Access Controls
Query Analyzer uses the same authentication and authorization model as other StarTree Cloud APIs. See [Manage Security](/corecapabilities/security/manage-security) for OIDC identity provider setup, RBAC policies, and API tokens.
| Control | Behavior |
| - | - |
| **Authentication** | Required for `POST /api/query-analyzer/static-analysis` and `POST /api/query-analyzer/runtime-analysis` |
| **Authorization** | Governed by StarTree Cloud RBAC and Pinot cluster policies for the tables in your query |
| **Workspace scoping** | Use the `workspace` header to scope requests to a specific workspace |
| **Feature availability** | Disabled by default. Contact your StarTree account team to enable. |
## Encryption
| Layer | Control |
| - | - |
| **In transit** | TLS 1.2+ between Data Portal, StarTree Cloud, Pinot, and the LLM provider |
| **At rest** | Query Analyzer does not write analysis data to disk; platform policies apply to operational logs |
| **Secrets** | LLM API keys are managed through StarTree's secure deployment configuration and are not embedded in customer-facing payloads |
## Platform Compliance
Query Analyzer runs within StarTree Cloud infrastructure and inherits platform certifications:
* **SOC 2 Type 2**
* **ISO 27001**
* **HIPAA readiness**
For full platform security details, see [Security and Compliance](/getstarted/deployment/security_certifications).
## Related Documentation
Feature overview, analysis modes, and what gets analyzed.
Platform-wide security controls and compliance certifications.
# How to Use Query Analyzer
Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/usage
Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment.
Query Analyzer is built into the **StarTree Data Portal Query Console**. No setup is required — open the console, write your query, and run an analysis.
## Running Static Analysis
Static analysis catches issues **before** your query runs.
1. Navigate to **Data Portal → Query Console**
2. Write or paste your SQL query
3. Select **Analyze → Static Analysis**
Query Analyzer fetches table metadata automatically and returns results within seconds.
***
## Running Runtime Analysis
Runtime analysis identifies issues **after** your query runs, using real execution statistics. Each risk flag includes the operator metrics and any correctness alerts relevant to that finding.
1. Write your SQL query in the Query Console
2. Select **Analyze → Runtime Analysis**
Runtime analysis executes your query against your Pinot cluster. For very large or expensive queries, run static analysis first to catch structural issues before proceeding.
***
## Acting on Recommendations
**Index additions** — Apply through **Data Portal → Table Management**. Index changes require a segment reload to take effect on existing data.
**Query rewrites** — Apply directly in the Query Console editor. The recommendation shows a before/after diff.
**Query hints** — Added inline to your SQL using `/*+ ... */` syntax. Take effect immediately without schema changes.
Start with rank 1 recommendations. If any risk flag carries a correctness alert, address it first — incomplete results affect how you should interpret the rest of the analysis.
For a full breakdown of what each output field and recommendation type means, see [Understanding Recommendations](/corecapabilities/ai/query-analyzer/reading-results).
***
## Using the API
Both analysis modes are available as REST endpoints on the StarTree Cloud data-manager service. Use these when integrating Query Analyzer into CI pipelines, query validation tooling, or custom dashboards.
### Prerequisites
* A valid **StarTree Cloud API token** (Bearer auth)
* The **workspace ID** for your environment (obtain from the Data Portal URL or your account team)
* Query Analyzer must be enabled for your environment
### Static Analysis via API
`POST /api/query-analyzer/static-analysis`
Analyzes a SQL query using table metadata and explain plan only — no query execution required.
```bash theme={null}
curl -X POST "https:///api/query-analyzer/static-analysis" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-H "workspace: " \
-d '{
"sql": "SELECT COUNT(*) FROM orders WHERE status = '\''shipped'\'' AND ts > 1700000000"
}'
```
### Runtime Analysis via API
`POST /api/query-analyzer/runtime-analysis`
Executes the query against your Pinot cluster, collects execution statistics, then runs analysis with evidence-backed findings.
```bash theme={null}
curl -X POST "https:///api/query-analyzer/runtime-analysis" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-H "workspace: " \
-d '{
"sql": "SELECT status, COUNT(*) FROM orders WHERE ts BETWEEN 1700000000 AND 1710000000 GROUP BY status"
}'
```
Runtime analysis executes your query against the cluster. For expensive queries, run static analysis first.
### Optional Request Parameters
Both endpoints accept the following optional fields in the request body:
| Parameter | Type | Description |
| - | - | - |
| `timeout` | string | ISO-8601 duration override for the LLM call (e.g. `"PT60S"`, `"PT5M"`). Minimum: 10 seconds. Maximum: 10 minutes. Default: server-configured value. |
| `thinkingMode` | boolean | Enable extended chain-of-thought reasoning in the LLM. Produces deeper analysis at the cost of higher latency. Default: server-configured value. |
For the full request and response schemas, see the [API Reference](/corecapabilities/ai/query-analyzer/api-reference).
***
## Tips
* **Use real queries**: Query Analyzer uses column cardinality and table size to produce accurate recommendations. Simplified test queries produce weaker results.
* **Static first, runtime second**: Static analysis is faster. Use runtime analysis for your most important or problematic queries.
* **Read the tradeoffs**: Every recommendation includes tradeoffs, for example, an index that helps one query pattern may not justify the storage cost.
# Scheduled Server Scaling
Source: https://docs.startree.ai/corecapabilities/cluster-operations/scheduled-server-scaling
This feature requires StarTree release 0.15.0 or later.
Scheduled Server Scaling lets you automatically scale Apache Pinot server replica groups up or down on a fixed schedule. This is useful when your query load is predictable — for example, running fewer server replica groups overnight or on weekends to save cost, and restoring full capacity before peak hours.
Scaling is driven by **cron schedules** defined per Pinot **tenant**. At each scheduled time, the operator asks the Pinot controller which servers to remove (or restore) to reach your target replica-group count, drains queries off the affected servers, and scales their StatefulSets accordingly.
Scaling operates at the granularity of **replica groups**, not individual servers. You specify a target number of replica groups; the Pinot controller computes the exact set of servers to remove or restore to achieve that target. See [Replica Group based Workload Isolation](/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation) for background on replica groups and pools.
Scaling down reduces redundancy for the tenant until the matching scale-up restores it. A schedule that targets `targetReplicaGroups: 1` leaves the tenant with **no replica group failover** for the duration of the scale-down window — if that single replica group has an issue, queries to the tenant's tables will fail. Choose a target that keeps the minimum redundancy your workload needs.
## Prerequisites
* Your cluster is on **StarTree release 0.15.0 or later**.
* Your cluster is managed by the StarTree Kubernetes operator via the `PinotCluster` custom resource. This feature is configured at the operator level by the StarTree Ops team — see [Enabling it](#enabling-it).
* The tenant's tables already use **pool-based, replica-group-aware** instance assignment — see [Controller requirements for scale-down](#controller-requirements-for-scale-down) for the exact conditions the controller checks before it will scale down a tenant.
## How it works
```mermaid theme={null}
flowchart TD
Cron["Cron time fires\nfor a schedule"] --> Action{"action"}
Action -->|SCHEDULED_SCALE_DOWN| AskDown["Ask Pinot controller which\nservers to remove\n(GET /serverReplicaGroupScaleDown)"]
AskDown --> Drain["Server marks itself unavailable\nand drains in-flight queries\n(up to 10 minutes)"]
Drain --> ScaleDownSS["Scale affected server\nStatefulSets to zero"]
Action -->|SCHEDULED_SCALE_UP| AskUp["Determine which previously-removed\nservers must come back"]
AskUp --> ScaleUpSS["Restore those server\nStatefulSets"]
Missed["Operator was down\nat cron time"] -.->|restarts within\nmissedExecutionWindow| Action
Missed -.->|restarts after\nmissedExecutionWindow| Skip["Run is skipped"]
```
1. Scheduled scaling is enabled in the `PinotCluster` spec under the `server` component, with one or more schedules defined per tenant (see [Enabling it](#enabling-it)).
2. The operator creates and maintains a `ScheduledServerScaling` custom resource for each tenant you configure.
3. At each schedule's cron time:
* **Scale down** — the operator calls the Pinot controller to determine which servers can be removed to reach the target replica-group count. Each affected server marks itself unavailable and drains its in-flight queries (up to 10 minutes) before its StatefulSet is scaled to zero.
* **Scale up** — the operator determines which previously-removed servers must come back to reach the target, and restores their StatefulSets.
4. If the operator was down when a schedule fired, it still executes the missed run as long as it restarts within the `missedExecutionWindow`. After that window passes, the missed run is skipped until the next occurrence.
While a scale operation is in progress, the operator suppresses normal replica reconciliation for the affected servers, so a manually-running cluster reconcile will not fight the schedule.
## Enabling it
Scheduled scaling is configured at the Kubernetes operator level, via the `PinotCluster` custom resource. This isn't currently exposed in the Data Portal or any self-serve UI, so **contact the StarTree Ops team** to have it enabled for your cluster.
Self-serve configuration of scheduled scaling from the Cloud Console is coming soon.
To enable it, share the following with StarTree Ops for each tenant you want to schedule:
| Input | Description |
| :- | :- |
| Tenant name | The Pinot server tenant to schedule (e.g. `DefaultTenant`). |
| Schedules | One or more `{name, action, cron, targetReplicaGroups}` combinations — see the [configuration reference](#configuration-reference) below for what each field means. |
| Desired replica-group counts | The `targetReplicaGroups` to scale down to (off-peak) and scale back up to (peak). |
| Cron times | When each scale-down/scale-up should fire, in **UTC**. |
To change or disable a schedule, or to remove scheduled scaling from a tenant altogether, reach out to StarTree Ops with the updated details. When a tenant is removed from the spec, the operator deletes its `ScheduledServerScaling` resource and restores any servers that were left scaled down.
## Configuration reference
`scheduledScaling` (under `spec.components.server`):
| Field | Type | Description |
| :- | :- | :- |
| `enabled` | boolean | Master switch for scheduled scaling on this cluster. |
| `tenants` | array | One entry per Pinot tenant you want to schedule. Each has `tenant` + `schedules`. |
Each entry in `tenants`:
| Field | Type | Description |
| :- | :- | :- |
| `tenant` | string | Pinot server tenant name (e.g. `DefaultTenant`). |
| `schedules` | array | One or more scheduled actions for this tenant. |
Each entry in `schedules`:
| Field | Type | Required | Description |
| :- | :- | :- | :- |
| `name` | string | yes | A label for the schedule, used in logs and status. |
| `action` | enum | yes | `SCHEDULED_SCALE_DOWN` or `SCHEDULED_SCALE_UP`. |
| `cron` | string | yes | 5-field Unix cron expression (`minute hour day-of-month month day-of-week`), evaluated in **UTC**. See examples below. |
| `targetReplicaGroups` | integer | yes | Desired number of server replica groups for the tenant **after** this action completes. Must be greater than 0. |
| `missedExecutionWindow` | string | no | Grace period after the scheduled time during which a missed run still executes if the operator was down (e.g. `30m`, `1h`). Defaults to `15m`. |
Before a server is scaled down, it marks itself unavailable and drains its in-flight queries, for up to **10 minutes**, before its StatefulSet is scaled to zero. This drain timeout is fixed and isn't currently configurable.
### Cron format
Schedules use standard **5-field Unix cron** (`minute hour day-of-month month day-of-week`). All times are evaluated in **UTC** — convert your local schedule to UTC before setting `cron`.
| Expression | Meaning (UTC) |
| :- | :- |
| `0 22 * * *` | Every day at 22:00 UTC |
| `0 6 * * 1-5` | Weekdays at 06:00 UTC |
| `0 0 * * 0` | Every Sunday at midnight UTC |
| `30 18 * * 5` | Every Friday at 18:30 UTC |
## Typical pattern: nightly scale-down, morning scale-up
Pair a `SCHEDULED_SCALE_DOWN` with a `SCHEDULED_SCALE_UP` to shrink capacity during off-hours and restore it before peak load:
* **Scale down** at night to a low `targetReplicaGroups` (e.g. `1`).
* **Scale up** in the morning back to your full `targetReplicaGroups` (e.g. `3`).
Use the same tenant for both schedules. The scale-up restores exactly the servers that the matching scale-down removed.
## Checking status
The operator tracks each tenant's scaling state in the `ScheduledServerScaling` resource:
```bash theme={null}
kubectl get scheduledserverscaling -n
kubectl get scheduledserverscaling -n -o yaml
```
Key status fields:
| Field | Description |
| :- | :- |
| `status.state` | `AVAILABLE`, `DELETING` (scale-down in progress), `DELETED` (scaled down), `RESTORING` (scale-up in progress), `RESTORED`. |
| `status.lastScalingOperation.action` | The most recent action: `SCHEDULED_SCALE_DOWN` or `SCHEDULED_SCALE_UP`. |
| `status.lastScalingOperation.lastOccurrence` | Timestamp of the cron fire time that triggered the operation. |
| `status.lastScalingOperation.serversAffected` | List of server StatefulSets affected by the operation. |
| `status.lastScalingOperation.targetReplicaGroups` | The target replica-group count of the last completed operation. |
## Notes and limitations
* `targetReplicaGroups` must be **greater than 0** — a schedule cannot remove every replica group.
* If `targetReplicaGroups` equals the tenant's current total replica groups, a scale-down is a no-op (nothing to remove).
* Cron times are evaluated in **UTC** — convert your local time before setting `cron`.
* Scheduled scaling only affects **server** StatefulSets for the named tenant. It does not change Zookeeper, Controller, Broker, or Minion components.
## Controller requirements for scale-down
At each scale-down, the operator asks the Pinot controller (`GET /serverReplicaGroupScaleDown`) which servers can be removed for the tenant. The controller only returns a server list when **all** of the following hold. If any fails, the scale-down is rejected and no servers are removed — fix the underlying condition and the next scheduled run (or a manual retry) will proceed.
**Tenant and topology**
* The tenant exists and has servers tagged for it (`_OFFLINE` / `_REALTIME`).
* **Servers are exclusive to the tenant** — no server tagged for this tenant may also carry tags for another tenant.
* Every tenant server has a **pool assignment**, and each pool matches at least one of the tenant's tags.
* If a server has both OFFLINE and REALTIME tags, both must point to the **same pool number**.
**Target value**
* `targetReplicaGroups` must be **≥ 1** and **≤ the current number of replica groups (pools)**.
* If `targetReplicaGroups` equals the current replica-group count, the call succeeds but returns **no servers** (nothing to remove).
**Table configuration (when the tenant has tables)**
* No table may use `instancePartitionsMap` (which bypasses pool-based assignment).
* All non-dimension tables must use **pool-based, replica-group-aware** instance assignment.
* Each table's configured `numReplicaGroups` (when non-zero) must **equal the current pool count**.
**No rebalance in progress**
* No table in the tenant may have an active or failed **table rebalance** job.
* The tenant may not have an active, aborted, cancelled, or unscheduled **tenant rebalance** job.
The controller evaluates these against a point-in-time snapshot of cluster state. The highest numbered pools are always selected for removal, so a given `(tenant, targetReplicaGroups)` request is deterministic.
## Troubleshooting
A scale-down is rejected, with no servers removed, whenever one of the [controller requirements](#controller-requirements-for-scale-down) above isn't met. Fix the underlying condition — the next scheduled run (or a manual retry) will proceed once it's resolved.
| Symptom | Likely cause | What to do |
| :- | :- | :- |
| Scale-down never removes any servers | A server tagged for the tenant is also tagged for another tenant | Make servers exclusive to one tenant, or scope the schedule to a tenant that already has dedicated servers. |
| Scale-down never removes any servers | A server is missing a pool assignment, or OFFLINE/REALTIME tags point to different pools | Fix the server's pool assignment so it matches the tenant's tags, and align OFFLINE/REALTIME pool numbers. |
| Scale-down never removes any servers | `targetReplicaGroups` is greater than the tenant's current replica-group (pool) count | Lower `targetReplicaGroups` to at most the current pool count. |
| Scale-down never removes any servers | A table uses `instancePartitionsMap` | Migrate the table to pool-based instance assignment before scheduling scale-down for its tenant. |
| Scale-down never removes any servers | A table's `numReplicaGroups` doesn't match the current pool count | Update the table's replica-group config to match the tenant's current pool count. |
| Scale-down never removes any servers | A table or tenant rebalance is active, aborted, cancelled, or unscheduled | Let the rebalance finish (or clear the stuck job), then wait for the next scheduled run or retry manually. |
| Scale-up doesn't fully restore capacity | The matching scale-down removed a different set of servers than expected | Check `status.lastScalingOperation.serversAffected` on the `ScheduledServerScaling` resource for both operations to confirm which servers were removed vs. restored. |
| A schedule didn't run at all | The operator was down past `missedExecutionWindow` at the cron time | Increase `missedExecutionWindow` if operator restarts routinely take longer than the current setting, or trigger the action manually. |
## Related
* [Replica Group based Workload Isolation](/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation) — background on replica groups and pool-based instance assignment.
* [Health checks](/corecapabilities/observability/health-checks) — check overall cluster health, including the replication and instance-pool checks relevant to scale-down eligibility.
# Troubleshooting resource pressure
Source: https://docs.startree.ai/corecapabilities/cluster-operations/troubleshooting
Diagnose out-of-memory conditions, high CPU, disk pressure, and degradation after an upgrade or a traffic peak.
This guide covers the cluster running out of something: memory, CPU, or disk. These problems announce themselves indirectly — as query timeouts, ingestion lag, or tasks failing in unrelated places — so the first job is recognising that resource pressure is what you are looking at.
## Recognising it
Resource pressure is the likely cause when:
* **Several unrelated things degrade at once.** Queries slow, ingestion falls behind, and tasks start failing, all within the same window. A single bad query does not do this.
* **Symptoms correlate with load rather than with a specific query or table.** The same query is fast at 03:00 and slow at 14:00.
* **Failures cluster in time.** Garbage-collection pauses and restarts produce bursts of timeouts separated by normal periods, rather than a steady failure rate.
* **A component restarted.** Anything that was killed and came back was almost certainly killed for consuming too much of something.
## Identify which component and which resource
Both halves matter. "The cluster is slow" is not diagnosable; "brokers are fine, servers on one tenant are at 90% heap" is.
| Component | Symptom when under pressure |
| - | - |
| **Server** | Query timeouts, slow scans, ingestion lag on tables it hosts, segments going unavailable |
| **Broker** | Timeouts affecting all tables equally, planning-phase failures, connection errors |
| **Controller** | Table operations hang or fail, segment state stops converging, tasks stop being scheduled |
| **Minion** | Batch and ingestion tasks fail in unrelated groups, tasks restart repeatedly |
Controllers and minions are easy to overlook because their symptoms surface elsewhere. If tables are healthy but nothing is *changing* — no new segments, no task progress — look at the controller.
* **Memory** — restarts, out-of-memory failures, pauses that look like timeouts, degradation that worsens gradually and resets on restart.
* **CPU** — latency that scales with load, no restarts, scheduler wait time rising while execution time stays flat.
* **Disk** — segment operations failing, ingestion stalling on commit, errors mentioning storage rather than compute.
Memory and CPU are frequently confused because both produce slow queries. The distinguishing signal is restarts: a component that keeps restarting is running out of memory.
## Out-of-memory conditions
Out-of-memory conditions almost always follow a change. Check for:
* A version upgrade — memory characteristics change between versions
* A new table, especially an upsert or dedup table
* Growth in an existing upsert table's primary-key count
* A change in query shape — new aggregations, larger result sets, joins that were not there before
* A change in data volume or retention
Upsert and dedup tables hold per-primary-key state in memory, and that state grows with distinct keys rather than with rows. A table that was comfortable at launch can exhaust memory months later with no change to its config.
Pinot exposes an API to estimate memory usage for an upsert table — use it to size before creating one, and to check what an existing one should be consuming against what it is. Consider off-heap options where they suit your workload: see [Off-heap upsert](/corecapabilities/manage-data/offheap-upsert) and [Off-heap dedup](/corecapabilities/manage-data/offheap-dedup).
Queries with large intermediate results — high-cardinality group-bys, large joins, big `DISTINCT` sets — allocate heavily. In `system_query_log`:
```sql theme={null}
SELECT logTimestampMs, tableName, timeUsedMs,
offlineTotalMemAllocatedBytes + realtimeTotalMemAllocatedBytes AS memAllocated,
numGroupsLimitReached, maxRowsInJoinReached, maxRowsInDistinctReached,
query
FROM system_query_log
WHERE logTimestampMs > now() - 3600000
ORDER BY memAllocated DESC
LIMIT 20
```
Queries hitting the limit flags are being trimmed — they are consuming as much as they are allowed to, which suggests they would consume more if permitted.
If analytical queries and latency-sensitive queries share the same servers, the analytical ones will periodically starve the others. StarTree supports isolating workloads so one class cannot exhaust the resources the other depends on — see [Replica group based workload isolation](/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation) and the [memory-throttled scheduler](/corecapabilities/query_data/advanced_operations/workload-isolation/memory-throttled-scheduler).
## High CPU
Rank recent queries by CPU:
```sql theme={null}
SELECT queryHash,
COUNT(*) AS executions,
AVG(timeUsedMs) AS avgLatencyMs,
SUM(offlineTotalCpuTimeNs + realtimeTotalCpuTimeNs) AS totalCpuNs,
ANY_VALUE(query) AS sampleQuery
FROM system_query_log
WHERE logTimestampMs > now() - 3600000
GROUP BY queryHash
ORDER BY totalCpuNs DESC
LIMIT 20
```
Aggregate CPU matters more than per-query CPU. A cheap query running constantly costs more than an expensive one running hourly, and only the grouped view shows that.
High CPU with high `numEntriesScannedInFilter` means the cluster is scanning data it should be skipping. That is an indexing problem wearing a capacity problem's clothes, and adding servers will not fix it. See [Queries failing or slow](/corecapabilities/query_data/troubleshooting#timeouts) and run [Query Analyzer](/corecapabilities/query_data/query-analyzer) on the worst offenders.
Rebalances, segment reloads, merges and purge tasks all consume CPU on the same servers that answer queries. If CPU rose without a change in query volume, check whether a maintenance operation is running. Check `TABLE_SEGMENT_ASSIGNMENT_CHECK` and `TABLE_SEGMENTS_RELOAD_CHECK` on the Health Dashboard.
## Disk and storage
* **Check what is actually stored.** The table Storage view in the Data Portal breaks down deep, local and tiered storage. See also [Table storage usage](/corecapabilities/manage-data/table-storage-usage).
* **Check retention is running.** `SEGMENT_RETENTION_CHECK` on the Health Dashboard. Data past retention that has not been removed consumes space it should not.
* **Check for segments that should not be there.** Failed or interrupted operations can leave segments behind that are no longer referenced but still occupy storage. If your segment count is much higher than your data volume suggests, this is worth raising.
* **Check whether you are near a segment-count limit.** [Segment count limits](/corecapabilities/manage-data/recipes/segment-count-limits#monitoring-and-troubleshooting) shows how to count where a table stands and what the cluster-wide picture looks like.
* **Check segment sizing.** `SEGMENT_SIZE_CHECK` and `SEGMENT_COUNT_CHECK` flag tables whose segments are too small or too numerous. Many small segments cost more than the same data in fewer, larger ones — in memory, in query fan-out, and in metadata overhead. See [Merge small segments](/recipes/merge-small-segments).
* **Consider tiered storage.** If most of your data is queried rarely, moving it to a cheaper tier reduces local pressure. See [Cloud tiered storage](/corecapabilities/manage-data/set-up-tiered-storage/motivation).
## Degradation after an upgrade
Several resource problems appear immediately after a version upgrade. When that is the sequence:
"This started after the upgrade" is a diagnosis-shortening fact. Note the previous version, the new version, and the exact time the symptom began.
See [StarTree release notes](/reference/startree-release-notes). If you skipped versions, read all of them — behaviour changes accumulate.
Upgrades can leave segments requiring reload before they behave as expected, and occasionally leave segments in an error state. Check [`TABLE_SEGMENTS_RELOAD_CHECK`](/corecapabilities/observability/health-checks#table_segments_reload_check) and [`IDEAL_STATE_EV_MISMATCH_CHECK`](/corecapabilities/observability/health-checks#ideal_state_ev_mismatch_check) on the Health Dashboard immediately after any upgrade.
A query that got slower after an upgrade may be planning differently. Compare scan counts before and after — if `numEntriesScannedInFilter` changed materially for the same query, the plan changed, and that is a more specific thing to report than "it got slower."
Capture a baseline before your next upgrade: p95 latency and scan counts for a handful of representative queries, current segment and document counts per table, and steady-state resource utilisation. Ten minutes beforehand turns "something feels slower" into a measurement.
## Planning capacity
To avoid the reactive version of all of the above:
* **Size before you create.** Estimate storage, and for upsert tables estimate primary-key memory, before adding a table to a tenant rather than after.
* **Know your headroom before a peak.** If you have a known traffic event, check utilisation against it in advance — scaling ahead of a peak is routine, scaling during one is not.
* **Watch upsert key growth.** It grows with distinct keys, not rows, and it does not shrink on its own.
* **Revisit sizing after workload changes.** New dashboards, new consumers and new query patterns change resource profiles without changing any config you own.
See [Scheduled server scaling](/corecapabilities/cluster-operations/scheduled-server-scaling) for scaling around predictable load, and [Query quotas](/corecapabilities/query_data/advanced_operations/query-quotas) for capping what any single workload can consume.
## Feature-specific guidance
For scaling operations that did not take effect, see [Scheduled server scaling → Troubleshooting](/corecapabilities/cluster-operations/scheduled-server-scaling#troubleshooting). External tables have their own operational failure modes: [External table troubleshooting](/corecapabilities/external-table/troubleshooting#queries) covers server OOM and pod restarts under query load, and [Operational guidance](/corecapabilities/external-table/operational-guidance#additional-troubleshooting) covers ZooKeeper packet-size limits as segment count grows. More at [Troubleshooting by feature](/corecapabilities/observability/troubleshooting-by-feature).
## Escalating
Include: which component, which resource, when it began, what changed, whether anything restarted, and your current tenant sizing. If queries are implicated, attach the CPU or memory ranking from `system_query_log`. See [what to collect](/corecapabilities/observability/troubleshooting#before-you-open-a-support-ticket).
# Best Practices and Configs
Source: https://docs.startree.ai/corecapabilities/external-table/best-practices-and-configs
The full config reference for External Tables — sync task, tier caching, server/cluster tuning, and per-query options — plus onboarding and performance best practices.
This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
Best practices for running External Tables, followed by the full reference for every configuration and query option: the table-level sync config, the S3 tier and caching properties, the server/cluster tuning knobs, and the per-query `SET` options.
Two 0.16.0 features have their own dedicated config reference: [Large External Tables](./segment-groups) (`segmentGroupConfig` and the cluster-level grouping gate) and [Deletion Vectors](./deletion-vectors) (`enableDeletionVectors` and the readiness/pruning config below).
## Best practices
**Onboarding**
* Enter the **bucket name only** (no `s3://`) and set the AWS **region in the connection config** — a region in config means you don't depend on the `AWS_REGION` environment variable.
* Grant the cluster access to the source bucket with an **assumed IAM role** (`roleArn` + optional `externalId`) or the cluster's node role; verify with `aws s3 ls s3:////` before onboarding.
* Keep the source bucket **in the same AWS region as your cluster**. Cross-region reads without an S3 interface endpoint go through the NAT Gateway and can add large AWS charges; see [Networking and data transfer costs](#networking-and-data-transfer-costs).
* For very large sources, **start with a smaller prefix subset** and scale the cluster before onboarding the full dataset.
* Set `continueOnFileError=true` if a sync run should **skip a file that fails to process** — unreadable, corrupt, or any other per-file failure, not just unreadable ones — instead of failing the whole run (default is to fail). Because it skips on *any* failure, it can silently drop more data than "unreadable file" suggests, so review sync logs after enabling it. This only affects files *during a sync* — it does not protect an already-ingested segment whose file is later deleted or compacted away (see the note below).
* **If your source compacts or rewrites files, onboard it through an Iceberg REST catalog, not the raw S3/GCS Data Lake source** — a REST catalog reconciles compaction via snapshot metadata, while raw sources pin segments to physical file paths that break when files are rewritten. See [Segments fail to load after upstream compaction](./troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction).
**Schema**
* **Keep the data types** the preview step infers; overriding them can break segment generation.
* Set a **time column** where the data has one — it enables time-based segment pruning and retention. Recommended but not mandatory; tables without a time column still work.
**Indexes**
* Add indexes for the columns you **filter and group by** — without them, queries scan remote data.
* For dictionary-backed indexes (inverted, FST, IFST), keep the forward index `RAW` and add an explicit **`dictionary` block**.
* **Avoid grouping by derived/computed columns** — it defeats segment pruning.
**Tiered storage — data & index caching**
* Enable the page cache (`enable.prefetch.page.cache`, a per-table tier setting) and ask your cluster operator to turn on `preload.enable` — a **server-level** setting that cannot be set per table — so index data is served locally. See [Caching and index pinning](#caching-and-index-pinning).
* **Pin small, hot indexes** (bloom filter, dictionary) with `preload.index.keys.override`; let the index cache serve the larger/colder ones.
* Turn on **index consolidation** to collapse a segment's per-column index files into one mmap, avoiding the OS `max_map_count` limit on wide tables — see [Caching and index pinning](#caching-and-index-pinning).
* Enable **pre-warm** (`pinot.parquet.prewarm.enabled`) to pay the cache-fill cost at segment load instead of on the first query.
**Large-scan queries (OOM protection)**
* For tables that get heavy scans or aggregations, enable the **query OOM killer** so a single runaway query is killed instead of crashing the server — see [Query OOM protection](#query-oom-protection-large-scans).
* Optionally enable the **pre-kill pause** so the server briefly pauses to let GC reclaim memory before killing — this can save a query that would otherwise be killed.
**Operations**
* Let the controller watcher schedule syncs; monitor health with the [observability endpoints](./observability).
***
## Networking and data transfer costs
External Tables and batch ingestion read data directly from S3, often in large volumes. Where the bucket sits relative to your StarTree cluster decides the network path those reads take, and that path decides what you pay AWS. This matters most for **BYOC** deployments, where the network charges land on your own AWS account.
| Bucket location | Network path | What you pay |
| - | - | - |
| **Same region as the cluster** | S3 gateway endpoint (StarTree creates it by default for BYOC on AWS) | No NAT Gateway or endpoint charges |
| **Different region, no endpoint** | Node → NAT Gateway → Internet Gateway → public S3 endpoint | NAT Gateway per-GB processing **plus** inter-region data transfer. This can be expensive for large datasets |
| **Different region, cross-region interface endpoint** | AWS PrivateLink to S3 in the source region | Interface endpoint hourly + per-GB processing **plus** inter-region data transfer. Usually much cheaper than NAT |
**Recommendations**
* **Co-locate the data and the cluster.** Keep source buckets in the same region as your StarTree cluster whenever you can. It is the cheapest and fastest setup.
* **Weigh how often the data is read.** For a one-time read, such as a single backfill, moving the data next to the cluster may not save much. Repeated reads of a large dataset are different: External Tables query S3 again and again, and the per-GB charges add up quickly.
* **Check the bucket region before onboarding.** Compare it with your cluster's region (`aws s3api get-bucket-location --bucket `). A gateway endpoint only covers buckets in its own region.
* **For cross-region buckets, add an S3 interface endpoint before you onboard.** Create a cross-region interface endpoint for S3 in your cluster's VPC so reads skip the NAT Gateway. See the AWS guides for [gateway endpoints for S3](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) and [cross-region PrivateLink support](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html). Contact StarTree support if you need help with the setup.
* **Watch NAT Gateway traffic during the first sync.** A spike in the NAT Gateway's CloudWatch byte counters during an initial onboarding usually means S3 reads are leaving the private network.
***
## Caching and index pinning
External Table queries read indexes from remote storage, so keeping index data local is what makes them fast. There are two ways to do that — **pinning** and the **index cache** — and they're used together.
| | Pinning (preload) | Index cache |
| - | - | - |
| Enabled by | `preload.enable=true` (**server setting**, see [Preload (server-level)](#preload-server-level)) + `preload.index.keys.override` (table setting) | `enable.prefetch.page.cache=true` (table setting) |
| What it does | Downloads the listed index files into local mmap at segment load and keeps them **resident** | Serves index byte ranges through an **on-demand, evictable** (LRU) cache |
| When it loads | Eagerly, at segment load | Lazily, on first access |
| Eviction | Never — always local | Evicted under memory pressure |
| Memory cost | Fixed and predictable | Elastic, bounded by the cache size |
**Pin the small, always-needed indexes; let the cache handle the rest.** Pinning is resident memory, so reserve it for compact indexes you hit on most queries — **bloom filters, dictionaries**, and small range/inverted indexes. Everything larger or colder (JSON, text, wide columns) should go through the index cache, which fetches on demand and evicts what isn't used.
`preload.enable` is a **server / cluster setting**, not a table config key — it cannot be set in `tierBackendProperties`. Putting it there is silently accepted and does nothing; even StarTree's own onboarding flow currently writes it into new table configs, where it has no effect. Ask your cluster operator to turn it on server-side. Until it's on, listing keys in `preload.index.keys.override` pins nothing.
```json theme={null}
"tierBackendProperties": {
"preload.index.keys.override": "*.bloom_filter,*.dictionary,*.range_index",
"enable.prefetch.page.cache": "true"
}
```
Pin keys are `.` (or `*` for all columns), where `indexKey` is the index's file key — e.g. `inverted_index`, `range_index`, `bloom_filter`, or `dictionary`. Examples: `payment_type.inverted_index`, `*.bloom_filter`, `*.dictionary`.
**Index consolidation.** On a wide table, each column's index is a separate memory-mapped file, and many segments × many columns can exhaust the OS memory-map limit (`max_map_count`) — surfacing as native-memory / mmap allocation failures and server restarts. Set `preload.enable.index.consolidation=true` to pack a segment's index regions into a single file per segment, drastically cutting the mmap count. Onboarding enables this by default.
***
## Where configs live
| Scope | Set in | Applies to |
| - | - | - |
| **Table config — `ExternalTableSyncTask`** | `task.taskTypeConfigsMap.ExternalTableSyncTask` in the table config | Catalog connection, scheduling, snapshot processing. |
| **Tier backend properties** | `tierConfigs[].tierBackendProperties` (S3 tier) | Per-table caching and preload behavior. |
| **Server / cluster config** | Helix cluster config or server `PinotConfiguration` | Page cache sizing, pre-warm, segment cache. |
| **Query option** | `SET key = 'value'` per query | Per-query cache/prefetch behavior. |
Server and cluster keys resolve in the order **cluster config (Helix) → JVM `-D` property → built-in default**. Prefer cluster config so values can change without a restart. **Exception:** in current releases the `pinot.parquet.page.cache.*` and `pinot.parquet.pagereader.*` keys (and `pinot.server.index.inverted.enable.startree.reader`) are **cluster-config-only** — their JVM `-D` fallbacks were removed.
***
## Table config — `ExternalTableSyncTask`
### Catalog & mode
| Key | Default | Description |
| - | - | - |
| `catalogType` | *(required)* | `s3`, `iceberg-rest`, or `gcs-interop`. Selects the catalog provider. |
| `executor` | *(none — omitting it silently falls back to the deprecated minion path)* | **Always set `controller`.** The minion executor path is legacy and unsupported for the current feature set — snapshot processing, the observability endpoints, ingestion-time segment grouping, and IAM assume-role auth all require the controller path. There is no default that pushes new tables toward `controller`: a hand-written config that omits this key runs on minion without any error or warning. Onboarding sets it automatically, so this only bites hand-written configs. |
| `enabled` | `true` | Enable/disable the per-table sync. |
| `forceConfig` | `false` | Bypass external-table config validation **errors** (RAW encoding, unsupported index types, transform configs) and proceed anyway. Use only when you understand why validation is failing. |
External Tables read **Parquet files only**. There is no `inputFormat` config — the sync engine doesn't read one, so setting any such key has no effect.
`glue` is also accepted as a `catalogType` value when a table is created, but sync then fails at runtime — it is not a supported catalog type. Onboard AWS Glue-backed sources with `catalogType=iceberg-rest` and the Glue service type under `catalog.iceberg-rest.*` instead; see [AWS Glue: Onboarding via API](/corecapabilities/external-table/glue/onboarding-api#step-1-validate-and-browse-the-connection).
### Connection keys
The catalog connection keys are prefixed `catalog.s3.*` or `catalog.iceberg-rest.*` (REST URI, service type — `glue`, `s3Tables`, `unity`, or `rest` (Nessie and other generic Iceberg REST catalogs) — warehouse/prefix, namespace/table, and `auth.rest.*` / `auth.storage.*` credentials). The full list with examples is in [AWS Glue: Onboarding via API](/corecapabilities/external-table/glue/onboarding-api#step-1-validate-and-browse-the-connection), [Unity Catalog: Onboarding via API](/corecapabilities/external-table/unity/onboarding-api#step-1-validate-and-browse-the-connection) for Unity-specific auth, or [Nessie: Onboarding via API](/corecapabilities/external-table/nessie/onboarding-api#step-1-validate-and-browse-the-connection) for the generic `rest` adapter.
### Scheduling & task sizing
| Key | Default | Description |
| - | - | - |
| `schedule` | — | Quartz cron for sync frequency, e.g. `0 */5 * * * ?`. |
| `desiredSegmentSize` | `500MB` | Reserved for future use — currently has no effect. Each sync run generates one Pinot segment per source Parquet file; there is no size-based segment splitting or merging today. |
| `taskMaxNumFiles` | `100` | Max source files per task run (minion mode). |
| `taskMaxDataSize` | `1GB` | Max source bytes per task run (minion mode). |
| `tableMaxNumTasks` | *(unbounded)* | Not read by the External Table task generator itself; on ad-hoc `/tasks/schedule` triggers the shared task framework applies it, with an effective default of unbounded. (The Data Export Task reads the same key name with a different default — `10000`.) |
| `maxParallelOps` | *(CPU cores)* | Parallelism within a single run. |
| `continueOnFileError` | `false` | Skip an entire file that fails to process, **for any reason** — not limited to unreadable files — instead of failing the run. Because it's not scoped to a specific failure type, it can silently drop more data than "skip an unreadable file" implies. Applies only *during a sync run* — see the note below. Distinct from `continueOnError`. |
| `catalog.s3.startAfter` | *(unset)* | New in 0.16.0. Seeds the lower bound for the **very first** S3 listing call (`ListObjectsV2`'s `startAfter`), as a bucket-relative object key. Only used before any checkpoint exists — once a checkpoint is recorded, it's permanently ignored. Takes precedence over `catalog.s3.startFromLexLatestPartition`. |
| `catalog.s3.maxFilesToList` | `1000` | Max objects fetched per listing call for raw S3/GCS Data Lake sources (the per-run batching knob for `catalogType=s3`/`gcs-interop`; the Iceberg path batches by `iceberg.snapshotProcessing.batchSize` instead). |
| `catalog.s3.startFromLexLatestPartition` | `false` | Start the first listing from the lexicographically latest partition instead of the beginning of the prefix. Ignored when `catalog.s3.startAfter` is set. |
Don't use `catalog.s3.prefix` as a way to bound where ingestion *starts* within a large bucket — `prefix` is a permanent filter on which objects are ever considered, not a one-time starting cursor. Narrowing it to "skip ahead" permanently excludes everything outside that range, including future files, once the checkpoint reaches the end of the prefix. Use `catalog.s3.startAfter` instead, which only affects the first run.
**`continueOnFileError` vs. `continueOnError` — two different configs.** `continueOnFileError` (this `ExternalTableSyncTask` config) skips an **entire file that fails to process, for any reason** — not just unreadable ones — so one bad file doesn't fail the run. `continueOnError` (an OSS field in the table's `ingestionConfig`, not here) skips **individual bad rows** while reading a file or stream. Both apply only *during ingestion* — neither one protects a segment that is already ingested. In particular, if a source file is deleted or [compacted away upstream](./troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction) after its segment exists, the server fails to serve that segment regardless of either flag; the durable fix for compacting sources is an Iceberg REST catalog.
### Snapshot processing (Iceberg)
| Key | Default | Description |
| - | - | - |
| `iceberg.snapshotProcessing.initialVersionSelector` | `LATEST` | Where to start: `LATEST`, `EARLIEST`, or `SNAPSHOT_ID`. |
| `iceberg.snapshotProcessing.initialSnapshotId` | — | Snapshot ID to start from (when selector is `SNAPSHOT_ID`). |
| `iceberg.snapshotProcessing.batchSize` | `10000` | Max new files ingested per run. |
| `iceberg.snapshotProcessing.maxFilesScannedPerRun` | `100000` | Upper bound on files enumerated per run (≥ `batchSize`). |
| `iceberg.snapshotProcessing.maxRetries` | `3` | Consecutive failures before a snapshot is marked `FAILED`. |
| `iceberg.snapshotProcessing.retryFailedSnapshot` | `false` | Keep re-arming a `FAILED` snapshot from its last cursor on every cycle instead of abandoning it after `maxRetries`. See [Sync says COMPLETED but no data lands](./troubleshooting#sync-says-completed-but-no-data-lands). |
| `iceberg.snapshotProcessing.lagMaxDiffFiles` | `100000` | Cap on the manifest set-diff used by the exact ingestion-lag calculation (`?includeLag=true&pendingExact=true`). Beyond it, lag degrades to summary-based estimates with an explanatory `message`. Only affects the numbers returned by the [lag/status endpoint](./observability#ingestion-lag) — it has no effect on ingestion itself. |
### Checkpointing & schema evolution
| Key | Default | Description |
| - | - | - |
| `enableCheckpointing` | *(always on)* | Declared but not currently read — checkpointing is unconditionally enabled for `ExternalTableSyncTask`; this key has no effect. |
| `checkpointTTLSec` | *(unused)* | Declared but not currently read for this task type. |
| `schemaEvolution.enabled` | `false` | Refine the Pinot schema to the source schema before each sync. |
| `schemaEvolution.failFast` | `false` | Abort (vs. log and continue) when a schema change is rejected. |
| `schemaEvolution.typeWidening.enabled` | `false` | Apply a supported source-side type widening (`INT`→`LONG` or `FLOAT`→`DOUBLE`) instead of rejecting it. Requires `schemaEvolution.enabled=true`; takes effect on the periodic-watcher path only (not snapshot-sync). See the note below. |
| `storeNestedSchema.enabled` | `true` | Stamp the source's nested-schema metadata onto complex (JSON) columns so field names **inside** `struct` / `array` / `map<_,struct>` values resolve to your logical names. Only matters for Delta UniForm / Unity Catalog tables that use column mapping — see the note below. |
**How schema evolution resolves changes (0.16.0+).** When `schemaEvolution.enabled=true` on a `catalogType=iceberg-rest` source, each sync unions the current schema against the source's Iceberg field-id/alias history instead of matching by column name alone. This means a **source column rename is tracked as an alias** rather than dropping and re-adding a column — the existing Pinot column name is preserved. Resolution only covers **top-level** fields (nested-field renames aren't resolved). A detected **type widening** is rejected by default, unless `schemaEvolution.typeWidening.enabled=true` opts the table into applying it instead — set `schemaEvolution.failFast=true` if you want a rejection (widening or otherwise) to abort the run instead of being logged and skipped. When fail-fast aborts a run, the [status API](./observability#run-status) reports `failurePhase: UPDATING_SCHEMA`.
**Type widening (opt-in).** `schemaEvolution.typeWidening.enabled=true` lets a table adopt a source-side widening instead of rejecting it — `INT`→`LONG` and `FLOAT`→`DOUBLE` are the supported widenings; anything else is still rejected. It applies only on the **periodic-watcher path** (an `executor=controller, watcher.enabled=true` table) — snapshot-sync ingestion skips schema evolution entirely and keeps the registered schema unchanged. Widening is safe for External Tables specifically: a segment's forward index *is* the backing Parquet file, so adopting the wider type is a metadata-only edit, not a physical rewrite. It defaults to off because applying it opens a transient window, until every segment has reloaded onto the new type, where a query spanning both generations of segments on the widened column can fail (`ORDER BY` / `GROUP BY` / `DISTINCT`) or drop mismatching rows (a plain `SELECT`) — aggregations like `SUM` / `COUNT` are unaffected. Enabling it is a deliberate choice to accept that window during rollout.
`FLOAT`→`DOUBLE` is lossless but **changes what already-ingested rows print**: the float nearest `0.1` reads back as `0.10000000149011612` once served as a double, because that is the exact value the 4-byte encoding always held — only the rendering changes, and the value narrows back to the same float. An Iceberg-native engine over the same table prints the same digits, since Iceberg performs the identical JVM widening.
**Decimal precision widening needs no flag.** Iceberg permits only a precision increase at fixed scale, and `BIG_DECIMAL` (Pinot's only decimal type) carries no precision or scale of its own, so a decimal precision change produces an identical registered schema — there's no diff for `schemaEvolution.typeWidening` to force or reject. The physical Parquet encoding still varies by precision (`decimal(9,2)` in an `INT32`, `decimal(18,2)` in an `INT64`, wider in a `FIXED_LEN_BYTE_ARRAY`), and a widened table can hold a mix; each file's own footer resolves the scale on read.
**Nested logical names under column mapping (`storeNestedSchema.enabled`).** Delta UniForm tables created with `delta.columnMapping.mode` (`name` or `id`) don't store your column names in the Parquet files — each physical column is an opaque `col-`, and the logical name plus a stable field-id live only in table metadata. **Top-level** column names are always resolved back to logical names, with or without this flag. Names **inside** a complex value are only resolved when `storeNestedSchema.enabled` is on (the default) for the table's `ExternalTableSyncTask` config; set it to `false` to opt out, and nested keys are then stored under their physical names, so `JSON_EXTRACT_SCALAR(customer, '$.name', ...)` returns `NULL` even though the data is there. Only an explicit, case-insensitive `false` opts out — an unset, blank, or unrecognized value keeps the default on.
The flag applies to every schema-writing path (onboarding, preview, and the schema-evolution union during sync) and is independent of `schemaEvolution.enabled`. It defaults to on because leaving it off silently materializes the wrong JSON keys for complex columns on column-mapped tables; the only cost of stamping is a larger ZK-persisted schema on tables that don't need it, and it does nothing for native Iceberg or `add_files` tables where physical names already equal logical names. Changing it on an already-onboarded table takes effect only on the **next schema write** (re-preview/re-register, or the next sync if schema evolution is on), followed by a segment reload — resolution happens at read time, so segments don't need regenerating. A table onboarded before this default flipped gets that one-time schema update and segment reload automatically on its first sync after upgrade, if it has complex columns and no explicit `storeNestedSchema.enabled` setting. See [Unity Catalog — schema options for column mapping](./unity/onboarding-api#schema-options-for-column-mapping).
***
## Tier backend properties (caching & preload)
Set on the S3 tier in `tierConfigs[].tierBackendProperties`. The onboarding/preview flow sets sensible defaults.
**Two-tier scaffold for `skipSegmentPreprocess=true` (create-time only).** If the table config already sets `indexingConfig.skipSegmentPreprocess=true` when it's created — through the preview API or `CREATE TABLE ... WITH (type='iceberg', ...)` — the enrichment scaffolds **two** tiers instead of the usual single fixed-selector tier: `_preprocessed_tier` (`segmentAge=2h`) and `_s3_tier` (`segmentAge=0s`), both keyed on `creationTime`. They share the same bucket and region but get distinct `pathPrefix` values (the preprocessed tier appends `/preprocessed`), so a segment aging from one tier to the other is a real physical move, and matching `indexingConfig.tierOverwrites` entries are written so segment preprocessing runs on the preprocessed tier and stays skipped on the s3 tier. New segments land on the s3 tier and relocate to the preprocessed tier once they pass the 2-hour threshold. This applies only at create time: adding `skipSegmentPreprocess=true` to an existing table via `PUT /tables//config` does not scaffold the second tier, and turning the flag back off does not remove one already created.
### Common
| Key | Default | Description |
| - | - | - |
| `bucket` / `pathPrefix`² / `region` | *(required)* | S3 location of the tier storage. |
| `enable.prefetch.page.cache` | `false`¹ | Route this table's index reads through the page cache. |
| `preload.enable.index.consolidation` | `false`¹ | Pack a segment's index regions into one file to reduce mmap count on wide tables. |
| `enable.delegate.v2` | `true` | Use the v2 delegate segment format. Already on by default — onboarding also writes this key, but that has no effect since it was already `true`. |
| `enable.startree.index` | `false` | Load StarTree index files for this tier. **Off by default** — while it's off, a StarTree index built on an external table is generated at index-build time but never used to serve queries. Turn this on if the table has (or will have) a star-tree index. |
¹ The onboarding/preview flow sets these to `true` on **newly scaffolded** tables (`preload.enable.index.consolidation`, `enable.prefetch.page.cache`). Existing tables are never re-enriched — if a table predates these defaults, add the keys manually.
² `pathPrefix` is not validated. Nothing errors if it's missing or wrong — files simply land at the bucket root instead of under a per-table prefix. Always set it explicitly so each table's files get their own folder.
`preload.enable`, `preload.dir`, `preload.total.size`, and `preload.load.existing.buffers` are **not** tier/table settings, even though they sound like they belong here next to `preload.enable.index.consolidation`. They are server / cluster settings — see [Preload (server-level)](#preload-server-level) below.
### Preload tuning
| Key | Default | Description |
| - | - | - |
| `preload.index.keys.override` | *(empty — nothing is preloaded until you set this)* | Comma-separated index keys to preload for **this table**, e.g. `*.inverted_index,*.range_index`. This comma-separated format is **table-config-only** — the server/cluster-level equivalent of this setting uses a different key and format, so don't copy this syntax into a server config. Preloading also requires the server-level `preload.enable` to be on; with it off, this list pins nothing. |
The S3 tier also exposes advanced on-demand buffer, mmap, and read-ahead sub-properties (`ondemand.*`, `mmap.*`, `readAhead.*`). These are internal tuning knobs — leave them at defaults unless StarTree support advises otherwise.
***
## Server / cluster configs
### Parquet page cache sizing (in-memory)
| Key | Default | Description |
| - | - | - |
| `pinot.parquet.page.cache.memory.data.size.mb` | `256` | In-memory data page cache size. |
| `pinot.parquet.page.cache.memory.dict.size.mb` | `64` | In-memory dictionary page cache size. |
| `pinot.parquet.page.cache.prefetch.size.mb` | `1024` | Prefetch buffer size (decoded hot pages). |
These in-memory pools are a separate concern from the persistent disk cache below — they don't share a namespace or a budget with it.
### Pre-warm at segment load
| Key | Default | Description |
| - | - | - |
| `pinot.parquet.prewarm.enabled` | `false` | Pre-warm the first data page of each column when a segment loads. |
| `pinot.parquet.prewarm.max.inflight.per.table` | `64` | Per-table cap on concurrent pre-warm reads. |
| `pinot.parquet.prewarm.max.bytes.inflight.per.table` | `64MB` | Per-table cap on concurrent pre-warm bytes. |
| `pinot.parquet.prewarm.max.inflight.per.table.override` | `""` | Per-table overrides: `table1,N;table2,M`. |
### Page reader & prefetch
| Key | Default | Description |
| - | - | - |
| `pinot.parquet.pagereader.read.timeout.ms` | `30000` | Timeout for a single page S3 GET. **Cluster-config only** — despite appearing in older docs as a query option, it cannot be overridden per query; setting it with `SET` has no effect. |
| `pinot.parquet.pagereader.max.in.flight.chunk.reads` | `0` | Cap on concurrent chunk reads (`0` = unbounded). |
| `pinot.parquet.pagereader.pool.size` | `max(4, 2×cores)` | Thread-pool parallelism for async page reads. |
| `pinot.parquet.prefetch.deserialize` | `true` | Decode prefetched pages to values (not just raw bytes). |
| `pinot.parquet.prefetch.segment.lookahead.enabled` | `false` | Warm upcoming segments' chunks ahead of time. |
| `pinot.parquet.prefetch.segment.lookahead.max.inflight.chunks` | `12` | Server-wide cap on concurrent look-ahead chunk reads. |
| `pinot.parquet.reader.lazy.enabled` | `false` | Build the per-column Parquet forward-index reader lazily, on first use, instead of eagerly at segment load. Lazy readers cap concurrently-built readers via a pool so a server's segment capacity stops scaling with table width; the tradeoff is a first-read cost. Applies to scalar columns only — `JSON`/`MAP` columns keep their own reader path regardless. Ignored (readers stay eager) when pre-warm or a build/reload scope is active. Takes effect on the next segment load or reload, not for already-loaded segments. |
**Prefetch depth** is bounded by the query option `prefetch.projection.queue.size` (default `10`) — how many projection blocks are fetched ahead when the scan queue refills. The other prefetch-related knobs are the prefetch buffer size (`...prefetch.size.mb`), the look-ahead in-flight cap (`...segment.lookahead.max.inflight.chunks`), the in-flight chunk-read cap (`...max.in.flight.chunk.reads`), and the page-reader pool size (`...pagereader.pool.size`).
### Parquet read modes (PAGE / CHUNK)
Explicit physical-read mode for Parquet V6 query prefetch, letting a per-page range-read strategy be compared against a whole-column-chunk strategy without an adaptive policy choosing between them.
| Key | Default | Description |
| - | - | - |
| `pinot.parquet.prefetch.read.mode` | `PAGE` | `PAGE` reads each selected compressed page range. `CHUNK` reads each selected column chunk once and slices the response into the existing page-cache entries. Invalid values fail closed to `PAGE`. |
| `pinot.parquet.prefetch.read.chunk.enabled` | `true` | Immediate CHUNK kill switch — `false` forces new and queued prefetch work to `PAGE` even when the mode is `CHUNK`. Does not cancel a storage GET already issued. |
| `pinot.parquet.prefetch.read.max.inflight.requests` | `32` | JVM-wide physical-request admission limit shared by both modes. |
| `pinot.parquet.prefetch.read.max.inflight.bytes` | `67108864` | JVM-wide compressed-byte admission limit shared by both modes. A CHUNK larger than this limit falls back to `PAGE`. |
| `pinot.parquet.prefetch.read.page.coalesce.enabled` | `true` | PAGE mode only. Merges nearly-adjacent selected pages of one chunk into single storage range reads, additionally covering the page-header gaps between them (discarded when pages are sliced back out). Each merged range still stays within `...max.inflight.bytes`. On by default — merged reads request slightly more bytes than the selected pages alone (the bridged headers), a trade that cuts request counts to whole-chunk-read parity. Setting the switch to anything but `true`, including an invalid value, restores strictly per-page reads; no restart required. |
| `pinot.parquet.prefetch.read.page.coalesce.gap.bytes` | `4096` | Largest unselected byte gap a coalesced PAGE run may cover. `0` merges only strictly byte-adjacent pages. |
These are **cluster-config only**, not query options. `PAGE` remains the recommended default — StarTree's own benchmarks found no universal crossover point, and `CHUNK` can transfer significantly more bytes than needed on selective queries. Canary `CHUNK` only for scan-heavy, request-constrained workloads, and monitor the per-mode GET count, physical/useful bytes, and amplification metrics.
### Filter-ahead prefetch
Default-off stage that evaluates the query filter ahead of scanners and warms projection pages for matched rows on external Parquet scans. It is best effort — query execution always performs the authoritative filter regardless of prefetch outcome.
| Key | Default | Description |
| - | - | - |
| `pinot.parquet.prefetch.filter.ahead.enabled` | `false` | Enable the stage. SQL query option or cluster config. A cluster value of `false` (case-insensitive) is a fleet-wide hard stop the query cannot override; otherwise the query option wins when set, so a query can opt in when the cluster value is absent or opt out when the cluster default is `true`. |
| `pinot.parquet.prefetch.filter.ahead.depth` | `8` | Max segments filtered concurrently. SQL query option or cluster config. A positive cluster value sets both the server default and the ceiling for the query option — a query may lower it but not raise it. |
| `pinot.parquet.prefetch.filter.ahead.max.outstanding.futures` | `128` | Per-query sliding window of outstanding filter-ahead prefetch futures. Same cluster-value-as-ceiling behavior as `...depth`. |
| `pinot.parquet.prefetch.filter.ahead.max.active.filters` | `128` | **Cluster-config only** — not query-settable. Fair per-server ceiling on concurrent speculative filter-ahead segment lifecycles. Resizes live; invalid or removed values restore the default, and numeric values are clamped to at least `1`. |
| `pinot.parquet.prefetch.filter.ahead.max.inflight.chunks` | `128` | **Cluster-config only** — not query-settable. Per-server storage-read ceiling for filter-ahead warms, resized live the same way as `...max.active.filters`. |
### Persistent disk cache (`pinot.server.instance.disk.cache.*`)
**Breaking change (0.16.x).** Every persistent disk cache for External Tables and tiered storage — Parquet page cache, segment metadata cache, preload, mmap, and the deletion-vector Puffin cache — now shares one config namespace, `pinot.server.instance.disk.cache.*`, and one root directory. The old `pinot.parquet.page.cache.disk.*`, `pinot.server.instance.segment.cache.*`, and `pinot.server.instance.buffer.reference.manager.*` (preload/mmap) keys, and the table-level `iceberg.dv.fetch.mode` / `iceberg.dv.puffinCache.*` keys (see [Deletion Vectors](./deletion-vectors)), are retired — not read, not aliased. Migrate before upgrading. All settings below take effect at server startup only; changing or deleting one requires a restart.
The table below omits the shared `pinot.server.instance.disk.cache.` prefix. Sizes accept bytes or binary units (`K`/`M`/`G`/`T`/`P`, optionally followed by `B`); the page-capacity and fragment-size keys also accept a percentage of filesystem capacity, and `0` disables that pool or layer.
| Key | Default | Description |
| - | - | - |
| `enabled` | `true` | Global switch for every managed disk cache (Parquet pages, segment metadata, preload, mmap, Puffin). `false` leaves remote reads and the in-memory page cache above available. |
| `directory` | `pinot.server.instance.dataDir` | Single root for every managed cache (previously several independent directories). |
| `pages.dict.max.size` | `15%` per consumer | Dictionary-page capacity. `0` disables this pool. |
| `pages.data.max.size` | `15%` per consumer | Data-page capacity. `0` disables this pool. |
| `pages.fragment.size.mb` | `32` (Parquet pages) / `512` (segment index pages) | Fragment size in MiB, capped at `2047`. Append `.PARQUET_INDEX` or `.SEGMENT_INDEX` to size the two pools independently, e.g. `pages.dict.max.size.SEGMENT_INDEX`. |
| `pages.snapshot.enabled` | `true` | Persist/restore page snapshots across restarts (default flipped from `false`). |
| `preload.max.size` | `100G` | Maximum tracked preloaded-index bytes. `0` disables the layer at startup. Gates whether any table's `preload.index.keys.override` (tier-level) has any effect. |
| `preload.index.keys` | *(empty)* | Comma-separated index keys to preload globally, e.g. `col.forward_index,col.inverted_index`. |
| `preload.index.keys.override` | *(empty)* | Semicolon-separated `table_OFFLINE,col.forward_index,col.inverted_index` groups, overriding the global selection per table — the server/cluster-level equivalent of the tier's `preload.index.keys.override` (different key, different format; don't copy that syntax here). |
| `preload.index.consolidation.enabled` | `false` | Consolidate selected indexes into fewer local files. |
| `mmap.max.size` | `100G` | Maximum buffer bytes admitted by LFU membership. `0` disables the layer at startup. |
| `puffin.max.size` | `10G` | Soft LRU budget per table/URI-scheme for the deletion-vector Puffin cache (see [Deletion Vectors](./deletion-vectors)). `0` uses ranged reads (or stream-skip where ranges are unavailable) instead of caching whole files. |
| `puffin.download.timeout.ms` | `600000` | Timeout for a Puffin cache download. |
Fixed cache locations under `` (the resolved `directory`): External Table Parquet pages at `remote_data_cache`, tiered-storage index pages at `index_cache`, segment metadata/index headers/Parquet page indexes at `tieredStorage/segmentCache`, preloaded indexes at `tieredStorage/preload`, LFU mmap indexes at `tieredStorage/mmap`, and Puffin files at `dv-puffin-cache//`. Disk caching and restart reuse (files survive a graceful shutdown and are reused on the next start) are both on by default; to force a cold rebuild of one cache, stop the server and remove only that cache's directory, then restart — don't remove the whole data directory.
### Index reader
| Key | Default | Description |
| - | - | - |
| `pinot.server.index.inverted.enable.startree.reader` | `false` | Use an optimized inverted-index reader for cloud-stored (external table) segments. This speeds up **regular inverted indexes** — "StarTree" in the key name refers to the company, not the [StarTree index](/corecapabilities/manage-data/indexes/startree-index) feature. Flipping it does not affect star-tree index behavior. |
### Query OOM protection (large scans)
A query that scans a large amount of remote data can grow the server heap until the process OOMs. Server-side per-query memory accounting protects against this: when heap usage crosses a threshold, the most memory-hungry query is killed (or briefly paused first) instead of the whole server crashing. Recommended for tables that get heavy scans or aggregations.
On **StarTree Cloud**, `pinot.query.scheduler.accounting.enable.thread.memory.sampling` and `pinot.query.scheduler.accounting.oom.enable.killing.query` are already turned on by default for servers — your cluster is protected without any action. The steps and settings below matter only for **self-managed** clusters that haven't enabled them.
| Key | Default | Description |
| - | - | - |
| `pinot.query.scheduler.accounting.enable.thread.memory.sampling` | `false` | Track per-query memory. **Required** for memory-based killing to work. |
| `pinot.query.scheduler.accounting.oom.enable.killing.query` | `false` | Enable the OOM killer — kill the biggest query when heap is critical, instead of crashing the server. |
| `pinot.query.scheduler.accounting.oom.critical.heap.usage.ratio` | `0.96` | Heap ratio at which the heaviest query is killed. |
| `pinot.query.scheduler.accounting.oom.panic.heap.usage.ratio` | `0.99` | Heap ratio for panic — kill even without per-query sampling. |
| `pinot.query.scheduler.accounting.oom.alarming.usage.ratio` | `0.75` | Heap ratio at which usage is flagged (logging / back-pressure). |
| `pinot.query.scheduler.accounting.oom.panic.allow.pre.query.kill.pause` | `false` | **OOM pause:** at panic level, pause briefly to let GC reclaim memory *before* killing — can save a query that would otherwise be killed. |
| `pinot.query.scheduler.accounting.oom.pre.query.kill.pause.duration.ms` | `-1` | How long the pre-kill pause waits (ms). `-1` disables it. |
To protect a cluster running large External Table scans, set (full keys, all prefixed `pinot.query.scheduler.accounting.`): `...enable.thread.memory.sampling=true` and `...oom.enable.killing.query=true`. Add the pre-kill pause (`...oom.panic.allow.pre.query.kill.pause=true` with `...oom.pre.query.kill.pause.duration.ms=2000`) so transient spikes recover via GC instead of killing the query.
***
## Query options
Set per query with `SET "key" = 'value'`. Several of these are also tier or server configs — setting them as a query option overrides the config for that one query.
| Query option | Default | Also a config? | Effect |
| - | - | - | - |
| `enable.prefetch.page.cache` | `false` | Tier + server | Route index reads through the page cache for this query (overrides read-ahead). |
| `prefetch.projection.queue.size` | `10` | — | Projection blocks prefetched ahead on a scan queue refill (prefetch depth). |
| `readAhead.enable` | `false` | Tier | Enable read-ahead buffering for index reads on this query. |
| `skip.remote.table.cache` | `false` | Tier | Bypass the Parquet **page cache** — the cache that holds fetched remote-table data pages in memory/on disk — for both reads and writes. Covers **data pages only**. To measure a genuinely cold query, also set `skip.columnIndex.cache`; index reads otherwise still come from the local cache and the result will look faster than a real cold run. |
| `skip.columnIndex.cache` | `false` | Tier | Bypass the column index cache for this query. Pair with `skip.remote.table.cache` for a truly cold-cache measurement. |
| `snapshotVersionByTable` | — | — | Time-travel: pin the query to specific Iceberg snapshots, as a JSON map `{"table": snapshotId}`. See [Time travel](#time-travel-query-time-snapshot-pinning). |
| `concurrentProjectionPrefetch` | `true` | — | Concurrent per-column prefetch for wide projections (`SELECT *`). Set `'false'` as a kill switch if a wide scan misbehaves. |
| `pinot.parquet.pagereader.max.in.flight.chunk.reads` | `0` | Server | Per-query cap on concurrent chunk reads. |
| `pinot.parquet.prefetch.segment.lookahead.enabled` | `false` | Cluster | Enable segment look-ahead warming for this query. |
| `pinot.parquet.prefetch.filter.ahead.enabled` | `false` | Cluster | Enable [filter-ahead prefetch](#filter-ahead-prefetch) for this query. A cluster value of `false` is a fleet-wide hard stop this option cannot override. |
| `pinot.parquet.prefetch.filter.ahead.depth` | `8` | Cluster | Max segments filter-ahead filters concurrently for this query. Can lower, not raise, the cluster-set ceiling. |
| `pinot.parquet.prefetch.filter.ahead.max.outstanding.futures` | `128` | Cluster | Per-query cap on outstanding filter-ahead prefetch futures. Can lower, not raise, the cluster-set ceiling. |
| `enable.tiered.storage.trace` | `false` | — | Collect a tiered-storage trace for this query. The trace is written to **server logs only** — set the standard `trace` query option **together with** this one if you also want it back in the query response. |
**Example — bypass the cache for one query:**
```sql theme={null}
SET "skip.remote.table.cache" = 'true';
SELECT count(*) FROM nyc_taxi_trips WHERE fare > 100;
```
***
## Time travel (query-time snapshot pinning)
By default, a query against an Iceberg external table reads the **newest fully-synced snapshot**, resolved per query at the broker. The `snapshotVersionByTable` query option pins the query to a specific snapshot instead — the broker routes it to exactly that snapshot's file set, so you get the data (and row-level deletes) as of that snapshot:
```sql theme={null}
SET "snapshotVersionByTable" = '{"nyc_taxi_trips_OFFLINE": 1234567890123456789}';
SELECT count(*) FROM nyc_taxi_trips;
```
The value is a JSON map of table name → snapshot ID, so a single query can pin several tables (one entry per table). Table names are canonicalized — `nyc_taxi_trips`, `nyc_taxi_trips_OFFLINE`, and `mydb.nyc_taxi_trips` all resolve to the same entry. The option works on both the single-stage and multi-stage query engines.
### Caveats
* **Snapshot IDs only, no SQL clause.** There is no `FOR VERSION AS OF` / `FOR TIMESTAMP AS OF` syntax and no timestamp-based variant. Pass the numeric Iceberg snapshot ID (as a JSON number or numeric string) via the query option.
* **Bounded time-travel window.** Only snapshots still in the table's **active snapshot list** can be pinned. [Snapshot retention](./snapshot-retention) keeps the newest `iceberg.snapshotProcessing.retention.maxActiveVersions` versions (default **2**). A configured value below **2** is ignored outright and the default of 2 — also the floor — is used instead. This is *not* arbitrary Iceberg-history time travel — it is "pick one of the last N snapshots StarTree has synced and still retains."
* **Iceberg catalog sources only.** Supported for Iceberg REST catalog sources (AWS Glue, S3 Tables, Unity Catalog, generic REST). On raw S3/GCS Data Lake tables the option is **silently ignored** — file listings have no snapshot concept.
* **Expired or unknown snapshots fail the query.** Pinning a snapshot that is not (or no longer) active returns an error (`Iceberg snapshot is not active for table `) rather than silently falling back to the latest snapshot. Malformed JSON in the option also fails the query.
* **No pin = latest ready snapshot.** Without the option, each query reads the newest snapshot that is fully synced and cache-ready at that moment. Two queries issued at different times may therefore read different snapshots; pin a snapshot when you need repeatable reads across queries.
* **On some tables the broker sets it automatically.** Every query on a [deletion-vector](./deletion-vectors)-enabled table runs against a pinned snapshot automatically — and so does every query on a table with snapshot consistency enabled, or one using [segment groups](./segment-groups). You only set the option yourself to override the default (latest) choice.
* **Not the same as `initialSnapshotId`.** The table-config keys `iceberg.snapshotProcessing.initialVersionSelector` / `initialSnapshotId` (and the DDL `snapshot_mode` / `snapshot_id`) choose where **ingestion starts** when the table is created. They do not affect which snapshot a query reads.
* **Finding snapshot IDs.** `GET …/externalTable/status?includeLag=true` returns the newest synced snapshot (`lag.synced.snapshotId`) and the upstream head (`lag.upstream.snapshotId`) — see [Observability](./observability#ingestion-lag). For older retained snapshots, get IDs from the source catalog (e.g. the Iceberg table's `snapshots` metadata); there is currently no API that lists all active snapshot IDs.
# Data and Index caching
Source: https://docs.startree.ai/corecapabilities/external-table/data-and-index-caching
How the Parquet data cache, index cache, and footer cache work for External Tables and tiered-storage tables — sizing, eviction, and restart behavior.
## 1. Overview
This document describes how Parquet data cache and Pinot index cache work in case of External tables as well as regular tables configured with remote tiered storage.
External tables read their data from object storage on every query. The **Parquet Data Cache** keeps recently-read Parquet pages on the server so subsequent queries do not pay the round-trip.
Indexes for both external tables and Pinot tiered storage tables also live on object storage. The **Index Cache** keeps recently-read index byte ranges on the server.
| Cache | What it stores | Used by | On-disk location | When is it populated |
| - | - | - | - | - |
| **Parquet Data Cache** | Compressed Parquet column data and dictionary pages | External tables (Iceberg/S3/GCS) | `{dataDir}/remote_data_cache` | On every Parquet column read (always on) |
| **Index Cache** | Raw byte ranges of segment index files (`columns.psf`, `inverted_index`) | External tables and Pinot tiered storage tables | `{dataDir}/index_cache` | On query opt-in |
| **Footer Cache** | The Parquet file footer (row-group metadata + schema) for each segment, so it isn't re-fetched on every query. Persisted as part of the Parquet page index. | External tables | `{dataDir}/tieredStorage/segmentCache` (`parquet_page_index.bin`; migrates a legacy standalone `parquet_footer.bin` if present) | At segment load, whenever the persistent disk cache is enabled (on by default — see [Footer Cache](#3-3-footer-cache)) |
Both caches share the same underlying machinery and, by default, the same in-memory sizing configuration (it isn't scoped per cache):
* **Parquet Data Cache** populates an in-memory **prefetch tier** (decompressed, decoded values on heap, default 1 GB) in front of an mmap-backed **disk tier** (default 15% of disk per dictionary/data pool).
* **Index Cache** populates an in-memory raw-bytes tier (the same underlying memory cache used by the Parquet Data Cache — default 256 MB data / 64 MB dictionary, since the memory-size configs aren't per-consumer) in front of its own mmap-backed disk tier. Only the 1 GB prefetch tier is effectively Parquet-Data-Cache-only, since raw index byte ranges have no decoded form to keep there.
The two caches do not share quota, directory, or eviction. They are sized and tuned independently. Following image depicts the high level query flow and how it uses the different caches.
## 2. What's enabled out of the box
### 2.1 Parquet Data Cache — always on
This cache is always on and does not need to be explicitly enabled. The cache is created at server startup and serves every read against a remote-Parquet column.
| Tier | Default size | Backing store |
| - | - | - |
| Prefetch (in-memory) | 1 GB | Java heap (deserialized values) |
| Disk | 15% of total disk per pool (dictionary + data) | mmap'd files at `{dataDir}/remote_data_cache` |
### 2.2 Index Cache — opt-in per query
The Index Cache is populated only when a query is preceded by
```text theme={null}
SET `enable.prefetch.page.cache` = 'true'
```
Byte ranges of segment index files fetched from object storage during that query are persisted on disk, so subsequent queries (with or without the option) get cache hits. This applies to both external tables and Pinot tiered storage tables — any query that needs to read an index buffer from object storage benefits from this cache.
| Tier | Default size | Backing store |
| - | - | - |
| In-memory | 256 MB (data) / 64 MB (dict) — shared with the Parquet Data Cache's memory sizing | Java heap |
| Disk | 15% of total disk per pool (dictionary + data) | mmap'd files at `{dataDir}/index_cache` |
Without the query option and without StarTree bitmap reads, the Index Cache stays empty and the directory remains untouched.
***
## 3. Advanced operations: Enable, disable, skip
### 3.1 Parquet Data Cache
Always on at the server level and cannot be globally disabled. However, you can use a per-query bypass as shown below:
```sql theme={null}
SET `skip.remote.table.cache` = 'true';
SELECT ... FROM remoteTable WHERE ...;
```
When set, both tiers are bypassed for that query. Pages are read directly from object storage and not stored.
### 3.2 Index Cache
In this case, there is no global enable flag; enablement is a per-query decision. You can enable index cache for a specified query as shown below:
| Action | How |
| - | - |
| Enable for one query | `SET `enable.prefetch.page.cache` = 'true'` |
As mentioned previously, once set, subsequent queries can benefit from this index cache as applicable.
### 3.3 Footer Cache
The Footer Cache stores the Parquet file footer (row-group metadata and schema) for each segment so it doesn't need to be re-fetched from object storage on every query. It follows the persistent disk cache's global switch — on by default, no separate directory to configure:
```text theme={null}
pinot.server.instance.disk.cache.enabled = true # default
pinot.server.instance.disk.cache.directory = /path/to/cache/root # optional, defaults to the server data dir
```
Metadata caching (including the footer) is always on whenever the disk cache is enabled — there is no separate opt-in flag. The footer is persisted as part of the Parquet page index (`parquet_page_index.bin`) under `/tieredStorage/segmentCache` and always reloaded on server restart. A legacy standalone `parquet_footer.bin` from an older build is migrated into the page index and then removed. Setting `enabled = false` disables the footer cache along with every other managed disk cache, and the footer is then fetched from remote storage on each query.
For the full list of persistent disk cache configuration keys, see [Best Practices and Configs](./best-practices-and-configs#persistent-disk-cache-pinot-server-instance-disk-cache).
***
## 4. Behavior on restart
Restart reuse is **on by default** (changed in 0.16.x — it used to be opt-in). The disk tier reloads its key index from a snapshot file written during shutdown (and periodically every five minutes) and resumes serving immediately, for both caches identically:
```text theme={null}
pinot.server.instance.disk.cache.pages.snapshot.enabled = true # default
```
Set it to `false` to start the disk tier empty on every restart instead.
The prefetch tier is in-memory — it always starts empty and is not persisted across restarts.
***
## 5. Eviction policies
| Tier | Cache | Policy | Defaults |
| - | - | - | - |
| Prefetch (in-memory) | Parquet Data Cache only | Circular buffer on heap; oldest slot is overwritten on insert | 1 GB capacity, no TTL |
| Disk | Both | LRU over fixed-size mmap fragment files. | Parquet Data Cache: 32 MB fragments. Index Cache: 512 MB fragments. Background eviction sweep every 60s (fixed, not configurable). |
When the disk tier is full, the oldest fragment file is unmapped and deleted; new pages are written to a fresh fragment.
**TTL-based eviction was removed in 0.16.x.** The disk tier now evicts purely on capacity (LRU) and segment-version invalidation; there is no idle- or write-age TTL to configure, and the old `ttl.access.ms` / `ttl.write.ms` / `eviction.interval.ms` keys are ignored (the sweep interval is a fixed 60s). Entries restored from a restart snapshot are still safe to reuse — the source/version keying described in [Best Practices and Configs](./best-practices-and-configs#persistent-disk-cache-pinot-server-instance-disk-cache) prevents a stale entry from being served after a same-URI replacement.
### Disk cache configuration reference
The Parquet Data Cache and Index Cache disk tiers are two consumers of the shared persistent disk cache, `pinot.server.instance.disk.cache.*`. Sizing keys accept a `.PARQUET_INDEX` or `.SEGMENT_INDEX` suffix to target one consumer independently.
These configs are read when the cache is created at server startup. Changes take effect only after a server restart.
| Config | Effect | Default |
| - | - | - |
| `pinot.server.instance.disk.cache.pages.data.max.size` | Data-page capacity, as a size or a percent of total disk. | `15%` per consumer |
| `pinot.server.instance.disk.cache.pages.dict.max.size` | Dictionary-page capacity, as a size or a percent of total disk. | `15%` per consumer |
| `pinot.server.instance.disk.cache.pages.fragment.size.mb` | Fragment size in MiB (capped at `2047`); controls eviction granularity. | `32` for `PARQUET_INDEX`, `512` for `SEGMENT_INDEX` |
| `pinot.server.instance.disk.cache.pages.snapshot.enabled` | Persist the disk tier across restarts (see [Behavior on restart](#4-behavior-on-restart)). | `true` |
Percentage sizes round down to whole fragments and are capped at 320000 MiB per Parquet-page pool or 5120000 MiB per segment-index pool; set an explicit byte/unit size instead of a percentage to exceed that ceiling.
See [Persistent disk cache](./best-practices-and-configs#persistent-disk-cache-pinot-server-instance-disk-cache) for the global `enabled`/`directory` switches shared with every other managed cache.
***
## 6. Clearing caches
### HTTP endpoints
There are certain scenarios in which user may want to clear the caches. StarTree provides Cluster-wide APIs for this (controller fans out to every server):
| Endpoint | Effect |
| - | - |
| `DELETE /pageCache?consumerName=PARQUET_INDEX` | Clear Parquet Data Cache (all tiers) |
| `DELETE /pageCache?consumerName=SEGMENT_INDEX` | Clear Index Cache (all tiers) |
| `DELETE /pageCache` | Clear both |
This can also be done via a per-server API (skip the controller, hit one server directly):
```text theme={null}
DELETE http://:/pageCache?consumerName=PARQUET_INDEX
```
Response is JSON listing cleared cache.
# Data Export Task: Parquet Writer
Source: https://docs.startree.ai/corecapabilities/external-table/data-export-task
Export completed Pinot REALTIME segments back to Parquet on a filesystem or an Iceberg REST catalog with the DataExportTask minion task.
This feature requires StarTree Cloud with a minion tier enabled, and must be enabled on demand — contact StarTree support to activate it.
`DataExportTask` is a Pinot minion task that exports completed **REALTIME segments** to external storage as Parquet files. It is the complement of `ExternalTableSyncTask`: where that task reads external Parquet into Pinot, `DataExportTask` writes Pinot data back out to your data lake. Common uses include cold-tier archival, feeding downstream Iceberg catalogs, and cross-system data sharing without a separate ETL pipeline.
Two destinations are supported: a plain filesystem path (any S3-compatible URI via PinotFS) and an Iceberg REST catalog (AWS Glue, S3 Tables, or any Iceberg REST-compliant catalog).
***
## How it works
```mermaid theme={null}
flowchart LR
REALTIME["REALTIME Table\n(Pinot)"] --> Generator["Generator\n(Controller)"]
Generator --> |"Watermark +\nZK checkpoint"| EligibleSegments["Eligible\nCompleted Segments"]
EligibleSegments --> Executor["Executor\n(Minion)"]
Executor --> Download["Download segment\nfrom deep store"]
Download --> Convert["Convert to\nParquet"]
Convert --> Upload["Upload to\ndestination"]
Upload --> FilesystemDest["Filesystem\n(S3/GCS/HDFS)"]
Upload --> IcebergDest["Iceberg REST\n(Glue / S3 Tables)"]
```
Each cycle, the **Generator** runs on the controller: it loads a time-based watermark from ZooKeeper, filters REALTIME segments that are completed, past the `bufferTimePeriod`, and not already checkpointed or in a running task. It then emits one `PinotTaskConfig` per eligible segment, capped by `tableMaxNumTasks`.
The **Executor** runs on a minion: it downloads the segment from Pinot deep store, converts it to Parquet using the configured writer, and uploads the result to the destination. On success, it posts a COMPLETED checkpoint back so the segment is not re-exported.
***
## Supported destinations
| Destination | `dataExport.target` | Use with |
| - | - | - |
| Filesystem (PinotFS) | `filesystem` | Any S3-compatible URI, GCS, HDFS |
| Iceberg REST catalog | `iceberg-rest` | AWS Glue, AWS S3 Tables, any Iceberg REST catalog |
***
## Prerequisites
* A REALTIME table with completed segments exists in your StarTree cluster.
* A minion tier is provisioned in the cluster.
* The minion has network access to the destination (S3 bucket, GCS bucket, or Iceberg REST endpoint).
* For Iceberg REST: the target table must already exist in the catalog, must not be partitioned, and must use Parquet as its file format.
***
## Configuring the task
Add `DataExportTask` to the table's `task.taskTypeConfigsMap`.
#### AWS S3 using Basic Authentication
```json theme={null}
"task": {
"taskTypeConfigsMap": {
"DataExportTask": {
"schedule": "0 0 * * * ?",
"sourceTableName": "my_events_REALTIME",
"fileFormat": "parquet",
"dataExport.target": "filesystem",
"outputDirURI": "s3://my-bucket/exports/my_events/",
"output.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
"output.fs.prop.region": "",
"enableTaskOrchestration": "true",
"output.fs.prop.accessKey" : "ACCESS_KEY",
"output.fs.prop.secretKey" : "SECRET_KEY"
}
}
}
```
#### AWS S3 using IAM-Based Authentication
```json theme={null}
"task": {
"taskTypeConfigsMap": {
"DataExportTask": {
"schedule": "0 0 * * * ?",
"sourceTableName": "my_events_REALTIME",
"fileFormat": "parquet",
"dataExport.target": "filesystem",
"enableTaskOrchestration": "true",
"outputDirURI": "s3://my-bucket/exports/my_events/",
"output.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
"output.fs.prop.region": "",
"output.fs.prop.roleArn" : "ROLE_ARN",
"output.fs.prop.externalId": "env-12abc234xyz"
}
}
}
```
#### GCS
Use `GcsPinotFS` with a service account key:
```json theme={null}
"task": {
"taskTypeConfigsMap": {
"DataExportTask": {
"schedule": "0 0 * * * ?",
"sourceTableName": "my_events_REALTIME",
"fileFormat": "parquet",
"dataExport.target": "filesystem",
"enableTaskOrchestration": "true",
"outputDirURI": "gs://my-bucket/exports/my_events/",
"output.fs.className": "org.apache.pinot.plugin.filesystem.GcsPinotFS",
"output.fs.prop.projectId": "my-gcp-project",
"output.fs.prop.jsonKey": ""
}
}
}
```
#### Iceberg REST
For an Iceberg REST destination, replace the filesystem keys with the catalog connection:
```json theme={null}
"task": {
"taskTypeConfigsMap": {
"DataExportTask": {
"schedule": "0 0 * * * ?",
"sourceTableName": "my_events_REALTIME",
"fileFormat": "parquet",
"dataExport.target": "iceberg-rest",
"enableTaskOrchestration": "true",
"catalog.iceberg-rest.restUri": "https://glue..amazonaws.com/iceberg",
"catalog.iceberg-rest.table.namespace": "my_database",
"catalog.iceberg-rest.table.tableName": "my_events_archive",
"iceberg.commitThreshold": "250"
}
}
}
```
***
## Config reference
### Core settings
| Key | Default | Description |
| - | - | - |
| `schedule` | — | Quartz cron expression for how often the Generator runs, e.g. `0 0 * * * ?` (hourly). |
| `sourceTableName` | *(required)* | The REALTIME table to export from, with the `_REALTIME` suffix, e.g. `my_events_REALTIME`. |
| `fileFormat` | `parquet` | Output file format. Only `parquet` is supported. |
| `dataExport.target` | `filesystem` | Destination type. `filesystem` or `iceberg-rest`. |
| `bufferTimePeriod` | `1d` | Segments whose end time is within this window of now are excluded — allows segments to fully complete before export. |
| `initialWatermarkMs` | `0` | Epoch ms to use as the starting watermark on the first run. Segments older than this value are skipped. |
| `tableMaxNumTasks` | `10000` | Maximum number of segments exported per Generator cycle. Limits concurrent minion load. |
| `enableTaskOrchestration` | — | Opts into the minion task orchestration framework (MTOF), which auto-progresses across segment batches within a run. Set in all example configs above. |
***
### Filesystem target
Set these when `dataExport.target=filesystem`.
| Key | Default | Description |
| - | - | - |
| `outputDirURI` | *(required)* | Destination URI, e.g. `s3://my-bucket/exports/table/` or `gs://my-bucket/exports/table/`. Output files are written as `//.parquet`. |
#### AWS S3 using Basic Authentication
| Key | Default | Description |
| - | - | - |
| `output.fs.className` | — | Fully-qualified PinotFS class, e.g. `org.apache.pinot.plugin.filesystem.S3PinotFS`. Defaults to the cluster's configured deep-store FS if unset. |
| `output.fs.prop.region` | — | AWS region, e.g. `us-east-1`. |
| `output.fs.prop.accessKey` | — | Static AWS access key. |
| `output.fs.prop.secretKey` | — | Static AWS secret key. |
| `output.fs.prop.endpoint` | — | S3-compatible endpoint override, e.g. for MinIO or other non-AWS S3 implementations. |
#### AWS S3 using IAM-Based Authentication
| Key | Default | Description |
| - | - | - |
| `output.fs.className` | — | Fully-qualified PinotFS class, e.g. `org.apache.pinot.plugin.filesystem.S3PinotFS`. Defaults to the cluster's configured deep-store FS if unset. |
| `output.fs.prop.region` | — | AWS region, e.g. `us-east-1`. |
| `output.fs.prop.roleArn` | — | ARN of the IAM role the minion assumes to access the bucket, in place of static credentials. |
| `output.fs.prop.externalId` | — | External ID required by the role's trust policy, if configured. |
#### GCS
| Key | Default | Description |
| - | - | - |
| `output.fs.className` | — | Fully-qualified PinotFS class, e.g. `org.apache.pinot.plugin.filesystem.GcsPinotFS`. |
| `output.fs.prop.projectId` | — | The GCP project ID that owns the target bucket. |
| `output.fs.prop.jsonKey` | — | The string-encoded GCP service account JSON key used to authenticate to the bucket. |
***
### Parquet format
These apply to both destinations.
| Key | Default | Description |
| - | - | - |
| `fileFormat.parquet.compressionCodec` | `SNAPPY` | Parquet compression. `SNAPPY`, `GZIP`, `ZSTD`, or `UNCOMPRESSED`. |
| `fileFormat.parquet.rowGroupSizeBytes` | `134217728` (128 MB) | Target row group size in bytes. |
| `fileFormat.parquet.pageSizeBytes` | `1048576` (1 MB) | Target page size in bytes. |
| `fileFormat.parquet.enableDictionary` | `true` | Write Parquet dictionary pages. Disable for high-cardinality columns. |
| `fileFormat.parquet.json.maxDepth` | — | Maximum nesting depth when serializing JSON columns to Parquet `group` types. |
| `fileFormat.parquet.decimal.precision` | `38` | Override decimal precision for `BIG_DECIMAL` columns. |
| `fileFormat.parquet.decimal.scale` | `18` | Override decimal scale for `BIG_DECIMAL` columns. |
| `fileFormat.parquet.json.unknownKeysBehavior` | `warn` | What to do when a JSON column in the row data has keys not present in the Iceberg schema: `warn` (log and continue) or `fail` (abort the task). Applies to `iceberg-rest` target only. |
***
### Iceberg REST target
Set these when `dataExport.target=iceberg-rest`.
| Key | Default | Description |
| - | - | - |
| `catalog.iceberg-rest.restUri` | *(required)* | Iceberg REST catalog URI, e.g. `https://glue..amazonaws.com/iceberg`. |
| `catalog.iceberg-rest.table.namespace` | *(required)* | Catalog namespace (database) containing the target table. |
| `catalog.iceberg-rest.table.tableName` | *(required)* | Target table name inside the namespace. |
| `catalog.iceberg-rest.token` | — | Static bearer token for catalog authentication. |
| `catalog.iceberg-rest.credential` | — | OAuth2 client credential (`clientId:clientSecret`). |
| `catalog.iceberg-rest.rest.signing-region` | — | AWS region for SigV4 request signing (e.g. `us-east-1`). |
| `catalog.iceberg-rest.s3.access-key-id` | — | Static AWS access key for S3 file I/O (overrides vended credentials). |
| `catalog.iceberg-rest.s3.secret-access-key` | — | Static AWS secret key. |
For AWS Glue and S3 Tables, the catalog vends short-lived S3 credentials automatically — do not set static `s3.*` keys unless you want to override them. The minion's IAM role must have `glue:GetTable` and `s3:PutObject` (or equivalent S3 Tables) permissions on the target.
***
### Iceberg batch commits
By default, each minion subtask uploads a Parquet file and stages it. The Generator accumulates staged files in ZooKeeper and issues a single Iceberg snapshot commit (`AppendFiles.commit()`) when the batch is ready — one snapshot for many segments instead of one per segment. This reduces Iceberg snapshot churn on high-throughput tables.
The Generator runs an **EMIT / WAIT / COMMIT** decision each cycle:
| State | Condition | Action |
| - | - | - |
| `EMIT` | Eligible segments exist and in-flight count is below threshold | Schedule new subtasks |
| `WAIT` | Subtasks are still running **and** the uncommitted/staged count has reached `iceberg.commitThreshold` | Skip this cycle; wait for in-flight tasks to drain before committing |
| `COMMIT` | No subtasks are running, at least one file is staged, **and** either the staged count has reached `iceberg.commitThreshold` or no eligible segments remain | Flush the staged queue with one `AppendFiles.commit()` |
| Key | Default | Description |
| - | - | - |
| `iceberg.commitThreshold` | `250` | Number of staged files that triggers an immediate batch commit. Tune down for lower latency; tune up to reduce snapshot frequency on high-volume tables. |
***
## Scheduling and monitoring
`DataExportTask` uses the standard Pinot minion task scheduler. To trigger a run immediately without waiting for the next `schedule` tick, use the task schedule endpoint:
```bash theme={null}
curl -X POST "$BASE_URL/tasks/schedule?taskType=DataExportTask&tableName=my_events_REALTIME"
```
To check task status, use the standard Pinot tasks API — per-table state:
```bash theme={null}
curl "$BASE_URL/tasks/DataExportTask/my_events_REALTIME/state"
```
or subtask states across all tables running this task type:
```bash theme={null}
curl "$BASE_URL/tasks/DataExportTask/taskstates"
```
For a broader view of sync health and the ingestion checkpoint, see the [Observability](./observability) page.
# Data Type Mapping
Source: https://docs.startree.ai/corecapabilities/external-table/data-type-mapping
How Parquet and Iceberg source types map to Pinot types in an External Table, including time-column detection, granularity, and known limitations.
This feature is available starting in **StarTree release 0.15.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment.
When you onboard an External Table, StarTree infers a Pinot schema from the source metadata. The mapping depends on the source:
* **Raw S3 Parquet** (S3 Data Lake, `catalogType=s3`) — inferred from the Parquet file's physical and logical types.
* **Iceberg** (AWS Glue / S3 Tables, `catalogType=iceberg-rest`) — inferred from the Iceberg table schema.
Numeric columns (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `BIG_DECIMAL`) are inferred as **metrics**; everything else (`BOOLEAN`, `STRING`, `JSON`, `BYTES`, time columns) as **dimensions**. You can adjust this in the schema before creating the table.
***
## Raw S3 Parquet → Pinot
For the **S3 Data Lake** source. Types come from the Parquet file metadata.
## Primitive Types
| Parquet Type | Pinot Type | Notes |
| - | - | - |
| `BOOLEAN` | `BOOLEAN` | Optional and required variants both supported. |
| `INT32` | `INT` | |
| `INT64` | `LONG` | |
| `FLOAT` | `FLOAT` | |
| `DOUBLE` | `DOUBLE` | |
| `BINARY` | `BYTES` | Raw bytes are preserved losslessly rather than decoded as UTF-8. |
| `FIXED_LEN_BYTE_ARRAY` | `BYTES` | Fixed-width byte arrays are preserved losslessly rather than decoded as UTF-8. |
| `INT96` | `TIMESTAMP` | Deprecated Parquet type. Converted to milliseconds since epoch; no column statistics available. Avoid using as a time column. |
***
## Logical Types
### Integer Subtypes
| Parquet Logical Type | Pinot Type | Notes |
| - | - | - |
| `INT(8, signed)` | `INT` | Range: -128 to 127 |
| `INT(16, signed)` | `INT` | Range: -32,768 to 32,767 |
| `INT(32, signed)` | `INT` | |
| `INT(64, signed)` | `LONG` | |
| `INT(8, unsigned)` | `INT` | Range: 0 to 255 |
| `INT(16, unsigned)` | `INT` | Range: 0 to 65,535 |
| `INT(32, unsigned)` | `LONG` | **Known limitation:** values are stored as signed. Values above `2,147,483,647` may be corrupted. See [Unsigned Integer Limitation](#unsigned-integer-types). |
| `INT(64, unsigned)` | `LONG` | **Known limitation:** same signed-storage issue. Values above `Long.MAX_VALUE` are not preserved accurately. |
### String and Binary Types
| Parquet Logical Type | Pinot Type | Notes |
| - | - | - |
| `STRING` | `STRING` | UTF-8 encoded strings. |
| `ENUM` | `STRING` | Enum values stored as strings. |
| `JSON` | `JSON` | Parsed into Pinot's native `JSON` type, not preserved as a plain string. |
| `BSON` | `STRING` | Binary data is decoded as a UTF-8 string, not preserved as a distinct binary type. Non-UTF8-safe byte sequences may not round-trip cleanly — verify BSON columns after ingestion. |
| `UUID` | `BYTES` | The raw 16-byte value is preserved losslessly as bytes rather than decoded as a UTF-8 string. |
| `INTERVAL` | `STRING` | |
| `GEOMETRY`, `GEOGRAPHY` | `BYTES` | Stored as raw bytes; no geospatial functions or H3 index are available on external tables. |
### Temporal Types
| Parquet Logical Type | Pinot Type | Notes |
| - | - | - |
| `DATE` | `LONG` | Stored as epoch days. |
| `TIME(MILLIS)` | `LONG` | Milliseconds since midnight; range 0 to 86,399,999. |
| `TIME(MICROS)` | `LONG` | Microseconds since midnight; range 0 to 86,399,999,999. |
| `TIME(NANOS)` | `LONG` | Nanoseconds since midnight. |
| `TIMESTAMP(MILLIS, UTC)` | `TIMESTAMP` | |
| `TIMESTAMP(MILLIS, local)` | `TIMESTAMP` | |
| `TIMESTAMP(MICROS, UTC)` | `TIMESTAMP` | Native microsecond granularity is preserved (`EPOCH\|MICROSECONDS`) — see [Granularity](#granularity). |
| `TIMESTAMP(MICROS, local)` | `TIMESTAMP` | Native microsecond granularity is preserved — see [Granularity](#granularity). |
| `TIMESTAMP(NANOS, UTC)` | `TIMESTAMP` | Native nanosecond granularity is preserved (`EPOCH\|NANOSECONDS`) — see [Granularity](#granularity). |
| `TIMESTAMP(NANOS, local)` | `TIMESTAMP` | Native nanosecond granularity is preserved — see [Granularity](#granularity). |
### Decimal Types
All Parquet `DECIMAL` variants map to Pinot `BIG_DECIMAL`, preserving scale and precision.
| Parquet Physical + Logical Type | Pinot Type | Example |
| - | - | - |
| `INT32` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(9, 2)` |
| `INT64` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(18, 4)` |
| `FIXED_LEN_BYTE_ARRAY` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(28, 6)` |
| `BINARY` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(38, 10)` |
***
## Complex Types
| Source type | Pinot type | Notes |
| - | - | - |
| `LIST` / `ARRAY` of a primitive | Multi-value column of the element type | Element types: `INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`, `BOOLEAN`. A list of timestamps is stored as multi-value `LONG`. |
| `LIST` of a complex element | `JSON` | Lists of structs / maps / nested lists are serialized to a JSON string. |
| `MAP` | `MAP` | Native map for primitive values (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`). |
| `MAP` (other shapes) | `JSON` | Non-string keys or complex values are serialized to JSON. |
| `STRUCT` / nested group | `JSON` | Serialized to a JSON string; query with JSON functions. |
| `VARIANT` | `OPEN_STRUCT` | Detected from the Parquet VARIANT logical type annotation (or, for writers that predate it, the `metadata` + `value`/`typed_value` group shape the shredding spec fixes). Not the same as `JSON` — a VARIANT's key set and per-key types are data, not schema, and are read per-file from the shredded tree rather than declared up front. |
To use `JSON_MATCH` or `JSON_EXTRACT_SCALAR` on complex columns, you must enable a JSON index on those columns. Without it, queries will fail with a `Cannot apply JSON_MATCH on column without json index` error. Configure the index in the table's index settings.
Multi-value columns are capped at **1,000 entries per row** — the segment metadata is written with a fixed upper bound, and rows whose source array is longer are silently truncated at query time. If your lists can exceed 1,000 elements, keep the column as `JSON` instead.
***
## Iceberg → Pinot
For the **AWS Glue** and **AWS S3 Tables** sources. Types come from the Iceberg table schema.
| Iceberg type | Pinot type | Notes | |
| - | - | - | - |
| `boolean` | `BOOLEAN` | | |
| `int` | `INT` | | |
| `long` | `LONG` | | |
| `float` | `FLOAT` | | |
| `double` | `DOUBLE` | | |
| `decimal(p, s)` | `BIG_DECIMAL` | Precision and scale preserved. | |
| `string` | `STRING` | | |
| `uuid` | `BYTES` | Preserved losslessly as bytes rather than decoded as a UTF-8 string. | |
| `binary`, `fixed` | `BYTES` | Preserved losslessly as bytes rather than decoded as a UTF-8 string. `fixed[N]` length suffix is parsed. | |
| `date` | `LONG` | \`EPOCH | DAYS\`. |
| `time` | `LONG` | Stored as the raw long; not a time-column candidate. | |
| `timestamp`, `timestamptz` | `TIMESTAMP` | Granularity inferred from the type — see note below. | |
| `timestamp_ns`, `timestamptz_ns` | `TIMESTAMP` | \`EPOCH | NANOSECONDS\`. |
| `struct` | `JSON` | Serialized to a JSON string. | |
| `list` | Multi-value column of the element type | `INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`, `BOOLEAN`; `list` becomes multi-value `LONG`. | |
| `list` | `JSON` | | |
| `map` | `MAP` | | |
| `map` (other shapes) | `JSON` | | |
| `variant` | `OPEN_STRUCT` | Not the same as `JSON` — a variant's key set and per-key types are data, not schema, and are discovered per-file rather than declared up front. See the raw-Parquet `VARIANT` row above for the equivalent mapping on the S3 Data Lake source. | |
| (unrecognized) | `STRING` | Falls back to `STRING`. | |
**Timestamp granularity is inferred from the Iceberg type** (current releases):
* `timestamp_ns` / `timestamptz_ns` → `EPOCH|NANOSECONDS`
* `timestamp(p)` with an explicit precision → milliseconds for `p ≤ 3`, microseconds for `p ≤ 6`, nanoseconds above that
* plain `timestamp` / `timestamptz` → **`EPOCH|MICROSECONDS`**, matching Iceberg's on-disk microsecond storage
Older 0.16.0 builds inferred plain `timestamp` as `EPOCH|MILLISECONDS`, which mislabeled microsecond data (values \~1000× off). Tables onboarded on those builds keep their inferred format — set the time column's granularity explicitly if it doesn't match the data. See [Time column](#time-column).
***
## Time column
A Pinot table can designate one column as its **primary time column** — a `DateTimeFieldSpec` that records the event time of each row. See the Apache Pinot [DateTimeFieldSpec reference](https://docs.pinot.apache.org/configuration-reference/schema#datetimefieldspec) for the general schema requirement and format syntax.
### How the time column is used in an External Table
* **Segment pruning** — a query with a time filter skips segments whose time range can't match, so fewer remote files are read.
* **Retention** — if a retention period is configured on the table, segments older than it are dropped.
The time column is **optional**: without one, the table is still created and fully queryable, but loses time-based segment pruning and retention.
### How the time column is detected
During onboarding, after the schema is inferred, StarTree selects a time column automatically — choosing, in order:
1. The first non-nullable millisecond-granularity datetime column.
2. Otherwise, the first millisecond-granularity datetime column (nullable allowed).
3. Otherwise, the first datetime column of any unit.
Only columns that map to a Pinot `DateTimeFieldSpec` are candidates — `TIMESTAMP`, `DATE`, and (raw Parquet) `INT96`. A plain integer column holding an epoch value is **not** auto-detected; set it as the time column manually if needed. If the source has no timestamp/date column, no time column is set; you can assign one in the schema before creating the table.
If auto-detection picks an `INT96` column as the time column, segment generation now **fails by default** with an actionable error naming the column and pointing at the workaround, instead of an opaque decoding error. See [INT96 (Deprecated)](#int96-deprecated) below before onboarding a source whose only datetime column is `INT96`.
### Granularity
For **raw Parquet**, the inferred `DateTimeFieldSpec` preserves the source unit:
| Source timestamp | Inferred format | Granularity |
| - | - | - |
| `INT96` (legacy) | `EPOCH\|MILLISECONDS` | `1:MILLISECONDS` |
| `TIMESTAMP(millis)` | `EPOCH\|MILLISECONDS` | `1:MILLISECONDS` |
| `TIMESTAMP(micros)` | `EPOCH\|MICROSECONDS` | `1:MICROSECONDS` |
| `TIMESTAMP(nanos)` | `EPOCH\|NANOSECONDS` | `1:NANOSECONDS` |
| `DATE` | `EPOCH\|DAYS` | `1:DAYS` |
For **Iceberg**, the granularity follows the type: `timestamp_ns` → nanoseconds, `timestamp(p)` → from the precision, and plain `timestamp` / `timestamptz` → **microseconds** (older 0.16.0 builds inferred milliseconds — see the note in the Iceberg → Pinot section above).
Values are stored in the source unit — there is no implicit downscaling — so account for the unit in time-range queries. If the inferred granularity doesn't match what the underlying Parquet actually stores (for example, a table onboarded on an older build), set the time column's granularity explicitly.
***
## Known Limitations
### Unsigned Integer Types
Parquet `UINT32` and `UINT64` (unsigned 32-bit and 64-bit integers) are stored in Pinot as a signed `LONG`. This means:
* Values that fit within the signed range are accurate.
* Values above `2,147,483,647` (for UINT32) are stored as negative numbers or wrap around, silently corrupting approximately half the data.
If your dataset contains unsigned integers larger than `INT32_MAX`, treat the Pinot `LONG` values with caution or apply a transformation at ingestion time.
### Null Handling in Complex Types
`MAP` columns that contain null values in keys or values may not propagate null bitmaps correctly to Pinot segments. This can result in nulls being replaced by type defaults (e.g., `0` for integers, empty string for strings). Verify null behavior after ingestion.
Current onboarding flows set `nullHandlingEnabled: true` on the generated table config automatically, and when the source schema has nullable fields, the inferred Pinot schema marks them nullable and enables column-based null handling (`enableColumnBasedNullHandling`). Tables created on older builds may have null handling off — see [Advanced Guidance](/corecapabilities/external-table/operational-guidance).
### INT96 (Deprecated)
`INT96` has no column statistics in Parquet — its footer min/max are skipped rather than trusted, since the format leaves INT96's sort order undefined (they aren't guaranteed to be the chronological extremes, so pruning on them could drop live rows). This affects min/max pruning during segment generation. Scalar and list `INT96` values still decode correctly for reads.
It should not be used as a `timeColumnName`. If it is (including via auto-detection — see [How the time column is detected](#how-the-time-column-is-detected)), segment generation fails by default because no time range can be derived from the skipped statistics, with an error naming the column and the workaround. Set `segmentTimeValueCheck=false` in the table's `ingestionConfig` to ingest without time-range metadata for that column instead.
# Iceberg Deletion Vectors
Source: https://docs.startree.ai/corecapabilities/external-table/deletion-vectors
Apply Iceberg v3 Puffin deletion vectors at query time so External Tables reflect source-side row-level deletes and updates, not just appends.
This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it. It is offered as a preview capability; validate it in a non-production environment before relying on it.
Iceberg tables record row-level deletes and updates as **deletion vectors** (Puffin-format delete files) rather than rewriting data files in place. External Tables read and apply Iceberg v3 Puffin deletion vectors at query time, so query results stay correct as source data is updated or deleted — the source doesn't have to be append-only.
## How it works
1. **Server-side loading** — each server decodes the Iceberg Puffin deletion files relevant to its segments and applies them when answering a query, so deleted/updated rows are excluded from results.
2. **Atomic snapshot readiness** — before a new Iceberg snapshot becomes visible to queries, the controller polls every server until all of them confirm they've loaded the deletion vectors for that snapshot. This avoids a window where some servers see the new snapshot's deletes and others don't.
3. **Broker-side pinning and consistency** — the broker pins every query to a single snapshot per table so all servers involved in a query agree on which snapshot (and therefore which deletes) to use. By default it pins the newest active snapshot automatically and injects the `snapshotVersionByTable` option for the servers; a query can also pin an explicit snapshot (see [Query-time snapshot pinning](#query-time-snapshot-pinning)). An invalid explicitly-pinned snapshot fails the query rather than silently mixing snapshot state.
## Enabling deletion vectors
Set on the table's `ExternalTableSyncTask` config:
| Key | Default | Description |
| :- | :- | :- |
| `enableDeletionVectors` | `false` | Enable Iceberg deletion vector support for this table. When a table is created against a catalog that supports deletion vectors, onboarding sets this to `true` automatically; the `false` default applies only when the key is absent from an existing config. |
| `deletionVectorSource` | `iceberg` | Source format for deletion vectors. `iceberg` is the only supported value. |
Changes to `enableDeletionVectors` and `deletionVectorSource` (and to the table's catalog or storage credentials) take effect on the servers without a restart — the deletion-vector readers are rebuilt when the table config changes.
**Incompatible with the Star-tree index.** Table-config validation rejects a Star-tree index (explicit `starTreeIndexConfigs` or `enableDefaultStarTree`) on a table with `enableDeletionVectors=true`, and the reverse — enabling deletion vectors on a table that already has a Star-tree index configured. Deletion vectors filter rows per query, and Star-tree's precomputed aggregates can't be reconciled against that. Remove the Star-tree index configuration or disable deletion vectors. See [Supported Indexes](./indexes).
**How Puffin delete files are read is now a server setting, not a table setting.** The old per-table `iceberg.dv.fetch.mode`, `iceberg.dv.puffinCache.maxBytes`, and `iceberg.dv.puffinCache.downloadTimeoutMs` keys are retired (0.16.x) in favor of the server-wide persistent disk cache: `pinot.server.instance.disk.cache.puffin.max.size` (default `10G`) and `pinot.server.instance.disk.cache.puffin.download.timeout.ms` (default `600000`). There's no more explicit `ranged`/`cache`/`auto` choice — a positive `puffin.max.size` caches whole Puffin files locally (shared per table/URI-scheme, LRU-evicted), and `0` always uses ranged reads (or stream-skip where ranges aren't available). Unlike the table-level keys these removed, the replacements take effect only at server startup — see [Persistent disk cache](./best-practices-and-configs#persistent-disk-cache-pinot-server-instance-disk-cache) for the full settings table.
The number of snapshots kept queryable (and therefore pinnable — see [Query-time snapshot pinning](#query-time-snapshot-pinning)) is bounded by snapshot retention, `iceberg.snapshotProcessing.retention.maxActiveVersions` (default `5`).
### Controller: snapshot readiness
| Key | Default | Description |
| :- | :- | :- |
| `iceberg.snapshotProcessing.readiness.maxWaitMs` | `3600000` (1 h) | Max time the controller waits for all servers to confirm a snapshot is ready. |
| `iceberg.snapshotProcessing.readiness.pollIntervalMs` | `30000` | Interval between readiness polls. |
| `iceberg.snapshotProcessing.readiness.requestTimeoutMs` | `10000` | Per-request timeout for a readiness poll call. |
| `iceberg.snapshotProcessing.readiness.stallTimeoutMs` | `600000` (10 min) | Stop the poll early if no server makes load progress for this long, instead of waiting out `maxWaitMs`. Only applies while every not-ready server reports load counts (see the response fields below); a server on an older build falls back to the `maxWaitMs` deadline. |
Each **server** exposes a readiness endpoint, which the controller polls:
```
POST /tables/{tableNameWithType}/iceberg/isSnapshotReady?snapshotVersion=
```
Returns `{"ready": true|false}` plus additive load-progress fields the controller's poller uses for stall detection — `loaded`, `total`, `inFlight`, `consecutiveLoadFailures`, and `lastLoadError` (present only when set). Older pollers ignore the extra fields; a server that omits them (older build) is treated as deadline-only, never assumed to be stalled. The check is non-blocking: it answers from the server's in-memory state and, if the deletion vectors aren't loaded yet, kicks off a background load and returns `ready: false` — so the first poll for a new snapshot normally reports not-ready. Returns `412` when the table is not configured for deletion vectors (`enableDeletionVectors=false`) or the deletion-vector index for the requested snapshot is missing or unreadable.
### Controller: IS/EV convergence
Before polling for snapshot readiness, the controller waits for every affected segment's Helix ideal-state and external-view to converge:
| Key | Default | Description |
| :- | :- | :- |
| `iceberg.snapshotProcessing.convergence.maxWaitMs` | *(falls back to the effective `readiness.maxWaitMs`)* | Hard deadline for the ideal-state/external-view convergence wait. Unset by default so pre-split deployments keep their tuned readiness timeout. |
| `iceberg.snapshotProcessing.convergence.stallTimeoutMs` | `600000` (10 min) | Stop the convergence wait if no mismatched (segment, instance) pair resolves for this long. A pair in Helix `ERROR` on **every** replica stops the run immediately regardless of this window, since Helix never retries an `ERROR` replica on its own; a pair still `ERROR` on some replica when the window ends fails the run the same way. |
| `iceberg.snapshotProcessing.convergence.autoReset.maxErrorPairs` | `1000` | During the convergence wait, a segment `ERROR` on one replica while another replica is `ONLINE` is a server problem, not a segment problem — the segment is loadable. Once per run the controller issues a single bulk segment reset (`ERROR` → `OFFLINE` → reload) for every such pair in the table, then keeps waiting; this cap compares against the table-wide count of resettable pairs, and above it the run fails immediately with the count instead of resetting. `0` disables the reset, reproducing pre-auto-reset behavior: any in-scope `ERROR` pair stops the run at once. A pair `ERROR` on every replica always stops the run immediately regardless of this setting. |
### Broker: pruning and pinning
| Key | Default | Description |
| :- | :- | :- |
| `pinot.broker.external.table.cache.dir` | `/home/pinot/data/external-table/` | Local directory the broker uses for external-table snapshot/deletion-vector working data. |
| `pinot.broker.iceberg.dv.warmup.parallelism` | *(cores / 2)* | Parallelism for warming deletion-vector state at broker startup. |
| `pinot.broker.iceberg.dv.warmup.tableTimeoutMs` | `30000` | Per-table timeout during warmup. |
| `pinot.broker.iceberg.dv.warmup.totalTimeoutMs` | `300000` | Total warmup timeout across all tables. |
| `pinot.broker.iceberg.dv.cacheRetryIntervalMs` | `30000` | Retry interval when broker-side deletion-vector cache population fails. A value ≤ 0 disables background retries. |
Broker-side pruning also requires `pinot.broker.data.dir` to point at the cluster's deep-store root (StarTree Cloud sets this); without it the broker cannot read snapshot state and affected queries fail.
**The broker reads snapshot manifests with its own storage credentials, separate from the controller's.** For each active snapshot the broker downloads `{pinot.broker.data.dir}/snapshots///final.parquet` itself, so it needs a `PinotFS` for that scheme — `pinot.broker.storage.factory.class.` plus its properties, loaded at broker startup — with credentials that can read `snapshots/`. The table's `output.fs.className` / `output.fs.prop.*` task config take precedence when set; otherwise, for S3, the default AWS credential chain (env vars, IRSA/Pod Identity, instance profile) is used. Changing the broker's storage factory config or credentials requires a **broker restart** to take effect for tables that are already pruned — only a `pinot.broker.data.dir` change or a restart rebuilds an existing table's pruner.
A broker that has never loaded a snapshot for a table and lacks this access fails every query on it with `No active Iceberg snapshots are available for table `, even though the controller side and the active-snapshot list look healthy. A broker that loses access **after** loading snapshots does not fail queries — it keeps serving the newest snapshot it already loaded, even after that snapshot is retired by retention, until access is restored. Either case increments the `BROKER_ICEBERG_DV_PARQUET_DOWNLOAD_FAILURE` meter, so a steady rate there is the signal to check broker-side storage credentials even when queries are still returning results.
Broker-side snapshot pruning activates for **OFFLINE** External Tables when any of these is true on the table: `enableDeletionVectors=true`, `enableSnapshotConsistency=true` (a flag the onboarding/sync flow sets automatically for snapshotting catalogs — pins queries to the last completed snapshot so they never see a half-ingested one), or [segment groups](./segment-groups) enabled. It is a zero-cost no-op otherwise.
## Query-time snapshot pinning
You do **not** need to pin a snapshot yourself: for a pruning-enabled table the broker automatically resolves each query to the newest active snapshot and injects the server-required `snapshotVersionByTable` option. Set the option explicitly only to pin a query to a *specific* snapshot (time travel / reproducing an earlier state):
```sql theme={null}
SET snapshotVersionByTable = '{"nyc_taxi_trips_OFFLINE": 1234567890123456789}';
SELECT count(*) FROM nyc_taxi_trips;
```
If an explicitly pinned snapshot is not in the table's active set (expired, never existed, or not yet confirmed ready on all servers), the query fails with a `QUERY_EXECUTION` error rather than returning results computed from a mix of snapshot states. A query also fails when the table has no active snapshots at all yet (nothing fully synced), or when the broker hasn't finished loading the pinned snapshot's segment state locally (retried in the background; see `pinot.broker.iceberg.dv.cacheRetryIntervalMs`).
Only snapshots in the active set can be pinned, and the set's size is bounded by snapshot retention (`iceberg.snapshotProcessing.retention.maxActiveVersions`, default `5`) — so time travel reaches back at most that many snapshots.
## Storage format
The deletion-vector index itself is stored as a dictionary-encoded **Parquet** file (`dv-index.parquet`), with rows sorted by segment name within each row group and per-row-group bloom filters, which keeps heap usage low even for large delete sets and supports lazy point lookups. On the servers, cached Puffin files live under the persistent disk-cache root (`/dv-puffin-cache//`, see [Persistent disk cache](./best-practices-and-configs#persistent-disk-cache-pinot-server-instance-disk-cache)); the cache survives restarts and is cleaned up when the table is deleted.
## Observability
| Component | Metric | Meaning |
| :- | :- | :- |
| Controller | `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIME_MS` (timer) | Time for all servers to converge on a snapshot's deletion-vector state. |
| Controller | `EXTERNAL_TABLE_SYNC_SNAPSHOT_READINESS_POLL_TIME_MS` (timer) | Time spent polling servers for readiness. |
| Controller | `EXTERNAL_TABLE_SYNC_READINESS_POLL_SUCCESS` / `_ATTEMPTS` / `_FAST_FAIL` / `_TIMEOUT` / `_SKIPPED` (meters) | Outcome counters for the readiness poll loop. `_SKIPPED` fires when `enableDeletionVectors=false` or when the snapshot produced no deletion vectors (nothing to poll for). |
| Controller | `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIMEOUT` (meter) | A snapshot's ideal-state / external-view convergence timed out. |
| Controller | `EXTERNAL_TABLE_SYNC_UNHEALTHY` (gauge) | `1` when the table's sync health latch is set (set on a `FAILED` run, carried across snapshots and further failures, cleared only when a run completes `DONE`); `0` otherwise. Emitted every tick, including no-work ticks; removed on controller leadership handoff. |
| Controller | `EXTERNAL_TABLE_SYNC_CONSECUTIVE_FAILURES` (gauge) | Consecutive failed sync runs for the table. |
| Controller | `EXTERNAL_TABLE_SYNC_RUN_FAILED_STALL` / `_ERROR_REPLICA` / `_EXCEPTION` (meters) | Reason a sync run stopped: a wait (convergence or readiness) made no progress for its stall window, a (segment, instance) pair reached Helix `ERROR`, or any other exception. |
| Controller | `EXTERNAL_TABLE_SYNC_AUTO_RESET_PAIRS` (meter) | Table-wide `ERROR` (segment, instance) pairs swept by the one auto-reset a convergence wait may issue per run (see `iceberg.snapshotProcessing.convergence.autoReset.maxErrorPairs` above). A steady rate means servers keep flapping between `ERROR` and `ONLINE` and the reset is masking the cause. |
| Server | `DELETION_VECTORS_LOADED` (gauge) | Deletion vectors currently resident in memory, per table and snapshot. |
| Server | `DELETION_VECTOR_DELETED_ROWS` / `DELETION_VECTOR_LOAD_FAILURES` / `DELETION_VECTOR_INDEX_FAST_FAIL` (meters), `DELETION_VECTOR_LOAD_TIME_MS` (timer) | Deletion-vector load and application activity per server. |
| Broker | `BROKER_ICEBERG_DV_ACTIVE_SNAPSHOTS_COUNT`, `BROKER_ICEBERG_DV_LATEST_SNAPSHOT_NOT_LOADED` (gauges), `BROKER_ICEBERG_DV_FILTERED_SEGMENTS_PER_QUERY`, `BROKER_ICEBERG_DV_PARQUET_DOWNLOAD_FAILURE`, `BROKER_ICEBERG_DV_PRUNE_FAILURE`, `BROKER_ICEBERG_DV_PIN_INVALID_SNAPSHOT`, `BROKER_ICEBERG_DV_WARMUP_FAILURES` (meters), `BROKER_ICEBERG_DV_WARMUP_DURATION_MS` (timer) | Broker-side pruning and warmup health. `BROKER_ICEBERG_DV_LATEST_SNAPSHOT_NOT_LOADED` is `1`, per table, while this broker cannot yet serve the newest entry in the table's active-snapshot list (it falls back to the newest cache-ready older snapshot, or fails queries if none is ready) and `0` once it catches up; removed when the table is deleted or stops pruning. |
The run-status endpoint's `failurePhase` field (see [Observability](./observability#run-status)) can also report `WAITING_FOR_SEGMENTS_ONLINE` or `WAITING_FOR_SERVERS_READY` when a sync run fails during deletion-vector convergence.
## FAQs
### Do I need to set `snapshotVersionByTable` on every query once deletion vectors are enabled?
No — the broker pins each query to the newest active snapshot automatically and injects the option for the servers. Set it explicitly only for time travel to a specific snapshot; an explicit pin that isn't in the active set fails the query rather than returning potentially inconsistent results.
### Does this work for raw S3 Parquet (non-Iceberg) External Tables?
No — deletion vectors are an Iceberg-specific capability (`deletionVectorSource=iceberg`), tied to Iceberg's Puffin delete-file format.
# FAQ
Source: https://docs.startree.ai/corecapabilities/external-table/faq
Answers to common questions about StarTree External Tables — supported sources, onboarding, schema, indexes, performance, and operations.
This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
## General
**What is an External Table?** A Pinot table whose data stays in Parquet files in your object store (an Iceberg catalog such as AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris, Snowflake Open Catalog / Horizon, or Nessie, or a raw S3/GCS data lake) instead of being copied into Pinot's own segment format. Pinot reads the remote data at query time, with server-side caching to keep queries fast.
**Which sources are supported?** S3 Data Lake (raw Parquet), GCS Data Lake (raw Parquet via S3 interop), AWS Glue (Iceberg REST), AWS S3 Tables (Iceberg REST), Unity Catalog (Iceberg REST, including Delta Lake tables with UniForm enabled), Apache Polaris and Snowflake Open Catalog (Iceberg REST, OAuth2 principal credentials), Snowflake Horizon Catalog (Iceberg REST, PAT), and Nessie (Iceberg REST, via the generic `rest` service type with a branch/ref `prefix`).
**Does StarTree support Delta Lake?** Yes, indirectly: a Delta Lake table with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled publishes Iceberg-compatible metadata alongside its Delta log, so registering it in a **Unity Catalog** makes it queryable as an External Table through the same Iceberg REST path used for Glue and S3 Tables. There is no direct Delta-log reader — UniForm is required. Enabling UniForm means setting `'delta.enableIcebergCompatV3' = 'true'` (Databricks Runtime 18 LTS or later; `'delta.enableIcebergCompatV2' = 'true'` on older runtimes) and `'delta.universalFormat.enabledFormats' = 'iceberg'` on the Delta table, plus column mapping (`'delta.columnMapping.mode' = 'id'` recommended for new tables; existing tables can only be upgraded to `'name'`) — see the [Databricks UniForm requirements](https://docs.databricks.com/aws/en/delta/uniform#requirements) for the full prerequisites. Three operational caveats: Databricks generates the Iceberg metadata **asynchronously** after each Delta commit, so freshly written data reaches StarTree only after that conversion plus the next sync; **materialized views and streaming tables** can only be exposed through the pipeline-managed Compatibility V3 path (Databricks Runtime 17.3 or later, plus `delta.enableRowTracking` and `pipelines.externalMetadata.enabled`); and under Compatibility V3 **deletes may be written as deletion vectors** rather than by rewriting data files — StarTree applies Iceberg v3 Puffin position deletes at query time, but that path is a [preview feature](./deletion-vectors) you have to ask StarTree to enable (V2 tables keep copy-on-write deletes, which need no such setup). See [Unity Catalog: Onboarding via API](./unity/onboarding-api).
**Can I query Parquet on S3 or GCS without a catalog at all?** Yes — the **S3 Data Lake** and **GCS Data Lake** sources scan a raw bucket prefix directly, with no Iceberg table, Glue database, or catalog service of any kind. See [S3: Onboarding via API](./s3/onboarding-api) or [GCS: Onboarding via API](./gcs/onboarding-api).
**How is it different from a regular Pinot table?** A regular table ingests and stores a local copy of the data as Pinot segments. An External Table leaves the data in place and reads it remotely — so there's no data duplication and onboarding is fast, at the cost of relying on the object store plus cache for read performance.
**When should I use one — and when not?** Use it when you already have data in S3/Iceberg and want to query it without copying or re-ingesting — large or infrequently queried datasets, or data shared with other engines. Prefer a regular table for the lowest, most predictable query latency, or when you need an unsupported capability such as a sorted index.
**Does Pinot copy the Parquet into Pinot, or read it from S3 directly?** It reads directly from the Parquet files in S3 — the data is never duplicated into Pinot's segment format. Servers cache the bytes they read (data, index, and footer caches) so repeat queries are fast. See [Data and Index Caching](./data-and-index-caching).
***
## Onboarding & sync
**How do I onboard one?** Two ways: the Data Portal UI or the REST APIs. The Data Portal wizard covers [AWS Glue](/corecapabilities/external-table/glue/onboarding-data-portal), [Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-data-portal), [Unity Catalog](/corecapabilities/external-table/unity/onboarding-data-portal), [Apache Polaris & Snowflake Open Catalog](/corecapabilities/external-table/polaris/onboarding-data-portal), [Snowflake Horizon](/corecapabilities/external-table/horizon/onboarding-data-portal), [S3 Data Lake](/corecapabilities/external-table/s3/onboarding-data-portal), and [GCS Data Lake](/corecapabilities/external-table/gcs/onboarding-data-portal). The API guides cover those plus [Apache Polaris & Snowflake Open Catalog](/corecapabilities/external-table/polaris/onboarding-api), [Snowflake Horizon](/corecapabilities/external-table/horizon/onboarding-api), and [Nessie](/corecapabilities/external-table/nessie/onboarding-api) — Nessie is the only API-only source, with no Data Portal wizard yet.
**Do I have to trigger the first sync?** No. After the table is created, the controller's External Table watcher runs the first sync automatically and then re-syncs on the table's schedule. You can trigger a run manually to start sooner — see [Observability](./observability).
**How often does it sync? Can I change the schedule?** It syncs on the `schedule` (Quartz cron) in the table's `ExternalTableSyncTask` config. The Data Portal applies a default of every 5 minutes; via the API you set `schedule` yourself. Match it to your source's commit cadence — polling faster than the source changes just adds overhead.
**Can I pause and resume syncing?** Yes. In the Data Portal, open the table and select **Pause Sync** — this removes the sync schedule (stashing it so it can be restored) rather than touching the table's data or checkpoint. In-progress runs finish normally and data is preserved; select **Resume Sync** to pick up where it left off.
**Connection validation failed — what do I check?** Almost always a permission or region mismatch. Confirm the credentials can read both the catalog and the underlying S3 data, and that the region matches. For AWS Glue, the warehouse must be the numeric AWS account ID. Test Connection now checks storage credentials up front for every catalog type, and — for S3 Data Lake and S3 Tables specifically — the bucket region too; see [Onboarding fails with a region error](/corecapabilities/external-table/troubleshooting#onboarding-fails-with-a-region-error).
**Can multiple tables share the same credentials?** Yes — reuse the same catalog connection parameters across as many External Tables as you like.
**Do I put `s3://` in the bucket field?** No. The bucket field takes the bucket name only — not an `s3://...` URL — and the prefix should not have a trailing slash. Make sure the AWS region is set; a missing region is a common cause of onboarding errors. See [Troubleshooting](./troubleshooting#onboarding).
**How do I grant access to the source bucket?** Three options: an **assumed IAM role** (set `roleArn`, plus `externalId` if required), the cluster's **node IAM role**, or static **AWS access keys**. The role/keys need `s3:GetObject` and `s3:ListBucket` on the source bucket and prefix. Verify access from the cluster with `aws s3 ls s3:////` before onboarding.
**Can I onboard many tables at once?** Onboard each table through the Data Portal or the REST APIs ([AWS Glue](/corecapabilities/external-table/glue/onboarding-api) / [Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-api) / [Unity Catalog](/corecapabilities/external-table/unity/onboarding-api)). There is no single bulk table-creation API today; script the per-table API flow to automate it.
***
## Schema & data types
**How are source types mapped to Pinot types?** See [Data Type Mapping](./data-type-mapping) for the Parquet (raw S3) and Iceberg type tables. Complex types (struct, map, list) become JSON or multi-value columns.
**Can I edit the inferred schema or change a column's data type?** Avoid changing the data types that the preview/onboarding step infers. Forcing a different type (for example, switching a binary/string column to `INT`) breaks segment generation. Adjusting names, the time column, or null handling is fine.
**How do I add an inverted index (or another dictionary-backed index)?** Keep the column's forward index `RAW` and add an explicit `dictionary` block alongside the index — dictionary-backed indexes (inverted, FST, IFST) need it on External Tables:
```json theme={null}
"indexes": {
"forward": { "encodingType": "RAW" },
"dictionary": {},
"inverted": { "disabled": false }
}
```
**Can I rename columns on an External Table?** As of 0.16.0, conditionally yes: for `catalogType=iceberg-rest` sources with full Iceberg snapshot metadata access, enable `schemaEvolution.enabled=true` on the table (see [Best Practices & Configs — Checkpointing & schema evolution](./best-practices-and-configs#checkpointing--schema-evolution)). A rename in the source is tracked as an alias against the existing Pinot column rather than creating a new column, so the Pinot column name is preserved. This only resolves **top-level** field renames (not nested-field renames). Raw S3/GCS Parquet sources have no Iceberg field-id history to key off, so renames there still require a manual schema change; confirm with StarTree support whether your specific catalog adapter (Glue, S3 Tables, Unity, generic REST) has the metadata access this needs.
**Why do JSON path queries on a struct column return NULL on my Unity Catalog / Delta UniForm table?** The table was likely created with Delta column mapping (`delta.columnMapping.mode` = `name` or `id`), which stores every Parquet column under an opaque physical `col-` name. Top-level column names are always resolved back to your logical names, but names **inside** complex columns (`struct`, `array`, `map<_,struct>`) are only resolved when `storeNestedSchema.enabled` is on (the default) for the table's `ExternalTableSyncTask` config — if it was set to `false`, the nested keys are stored under their physical names and `JSON_EXTRACT_SCALAR(col, '$.yourField', ...)` returns `NULL`. See [Unity Catalog — Schema options for column mapping](./unity/onboarding-api#schema-options-for-column-mapping).
**How is the time column chosen?** Automatically, from the first suitable timestamp/date column, preserving the source granularity. See the [time column section](./data-type-mapping#time-column).
**Why are my timestamps off by \~1000×?** On current releases, plain Iceberg `timestamp` / `timestamptz` columns are inferred as **microseconds** (matching Iceberg's storage), `timestamp_ns` as nanoseconds, and `timestamp(p)` from its precision. Tables onboarded on older 0.16.0 builds were inferred as milliseconds regardless of the stored unit — that mismatch produces the \~1000× error. Fix it by setting the time column's granularity explicitly to match the data. See [Data Type Mapping](./data-type-mapping#granularity).
**Why did I get a "Requires RAW encoding" error?** External Tables require every column to use RAW (no-dictionary) encoding. A column was configured or defaulted to dictionary encoding. The preview/onboarding flow sets RAW automatically.
***
## Indexes
**Which indexes are supported?** Most of them — including inverted, range, timestamp, JSON, text, sparse, star-tree, bloom, FST, IFST, and vector (HNSW, with `storeInSegmentFile: true`). Only **sorted** and **geospatial/H3** are not supported. See [Supported Indexes](./indexes).
**Why isn't the sorted index supported?** A sorted index is the column's forward data physically stored in sorted order. External Tables read the Parquet files in place and don't own that layout, so the data can't be reordered.
**Can I add an index after creating the table?** Yes — update the table config to add a supported index, the same as any Pinot table. The index is built over the existing remote data.
***
## Performance & caching
**Where is the data cached?** On each server: a Parquet data cache, an index cache, and a Parquet footer cache. See [Data and Index Caching](./data-and-index-caching).
**The first query on a column is slow, then it's fast. Why?** The first read populates the cache from object storage; later reads hit the cache. Enable pre-warm / preload to populate caches at segment load instead of on first query.
**How do I make queries faster?** Add supported indexes for your filters, enable the page cache and preload for the table, and pre-warm at segment load. See [Best Practices and Configs](./best-practices-and-configs) for the relevant knobs.
**How do I clear a cache?** Use the server page cache endpoint — see [Data and Index Caching](./data-and-index-caching#clearing-caches).
***
## Operations
**What happens when the source data changes or files are deleted?** Each scheduled sync advances to the latest Iceberg snapshot (or, for raw S3/GCS, scans for new files) and ingests what's new; the watermark is visible as `checkpointValue` in the [status endpoint](./observability#run-status). **Append-only sources are fully handled.** How deletes and compaction are handled depends on the source type — see the next two questions.
**My source compacts or rewrites files — which source type should I use? (Iceberg REST catalog.)** Use an **Iceberg REST catalog** (`catalogType=iceberg-rest` — AWS Glue, S3 Tables, Unity Catalog, Apache Polaris / Snowflake, or Nessie), not the raw S3/GCS Data Lake source. A raw Data Lake source pins each ingested segment to a **physical Parquet file path**; when the source compacts — rewriting small files into larger ones and deleting the originals — those paths break and the server can no longer load or serve the affected segments (you'll see `java.io.FileNotFoundException` in server logs). A REST catalog instead tracks the table's live file set through **snapshot metadata**, so each sync reconciles to the current post-compaction files. Raw S3/GCS is the right fit only for **append-only** sources whose files are never rewritten. See [Segments fail to load after upstream compaction](./troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction).
**Does `continueOnFileError` protect against files deleted or compacted after ingestion?** No. `continueOnFileError=true` only lets a **sync run** skip a file it can't read *while it is generating segments* — it has no effect once a segment is already in the table. If that segment's backing file later disappears (e.g. compacted away upstream), the server fails to load or serve it regardless of this setting; the fix is to use an Iceberg REST catalog for compacting sources (above).
**What's the difference between `continueOnError` and `continueOnFileError`?** They are two separate configs, set in different places, and easy to confuse. `continueOnError` is an OSS `ingestionConfig` field that skips **individual bad rows** while reading a file or stream. `continueOnFileError` is an `ExternalTableSyncTask` field that skips an **entire unreadable file** so one bad file doesn't fail the whole run. Both apply only during ingestion/sync — neither repairs an already-ingested segment.
***
## Monitoring & observability
**How do I check whether my last sync succeeded?** Call the [status endpoint](./observability#run-status) and read `fileOnboardingRun.status` (`COMPLETED` / `RUNNING` / `FAILED` / `IDLE`). On failure, `failurePhase` and `errorMessage` say where and why. See [Observability](./observability).
**Does `status: COMPLETED` mean my data is current?** No. Run status only means the last sync run *finished* — the source may have committed more data since. To judge freshness, call `…/externalTable/status?includeLag=true` and read `syncState`, `lag.percentDataIngested`, and `lag.snapshotsBehind`. See [Ingestion lag](./observability#ingestion-lag).
**How far behind the source is my data?** With `?includeLag=true`, subtract `lag.synced.committedAt` from `lag.upstream.committedAt` for a time delta, or read `lag.snapshotsBehind` / `lag.percentDataIngested`. There is deliberately no single "lag in seconds" field.
**My sync shows `COMPLETED` but the table is empty or stale — why?** A snapshot whose segment generation keeps failing is retried up to `maxRetries` (default 3) runs, then abandoned; a later run with no newer snapshot then reports `COMPLETED` with `segmentsUploaded: 0`, and `fileOnboardingRun` — which only ever describes the run happening now — hides the earlier failures. Compare `filesDiscovered` vs `segmentsUploaded`, check `syncState`, and check `snapshotIngestion.consecutiveFailedRuns` / `lastRunError` for the hidden failure streak. See [Sync says COMPLETED but no data lands](./troubleshooting#sync-says-completed-but-no-data-lands).
**Can I monitor sync from dashboards instead of polling the API?** The controller emits per-phase sync timers (`EXTERNAL_TABLE_SYNC_*_TIME_MS`) for latency dashboards, but there is no lag or per-run success/failure metric — alert on freshness and failures through the status API. See [Metrics and alerting](./observability#metrics-and-alerting).
# GCS Data Lake: Onboarding via API
Source: https://docs.startree.ai/corecapabilities/external-table/gcs/onboarding-api
Onboard raw Parquet files on Google Cloud Storage as a StarTree External Table via the REST API, using GCS's S3-compatible interop endpoint and HMAC keys.
This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it.
This page shows how to onboard a GCS Data Lake External Table with the StarTree controller REST APIs instead of the [Data Portal UI](./onboarding-data-portal). Use it for automation, infrastructure-as-code, or onboarding many tables at once.
GCS Data Lake uses Google Cloud Storage accessed through its **S3-compatible ("interop") endpoint** (`storage.googleapis.com`). The catalog type is `gcs-interop` and it reuses the same S3 wiring as [S3 Data Lake](../s3/onboarding-api) — only the endpoint, addressing style, and credential source differ.
## How it works
Onboarding is four calls. Each one feeds the next:
```text theme={null}
1. Validate & browse POST /connections/browse → validate; list files under the GCS prefix
2. Preview POST /tables/preview → infer schema + enriched table config
3. Create schema POST /schemas → register the schema
4. Create table POST /tables → register the External Table
```
The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output.
Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding).
All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` — the StarTree Cloud data-plane proxy that fronts the controller.
If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity.
## Prerequisites
* StarTree release with GCS External Table support enabled, and **tiered storage configured** on the cluster with a `GCS_INTEROPERABLE` tier backend. Contact StarTree support if unsure.
* Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth.
* A GCS bucket with Parquet files and a GCS HMAC key (access ID + secret) with `storage.objects.get` and `storage.objects.list` on the bucket. Generate HMAC keys in the [Google Cloud Console → Cloud Storage → Settings → Interoperability](https://console.cloud.google.com/storage/settings;tab=interoperability).
* The GCS bucket name, key prefix, and the project's region (used as a placeholder — GCS ignores it but the field is required).
## Limitations
* **HMAC keys are the only credential type** — there's no IAM-role or workload-identity option for GCS Data Lake today.
* **`disable.integrity.protections` must be `true`** for every GCS connection — GCS's checksum behavior differs from AWS S3's, and the AWS SDK mis-validates it otherwise. See the Note below.
* **No schema evolution** — same limitation as the S3 Data Lake source; a source-side column rename requires a manual schema change.
## Authentication
GCS Data Lake authenticates using HMAC keys (Google's S3-interop credentials) — there's no assumed-role equivalent since this path goes through the AWS SDK's S3-compatible client, not native GCP IAM. Choose one of two methods.
`disable.integrity.protections` must be set to `true` for all GCS connections, regardless of method. GCS rejects the AWS SDK's default request checksums on writes and returns whole-object checksums on range GETs that the SDK mis-validates. Setting this flag relaxes both to `WHEN_REQUIRED`, which is required for reads and writes to succeed.
### Inline HMAC keys
Quick tests, or when Secret Manager is not available. Generate HMAC keys in the [Google Cloud Console → Cloud Storage → Settings → Interoperability](https://console.cloud.google.com/storage/settings;tab=interoperability).
```json theme={null}
{
"catalog.gcs-interop.bucketName": "",
"catalog.gcs-interop.prefix": "",
"catalog.gcs-interop.region": "auto",
"catalog.gcs-interop.endpoint": "https://storage.googleapis.com",
"catalog.gcs-interop.accessKey": "",
"catalog.gcs-interop.secretKey": "",
"catalog.gcs-interop.disable.integrity.protections": "true"
}
```
### GCP Secret Manager (recommended for production)
`accessKey`/`secretKey` are treated as **secret names** rather than literal values, and resolved from GCP Secret Manager at read time — no HMAC material stored in the table config.
```json theme={null}
{
"catalog.gcs-interop.bucketName": "",
"catalog.gcs-interop.prefix": "",
"catalog.gcs-interop.region": "auto",
"catalog.gcs-interop.endpoint": "https://storage.googleapis.com",
"catalog.gcs-interop.accessKey": "",
"catalog.gcs-interop.secretKey": "",
"catalog.gcs-interop.keyType": "SECRET",
"catalog.gcs-interop.secretmanagertype": "GCS",
"catalog.gcs-interop.gcpprojectid": "",
"catalog.gcs-interop.gcpkeypath": "",
"catalog.gcs-interop.disable.integrity.protections": "true"
}
```
The service account at `gcpkeypath` needs Secret Manager read access to the two named secrets.
***
## Step 1: Validate and browse the connection
**`POST /connections/browse`**
There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your credentials and connectivity.
* Set `path` to `""` to browse the **root prefix** — this both validates the connection and lists files/directories.
### Request
| Field | Type | Required | Description |
| - | - | - | - |
| `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. |
| `connection.params` | object | Yes | GCS connection settings (see examples below). |
| `path` | string | No | Where to browse. `""` or omitted = root prefix. |
```bash theme={null}
curl -X POST "$BASE_URL/connections/browse" \
-H "Content-Type: application/json" \
-d '{
"connection": {
"type": "CATALOG",
"params": {
"catalogType": "gcs-interop",
"catalog.gcs-interop.bucketName": "",
"catalog.gcs-interop.prefix": "",
"catalog.gcs-interop.region": "auto",
"catalog.gcs-interop.endpoint": "https://storage.googleapis.com",
"catalog.gcs-interop.accessKey": "",
"catalog.gcs-interop.secretKey": "",
"catalog.gcs-interop.disable.integrity.protections": "true"
}
},
"path": ""
}'
```
```bash theme={null}
curl -X POST "$BASE_URL/connections/browse" \
-H "Content-Type: application/json" \
-d '{
"connection": {
"type": "CATALOG",
"params": {
"catalogType": "gcs-interop",
"catalog.gcs-interop.bucketName": "",
"catalog.gcs-interop.prefix": "",
"catalog.gcs-interop.region": "auto",
"catalog.gcs-interop.endpoint": "https://storage.googleapis.com",
"catalog.gcs-interop.accessKey": "",
"catalog.gcs-interop.secretKey": "",
"catalog.gcs-interop.keyType": "SECRET",
"catalog.gcs-interop.secretmanagertype": "GCS",
"catalog.gcs-interop.gcpprojectid": "",
"catalog.gcs-interop.gcpkeypath": "",
"catalog.gcs-interop.disable.integrity.protections": "true"
}
},
"path": ""
}'
```
### Response
```json theme={null}
{
"items": [
{ "name": "2024/01/", "type": "DIR" },
{ "name": "data.parquet", "type": "FILE" }
]
}
```
| `items[].type` | Meaning |
| - | - |
| `DIR` | A directory you can drill into — pass its `name` as `path`. |
| `FILE` | A Parquet file under the prefix. |
***
## Step 2: Preview the schema
**`POST /tables/preview`**
Samples the Parquet files, infers a Pinot schema, and returns an enriched table config plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4.
The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`.
### Request
**Top-level fields:**
| Field | Type | Required | Description |
| - | - | - | - |
| `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. |
| `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. |
| `schema` | object | No | An explicit schema to use instead of inferring one. |
**`config.inference` — how the schema is derived:**
| Field | Type | Default | Description |
| - | - | - | - |
| `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Parquet metadata. **Set this to `true` for External Tables.** |
| `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. |
**`config.sampling` — how rows are sampled:**
| Field | Type | Default | Description |
| - | - | - | - |
| `min` | integer | `1` | Minimum records to sample. |
| `max` | integer | `100` | Maximum records to sample. |
To list source files without sampling any data, set `config.previewFiles.previewFilesOnly = true`. The response returns matching file URIs in `sourceFiles` and skips schema inference. (OFFLINE only.)
```bash theme={null}
curl -X POST "$BASE_URL/tables/preview" \
-H "Content-Type: application/json" \
-d '{
"tableConfig": {
"tableName": "my_gcs_table_OFFLINE",
"tableType": "OFFLINE",
"task": {
"taskTypeConfigsMap": {
"ExternalTableSyncTask": {
"catalogType": "gcs-interop",
"executor": "controller",
"inputFormat": "parquet",
"catalog.gcs-interop.bucketName": "",
"catalog.gcs-interop.prefix": "",
"catalog.gcs-interop.region": "auto",
"catalog.gcs-interop.endpoint": "https://storage.googleapis.com",
"catalog.gcs-interop.accessKey": "",
"catalog.gcs-interop.secretKey": "",
"catalog.gcs-interop.table.namespace": "default",
"catalog.gcs-interop.table.tableName": "my_gcs_table",
"catalog.gcs-interop.disable.integrity.protections": "true"
}
}
}
},
"config": {
"inference": { "embeddedSchemaEnabled": true },
"sampling": { "min": 1, "max": 100 }
}
}'
```
```bash theme={null}
curl -X POST "$BASE_URL/tables/preview" \
-H "Content-Type: application/json" \
-d '{
"tableConfig": {
"tableName": "my_gcs_table_OFFLINE",
"tableType": "OFFLINE",
"task": {
"taskTypeConfigsMap": {
"ExternalTableSyncTask": {
"catalogType": "gcs-interop",
"executor": "controller",
"inputFormat": "parquet",
"catalog.gcs-interop.bucketName": "",
"catalog.gcs-interop.prefix": "",
"catalog.gcs-interop.region": "auto",
"catalog.gcs-interop.endpoint": "https://storage.googleapis.com",
"catalog.gcs-interop.accessKey": "",
"catalog.gcs-interop.secretKey": "",
"catalog.gcs-interop.keyType": "SECRET",
"catalog.gcs-interop.secretmanagertype": "GCS",
"catalog.gcs-interop.gcpprojectid": "",
"catalog.gcs-interop.gcpkeypath": "",
"catalog.gcs-interop.table.namespace": "default",
"catalog.gcs-interop.table.tableName": "my_gcs_table",
"catalog.gcs-interop.disable.integrity.protections": "true"
}
}
}
},
"config": {
"inference": { "embeddedSchemaEnabled": true },
"sampling": { "min": 1, "max": 100 }
}
}'
```
### Response
| Field | Type | Description |
| - | - | - |
| `schema` | object | The resolved Pinot schema. **Use this in step 3.** |
| `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. |
| `rows` | array | Sample records after schema + ingestion transforms. |
| `sourceRows` | array | Raw records before transformation. |
| `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). |
***
## Step 3: Create the schema
**`POST /schemas`**
Send the `schema` object from the preview response. Rename its `schemaName` to match your table.
```bash theme={null}
curl -X POST "$BASE_URL/schemas" \
-H "Content-Type: application/json" \
-d @schema.json
```
```json theme={null}
{ "status": "my_gcs_table successfully added" }
```
***
## Step 4: Create the table
**`POST /tables`**
Send the enriched `tableConfigs.offline` object from the preview response. The `ExternalTableSyncTask` block marks the table as external, and the `tierConfigs` array pins it to the `GCS_INTEROPERABLE` tier.
```bash theme={null}
curl -X POST "$BASE_URL/tables" \
-H "Content-Type: application/json" \
-d @tableConfig.json
```
```json theme={null}
{ "status": "Table my_gcs_table_OFFLINE successfully added" }
```
The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call.
The `tableConfigs.offline` returned by `/tables/preview` includes a `tierConfigs` block pre-configured for `GCS_INTEROPERABLE`. Verify that the tier backend credentials match the source catalog credentials (they can differ if you use separate HMAC keys or Secret Manager secrets for the deep store).
The first sync runs on the watcher's next tick. To kick it off immediately:
```bash theme={null}
curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=my_gcs_table_OFFLINE"
```
***
## Quickstart: onboard a table end-to-end
The whole flow as one script (inline HMAC mode). Fill in the variables at the top, run the script, and the first sync starts automatically.
```bash theme={null}
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="https://dp..cp..startree.cloud/api/pinot"
AUTH="Authorization: Bearer "
BUCKET=""
PREFIX=""
REGION="" # Required field; GCS ignores the value
HMAC_KEY=""
HMAC_SECRET=""
TABLE="my_gcs_table"
# 1. Validate & browse — confirms credentials and lists files.
curl -sf -X POST "$BASE_URL/connections/browse" \
-H "$AUTH" -H 'Content-Type: application/json' \
--data-raw "{
\"connection\": {
\"type\": \"CATALOG\",
\"params\": {
\"catalogType\": \"gcs-interop\",
\"catalog.gcs-interop.bucketName\": \"$BUCKET\",
\"catalog.gcs-interop.prefix\": \"$PREFIX\",
\"catalog.gcs-interop.region\": \"auto\",
\"catalog.gcs-interop.endpoint\": \"https://storage.googleapis.com\",
\"catalog.gcs-interop.accessKey\": \"$HMAC_KEY\",
\"catalog.gcs-interop.secretKey\": \"$HMAC_SECRET\",
\"catalog.gcs-interop.disable.integrity.protections\": \"true\"
}
},
\"path\": \"\"
}" | jq .
# 2. Preview — infer schema + enriched table config.
curl -sf -X POST "$BASE_URL/tables/preview" \
-H "$AUTH" -H 'Content-Type: application/json' \
--data-raw "{
\"tableConfig\": {
\"tableName\": \"${TABLE}_OFFLINE\",
\"tableType\": \"OFFLINE\",
\"task\": {
\"taskTypeConfigsMap\": {
\"ExternalTableSyncTask\": {
\"catalogType\": \"gcs-interop\",
\"executor\": \"controller\",
\"inputFormat\": \"parquet\",
\"catalog.gcs-interop.bucketName\": \"$BUCKET\",
\"catalog.gcs-interop.prefix\": \"$PREFIX\",
\"catalog.gcs-interop.region\": \"auto\",
\"catalog.gcs-interop.endpoint\": \"https://storage.googleapis.com\",
\"catalog.gcs-interop.accessKey\": \"$HMAC_KEY\",
\"catalog.gcs-interop.secretKey\": \"$HMAC_SECRET\",
\"catalog.gcs-interop.table.namespace\": \"default\",
\"catalog.gcs-interop.table.tableName\": \"$TABLE\",
\"catalog.gcs-interop.disable.integrity.protections\": \"true\"
}
}
}
},
\"config\": {
\"inference\": { \"embeddedSchemaEnabled\": true },
\"sampling\": { \"min\": 1, \"max\": 100 }
}
}" > preview.json
# 3. Extract schema and table config from preview response.
jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json
jq '.tableConfigs.offline' preview.json > tableConfig.json
# 4. Create the schema.
curl -sf -X POST "$BASE_URL/schemas" \
-H "$AUTH" -H 'Content-Type: application/json' -d @schema.json
# 5. Create the table — watcher starts the first sync automatically.
curl -sf -X POST "$BASE_URL/tables" \
-H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json
# 6. (optional) Poll until sync completes.
curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \
| jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered'
```
Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable).
## Monitor onboarding
Three read-only endpoints report ingestion progress — run status, ingestion checkpoint, and source file count — and require `executor=controller` (set automatically). See [Observability](../observability) for full request and response details.
## Verify it's queryable
When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live:
```sql theme={null}
SELECT count(*) FROM my_gcs_table;
```
The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting).
***
## GCS-specific config reference
All keys go under the `ExternalTableSyncTask` block in `taskTypeConfigsMap`.
| Key | Required | Description |
| - | - | - |
| `catalog.gcs-interop.bucketName` | Yes | GCS bucket name. |
| `catalog.gcs-interop.prefix` | Yes | Key prefix for the Parquet data (e.g. `path/to/data/`). |
| `catalog.gcs-interop.region` | Yes | Region string. GCS ignores the value but the field must be present with the value "auto". |
| `catalog.gcs-interop.endpoint` | Yes | Always `https://storage.googleapis.com`. |
| `catalog.gcs-interop.accessKey` | Yes | HMAC access ID, or GCP Secret Manager secret name if `keyType=SECRET`. |
| `catalog.gcs-interop.secretKey` | Yes | HMAC secret, or GCP Secret Manager secret name if `keyType=SECRET`. |
| `catalog.gcs-interop.disable.integrity.protections` | Yes | Must be `"true"`. Required for GCS range-GET and PUT compatibility with the AWS SDK. |
| `catalog.gcs-interop.keyType` | No | Set to `SECRET` to resolve `accessKey`/`secretKey` via GCP Secret Manager. |
| `catalog.gcs-interop.secretmanagertype` | If `keyType=SECRET` | Must be `GCS`. |
| `catalog.gcs-interop.gcpprojectid` | If `keyType=SECRET` | GCP project ID for Secret Manager lookups. |
| `catalog.gcs-interop.gcpkeypath` | If `keyType=SECRET` | Path to a service-account key JSON with Secret Manager read access. |
| `catalog.gcs-interop.table.namespace` | No | Logical namespace. Use `default` for a flat prefix. |
| `catalog.gcs-interop.table.tableName` | No | Logical table name (any value; used internally). |
***
## What's next
Now that the table is created and data is loading, these are the highest-impact follow-up steps:
1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes)
2. **Enable caching and preload.** Set `enable.prefetch.page.cache=true` and `preload.enable=true` on the S3 tier so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching)
3. **Protect large-scan queries from OOM.** For tables that receive heavy aggregations or wide scans, enable the query OOM killer so a runaway query is killed instead of crashing the server. → [Best Practices & Configs — Query OOM protection](../best-practices-and-configs#query-oom-protection-large-scans)
4. **Monitor ongoing syncs.** Use the observability endpoints to check run status, ingestion checkpoint, and source file count after each scheduled sync. → [Observability](../observability)
***
For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting).
# GCS Data Lake: Onboarding via Data Portal
Source: https://docs.startree.ai/corecapabilities/external-table/gcs/onboarding-data-portal
Connect raw Parquet files on GCS to StarTree as an External Table with the point-and-click Data Portal wizard, via the S3-compatible interop endpoint.
This feature is available starting in **StarTree release 0.16.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment.
This guide walks through connecting a GCS Data Lake to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface.
> Looking for the API-based approach? See [GCS Data Lake: Onboarding via API](./onboarding-api).
***
## Overview
**GCS Data Lake** connects to raw Parquet files stored in a Google Cloud Storage bucket, accessed through GCS's S3-compatible ("interop") endpoint (`storage.googleapis.com`). StarTree scans the specified GCS prefix, discovers Parquet files, and makes them queryable through Pinot.
Authentication uses **HMAC keys** (Google's S3-interop credentials). Generate HMAC keys in the [Google Cloud Console → Cloud Storage → Settings → Interoperability](https://console.cloud.google.com/storage/settings;tab=interoperability).
***
## Prerequisites
Before starting, ensure you have:
* **StarTree 0.16.0 or later** with the external table Beta feature enabled and a `GCS_INTEROPERABLE` tier backend configured for your environment.
* A GCS HMAC key (access ID + secret) with `storage.objects.get` and `storage.objects.list` permissions on the source bucket.
* The GCS bucket name, key prefix, and GCP project region.
***
## Step 1: Open the External Tables
1. Log in to **Data Portal**.
2. In the left navigation, go to **Tables**.
3. Click **+ Connect External Table**.
The wizard opens with a connection configuration screen.
***
## Step 2: Select GCS Data Lake
Choose **GCS Data Lake** as the catalog type.
Fill in the connection details:
| Field | Description |
| - | - |
| **Bucket Name** | Name of the GCS bucket containing your Parquet files. |
| **Region** | GCS S3-interoperability region — the default `auto` works for most buckets. |
| **Endpoint** | The GCS S3-interop endpoint; keep the default `https://storage.googleapis.com`. |
| **Authentication Type** | **Inline HMAC Keys** or **GCP Secret Manager**. |
With **Inline HMAC Keys**:
| Field | Description |
| - | - |
| **HMAC Access ID** | The HMAC access ID from the Google Cloud Console Interoperability settings. |
| **HMAC Secret** | The HMAC secret corresponding to the access ID. |
With **GCP Secret Manager** (credentials resolved from Secret Manager instead of entered inline):
| Field | Description |
| - | - |
| **Access ID Secret Name** | Secret Manager secret holding the HMAC access ID. |
| **Secret Key Secret Name** | Secret Manager secret holding the HMAC secret. |
| **GCP Project ID** | Project that owns the secrets. |
| **Service Account Key Path** | Path to a GCP service-account JSON key used to access Secret Manager. |
Click **Validate Connection**. Data Portal confirms credentials and connectivity via the GCS interop endpoint before proceeding.
***
## Step 3: Browse and Select a Table
Once the connection is validated, enter the **Prefix** — the key prefix (folder path) pointing to the Parquet data, e.g. `path/to/parquet/data` (no leading slash; the field auto-sanitizes the value). Then:
1. Data Portal lists the available **GCS prefixes** (directories) under your configured prefix.
2. Select a prefix to view the Parquet files it contains.
3. Click the **table** (prefix) you want to onboard.
StarTree samples the Parquet files to derive a schema automatically.
***
## Step 4: Review the Schema
The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity.
Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct.
***
## Step 5: Configure the Table
Set the final table options:
| Setting | Default | Notes |
| - | - | - |
| **Table Name** | — | Name of the Pinot table to create. |
| **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. |
| **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. |
The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config.
Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required.
***
## Step 6: Monitor Onboarding
Once the table is created, onboarding starts automatically. The table detail view shows the status in real time:
* **Running** — the task is actively reading Parquet files and building Pinot segments.
* **Completed** — onboarding finished successfully.
* **Failed** — onboarding encountered an error. The error message and the number of files discovered vs. segments generated are surfaced to help diagnose the issue.
For deeper observability — watcher status, checkpoint values, and per-file counts — see the [Observability](../observability) page.
***
## Pausing Onboarding
To pause scheduled onboarding from Data Portal:
1. Open the table in the **Tables** view.
2. Click **Pause Sync**. This removes the `schedule` (cron trigger) from the table's `ExternalTableSyncTask` config and stashes its value under `schedule.paused`, so the watcher stops firing new runs. Any run currently in progress completes normally, and existing segments and the last checkpoint are untouched. Click **Resume Sync** to restore the schedule from `schedule.paused` — onboarding resumes from where it left off; it does not restart from scratch.
This is a separate mechanism from the `enabled` key documented in [Best Practices & Configs](../best-practices-and-configs#table-config--externaltablesynctask) — that key can also be set to `false` directly via the table config or API to stop new runs, but the Data Portal's **Pause Sync** button acts on `schedule`/`schedule.paused`, not `enabled`.
***
## Frequently Asked Questions
**The Validate step fails — what should I check?**
* Confirm the HMAC key has `storage.objects.get` and `storage.objects.list` permissions on the bucket.
* Verify the bucket name and prefix are correct — GCS bucket names are globally unique.
* Ensure the HMAC key was generated under the correct GCP project.
***
**Can I onboard multiple prefixes from the same bucket?**
Yes. Each prefix is registered as an independent Pinot table with its own onboarding schedule. Start the wizard again and use the same bucket with a different prefix.
***
**The table was created but onboarding hasn't started — what should I check?**
Data Portal triggers the first onboarding run automatically after table creation. If onboarding hasn't started, check the table's detail page for an error status and review the error message. You can also trigger a run manually via the [trigger API](../observability#trigger-a-sync-run-optional).
# Getting Started: Connect a Data Catalog
Source: https://docs.startree.ai/corecapabilities/external-table/getting-started/connect-a-catalog
Connect StarTree to an Iceberg-compatible data catalog — AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris, Snowflake Open Catalog / Horizon, or Nessie — and onboard a registered table as an External Table.
A data catalog is a central registry that tracks a table's schema, partitions, and file locations, so a query engine can look those up instead of scanning a bucket to rediscover them. If your tables are already registered in one of the catalogs below, StarTree reads that registry directly — you get schema inference, partition awareness, and incremental snapshot sync for free, instead of the raw file-scanning approach in [Query Parquet Directly](./query-without-a-catalog).
**A catalog is also the right choice when your source compacts.** Because a REST catalog tracks the table's live file set through snapshot metadata, each sync reconciles to the current files even after the source rewrites and deletes Parquet during compaction. The raw [S3/GCS Data Lake](./query-without-a-catalog) source pins segments to physical file paths, so compaction breaks them — see [Segments fail to load after upstream compaction](../troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction). If your pipeline compacts, connect a catalog rather than pointing at the bucket directly.
## Which catalog do you have?
All of them are reached the same way — `catalogType=iceberg-rest` — over the **Iceberg REST protocol**; only the service endpoint and authentication differ.
| Catalog | What it is | Onboarding guide |
| - | - | - |
| **AWS Glue** | AWS's managed Iceberg REST catalog, backed by the Glue Data Catalog | [Onboarding via API](../glue/onboarding-api) · [via Data Portal](../glue/onboarding-data-portal) |
| **Amazon S3 Tables** | AWS's managed Iceberg tables, addressed by table-bucket ARN | [Onboarding via API](../s3tables/onboarding-api) · [via Data Portal](../s3tables/onboarding-data-portal) |
| **Unity Catalog** | Databricks-managed or OSS Unity Catalog — also covers Delta Lake tables with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled | [Onboarding via API](../unity/onboarding-api) · [via Data Portal](../unity/onboarding-data-portal) |
| **Apache Polaris / Snowflake Open Catalog** | Self-managed Apache Polaris or Snowflake Open Catalog, over the generic Iceberg REST adapter with OAuth2 principal credentials | [Onboarding via API](../polaris/onboarding-api) · [via Data Portal](../polaris/onboarding-data-portal) |
| **Snowflake Horizon Catalog** | Snowflake-managed Iceberg tables, served under `/polaris/api/catalog` on the account host, authenticated with a PAT | [Onboarding via API](../horizon/onboarding-api) · [via Data Portal](../horizon/onboarding-data-portal) |
| **Nessie** | A Git-like, branch/ref-aware Iceberg catalog, self-hosted or managed | [Onboarding via API](../nessie/onboarding-api) |
Not sure which authentication method to use for your catalog, or whether a given feature (assumed-role auth, vended credentials, OAuth2) is supported? Each catalog's onboarding guide has an Authentication section with a runnable config sample per method.
## How onboarding works, regardless of catalog
Every catalog above follows the same four calls: **validate & browse** the connection, **preview** to infer a schema, **create the schema**, then **create the table** — after which the controller's watcher takes over and syncs on a schedule with no further manual steps. Each catalog's onboarding guide walks through all four with runnable examples.
Validate & browse also checks that the storage credentials can actually reach the data bucket, not just the catalog endpoint — for S3 Data Lake and S3 Tables this includes confirming the configured region matches the bucket at that step; for the other catalogs, where no bucket is known until a table is picked, the storage region is confirmed later, once a table is selected in preview or `CREATE TABLE`. See [Support Matrix](../support-matrix#connection-test-storage-validation).
## What's next
Once the table is created, add [indexes](../indexes) for the columns you filter and group by, enable [caching](../data-and-index-caching), and track sync health through [Observability](../observability).
# Getting Started: Query Parquet Without a Catalog
Source: https://docs.startree.ai/corecapabilities/external-table/getting-started/query-without-a-catalog
Query raw Parquet files on S3 or GCS as a StarTree External Table with no Iceberg table, Glue database, or catalog service in front of them.
**Yes — if your Parquet files just sit under a bucket prefix with nothing else in front of them, StarTree can query them directly as an External Table.** No Iceberg metadata, no Glue database, no catalog REST endpoint, no separate registration step. Point StarTree at the bucket and prefix and it discovers the files itself.
## When this is the right fit
* You have a data pipeline (Spark, Flink, a batch job) that lands Parquet files in S3 or GCS, with no catalog layer on top.
* You want to try External Tables against a sample dataset before setting up a full Iceberg catalog.
* Your data lake uses a catalog for *other* consumers, but you'd rather point StarTree straight at the underlying files.
If your data is already registered in a catalog — AWS Glue, Amazon S3 Tables, Databricks/OSS Unity Catalog, or Nessie — skip ahead to [Connect a Catalog](./connect-a-catalog) instead; StarTree can use the catalog's schema and file-listing directly rather than re-discovering it from raw files.
**Raw S3/GCS is for append-only sources.** A Data Lake source pins each ingested segment to a physical Parquet file path, so it works cleanly only when files are added and never rewritten. If your pipeline **compacts** the data — rewriting small files into larger ones and deleting the originals — those paths break and the affected segments can no longer be loaded or served (`java.io.FileNotFoundException`). For a source that compacts, use an [Iceberg REST catalog](./connect-a-catalog) instead: it tracks the live file set through snapshot metadata and reconciles compaction on each sync. See [Segments fail to load after upstream compaction](../troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction).
## Two sources, same shape
Both scan a bucket prefix, infer a schema by sampling the Parquet files, and sync on a schedule — the only difference is which object store and credential type you use.
| Source | Object store | Credentials | Onboarding guide |
| - | - | - | - |
| **S3 Data Lake** | Amazon S3 | Assumed IAM role, cluster node role, or static access keys | [S3: Onboarding via API](../s3/onboarding-api) · [via Data Portal](../s3/onboarding-data-portal) |
| **GCS Data Lake** | Google Cloud Storage (via its S3-compatible interop endpoint) | HMAC keys (inline or via GCP Secret Manager) | [GCS: Onboarding via API](../gcs/onboarding-api) · [via Data Portal](../gcs/onboarding-data-portal) |
Prefer the REST API for automation and scripting multiple tables; use the Data Portal wizard for a one-off table with no JSON to write.
Also see the [SQL DDL](../sql-ddl) shortcut — `CREATE TABLE ... WITH (type = 's3', ...)` creates the same table in one statement instead of the 4-step REST flow.
## What's next
Once the table is created, add [indexes](../indexes) for the columns you filter and group by, enable [caching](../data-and-index-caching) so repeat queries don't re-fetch from object storage, and track sync health through [Observability](../observability).
# Getting Started: Speed Up External Table Queries
Source: https://docs.startree.ai/corecapabilities/external-table/getting-started/speed-up-queries
Make StarTree External Table queries fast with the right indexes, caching and preload settings, and protection against out-of-memory scans.
**External Table queries read Parquet from object storage at query time instead of a local segment, so the fastest path to low latency is indexes and caching — not more compute.** Three changes cover almost every slow-query case:
## 1. Add indexes for your filter and group-by columns
Without an index, every query does a full remote scan. A range index on time/numeric columns and an inverted index on the low-cardinality columns you filter on most will fix the majority of slow queries.
→ [Supported Indexes](../indexes) — what's supported, how to add one, and why every column starts out RAW (no dictionary).
## 2. Turn on caching and preload
The first read of any column pays the cost of fetching it from object storage; caching keeps that data local so repeat queries don't pay it again. Preload goes further, pulling small hot indexes (bloom filters, dictionaries) into memory eagerly at segment load instead of on first query.
→ [Data and Index Caching](../data-and-index-caching) — how the Parquet data cache, index cache, and footer cache work.
→ [Best Practices and Configs — Caching and index pinning](../best-practices-and-configs#caching-and-index-pinning) — which indexes to pin vs. leave to the cache.
## 3. Protect large scans from OOM
Tables that get heavy aggregations or wide scans can push a server's heap to its limit. Server-side per-query memory accounting kills the heaviest query instead of crashing the server when that happens.
→ [Best Practices and Configs — Query OOM protection](../best-practices-and-configs#query-oom-protection-large-scans)
## Still slow?
If you've done all three and a specific query is still slow or times out, check [Troubleshooting — Query times out or returns "servers not responded"](../troubleshooting#query-times-out-or-returns-servers-not-responded) for the full diagnostic order, including under-provisioned servers and derived-column group-bys that defeat pruning.
# AWS Glue: Onboarding via API
Source: https://docs.startree.ai/corecapabilities/external-table/glue/onboarding-api
Onboard an AWS Glue Iceberg table as a StarTree External Table over the REST catalog protocol, with IAM-role, node-role, or static-key auth.
This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
This page shows how to onboard an **AWS Glue** Iceberg External Table with the StarTree controller REST APIs instead of the [Data Portal UI](./onboarding-data-portal). Use it for automation, infrastructure-as-code, or onboarding many tables at once.
Glue tables are accessed over the **Iceberg REST protocol** (`catalogType=iceberg-rest`, `serviceType=glue`) with AWS SigV4 authentication. Data files must be **Parquet**.
> Using **Amazon S3 Tables** instead? See [Amazon S3 Tables: Onboarding via API](../s3tables/onboarding-api).
>
> Using **Nessie** instead? See [Nessie: Onboarding via API](../nessie/onboarding-api).
## How it works
Onboarding is four calls. Each one feeds the next:
```text theme={null}
1. Validate & browse POST /connections/browse → validate; list namespaces/tables
2. Preview POST /tables/preview → infer schema + enriched table config
3. Create schema POST /schemas → register the schema
4. Create table POST /tables → register the External Table
```
The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output.
Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding).
All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` — the StarTree Cloud data-plane proxy that fronts the controller.
If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity.
## Prerequisites
* StarTree 0.15.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure.
* Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth.
* AWS credentials (access key + secret key) with read access to the Glue catalog **and** the underlying S3 data.
* Your AWS account ID (the catalog warehouse) and the Glue database name.
**Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly.
If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details.
## Limitations
* **Iceberg tables only.** This path reads Glue's Iceberg REST Catalog API — plain Hive/Parquet tables registered in Glue (non-Iceberg) aren't reachable here. A separate native AWS Glue Data Catalog code path exists, but it can only browse databases/tables today, not complete onboarding.
* **`catalog.iceberg-rest.auth.rest.service` must be set to `glue` explicitly** or the SigV4 signature is computed for the wrong AWS service — see the Warning under [Catalog (REST) authentication](#catalog-rest-authentication-aws-sigv4).
## Authentication
An External Table authenticates in two places, configured independently so the catalog and the data can use different credentials:
* **Catalog (REST)** — keys under `catalog.iceberg-rest.auth.rest.*`. Calls to the Glue Iceberg REST endpoint always use AWS SigV4; the signing credentials come from static keys or an assumed IAM role.
* **Storage (S3 data files)** — keys under `catalog.iceberg-rest.auth.storage.*`. Choose any one of three methods.
The role or keys need `s3:GetObject` and `s3:ListBucket` on the source bucket/prefix, plus Glue read access (`glue:GetDatabase`, `glue:GetTable`, `glue:GetPartitions`) on the catalog. Verify S3 access from the cluster before onboarding:
```bash theme={null}
aws s3 ls s3:////
```
### Catalog (REST) authentication: AWS SigV4
The Glue Iceberg REST endpoint accepts only SigV4-signed requests. The signing credentials can come from **static keys** or an **assumed IAM role** — pick one.
#### Static keys
`authType` is optional — it's auto-detected as `aws-sigv4` once `accessKeyId`/`secretAccessKey`/`region` are all present.
```json theme={null}
{
"catalog.iceberg-rest.auth.rest.authType": "aws-sigv4",
"catalog.iceberg-rest.auth.rest.accessKeyId": "",
"catalog.iceberg-rest.auth.rest.secretAccessKey": "",
"catalog.iceberg-rest.auth.rest.region": "",
"catalog.iceberg-rest.auth.rest.service": "glue"
}
```
#### Assumed IAM role
SigV4 signing with role-vended credentials — no static secrets on the catalog surface. `auth.rest.region` is **required** when a catalog-side `roleArn` is set (falls back to `auth.storage.region` if that is set instead); static `auth.rest` keys are ignored once a role is configured; and the `AssumeRole` call authenticates with the cluster's ambient AWS identity (node role / IRSA), which the role's trust policy must trust.
```json theme={null}
{
"catalog.iceberg-rest.auth.rest.roleArn": "arn:aws:iam::123456789012:role/pinot-glue-catalog-reader",
"catalog.iceberg-rest.auth.rest.externalId": "",
"catalog.iceberg-rest.auth.rest.region": "",
"catalog.iceberg-rest.auth.rest.service": "glue"
}
```
If you set a catalog-side `roleArn` but **no** storage-side `roleArn`, data-file reads through the catalog's default S3 client also assume the catalog role — set `auth.storage.roleArn` explicitly if storage should use a different role.
Always set `catalog.iceberg-rest.auth.rest.service` to `glue` explicitly. If left unset it defaults to `s3tables`, and Glue rejects the resulting SigV4 signature.
### Storage authentication: choose one
#### Assumed IAM role (recommended for production)
No static secrets in the table config — every S3 read assumes `roleArn` via STS. Three things to know: `auth.storage.region` is **required** whenever a `roleArn` is set (the sync fails fast without it); any static `auth.storage` access keys are **ignored** once a `roleArn` is present; and the `AssumeRole` call itself authenticates with the cluster's ambient AWS identity (node role / IRSA), so the target role's trust policy must trust that identity (plus the `externalId`, if you set one).
```json theme={null}
{
"catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader",
"catalog.iceberg-rest.auth.storage.externalId": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
```
#### Cluster node role
Omit every `auth.storage.*` credential key — only `region` is required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have S3 access.
```json theme={null}
{
"catalog.iceberg-rest.auth.storage.region": ""
}
```
#### Static access keys
Quick tests, or when role-based access isn't available. Use the canonical `accessKeyId` / `secretAccessKey` spellings (an optional `sessionToken` is also accepted). Avoid the shorter `accessKey`/`secretKey` aliases: only part of the pipeline honors them (segment generation does, Iceberg manifest and data-file reads do not), so a short-alias config falls back to the cluster's ambient credentials for catalog file reads and can fail in confusing ways.
```json theme={null}
{
"catalog.iceberg-rest.auth.storage.accessKeyId": "",
"catalog.iceberg-rest.auth.storage.secretAccessKey": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
```
***
## Step 1: Validate and browse the connection
**`POST /connections/browse`**
There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your credentials and connectivity. Bad credentials, an unreachable host, or an unknown catalog type return an error.
* Set `path` to `""` to browse the **root** — this both validates the connection and lists Glue databases (namespaces).
* Set `path` to a database name to list the **tables** inside it.
### Request
| Field | Type | Required | Description |
| - | - | - | - |
| `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. |
| `connection.params` | object | Yes | Catalog-specific connection settings (see example below). |
| `path` | string | No | Where to browse. `""` or omitted = root. A namespace name = that namespace's tables. |
```bash theme={null}
curl -X POST "$BASE_URL/connections/browse" \
-H "Content-Type: application/json" \
-d '{
"connection": {
"type": "CATALOG",
"params": {
"catalogType": "iceberg-rest",
"catalog.iceberg-rest.restUri": "https://glue..amazonaws.com",
"catalog.iceberg-rest.serviceType": "glue",
"catalog.iceberg-rest.warehouse": "123456789012",
"catalog.iceberg-rest.auth.rest.authType": "aws-sigv4",
"catalog.iceberg-rest.auth.rest.accessKeyId": "",
"catalog.iceberg-rest.auth.rest.secretAccessKey": "",
"catalog.iceberg-rest.auth.rest.region": "",
"catalog.iceberg-rest.auth.rest.service": "glue",
"catalog.iceberg-rest.auth.storage.authType": "aws-sigv4",
"catalog.iceberg-rest.auth.storage.accessKeyId": "",
"catalog.iceberg-rest.auth.storage.secretAccessKey": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
},
"path": ""
}'
```
| Key | Value | Description |
| - | - | - |
| `catalog.iceberg-rest.restUri` | `https://glue..amazonaws.com` | The Glue endpoint for your region. |
| `catalog.iceberg-rest.serviceType` | `glue` | Selects the built-in Glue adapter. |
| `catalog.iceberg-rest.warehouse` | AWS account ID | The Glue warehouse identifier. |
| `catalog.iceberg-rest.auth.rest.service` | `glue` | The SigV4 signing service name. |
### Response
```json theme={null}
{
"items": [
{ "name": "transportation", "type": "NAMESPACE" },
{ "name": "nyc_taxi_trips", "type": "TABLE" }
]
}
```
| `items[].type` | Meaning |
| - | - |
| `NAMESPACE` | A Glue database you can drill into — pass its `name` as `path`. |
| `TABLE` | A selectable table. |
***
## Step 2: Preview the schema
**`POST /tables/preview`**
Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4.
The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`.
### Request
**Top-level fields:**
| Field | Type | Required | Description |
| - | - | - | - |
| `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. |
| `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. |
| `schema` | object | No | An explicit schema to use instead of inferring one. |
**`config.inference` — how the schema is derived:**
| Field | Type | Default | Description |
| - | - | - | - |
| `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Iceberg metadata. **Set this to `true` for External Tables.** |
| `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. |
Resolution order: explicit `schema` → embedded schema → inferred schema.
**`config.sampling` — how rows are sampled:**
| Field | Type | Default | Description |
| - | - | - | - |
| `min` | integer | `1` | Minimum records to sample. |
| `max` | integer | `100` | Maximum records to sample. |
To list source files without sampling any data, set `config.previewFiles.previewFilesOnly = true`. The response returns matching file URIs in `sourceFiles` and skips schema inference. (OFFLINE only.)
```bash theme={null}
curl -X POST "$BASE_URL/tables/preview" \
-H "Content-Type: application/json" \
-d '{
"tableConfig": {
"tableName": "nyc_taxi_trips_OFFLINE",
"tableType": "OFFLINE",
"task": {
"taskTypeConfigsMap": {
"ExternalTableSyncTask": {
"catalogType": "iceberg-rest",
"executor": "controller",
"inputFormat": "parquet",
"catalog.iceberg-rest.restUri": "https://glue..amazonaws.com",
"catalog.iceberg-rest.serviceType": "glue",
"catalog.iceberg-rest.warehouse": "123456789012",
"catalog.iceberg-rest.table.namespace": "transportation",
"catalog.iceberg-rest.table.tableName": "nyc_taxi_trips",
"catalog.iceberg-rest.auth.rest.authType": "aws-sigv4",
"catalog.iceberg-rest.auth.rest.accessKeyId": "",
"catalog.iceberg-rest.auth.rest.secretAccessKey": "",
"catalog.iceberg-rest.auth.rest.region": "",
"catalog.iceberg-rest.auth.rest.service": "glue",
"catalog.iceberg-rest.auth.storage.authType": "aws-sigv4",
"catalog.iceberg-rest.auth.storage.accessKeyId": "",
"catalog.iceberg-rest.auth.storage.secretAccessKey": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
}
}
},
"config": { "inference": { "embeddedSchemaEnabled": true } }
}'
```
**Setting the namespace and table.** Take them from the [browse](#step-1-validate-and-browse-the-connection) response — set `catalog.iceberg-rest.table.namespace` / `.tableName` to the `NAMESPACE` (Glue database) and `TABLE` you selected.
**Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4.
### Response
| Field | Type | Description |
| - | - | - |
| `schema` | object | The resolved Pinot schema. **Use this in step 3.** |
| `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. |
| `rows` | array | Sample records after schema + ingestion transforms. |
| `sourceRows` | array | Raw records before transformation. |
| `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). |
Adjust the schema (time column, column names, null handling) before moving on.
***
## Step 3: Create the schema
**`POST /schemas`**
Send the `schema` object from the preview response. Rename its `schemaName` to match your table.
```bash theme={null}
curl -X POST "$BASE_URL/schemas" \
-H "Content-Type: application/json" \
-d @schema.json
```
```json theme={null}
{ "status": "nyc_taxi_trips successfully added" }
```
***
## Step 4: Create the table
**`POST /tables`**
Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external.
```bash theme={null}
curl -X POST "$BASE_URL/tables" \
-H "Content-Type: application/json" \
-d @tableConfig.json
```
```json theme={null}
{ "status": "Table nyc_taxi_trips_OFFLINE successfully added" }
```
The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call.
The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run:
```bash theme={null}
curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE"
```
***
## Quickstart: onboard a table end-to-end
The whole flow as one script. Fill in the variables at the top, run the script, and the first sync starts automatically.
```bash theme={null}
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="https://dp..cp..startree.cloud/api/pinot"
AUTH="Authorization: Bearer "
GLUE_URI="https://glue..amazonaws.com"
WAREHOUSE="123456789012" # AWS account ID
NAMESPACE="transportation" # Glue database
TABLE="nyc_taxi_trips"
REGION=""
ACCESS_KEY=""
SECRET_KEY=""
# 1. Validate & browse root — lists Glue namespaces.
curl -sf -X POST "$BASE_URL/connections/browse" \
-H "$AUTH" -H 'Content-Type: application/json' \
--data-raw "{
\"connection\": {
\"type\": \"CATALOG\",
\"params\": {
\"catalogType\": \"iceberg-rest\",
\"catalog.iceberg-rest.restUri\": \"$GLUE_URI\",
\"catalog.iceberg-rest.serviceType\": \"glue\",
\"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\",
\"catalog.iceberg-rest.auth.rest.authType\": \"aws-sigv4\",
\"catalog.iceberg-rest.auth.rest.accessKeyId\": \"$ACCESS_KEY\",
\"catalog.iceberg-rest.auth.rest.secretAccessKey\": \"$SECRET_KEY\",
\"catalog.iceberg-rest.auth.rest.region\": \"$REGION\",
\"catalog.iceberg-rest.auth.rest.service\": \"glue\",
\"catalog.iceberg-rest.auth.storage.authType\": \"aws-sigv4\",
\"catalog.iceberg-rest.auth.storage.accessKeyId\": \"$ACCESS_KEY\",
\"catalog.iceberg-rest.auth.storage.secretAccessKey\": \"$SECRET_KEY\",
\"catalog.iceberg-rest.auth.storage.region\": \"$REGION\"
}
},
\"path\": \"\"
}" | jq .
# 2. Preview — infer schema + enriched table config.
curl -sf -X POST "$BASE_URL/tables/preview" \
-H "$AUTH" -H 'Content-Type: application/json' \
--data-raw "{
\"tableConfig\": {
\"tableName\": \"${TABLE}_OFFLINE\",
\"tableType\": \"OFFLINE\",
\"task\": {
\"taskTypeConfigsMap\": {
\"ExternalTableSyncTask\": {
\"catalogType\": \"iceberg-rest\",
\"executor\": \"controller\",
\"inputFormat\": \"parquet\",
\"catalog.iceberg-rest.restUri\": \"$GLUE_URI\",
\"catalog.iceberg-rest.serviceType\": \"glue\",
\"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\",
\"catalog.iceberg-rest.table.namespace\": \"$NAMESPACE\",
\"catalog.iceberg-rest.table.tableName\": \"$TABLE\",
\"catalog.iceberg-rest.auth.rest.authType\": \"aws-sigv4\",
\"catalog.iceberg-rest.auth.rest.accessKeyId\": \"$ACCESS_KEY\",
\"catalog.iceberg-rest.auth.rest.secretAccessKey\": \"$SECRET_KEY\",
\"catalog.iceberg-rest.auth.rest.region\": \"$REGION\",
\"catalog.iceberg-rest.auth.rest.service\": \"glue\",
\"catalog.iceberg-rest.auth.storage.authType\": \"aws-sigv4\",
\"catalog.iceberg-rest.auth.storage.accessKeyId\": \"$ACCESS_KEY\",
\"catalog.iceberg-rest.auth.storage.secretAccessKey\": \"$SECRET_KEY\",
\"catalog.iceberg-rest.auth.storage.region\": \"$REGION\"
}
}
}
},
\"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } }
}" > preview.json
# 3. Extract schema and table config from preview response.
jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json
jq '.tableConfigs.offline' preview.json > tableConfig.json
# 4. Create the schema.
curl -sf -X POST "$BASE_URL/schemas" \
-H "$AUTH" -H 'Content-Type: application/json' -d @schema.json
# 5. Create the table — watcher starts the first sync automatically.
curl -sf -X POST "$BASE_URL/tables" \
-H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json
# 6. (optional) Poll until sync completes.
curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \
| jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered'
```
Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable).
## Monitor onboarding
Three read-only endpoints report ingestion progress — run status, ingestion checkpoint, and source file count — and require `executor=controller` (set automatically). See [Observability](../observability) for full request and response details.
## Verify it's queryable
When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live:
```sql theme={null}
SELECT count(*) FROM nyc_taxi_trips;
```
The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting).
***
## What's next
Now that the table is created and data is loading, these are the highest-impact follow-up steps:
1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes)
2. **Enable caching and preload.** Set `enable.prefetch.page.cache=true` and `preload.enable=true` on the S3 tier so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching)
3. **Protect large-scan queries from OOM.** For tables that receive heavy aggregations or wide scans, enable the query OOM killer so a runaway query is killed instead of crashing the server. → [Best Practices & Configs — Query OOM protection](../best-practices-and-configs#query-oom-protection-large-scans)
4. **Monitor ongoing syncs.** Use the observability endpoints to check run status, ingestion checkpoint, and source file count after each scheduled sync. → [Observability](../observability)
***
For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting).
# AWS Glue: Onboarding via Data Portal
Source: https://docs.startree.ai/corecapabilities/external-table/glue/onboarding-data-portal
Connect an AWS Glue Iceberg catalog to StarTree as an External Table with the point-and-click Data Portal wizard — no API calls required.
This feature is available starting in **StarTree release 0.15.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment.
This guide walks through connecting an **AWS Glue** Iceberg catalog to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface.
Glue tables are accessed over the **Iceberg REST protocol** with AWS SigV4 authentication.
> Looking for the API-based approach? See [AWS Glue: Onboarding via API](./onboarding-api).
>
> Using **Amazon S3 Tables** instead? See [Amazon S3 Tables: Onboarding via Data Portal](../s3tables/onboarding-data-portal).
***
## Prerequisites
Before starting, ensure you have:
* **StarTree 0.15.0 or later** with the external table Beta feature enabled for your environment.
* AWS credentials (access key + secret key) with read permissions on both the Glue catalog service and the underlying S3 data.
* Your **AWS account ID** (used as the Glue warehouse identifier).
* The target **Glue database** name.
**Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly.
If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details.
***
## Step 1: Open the External Tables
1. Log in to **Data Portal**.
2. In the left navigation, go to **Tables**.
3. Click **+ Connect External Table**.
The wizard opens with a connection configuration screen.
***
## Step 2: Configure the Glue Connection
Choose **AWS Glue** as the catalog type.
Fill in the credentials and connection details:
**Details**
| Field | Description |
| - | - |
| **Connection Name** | A unique name to identify this connection in Data Portal. |
**Metastore** — how StarTree authenticates with the AWS Glue catalog API.
| Field | Description |
| - | - |
| **AWS Account ID** | The AWS account that owns the Glue Data Catalog — used as the warehouse identifier. |
| **Authentication Type** | **Basic Authentication** (access key + secret key), **IAM Node Role** (the cluster node's own IAM identity — the form displays its ARN under **IAM Role Name** so you can grant it access in your AWS account), or **IAM Role** (a cross-account role StarTree assumes). |
| **Access Key** / **Secret Key** | With **Basic Authentication**: static AWS credentials with Glue read access. |
| **IAM Role ARN** | With **IAM Role**: the ARN of the role to assume. The form shows **Steps to create IAM Policy** — a ready-made trust policy including the StarTree account and External ID to configure on the role. |
| **Region** | AWS region where your Glue catalog is located, e.g. `us-east-1`. |
**Storage** — credentials for reading the underlying Parquet data files from S3.
| Field | Description |
| - | - |
| **Reuse metastore credentials** | **Yes** reuses the Metastore authentication for S3 data reads (the same three auth types apply). **No** opens a separate set of storage fields. |
| **Region** | With **No**: AWS region where the S3 data files are stored. |
| **Authentication Type** | With **No**: **Basic Authentication** (Access Key + Secret Key), **IAM Node Role**, or **IAM Role** (IAM Role ARN + trust-policy steps) — same choices as the Metastore section, configured independently. |
Click **Validate Connection**. Data Portal calls the catalog's validate endpoint and confirms credentials and connectivity before proceeding.
***
## Step 3: Browse and Select a Table
Once the connection is validated:
1. Data Portal lists the available **Glue databases** (namespaces).
2. Select a database to expand its tables.
3. Click the **table** you want to onboard.
Data Portal reads the Iceberg schema and derives a Pinot schema automatically.
***
## Step 4: Review the Schema
The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity.
Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct.
***
## Step 5: Configure the Table
Set the final table options:
| Setting | Default | Notes |
| - | - | - |
| **Table Name** | — | Name of the Pinot table to create. |
| **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. |
| **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. |
The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config.
Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required.
***
## Step 6: Monitor Onboarding
Once the table is created, onboarding starts automatically. The table detail view shows the status in real time:
* **Running** — the task is actively reading Iceberg snapshots and building Pinot segments.
* **Completed** — onboarding finished successfully. The last ingested snapshot ID is shown.
* **Failed** — onboarding encountered an error. The error message and the number of files discovered vs. segments generated are surfaced to help diagnose the issue.
For deeper observability — watcher status, checkpoint values, and per-snapshot file counts — see the [Observability](../observability) page.
***
## Pausing Onboarding
To pause scheduled onboarding from Data Portal:
1. Open the table in the **Tables** view.
2. Click **Pause Sync**. This removes the `schedule` (cron trigger) from the table's `ExternalTableSyncTask` config and stashes its value under `schedule.paused`, so the watcher stops firing new runs. Any run currently in progress completes normally, and existing segments and the last checkpoint are untouched. Click **Resume Sync** to restore the schedule from `schedule.paused` — onboarding resumes from where it left off; it does not restart from scratch.
This is a separate mechanism from the `enabled` key documented in [Best Practices & Configs](../best-practices-and-configs#table-config--externaltablesynctask) — that key can also be set to `false` directly via the table config or API to stop new runs, but the Data Portal's **Pause Sync** button acts on `schedule`/`schedule.paused`, not `enabled`.
***
## Frequently Asked Questions
**The Validate step fails — what should I check?**
* Confirm the access key has `glue:GetCatalog`, `glue:GetDatabase(s)`, `glue:GetTable(s)`, `glue:GetPartition(s)`, `glue:BatchGetPartition`, and `s3:GetObject` permissions (plus `lakeformation:GetDataAccess` if the database is registered with Lake Formation). The **Steps to create IAM Policy** panel in the connection form generates the exact policy for you.
* Verify the region matches where your Glue database lives.
* Ensure the warehouse value is your numeric AWS account ID, not an account alias.
***
**Can I onboard multiple tables from the same catalog?**
Yes. After creating the first table, start the wizard again and reuse the same connection credentials. Each table is registered as an independent Pinot table with its own onboarding schedule.
***
**The table was created but onboarding hasn't started — what should I check?**
Data Portal triggers the first onboarding run automatically after table creation. If onboarding hasn't started, check the table's detail page for an error status and review the error message. You can also trigger a run manually via the [trigger API](../observability#trigger-a-sync-run-optional).
# Snowflake Horizon Catalog: Onboarding via API
Source: https://docs.startree.ai/corecapabilities/external-table/horizon/onboarding-api
Query Snowflake-managed Iceberg tables in StarTree — onboard a Snowflake Horizon Catalog table as an External Table over the Iceberg REST API, authenticating with a Snowflake programmatic access token (PAT).
This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it.
This page shows how to onboard a **Snowflake Horizon Catalog** External Table with the StarTree controller REST APIs.
Prefer a point-and-click setup? The Data Portal wizard has a dedicated **Snowflake Horizon Catalog** tile — see [Onboarding via Data Portal](./onboarding-data-portal).
Snowflake Horizon exposes Snowflake-managed Iceberg tables over the same **Iceberg REST protocol** as Apache Polaris, served on the Snowflake account host under `/polaris/api/catalog`. StarTree reaches it through the generic spec-compliant adapter — `catalogType=iceberg-rest`, `serviceType=rest`. Data files must be **Parquet**.
> Running **self-managed Apache Polaris** or **Snowflake Open Catalog** instead? Those use principal `client-id`/`client-secret` credentials — see [Apache Polaris & Snowflake Open Catalog: Onboarding via API](../polaris/onboarding-api).
## How Horizon differs from Apache Polaris
Horizon speaks the Polaris REST protocol, but authentication and identifier mapping follow Snowflake's model, not Polaris's:
| | Snowflake Horizon Catalog |
| - | - |
| REST base URI | `https://-.snowflakecomputing.com/polaris/api/catalog` |
| Identity | Snowflake **service user** + a **programmatic access token (PAT)** (or key-pair JWT) |
| Catalog credential | the **PAT**, passed via `catalog.iceberg-rest.credential` (see [Authentication](#authentication)) |
| `oauthScopes` | `session:role:` — Polaris-style `PRINCIPAL_ROLE:ALL` returns `401` |
| `warehouse` | the Snowflake **database** name |
| Namespace | the Snowflake **schema** (single level; the database is the prefix) |
| `prefix` | **required** — set it to the database name (same value as `warehouse`) |
| Access control | native Snowflake RBAC (`GRANT SELECT` / `USAGE`); the PAT also needs a **network policy** |
**Why the PAT goes in `catalog.iceberg-rest.credential`.** Horizon's token endpoint expects a `client_credentials` exchange that sends the PAT as `client_secret` with **no** `client_id`. Passing the PAT through `catalog.iceberg-rest.credential` produces exactly that request. (Setting `oauthClientSecret` with an empty `oauthClientId` does **not** currently work — the empty client id causes the credential to be dropped and the catalog call goes out unauthenticated. Use `catalog.iceberg-rest.credential` until first-class secret-only support lands.)
## How it works
Onboarding is four calls. Each one feeds the next:
```text theme={null}
1. Validate & browse POST /connections/browse → validate; list namespaces/tables
2. Preview POST /tables/preview → infer schema + enriched table config
3. Create schema POST /schemas → register the schema
4. Create table POST /tables → register the External Table
```
The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output.
Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding).
All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` (the StarTree Cloud data-plane proxy). A browse call then posts to `$BASE_URL/connections/browse`.
If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity.
## Prerequisites
* StarTree 0.16.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure.
* Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth.
* A Snowflake account with **Horizon Catalog** and the target Iceberg table backed by an **external volume** (S3).
* A Snowflake **service user** with a **programmatic access token (PAT)**, a role that can read the table, and a **network policy** governing the user (see [Snowflake setup](#snowflake-setup) below).
* The Snowflake **database** (used as both `warehouse` and `prefix`), the **schema** (the namespace), and the **table** to onboard.
* Read access to the underlying object storage. This connector supports **S3-compatible storage** for the data files.
## Snowflake setup
Run these once in Snowflake (or have an ACCOUNTADMIN do so). Replace the placeholders with your own names.
```sql theme={null}
-- 1. A role that can read the Iceberg table's database, schema, and external volume.
GRANT USAGE ON DATABASE TO ROLE ;
GRANT USAGE ON SCHEMA . TO ROLE ;
GRANT SELECT ON ALL TABLES IN SCHEMA . TO ROLE ;
GRANT USAGE ON EXTERNAL VOLUME TO ROLE ;
-- 2. A dedicated service user, and the role granted to it.
CREATE USER IF NOT EXISTS DEFAULT_ROLE = ;
GRANT ROLE TO USER ;
-- 3. A network policy is REQUIRED before a PAT can be created or used.
ALTER USER SET NETWORK_POLICY = ;
-- 4. The programmatic access token. Copy the secret it prints — it is shown only once.
ALTER USER IF EXISTS
ADD PAT startree_external_table
DAYS_TO_EXPIRY = 90
ROLE_RESTRICTION = ''
COMMENT = 'StarTree External Table catalog access';
```
The `ROLE_RESTRICTION` role must match the role named in the `session:role:` scope you configure below, and it must hold the grants from step 1. Without a **network policy** in effect, `ADD PAT` fails and the token endpoint later returns `401 unauthorized_client`.
## Authentication
An External Table authenticates in two places, configured independently:
* **Catalog (REST)** — the Snowflake PAT, supplied via `catalog.iceberg-rest.credential` plus the token URI and scope.
* **Storage (S3 data files)** — keys under `catalog.iceberg-rest.auth.storage.*`. Choose one of the methods below.
### Catalog (REST) authentication
Pass the **PAT** as the catalog credential. StarTree exchanges it at the Snowflake token endpoint for a short-lived token, which it caches and refreshes automatically.
```json theme={null}
{
"catalog.iceberg-rest.auth.rest.authType": "oauth2",
"catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens",
"catalog.iceberg-rest.auth.rest.oauthScopes": "session:role:",
"catalog.iceberg-rest.credential": ""
}
```
| Key | Value | Description |
| - | - | - |
| `catalog.iceberg-rest.credential` | `` | The Snowflake **programmatic access token**. Sent to the token endpoint as `client_secret` with no client id. |
| `catalog.iceberg-rest.auth.rest.oauthTokenUri` | `https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens` | Snowflake's OAuth token endpoint (the account host, `/polaris/api/catalog/v1/oauth/tokens`). |
| `catalog.iceberg-rest.auth.rest.oauthScopes` | `session:role:` | Activates the Snowflake role for the session. Must match the PAT's `ROLE_RESTRICTION`. |
| `catalog.iceberg-rest.auth.rest.authType` | `oauth2` | Optional; auto-detected as `oauth2` when a credential is present. |
The PAT is a **secret** — treat the table config as sensitive and rotate the token on the `DAYS_TO_EXPIRY` cadence. A static PAT is not auto-rotated; when it expires, syncs fail until you update the table config with a new one. Key-pair JWT auth (a signed JWT in place of the PAT) works the same way — put the JWT in `catalog.iceberg-rest.credential`.
### Storage authentication: choose one
#### Static access keys
What most Snowflake external-volume setups use. Use the canonical `accessKeyId` / `secretAccessKey` spellings (an optional `sessionToken` is also accepted). Avoid the shorter `accessKey`/`secretKey` aliases: only part of the pipeline honors them (segment generation does, Iceberg manifest and data-file reads do not), so a short-alias config falls back to the cluster's ambient credentials for catalog file reads and can fail in confusing ways.
```json theme={null}
{
"catalog.iceberg-rest.auth.storage.accessKeyId": "",
"catalog.iceberg-rest.auth.storage.secretAccessKey": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
```
#### Assumed IAM role (recommended for production)
No static secrets in the table config — every S3 read assumes `roleArn` via STS. Three things to know: `auth.storage.region` is **required** whenever a `roleArn` is set (the sync fails fast without it); any static `auth.storage` access keys are **ignored** once a `roleArn` is present; and the `AssumeRole` call itself authenticates with the cluster's ambient AWS identity (node role / IRSA), so the target role's trust policy must trust that identity (plus the `externalId`, if you set one).
```json theme={null}
{
"catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader",
"catalog.iceberg-rest.auth.storage.externalId": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
```
#### Cluster node role
Omit every `auth.storage.*` credential key — only `region` is required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have S3 access.
```json theme={null}
{
"catalog.iceberg-rest.auth.storage.region": ""
}
```
Horizon can vend short-lived storage credentials, but the generic `rest` adapter does not consume vended credentials — configure static keys, an assumed role, or the cluster's node role for storage explicitly.
### Selecting the database
Set both `catalog.iceberg-rest.warehouse` and `catalog.iceberg-rest.prefix` to the Snowflake **database** name. `prefix` is **required** for Horizon: the generic adapter doesn't send the `warehouse` query parameter that Snowflake's `/v1/config` gates on, so without an explicit prefix, browse falls back to the un-prefixed `/v1/namespaces` path and fails with `404 Unable to find matching target resource method`.
```json theme={null}
{
"catalog.iceberg-rest.warehouse": "",
"catalog.iceberg-rest.prefix": ""
}
```
The Snowflake **schema** is the Iceberg namespace — set `catalog.iceberg-rest.table.namespace` to the schema name and `catalog.iceberg-rest.table.tableName` to the table.
***
## Step 1: Validate and browse the connection
**`POST /connections/browse`**
There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your PAT, scope, and connectivity.
* Set `path` to `""` to browse the **root** — this both validates the connection and lists schemas (namespaces).
* Set `path` to a schema name to list the **tables** inside it.
### Request
| Field | Type | Required | Description |
| - | - | - | - |
| `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. |
| `connection.params` | object | Yes | Catalog-specific connection settings (see example below). |
| `path` | string | No | Where to browse. `""` or omitted = root. A schema name = that schema's tables. |
```bash theme={null}
curl -X POST "$BASE_URL/connections/browse" \
-H "Content-Type: application/json" \
-d '{
"connection": {
"type": "CATALOG",
"params": {
"catalogType": "iceberg-rest",
"catalog.iceberg-rest.serviceType": "rest",
"catalog.iceberg-rest.restUri": "https://-.snowflakecomputing.com/polaris/api/catalog",
"catalog.iceberg-rest.warehouse": "",
"catalog.iceberg-rest.prefix": "",
"catalog.iceberg-rest.auth.rest.authType": "oauth2",
"catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens",
"catalog.iceberg-rest.auth.rest.oauthScopes": "session:role:",
"catalog.iceberg-rest.credential": "",
"catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader",
"catalog.iceberg-rest.auth.storage.externalId": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
},
"path": ""
}'
```
| Key | Value | Description |
| - | - | - |
| `catalog.iceberg-rest.restUri` | `https://-.snowflakecomputing.com/polaris/api/catalog` | The Snowflake Iceberg REST base URI. |
| `catalog.iceberg-rest.serviceType` | `rest` | Selects the generic spec-compliant adapter — there's no Snowflake-specific `serviceType`. |
| `catalog.iceberg-rest.warehouse` | `` | The Snowflake database to browse. |
| `catalog.iceberg-rest.prefix` | `` | Pins the catalog prefix (the database). Required for Horizon. |
### Response
```json theme={null}
{
"items": [
{ "name": "SCHEMA_ONE", "type": "NAMESPACE" },
{ "name": "nyc_taxi_trips", "type": "TABLE" }
]
}
```
| `items[].type` | Meaning |
| - | - |
| `NAMESPACE` | A schema you can drill into — pass its `name` as `path`. |
| `TABLE` | A selectable table. |
***
## Step 2: Preview the schema
**`POST /tables/preview`**
Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4.
The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`.
### Request
**Top-level fields:**
| Field | Type | Required | Description |
| - | - | - | - |
| `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. |
| `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. |
| `schema` | object | No | An explicit schema to use instead of inferring one. |
**`config.inference` — how the schema is derived:**
| Field | Type | Default | Description |
| - | - | - | - |
| `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Iceberg metadata. **Set this to `true` for External Tables.** |
| `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. |
Resolution order: explicit `schema` → embedded schema → inferred schema.
**`config.sampling` — how rows are sampled:**
| Field | Type | Default | Description |
| - | - | - | - |
| `min` | integer | `1` | Minimum records to sample. |
| `max` | integer | `100` | Maximum records to sample. |
```bash theme={null}
curl -X POST "$BASE_URL/tables/preview" \
-H "Content-Type: application/json" \
-d '{
"tableConfig": {
"tableName": "nyc_taxi_trips_OFFLINE",
"tableType": "OFFLINE",
"task": {
"taskTypeConfigsMap": {
"ExternalTableSyncTask": {
"catalogType": "iceberg-rest",
"executor": "controller",
"inputFormat": "parquet",
"catalog.iceberg-rest.serviceType": "rest",
"catalog.iceberg-rest.restUri": "https://-.snowflakecomputing.com/polaris/api/catalog",
"catalog.iceberg-rest.warehouse": "",
"catalog.iceberg-rest.prefix": "",
"catalog.iceberg-rest.table.namespace": "",
"catalog.iceberg-rest.table.tableName": "nyc_taxi_trips",
"catalog.iceberg-rest.auth.rest.authType": "oauth2",
"catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens",
"catalog.iceberg-rest.auth.rest.oauthScopes": "session:role:",
"catalog.iceberg-rest.credential": "",
"catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader",
"catalog.iceberg-rest.auth.storage.externalId": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
}
}
},
"config": { "inference": { "embeddedSchemaEnabled": true } }
}'
```
**Setting the namespace and table.** Take them from the [browse](#step-1-validate-and-browse-the-connection) response — set `catalog.iceberg-rest.table.namespace` (the Snowflake schema) and `.tableName` to the `NAMESPACE` and `TABLE` you selected.
**Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4.
### Response
| Field | Type | Description |
| - | - | - |
| `schema` | object | The resolved Pinot schema. **Use this in step 3.** |
| `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. |
| `rows` | array | Sample records after schema + ingestion transforms. |
| `sourceRows` | array | Raw records before transformation. |
| `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). |
Adjust the schema (time column, column names, null handling) before moving on.
***
## Step 3: Create the schema
**`POST /schemas`**
Send the `schema` object from the preview response. Rename its `schemaName` to match your table.
```bash theme={null}
curl -X POST "$BASE_URL/schemas" \
-H "Content-Type: application/json" \
-d @schema.json
```
```json theme={null}
{ "status": "nyc_taxi_trips successfully added" }
```
***
## Step 4: Create the table
**`POST /tables`**
Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external.
```bash theme={null}
curl -X POST "$BASE_URL/tables" \
-H "Content-Type: application/json" \
-d @tableConfig.json
```
```json theme={null}
{ "status": "Table nyc_taxi_trips_OFFLINE successfully added" }
```
The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call.
The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run:
```bash theme={null}
curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE"
```
***
## Quickstart: onboard a table end-to-end
The whole flow as one script. Fill in the variables at the top, run the script, and the first sync starts automatically.
```bash theme={null}
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="https://dp..cp..startree.cloud/api/pinot"
AUTH="Authorization: Bearer "
ACCOUNT_HOST="-.snowflakecomputing.com"
REST_URI="https://$ACCOUNT_HOST/polaris/api/catalog"
TOKEN_URI="https://$ACCOUNT_HOST/polaris/api/catalog/v1/oauth/tokens"
DATABASE="" # used as both warehouse and prefix
SCHEMA="" # the Iceberg namespace
TABLE="nyc_taxi_trips"
ROLE=""
SCOPES="session:role:$ROLE"
PAT=""
STORAGE_REGION=""
STORAGE_ROLE_ARN="arn:aws:iam::123456789012:role/pinot-external-table-reader"
STORAGE_EXTERNAL_ID=""
# 1. Validate & browse root — lists schemas in the Snowflake database.
curl -sf -X POST "$BASE_URL/connections/browse" \
-H "$AUTH" -H 'Content-Type: application/json' \
--data-raw "{
\"connection\": {
\"type\": \"CATALOG\",
\"params\": {
\"catalogType\": \"iceberg-rest\",
\"catalog.iceberg-rest.serviceType\": \"rest\",
\"catalog.iceberg-rest.restUri\": \"$REST_URI\",
\"catalog.iceberg-rest.warehouse\": \"$DATABASE\",
\"catalog.iceberg-rest.prefix\": \"$DATABASE\",
\"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\",
\"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\",
\"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\",
\"catalog.iceberg-rest.credential\": \"$PAT\"
}
},
\"path\": \"\"
}" | jq .
# 2. Preview — infer schema + enriched table config.
curl -sf -X POST "$BASE_URL/tables/preview" \
-H "$AUTH" -H 'Content-Type: application/json' \
--data-raw "{
\"tableConfig\": {
\"tableName\": \"${TABLE}_OFFLINE\",
\"tableType\": \"OFFLINE\",
\"task\": {
\"taskTypeConfigsMap\": {
\"ExternalTableSyncTask\": {
\"catalogType\": \"iceberg-rest\",
\"executor\": \"controller\",
\"inputFormat\": \"parquet\",
\"catalog.iceberg-rest.serviceType\": \"rest\",
\"catalog.iceberg-rest.restUri\": \"$REST_URI\",
\"catalog.iceberg-rest.warehouse\": \"$DATABASE\",
\"catalog.iceberg-rest.prefix\": \"$DATABASE\",
\"catalog.iceberg-rest.table.namespace\": \"$SCHEMA\",
\"catalog.iceberg-rest.table.tableName\": \"$TABLE\",
\"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\",
\"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\",
\"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\",
\"catalog.iceberg-rest.credential\": \"$PAT\",
\"catalog.iceberg-rest.auth.storage.roleArn\": \"$STORAGE_ROLE_ARN\",
\"catalog.iceberg-rest.auth.storage.externalId\": \"$STORAGE_EXTERNAL_ID\",
\"catalog.iceberg-rest.auth.storage.region\": \"$STORAGE_REGION\"
}
}
}
},
\"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } }
}" > preview.json
# 3. Extract schema and table config from preview response.
jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json
jq '.tableConfigs.offline' preview.json > tableConfig.json
# 4. Create the schema.
curl -sf -X POST "$BASE_URL/schemas" \
-H "$AUTH" -H 'Content-Type: application/json' -d @schema.json
# 5. Create the table — watcher starts the first sync automatically.
curl -sf -X POST "$BASE_URL/tables" \
-H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json
# 6. (optional) Poll until sync completes.
curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \
| jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered'
```
Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable).
## Monitor onboarding
The [status endpoint](../observability#run-status) reports each sync run's progress, and `?includeLag=true` adds how far the table trails the upstream catalog. Both require `executor=controller` (set automatically by this flow). See [Observability](../observability) for full request and response details.
## Verify it's queryable
When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live:
```sql theme={null}
SELECT count(*) FROM nyc_taxi_trips;
```
The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting).
***
## What's next
Now that the table is created and data is loading, these are the highest-impact follow-up steps:
1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes)
2. **Enable caching and preload.** Set `enable.prefetch.page.cache=true` and `preload.enable=true` on the S3 tier so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching)
3. **Protect large-scan queries from OOM.** For tables that receive heavy aggregations or wide scans, enable the query OOM killer so a runaway query is killed instead of crashing the server. → [Best Practices & Configs — Query OOM protection](../best-practices-and-configs#query-oom-protection-large-scans)
4. **Monitor ongoing syncs.** Use the status endpoint (with `includeLag=true` for ingestion lag) after each scheduled sync. → [Observability](../observability)
***
For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting).
# Snowflake Horizon Catalog: Onboarding via Data Portal
Source: https://docs.startree.ai/corecapabilities/external-table/horizon/onboarding-data-portal
Connect Snowflake-managed Iceberg tables (Horizon Catalog) to StarTree as an External Table with the point-and-click Data Portal wizard, authenticating with a Snowflake programmatic access token (PAT).
This feature is available starting in **StarTree release 0.16.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment.
This guide walks through connecting a **Snowflake Horizon Catalog** to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface.
Snowflake Horizon exposes Snowflake-managed Iceberg tables over the standard **Iceberg REST protocol**, served on the Snowflake account host under `/polaris/api/catalog`, authenticated with a **programmatic access token (PAT)**. Data files must be **Parquet**.
> Looking for the API-based approach? See [Snowflake Horizon Catalog: Onboarding via API](./onboarding-api).
>
> Running **self-managed Apache Polaris** or **Snowflake Open Catalog** instead? Those use principal client-id/client-secret credentials — see [Apache Polaris & Snowflake Open Catalog: Onboarding via Data Portal](../polaris/onboarding-data-portal).
***
## Prerequisites
Before starting, ensure you have:
* **StarTree 0.16.0 or later** with the External Table feature enabled and **tiered storage configured** for your environment. Contact StarTree support if unsure.
* A Snowflake account with **Snowflake-managed Iceberg tables** and a **programmatic access token (PAT)** whose role restriction covers a role with read access to them.
* The Snowflake **database** backing the Iceberg tables and the schema/table you want to onboard.
* Read access to the underlying S3 object storage (the external volume's bucket) — static keys or an IAM role StarTree can assume. See [Onboarding via API — Authentication](./onboarding-api#authentication) for the permissions required.
***
## Step 1: Open the External Tables
1. Log in to **Data Portal**.
2. In the left navigation, go to **Tables**.
3. Click **+ Connect External Table**.
The wizard opens with a connection configuration screen.
***
## Step 2: Configure the Horizon Connection
Choose **Snowflake Horizon Catalog** as the catalog type — it appears as a top-level tile alongside AWS Glue, Amazon S3 Tables, and Unity Catalog.
Fill in the connection details:
**Details**
| Field | Description |
| - | - |
| **Connection Name** | A unique name to identify this connection in Data Portal. |
**Metastore** — how StarTree authenticates with Horizon's Iceberg REST API.
| Field | Description |
| - | - |
| **Catalog Endpoint URI** | The Horizon Iceberg REST endpoint: `https://-.snowflakecomputing.com/polaris/api/catalog`. |
| **Database** | The Snowflake database backing the Iceberg tables. |
| **REST Authentication Type** | **OAuth** — the PAT is exchanged through the OAuth2 token endpoint. |
| **OAuth Token URI** | The token endpoint on the account host: `https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens`. |
| **Personal Access Token (PAT)** | The Snowflake programmatic access token — sent as the OAuth2 client secret. |
| **OAuth Scopes** | `session:role:` — must match the PAT's role restriction. |
**Storage** — credentials for reading the underlying Parquet data files from S3.
| Field | Description |
| - | - |
| **Region** | AWS region where the S3 data files are stored. Always required. |
| **Authentication Type** | **Basic Authentication** (Access Key + Secret Key), **IAM Node Role** (the cluster node's own IAM identity — Data Portal shows the resolved role as a read-only **IAM Role Name** field so you can grant it access in your AWS account), or **IAM Role** (a cross-account role StarTree assumes). |
| **Access Key** / **Secret Key** | With **Basic Authentication**: static AWS credentials for S3 data access. |
| **IAM Role ARN** | With **IAM Role**: the ARN of the role to assume. The form shows **Steps to create IAM Policy** — a ready-made trust policy including the StarTree account and External ID to configure on the role. |
Click **Validate Connection**. Data Portal calls the catalog's validate endpoint and confirms credentials and connectivity before proceeding.
***
## Step 3: Browse and Select a Table
Once the connection is validated:
1. Data Portal lists the available **schemas** in the database — the field is labeled **Schema**.
2. Select a schema to expand its tables.
3. Click the **table** you want to onboard.
Data Portal reads the Iceberg schema and derives a Pinot schema automatically.
***
## Step 4: Review the Schema
The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity.
Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct.
***
## Step 5: Configure the Table
Set the final table options:
| Setting | Default | Notes |
| - | - | - |
| **Table Name** | — | Name of the Pinot table to create. |
| **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. |
| **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. |
The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config.
Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required.
***
## Step 6: Monitor Onboarding
The table detail view shows sync status in real time — files discovered, segments generated, last synced snapshot, and ingestion lag. Use **Sync Now** to trigger an immediate sync, or **Pause Sync** / **Resume Sync** from the table menu (while the sync task is paused, the primary button itself reads **Resume Sync**). For API-based monitoring, see [Observability](../observability).
# Supported Indexes
Source: https://docs.startree.ai/corecapabilities/external-table/indexes
Which Pinot indexes External Tables support — inverted, range, JSON, text, bloom, star-tree, and more — with config examples and why columns are RAW.
This feature is available starting in **StarTree release 0.15.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment.
External tables support a broad set of Pinot indexes out of the box. Indexes are key to getting low-latency query performance from external tables. Without them, every query performs a full segment scan across data read from object storage. With the right indexes configured for your query patterns, Pinot can skip irrelevant segments and data blocks entirely, reducing query time from minutes/seconds to seconds/milliseconds on large datasets, while also cutting S3 I/O costs.
The most impactful indexes to configure based on your workload:
* **JSON index** — required for filtering or extracting nested fields from complex columns (`STRUCT`, `LIST`, `MAP`).
* **Text index** — enables full-text search on free-text string columns.
* **Range index** — speeds up range predicates on numeric or timestamp columns (e.g., `WHERE ts > X AND ts < Y`).
* **Inverted index** — efficient equality filtering on low-cardinality columns (e.g., `WHERE status = 'active'`).
***
## Supported
| Index | Status | Notes |
| - | - | - |
| **Forward index (remote)** | ✅ Supported | Reads raw values directly from Parquet on object storage. |
| **Inverted index** | ✅ Supported | Dictionary is retained in the index sidecar; see below. |
| **Range index** | ✅ Supported | |
| **Timestamp index** | ✅ Supported | |
| **JSON index** | ✅ Supported | Required for `JSON_MATCH` and `JSON_EXTRACT_SCALAR` on complex columns (`STRUCT`, `LIST`, `MAP`). |
| **Composite JSON index** | ✅ Supported | |
| **Text index (Lucene)** | ✅ Supported | Enables full-text search on string columns. |
| **Sparse index** | ✅ Supported | |
| **Min/Max index** | ✅ Supported | No dictionary required. |
| **Star-tree index** | ✅ Supported | Pre-aggregates on low-cardinality dimensions. Rejected at table-config validation on a table with Iceberg deletion vectors enabled (`enableDeletionVectors=true`) — deletion vectors filter rows per query, which precomputed aggregates can't honor. See [Deletion Vectors](./deletion-vectors). |
| **Bloom filter** | ✅ Supported | |
| **FST index** | ✅ Supported | |
| **IFST index** | ✅ Supported | |
| **Null value vector** | ✅ Supported | Used internally when `nullHandlingEnabled: true` is set on the table. |
| **Vector index (HNSW)** | ✅ Supported | The index config must set `"storeInSegmentFile": true` — without it the HNSW index is not loaded and vector queries silently fall back to exact scan (`VECTOR_SIMILARITY_EXACT_SCAN`). |
| **Sorted index** | 🚫 Not supported | Requires data to be sorted at ingestion time; not applicable to external tables. |
| **Geospatial / H3 index** | 🚫 Not supported | |
Sorted and H3 are the only index *types* the controller rejects on an external table. Two related validations to be aware of: every non-virtual column must have a `fieldConfig` with `encodingType: RAW` (a column without one would default to dictionary encoding and is rejected), and derived columns / `transformConfigs` are not allowed.
For complex (`STRUCT`/nested-JSON) columns, the `open_struct` settings that choose **which keys get materialized** — `maxDenseKeys`, `denseKeys`, `denseKeyMinFillRate`, `maxNestedKeyDepth` — are rejected at table creation on an external table. Those settings are applied by the dense/sparse key splitter while a segment's column data is written; an external table never writes column data, so the split is whatever the source Parquet file's shredding already chose, and the setting can only be ignored.
Indexes are **not** rejected, in either of the two places an OPEN\_STRUCT (`VARIANT`) column can carry them: per-key `valueFieldConfigs` over a materialized key, and `sparseFieldConfig` (or its older spelling `sparseJsonIndex`) over the shared sparse blob. Both name indexes over columns that already exist rather than a split, and a reload builds them by reading back through the Parquet reader — the same path a per-key index already took.
***
## Why are columns RAW (no dictionary)?
In a standard Pinot table, every column is dictionary-encoded by default. Pinot builds a lookup table that maps each unique value to a compact integer ID and stores those IDs in the forward index. This dictionary is created during ingestion, when data is converted into Pinot's native segment format.
External tables never go through that ingestion step — they read Parquet files directly from object storage at query time. Because the data is never materialized into Pinot segments, there is no opportunity to build a dictionary, so every column is encoded as `RAW` (no dictionary).
You can explicitly see this in the column config generated by the onboarding flow — every column gets RAW encoding and a RAW forward index, and nothing else:
```json theme={null}
{
"name": "payment_type",
"encodingType": "RAW",
"indexes": {
"forward": { "encodingType": "RAW" }
}
}
```
This means the forward index stores the actual values from Parquet, not dictionary IDs.
**Indexes that need a dictionary — inverted, FST, IFST, star-tree** — still work because Pinot builds a per-segment dictionary sidecar when it writes the index files alongside the Parquet data. The main forward index stays RAW; only the index sidecar carries the dictionary mapping. You opt in by adding a `"dictionary": {}` block to the column's `indexes` config (see examples below).
**Sorted index is ruled out entirely** because it requires data to be physically sorted during the ingestion write path. External tables have no write path — they read whatever order the Parquet files are in — so a sorted index cannot be built or maintained.
***
## Adding an index
### Choosing the right index
| Query pattern | Use | Cardinality guidance |
| - | - | - |
| Equality / IN filter | Inverted | Best for low-to-medium cardinality |
| Range filter (`BETWEEN`), including time | Range | Any cardinality; ideal for numeric/timestamp |
| "Does this value exist?" segment pruning | Bloom filter | Best for high-cardinality equality filters |
| Filtering/extracting nested JSON fields | JSON | Works on raw JSON regardless of cardinality |
| Full-text / token search | Text | For free-text columns; use `TEXT_MATCH` |
| Prefix / fuzzy string match | FST | Dictionary-backed; low-to-medium cardinality |
| Repeated aggregations / group-bys | Star-tree | Pre-aggregates on low-cardinality dimensions |
| Time-bucketed queries | Timestamp | On the time column |
If you're unsure, start with a range index on time and numeric filters and an inverted index on the low-cardinality string columns you filter on most. Add others only when a query needs them.
### Inverted index (low-cardinality equality)
Dictionary-backed, so keep the forward index RAW and add a dictionary block:
```json theme={null}
{
"name": "payment_type",
"encodingType": "RAW",
"indexes": {
"forward": { "encodingType": "RAW" },
"dictionary": {},
"inverted": { "disabled": false }
}
}
```
Example query:
```sql theme={null}
SELECT count(*) FROM nyc_taxi_trips WHERE payment_type = 'CASH';
```
Optional tuning: for External (tiered) tables, the cluster config `pinot.server.index.inverted.enable.startree.reader = true` selects the StarTree S3-optimized inverted index reader.
### Range index (numeric / time ranges)
No dictionary needed:
```json theme={null}
{
"name": "fare_amount",
"encodingType": "RAW",
"indexes": { "range": {} }
}
```
Example query:
```sql theme={null}
SELECT count(*) FROM nyc_taxi_trips WHERE fare_amount BETWEEN 10 AND 50;
```
### Bloom filter (high-cardinality equality)
```json theme={null}
{
"name": "trip_id",
"encodingType": "RAW",
"indexes": { "bloom": {} }
}
```
Example query:
```sql theme={null}
SELECT * FROM nyc_taxi_trips WHERE trip_id = 't_8f3a91';
```
### JSON index
```json theme={null}
{
"name": "surcharges",
"encodingType": "RAW",
"indexes": { "json": {} }
}
```
Example query:
```sql theme={null}
SELECT count(*) FROM nyc_taxi_trips WHERE JSON_MATCH(surcharges, '"$.type" = ''airport''');
```
### Text index (full-text search)
```json theme={null}
{
"name": "notes",
"encodingType": "RAW",
"indexes": { "text": {} }
}
```
Example query:
```sql theme={null}
SELECT count(*) FROM nyc_taxi_trips WHERE TEXT_MATCH(notes, 'airport AND delay');
```
The onboarding/preview flow generates the base `fieldConfigList` with RAW encoding for every column. To add an index, merge the `indexes` block above into the matching column entry — don't change `encodingType`.
# Observability
Source: https://docs.startree.ai/corecapabilities/external-table/observability
REST endpoints to check an External Table sync run's status and ingestion lag against the upstream table, or trigger a run on demand.
Once an External Table is created, the controller's External Table watcher ingests data on a schedule. These read-only endpoints let you check each run's status and see how far the queryable data lags behind the upstream table. There is also an endpoint to trigger a run immediately instead of waiting for the next scheduled tick.
The observability endpoints require `executor=controller` in the table's `ExternalTableSyncTask` config (the default for tables onboarded via the Data Portal or the preview API).
All paths are relative to your data-plane base URL — set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` (the StarTree Cloud data-plane proxy). Add an `Authorization` header if your controller requires auth.
**Run status is not the same as data freshness.** A `fileOnboardingRun.status` of `COMPLETED` only means the last sync run finished — the table can still be behind the source, because the upstream table may have committed more data since. To judge whether your data is current, use [`syncState`](#run-status) (with `lag.percentDataIngested` and `lag.snapshotsBehind` for detail), not the run status.
***
## Common tasks
A map from what you want to know to the call that answers it and the field to read. Each row links to the full reference below.
| I want to… | Call | Look at |
| - | - | - |
| Check the last sync succeeded | `GET …/externalTable/status` | [`fileOnboardingRun.status`](#run-status) |
| See what an in-progress run is doing right now | `GET …/externalTable/status` | [`fileOnboardingRun.currentPhase`](#run-status) + `currentPhaseStartTime` |
| Diagnose a stuck or failed sync | `GET …/externalTable/status` | [`fileOnboardingRun.failurePhase`](#run-status) + `errorMessage`, then [Troubleshooting](./troubleshooting#sync-not-progressing) |
| Check my table is caught up with the source | `GET …/externalTable/status?includeLag=true` | [`syncState`](#run-status) / `lag.percentDataIngested` |
| Know how far behind my data is, as a duration | `GET …/externalTable/status?includeLag=true` | `lag.upstream.committedAtMs` − `lag.synced.committedAtMs` |
| Track progress on a snapshot too large for one run | `GET …/externalTable/status` | [`snapshotIngestion`](#snapshot-ingestion-cross-run-progress) |
| See when the next sync is scheduled | `GET …/externalTable/status` | `nextRunTime` |
| Check whether the current snapshot is fully ingested | `GET …/externalTable/checkpoint` | [`state`](#ingestion-watermark) |
| Trigger a sync now instead of waiting | `POST /tasks/schedule` | [Trigger a sync run](#trigger-a-sync-run-optional) |
Prefer clicking to curl? A table's details page in the Data Portal shows the same run status and sync controls — see [Trigger a sync run](#trigger-a-sync-run-optional).
***
## API Endpoints Quick Reference
| Method | Endpoint | Purpose |
| - | - | - |
| `GET` | `/tables/{tableNameWithType}/externalTable/status` | Status of the last (or current) sync run. Add `includeLag=true` for [ingestion lag](#ingestion-lag). |
| `GET` | `/tables/{tableNameWithType}/externalTable/checkpoint` | The [ingestion watermark](#ingestion-watermark): which snapshot the table is on and whether it is fully ingested. |
| `POST` | `/tasks/schedule` | Optionally trigger a sync run now instead of waiting for the schedule. |
| `POST` | `/tables/{tableNameWithType}/iceberg/isSnapshotReady` | Check whether a specific Iceberg snapshot's [deletion vectors](./deletion-vectors) have converged on all servers. |
***
## Run status
`GET /tables/{tableNameWithType}/externalTable/status`
*Use this to answer "did the last sync run work?" and — with `includeLag=true` — "is my data current?"*
Returns the External Table sync watcher's last run status for the table, under a `fileOnboardingRun` object. With `includeLag=true`, the response additionally reports how far the table's queryable data is behind the upstream Iceberg table — see [Ingestion lag](#ingestion-lag).
| Query parameter | Default | Description |
| - | - | - |
| `includeLag` | `false` | Also return ingestion lag as a nested `lag` object (available starting in **StarTree release 0.16.0**), and compute [`syncState`](#run-status). Adds one catalog read; the default call stays cheap (a single ZooKeeper read, no catalog call). |
| `pendingExact` | **`true`** | Only with `includeLag=true`: compute pending files and bytes from an exact manifest diff instead of snapshot summary totals. Exact is now the default because summary arithmetic cannot answer the pending question at all once the source has non-append commits (compaction, overwrite, delete) — it yields no estimate rather than a wrong one. Pass `pendingExact=false` to opt out to cheaper summary arithmetic. See [Exact pending mode](#exact-pending-mode). |
**Top-level fields:**
| Field | Type | Description |
| - | - | - |
| `tableNameWithType` | string | Table name with the `_OFFLINE` suffix. |
| `fileOnboardingRun` | object | The run happening right now — see below. Reports `IDLE` between runs; it never restates a finished run once history has recorded it. |
| `snapshotIngestion` | object \| null | Cross-run view of the snapshot currently being ingested — see [Snapshot ingestion](#snapshot-ingestion-cross-run-progress). `null` when the table has neither a watermark nor a finished run. |
| `syncState` | string | `CAUGHT_UP` \| `CATCHING_UP` \| `STALLED` \| `UNKNOWN`. One-value ingestion health verdict. Requires `includeLag=true` — without it, always `UNKNOWN` (never assumed healthy). |
| `nextRunTime` / `nextRunTimeMs` | string \| null / long \| null | Next scheduled sync, read from the table's live cron trigger. `null` when the table has no cron schedule or cron scheduling is off on this controller. |
| `lag` | object \| null | Only with `includeLag=true` (the key is absent otherwise): the [ingestion lag object](#ingestion-lag), or `null` when lag does not apply to the table or could not be computed. |
| `lagMessage` | string \| null | Only with `includeLag=true` (the key is absent otherwise): explains why `lag` is `null`; `null` when lag was computed. |
**`fileOnboardingRun` fields:**
| Field | Type | Description |
| - | - | - |
| `status` | string | `IDLE` \| `RUNNING` \| `COMPLETED` \| `FAILED`. |
| `startTime` | string \| null | Run start, ISO-8601 UTC (e.g. `2026-02-09T18:53:20Z`); `null` if never started. |
| `startTimeMs` | long | Run start as raw epoch milliseconds; `0` if never started. |
| `endTime` | string \| null | Run end, ISO-8601 UTC; `null` while `RUNNING` or if never finished. |
| `endTimeMs` | long | Run end as raw epoch milliseconds; `0` while `RUNNING` or if never finished. |
| `filesDiscovered` | int | Source files found in this run. |
| `segmentsUploaded` | int | Segments successfully created and uploaded. Stays `0` until the upload phase — a run generates and compresses every segment for its page before uploading them as one batch, so `currentPhase`, not this counter, is the progress signal for an in-progress run. |
| `filesFailed` | int \| null | Files this run could not turn into segments and skipped (only possible under `continueOnFileError`). `null` means the run reported no outcome for it yet, which is not the same as `0`. A `COMPLETED` run with a non-zero count is a partial success. |
| `currentPhase` | string \| null | Phase the in-flight run is currently executing — same vocabulary as `failurePhase` below. `null` once the run reaches a terminal state. |
| `currentPhaseStartTime` / `currentPhaseStartTimeMs` | string \| null / long \| null | When the run entered `currentPhase`; time-in-phase is `now − currentPhaseStartTimeMs`. `null` whenever `currentPhase` is `null` — this is what surfaces a run parked for an hour in `WAITING_FOR_SEGMENTS_ONLINE` instead of just reading `RUNNING`. |
| `errorMessage` | string \| null | Populated only when `status=FAILED`. |
| `failurePhase` | string \| null | When failed, where it failed: `LISTING_FILES` (credentials/path), `GENERATING_SEGMENTS` (type mismatch), `COMPRESSING_SEGMENTS` (server resources), `UPLOADING_SEGMENTS` (deep-store permissions), `PUBLISHING_SNAPSHOT`, `WAITING_FOR_SEGMENTS_ONLINE`, `WAITING_FOR_SERVERS_READY`, `SAVING_PROGRESS` (ZooKeeper), or `UPDATING_SCHEMA`. `null` on success. See [Troubleshooting](./troubleshooting#sync-not-progressing). |
| `checkpointValue` | string \| null | Snapshot ID / watermark after a successful run. |
| `requestId` | string \| null | Identifier of the run, for log correlation. |
**`failurePhase` / `currentPhase` values were renamed** to describe what the sync is doing rather than Pinot's internal step names: `FILE_LISTING`→`LISTING_FILES`, `SCHEMA_UPDATE`→`UPDATING_SCHEMA`, `SEGMENT_GENERATION`→`GENERATING_SEGMENTS`, `SEGMENT_COMPRESSION`→`COMPRESSING_SEGMENTS`, `SEGMENT_UPLOAD`→`UPLOADING_SEGMENTS`, `SNAPSHOT_FINALIZE`→`PUBLISHING_SNAPSHOT`, `IS_EV_CONVERGENCE`→`WAITING_FOR_SEGMENTS_ONLINE`, `SNAPSHOT_READINESS_POLL`→`WAITING_FOR_SERVERS_READY`, `CHECKPOINT_SAVE`→`SAVING_PROGRESS`. No alias is kept — update any dashboard, alert, or script matching on the old strings.
| `status` | Meaning |
| - | - |
| `IDLE` | No run yet, or sitting between scheduled runs. |
| `RUNNING` | A run is in progress — check `currentPhase` for what it's doing right now. |
| `COMPLETED` | Last run succeeded — check `checkpointValue`. |
| `FAILED` | Last run failed — check `errorMessage` and `failurePhase`. See [Troubleshooting → Sync not progressing](./troubleshooting#sync-not-progressing) for a diagnosis guide per phase. |
```bash theme={null}
curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/status"
```
```json theme={null}
{
"tableNameWithType": "nyc_taxi_trips_OFFLINE",
"fileOnboardingRun": {
"status": "COMPLETED",
"startTime": "2026-02-09T18:53:20Z",
"startTimeMs": 1770663200000,
"endTime": "2026-02-09T18:54:20Z",
"endTimeMs": 1770663260000,
"filesDiscovered": 15,
"segmentsUploaded": 15,
"filesFailed": null,
"currentPhase": null,
"currentPhaseStartTime": null,
"currentPhaseStartTimeMs": null,
"errorMessage": null,
"failurePhase": null,
"checkpointValue": "1234567890123456789",
"requestId": "a1b2c3d4"
},
"snapshotIngestion": {
"snapshotId": "1234567890123456789",
"state": "DONE",
"runs": 1,
"totalFiles": 15,
"filesDiscovered": 15,
"segmentsUploaded": 15,
"filesFailed": null,
"startedAt": "2026-02-09T18:53:20Z",
"startedAtMs": 1770663200000,
"updatedAt": "2026-02-09T18:54:20Z",
"updatedAtMs": 1770663260000,
"consecutiveFailedRuns": 0,
"lastRunStatus": "COMPLETED",
"lastRunAt": "2026-02-09T18:54:20Z",
"lastRunAtMs": 1770663260000,
"lastRunError": null
},
"nextRunTime": "2026-02-09T19:53:20Z",
"nextRunTimeMs": 1770666800000,
"syncState": "UNKNOWN"
}
```
**Status codes:**
| Code | When |
| - | - |
| `200` + `lag: null` + `lagMessage` | OK, but lag does not apply — the table is not `catalogType=iceberg-rest` with `executor=controller`, or has no sync task config. |
| `200` + `lag` with some `null` fields + `lag.message` | OK — a transient catalog/ZooKeeper issue degraded individual lag fields. Lag never returns `500`. |
| `404` | Table not found. |
| `500` | Internal error (does not apply to lag computation, which degrades to `null` fields instead). |
### Ingestion lag
`GET /tables/{tableNameWithType}/externalTable/status?includeLag=true`
*Use this when you need to know whether the table is current with the source — and, if not, by how much.*
Ingestion lag reporting is available starting in **StarTree release 0.16.0**.
Reports how far the table's queryable data is behind the upstream Iceberg table. Lag is expressed as a snapshot delta rather than a time lag: the last fully synced snapshot versus the upstream head snapshot, plus the files and bytes still pending.
Lag applies to `catalogType=iceberg-rest` tables synced with `executor=controller`. For any other table the endpoint still returns `200` with `lag: null` and a `lagMessage` explaining why; the status response is never failed because of lag. Snapshot lag is undefined for raw S3 listings, for example.
```bash theme={null}
curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/status?includeLag=true"
```
```json theme={null}
{
"tableNameWithType": "nyc_taxi_trips_OFFLINE",
"fileOnboardingRun": {
"status": "COMPLETED",
"startTime": "2026-02-09T18:53:20Z",
"startTimeMs": 1770663200000,
"endTime": "2026-02-09T18:54:20Z",
"endTimeMs": 1770663260000,
"filesDiscovered": 15,
"segmentsUploaded": 15,
"filesFailed": null,
"currentPhase": null,
"currentPhaseStartTime": null,
"currentPhaseStartTimeMs": null,
"errorMessage": null,
"failurePhase": null,
"checkpointValue": "1234567890123456789",
"requestId": "a1b2c3d4"
},
"snapshotIngestion": {
"snapshotId": "1234567890123456789",
"state": "DONE",
"runs": 1,
"totalFiles": 120,
"filesDiscovered": 120,
"segmentsUploaded": 120,
"filesFailed": null,
"startedAt": "2026-02-09T18:53:20Z",
"startedAtMs": 1770663200000,
"updatedAt": "2026-02-09T18:54:20Z",
"updatedAtMs": 1770663260000,
"consecutiveFailedRuns": 0,
"lastRunStatus": "COMPLETED",
"lastRunAt": "2026-02-09T18:54:20Z",
"lastRunAtMs": 1770663260000,
"lastRunError": null
},
"nextRunTime": "2026-02-09T19:53:20Z",
"nextRunTimeMs": 1770666800000,
"lag": {
"tableNameWithType": "nyc_taxi_trips_OFFLINE",
"synced": {
"snapshotId": "1234567890123456789",
"sequenceNumber": 41,
"committedAtMs": 1770663000000,
"committedAt": "2026-02-09T18:50:00Z",
"totalFiles": 120,
"totalSize": "5.00 GB",
"totalRecords": 48000000
},
"upstream": {
"snapshotId": "2345678901234567890",
"sequenceNumber": 44,
"committedAtMs": 1770663480000,
"committedAt": "2026-02-09T18:58:00Z",
"totalFiles": 192,
"totalSize": "8.00 GB",
"totalRecords": 76800000
},
"snapshotsBehind": 3,
"filesPending": 72,
"bytesPending": 3221225472,
"pendingExact": true,
"pendingState": "EXACT",
"unavailableReason": null,
"percentDataIngested": 62.5,
"syncedAt": "2026-02-09T18:54:00Z",
"syncedAtMs": 1770663240000,
"controllerTimeMs": 1770663300000,
"controllerTime": "2026-02-09T18:55:00Z",
"message": null
},
"lagMessage": null,
"syncState": "CATCHING_UP"
}
```
**`lag.caughtUp` was removed.** It was fully derivable from the top-level `syncState`: `true` always meant `CAUGHT_UP`, `false` never did, `null` meant `UNKNOWN`. Read `syncState` instead — it collapses ingestion health to one of `CAUGHT_UP` / `CATCHING_UP` / `STALLED` / `UNKNOWN` so a client never has to infer health from a nullable boolean and a percent.
**`lag` object fields:**
| Field | Type | Description |
| - | - | - |
| `tableNameWithType` | string | Echoes the table name with the `_OFFLINE` suffix. |
| `synced` | object \| null | The last fully synced snapshot: the newest snapshot whose data is actually queryable. It is recorded only after the ingested segments have converged on all servers. `null` when the table has never synced or when the synced state could not be read; `message` distinguishes the two cases. |
| `upstream` | object \| null | The upstream Iceberg head snapshot, read from the catalog. `null` when the catalog is unreachable or the upstream table has no snapshots. |
| `snapshotsBehind` | int \| null | How many upstream snapshots sit between the synced snapshot and the head. Best-effort: exact for the linear histories ingestion produces, but branches, tags, WAP snapshots, or retention can skew it. `null` on Iceberg v1 metadata. |
| `filesPending` | long \| null | Data files not yet queryable. An exact manifest diff by default (`pendingState=EXACT`), or a net delta of snapshot summary totals with `pendingExact=false` (`pendingState=ESTIMATED`). `null` when it cannot be computed honestly (`pendingState=UNAVAILABLE`; see `unavailableReason`). |
| `bytesPending` | long \| null | Bytes not yet queryable, as a raw number, computed the same way as `filesPending`. |
| `pendingExact` | boolean | Whether the pending numbers actually came from the exact manifest diff — equivalent to `pendingState == EXACT` (including a caught-up table, where pending is `0` by definition). This is **not** an echo of the `?pendingExact` request flag: a diff that was requested but could not be delivered (cap or concurrency limit hit) reports `false` here. |
| `pendingState` | string | `EXACT` \| `ESTIMATED` \| `UNAVAILABLE` — how `filesPending` / `bytesPending` / `percentDataIngested` were arrived at. |
| `unavailableReason` | string \| null | Set only when `pendingState=UNAVAILABLE`: `SYNCED_SNAPSHOT_EXPIRED`, `NON_APPEND_HISTORY`, `DIFF_CAP_EXCEEDED`, `CONCURRENCY_LIMIT`, `EXACT_DIFF_FAILED`, `CATALOG_UNREACHABLE`, `UPSTREAM_HAS_NO_SNAPSHOTS`, `SYNCED_STATE_UNREADABLE`, or `SYNCED_AHEAD_OF_HEAD`. `message` carries the human-readable detail. |
| `percentDataIngested` | double \| null | Percent of upstream bytes already queryable. `100` when caught up, `0` when never synced, `null` when it cannot be computed. |
| `syncedAt` / `syncedAtMs` | string \| null / long \| null | When *this table* committed the synced snapshot (from the internal `ActiveSnapshots` record) — not the upstream commit time. Unlike `synced.committedAt`, this survives the synced snapshot expiring out of source retention, so it stays populated even when `synced.totalFiles` etc. go `null`. |
| `controllerTimeMs` | long | Controller clock at computation time, raw epoch milliseconds. |
| `controllerTime` | string | Controller clock at computation time, ISO-8601 UTC (display companion of `controllerTimeMs`). |
| `message` | string \| null | Explains any degraded or missing fields, for example summary totals being unavailable or the exact-diff cap being hit. `null` when everything was computed cleanly. |
**Snapshot fields (`synced` and `upstream`):**
| Field | Type | Description |
| - | - | - |
| `snapshotId` | string \| null | Iceberg snapshot ID, serialized as a JSON **string** — Iceberg ids exceed 2^53, so a JSON number silently loses precision in a JavaScript client. |
| `sequenceNumber` | long \| null | Iceberg sequence number. |
| `committedAtMs` | long \| null | The upstream Iceberg commit time as raw epoch milliseconds. This is always catalog commit time, never Pinot ingest time. |
| `committedAt` | string \| null | The commit time as ISO-8601 UTC, truncated to seconds (display companion of `committedAtMs`). |
| `totalFiles` | long \| null | Total data files at this snapshot, from the snapshot summary. `null` when the snapshot has expired from table metadata or the summary keys are absent. |
| `totalSize` | string \| null | Total data size at this snapshot, human-readable (e.g. `"5.00 GB"`), from the snapshot summary. Same nullability as `totalFiles`. |
| `totalRecords` | long \| null | Total records at this snapshot, from the snapshot summary. Same nullability as `totalFiles`. |
There is deliberately no `lagMs` field. To express lag as a duration, subtract `lag.synced.committedAtMs` from `lag.upstream.committedAtMs` (raw epoch milliseconds; the ISO-8601 `committedAt` strings are seconds-precision display companions). Both timestamps come from the same upstream catalog clock, so the difference is not skewed by clock drift between the catalog and the Pinot controller.
Transient catalog or ZooKeeper failures never turn into a `500`. The affected lag fields are returned as `null` with an explanatory `message`, and a synced state that exists but cannot be read is reported as unavailable rather than as a confident "nothing ingested".
#### Exact pending mode
`pendingExact` now **defaults to `true`**: the controller diffs the live data-file sets of the head and synced snapshots, giving exact `filesPending` / `bytesPending`. This replaced defaulting to the cheap summary-totals estimate, because netting cumulative summary totals cannot answer the pending question at all once the source has non-append commits (compaction, overwrite, delete) between the synced and head snapshots — it silently reports nothing (`pendingState=UNAVAILABLE`) rather than a wrong number.
The exact diff enumerates files and is capped at `iceberg.snapshotProcessing.lagMaxDiffFiles` files per snapshot enumeration (default 100,000) — a per-table key in the `ExternalTableSyncTask` config. It is also bounded controller-wide by the config below; either cap degrades the response to the summary estimate and says so in `lag.message`, with `pendingState=UNAVAILABLE` (or `ESTIMATED` if the fallback succeeds) and `pendingExact: false`.
Pass `?pendingExact=false` to opt out to the O(1) summary-totals estimate up front.
| Controller config | Default | Description |
| - | - | - |
| `controller.externalTable.lag.maxConcurrentExactDiffs` | `4` | Controller-wide ceiling on exact-diff lag computations running at once. Each in-flight diff holds a path set of up to `lagMaxDiffFiles` entries for the whole enumeration, so this — not the `pendingExact` query param — is what bounds cost against a polling dashboard. A request over the ceiling degrades to the summary estimate rather than queueing. |
### Integration notes
* **Pending fields can be `null` even when `syncState` is not `CAUGHT_UP`.** A compaction, overwrite, or delete between snapshots can leave both the exact diff and the summary estimate unable to answer (`pendingState=UNAVAILABLE`), so `filesPending`, `bytesPending`, and `percentDataIngested` may come back `null`. Fall back to `snapshotsBehind`; `lag.message` and `lag.unavailableReason` explain why.
* **Two different message fields.** Top-level `lagMessage` means `lag` itself is `null` (lag does not apply to the table); without `includeLag=true` neither key appears at all. `lag.message` means `lag` is present but an individual field is degraded.
* **Display strings vs numbers.** `startTime`, `endTime`, `committedAt`, `controllerTime`, `syncedAt`, and `totalSize` are display strings. Every timestamp string has a raw epoch-milliseconds companion (`startTimeMs`, `endTimeMs`, `committedAtMs`, `controllerTimeMs`, `syncedAtMs`) — use those for arithmetic. `filesPending`, `bytesPending`, `percentDataIngested`, `totalFiles`, `totalRecords`, and `snapshotsBehind` remain numbers.
***
## Snapshot ingestion (cross-run progress)
*Use this when a single snapshot is too large to ingest in one run, and you need to know how far the ingestion as a whole has got — not just the last batch.*
`fileOnboardingRun` is overwritten with `RUNNING` the moment the next run starts and reports `IDLE` between runs, and each run's own counts (`filesDiscovered`, `segmentsUploaded`) cover only that run's page of at most `iceberg.snapshotProcessing.batchSize` files. Neither one answers "how far through this snapshot are we" for a snapshot ingested over many runs. The `snapshotIngestion` object in the [Run status](#run-status) response does: it accumulates `runs`, `filesDiscovered`, `segmentsUploaded`, and `filesFailed` across every run that has processed the snapshot the table is currently on, alongside the snapshot's own `totalFiles` for context.
It also carries the outcome of the **last run that actually finished** — `lastRunStatus`, `lastRunAt`, `lastRunError` — plus `consecutiveFailedRuns`, since a fresh `RUNNING` record can otherwise hide a failure for the length of one cron tick. The failure streak is counted per run, not per snapshot, so a run that re-targets a newer upstream snapshot does not reset it.
| Field | Type | Description |
| - | - | - |
| `snapshotId` | string \| null | The snapshot these totals belong to — compare against the ingestion watermark's `checkpointValue` before trusting the counts as current for a snapshot in progress. |
| `state` | string \| null | The ingestion watermark's [state](#ingestion-watermark) (`IN_PROGRESS` \| `DONE` \| `FAILED`) for this snapshot. |
| `runs` | int \| null | Runs that have processed this snapshot, successful or not. Well above `ceil(files / batchSize)` means runs are failing and retrying. `null` until the snapshot's first run finishes. |
| `totalFiles` | long \| null | Data files in the snapshot from its own metadata summary — its size, not a work estimate (use `lag.filesPending` for what remains). `null` for catalogs without snapshot summaries (`catalogType=s3`) or when the summary could not be read. |
| `filesDiscovered` | int \| null | Files picked up for ingestion across those runs (attempts, not distinct files — a retried run's files are counted again). |
| `segmentsUploaded` | int \| null | Segments uploaded across those runs, counting only work that stuck. Progress on the snapshot is `segmentsUploaded / lag.filesPending`, since `filesPending` holds still at the snapshot's full new-file count until the snapshot publishes. |
| `filesFailed` | int \| null | Files skipped across those runs (only under `continueOnFileError`). `null` means no run has reported a skip outcome yet, not zero. |
| `startedAt` / `startedAtMs` | string \| null / long \| null | When the first run for this snapshot started. |
| `updatedAt` / `updatedAtMs` | string \| null / long \| null | When the most recent run for this snapshot finished. |
| `consecutiveFailedRuns` | int | Consecutive terminal runs that failed, table-scoped. Reset to `0` by a success; not reset by a run re-targeting a newer snapshot. |
| `lastRunStatus` | string \| null | `COMPLETED` \| `FAILED` — outcome of the last run to reach a terminal state. `null` until one has. |
| `lastRunAt` / `lastRunAtMs` | string \| null / long \| null | When that last completed run finished. |
| `lastRunError` | string \| null | Populated only when `lastRunStatus=FAILED`. |
`snapshotIngestion` is `null` when the table has neither an ingestion watermark nor a finished run yet (a brand-new table before its first sync).
***
## Ingestion watermark
`GET /tables/{tableNameWithType}/externalTable/checkpoint`
*Use this to answer "is the snapshot fully ingested, or only partly?"*
A large snapshot is ingested over many runs, one page of files at a time. A run that reports `COMPLETED` therefore says nothing about whether the snapshot as a whole is finished — it means that page succeeded. The watermark is what tracks the snapshot itself: which one the table is on, and how far through it ingestion has got.
Requires `executor=controller`; legacy minion-executor tables are rejected with `400`, because they never write a watermark and would otherwise be indistinguishable from a table that has not run yet. All value fields are `null` when the table has never synced. The Data Portal does not surface this endpoint — call it directly.
| Field | Type | Description |
| - | - | - |
| `tableNameWithType` | string | Table name with the `_OFFLINE` suffix. |
| `checkpointType` | string \| null | Provider type: `iceberg-rest` for every Iceberg catalog, `s3` for bucket-backed tables (including GCS via the S3-interop path). |
| `checkpointValue` | string \| null | The Iceberg snapshot ID being ingested, or the S3 sentinel `CURRENT`. |
| `state` | string \| null | Ingestion state for that snapshot — see below. |
| `updatedAtMs` | long \| null | When the watermark was last written. On `FAILED`, this is the "stalled since" time. |
| `consecutiveFailures` | int \| null | Failed runs in a row on this snapshot. Resets to `0` on a successful page; the snapshot goes `FAILED` when it reaches `iceberg.snapshotProcessing.maxRetries`. |
| `errorMessage` | string \| null | Error from the most recent failed run on this snapshot, `null` after a success. |
| `pageCursor` | object \| null | How far the file enumeration got: `{providerType, manifestIndex, entryPosition}` for Iceberg, `{providerType, lastKey}` for S3. `null` at a fresh start, and on an Iceberg snapshot that is `DONE`. |
| `state` | Meaning |
| - | - |
| `IN_PROGRESS` | The snapshot is partly ingested — more pages remain. Trigger further runs, or wait for the schedule. |
| `DONE` | The snapshot was ingested completely. The next run picks up the next upstream snapshot, if there is one. |
| `FAILED` | Ingestion stopped after `iceberg.snapshotProcessing.maxRetries` consecutive failures. `errorMessage` on this response says why; `failurePhase` on [Run status](#run-status) says where. The snapshot stays `FAILED` until a newer upstream snapshot arrives, unless `iceberg.snapshotProcessing.retryFailedSnapshot=true`. See [Troubleshooting](./troubleshooting#sync-not-progressing). |
| `TIMEDOUT` | Reserved; not currently emitted. |
```bash theme={null}
curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/checkpoint"
```
```json theme={null}
{
"tableNameWithType": "nyc_taxi_trips_OFFLINE",
"checkpointType": "iceberg-rest",
"checkpointValue": "1234567890123456789",
"state": "IN_PROGRESS",
"updatedAtMs": 1770663260000,
"consecutiveFailures": 0,
"errorMessage": null,
"pageCursor": { "providerType": "iceberg-rest", "manifestIndex": 3, "entryPosition": 1200 }
}
```
`state: DONE` is the signal that a table is at a snapshot boundary — the point at which it is safe to change how the table ingests, for example when [enabling segment groups](./segment-groups#enabling-grouping-on-a-live-table).
***
## Metrics and alerting
The status and lag endpoints answer on-demand questions. For dashboards and continuous monitoring, the controller also emits per-sync metrics (available wherever you scrape StarTree Pinot controller metrics).
**Per-phase sync timers** (milliseconds) — how long each stage of a run takes; useful for latency dashboards and for finding where a slow sync spends its time:
| Metric | Stage |
| - | - |
| `EXTERNAL_TABLE_SYNC_FILE_LISTING_TIME_MS` | Listing source files from the catalog |
| `EXTERNAL_TABLE_SYNC_SEGMENT_GENERATION_TIME_MS` | Converting Parquet to segments |
| `EXTERNAL_TABLE_SYNC_SEGMENT_COMPRESSION_TIME_MS` | Compressing segments |
| `EXTERNAL_TABLE_SYNC_SEGMENT_UPLOAD_TIME_MS` | Uploading segments to deep store |
| `EXTERNAL_TABLE_SYNC_SNAPSHOT_FINALIZE_TIME_MS` | Finalizing the snapshot |
| `EXTERNAL_TABLE_SYNC_CHECKPOINT_SAVE_TIME_MS` | Persisting the checkpoint / watermark |
| `EXTERNAL_TABLE_SYNC_OVERALL_TIME_MS` | End-to-end run duration |
**Deletion-vector convergence** — only when [deletion vectors](./deletion-vectors) are enabled; covers the server-readiness handshake:
| Metric | Meaning |
| - | - |
| `EXTERNAL_TABLE_SYNC_READINESS_POLL_SUCCESS` / `_TIMEOUT` / `_FAST_FAIL` / `_SKIPPED` / `_ATTEMPTS` | Outcomes of polling servers for snapshot readiness |
| `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIMEOUT` | A snapshot's ideal-state / external-view convergence timed out |
| `EXTERNAL_TABLE_SYNC_SNAPSHOT_READINESS_POLL_TIME_MS` / `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIME_MS` | Time spent polling for readiness and waiting for convergence |
**Sync health** — a per-table latch driven by run outcomes, so a dashboard or alert doesn't have to poll the status API:
| Metric | Meaning |
| - | - |
| `EXTERNAL_TABLE_SYNC_UNHEALTHY` (gauge) | `1` when the table's health latch is set (set on a `FAILED` run, carried across further failures, cleared only when a run completes); `0` otherwise. Emitted every tick, including no-work ticks. |
| `EXTERNAL_TABLE_SYNC_CONSECUTIVE_FAILURES` (gauge) | Consecutive failed sync runs for the table. |
| `EXTERNAL_TABLE_SYNC_RUN_FAILED_STALL` / `_ERROR_REPLICA` / `_EXCEPTION` (meters) | Reason a sync run stopped: a wait made no progress for its stall window, a (segment, instance) pair reached Helix `ERROR`, or any other exception. |
| `EXTERNAL_TABLE_SYNC_AUTO_RESET_PAIRS` (meter) | Table-wide `ERROR` (segment, instance) pairs swept by the one IS/EV convergence auto-reset a run may issue. See [`iceberg.snapshotProcessing.convergence.autoReset.maxErrorPairs`](./deletion-vectors#controller-is-ev-convergence). |
| `EXTERNAL_TABLE_SYNC_TASK_GENERATION_FAILURE` (meter, per `.`) | A sync pass failed for the table, counted once per failed pass regardless of cause — the total to alert on. The `_RUN_FAILED_*` meters above break the controller-executor subset down by reason (summing to at most this total); a failure raised before the run starts (task-config validation, catalog provider construction, an active-task conflict) lands only here. Exported as `pinot_controller_externalTableSyncTaskGenerationFailure_Count`. Previously these failures incremented the shared `FILE_INGESTION_FAILURE` meter, so they paged as `FileIngestionTask` failures — this is now a dedicated series. |
There is **no metric for ingestion lag** today — monitor that through the status API:
* **Data falling behind** — poll `…?includeLag=true` and alert when `syncState` is `STALLED` (or `CATCHING_UP` for longer than one sync interval), or `lag.percentDataIngested` stays below your threshold. Note `syncState` can flicker `STALLED` → `CATCHING_UP` on a fast-committing source: a permanently broken table's watermark cycles `FAILED` → `IN_PROGRESS` → `FAILED` as it re-targets each new upstream snapshot, so alert on the sustained condition, or on `snapshotIngestion.consecutiveFailedRuns` instead, which does not reset on a re-target.
* **Snapshot silently skipped** — a run that reports `COMPLETED` with `segmentsUploaded: 0` while `syncState` is not `CAUGHT_UP`. See [Sync says COMPLETED but no data lands](./troubleshooting#sync-says-completed-but-no-data-lands).
Per-run success/failure now has a metric too — `EXTERNAL_TABLE_SYNC_UNHEALTHY` above — but for triage detail (`failurePhase`, `errorMessage`) still poll `…/externalTable/status`.
***
## Trigger a sync run (optional)
`POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=`
You normally do **not** need this. After a table is created, the controller's watcher runs the first sync automatically and then re-syncs on the table's schedule. Use this endpoint only to start a run immediately instead of waiting for the next tick.
| Query parameter | Required | Description |
| - | - | - |
| `taskType` | Yes | Always `ExternalTableSyncTask`. |
| `tableName` | Yes | Table name **with the `_OFFLINE` suffix**, e.g. `nyc_taxi_trips_OFFLINE`. |
```bash theme={null}
curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE"
```
```json theme={null}
{ "ExternalTableSyncTask": "Task_ExternalTableSyncTask_3f2a1c9e-4b7d-4a1f-9c3e-2d8f5a6b7c9d_1707500000000" }
```
A table's details page in the Data Portal also exposes **Sync Now**, **Pause Sync**, and **Resume Sync** for the sync task, so you don't need to call these endpoints directly. Pausing removes the table's sync schedule (already-ingested data is untouched); resuming restores it — the primary action button itself becomes **Resume Sync** while the task is paused. See [Task Observability](/corecapabilities/observability/task-observability) for the broader task-monitoring view these controls sit alongside.
# Advanced Sizing & Operational Guidance
Source: https://docs.startree.ai/corecapabilities/external-table/operational-guidance
Capacity guardrails, deployment recommendations, and additional troubleshooting fixes for External Tables, beyond the core Best Practices and Troubleshooting references.
This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
Supplemental guidance for running External Tables at scale — capacity guardrails, deployment recommendations, and a couple of additional error fixes not covered in [Best Practices and Configs](./best-practices-and-configs) or [Troubleshooting](./troubleshooting).
***
## Why remote Parquet queries are fast
Most engines that query Iceberg/Parquet directly (e.g. Trino, Athena) read at the **row-group / column-chunk** level. Pinot reads at the **Parquet page** level — a finer granularity — and uses its own indexes (inverted, range) to skip straight to the pages that matter instead of scanning a whole column chunk. This page-level skip is what makes an indexed External Table competitive with a native Pinot segment despite the data living remotely.
External Tables also federate cleanly with regular Pinot tables. A common pattern keeps recent data (e.g. the last 24h) in a REALTIME table fed from Kafka, while the same events land in Iceberg for long-term retention — a single query can span both, giving one query surface across the full time range without copying the archive into Pinot.
***
## Capacity guardrails
These aren't hard limits, but the confident, well-tested operating range today:
| Guardrail | Guidance |
| - | - |
| Files per table | Up to **\~350,000** Parquet files. Each source file maps 1:1 to a pseudo-segment today, so file count directly drives segment count. |
| Segments per cluster | Stay well inside **\~500,000** total — treat this and the per-table file guardrail as overlapping budgets, not independent ones. |
| Segments per server | Rule of thumb: **don't exceed \~10,000 segments per server**. |
| File size | **\< 100 MB** is too small — too many files for the benefit; ask the source to compact. **128 MB – 512 MB** is the well-tested sweet spot. Up to **\~2 GB** generally works; beyond that (up to \~4 GB) is less exercised. |
**The per-table guardrail is also enforced.** The controller's segment limit checker — which applies to external tables only — fails ingestion, task generation, and segment upload once a table crosses `controller.segmentLimitChecker.maxSegmentsPerTable` (default **350,000**) or the cluster crosses `controller.segmentLimitChecker.maxSegmentsInCluster` (default **1,000,000**). The \~500,000 cluster figure above remains the recommended comfortable operating range; the enforced ceiling is higher.
**Millions of files isn't viable today.** If a source has millions of small files, either compact upstream to raise average file size and cut file count below the guardrail, or onboard a subset first (e.g. the most recent partitions) and expand later rather than pointing at the full dataset on day one.
**Full-refresh Iceberg tables** (where a snapshot replaces all existing files rather than appending) are supported, but a full replace can spike the segment count for that cycle. Before onboarding one, check how many files the refreshed snapshot contains, whether old segments need cleanup after each refresh, and whether the file count stays inside the guardrails above.
***
## Deployment
* **Use a dedicated tenant** for External Table workloads rather than an existing production tenant — it isolates ingestion and query resource usage from other traffic on the cluster.
* **Default replication factor is 1**, chosen to maximize ingestion throughput; pseudo-segments are small (single-digit KB to a few MB before indexes are attached), so the cost of raising it later — e.g. to 2 once onboarding is stable — is low.
* Both **SSE and MSE** query execution modes are supported against External Tables.
* For **AWS Glue** sources specifically, the cluster's IAM role/credentials also need Glue API access, in addition to the S3 permissions in [Troubleshooting → Access Denied](./troubleshooting#access-denied-http-403-when-creating-the-table): `glue:GetTable`, `glue:GetTables`, `glue:GetCatalog`, `glue:GetDatabase`, `glue:GetDatabases`, `glue:GetPartition`, `glue:GetPartitions`, `glue:BatchGetPartition`, plus `lakeformation:GetDataAccess` if the database is registered with Lake Formation. Scope these to the specific Glue databases/tables the cluster needs, and prefer an assumed IAM role over static access keys.
***
## Ingestion mechanics
Each sync run has two phases, which explains a couple of behaviors that are otherwise easy to mistake for a bug:
1. **Segment generation (controller).** Source files are processed in batches (`iceberg.snapshotProcessing.batchSize`, default `1000`, capped by `iceberg.snapshotProcessing.maxFilesScannedPerRun`; the `taskMaxNumFiles` cap applies only to the legacy minion-executor path). The controller reads Parquet footer statistics (row count, min/max) and creates pseudo-segments — metadata and index pointers, not materialized data.
2. **Download and index (servers).** Servers download the segment artifacts and build indexes. **Queries can return partial results while this phase is in flight** — for example, a response noting some segments are unavailable — until all servers finish loading.
**Sort or cluster upstream data** on the column most often used in equality filters (in the Iceberg/Parquet writer, before StarTree ever sees it) — this reduces random scans at read time, on top of whatever indexes are added on the Pinot side.
**Disable the sync schedule after the initial bootstrap** for a table that only watches an S3 path for a one-time or occasional load (`enabled=false` on `ExternalTableSyncTask`). Leaving it on means every scheduled tick re-scans and bootstraps newly-discovered files, so the segment count keeps growing even when there's no new data to justify it. For catalog-backed incremental sources, rely on checkpointing (see [Best Practices & Configs — Checkpointing & schema evolution](./best-practices-and-configs#checkpointing--schema-evolution)) instead of re-bootstrapping.
***
## Null handling — why it matters
Without null handling, Pinot doesn't build null bitmap vectors during segment generation, so columns containing nulls in the source Parquet aren't tracked — `IS NULL`, `IS NOT NULL`, and aggregations that depend on the distinction (`COUNT(col)` vs. `COUNT(*)`) will be wrong on those columns.
**Current onboarding flows handle this automatically:** the preview/onboarding step sets `nullHandlingEnabled: true` on every generated table config, and when the source schema has nullable fields the inferred Pinot schema marks those columns nullable and enables column-based null handling.
For tables created on older builds (where onboarding left it off), turn it on explicitly:
```json theme={null}
"tableIndexConfig": {
"nullHandlingEnabled": true
}
```
Verify null behavior after ingestion either way — see [Null Handling in Complex Types](/corecapabilities/external-table/data-type-mapping#null-handling-in-complex-types) for a known `MAP`-column caveat.
***
## Missing time-column statistics — tolerated by default on new tables
Source Parquet can legitimately carry no min/max footer statistics for the time column — an all-null chunk, a producer-side writer change, or Iceberg schema evolution. Under Pinot's strict default this fails segment generation with `No time statistics available for datetime column ... Cannot determine time range.`
**Tables created through the preview API or `CREATE TABLE ... WITH (type = 'iceberg', ...)` are scaffolded with `ingestionConfig.segmentTimeValueCheck: false`**, so files missing time-column stats ingest instead of failing. A segment built from such a file has no derivable time range and is therefore **not pruned by time** — a time-filtered query scans it. Files whose Parquet does carry statistics are unaffected and keep their full time range. Corrupt or out-of-range time values are still rejected regardless of this flag.
To restore strict checking, set `"segmentTimeValueCheck": true` under `ingestionConfig` — in the config returned by the preview API before creating the table, or via a table-config edit afterwards for a DDL-created or existing table.
**Existing tables, direct `POST /tables` with a hand-written config, and Pinot-managed (non-External) tables are unaffected** — the tolerant default applies only to preview- and DDL-scaffolded External Tables created after upgrading.
***
## Additional troubleshooting
### ZooKeeper packet-size overflow as segment count grows
**Symptom:**
```
Caused by: org.apache.zookeeper.KeeperException$...
Packet len ... is out of range
```
**Cause:** The serialized ZNode record for the table has grown past ZooKeeper's default 1 MB packet limit — this shows up as segment count increases.
**Fix:**
1. Cluster config: `"zk.serializer.znrecord.write.size.limit.bytes": "62914560"` (60 MB — raise further if the error persists as more segments are added).
2. JVM flag on **both** Controller and Server: `-Djute.maxbuffer=62914560`.
3. Restart Controller and Server.
### 403 on segment upload with 1970-era timestamps
**Symptom:**
```
Got error status code: 403 (Forbidden) with reason:
"Invalid segment start/end time: 1970-01-21T11:51:21.667Z / ... for segment: ..."
```
**Cause:** Iceberg-generated Parquet files store timestamps in milliseconds; Pinot's segment time-value sanity check misreads them as seconds, producing 1970-era dates and rejecting the segment.
**Fix:** Add to the table's `ingestionConfig` (at creation, or via a config update followed by re-triggering ingestion):
```json theme={null}
"ingestionConfig": {
"segmentTimeValueCheck": false
}
```
# External Tables Overview
Source: https://docs.startree.ai/corecapabilities/external-table/overview
Query Iceberg (AWS Glue, S3 Tables, Unity Catalog, Apache Polaris, Snowflake, Nessie) and raw Parquet (S3, GCS) tables directly from Pinot, with no ingestion or data duplication.
This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
An External Table is a Pinot table whose data stays in Parquet files in your object store — an Iceberg catalog (AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris, Snowflake Open Catalog / Horizon, Nessie) or an S3/GCS data lake — instead of being copied into Pinot's own segment format. Pinot reads the remote Parquet at query time and exposes it through standard SQL. There is no ETL pipeline, no data duplication, and onboarding takes minutes instead of hours.
A watcher on the controller detects source-side changes on a schedule and builds segment index files (bloom filters, inverted indexes, range indexes, etc.) alongside the Parquet. Queries use server-side caches — a Parquet data cache, an index cache, and a footer cache — so repeat reads avoid paying S3 round-trip latency. With the right indexes configured, query times drop from minutes to milliseconds on large datasets.
For a single-page view of everything External Tables support — connectors, authentication methods, table formats, and features — see the [Support Matrix](/corecapabilities/external-table/support-matrix).
***
## How it works
```mermaid theme={null}
flowchart LR
UserSQL["SQL Query"] --> Broker["Pinot Broker"]
Broker --> Server["Pinot Server"]
Server --> CacheCheck{"Cache hit?"}
CacheCheck -->|"Yes"| Result["Return result"]
CacheCheck -->|"No"| S3["S3 / Object Store\nParquet files"]
S3 --> Server
Server --> LocalCache["Local caches\ndata · index · footer"]
LocalCache --> Result
```
At query time, the server checks its local caches first. On a miss, it fetches the required Parquet column pages or index byte ranges from object storage and stores them for subsequent queries. Index files (built by the watcher at sync time) live in tiered storage and are also cached locally, so filters and aggregations avoid full column scans.
***
## Supported sources
| Source | Protocol | `catalogType` |
| - | - | - |
| Iceberg (AWS Glue) | Iceberg REST | `iceberg-rest` (`serviceType=glue`) |
| Iceberg (AWS S3 Tables) | Iceberg REST | `iceberg-rest` (`serviceType=s3Tables`) |
| Unity Catalog | Iceberg REST | `iceberg-rest` (`serviceType=unity`) |
| Nessie | Iceberg REST | `iceberg-rest` (`serviceType=rest`, with `prefix` set to a branch/ref) |
| Apache Polaris / Snowflake Open Catalog | Iceberg REST | `iceberg-rest` (`serviceType=rest`, with `warehouse` set to the catalog name) |
| Snowflake Horizon Catalog | Iceberg REST | `iceberg-rest` (`serviceType=rest`, served under `/polaris/api/catalog` on the Snowflake account host) |
| S3 Data Lake | Raw Parquet files under an S3 prefix | `s3` |
| GCS Data Lake | Raw Parquet files in GCS via S3 interop | `gcs-interop` |
`catalogType=iceberg-rest` works with any Iceberg REST–compliant catalog (built-in service adapters exist for AWS Glue, AWS S3 Tables, Unity Catalog, and generic REST — the last of which also covers Nessie, addressed via its branch/ref as the REST `prefix`). Data files must be Parquet.
There is no dedicated `catalogType` for Delta Lake's native log format. A **Delta Lake table with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled** publishes Iceberg-compatible metadata alongside its Delta log — registering it in a **Unity Catalog** (`serviceType=unity`) makes it queryable as an External Table through the same Iceberg REST path used for Glue and S3 Tables.
***
## Where to start
New here? Start with a task instead of a vendor:
* **["I just have Parquet files on S3/GCS"](./getting-started/query-without-a-catalog)** — no catalog, no Iceberg, query the bucket directly.
* **["I already have a catalog"](./getting-started/connect-a-catalog)** — AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris / Snowflake, or Nessie.
* **["My queries are slow"](./getting-started/speed-up-queries)** — indexes, caching, and OOM protection.
Otherwise, choose your catalog type and pick your preferred onboarding path:
**AWS Glue**
* [Onboarding via Data Portal](./glue/onboarding-data-portal) — wizard-based setup for AWS Glue
* [Onboarding via API](./glue/onboarding-api) — 4-step REST flow with bash examples and a quickstart script
**Amazon S3 Tables**
* [Onboarding via Data Portal](./s3tables/onboarding-data-portal) — wizard-based setup for Amazon S3 Tables
* [Onboarding via API](./s3tables/onboarding-api) — 4-step REST flow with bash examples and a quickstart script
**Unity Catalog**
* [Onboarding via Data Portal](./unity/onboarding-data-portal) — wizard-based setup for Databricks-managed and OSS Unity Catalog
* [Onboarding via API](./unity/onboarding-api) — 4-step REST flow; supports Databricks-managed and OSS Unity Catalog, including Delta Lake tables with UniForm enabled
**Apache Polaris & Snowflake Open Catalog**
* [Onboarding via Data Portal](./polaris/onboarding-data-portal) — wizard-based setup with OAuth2 principal credentials
* [Onboarding via API](./polaris/onboarding-api) — 4-step REST flow using the generic Iceberg REST adapter with OAuth2 principal credentials
**Snowflake Horizon Catalog**
* [Onboarding via Data Portal](./horizon/onboarding-data-portal) — wizard-based setup with a Snowflake programmatic access token (PAT)
* [Onboarding via API](./horizon/onboarding-api) — 4-step REST flow authenticating with a Snowflake programmatic access token (PAT)
**Nessie**
* [Onboarding via API](./nessie/onboarding-api) — 4-step REST flow using the generic Iceberg REST adapter with a branch/ref `prefix`
* Onboarding via Data Portal is not yet available — the API flow is the only option today
**S3 Data Lake**
* [Onboarding via Data Portal](./s3/onboarding-data-portal) — wizard-based setup for raw Parquet on S3
* [Onboarding via API](./s3/onboarding-api) — 4-step REST flow with bash examples and a quickstart script
**GCS Data Lake**
* [Onboarding via Data Portal](./gcs/onboarding-data-portal) — wizard-based setup for Parquet files in GCS
* [Onboarding via API](./gcs/onboarding-api) — 4-step REST flow using GCS S3 interop with HMAC keys
**After onboarding**
* **Monitoring sync progress** → [Observability](./observability) — sync status, checkpoint watermark, and source file count APIs
* **Queries are slow** → [Indexes](./indexes) to add the right indexes, then [Best Practices & Configs](./best-practices-and-configs) for caching and tuning
* **Something is broken** → [Troubleshooting](./troubleshooting) for symptom-based fixes, or [FAQ](./faq) for common questions
**New in 0.16.0**
* **[SQL DDL](./sql-ddl)** — create an External Table with a single `CREATE TABLE ... WITH (...)` statement instead of the 4-step REST flow.
* **[Large External Tables](./segment-groups)** — segment groups track a batch of small member segments as one unit, keeping cluster metadata manageable for tables backed by millions of source files.
* **[Deletion Vectors](./deletion-vectors)** (preview) — apply Iceberg row-level deletes/updates at query time instead of requiring append-only sources.
***
## Page map
| Page | What it covers |
| - | - |
| [Query Parquet Directly](./getting-started/query-without-a-catalog) | Task-based entry point for raw Parquet on S3/GCS with no catalog |
| [Connect a Catalog](./getting-started/connect-a-catalog) | Task-based entry point for Glue, S3 Tables, Unity, Polaris/Snowflake, and Nessie |
| [Speed Up Queries](./getting-started/speed-up-queries) | Task-based entry point for indexes, caching, and OOM protection |
| [AWS Glue: Onboarding via Data Portal](./glue/onboarding-data-portal) | Point-and-click wizard for AWS Glue |
| [AWS Glue: Onboarding via API](./glue/onboarding-api) | REST API 4-step flow for AWS Glue (Iceberg REST) |
| [Amazon S3 Tables: Onboarding via Data Portal](./s3tables/onboarding-data-portal) | Point-and-click wizard for Amazon S3 Tables |
| [Amazon S3 Tables: Onboarding via API](./s3tables/onboarding-api) | REST API 4-step flow for Amazon S3 Tables (Iceberg REST) |
| [Unity Catalog: Onboarding via Data Portal](./unity/onboarding-data-portal) | Point-and-click wizard for Unity Catalog, including Delta+UniForm tables |
| [Unity Catalog: Onboarding via API](./unity/onboarding-api) | REST API 4-step flow for Unity Catalog (Iceberg REST), including Delta+UniForm tables |
| [Apache Polaris & Snowflake Open Catalog: Onboarding via Data Portal](./polaris/onboarding-data-portal) | Point-and-click wizard for Apache Polaris and Snowflake Open Catalog |
| [Apache Polaris & Snowflake Open Catalog: Onboarding via API](./polaris/onboarding-api) | REST API 4-step flow with OAuth2 principal credentials (generic `rest` adapter) |
| [Snowflake Horizon Catalog: Onboarding via Data Portal](./horizon/onboarding-data-portal) | Point-and-click wizard for Snowflake-managed Iceberg tables |
| [Snowflake Horizon Catalog: Onboarding via API](./horizon/onboarding-api) | REST API 4-step flow with a Snowflake programmatic access token (PAT) |
| [Nessie: Onboarding via API](./nessie/onboarding-api) | REST API 4-step flow for Nessie (Iceberg REST, generic `rest` adapter with a branch/ref `prefix`) |
| [S3: Onboarding via Data Portal](./s3/onboarding-data-portal) | Point-and-click wizard for raw Parquet on S3 |
| [S3: Onboarding via API](./s3/onboarding-api) | REST API 4-step flow for S3 Data Lake |
| [GCS Data Lake: Onboarding via Data Portal](./gcs/onboarding-data-portal) | Point-and-click wizard for Parquet in GCS via S3 interop |
| [GCS Data Lake: Onboarding via API](./gcs/onboarding-api) | REST API 4-step flow using GCS HMAC keys and S3 interop |
| [Observability](./observability) | Sync run status, ingestion checkpoint, source file count, and manual trigger APIs |
| [Data Type Mapping](./data-type-mapping) | Parquet → Pinot and Iceberg → Pinot type mapping tables, plus time column detection |
| [Indexes](./indexes) | Supported indexes, why columns are RAW, and per-index config examples |
| [Data and Index Caching](./data-and-index-caching) | Three caches (data, index, footer), eviction, restart behavior, and how to clear them |
| [Best Practices & Configs](./best-practices-and-configs) | Full config reference: sync task, tier backend, server/cluster, query options, and OOM protection |
| [FAQ](./faq) | Common questions by category: general, onboarding, schema, indexes, performance, operations |
| [DataExportTask](./data-export-task) | Export completed REALTIME segments to Parquet on filesystem or Iceberg REST |
| [Troubleshooting](./troubleshooting) | Symptom-based diagnostic playbook with exact error strings and escalation guidance |
| [SQL DDL](./sql-ddl) | Create an External Table with `CREATE TABLE ... WITH (...)` instead of the REST flow |
| [Large External Tables](./segment-groups) | Segment groups: keep cluster metadata manageable for tables with millions of files |
| [Deletion Vectors](./deletion-vectors) | Apply Iceberg row-level deletes/updates at query time (preview) |
# Apache Polaris & Snowflake Open Catalog: Onboarding via API
Source: https://docs.startree.ai/corecapabilities/external-table/polaris/onboarding-api
Onboard an Apache Polaris or Snowflake Open Catalog Iceberg table as a StarTree External Table over the Iceberg REST API, authenticating with OAuth2 principal (client-id/client-secret) credentials.
This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it.
This page shows how to onboard an **Apache Polaris** or **Snowflake Open Catalog** External Table with the StarTree controller REST APIs.
Prefer a point-and-click setup? The Data Portal wizard has dedicated **Apache Polaris** and **Snowflake Open Catalog** tiles — see [Onboarding via Data Portal](./onboarding-data-portal).
Polaris tables are accessed over the standard **Iceberg REST protocol**. There's no dedicated `serviceType` for Polaris — use the generic spec-compliant adapter: `catalogType=iceberg-rest`, `serviceType=rest`, with `catalog.iceberg-rest.warehouse` set to the Polaris **catalog name**. Data files must be **Parquet**.
> Using **Nessie**? It rides the same generic `rest` adapter with a branch/ref `prefix` — see [Nessie: Onboarding via API](../nessie/onboarding-api).
## Apache Polaris and Snowflake Open Catalog
**Snowflake Open Catalog** is Snowflake's managed Apache Polaris service. It speaks the same Iceberg REST protocol and uses the same **principal**-based OAuth2 (`client-id` / `client-secret`, `PRINCIPAL_ROLE:ALL` scope) as self-managed Polaris — only the host and token-endpoint path differ. Self-managed Polaris serves the REST API under `/api/catalog`; Open Catalog serves it under `https://-.snowflakecomputing.com/polaris/api/catalog`. Everything else on this page — principal credentials, scope, catalog-name `warehouse` — applies to both.
Using **Snowflake Horizon Catalog** instead? Snowflake has stopped provisioning new Open Catalog accounts and now routes customers to **Horizon**, which exposes the same Polaris REST API on the Snowflake account host but authenticates differently (a service-user **PAT** passed as the catalog credential, a `session:role:` scope, and a required `prefix`). It has its own page: [Snowflake Horizon Catalog: Onboarding via API](../horizon/onboarding-api).
## How it works
Onboarding is four calls. Each one feeds the next:
```text theme={null}
1. Validate & browse POST /connections/browse → validate; list namespaces/tables
2. Preview POST /tables/preview → infer schema + enriched table config
3. Create schema POST /schemas → register the schema
4. Create table POST /tables → register the External Table
```
The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output.
Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding).
All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` (the StarTree Cloud data-plane proxy). A browse call then posts to `$BASE_URL/connections/browse`.
If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity.
## Prerequisites
* StarTree 0.16.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure.
* Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth.
* A running Polaris deployment with its Iceberg REST API reachable: self-managed Apache Polaris serves it under `/api/catalog`; Snowflake Open Catalog serves it under `https://-.snowflakecomputing.com/polaris/api/catalog`.
* A Polaris **principal** with client credentials (`client-id` / `client-secret`) whose principal role maps, through a catalog role, to at least read grants on the target catalog (for example `TABLE_READ_DATA`, which implies the list/metadata grants needed for browsing). See the [Polaris access-control docs](https://polaris.apache.org/) for creating principals, principal roles, catalog roles, and grants.
* The Polaris **catalog name** (used as the `warehouse`) and the namespace/table to onboard.
* Read access to the underlying object storage. This connector supports **S3-compatible storage** for the data files.
## Limitations
* **No credential vending.** Polaris can vend short-lived storage credentials to engines that request them, but the generic `rest` adapter does not consume vended credentials — configure static keys, an assumed role, or the cluster's node role for storage explicitly.
## Authentication
An External Table authenticates in two places, configured independently:
* **Catalog (REST)** — keys under `catalog.iceberg-rest.auth.rest.*`. Polaris always requires authentication; choose one of the methods below.
* **Storage (S3 data files)** — keys under `catalog.iceberg-rest.auth.storage.*`. Choose one of the three methods below.
### Catalog (REST) authentication: choose one
#### OAuth2 client credentials (recommended)
The standard Polaris method: the principal's `client-id` / `client-secret` are exchanged at the Polaris token endpoint for a short-lived token, which StarTree caches and refreshes automatically. `authType` is optional; it's auto-detected as `oauth2` once the three required keys below are all present.
Polaris requires a scope naming the principal role to activate — use `PRINCIPAL_ROLE:ALL` to activate all of the principal's roles, or `PRINCIPAL_ROLE:` to restrict the session to one role.
```json theme={null}
{
"catalog.iceberg-rest.auth.rest.authType": "oauth2",
"catalog.iceberg-rest.auth.rest.oauthTokenUri": "https:///api/catalog/v1/oauth/tokens",
"catalog.iceberg-rest.auth.rest.oauthClientId": "",
"catalog.iceberg-rest.auth.rest.oauthClientSecret": "",
"catalog.iceberg-rest.auth.rest.oauthScopes": "PRINCIPAL_ROLE:ALL"
}
```
For **Snowflake Open Catalog**, the token endpoint is the same path on the account host: `https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens`.
#### OAuth2 against an external identity provider
Polaris deployments configured to trust an external IdP (Keycloak, Okta, and so on) accept tokens minted by that IdP instead of the built-in token endpoint. Use the same four keys, pointing `oauthTokenUri` at the IdP's client-credentials token endpoint and setting whatever scopes the IdP requires.
```json theme={null}
{
"catalog.iceberg-rest.auth.rest.authType": "oauth2",
"catalog.iceberg-rest.auth.rest.oauthTokenUri": "https:///realms//protocol/openid-connect/token",
"catalog.iceberg-rest.auth.rest.oauthClientId": "",
"catalog.iceberg-rest.auth.rest.oauthClientSecret": "",
"catalog.iceberg-rest.auth.rest.oauthScopes": ""
}
```
#### Bearer token
If you already hold a valid token (for example, minted out-of-band from your IdP), pass it directly. `authType` doesn't need to be set explicitly — it's auto-detected as token auth when `.token` is present. Note that a static token is **not refreshed**: when it expires, syncs start failing until the table config is updated, so prefer the OAuth2 client-credentials flow for anything long-running.
```json theme={null}
{
"catalog.iceberg-rest.auth.rest.token": ""
}
```
### Storage authentication: choose one
#### Assumed IAM role (recommended for production)
No static secrets in the table config — every S3 read assumes `roleArn` via STS. Three things to know: `auth.storage.region` is **required** whenever a `roleArn` is set (the sync fails fast without it); any static `auth.storage` access keys are **ignored** once a `roleArn` is present; and the `AssumeRole` call itself authenticates with the cluster's ambient AWS identity (node role / IRSA), so the target role's trust policy must trust that identity (plus the `externalId`, if you set one).
```json theme={null}
{
"catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader",
"catalog.iceberg-rest.auth.storage.externalId": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
```
#### Cluster node role
Omit every `auth.storage.*` credential key — only `region` is required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have S3 access.
```json theme={null}
{
"catalog.iceberg-rest.auth.storage.region": ""
}
```
#### Static access keys
Quick tests, or when role-based access isn't available. Use the canonical `accessKeyId` / `secretAccessKey` spellings (an optional `sessionToken` is also accepted). Avoid the shorter `accessKey`/`secretKey` aliases: only part of the pipeline honors them (segment generation does, Iceberg manifest and data-file reads do not), so a short-alias config falls back to the cluster's ambient credentials for catalog file reads and can fail in confusing ways.
```json theme={null}
{
"catalog.iceberg-rest.auth.storage.accessKeyId": "",
"catalog.iceberg-rest.auth.storage.secretAccessKey": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
```
### Selecting the catalog
A Polaris server hosts multiple catalogs. Set `catalog.iceberg-rest.warehouse` to the catalog name — the adapter passes it to the server's config endpoint, which resolves the per-catalog prefix automatically.
```json theme={null}
{
"catalog.iceberg-rest.warehouse": "",
"catalog.iceberg-rest.prefix": ""
}
```
Also set `catalog.iceberg-rest.prefix` to the catalog name whenever the prefix can't be negotiated automatically. This is **required for Snowflake Open Catalog** and any deployment whose `/v1/config` endpoint gates on the `warehouse` query parameter, or that sits behind a proxy which doesn't forward the config handshake. Without it, browse fails with `404 Unable to find matching target resource method`, because the request falls back to the un-prefixed `/v1/namespaces` path. The examples below set both `warehouse` and `prefix`.
***
## Step 1: Validate and browse the connection
**`POST /connections/browse`**
There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your credentials, scope, and connectivity.
* Set `path` to `""` to browse the **root** — this both validates the connection and lists namespaces.
* Set `path` to a namespace name to list the **tables** inside it.
### Request
| Field | Type | Required | Description |
| - | - | - | - |
| `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. |
| `connection.params` | object | Yes | Catalog-specific connection settings (see example below). |
| `path` | string | No | Where to browse. `""` or omitted = root. A namespace name = that namespace's tables. |
```bash theme={null}
curl -X POST "$BASE_URL/connections/browse" \
-H "Content-Type: application/json" \
-d '{
"connection": {
"type": "CATALOG",
"params": {
"catalogType": "iceberg-rest",
"catalog.iceberg-rest.serviceType": "rest",
"catalog.iceberg-rest.restUri": "https:///api/catalog",
"catalog.iceberg-rest.warehouse": "",
"catalog.iceberg-rest.prefix": "",
"catalog.iceberg-rest.auth.rest.authType": "oauth2",
"catalog.iceberg-rest.auth.rest.oauthTokenUri": "https:///api/catalog/v1/oauth/tokens",
"catalog.iceberg-rest.auth.rest.oauthClientId": "",
"catalog.iceberg-rest.auth.rest.oauthClientSecret": "",
"catalog.iceberg-rest.auth.rest.oauthScopes": "PRINCIPAL_ROLE:ALL",
"catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader",
"catalog.iceberg-rest.auth.storage.externalId": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
},
"path": ""
}'
```
| Key | Value | Description |
| - | - | - |
| `catalog.iceberg-rest.restUri` | `https:///api/catalog` | The Polaris Iceberg REST base URI (Snowflake Open Catalog: `https://-.snowflakecomputing.com/polaris/api/catalog`). |
| `catalog.iceberg-rest.serviceType` | `rest` | Selects the generic spec-compliant adapter — there's no Polaris-specific `serviceType`. |
| `catalog.iceberg-rest.warehouse` | `` | The Polaris catalog to browse; the server resolves the per-catalog prefix from it. |
| `catalog.iceberg-rest.prefix` | `` | Pins the catalog prefix explicitly. Required for Snowflake Open Catalog and any proxy that doesn't forward the config handshake. |
### Response
```json theme={null}
{
"items": [
{ "name": "transportation", "type": "NAMESPACE" },
{ "name": "nyc_taxi_trips", "type": "TABLE" }
]
}
```
| `items[].type` | Meaning |
| - | - |
| `NAMESPACE` | A namespace you can drill into — pass its `name` as `path`. |
| `TABLE` | A selectable table. |
***
## Step 2: Preview the schema
**`POST /tables/preview`**
Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4.
The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`.
### Request
**Top-level fields:**
| Field | Type | Required | Description |
| - | - | - | - |
| `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. |
| `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. |
| `schema` | object | No | An explicit schema to use instead of inferring one. |
**`config.inference` — how the schema is derived:**
| Field | Type | Default | Description |
| - | - | - | - |
| `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Iceberg metadata. **Set this to `true` for External Tables.** |
| `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. |
Resolution order: explicit `schema` → embedded schema → inferred schema.
**`config.sampling` — how rows are sampled:**
| Field | Type | Default | Description |
| - | - | - | - |
| `min` | integer | `1` | Minimum records to sample. |
| `max` | integer | `100` | Maximum records to sample. |
```bash theme={null}
curl -X POST "$BASE_URL/tables/preview" \
-H "Content-Type: application/json" \
-d '{
"tableConfig": {
"tableName": "nyc_taxi_trips_OFFLINE",
"tableType": "OFFLINE",
"task": {
"taskTypeConfigsMap": {
"ExternalTableSyncTask": {
"catalogType": "iceberg-rest",
"executor": "controller",
"inputFormat": "parquet",
"catalog.iceberg-rest.serviceType": "rest",
"catalog.iceberg-rest.restUri": "https:///api/catalog",
"catalog.iceberg-rest.warehouse": "",
"catalog.iceberg-rest.prefix": "",
"catalog.iceberg-rest.table.namespace": "transportation",
"catalog.iceberg-rest.table.tableName": "nyc_taxi_trips",
"catalog.iceberg-rest.auth.rest.authType": "oauth2",
"catalog.iceberg-rest.auth.rest.oauthTokenUri": "https:///api/catalog/v1/oauth/tokens",
"catalog.iceberg-rest.auth.rest.oauthClientId": "",
"catalog.iceberg-rest.auth.rest.oauthClientSecret": "",
"catalog.iceberg-rest.auth.rest.oauthScopes": "PRINCIPAL_ROLE:ALL",
"catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader",
"catalog.iceberg-rest.auth.storage.externalId": "",
"catalog.iceberg-rest.auth.storage.region": ""
}
}
}
},
"config": { "inference": { "embeddedSchemaEnabled": true } }
}'
```
**Setting the namespace and table.** Take them from the [browse](#step-1-validate-and-browse-the-connection) response — set `catalog.iceberg-rest.table.namespace` / `.tableName` to the `NAMESPACE` and `TABLE` you selected.
**Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4.
### Response
| Field | Type | Description |
| - | - | - |
| `schema` | object | The resolved Pinot schema. **Use this in step 3.** |
| `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. |
| `rows` | array | Sample records after schema + ingestion transforms. |
| `sourceRows` | array | Raw records before transformation. |
| `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). |
Adjust the schema (time column, column names, null handling) before moving on.
***
## Step 3: Create the schema
**`POST /schemas`**
Send the `schema` object from the preview response. Rename its `schemaName` to match your table.
```bash theme={null}
curl -X POST "$BASE_URL/schemas" \
-H "Content-Type: application/json" \
-d @schema.json
```
```json theme={null}
{ "status": "nyc_taxi_trips successfully added" }
```
***
## Step 4: Create the table
**`POST /tables`**
Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external.
```bash theme={null}
curl -X POST "$BASE_URL/tables" \
-H "Content-Type: application/json" \
-d @tableConfig.json
```
```json theme={null}
{ "status": "Table nyc_taxi_trips_OFFLINE successfully added" }
```
The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call.
The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run:
```bash theme={null}
curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE"
```
***
## Quickstart: onboard a table end-to-end
The whole flow as one script. Fill in the variables at the top, run the script, and the first sync starts automatically.
```bash theme={null}
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="https://dp..cp..startree.cloud/api/pinot"
AUTH="Authorization: Bearer "
REST_URI="https:///api/catalog"
TOKEN_URI="https:///api/catalog/v1/oauth/tokens"
WAREHOUSE=""
PREFIX=""
CLIENT_ID=""
CLIENT_SECRET=""
SCOPES="PRINCIPAL_ROLE:ALL"
NAMESPACE="transportation"
TABLE="nyc_taxi_trips"
STORAGE_ROLE_ARN="arn:aws:iam::123456789012:role/pinot-external-table-reader"
STORAGE_EXTERNAL_ID=""
STORAGE_REGION=""
# 1. Validate & browse root — lists namespaces in the Polaris catalog.
curl -sf -X POST "$BASE_URL/connections/browse" \
-H "$AUTH" -H 'Content-Type: application/json' \
--data-raw "{
\"connection\": {
\"type\": \"CATALOG\",
\"params\": {
\"catalogType\": \"iceberg-rest\",
\"catalog.iceberg-rest.serviceType\": \"rest\",
\"catalog.iceberg-rest.restUri\": \"$REST_URI\",
\"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\",
\"catalog.iceberg-rest.prefix\": \"$PREFIX\",
\"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\",
\"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\",
\"catalog.iceberg-rest.auth.rest.oauthClientId\": \"$CLIENT_ID\",
\"catalog.iceberg-rest.auth.rest.oauthClientSecret\": \"$CLIENT_SECRET\",
\"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\"
}
},
\"path\": \"\"
}" | jq .
# 2. Preview — infer schema + enriched table config.
curl -sf -X POST "$BASE_URL/tables/preview" \
-H "$AUTH" -H 'Content-Type: application/json' \
--data-raw "{
\"tableConfig\": {
\"tableName\": \"${TABLE}_OFFLINE\",
\"tableType\": \"OFFLINE\",
\"task\": {
\"taskTypeConfigsMap\": {
\"ExternalTableSyncTask\": {
\"catalogType\": \"iceberg-rest\",
\"executor\": \"controller\",
\"inputFormat\": \"parquet\",
\"catalog.iceberg-rest.serviceType\": \"rest\",
\"catalog.iceberg-rest.restUri\": \"$REST_URI\",
\"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\",
\"catalog.iceberg-rest.prefix\": \"$PREFIX\",
\"catalog.iceberg-rest.table.namespace\": \"$NAMESPACE\",
\"catalog.iceberg-rest.table.tableName\": \"$TABLE\",
\"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\",
\"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\",
\"catalog.iceberg-rest.auth.rest.oauthClientId\": \"$CLIENT_ID\",
\"catalog.iceberg-rest.auth.rest.oauthClientSecret\": \"$CLIENT_SECRET\",
\"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\",
\"catalog.iceberg-rest.auth.storage.roleArn\": \"$STORAGE_ROLE_ARN\",
\"catalog.iceberg-rest.auth.storage.externalId\": \"$STORAGE_EXTERNAL_ID\",
\"catalog.iceberg-rest.auth.storage.region\": \"$STORAGE_REGION\"
}
}
}
},
\"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } }
}" > preview.json
# 3. Extract schema and table config from preview response.
jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json
jq '.tableConfigs.offline' preview.json > tableConfig.json
# 4. Create the schema.
curl -sf -X POST "$BASE_URL/schemas" \
-H "$AUTH" -H 'Content-Type: application/json' -d @schema.json
# 5. Create the table — watcher starts the first sync automatically.
curl -sf -X POST "$BASE_URL/tables" \
-H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json
# 6. (optional) Poll until sync completes.
curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \
| jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered'
```
The script uses an assumed IAM role for storage (`auth.storage.roleArn`); swap in static keys or the node-role default (`region` only) from [Authentication](#authentication) if that fits your setup better.
Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable).
## Monitor onboarding
The [status endpoint](../observability#run-status) reports each sync run's progress, and `?includeLag=true` adds how far the table trails the upstream catalog. Both require `executor=controller` (set automatically by this flow). See [Observability](../observability) for full request and response details.
## Verify it's queryable
When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live:
```sql theme={null}
SELECT count(*) FROM nyc_taxi_trips;
```
The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting).
***
## What's next
Now that the table is created and data is loading, these are the highest-impact follow-up steps:
1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes)
2. **Enable caching and preload.** Set `enable.prefetch.page.cache=true` and `preload.enable=true` on the S3 tier so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching)
3. **Protect large-scan queries from OOM.** For tables that receive heavy aggregations or wide scans, enable the query OOM killer so a runaway query is killed instead of crashing the server. → [Best Practices & Configs — Query OOM protection](../best-practices-and-configs#query-oom-protection-large-scans)
4. **Monitor ongoing syncs.** Use the status endpoint (with `includeLag=true` for ingestion lag) after each scheduled sync. → [Observability](../observability)
***
For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting).
# Apache Polaris & Snowflake Open Catalog: Onboarding via Data Portal
Source: https://docs.startree.ai/corecapabilities/external-table/polaris/onboarding-data-portal
Connect an Apache Polaris or Snowflake Open Catalog Iceberg table to StarTree as an External Table with the point-and-click Data Portal wizard, authenticating with OAuth2 principal credentials.
This feature is available starting in **StarTree release 0.16.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment.
This guide walks through connecting an **Apache Polaris** or **Snowflake Open Catalog** table to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface.
Both catalogs speak the standard **Iceberg REST protocol** and authenticate with **OAuth2 principal credentials** (client ID + client secret). The wizard has a dedicated tile for each; the two forms are identical apart from defaults, so this page covers both. Data files must be **Parquet**.
> Looking for the API-based approach? See [Apache Polaris & Snowflake Open Catalog: Onboarding via API](./onboarding-api).
>
> Using **Snowflake Horizon Catalog** (Snowflake-managed Iceberg tables, PAT auth) instead? See [Snowflake Horizon: Onboarding via Data Portal](../horizon/onboarding-data-portal).
***
## Prerequisites
Before starting, ensure you have:
* **StarTree 0.16.0 or later** with the External Table feature enabled and **tiered storage configured** for your environment. Contact StarTree support if unsure.
* A running catalog:
* **Apache Polaris** — a reachable Polaris server and a principal (client ID + client secret) whose principal role can read the catalog.
* **Snowflake Open Catalog** — an Open Catalog account and a service connection's principal credentials.
* The **catalog name** (Polaris/Open Catalog catalog backing the Iceberg tables) and the namespace/table you want to onboard.
* Read access to the underlying S3 object storage — static keys or an IAM role StarTree can assume. See [Onboarding via API — Authentication](./onboarding-api#authentication) for the permissions required.
***
## Step 1: Open the External Tables
1. Log in to **Data Portal**.
2. In the left navigation, go to **Tables**.
3. Click **+ Connect External Table**.
The wizard opens with a connection configuration screen.
***
## Step 2: Configure the Catalog Connection
Choose **Apache Polaris** or **Snowflake Open Catalog** as the catalog type — they appear as top-level tiles alongside AWS Glue, Amazon S3 Tables, and Unity Catalog.
Fill in the connection details:
**Details**
| Field | Description |
| - | - |
| **Connection Name** | A unique name to identify this connection in Data Portal. |
**Metastore** — how StarTree authenticates with the Iceberg REST API.
| Field | Description |
| - | - |
| **Catalog Endpoint URI** | The Iceberg REST endpoint — `https:///api/catalog` for Apache Polaris, or `https://.snowflakecomputing.com/polaris/api/catalog` for Snowflake Open Catalog. |
| **Catalog Name** | The Polaris / Open Catalog **catalog** backing the Iceberg tables. |
| **Prefix** | *(Snowflake Open Catalog only)* Multi-catalog REST prefix; pre-filled from the Catalog Name — leave the default unless your deployment uses a different prefix. |
| **REST Authentication Type** | **OAuth** — the only option for these catalogs. |
| **OAuth Token URI** | The catalog's OAuth2 token endpoint, e.g. `.../api/catalog/v1/oauth/tokens` (Polaris) or `https://.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens` (Open Catalog). |
| **OAuth Client ID** | The principal's client ID. |
| **OAuth Client Secret** | The principal's client secret. |
| **OAuth Scopes** | The principal-role scope, default `PRINCIPAL_ROLE:ALL`. Narrow it to a specific principal role if your security model requires it. |
**Storage** — credentials for reading the underlying Parquet data files from S3.
| Field | Description |
| - | - |
| **Region** | AWS region where the S3 data files are stored. Always required. |
| **Authentication Type** | **Basic Authentication** (Access Key + Secret Key), **IAM Node Role** (the cluster node's own IAM identity — Data Portal shows the resolved role as a read-only **IAM Role Name** field so you can grant it access in your AWS account), or **IAM Role** (a cross-account role StarTree assumes). |
| **Access Key** / **Secret Key** | With **Basic Authentication**: static AWS credentials for S3 data access. |
| **IAM Role ARN** | With **IAM Role**: the ARN of the role to assume. The form shows **Steps to create IAM Policy** — a ready-made trust policy including the StarTree account and External ID to configure on the role. |
Click **Validate Connection**. Data Portal calls the catalog's validate endpoint and confirms credentials and connectivity before proceeding.
***
## Step 3: Browse and Select a Table
Once the connection is validated:
1. Data Portal lists the available **namespaces** — the field is labeled **Namespace** (Apache Polaris) or **Schema** (Snowflake Open Catalog).
2. Select one to expand its tables.
3. Click the **table** you want to onboard.
Data Portal reads the Iceberg schema and derives a Pinot schema automatically.
***
## Step 4: Review the Schema
The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity.
Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct.
***
## Step 5: Configure the Table
Set the final table options:
| Setting | Default | Notes |
| - | - | - |
| **Table Name** | — | Name of the Pinot table to create. |
| **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. |
| **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. |
The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config.
Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required.
***
## Step 6: Monitor Onboarding
The table detail view shows sync status in real time — files discovered, segments generated, last synced snapshot, and ingestion lag. Use **Sync Now** to trigger an immediate sync, or **Pause Sync** / **Resume Sync** from the table menu (while the sync task is paused, the primary button itself reads **Resume Sync**). For API-based monitoring, see [Observability](../observability).
# S3: Onboarding via API
Source: https://docs.startree.ai/corecapabilities/external-table/s3/onboarding-api
Onboard raw Parquet files under an S3 prefix as a StarTree External Table via the REST API — no Iceberg or catalog service required.
This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
This page shows how to onboard an S3 Data Lake External Table with the StarTree controller REST APIs instead of the [Data Portal UI](./onboarding-data-portal). Use it for automation, infrastructure-as-code, or onboarding many tables at once.
## How it works
Onboarding is four calls. Each one feeds the next:
```text theme={null}
1. Validate & browse POST /connections/browse → validate; list files under the S3 prefix
2. Preview POST /tables/preview → infer schema + enriched table config
3. Create schema POST /schemas → register the schema
4. Create table POST /tables → register the External Table
```
The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output.
Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding).
All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` — the StarTree Cloud data-plane proxy that fronts the controller.
If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity.
## Prerequisites
* StarTree 0.15.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure.
* Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth.
* AWS credentials (access key + secret key) with `s3:GetObject` and `s3:ListBucket` on the source bucket and prefix.
* The S3 bucket name, key prefix, and region.
**Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly.
If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details.
## Limitations
* **No catalog metadata** — namespace and table names are values you choose, not read from a catalog; browse only validates the connection and lists files/directories.
* **No schema evolution.** Raw Parquet has no Iceberg field-id history to key a rename against, so a source-side column rename always requires a manual schema change (unlike `catalogType=iceberg-rest` sources with `schemaEvolution.enabled=true`).
* **`catalog.s3.prefix` is a permanent filter, not a starting cursor** — see the Warning further down this page, under the `catalog.s3.*` reference table, before using it to try to skip ahead in a large bucket.
## Authentication
Unlike the Iceberg-catalog sources, there's only one auth surface here — `catalog.s3.*` covers both listing the bucket and reading the Parquet data. Choose one of three credential methods.
The role or keys need `s3:GetObject` and `s3:ListBucket` on the source bucket and prefix. Verify from the cluster before onboarding:
```bash theme={null}
aws s3 ls s3:////
```
### Assumed IAM role (recommended for production)
No static secrets in the table config — every S3 call assumes `roleArn` via STS.
```json theme={null}
{
"catalog.s3.bucketName": "",
"catalog.s3.prefix": "",
"catalog.s3.region": "",
"catalog.s3.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader",
"catalog.s3.externalId": ""
}
```
### Cluster node role
Omit `roleArn`/`accessKey`/`secretKey` entirely — only `bucketName`, `prefix`, and `region` are required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have access to the bucket.
```json theme={null}
{
"catalog.s3.bucketName": "",
"catalog.s3.prefix": "