# Get progress and estimated completion time for the task Source: https://docs.startree.ai/api-reference/altertable/get-progress-and-estimated-completion-time-for-the-task get /alterTable/{tableNameWithType}/estimatedTime # Get appconfigs Source: https://docs.startree.ai/api-reference/appconfigs/get-appconfigs get /appconfigs # Get all application qps quotas Source: https://docs.startree.ai/api-reference/application/get-all-application-qps-quotas get /applicationQuotas Get all application qps quotas # Get application qps quota Source: https://docs.startree.ai/api-reference/application/get-application-qps-quota get /applicationQuotas/{appName} Get application qps quota # Update application quota Source: https://docs.startree.ai/api-reference/application/update-application-quota post /applicationQuotas/{appName} Update application quota # Delete the minion task metadata Source: https://docs.startree.ai/api-reference/atomicingestion/delete-the-minion-task-metadata post /segments/{tableName}/deleteMinionTaskMetadata # Get Segment to rows factor from customMap of progress metadata Source: https://docs.startree.ai/api-reference/atomicingestion/get-segment-to-rows-factor-from-custommap-of-progress-metadata post /segments/{tableName}/getSegmentToRowsFactor # Mark the end of data ingestion to upload multiple segments Source: https://docs.startree.ai/api-reference/atomicingestion/mark-the-end-of-data-ingestion-to-upload-multiple-segments post /segments/{tableName}/endDataIngestRequest # Mark the start of data ingestion to upload multiple segments Source: https://docs.startree.ai/api-reference/atomicingestion/mark-the-start-of-data-ingestion-to-upload-multiple-segments post /segments/{tableName}/startDataIngestRequest # Check whether authentication is enabled Source: https://docs.startree.ai/api-reference/auth/check-whether-authentication-is-enabled get /auth/verify # Retrieve auth workflow info Source: https://docs.startree.ai/api-reference/auth/retrieve-auth-workflow-info get /auth/info # Get the list of servers to restart in sequence Source: https://docs.startree.ai/api-reference/batchrestart/get-the-list-of-servers-to-restart-in-sequence get /serverBatchRestartSequence # Enable/disable the query rate limiting for a broker instance Source: https://docs.startree.ai/api-reference/broker/enabledisable-the-query-rate-limiting-for-a-broker-instance post /brokers/instances/{instanceName}/qps Enable/disable the query rate limiting for a broker instance # List brokers for a given table Source: https://docs.startree.ai/api-reference/broker/list-brokers-for-a-given-table get /brokers/tables/{tableName} List brokers for a given table # List brokers for a given table Source: https://docs.startree.ai/api-reference/broker/list-brokers-for-a-given-table-1 get /v2/brokers/tables/{tableName} List brokers for a given table # List brokers for a given tenant Source: https://docs.startree.ai/api-reference/broker/list-brokers-for-a-given-tenant get /brokers/tenants/{tenantName} List brokers for a given tenant # List brokers for a given tenant Source: https://docs.startree.ai/api-reference/broker/list-brokers-for-a-given-tenant-1 get /v2/brokers/tenants/{tenantName} List brokers for a given tenant # List tables to brokers mappings Source: https://docs.startree.ai/api-reference/broker/list-tables-to-brokers-mappings get /brokers/tables List tables to brokers mappings # List tables to brokers mappings Source: https://docs.startree.ai/api-reference/broker/list-tables-to-brokers-mappings-1 get /v2/brokers/tables List tables to brokers mappings # List tenants and tables to brokers mappings Source: https://docs.startree.ai/api-reference/broker/list-tenants-and-tables-to-brokers-mappings get /brokers List tenants and tables to brokers mappings # List tenants and tables to brokers mappings Source: https://docs.startree.ai/api-reference/broker/list-tenants-and-tables-to-brokers-mappings-1 get /v2/brokers List tenants and tables to brokers mappings # List tenants to brokers mappings Source: https://docs.startree.ai/api-reference/broker/list-tenants-to-brokers-mappings get /brokers/tenants List tenants to brokers mappings # List tenants to brokers mappings Source: https://docs.startree.ai/api-reference/broker/list-tenants-to-brokers-mappings-1 get /v2/brokers/tenants List tenants to brokers mappings # Delete cluster configuration Source: https://docs.startree.ai/api-reference/cluster/delete-cluster-configuration delete /cluster/configs/{configName} # Get cluster Info Source: https://docs.startree.ai/api-reference/cluster/get-cluster-info get /cluster/info Get cluster Info # Get debug information for segment. Source: https://docs.startree.ai/api-reference/cluster/get-debug-information-for-segment get /debug/segments/{tableName}/{segmentName} Debug information for segment. # Get debug information for table. Source: https://docs.startree.ai/api-reference/cluster/get-debug-information-for-table get /debug/tables/{tableName} Debug information for table. # Get the configuration for Groovy Static analysis Source: https://docs.startree.ai/api-reference/cluster/get-the-configuration-for-groovy-static-analysis get /cluster/configs/groovy/staticAnalyzerConfig Get the configuration for Groovy static analysis # Get the default configuration for Groovy Static analysis Source: https://docs.startree.ai/api-reference/cluster/get-the-default-configuration-for-groovy-static-analysis get /cluster/configs/groovy/staticAnalyzerConfig/default Get the default configuration for Groovy static analysis # List cluster configurations Source: https://docs.startree.ai/api-reference/cluster/list-cluster-configurations get /cluster/configs List cluster level configurations # Update cluster configuration Source: https://docs.startree.ai/api-reference/cluster/update-cluster-configuration post /cluster/configs # Update Groovy static analysis configuration Source: https://docs.startree.ai/api-reference/cluster/update-groovy-static-analysis-configuration post /cluster/configs/groovy/staticAnalyzerConfig # Enable / disable the periodic cluster health check task. Note that this setting isn't persisted across controller restarts and /cluster/configs should be used to disable it permanently Source: https://docs.startree.ai/api-reference/clusterhealth/enable-disable-the-periodic-cluster-health-check-task-note-that-this-setting-isnt-persisted-across-controller-restarts-and-clusterconfigs-should-be-used-to-disable-it-permanently put /clusterHealth # Get all available cluster health checks and their details Source: https://docs.startree.ai/api-reference/clusterhealth/get-all-available-cluster-health-checks-and-their-details get /clusterHealth/list # Get cached cluster health details for all pinot entities Source: https://docs.startree.ai/api-reference/clusterhealth/get-cached-cluster-health-details-for-all-pinot-entities get /clusterHealth # Get cluster properties if deployed in AZ aware mode Source: https://docs.startree.ai/api-reference/clusterhealth/get-cluster-properties-if-deployed-in-az-aware-mode get /azAwareCluster # browse Source: https://docs.startree.ai/api-reference/connection/browse post /connections/browse Browse resources in a connection. Eg list directories/files in a s3 bucket, list tables in a database, list topics in a Kafka cluster etc. # Update consistentPush state Source: https://docs.startree.ai/api-reference/consistentpush/update-consistentpush-state post /segments/{tableNameWithType}/consistentPushRequest # Delete all tables in given database name Source: https://docs.startree.ai/api-reference/database/delete-all-tables-in-given-database-name delete /databases/{databaseName} Delete all tables in given database name # Get database quota configs Source: https://docs.startree.ai/api-reference/database/get-database-quota-configs get /databases/{databaseName}/quotas Get database quota configs # List all database names Source: https://docs.startree.ai/api-reference/database/list-all-database-names get /databases Lists all database names # Update database quotas Source: https://docs.startree.ai/api-reference/database/update-database-quotas post /databases/{databaseName}/quotas Update database quotas # Query Data Source: https://docs.startree.ai/api-reference/dataplane/query POST /query/sql Run SQL queries on StarTree Cloud's real-time analytics engine. # Add metadata location of a new dedup snapshot Source: https://docs.startree.ai/api-reference/dedupsnapshot/add-metadata-location-of-a-new-dedup-snapshot post /dedupSnapshots/{tableNameWithType}/{snapshotName}/location # Add metadata of a new dedup snapshot Source: https://docs.startree.ai/api-reference/dedupsnapshot/add-metadata-of-a-new-dedup-snapshot post /dedupSnapshots/{tableNameWithType} # Delete dedup snapshot and its metadata Source: https://docs.startree.ai/api-reference/dedupsnapshot/delete-dedup-snapshot-and-its-metadata delete /dedupSnapshots/{tableNameWithType}/snapshotAndMetadata # Get all dedup snapshot names for the table Source: https://docs.startree.ai/api-reference/dedupsnapshot/get-all-dedup-snapshot-names-for-the-table get /dedupSnapshots/{tableNameWithType}/names # Get dedup snapshot metadata by the snapshot name Source: https://docs.startree.ai/api-reference/dedupsnapshot/get-dedup-snapshot-metadata-by-the-snapshot-name get /dedupSnapshots/{tableNameWithType}/{snapshotName}/metadata # Get latest dedup snapshot names for all partitions Source: https://docs.startree.ai/api-reference/dedupsnapshot/get-latest-dedup-snapshot-names-for-all-partitions get /dedupSnapshots/{tableNameWithType}/latest # Check controller health Source: https://docs.startree.ai/api-reference/health/check-controller-health get /pinot-controller/admin # Check controller health Source: https://docs.startree.ai/api-reference/health/check-controller-health-1 get /health # Get controller start time Source: https://docs.startree.ai/api-reference/health/get-controller-start-time get /start-time # Get controller uptime Source: https://docs.startree.ai/api-reference/health/get-controller-uptime get /uptime # Check if it's safe to drop the given instances. If not list all the reasons why its not safe. Source: https://docs.startree.ai/api-reference/instance/check-if-its-safe-to-drop-the-given-instances-if-not-list-all-the-reasons-why-its-not-safe get /instances/dropInstance/validate # Check if it's safe to update the tags of the given instances. If not list all the reasons. Source: https://docs.startree.ai/api-reference/instance/check-if-its-safe-to-update-the-tags-of-the-given-instances-if-not-list-all-the-reasons post /instances/updateTags/validate # Create a new instance Source: https://docs.startree.ai/api-reference/instance/create-a-new-instance post /instances Creates a new instance with given instance config # Drop an instance Source: https://docs.startree.ai/api-reference/instance/drop-an-instance delete /instances/{instanceName} Drop an instance # Enable/disable an instance Source: https://docs.startree.ai/api-reference/instance/enabledisable-an-instance put /instances/{instanceName}/state Enable/disable an instance # Enable/disable/drop an instance Source: https://docs.startree.ai/api-reference/instance/enabledisabledrop-an-instance post /instances/{instanceName}/state Enable/disable/drop an instance # Get instance information Source: https://docs.startree.ai/api-reference/instance/get-instance-information get /instances/{instanceName} # List all instances Source: https://docs.startree.ai/api-reference/instance/list-all-instances get /instances # List all live instances Source: https://docs.startree.ai/api-reference/instance/list-all-live-instances get /liveinstances # Update the specified instance Source: https://docs.startree.ai/api-reference/instance/update-the-specified-instance put /instances/{instanceName} Update specified instance with given instance config # Update the tables served by the specified broker instance in the broker resource Source: https://docs.startree.ai/api-reference/instance/update-the-tables-served-by-the-specified-broker-instance-in-the-broker-resource post /instances/{instanceName}/updateBrokerResource Broker resource should be updated when a new broker instance is added, or the tags for an existing broker are changed. Updating broker resource requires reading all the table configs, which can be costly for large cluster. Consider updating broker resource for each table individually. # Update the tags of the specified instance Source: https://docs.startree.ai/api-reference/instance/update-the-tags-of-the-specified-instance put /instances/{instanceName}/updateTags Update the tags of the specified instance # Introduction Source: https://docs.startree.ai/api-reference/introduction Learn about StarTree Cloud APIs for managing tables and querying data at scale. # StarTree Cloud APIs StarTree Cloud provides **powerful APIs**, enabling users to **manage tables, schemas, and query data efficiently at scale**. > **[Pinot Controller APIs](/api-reference/pinot-controller/introduction)** > > Create and manage tables, schemas, segments, tenants, and tasks. > **[StarTree Extension APIs](/api-reference/startree-extensions/introduction)** > > Controller endpoints only StarTree Cloud has: atomic ingestion, restream, snapshots, and more. > **[Broker APIs](/api-reference/dataplane/query)** > > Query real-time data at scale with low latency and high concurrency. > **[RBAC Manager API](/api-reference/rbac/introduction)** > > Manage policies, roles, role assignments, and service tokens. *** ## **Authentication & Prerequisites** To interact with StarTree Cloud APIs, follow these steps: **1. Log in to StarTree Cloud** Access the [StarTree Cloud Console](https://startree.ai/saas-signup) and log in to your account. **2. Generate a Bearer Token** Navigate to the **API Access** section in the console and generate your **Bearer Token**.\ This token must be included in every API request for authentication. **3. Retrieve Your Workspace ID** The **Workspace ID** can be found in the **top navigation bar** of the console.\ This ID must be passed in the `database` HTTP header for all API calls. > [!NOTE] > > Workspace ID is required **only for multi-tenant clusters** (e.g., Free Tier).\ > It is **not needed for dedicated clusters** (Standard or Premium plans). **4. Start Calling APIs** Explore the API reference in the following pages to learn how to use the Bearer Token and Workspace ID in API requests. # Gets leader for a given table Source: https://docs.startree.ai/api-reference/leader/gets-leader-for-a-given-table get /leader/tables/{tableName} Gets leader for a given table # Gets leaders for all tables Source: https://docs.startree.ai/api-reference/leader/gets-leaders-for-all-tables get /leader/tables Gets leaders for all tables # Collect log files from a given instance Source: https://docs.startree.ai/api-reference/logger/collect-log-files-from-a-given-instance get /loggers/instances/{instanceName} # Collect log files from all the instances Source: https://docs.startree.ai/api-reference/logger/collect-log-files-from-all-the-instances get /loggers/instances # Download a log file Source: https://docs.startree.ai/api-reference/logger/download-a-log-file get /loggers/download # Download a log file from a given instance Source: https://docs.startree.ai/api-reference/logger/download-a-log-file-from-a-given-instance get /loggers/instances/{instanceName}/download # Get all local log files Source: https://docs.startree.ai/api-reference/logger/get-all-local-log-files get /loggers/files # Get all the loggers Source: https://docs.startree.ai/api-reference/logger/get-all-the-loggers get /loggers Return all the logger names # Get logger configs Source: https://docs.startree.ai/api-reference/logger/get-logger-configs get /loggers/{loggerName} Return logger info # Set logger level Source: https://docs.startree.ai/api-reference/logger/set-logger-level put /loggers/{loggerName} Set logger level for a given logger # Get perf advisor recommendations given a query Source: https://docs.startree.ai/api-reference/perfadvisor/get-perf-advisor-recommendations-given-a-query get /perfadvisor/advise # Get comma-delimited list of all available periodic task names. Source: https://docs.startree.ai/api-reference/periodictask/get-comma-delimited-list-of-all-available-periodic-task-names get /periodictask/names # Run periodic task against table. If table name is missing, task will run against all tables. Source: https://docs.startree.ai/api-reference/periodictask/run-periodic-task-against-table-if-table-name-is-missing-task-will-run-against-all-tables get /periodictask/run # Pinot Controller APIs Source: https://docs.startree.ai/api-reference/pinot-controller/introduction Reference for the Apache Pinot controller REST API as served by StarTree Cloud: tables, schemas, segments, tenants, instances, tasks, and cluster operations. The controller is the management plane of a Pinot cluster. Use these endpoints to create and change tables and schemas, upload and inspect segments, manage tenants and instances, schedule minion tasks, and read cluster state. Every call goes to the controller URL of your StarTree Cloud environment with an `Authorization: Bearer ` header. On multi-tenant clusters, also pass your workspace ID in the `database` header. See the [API introduction](/api-reference/introduction) for how to get both. The groups below follow the controller's own tags. The most used ones: > **[Table](/api-reference/table/lists-all-tables-in-cluster)** > > Create, update, validate, rebalance, pause, and delete tables. > **[Schema](/api-reference/schema/list-all-schema-names)** > > Add and update schemas, and validate them before use. > **[Segment](/api-reference/segment/get-the-metadata-for-a-segment)** > > Upload, reload, delete, and inspect segments and their metadata. > **[Task](/api-reference/task/fetch-cron-scheduler-job-keys)** > > Schedule and monitor minion tasks and their cron jobs. StarTree adds its own controller endpoints on top of the Pinot ones, for example atomic ingestion, restream, and upsert snapshots. Those are listed separately under [StarTree Extension APIs](/api-reference/startree-extensions/introduction). # Cancel a query as identified by the clientQueryId Source: https://docs.startree.ai/api-reference/query/cancel-a-query-as-identified-by-the-clientqueryid delete /clientQuery/{brokerId}/{clientQueryId} No effect if no query exists for the given clientQueryId on the requested broker. Query may continue to run for a short while after callingcancel as it's done in a non-blocking manner. The cancel method can be called multiple times. # Cancel a query as identified by the clientQueryId Source: https://docs.startree.ai/api-reference/query/cancel-a-query-as-identified-by-the-clientqueryid-1 delete /clientQuery/{clientQueryId} No effect if no query exists forthe given clientQueryId on any broker. Query may continue to run for a short while after callingcancel as it's done in a non-blocking manner. The cancel method can be called multiple times. # Cancel a query as identified by the queryId Source: https://docs.startree.ai/api-reference/query/cancel-a-query-as-identified-by-the-queryid delete /query/{brokerId}/{queryId} No effect if no query exists for the given queryId on the requested broker. Query may continue to run for a short while after calling cancel as it's done in a non-blocking manner. The cancel method can be called multiple times. # Get running queries from all brokers Source: https://docs.startree.ai/api-reference/query/get-running-queries-from-all-brokers get /queries The queries are returned with brokers running them # Update rate limiter configs Source: https://docs.startree.ai/api-reference/ratelimiter/update-rate-limiter-configs post /upsertRateLimitConfigs Update all the rate limiter configs # Assign a role to a subject Source: https://docs.startree.ai/api-reference/rbac/assign-a-role-to-a-subject api-reference/rbac/openapi.yaml POST /roles/{roleSrn}/create-assignment Grants the role to a user, an identity-provider group, an email domain, or a service token. Assignments cannot be edited: to move a subject to a different role, delete the assignment and create a new one. # Attach a policy to a role Source: https://docs.startree.ai/api-reference/rbac/attach-a-policy-to-a-role api-reference/rbac/openapi.yaml POST /roles/{roleSrn}/attach-policy Links a policy to a role. Call once per policy to attach several policies to the same role. # Create a policy Source: https://docs.startree.ai/api-reference/rbac/create-a-policy api-reference/rbac/openapi.yaml POST /policies Creates a policy. The policy document is passed as a JSON string in `policyDocumentJson`. See [Custom Policy Configuration](/corecapabilities/security/custom-policies) for the document format, the list of actions, and SRN syntax. # Create a role Source: https://docs.startree.ai/api-reference/rbac/create-a-role api-reference/rbac/openapi.yaml POST /roles Creates an empty role. Attach policies with `POST /roles/{roleSrn}/attach-policy`. # Create a service token Source: https://docs.startree.ai/api-reference/rbac/create-a-service-token api-reference/rbac/openapi.yaml POST /service-tokens Creates a service token and returns its access key, secret key, and the full bearer token (`st--`). Save the secret key and bearer token from this response: they cannot be retrieved again. A new token has no permissions. Assign it a role with `POST /roles/{roleSrn}/create-assignment`, using `subjectType: service-token` and the access key as the `subject`. # Delete a policy Source: https://docs.startree.ai/api-reference/rbac/delete-a-policy api-reference/rbac/openapi.yaml DELETE /policies/{policySrn} Deletes a policy. Fails if the policy is still attached to any role; detach it first. # Delete a role Source: https://docs.startree.ai/api-reference/rbac/delete-a-role api-reference/rbac/openapi.yaml DELETE /roles/{roleSrn} Deletes a role. Fails if any subject is still assigned to the role; remove the assignments first. # Delete a service token Source: https://docs.startree.ai/api-reference/rbac/delete-a-service-token api-reference/rbac/openapi.yaml DELETE /service-tokens/{tokenSrn} Revokes the token permanently. To disable a token temporarily, remove its role assignments instead. # Detach a policy from a role Source: https://docs.startree.ai/api-reference/rbac/detach-a-policy-from-a-role api-reference/rbac/openapi.yaml POST /roles/{roleSrn}/detach-policy Removes the link between a policy and a role. The policy itself is not deleted. # Explain a role Source: https://docs.startree.ai/api-reference/rbac/explain-a-role api-reference/rbac/openapi.yaml GET /roles/{roleSrn}/explain Returns the full policy, including the policy document, for every policy attached to the role. Saves a `GET /policies/{policySrn}` call per policy. # Get a policy Source: https://docs.startree.ai/api-reference/rbac/get-a-policy api-reference/rbac/openapi.yaml GET /policies/{policySrn} Returns the full policy, including its policy document and audit fields. # Get a role Source: https://docs.startree.ai/api-reference/rbac/get-a-role api-reference/rbac/openapi.yaml GET /roles/{roleSrn} Returns a role and its audit fields. Use `/roles/{roleSrn}/policies` or `/roles/{roleSrn}/explain` to see what it grants. # Get a service token Source: https://docs.startree.ai/api-reference/rbac/get-a-service-token api-reference/rbac/openapi.yaml GET /service-tokens/{tokenSrn} Returns the token's description, access key, and audit fields. The secret key is not included. # RBAC Manager API Source: https://docs.startree.ai/api-reference/rbac/introduction Reference for the StarTree RBAC Manager API: policies, roles, role assignments, service tokens, and access simulation. The RBAC Manager API is the programmatic way to do everything the Security Manager UI does: define policies, group them into roles, assign roles to users, groups, domains, and service tokens, and check what a subject is allowed to do. The base URL is `https://rbac-manager..startree.cloud/api/v1/rbac-manager`, where `` is your environment. Every call needs an `Authorization: Bearer ` header. See [Generating an API Token](/corecapabilities/security/manage-api-tokens#generating-an-api-token). Resources are addressed by StarTree Resource Name (SRN), for example `srn2:policy#my-policy` or `srn2:role#table-reader`. In a URL path, encode the `#` as `%23`. > **[Using the RBAC API](/corecapabilities/security/using-rbac-api)** > > Step-by-step walkthrough: create a policy, create a role, attach the policy, assign the role. > **[Custom policies](/corecapabilities/security/custom-policies)** > > Policy document format, SRN syntax, and example policies for common roles. # List all role assignments Source: https://docs.startree.ai/api-reference/rbac/list-all-role-assignments api-reference/rbac/openapi.yaml GET /assignments Returns every subject-to-role assignment in the environment, across all roles. # List assignments for a role Source: https://docs.startree.ai/api-reference/rbac/list-assignments-for-a-role api-reference/rbac/openapi.yaml GET /roles/{roleSrn}/assignments Returns the subjects assigned to the role. # List policies Source: https://docs.startree.ai/api-reference/rbac/list-policies api-reference/rbac/openapi.yaml GET /policies Returns the name and SRN of every policy in the environment, including the predefined StarTree policies. # List policies attached to a role Source: https://docs.startree.ai/api-reference/rbac/list-policies-attached-to-a-role api-reference/rbac/openapi.yaml GET /roles/{roleSrn}/policies Returns one attachment record per policy attached to the role. # List roles Source: https://docs.startree.ai/api-reference/rbac/list-roles api-reference/rbac/openapi.yaml GET /roles Returns the name and SRN of every role in the environment, including the predefined StarTree roles. # List service tokens Source: https://docs.startree.ai/api-reference/rbac/list-service-tokens api-reference/rbac/openapi.yaml GET /service-tokens Returns a summary of each service token. Secret keys are never returned after creation. # Remove a role assignment Source: https://docs.startree.ai/api-reference/rbac/remove-a-role-assignment api-reference/rbac/openapi.yaml POST /roles/{roleSrn}/delete-assignment Revokes the role from a subject. Pass the same `subject` and `subjectType` that were used to create the assignment. # Simulate access Source: https://docs.startree.ai/api-reference/rbac/simulate-access api-reference/rbac/openapi.yaml POST /simulate-access Evaluates whether a subject would be allowed to perform an action on a resource, taking every role assigned to the subject into account. Pass `policyDocumentJson` to test a policy document that is not saved yet. Policy changes can take up to an hour to propagate, so test here before saving. # Update a policy Source: https://docs.startree.ai/api-reference/rbac/update-a-policy api-reference/rbac/openapi.yaml PUT /policies/{policySrn} Replaces the description and policy document of an existing policy. The name and SRN cannot change. Changes can take up to an hour to propagate; use [Simulate access](#tag/Access-simulation) to test a document before saving it. # Update a role Source: https://docs.startree.ai/api-reference/rbac/update-a-role api-reference/rbac/openapi.yaml PUT /roles/{roleSrn} Updates the description of a role. The name and SRN cannot change. # Cancel a restream operation Source: https://docs.startree.ai/api-reference/restream/cancel-a-restream-operation delete /tables/{tableNameWithType}/restream Cancel a restream operation # DEBUG ONLY: Process the next stage of the restream workflow. Source: https://docs.startree.ai/api-reference/restream/debug-only:-process-the-next-stage-of-the-restream-workflow put /tables/{tableNameWithType}/restream DEBUG ONLY: Process the next stage of the restream workflow. # Gets detailed stats of a restream operation Source: https://docs.startree.ai/api-reference/restream/gets-detailed-stats-of-a-restream-operation get /tables/{tableNameWithType}/restream Gets detailed stats of a restream operation # Restream data from a stream to a new table. Then replace the table Source: https://docs.startree.ai/api-reference/restream/restream-data-from-a-stream-to-a-new-table-then-replace-the-table post /tables/{tableName}/restream Restream a table # Add a new schema Source: https://docs.startree.ai/api-reference/schema/add-a-new-schema post /schemas Adds a new schema # Delete a schema Source: https://docs.startree.ai/api-reference/schema/delete-a-schema delete /schemas/{schemaName} Deletes a schema by name # Get a schema Source: https://docs.startree.ai/api-reference/schema/get-a-schema get /schemas/{schemaName} Gets a schema by name # Get fieldSpec metadata Source: https://docs.startree.ai/api-reference/schema/get-fieldspec-metadata get /schemas/fieldSpec Get fieldSpec metadata # Get table schema Source: https://docs.startree.ai/api-reference/schema/get-table-schema get /tables/{tableName}/schema Read table schema # List all schema names Source: https://docs.startree.ai/api-reference/schema/list-all-schema-names get /schemas Lists all schema names # List all schemas info with count of field specs Source: https://docs.startree.ai/api-reference/schema/list-all-schemas-info-with-count-of-field-specs get /schemas/info Lists all schemas with field count details # Update a schema Source: https://docs.startree.ai/api-reference/schema/update-a-schema put /schemas/{schemaName} Updates a schema # Validate schema Source: https://docs.startree.ai/api-reference/schema/validate-schema post /schemas/validate This API returns the schema that matches the one you get from 'GET /schema/{schemaName}'. This allows us to validate schema before apply. # Delete a segment Source: https://docs.startree.ai/api-reference/segment/delete-a-segment delete /segments/{tableName}/{segmentName} Delete a segment # Delete selected segments. An optional 'excludeReplacedSegments' parameter is used to get the list of segments which has not yet been replaced (determined by segment lineage entries) and can be queried from the table. The value is false by default. Source: https://docs.startree.ai/api-reference/segment/delete-selected-segments-an-optional-excludereplacedsegments-parameter-is-used-to-get-the-list-of-segments-which-has-not-yet-been-replaced-determined-by-segment-lineage-entries-and-can-be-queried-from-the-table-the-value-is-false-by-default delete /segments/{tableName}/choose List all segments # Delete the list of segments provided in the queryParam else all segments Source: https://docs.startree.ai/api-reference/segment/delete-the-list-of-segments-provided-in-the-queryparam-else-all-segments delete /segments/{tableName} Delete the list of segments provided in the queryParam else all segments # Delete the segments in the JSON array payload Source: https://docs.startree.ai/api-reference/segment/delete-the-segments-in-the-json-array-payload post /segments/{tableName}/delete Delete the segments in the JSON array payload # Download a segment Source: https://docs.startree.ai/api-reference/segment/download-a-segment get /segments/{tableName}/{segmentName} Download a segment # End to replace segments Source: https://docs.startree.ai/api-reference/segment/end-to-replace-segments post /segments/{tableName}/endReplaceSegments End to replace segments # Get a map from segment to CRC of the segment (only apply to OFFLINE table) Source: https://docs.startree.ai/api-reference/segment/get-a-map-from-segment-to-crc-of-the-segment-only-apply-to-offline-table get /segments/{tableName}/crc Get a map from segment to CRC of the segment (only apply to OFFLINE table) # Get a map from server to segments hosted by the server Source: https://docs.startree.ai/api-reference/segment/get-a-map-from-server-to-segments-hosted-by-the-server get /segments/{tableName}/servers Get a map from server to segments hosted by the server # Get selected segments timestamps Source: https://docs.startree.ai/api-reference/segment/get-selected-segments-timestamps get /segments/{tableName}/select Get the selected segments given the (inclusive) start and (exclusive) end timestamps in milliseconds. These timestamps will be compared against minmax values of the time column in each segment. If the table is a refresh use case, the value of start and end timestamp is voided, since there is no time column for refresh use case; instead, the whole qualified segments will be returned. If no timestamps are provided, all the qualified segments will be returned. For the segments that partially belong to the time range, the boolean flag 'excludeOverlapping' is introduced in order for user to determine whether to exclude this kind of segments in the response. # Get status for a submitted reload operation Source: https://docs.startree.ai/api-reference/segment/get-status-for-a-submitted-reload-operation get /segments/segmentReloadStatus/{jobId} Get status for a submitted reload operation # Get storage tier for all segments in the given table Source: https://docs.startree.ai/api-reference/segment/get-storage-tier-for-all-segments-in-the-given-table get /segments/{tableName}/tiers Get storage tier for all segments in the given table # Get storage tiers for the given segment Source: https://docs.startree.ai/api-reference/segment/get-storage-tiers-for-the-given-segment get /segments/{tableName}/{segmentName}/tiers Get storage tiers for the given segment # Get the metadata for a segment Source: https://docs.startree.ai/api-reference/segment/get-the-metadata-for-a-segment get /segments/{tableName}/{segmentName}/metadata Get the metadata for a segment # Get the server metadata for all table segments Source: https://docs.startree.ai/api-reference/segment/get-the-server-metadata-for-all-table-segments get /segments/{tableName}/metadata Get the server metadata for all table segments # Get the zookeeper metadata for all table segments Source: https://docs.startree.ai/api-reference/segment/get-the-zookeeper-metadata-for-all-table-segments get /segments/{tableName}/zkmetadata Get the zookeeper metadata for all table segments # Gets a list of segments that are stale from servers hosting the table Source: https://docs.startree.ai/api-reference/segment/gets-a-list-of-segments-that-are-stale-from-servers-hosting-the-table get /segments/{tableNameWithType}/isStale Gets a list of segments that are stale from servers hosting the table # Gets the metadata of reload segments check from servers hosting the table Source: https://docs.startree.ai/api-reference/segment/gets-the-metadata-of-reload-segments-check-from-servers-hosting-the-table get /segments/{tableNameWithType}/needReload Returns true if reload is needed on the table from any one of the servers # List all segments. An optional 'excludeReplacedSegments' parameter is used to get the list of segments which has not yet been replaced (determined by segment lineage entries) and can be queried from the table. The value is false by default. Source: https://docs.startree.ai/api-reference/segment/list-all-segments-an-optional-excludereplacedsegments-parameter-is-used-to-get-the-list-of-segments-which-has-not-yet-been-replaced-determined-by-segment-lineage-entries-and-can-be-queried-from-the-table-the-value-is-false-by-default get /segments/{tableName} List all segments # List segment lineage Source: https://docs.startree.ai/api-reference/segment/list-segment-lineage get /segments/{tableName}/lineage List segment lineage in chronologically sorted order # Reingest a realtime segment Source: https://docs.startree.ai/api-reference/segment/reingest-a-realtime-segment post /segments/reingested Reingest a segment as multipart file # Reload a segment Source: https://docs.startree.ai/api-reference/segment/reload-a-segment post /segments/{tableName}/{segmentName}/reload Reload a segment # Reload all segments Source: https://docs.startree.ai/api-reference/segment/reload-all-segments post /segments/{tableName}/reload Reload all segments # Resets a segment by first disabling it, waiting for external view to stabilize, and finally enabling it again Source: https://docs.startree.ai/api-reference/segment/resets-a-segment-by-first-disabling-it-waiting-for-external-view-to-stabilize-and-finally-enabling-it-again post /segments/{tableNameWithType}/{segmentName}/reset Resets a segment by disabling and then enabling it # Resets all segments (when errorSegmentsOnly = false) or segments with Error state (when errorSegmentsOnly = true) of the table, by first disabling them, waiting for external view to stabilize, and finally enabling them Source: https://docs.startree.ai/api-reference/segment/resets-all-segments-when-errorsegmentsonly-=-false-or-segments-with-error-state-when-errorsegmentsonly-=-true-of-the-table-by-first-disabling-them-waiting-for-external-view-to-stabilize-and-finally-enabling-them post /segments/{tableNameWithType}/reset Resets segments by disabling and then enabling them # Revert segments replacement Source: https://docs.startree.ai/api-reference/segment/revert-segments-replacement post /segments/{tableName}/revertReplaceSegments Revert segments replacement # Start to replace segments Source: https://docs.startree.ai/api-reference/segment/start-to-replace-segments post /segments/{tableName}/startReplaceSegments Start to replace segments # Update the start and end time of the segments based on latest schema Source: https://docs.startree.ai/api-reference/segment/update-the-start-and-end-time-of-the-segments-based-on-latest-schema post /segments/{tableNameWithType}/updateZKTimeInterval Update the start and end time of the segments based on latest schema # Upload a batch of segments Source: https://docs.startree.ai/api-reference/segment/upload-a-batch-of-segments post /segments/batchUpload Upload a batch of segments with METADATA upload type # Upload a segment Source: https://docs.startree.ai/api-reference/segment/upload-a-segment post /segments Upload a segment as binary # Upload a segment Source: https://docs.startree.ai/api-reference/segment/upload-a-segment-1 post /v2/segments Upload a segment as binary # StarTree Extension APIs Source: https://docs.startree.ai/api-reference/startree-extensions/introduction Reference for the controller endpoints StarTree adds on top of Apache Pinot: atomic ingestion, restream, consistent push, dedup and upsert snapshots, rate limiting, tuners, and cluster health. These endpoints are served by the same controller as the [Pinot Controller APIs](/api-reference/pinot-controller/introduction), with the same base URL and headers, but they only exist in StarTree Cloud. They cover operations that open source Pinot does not expose, such as uploading several segments as one atomic batch, replacing a table from a stream without downtime, or taking snapshots of dedup and upsert metadata. > **[Atomic ingestion](/api-reference/atomicingestion/mark-the-start-of-data-ingestion-to-upload-multiple-segments)** > > Upload a batch of segments that becomes visible all at once, or not at all. > **[Restream](/api-reference/restream/restream-data-from-a-stream-to-a-new-table-then-replace-the-table)** > > Rebuild a real-time table from its stream and swap it in place. > **[Upsert snapshot](/api-reference/upsertsnapshot/add-metadata-of-a-new-upsert-snapshot)** > > Record and restore snapshots of upsert metadata. > **[Cluster health](/api-reference/clusterhealth/get-cached-cluster-health-details-for-all-pinot-entities)** > > Read cached health details for every Pinot entity in the cluster. # Add the TableConfigs using the tableConfigsStr json Source: https://docs.startree.ai/api-reference/table/add-the-tableconfigs-using-the-tableconfigsstr-json post /tableConfigs Add the TableConfigs using the tableConfigsStr json # Adds a table Source: https://docs.startree.ai/api-reference/table/adds-a-table post /tables Adds a table # Assign server instances to a table Source: https://docs.startree.ai/api-reference/table/assign-server-instances-to-a-table post /tables/{tableName}/assignInstances # Cancel all rebalance jobs for the given table, and noop if no rebalance is running Source: https://docs.startree.ai/api-reference/table/cancel-all-rebalance-jobs-for-the-given-table-and-noop-if-no-rebalance-is-running delete /tables/{tableName}/rebalance Cancel all rebalance jobs for the given table, and noop if no rebalance is running # Create/update the instance partitions Source: https://docs.startree.ai/api-reference/table/createupdate-the-instance-partitions put /tables/{tableName}/instancePartitions # Delete hybrid table query time boundary Source: https://docs.startree.ai/api-reference/table/delete-hybrid-table-query-time-boundary delete /tables/{tableName}/timeBoundary Delete hybrid table query time boundary # Delete tables partitions Source: https://docs.startree.ai/api-reference/table/delete-tables-partitions delete /tables/{tableNameWithType}/partitions # Delete the TableConfigs Source: https://docs.startree.ai/api-reference/table/delete-the-tableconfigs delete /tableConfigs/{tableName} Delete the TableConfigs # Deletes a table Source: https://docs.startree.ai/api-reference/table/deletes-a-table delete /tables/{tableName} Deletes a table # Enable/disable a table Source: https://docs.startree.ai/api-reference/table/enabledisable-a-table put /tables/{tableName}/state Enable/disable a table # Force commit the current consuming segments Source: https://docs.startree.ai/api-reference/table/force-commit-the-current-consuming-segments post /tables/{tableName}/forceCommit Force commit the current segments in consuming state and restart consumption. This should be used after schema/table config changes. Please note that this is an asynchronous operation, and 200 response does not mean it has actually been done already.If specific partitions or consuming segments are provided, only those partitions or consuming segments will be force committed. # Get current table state Source: https://docs.startree.ai/api-reference/table/get-current-table-state get /tables/{tableName}/state Get current table state # Get list of controller jobs for this table Source: https://docs.startree.ai/api-reference/table/get-list-of-controller-jobs-for-this-table get /table/{tableName}/jobs Get list of controller jobs for this table # Get segment names to segment status map Source: https://docs.startree.ai/api-reference/table/get-segment-names-to-segment-status-map get /tables/{tableName}/segmentsStatus Get segment statuses of each segment # Get status for a submitted force commit operation Source: https://docs.startree.ai/api-reference/table/get-status-for-a-submitted-force-commit-operation get /tables/forceCommitStatus/{jobId} Get status for a submitted force commit operation # Get table external view Source: https://docs.startree.ai/api-reference/table/get-table-external-view get /tables/{tableName}/externalview Get table external view # Get table ideal state Source: https://docs.startree.ai/api-reference/table/get-table-ideal-state get /tables/{tableName}/idealstate Get table ideal state # Get the aggregate index details of all segments for a table Source: https://docs.startree.ai/api-reference/table/get-the-aggregate-index-details-of-all-segments-for-a-table get /tables/{tableName}/indexes Get the aggregate index details of all segments for a table # Get the aggregate metadata of all segments for a table Source: https://docs.startree.ai/api-reference/table/get-the-aggregate-metadata-of-all-segments-for-a-table get /tables/{tableName}/metadata Get the aggregate metadata of all segments for a table # Get the aggregate validDocIds metadata of all segments for a table Source: https://docs.startree.ai/api-reference/table/get-the-aggregate-validdocids-metadata-of-all-segments-for-a-table get /tables/{tableName}/validDocIdsMetadata Get the aggregate validDocIds metadata of all segments for a table # Get the instance partitions Source: https://docs.startree.ai/api-reference/table/get-the-instance-partitions get /tables/{tableName}/instancePartitions # Get the TableConfigs for a given raw tableName Source: https://docs.startree.ai/api-reference/table/get-the-tableconfigs-for-a-given-raw-tablename get /tableConfigs/{tableName} Get the TableConfigs for a given raw tableName # Gets detailed stats of a rebalance operation Source: https://docs.startree.ai/api-reference/table/gets-detailed-stats-of-a-rebalance-operation get /rebalanceStatus/{jobId} Gets detailed stats of a rebalance operation # Ingest a file Source: https://docs.startree.ai/api-reference/table/ingest-a-file post /ingestFromFile Creates a segment using given file and pushes it to Pinot. All steps happen on the controller. This API is NOT meant for production environments/large input files. Example usage (query params need encoding): ``` curl -X POST -F file=@data.json -H "Content-Type: multipart/form-data" "http://localhost:9000/ingestFromFile?tableNameWithType=foo_OFFLINE& batchConfigMapStr={ "inputFormat":"csv", "recordReader.prop.delimiter":"|" }" ``` # Ingest from the given URI Source: https://docs.startree.ai/api-reference/table/ingest-from-the-given-uri post /ingestFromURI Creates a segment using file at the given URI and pushes it to Pinot. All steps happen on the controller. This API is NOT meant for production environments/large input files. Example usage (query params need encoding): ``` curl -X POST "http://localhost:9000/ingestFromURI?tableNameWithType=foo_OFFLINE &batchConfigMapStr={ "inputFormat":"json", "input.fs.className":"org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region":"us-central", "input.fs.prop.accessKey":"foo", "input.fs.prop.secretKey":"bar" } &sourceURIStr=s3://test.bucket/path/to/json/data/data.json" ``` # List table instances Source: https://docs.startree.ai/api-reference/table/list-table-instances get /tables/{tableName}/instances List instances of the given table # List tables to live brokers mappings Source: https://docs.startree.ai/api-reference/table/list-tables-to-live-brokers-mappings get /tables/livebrokers List tables to live brokers mappings based on EV # List the brokers serving a table Source: https://docs.startree.ai/api-reference/table/list-the-brokers-serving-a-table get /tables/{tableName}/livebrokers List live brokers of the given table based on EV # Lists all TableConfigs in cluster Source: https://docs.startree.ai/api-reference/table/lists-all-tableconfigs-in-cluster get /tableConfigs Lists all TableConfigs in cluster # Lists all tables in cluster Source: https://docs.startree.ai/api-reference/table/lists-all-tables-in-cluster get /mytables Lists all tables in cluster # Lists all tables in cluster Source: https://docs.startree.ai/api-reference/table/lists-all-tables-in-cluster-1 get /tables Lists all tables in cluster # Lists the table configs Source: https://docs.startree.ai/api-reference/table/lists-the-table-configs get /tables/{tableName} # Pause consumption of a realtime table Source: https://docs.startree.ai/api-reference/table/pause-consumption-of-a-realtime-table post /tables/{tableName}/pauseConsumption Pause the consumption of a realtime table # preview Source: https://docs.startree.ai/api-reference/table/preview post /tables/preview Simulates a preview of the table given the table config and schema # Read table sizes Source: https://docs.startree.ai/api-reference/table/read-table-sizes get /tables/{tableName}/size Get table size details. Table size is the size of untarred segments including replication # Rebalances a table (reassign instances and segments for a table) Source: https://docs.startree.ai/api-reference/table/rebalances-a-table-reassign-instances-and-segments-for-a-table post /tables/{tableName}/rebalance Rebalances a table (reassign instances and segments for a table) # Rebuild broker resource for table Source: https://docs.startree.ai/api-reference/table/rebuild-broker-resource-for-table post /tables/{tableName}/rebuildBrokerResourceFromHelixTags when new brokers are added # Recommend config Source: https://docs.startree.ai/api-reference/table/recommend-config put /tables/recommender Recommend a config with input json # Remove the instance partitions Source: https://docs.startree.ai/api-reference/table/remove-the-instance-partitions delete /tables/{tableName}/instancePartitions # Replace an instance in the instance partitions Source: https://docs.startree.ai/api-reference/table/replace-an-instance-in-the-instance-partitions post /tables/{tableName}/replaceInstance # Resume consumption of a realtime table Source: https://docs.startree.ai/api-reference/table/resume-consumption-of-a-realtime-table post /tables/{tableName}/resumeConsumption Resume the consumption for a realtime table. ConsumeFrom parameter indicates from which offsets consumption should resume. Recommended value is 'lastConsumed', which indicates consumption should continue based on the offsets in segment ZK metadata, and in case the offsets are already gone, the first available offsets are picked to minimize the data loss. # Return pause status of a realtime table Source: https://docs.startree.ai/api-reference/table/return-pause-status-of-a-realtime-table get /tables/{tableName}/pauseStatus Return pause status of a realtime table along with list of consuming segments. # Returns state of consuming segments Source: https://docs.startree.ai/api-reference/table/returns-state-of-consuming-segments get /tables/{tableName}/consumingSegmentsInfo Gets the status of consumers from all servers.Note that the partitionToOffsetMap has been deprecated and will be removed in the next release. The info is now embedded within each partition's state as currentOffsetsMap. # Returns state of pauseless table Source: https://docs.startree.ai/api-reference/table/returns-state-of-pauseless-table get /tables/{tableName}/pauselessDebugInfo Gets the segments that are in error state and segments with COMMITTING status in ZK metadata # Set hybrid table query time boundary based on offline segments' metadata Source: https://docs.startree.ai/api-reference/table/set-hybrid-table-query-time-boundary-based-on-offline-segments-metadata post /tables/{tableName}/timeBoundary Set hybrid table query time boundary based on offline segments' metadata # table stats Source: https://docs.startree.ai/api-reference/table/table-stats get /tables/{tableName}/stats Provides metadata info/stats about the table. # table status Source: https://docs.startree.ai/api-reference/table/table-status get /tables/{tableName}/status Provides status of the table including ingestion status # Update the TableConfigs provided by the tableConfigsStr json Source: https://docs.startree.ai/api-reference/table/update-the-tableconfigs-provided-by-the-tableconfigsstr-json put /tableConfigs/{tableName} Update the TableConfigs provided by the tableConfigsStr json # Updates table config for a table Source: https://docs.startree.ai/api-reference/table/updates-table-config-for-a-table put /tables/{tableName} Updates table config for a table # Validate table config for a table Source: https://docs.startree.ai/api-reference/table/validate-table-config-for-a-table post /tables/validate This API returns the table config that matches the one you get from 'GET /tables/{tableName}'. This allows us to validate table config before apply. # Validate the TableConfigs Source: https://docs.startree.ai/api-reference/table/validate-the-tableconfigs post /tableConfigs/validate Validate the TableConfigs # Clean up finished tasks (COMPLETED, FAILED) for the given task type Source: https://docs.startree.ai/api-reference/task/clean-up-finished-tasks-completed-failed-for-the-given-task-type put /tasks/{taskType}/cleanup # Count of all tasks for the given task type Source: https://docs.startree.ai/api-reference/task/count-of-all-tasks-for-the-given-task-type get /tasks/{taskType}/tasks/count # Delete a single task given its task name Source: https://docs.startree.ai/api-reference/task/delete-a-single-task-given-its-task-name delete /tasks/task/{taskName} # Delete all tasks (as well as the task queue) for the given task type Source: https://docs.startree.ai/api-reference/task/delete-all-tasks-as-well-as-the-task-queue-for-the-given-task-type delete /tasks/{taskType} # Delete task metadata for the given task type and table Source: https://docs.startree.ai/api-reference/task/delete-task-metadata-for-the-given-task-type-and-table delete /tasks/{taskType}/{tableNameWithType}/metadata # Execute a task on minion Source: https://docs.startree.ai/api-reference/task/execute-a-task-on-minion post /tasks/execute # Fetch count of sub-tasks for each of the tasks for the given task type Source: https://docs.startree.ai/api-reference/task/fetch-count-of-sub-tasks-for-each-of-the-tasks-for-the-given-task-type get /tasks/{taskType}/taskcounts # Fetch cron scheduler information Source: https://docs.startree.ai/api-reference/task/fetch-cron-scheduler-information get /tasks/scheduler/information # Fetch cron scheduler job keys Source: https://docs.startree.ai/api-reference/task/fetch-cron-scheduler-job-keys get /tasks/scheduler/jobKeys # Fetch information for all the tasks for the given task type Source: https://docs.startree.ai/api-reference/task/fetch-information-for-all-the-tasks-for-the-given-task-type get /tasks/{taskType}/debug # Fetch information for all the tasks for the given task type and table Source: https://docs.startree.ai/api-reference/task/fetch-information-for-all-the-tasks-for-the-given-task-type-and-table get /tasks/{taskType}/{tableNameWithType}/debug # Fetch information for the given task name Source: https://docs.startree.ai/api-reference/task/fetch-information-for-the-given-task-name get /tasks/task/{taskName}/debug # Fetch job details for table tasks Source: https://docs.startree.ai/api-reference/task/fetch-job-details-for-table-tasks get /tasks/scheduler/jobDetails # Fetch task generation information for the recent runs of the given task for the given table Source: https://docs.startree.ai/api-reference/task/fetch-task-generation-information-for-the-recent-runs-of-the-given-task-for-the-given-table get /tasks/generator/{tableNameWithType}/{taskType}/debug # Get a map from task to task state for the given task type Source: https://docs.startree.ai/api-reference/task/get-a-map-from-task-to-task-state-for-the-given-task-type get /tasks/{taskType}/taskstates # Get progress of all subtasks with specified state tracked by minion worker in memory Source: https://docs.startree.ai/api-reference/task/get-progress-of-all-subtasks-with-specified-state-tracked-by-minion-worker-in-memory get /tasks/subtask/workers/progress # Get progress of specified sub tasks for the given task tracked by minion worker in memory Source: https://docs.startree.ai/api-reference/task/get-progress-of-specified-sub-tasks-for-the-given-task-tracked-by-minion-worker-in-memory get /tasks/subtask/{taskName}/progress # Get task metadata for the given task type and table Source: https://docs.startree.ai/api-reference/task/get-task-metadata-for-the-given-task-type-and-table get /tasks/{taskType}/{tableNameWithType}/metadata # Get the configs of specified sub tasks for the given task Source: https://docs.startree.ai/api-reference/task/get-the-configs-of-specified-sub-tasks-for-the-given-task get /tasks/subtask/{taskName}/config # Get the state (task queue state) for the given task type Source: https://docs.startree.ai/api-reference/task/get-the-state-task-queue-state-for-the-given-task-type get /tasks/{taskType}/state # Get the states of all the sub tasks for the given task Source: https://docs.startree.ai/api-reference/task/get-the-states-of-all-the-sub-tasks-for-the-given-task get /tasks/subtask/{taskName}/state # Get the task config (a list of child task configs) for the given task Source: https://docs.startree.ai/api-reference/task/get-the-task-config-a-list-of-child-task-configs-for-the-given-task get /tasks/task/{taskName}/config # Get the task runtime config for the given task Source: https://docs.startree.ai/api-reference/task/get-the-task-runtime-config-for-the-given-task get /tasks/task/{taskName}/runtime/config # Get the task state for the given task Source: https://docs.startree.ai/api-reference/task/get-the-task-state-for-the-given-task get /tasks/task/{taskName}/state # List all task types Source: https://docs.startree.ai/api-reference/task/list-all-task-types get /tasks/tasktypes # List all tasks for the given task type Source: https://docs.startree.ai/api-reference/task/list-all-tasks-for-the-given-task-type get /tasks/{taskType}/tasks # List all tasks for the given task type Source: https://docs.startree.ai/api-reference/task/list-all-tasks-for-the-given-task-type-1 get /tasks/{taskType}/{tableNameWithType}/state # Resume all stopped tasks (as well as the task queue) for the given task type Source: https://docs.startree.ai/api-reference/task/resume-all-stopped-tasks-as-well-as-the-task-queue-for-the-given-task-type put /tasks/{taskType}/resume # Schedule tasks and return a map from task type to task name scheduled. If task type is missing, schedules all tasks. If table name is missing, schedules tasks for all tables in the database. If database is missing in headers, uses default. Source: https://docs.startree.ai/api-reference/task/schedule-tasks-and-return-a-map-from-task-type-to-task-name-scheduled-if-task-type-is-missing-schedules-all-tasks-if-table-name-is-missing-schedules-tasks-for-all-tables-in-the-database-if-database-is-missing-in-headers-uses-default post /tasks/schedule # Stop all running/pending tasks (as well as the task queue) for the given task type Source: https://docs.startree.ai/api-reference/task/stop-all-runningpending-tasks-as-well-as-the-task-queue-for-the-given-task-type put /tasks/{taskType}/stop # Change tenant state Source: https://docs.startree.ai/api-reference/tenant/change-tenant-state post /tenants/{tenantName}/metadata # Create a tenant Source: https://docs.startree.ai/api-reference/tenant/create-a-tenant post /tenants # Delete a tenant Source: https://docs.startree.ai/api-reference/tenant/delete-a-tenant delete /tenants/{tenantName} # enable/disable a tenant Source: https://docs.startree.ai/api-reference/tenant/enabledisable-a-tenant post /tenants/{tenantName} # Get tenant information Source: https://docs.startree.ai/api-reference/tenant/get-tenant-information get /tenants/{tenantName}/metadata # Get the instance partitions of a tenant Source: https://docs.startree.ai/api-reference/tenant/get-the-instance-partitions-of-a-tenant get /tenants/{tenantName}/instancePartitions # Gets detailed stats of a tenant rebalance operation Source: https://docs.startree.ai/api-reference/tenant/gets-detailed-stats-of-a-tenant-rebalance-operation get /tenants/rebalanceStatus/{jobId} Gets detailed stats of a tenant rebalance operation # List all tenants Source: https://docs.startree.ai/api-reference/tenant/list-all-tenants get /tenants # List instance for a tenant Source: https://docs.startree.ai/api-reference/tenant/list-instance-for-a-tenant get /tenants/{tenantName} # List tables on a server or broker tenant Source: https://docs.startree.ai/api-reference/tenant/list-tables-on-a-server-or-broker-tenant get /tenants/{tenantName}/tables # Rebalances all the tables that are part of the tenant Source: https://docs.startree.ai/api-reference/tenant/rebalances-all-the-tables-that-are-part-of-the-tenant post /tenants/{tenantName}/rebalance # Update a tenant Source: https://docs.startree.ai/api-reference/tenant/update-a-tenant put /tenants # Update an instance partition for a server type in a tenant Source: https://docs.startree.ai/api-reference/tenant/update-an-instance-partition-for-a-server-type-in-a-tenant put /tenants/{tenantName}/instancePartitions # Apply specific tuner to a table Source: https://docs.startree.ai/api-reference/tuner/apply-specific-tuner-to-a-table post /tuner/{tableName} # Apply tuner(s) to a table Source: https://docs.startree.ai/api-reference/tuner/apply-tuners-to-a-table get /tuner/{tableName} # Estimate memory usage for an upsert table Source: https://docs.startree.ai/api-reference/upsert/estimate-memory-usage-for-an-upsert-table post /upsert/estimateHeapUsage This API returns the estimated heap usage based on primary key column stats. This allows us to estimate table size before onboarding. # Add metadata location of a new upsert snapshot Source: https://docs.startree.ai/api-reference/upsertsnapshot/add-metadata-location-of-a-new-upsert-snapshot post /upsertSnapshots/{tableNameWithType}/{snapshotName}/location # Add metadata of a new upsert snapshot Source: https://docs.startree.ai/api-reference/upsertsnapshot/add-metadata-of-a-new-upsert-snapshot post /upsertSnapshots/{tableNameWithType} # Delete upsert snapshot and its metadata Source: https://docs.startree.ai/api-reference/upsertsnapshot/delete-upsert-snapshot-and-its-metadata delete /upsertSnapshots/{tableNameWithType}/snapshotAndMetadata # Delete upsert snapshot metadata Source: https://docs.startree.ai/api-reference/upsertsnapshot/delete-upsert-snapshot-metadata delete /upsertSnapshots/{tableNameWithType}/metadata # Get all upsert snapshot names for the table Source: https://docs.startree.ai/api-reference/upsertsnapshot/get-all-upsert-snapshot-names-for-the-table get /upsertSnapshots/{tableNameWithType}/names # Get latest upsert snapshot names for all partitions Source: https://docs.startree.ai/api-reference/upsertsnapshot/get-latest-upsert-snapshot-names-for-all-partitions get /upsertSnapshots/{tableNameWithType}/latest # Get upsert snapshot metadata by the snapshot name Source: https://docs.startree.ai/api-reference/upsertsnapshot/get-upsert-snapshot-metadata-by-the-snapshot-name get /upsertSnapshots/{tableNameWithType}/{snapshotName}/metadata # Add a user Source: https://docs.startree.ai/api-reference/user/add-a-user post /users Add a user # Delete a user Source: https://docs.startree.ai/api-reference/user/delete-a-user delete /users/{username} Delete a user # Get an user in cluster Source: https://docs.startree.ai/api-reference/user/get-an-user-in-cluster get /users/{username} Get an user in cluster # List all uses in cluster Source: https://docs.startree.ai/api-reference/user/list-all-uses-in-cluster get /users List all users in cluster # Update user config for a user Source: https://docs.startree.ai/api-reference/user/update-user-config-for-a-user put /users/{username} Update user config for user # Get version number of Pinot components Source: https://docs.startree.ai/api-reference/version/get-version-number-of-pinot-components get /version # Create a node at a given path Source: https://docs.startree.ai/api-reference/zookeeper/create-a-node-at-a-given-path post /zk/create # Delete the znode at this path Source: https://docs.startree.ai/api-reference/zookeeper/delete-the-znode-at-this-path delete /zk/delete # Get all child znodes Source: https://docs.startree.ai/api-reference/zookeeper/get-all-child-znodes get /zk/getChildren # Get content of the znode Source: https://docs.startree.ai/api-reference/zookeeper/get-content-of-the-znode get /zk/get # Get the stat Source: https://docs.startree.ai/api-reference/zookeeper/get-the-stat get /zk/stat Use this api to fetch additional details of a znode such as creation time, modified time, numChildren etc # List the child znodes Source: https://docs.startree.ai/api-reference/zookeeper/list-the-child-znodes get /zk/ls # List the child znodes along with Stats Source: https://docs.startree.ai/api-reference/zookeeper/list-the-child-znodes-along-with-stats get /zk/lsl # Update the content of multiple znRecord node under the same path Source: https://docs.startree.ai/api-reference/zookeeper/update-the-content-of-multiple-znrecord-node-under-the-same-path put /zk/putChildren # Update the content of the node Source: https://docs.startree.ai/api-reference/zookeeper/update-the-content-of-the-node put /zk/put # Overview Source: https://docs.startree.ai/concepts/about-startree-cloud StarTree Cloud delivers real-time analytics at scale and is supported on all major cloud platforms. StarTree Cloud is a fully managed analytics-as-a-service platform built on Apache Pinot™, designed to deliver real-time, high-concurrency analytics at scale. Supported on on all major cloud platforms, including AWS, GCP, and Azure, it empowers organizations to unlock the value of their data by enabling ultra-fast queries on fresh data with minimal infrastructure overhead. With built-in connectors, intuitive UI, and enterprise-grade features, StarTree Cloud makes it easy for teams to ingest, manage, and visualize operational insights — whether it’s powering user-facing analytics, anomaly detection, or internal dashboards. Businesses choose StarTree Cloud to turn their data into action, instantly. ![04232025 Observability Arch Diagram Tall V1 Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/images/04232025-observability-arch_diagram-tall-v1.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=84d16cdda07d6371be139e536c77b518) StarTree Cloud offers public SaaS (software-as-a-service) and private BYOC (bring your own cloud) deployment options. For more information about our public or private SaaS offering, [schedule a demo](https://startree.ai/demo) with our StarTree solutions architect team. ## Why use StarTree Cloud? We recommend using StarTree Cloud for the following use cases: * User facing analytics * Business metrics * Anomaly detection * Root cause analysis * Dashboards * Log analytics * Cohort analytics * Ad hoc exploration | | **Query throughput (queries/sec)** | **Query latency (p95th)** | **Consistency/accuracy** | **Query flexibility** | | - | - | - | - | - | | **User facing analytics** | Very high: 10k-100k | 10ms-100ms | Best effort | Low | | **Personalization** | Very high: 10k-100k | 10ms-100ms | Best effort | Low | | **Metrics** | High: 100s-10k | 10ms-100ms | Accurate | Low | | **Anomaly detection** | Moderate: 100s-1k | 10ms-100ms | Best effort | Low | | **Root cause analysis** | High: 100s-10k | 10ms-100ms | Best effort | Low | | **Visualization/dashboarding** | Moderate: 100s-1k | 100ms | Best effort | Medium | | **Ad hoc analytics** | Low | Single-digit seconds | Best effort | High | | **Log analytics/text search** | Moderate: 100s-1k | Sub-second | Best effort | Medium | # What is Real-Time Analytics? Source: https://docs.startree.ai/concepts/what-is-real-time-analytics Real-time analytics enable users to make better and more timely decisions. Real-time analytics products pull data in as soon as it happens, pull data out as soon as it gets pulled in, and do so at scale. Many modern software or web applications offer analytics capabilities to their users. But what does it mean for those analytics to be real-time? [YouTube video player](https://www.youtube.com/embed/1pZmafdvsmk) *Tim Berglund explains Real-Time Analytics* ## Types of analytics There are, broadly speaking, three types of analytics: #### Dashboards and BI Tools Normally used for internal purposes. BI Analysts are often the ones that are assessing this data. #### User-Facing Analytics Analytics that you provide to the end users of your software or web applications. #### Machine Learning Also referred to as machine-powered, or machine-fed analytics. These are when you feed analytics or events directly into your systems and then have your systems do the processing, automatically. Anomaly detection and fraud detection fit into this category, or any time a machine is generating insights without human interaction. ## Why is fresh data important? When you're working with very large data sets that are constantly changing, you need to make sure the analytics that you have are based on the most recent data. Data gets outdated quickly and not having the freshest data can set you back in each of the type of analytics defined above. * Dashboards need to be able to monitor the constant changes. * You can build more interactive applications on top of this fresh data with actionable triggers for your end users. Organizations like LinkedIn provide their end users with real-time data through features like \_Who viewed my profile \_where a user can message other users upon seeing that they viewed their profile. Another example is in LinkedIn's news feed where a user can react, comment, or like a post as soon as they see it. * For machine learning analytics you want your systems to know right away when something is happening, when there are changes in patterns or trends, so that you or your systems can take action. ## Building a real-time analytics system What does it take to build a real-time analytics system? There are three requirements that these systems must satisfy: ### Speed of ingestion Your application has data coming from a bunch of different sources, including batch and streaming sources. Streaming sources are built to pull new data as soon as it comes in. The data ingestion process needs to be really really fast. As soon as any changes happen, you should see this reflected in the real-time analytics system. ### Speed of queries After you pull this data in, you need to make sure that the data can be queried as quickly as possible. If you're building dashboards, providing your end users with their own analytics, or feeding your data into some kind of machine learning algorithm, you need to make sure that all the freshly ingested data can be pulled out as quickly as it was pulled in. ### Scalability Your system need to be able to handle a large number of queries run at the same time. It's not uncommon for real-time analytics systems to process 100s of thousands of queries per second. # Claude Desktop Source: https://docs.startree.ai/corecapabilities/ai/mcp/claude # Claude Desktop Configuration Configure Claude Desktop to connect with your StarTree MCP Server for natural language analytics. ## Prerequisites * Claude Desktop installed ([download here](https://claude.ai/download)) * StarTree MCP Server [installed and running](./installation) ## Configuration Steps ### 1. Locate Claude's Configuration File **macOS** ```bash theme={null} ~/Library/Application Support/Claude/claude_desktop_config.json ``` **Windows** ```bash theme={null} %APPDATA%/Claude/claude_desktop_config.json ``` ### 2. Add MCP Server Configuration Open the configuration file and add your MCP server entry: ```json theme={null} { "mcpServers": { "pinot_mcp_claude": { "command": "/path/to/uv", "args": [ "--directory", "/path/to/mcp-pinot", "run", "mcp_pinot/server.py" ], "env": {} } } } ``` Replace the paths with your actual system paths: ```bash theme={null} # Find your uv path which uv # Example: /Users/username/.local/bin/uv # Use absolute path to your mcp-pinot directory # Example: /Users/username/projects/mcp-pinot ``` ### 3. Environment Configuration (Optional) You can include environment variables directly in the configuration instead of using the `.env` file: ``` { "mcpServers": { "pinot_mcp_claude": { "command": "/Users/username/.local/bin/uv", "args": [ "--directory", "/Users/username/projects/mcp-pinot", "run", "mcp_pinot/server.py" ], "env": { "PINOT_CONTROLLER_URL": "https://pinot.example.startree.cloud", "PINOT_BROKER_HOST": "broker.pinot.example.startree.cloud", "PINOT_BROKER_PORT": "443", "PINOT_BROKER_SCHEME": "https", "PINOT_TOKEN": "Bearer your_token_here", "PINOT_DATABASE": "your_workspace_id" } } } } ``` ### 4. Restart Claude Desktop Completely restart Claude Desktop for the configuration to take effect. ### 5. Verify Connection 1. Look for the **hammer icon** 🛠️ in the Claude chat interface 2. Click the hammer to see available tools 3. Verify **Pinot tools** are listed (list-tables, read-query, etc.) ![Screenshot2025 06 02at12 03 17PM Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/images/Screenshot2025-06-02at12.03.17PM.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=bdc6adf39f818a8353060dcdd827f565) ## Start Querying You're ready to analyze your data! Try these sample prompts: ``` Can you help me analyze my data in Pinot? What tables are available? ``` ``` Can you do a histogram plot on the GitHub events against time? ``` ``` How many orders were placed in 24 hours on average? Can you help me ``` # Cursor IDE Source: https://docs.startree.ai/corecapabilities/ai/mcp/cursor # Cursor IDE Integration Configure Cursor IDE to connect with your StarTree MCP Server for natural language analytics directly within your development environment. ## Prerequisites * Cursor IDE installed ([download here](https://cursor.sh/)) * **StarTree MCP Server already installed** ([see installation guide](https://docs.startree.ai/corecapabilities/ai/mcp/installation)) ## Configuration Steps Since you already have a working `.env` file in your mcp-pinot directory, Cursor will automatically use those settings. No need to duplicate configuration! Create or edit `~/.cursor/mcp.json` ```json theme={null} { "mcpServers": { "pinot": { "command": "uv", "args": [ "--directory", "/path/to/mcp-pinot", "run", "mcp_pinot/server.py" ] } } } ``` Replace `/path/to/mcp-pinot` with the absolute path to your mcp-pinot directory Add this config in Cursor MCP global settings ![Screenshot2025 06 04at7 42 51AM Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/images/Screenshot2025-06-04at7.42.51AM.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=875c61eb96e5b3b93db34f57127a4184) ### Restart Cursor IDE Completely restart Cursor IDE for the configuration to take effect. ## Verify Connection Try asking questions related to Pinot: ![Screenshot2025 06 04at7 44 46AM Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/images/Screenshot2025-06-04at7.44.46AM.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=7b7045352a54cf86e99fe8bbb65f6cb6) # Installation Source: https://docs.startree.ai/corecapabilities/ai/mcp/installation # MCP Server Installation This guide covers installing and configuring the StarTree MCP Server for Apache Pinot & StarTree Cloud ## Prerequisites ### System Requirements * **Python 3.9+** installed on your system * **Apache Pinot cluster & StarTree Cloud** * **Git** for cloning the repository ### Install uv Package Manager [uv](https://github.com/astral-sh/uv) is a fast Python package installer and resolver, written in Rust. It's designed to be a drop-in replacement for pip with significantly better performance. ```bash theme={null} curl -LsSf https://astral.sh/uv/install.sh | sh # Reload your shell configuration source ~/.bashrc # or ~/.zshrc # Alternatively, restart your terminal # Verify installation uv --version ``` ## Installation Steps ### 1. Clone the Repository ``` git clone https://github.com/startreedata/mcp-pinot.git cd mcp-pinot ``` ### 2. Install Dependencies ```bash theme={null} # Install the MCP server uv pip install -e . # For development (includes testing tools) uv pip install -e .[dev] ``` ### 3. Configure Pinot Connection The MCP server requires a `.env` file to configure the Pinot cluster connection. ```bash theme={null} # Copy the example configuration cp .env.example .env ``` Edit the `.env` file with your Pinot cluster details: #### Local Pinot Quickstart ``` # .env file for local Pinot PINOT_CONTROLLER_URL=http://localhost:9000 PINOT_BROKER_HOST=localhost PINOT_BROKER_PORT=8000 PINOT_BROKER_SCHEME=http PINOT_USE_MSQE=true ``` #### StarTree Cloud ``` # .env file for StarTree Cloud PINOT_CONTROLLER_URL=https://pinot.celpxu.cp.s7e.startree.cloud PINOT_BROKER_HOST=broker.pinot.celpxu.cp.s7e.startree.cloud PINOT_BROKER_PORT=443 PINOT_BROKER_SCHEME=https PINOT_TOKEN=Bearer PINOT_USE_MSQE=true PINOT_DATABASE=ws_2kc8eddzzb0 (startree workspace id) ``` ### 4. Run the MCP Server ``` uv --directory . run mcp_pinot/server.py ``` You should see logs indicating that the server is running and listening on STDIO: ``` INFO: MCP server started successfully INFO: Listening on STDIO for MCP connections ``` ### 5. Verify With your MCP server running, test the connection in another terminal: ```bash theme={null} uv --directory . run tests/test_service/test_pinot_quickstart.py ``` Expected output: ``` ✓ MCP server connection successful ✓ Available tools discovered: ['list-tables', 'read-query', ...] ✓ Sample query executed successfully ✓ All tests passed ``` ## Configure MCP Clients With your MCP server installed and running, the final step is to configure your MCP client of choice to connect and start querying your data with natural language. > **[Claude Desktop](/corecapabilities/ai/mcp/claude)** > > Claude Desktop is a desktop application that interacts with LLMs and connects to the StarTree MCP server. > **[LibreChat](/corecapabilities/ai/mcp/librechat)** > > LibreChat is a self-hosted chat interface that interacts with private LLMs (including AWS Bedrock models) and connects to the StarTree MCP server for secure, on-premises AI analytics. > **[Cursor IDE](/corecapabilities/ai/mcp/cursor)** > > Cursor IDE is an AI-powered code editor that integrates MCP pinot servers with your broader development workflow # Librechat Source: https://docs.startree.ai/corecapabilities/ai/mcp/librechat # LibreChat Integration Configure LibreChat to connect with your StarTree MCP Server and AWS Bedrock for natural language analytics with powerful foundation models. ## Prerequisites * Docker and Docker Compose installed * AWS Account with Bedrock access and appropriate IAM permissions * StarTree MCP Server [installed and running](/corecapabilities/ai/mcp/installation) * Node.js 18+ (for MCP Supergateway) * LibreChat repository cloned locally ## Configuration Steps ### 1. Clone and Setup LibreChat ``` # Clone the repository git clone https://github.com/danny-avila/LibreChat.git cd LibreChat # Copy environment file cp .env.example .env ``` ### 2. Configure Environment Variables Edit the `.env` file with your AWS Bedrock credentials: ```yaml theme={null} # Basic Settings HOST=localhost PORT=3080 MONGO_URI=mongodb://mongodb:27017/LibreChat # User/Group IDs (to avoid Docker warnings) UID=1000 GID=1000 # AWS Bedrock Configuration BEDROCK_AWS_DEFAULT_REGION=us-east-1 BEDROCK_AWS_ACCESS_KEY_ID=your_access_key_id BEDROCK_AWS_SECRET_ACCESS_KEY=your_secret_access_key # Specify available Bedrock models BEDROCK_AWS_MODELS=anthropic.claude-3-5-haiku-20241022-v1:0,anthropic.claude-3-sonnet-20240229-v1:0,meta.llama3-70b-instruct-v1:0 # Application Settings APP_TITLE="LibreChat with Pinot Analytics" ALLOW_REGISTRATION=true ``` ### 3. Setup MCP Supergateway Since mcp-pinot is Python-based, we use MCP Supergateway to bridge it to LibreChat: ```yaml theme={null} # Install supergateway globally npm install -g supergateway # Start the gateway (from your mcp-pinot directory) cd /path/to/mcp-pinot supergateway --stdio "uv --directory . run mcp_pinot/server.py" --port 8110 # Alternative: Run in background using screen screen -S pinot-gateway supergateway --stdio "uv --directory . run mcp_pinot/server.py" --port 8110 # Press Ctrl+A, then D to detach ``` ### 4. Create LibreChat Configuration Create `librechat.yaml` in your LibreChat root directory: ```yaml theme={null} # LibreChat Configuration version: 1.1.5 cache: true # Interface settings interface: privacyPolicy: externalUrl: 'https://librechat.ai/privacy-policy' openNewTab: true termsOfService: externalUrl: 'https://librechat.ai/tos' openNewTab: true # Enable registration and social logins registration: socialLogins: ['github', 'google'] # AWS Bedrock endpoint configuration endpoints: bedrock: titleModel: 'anthropic.claude-3-5-haiku-20241022-v1:0' streamRate: 35 availableRegions: - "us-east-1" - "us-west-2" modelDisplayLabel: "AWS Bedrock" # MCP Server configurations mcpServers: # StarTree Pinot MCP Server via Supergateway pinot: type: sse url: "http://host.docker.internal:8110/sse" timeout: 30000 initTimeout: 10000 iconPath: "/app/client/public/assets/logo.svg" # Optional: Additional MCP servers memory: command: npx args: - -y - "@modelcontextprotocol/server-memory" timeout: 30000 filesystem: command: npx args: - -y - "@modelcontextprotocol/server-filesystem" - "/app/uploads" timeout: 30000 ``` ### 5. Create Docker Override Configuration Create `docker-compose.override.yml`: ```yaml theme={null} version: '3.4' services: api: volumes: - type: bind source: ./librechat.yaml target: /app/librechat.yaml extra_hosts: # Allow container to access host services - "host.docker.internal:host-gateway" environment: # Pass AWS credentials to container - BEDROCK_AWS_DEFAULT_REGION=${BEDROCK_AWS_DEFAULT_REGION} - BEDROCK_AWS_ACCESS_KEY_ID=${BEDROCK_AWS_ACCESS_KEY_ID} - BEDROCK_AWS_SECRET_ACCESS_KEY=${BEDROCK_AWS_SECRET_ACCESS_KEY} - BEDROCK_AWS_MODELS=${BEDROCK_AWS_MODELS} ``` ### 6. Start LibreChat ```bash theme={null} # Launch the application docker compose down # Stop any existing containers docker compose up -d # Start in detached mode # Verify startup docker compose logs -f api ``` Look for these success messages in the logs: ```bash theme={null} LibreChat | 2025-06-04 13:58:17 info: Server listening on all interfaces at port 3080. Use http://localhost:3080 to access it LibreChat | 2025-06-04 13:58:36 info: [MCP][User: 68404ef09963a7b64176c39b][pinot] Establishing new connection LibreChat | 2025-06-04 13:58:36 info: [MCP][User: 68404ef09963a7b64176c39b][pinot] Creating SSE transport: http://host.docker.internal:8110/sse LibreChat | 2025-06-04 13:58:36 info: [MCP][User: 68404ef09963a7b64176c39b][pinot] Connection successfully established ``` ## Verify Connection ### Access LibreChat Interface * Open your browser and go to [http://localhost:3080](http://localhost:3080) * Register a new account or login with existing credentials * Look for "AWS Bedrock" in the model provider dropdown * Look for pinot mcp in the chat dialog ![Screenshot2025 06 04at7 07 38AM Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/images/Screenshot2025-06-04at7.07.38AM.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=0751f008e75b71d987c7e9837463ece6) ## Start Querying You're ready to analyze your data! Try these sample prompts: ```bash theme={null} "What tables are available in my Pinot cluster?" "Show me the schema for the airlineStats table" "How many records are in the GitHub events table?" ``` ![Screenshot2025 06 04at7 09 13AM Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/images/Screenshot2025-06-04at7.09.13AM.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=f15c620c8b1c6a3c68f09576621f43a1) # Overview Source: https://docs.startree.ai/corecapabilities/ai/mcp/overview # StarTree MCP Server for Apache Pinot StarTree MCP Server represents a profound transformation in how businesses engage with their data. Move beyond traditional query consoles and complex SQL—MCP Server is the database client reinvented for the AI age. Built on the Model Context Protocol (MCP) standard pioneered by Anthropic, this server creates a standardized bridge between large language models and real-time analytical data, serving everyone—AI agents, data analysts, and business users who've never written SQL. ### Your New Query Console Traditional query consoles require complex SQL knowledge and manual data exploration. Whether you're a business user needing quick insights or an AI agent requiring contextual data access, StarTree MCP Server eliminates this friction by enabling natural language interactions with your analytical data at machine speed. ![05222025 Mcp Server Blog Diagram 1 V3 Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/images/05222025-mcp_server-blog_diagram_1-v3.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=69dac575066a22db2704e3c55a530d3b) ## Key Features | **Feature** | **Description** | | :- | :- | | **Real-time Data Freshness** | Access the most current data available with Pinot's real-time ingestion capabilities, ensuring AI agents and analysts work with up-to-the-second information. | | **Millisecond Query Performance** | Leverage Pinot's industry-leading query speed for instantaneous responses, critical for maintaining AI agent context and fluid human interactions. | | **Natural Language Interface** | Transform plain English questions directly into precise database results without writing SQL—perfect for business users and AI agents alike. | | **Schema Discovery** | Automatically identify table relationships, columns, and data types | | **Multi-language Support** | Query data in your preferred language with automatic translation | | **Metadata Integration** | Access table information, segment details, and performance metrics via API | | **Universal Client Support** | Compatible with most MCP clients including Claude Desktop, LibreChat, Cursor, Windsurf & Agents. | ## Supported Operations | Tool | Purpose | | :- | :- | | `list-tables` | Enumerate available tables | | `read-query` | Execute SELECT queries | | `table-details` | Get table metadata | | `segment-list` | List table segments | | `index-column-details` | Examine segment indexes | | `segment-metadata-details` | Get segment metadata | | `tableconfig-schema-details` | Get table configuration | # Overview Source: https://docs.startree.ai/corecapabilities/ai/overview # AI-Powered Analytics at StarTree The future of analytics is agent-facing. As artificial intelligence transforms how we interact with data, StarTree is pioneering the next generation of AI-native analytical experiences that bridge the gap between intelligent agents and real-time business insights. ## The Analytics SuperBrain Vision Traditional analytics systems were built for human queries and dashboards. The next generation of intelligent systems—autonomous agents, conversational AI, and adaptive applications—requires fundamentally different data infrastructure: the ability to access thousands of contextual data points per conversation at machine speed, with real-time freshness and millisecond responsiveness. StarTree's AI strategy centers on building a comprehensive **Analytics SuperBrain** architecture that transforms how intelligent systems interact with enterprise data. This isn't just about faster queries—it's about creating the data foundation that powers the next wave of AI-native applications, where systems continuously adapt, reason, and act based on live business context. ![Startree Ai Pn](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/images/startree-ai.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=c7c543d1d06586a77b5c90a9b5eae555) ### Key Capabilities **Next-Generation Query Console** Move beyond traditional SQL interfaces with a reimagined database client that enables natural language interactions for both human analysts and AI agents. **Real-time AI Data Access** Seamlessly connect AI agents and business users to live business data with millisecond query performance, ensuring responses are based on the most current information available. **Enterprise-Grade AI Integration** Deploy AI-powered analytics with the security, governance, and scale required for mission-critical business applications, serving both internal users and external AI systems. > [!NOTE] > > Dive deeper into StarTree's AI strategy with our founder and CEO Kishore Gopalakrishna's keynote presentation on building the Analytics SuperBrain. > > [**Watch the Keynote: "Building the Analytics SuperBrain"**](https://www.youtube.com/watch?v=your-keynote-video) > > *In this presentation, Kishore outlines how Apache Pinot's proven user-facing analytics capabilities are evolving to meet the demands of agent-facing analytics, where AI systems require continuous interaction with live business data.* ## AI Solutions > **[CosmiQ](/corecapabilities/ai/cosmiq/overview)** > > Conversational analytics agent for Apache Pinot. Ask questions in plain English and get answers backed by real Pinot SQL, grounded in curated Spaces. > **[Model Context Protocol (MCP) Server](/corecapabilities/ai/mcp/overview)** > > Your new query console reimagined for the AI age. Natural language analytics for business users and AI agents. > **[Query Analyzer](/corecapabilities/ai/query-analyzer/overview)** > > AI-powered query optimization that analyzes your Pinot multi-stage queries and produces prioritized, evidence-backed recommendations. > **Native Vector Auto Embedding (Coming Soon)** > > Automatic vector embeddings for semantic search combined with real-time analytics on structured data # Query Analyzer API Reference Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/api-reference REST API reference for the Query Analyzer static and runtime analysis endpoints. > [!NOTE] > > Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment. Query Analyzer exposes two REST endpoints on the StarTree Cloud data-manager service. Both return the same unified response structure. ## Endpoints | Method | Path | Purpose | | - | - | - | | `POST` | `/api/data/manager/api/query-analyzer/static-analysis` | Analyze a query using table metadata and explain plan — no execution required | | `POST` | `/api/data/manager/api/query-analyzer/runtime-analysis` | Execute the query, collect stage statistics, then analyze with evidence-backed findings | ## Authentication and Headers All requests require a valid StarTree Cloud Bearer token. The `workspace` header scopes the request to a specific workspace. | Header | Required | Description | | - | - | - | | `Authorization` | Yes | `Bearer ` | | `Content-Type` | Yes | `application/json` | | `workspace` | No | Workspace ID. When omitted, the default workspace for the principal is used. | *** ## POST /api/data/manager/api/query-analyzer/static-analysis Analyzes a SQL query using only table metadata and explain plan. Does not execute the query. Returns results within seconds. ### Request Body ```json theme={null} { "sql": "SELECT COUNT(*) FROM orders WHERE status = 'shipped' AND ts > 1700000000", "timeout": "PT60S" } ``` | Field | Type | Required | Description | | - | - | - | - | | `sql` | string | Yes | The SQL query to analyze | | `timeout` | string | No | ISO-8601 duration override for the LLM call (e.g. `"PT60S"`, `"PT5M"`). Min: 10s. Max: 10 minutes. | ### Example Request ```bash theme={null} curl -X POST "https:///api/data/manager/api/query-analyzer/static-analysis" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -H "workspace: " \ -d '{ "sql": "SELECT COUNT(*) FROM orders WHERE status = '\''shipped'\'' AND ts > 1700000000" }' ``` ### Example Response ```json theme={null} { "querySummary": "Counts all orders with status 'shipped' after a given Unix timestamp.", "analysisType": "STATIC", "findings": [ { "type": "MISSING_INVERTED_INDEX", "severity": "HIGH", "description": "Column 'status' is used in an equality predicate but has no inverted index. Every segment will be fully scanned.", "evidence": [ { "type": "MISSING_INDEX", "detail": "status: no inverted index found" } ], "confidence": "HIGH" } ], "recommendations": [ { "rank": 1, "title": "Add inverted index on 'status'", "type": "ADD_INVERTED_INDEX", "description": "An inverted index on 'status' will allow Pinot to skip segments that do not contain the predicate value, eliminating the full scan.", "suggestedChange": { "before": "\"invertedIndexColumns\": []", "after": "\"invertedIndexColumns\": [\"status\"]" }, "expectedImpact": "HIGH", "confidence": "HIGH", "tradeoffs": "Adds a small amount of index storage per segment. Requires a segment reload to take effect on existing data." } ] } ``` *** ## POST /api/data/manager/api/query-analyzer/runtime-analysis Executes the query against your Pinot cluster and collects execution statistics, using stage statistics wherever they're available. It then produces evidence-backed findings tied to the measured metrics. If the query can't be executed or doesn't return execution statistics, the service falls back to static analysis. The response `analysisType` field shows which mode was used, and `mseIncompatibleReason` explains the fallback. ### Request Body ```json theme={null} { "sql": "SELECT status, COUNT(*) FROM orders WHERE ts BETWEEN 1700000000 AND 1710000000 GROUP BY status", "timeout": "PT120S" } ``` | Field | Type | Required | Description | | - | - | - | - | | `sql` | string | Yes | The SQL query to execute and analyze | | `timeout` | string | No | ISO-8601 duration override for the LLM call. Min: 10s. Max: 10 minutes. | > [!WARNING] > > Runtime analysis executes your query against the cluster. For very large or expensive queries, run static analysis first to catch structural issues before spending compute. ### Example Request ```bash theme={null} curl -X POST "https:///api/data/manager/api/query-analyzer/runtime-analysis" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -H "workspace: " \ -d '{ "sql": "SELECT status, COUNT(*) FROM orders WHERE ts BETWEEN 1700000000 AND 1710000000 GROUP BY status" }' ``` ### Example Response ```json theme={null} { "querySummary": "Counts orders grouped by status within a time window.", "analysisType": "RUNTIME", "findings": [ { "type": "MISSING_RANGE_INDEX", "severity": "HIGH", "description": "Column 'ts' is used in a BETWEEN predicate but has no range index. All segments are being scanned.", "evidence": [ { "type": "MISSING_INDEX", "detail": "ts: no range index found" }, { "type": "HIGH_POST_FILTER_SCAN", "detail": "numEntriesScannedPostFilter: 48,200,000" } ], "confidence": "HIGH", "operator": "TABLE_SCAN", "metrics": { "executionTimeMs": 3420, "memoryAllocatedBytes": 0, "rowsProduced": 12400 } } ], "recommendations": [ { "rank": 1, "title": "Add range index on 'ts'", "type": "ADD_RANGE_INDEX", "description": "A range index on 'ts' allows Pinot to prune segments outside the time window, reducing the scan to only matching segments.", "suggestedChange": { "before": "\"rangeIndexColumns\": []", "after": "\"rangeIndexColumns\": [\"ts\"]" }, "expectedImpact": "HIGH", "confidence": "HIGH", "tradeoffs": "Adds index storage per segment. Requires a segment reload to take effect on existing data." } ] } ``` *** ## Response Schema Both endpoints return a `QueryAnalysisResponse` object. | Field | Type | Description | | - | - | - | | `querySummary` | string | Plain-English description of what the query does | | `analysisType` | string | `RUNTIME` when query execution succeeded; `STATIC` when the service fell back to static analysis | | `mseIncompatibleReason` | string | Reason runtime analysis fell back to static analysis. Null when `analysisType` is `RUNTIME`. | | `findings` | array | Risk flags detected, each with type, severity, description, evidence, confidence, and (runtime only) operator and metrics | | `recommendations` | array | Ranked, actionable suggestions with title, type, description, suggested change, expected impact, confidence, evidence references, tradeoffs, and correctness notes | | `error` | object | Structured error. Null on success. See [Error Response](#error-response). | *** ## Error Response When a request fails, the response body contains an `error` object and the HTTP status reflects the failure category. ```json theme={null} { "error": { "errorType": "BAD_REQUEST", "description": "Invalid timeout value 'PT1S'. Expected ISO-8601 duration (e.g. PT60S, PT5M)." } } ``` | Field | Type | Description | | - | - | - | | `errorType` | string | Machine-readable error category (e.g. `BAD_REQUEST`, `PINOT_UNAVAILABLE`, `LLM_ERROR`, `UNKNOWN`) | | `description` | string | Human-readable explanation of the failure | | `cause` | object | Optional nested cause chain for debugging | Common HTTP status codes: | Status | When | | - | - | | `200` | Analysis completed successfully | | `400` | Malformed request (e.g. missing `sql`, invalid `timeout` format, timeout out of bounds) | | `401` | Missing or invalid authentication token | | `403` | The authenticated principal does not have access to the requested tables | | `500` | Internal error — contact support if it persists | *** ## Related Documentation > **[How to Use Query Analyzer](/corecapabilities/ai/query-analyzer/usage)** > > Step-by-step guide to running analysis from the Query Console and via API. > **[Understanding Recommendations](/corecapabilities/ai/query-analyzer/reading-results)** > > Full reference for every finding type, recommendation category, operator type, and evidence field in the response. > **[Security and Compliance](/corecapabilities/ai/query-analyzer/security-and-compliance)** > > What data is sent to the LLM, access controls, and platform compliance posture. # Query Analyzer Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/overview > [!NOTE] > > Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment. Query Analyzer is an AI-powered tool built into StarTree Cloud that helps you understand and optimize Apache Pinot queries. It analyzes your SQL query alongside table metadata, explain plans, and execution statistics to produce prioritized, evidence-backed recommendations — all without requiring deep Pinot expertise. ## The Problem Optimizing Pinot queries is hard because the debugging surface is **fragmented and expert-only**: * **Explain plans** are detailed but difficult to interpret for most users * **Stage statistics** show where time was spent, but not what to change * **Table metadata** (indexes, cardinality, data types) is spread across multiple pages * Users end up guessing, iterating blindly, or escalating to support This leads to slow time-to-resolution, wasted compute, and repetitive support loops. ## How to Access Query Analyzer is available directly in the **StarTree Data Portal Query Console**. Once enabled for your environment, no additional configuration is required — write your SQL query and use the built-in Analyze options to get instant, AI-generated optimization guidance. > [!NOTE] > > Navigate to **Data Portal → Query Console**, write your query, and select **Analyze** to get started. ## What You Get Back Query Analyzer returns three top-level sections for every analysis: | Output | Description | | - | - | | **Query Summary** | Plain-English description of what the query does | | **Risk Flags** | Issues found, ranked by severity. Each flag carries its own supporting evidence, bottleneck metrics (runtime), and any correctness alerts — so context and cause are always together | | **Recommendations** | Ranked, actionable suggestions with expected impact, a before/after change, and tradeoffs | Every recommendation is grounded in measured evidence, not generic advice. ## Two Analysis Modes > **[Static Analysis](/corecapabilities/ai/query-analyzer/usage#running-static-analysis)** > > Run **before** executing your query. Flags likely problems using table metadata, index configuration, and the explain plan — so you can fix issues before spending compute. > **[Runtime Analysis](/corecapabilities/ai/query-analyzer/usage#running-runtime-analysis)** > > Run **after** executing your query. Uses real execution statistics to pinpoint actual bottlenecks and provide evidence-based fixes. > [!NOTE] > > **Proactive Analyzer:** Query Analyzer also supports a proactive mode that runs on a schedule, ranks your most expensive real queries from the query log, and analyzes them automatically — no query submission required. Disabled by default. See [Proactive Analyzer](/corecapabilities/ai/query-analyzer/proactive-analyzer/overview). ## What Gets Analyzed **Index gaps** Missing inverted, range, sorted, bloom filter, text, FST, JSON, star-tree, timestamp, geospatial, sparse, composite JSON, and vector indexes, as well as full scan risk. **Query patterns** Wrong join order, filters not pushed below joins, broadcast join risk, cross-join risk, OFFSET-based pagination, `SELECT *`, leading-wildcard `LIKE`, high result LIMIT, unnecessary sort, high-cardinality `GROUP BY`, group limit risk, heavy aggregation, suboptimal column encoding, missing cross-table filter, large table without pre-filter, and missing date range propagation. **Operator-level execution** *(runtime only)* Each risk flag produced during runtime analysis includes measured metrics from the relevant operator — execution time, memory allocation, rows produced, bytes shuffled, and correctness signals such as whether the aggregation group limit was hit. ## Security and Data Privacy Query Analyzer uses an AI model to generate recommendations. Your SQL and table metadata are processed as part of every analysis — review what is sent, how it is protected, and how it maps to your security and compliance requirements **before** enabling or using the feature. > **[Security and Compliance](/corecapabilities/ai/query-analyzer/security-and-compliance)** > > **Essential reading for security, compliance, and data governance teams.** > > * AI model and cloud provider by deployment > * What data is sent to the LLM — and what is explicitly excluded > * Credential masking, data flows, and retention > * Access controls, encryption, and platform certifications (SOC 2, ISO 27001, HIPAA readiness) ## Guardrails Query Analyzer is designed to produce high-signal, actionable output: * Every recommendation includes specific evidence from the query plan or execution statistics * Low-confidence and low-impact findings are suppressed — only high-signal results are shown * Correctness alerts (such as results being potentially incomplete due to group limits) are surfaced prominently * Query rewrite suggestions are validated against Pinot SQL syntax rules * All suggestions include tradeoffs so you can make informed decisions ## Next Steps > **[How to Use Query Analyzer](/corecapabilities/ai/query-analyzer/usage)** > > Step-by-step guide to running static and runtime analysis from the Query Console, and how to act on the results. > **[API Reference](/corecapabilities/ai/query-analyzer/api-reference)** > > Full reference for the static and runtime analysis REST endpoints — request schemas, response fields, curl examples, and error handling. > **[Proactive Analyzer](/corecapabilities/ai/query-analyzer/proactive-analyzer/overview)** > > Scheduled background mode that finds and analyzes your most expensive real queries automatically. Disabled by default. > **[Understanding Recommendations](/corecapabilities/ai/query-analyzer/reading-results)** > > Reference guide to every output field, risk flag type, recommendation category, and what each means for your query. > **[Security and Compliance](/corecapabilities/ai/query-analyzer/security-and-compliance)** > > AI model, data privacy, access controls, and platform compliance. # Proactive Analyzer Configuration and Limitations Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/proactive-analyzer/configuration Configuration keys, the API endpoints, and current known limitations for Proactive Analyzer. > [!WARNING] > > **Disabled by default.** Proactive Analyzer is off by default and requires the reactive [Query Analyzer](/corecapabilities/ai/query-analyzer/overview) to be enabled first. Contact your StarTree account team to have it enabled for your environment. ## Enablement There is no single on/off flag for Proactive Analyzer. It activates when **both** of these are true: 1. The reactive analyzer is on (`query-analyzer.enabled=true`) — Proactive Analyzer reuses the same analysis engine. 2. At least one **delivery target** is configured: a Slack webhook URL or an email recipients list (see below). If either condition is missing, the scheduled job never loads and the proactive API endpoints are not available. ## Configuration All keys live under the `query-analyzer.proactive.` prefix: | Setting | Default | Controls | | - | - | - | | `cron` | `0 0 9 * * *` | When the scan runs — daily at 09:00 **UTC** by default | | `interval` | `PT24H` | The lookback window used to rank queries from the log | | `query-engines` | `MSE,SSE` | Which query engines are scanned — one ranking pass per engine (multi-stage and single-stage by default) | | `min-time-used-ms` | `100` | Minimum latency for a query shape to qualify | | `max-analyses-per-run` | `5` | Cap on analyses **per engine** per run — up to 10 total with the default two engines | | `system-query-log-table` | `system_query_log` | The query log table the scan reads | | `notify.slack.webhook-url` | *(empty)* | Slack digest delivery — setting a webhook URL turns Slack delivery on | | `notify.email.recipients` | *(empty)* | Accepted as a delivery target for activation, but email delivery is not implemented yet | The skip-when-unchanged behavior (re-analyzing a query shape only when the table configuration has changed since the last analysis) is built in and not configurable. ## API | Endpoint | Behavior | | - | - | | `POST /api/query-analyzer/proactive/run` | Starts one scan immediately on a background thread, without waiting for the schedule. Returns `202 Accepted`, or `409 Conflict` if a scan is already in progress. | | `GET /api/query-analyzer/proactive/recommendations` | Lists the stored recommendations for the workspace, most expensive first. | | `GET /api/query-analyzer/proactive/status` | Reports the effective configuration and a summary of the last run — candidates ranked, analyzed, skipped, failed, and the trigger (scheduled or manual). | ## Current Limitations * **Recommendations are stored in memory** and reset when the service restarts. * **Scans a single workspace on a single cluster per run.** Multi-cluster and multi-workspace scanning are not yet supported. * **No automatic ticketing and no dedicated UI.** The digest points a human at the work; it does not file tickets or change any cluster setting on its own. * **Email delivery is not yet available.** Slack and the log are the supported channels today. ## Related > **[Proactive Analyzer Overview](/corecapabilities/ai/query-analyzer/proactive-analyzer/overview)** > > What Proactive Analyzer does, how it works, and how it delivers results. # Proactive Analyzer Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/proactive-analyzer/overview A scheduled job that finds your most expensive queries from real traffic and analyzes them automatically — no query submission required. > [!WARNING] > > **Disabled by default.** Proactive Analyzer is off by default and requires the reactive [Query Analyzer](/corecapabilities/ai/query-analyzer/overview) to be enabled first. Contact your StarTree account team to have it enabled for your environment. ## What Is Proactive Analyzer? Static and Runtime Analysis both require a deliberate trigger — you submit a query and ask for it to be analyzed. Proactive Analyzer removes that requirement. Proactive Analyzer runs on a **schedule** (once a day by default), scans the queries that actually ran on your cluster, ranks them by cost, and runs the same analysis engine against the worst offenders automatically. It does this **without re-running any query** — it reads execution statistics that Pinot already recorded. The goal is to catch expensive query patterns as they occur in production — the queries that quietly burn cluster resources without anyone flagging them — rather than waiting for a user to notice a slowdown and submit it for analysis. ## How It Works Pinot records the queries it runs in a system table, `system_query_log` (by default, slow and failed queries), including the SQL, a stable hash identifying the query shape, execution time, CPU counters, scan volume, and per-stage execution statistics. Proactive Analyzer's scheduled job reads that log and, on each run: 1. **Checks the log is available.** If `system_query_log` is not queryable on the cluster, the run skips cleanly. 2. **Ranks query shapes by cost, per query engine.** The scan runs one ranking pass for each configured query engine (multi-stage and single-stage by default). Within each pass, shapes are grouped by query hash and ranked by **mean execution time** over the lookback window (24 hours by default) — a shape that is consistently slow ranks above one that is merely popular. Failed runs are excluded, and a minimum latency floor filters out trivially fast queries. 3. **Selects the worst run of each shape.** For each of the top-ranked shapes (up to 5 per engine by default), the single slowest successful run in the window is pulled, carrying its SQL and recorded execution statistics. 4. **Filters.** Queries against the log table itself are excluded, as are shapes with no matching successful run in the window and queries that consist entirely of SQL comments. 5. **Analyzes each one**, reusing the existing analysis engine (see below), up to a per-engine cap (5 by default — up to 10 analyses per run with the default two engines). 6. **Stores the results**, keyed by workspace, query shape, and query engine. 7. **Delivers a digest** summarizing what it found. The schedule, lookback window, ranking thresholds, and per-run caps are all configurable — see [Configuration and Limitations](/corecapabilities/ai/query-analyzer/proactive-analyzer/configuration). ### It never re-runs a slow query Re-running the exact queries just flagged as expensive would place the heaviest possible load back on the cluster — the opposite of what a tuning tool should do. Proactive Analyzer avoids this by reusing the execution statistics already captured in the query log. The only live data it fetches during analysis is cheap and current: table config, schema, table stats, and the explain plan — so recommendations reflect how the table is set up today, without replaying any heavy query. (When the analyzer proposes a rewritten query, the *rewrite* is validated against the cluster as usual; the original slow query is still never re-executed.) ### It reuses the same analysis engine Proactive Analyzer does not introduce a second recommendation engine. It selects which queries to analyze, then hands each one to the same engine used by [Runtime Analysis](/corecapabilities/ai/query-analyzer/usage#running-runtime-analysis) — same prompt, same evidence gathering, same validation and guardrail checks — with one difference: instead of executing the query to capture statistics, it injects the statistics already recorded in the query log. This matters for quality: the analyzer's most useful checks (for example, "this query is already fast enough" or "this ran out of memory") depend on seeing real runtime numbers. Because Proactive Analyzer feeds in the real numbers from the logged slow run, those checks behave exactly as they do for a manually submitted query — the output isn't a lower-quality approximation. ### Skips work that hasn't changed Before analyzing a query shape, the job checks whether it has already analyzed it. It fingerprints the current table configuration and compares it to the fingerprint from the last analysis. If nothing has changed, there is nothing new to recommend — the job refreshes the cost numbers and skips the analysis call for that shape entirely, keeping run cost predictable. ## Delivery Every run stores its recommendations and, on top of that, sends a digest: | Channel | Behavior | | - | - | | **Log** | Always on. A formatted digest is written to the service log. | | **Slack** | Optional. Configuring an incoming webhook URL turns it on; the run posts a summary of newly analyzed recommendations. | | **Email** | Not yet available. A recipients setting is accepted (and counts as a delivery target for activation), but email delivery is not implemented yet. | The digest also identifies where it came from — organization, environment, and a link back to the Query Console — so a shared Slack channel receiving digests from several environments stays readable. If a run finds nothing to report (no log table, or zero new recommendations), the digest is skipped rather than sent empty. A failure in one delivery channel doesn't affect the others or the run itself. ## Available Today While you evaluate Proactive Analyzer, both on-demand analysis modes are fully available: > **[Static Analysis](/corecapabilities/ai/query-analyzer/usage#running-static-analysis)** > > Run before executing your query. Catches structural issues using table metadata, index configuration, and the explain plan. > **[Runtime Analysis](/corecapabilities/ai/query-analyzer/usage#running-runtime-analysis)** > > Run after executing your query. Uses real execution statistics to pinpoint actual bottlenecks with measured evidence. Both modes are available from the Query Console and via the [Query Analyzer API](/corecapabilities/ai/query-analyzer/api-reference). > **[Configuration and Limitations](/corecapabilities/ai/query-analyzer/proactive-analyzer/configuration)** > > Config keys, the manual trigger endpoint, and current known limitations. # Understanding Recommendations Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/reading-results > [!NOTE] > > Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment. This page is a reference for everything Query Analyzer returns. Use it to understand what each field means, what triggered a specific finding, and what the different recommendation and risk flag categories represent. ## Output Sections ### Risk Flags The main output section. Each risk flag represents one detected issue and bundles everything related to that issue in one place: what was found, why it matters, the measured evidence behind it, and — for runtime analysis — the operator metrics and any correctness concerns. | Field | Description | | - | - | | **Type** | The category of issue detected (see [Risk Flag Types](#risk-flag-types) below) | | **Severity** | How serious the issue is — `HIGH` or `MEDIUM` | | **Description** | A plain-English explanation of what was found and why it matters | | **Evidence** | The specific data that triggered the flag — for example, a column name, cardinality value, index configuration, or measured execution metric | | **Confidence** | How confident the analyzer is in the finding — `HIGH` or `MEDIUM` | | **Operator** *(runtime)* | The execution operator this flag relates to — for example, Hash Join or Table Scan (see [Operator Types](#operator-types)) | | **Metrics** *(runtime)* | Measured execution values for the operator — time, memory, rows, bytes shuffled | Only `HIGH` and `MEDIUM` confidence findings are shown. Low-confidence results are suppressed to keep output focused. > [!WARNING] > > When a risk flag carries a correctness alert, address it before acting on performance recommendations. A correctness issue — such as the group limit being reached — means query results may already be incomplete, which affects how you should interpret the rest of the analysis. ### Recommendations Ranked, actionable suggestions. Each recommendation includes: | Field | Description | | - | - | | **Rank** | Priority order — rank 1 has the highest expected impact | | **Title** | Short description of the action to take | | **Type** | The category of change recommended (see [Recommendation Types](#recommendation-types) below) | | **Description** | Detailed explanation of why this change helps | | **Suggested Change** | A before/after view of the specific change — a query snippet, index config, or hint | | **Expected Impact** | How much improvement is expected — `HIGH` or `MEDIUM` | | **Confidence** | How confident the analyzer is in this recommendation — `HIGH` or `MEDIUM` | | **Evidence** | References to the supporting evidence items that justify this recommendation | | **Tradeoffs** | Side effects or caveats to consider before applying the change | | **Correctness Notes** | Semantic considerations — for example, whether a join reorder affects NULL handling | Recommendations are ranked by expected impact first, then confidence. A query rewrite that eliminates an expensive operation ranks above an index addition at the same impact level. *** ## Risk Flag Types Risk flags are detected during both **static and runtime analysis**. There are two categories. ### Index Gaps These flags indicate that a column used in a filter, join, or aggregation is missing an index that would significantly speed up the query. | Risk Flag | What It Means | | - | - | | **Missing Inverted Index** | A column used in an equality or IN predicate has no inverted index, requiring a full scan of each segment | | **Missing Range Index** | A column used in a range comparison (`>`, `<`, `BETWEEN`) has no range index | | **Missing Sorted Index** | A high-selectivity column could reduce segment pruning with a sorted index | | **Missing Bloom Filter** | A low-cardinality column with equality predicates could benefit from a bloom filter to skip segments quickly | | **Missing Text Index** | A column used with `LIKE` or text search has no full-text index | | **Missing FST Index** | A column used with regex predicates has no FST (Finite State Transducer) index | | **Missing JSON Index** | A column with JSON path access has no JSON index | | **Missing Star-Tree Index** | An aggregation pattern could be pre-computed with a star-tree index, potentially eliminating the scan entirely | | **Missing Timestamp Index** | A time column used in time-bucketing functions has no timestamp index | | **Missing Geospatial Index** | A column used in spatial predicates has no geospatial index | | **Missing Sparse Index** | A column stored in cold/tiered storage could benefit from a sparse index to reduce remote reads | | **Missing Composite JSON Index** | Multiple JSON path accesses could be accelerated with a composite JSON index | | **Missing Vector Index** | A column used in vector similarity search has no vector index | | **Full Scan Risk** | The query structure is likely to cause a full segment scan regardless of indexes | ### Query Pattern Issues These flags indicate structural patterns in the SQL or query plan that are likely to cause poor performance. | Risk Flag | What It Means | | - | - | | **Wrong Join Order** | The larger table appears on the right (hash build) side of a join. Pinot builds the in-memory hash table from the right input — putting the larger table there wastes memory and slows the build | | **Filter Not Pushed Down** | A filter appears above a join in the execution plan instead of below it. Pushing filters down reduces the number of rows that reach the join | | **Broadcast Join** | The right-side input of a join is broadcast to every worker, which is expensive for large tables | | **Self Join** | A table is joined with itself, which may indicate a query design issue | | **Cross Join Risk** | A join condition is non-equality, which can produce a Cartesian product | | **Offset Pagination** | `OFFSET` is used for pagination. For large offsets this requires scanning and discarding thousands of rows — use keyset (cursor) pagination instead | | **Select Star** | `SELECT *` fetches all columns including potentially large or unused ones | | **Leading Wildcard** | A `LIKE '%...'` pattern cannot use index prefix optimization and requires scanning every value | | **High Result Limit** | A very large `LIMIT` value may produce excessive result sets | | **Unnecessary Sort** | An `ORDER BY` clause has no corresponding `LIMIT`, meaning all rows are sorted | | **High Cardinality Group By** | `GROUP BY` on a high-cardinality column produces a large intermediate state that requires significant memory | | **Group Limit Risk** | The estimated number of groups may approach or exceed the default limit of 100,000 groups | | **Heavy Aggregation** | A memory-intensive aggregate function such as `DISTINCTCOUNT` is used | | **Suboptimal Encoding** | A column uses raw encoding where dictionary encoding would improve index effectiveness | ### Filter Propagation Issues These flags identify missed opportunities to reduce data early by pushing filters across the query plan. | Risk Flag | What It Means | | - | - | | **Missing Cross-Table Filter** | A filter applied to one side of a join could also be applied to the other side, but isn't — the engine ends up joining more rows than necessary | | **Large Table Without Pre-filter** | A large table is scanned with no pre-filter, causing a full scan before the join or aggregation that follows | | **Missing Date Range Propagation** | A date or time range filter on one side of a join is not propagated to the other side, missing an opportunity to prune data before the join executes | ### Diagnostic Values These values may appear in the `type` field of an API response but are not user-facing risk flags. They are used internally to drive post-processing logic and short-circuit gates. | Value | What It Means | | - | - | | `QUERY_FAILED` | The query did not execute successfully. When present, the analyzer may suppress or adjust downstream recommendations that depend on execution data. | | `STAR_TREE_FAST_PATH_FIRED` | A positive signal — an existing star-tree index's metadata fast-path served the query without a full scan. Used to trigger the "already optimal" short-circuit. | *** ## Operator Types Operator types appear within risk flags during runtime analysis. The **API value** column shows the exact string returned in the `operator` field of a `Finding` object. | Operator | API Value | What It Does | Key Metrics to Watch | | - | - | - | - | | **Table Scan** | `LEAF` | Scans segments and applies filters | Entries scanned in-filter (high = indexes not being used); entries scanned post-filter (high = star-tree opportunity) | | **Hash Join** | `HASH_JOIN` | Builds a hash table from the right input and probes it with the left | Hash table build time; memory allocated; rows emitted (high rows = join explosion) | | **Data Send** | `MAILBOX_SEND` | Sends rows to downstream workers across stages | Bytes serialized; fan-out (number of workers data is sent to) | | **Data Receive** | `MAILBOX_RECEIVE` | Receives rows from upstream workers | Bytes deserialized; upstream wait time (high = upstream stage is slow) | | **Aggregation** | `AGGREGATE` | Groups rows and computes aggregate functions | Memory allocated; groups emitted; whether the group limit was hit | | **Window Function** | `WINDOW` | Computes window functions over a partition | Execution time; memory allocated (high = large partitions or wide frames) | | **Sort** | `SORT` | Sorts rows, typically in combination with a LIMIT | Execution time and memory for sort buffer | | **Filter** | `FILTER` | Applies a predicate to filter rows mid-plan | Execution time; rows emitted (high = filter is not selective enough) | | **Transform** | `TRANSFORM` | Applies column transformations or expression evaluation | Execution time; memory allocated (high = expensive expressions or many columns) | | **Union** | `UNION` | Combines row sets from multiple inputs (`UNION ALL`) | Execution time; total rows emitted across all inputs | | **Intersect** | `INTERSECT` | Returns rows present in all inputs (`INTERSECT`) | Execution time; memory allocated for set tracking | | **Minus** | `MINUS` | Returns rows from the first input not present in subsequent inputs (`EXCEPT`) | Execution time; memory allocated for set tracking | *** ## Evidence Types Evidence is embedded within each risk flag to show what triggered it. The type tells you where the finding came from. ### Found in Both Modes | Evidence Type | What It Means | | - | - | | **Full Scan** | A full segment scan was detected in the explain plan | | **Index Used** | A specific index was used — for example, an inverted index for an equality predicate | | **Join Order** | Join order analysis based on table row counts | | **Missing Index** | A column is missing an index that would help — covers inverted, range, text, FST, JSON, bloom filter, star-tree, timestamp, geospatial, sparse, composite JSON, and vector index types | | **Filter-Only Join** | The right side of a join is used only to filter rows, making it a candidate for rewriting as `WHERE col IN (SELECT ...)` | | **Star-Tree Blocked** | A star-tree index cannot be used because the aggregation depends on a cross-table join | | **Column Cardinality** | Cardinality data from table statistics, used to assess index recommendations | | **Predicate Pattern** | Analysis of the query's filter structure | | **Suboptimal Encoding** | Column encoding is suboptimal for the query's access pattern | ### Found During Static Analysis Only | Evidence Type | What It Means | | - | - | | **Plan Structure** | A structural observation from the explain plan | | **Exchange Type** | The data distribution pattern between stages — hash, broadcast, or random | ### Found During Runtime Analysis Only | Evidence Type | What It Means | | - | - | | **High Post-Filter Scan** | Many entries were scanned after the filter, suggesting a star-tree index opportunity | | **Join Build Time** | High hash table build time indicates the right-side table is too large | | **Join Explosion** | Large number of output rows from the join, indicating a many-to-many relationship | | **Group Limit Hit** | The 100,000 group limit was reached — results may be incomplete | | **High Memory** | An operator allocated significant heap memory | | **Heavy Shuffle** | Large volumes of data were sent between workers | | **Upstream Wait** | A stage spent significant time waiting for its upstream stage to produce data | | **Downstream Wait** | A stage was blocked waiting for the downstream consumer to process rows | | **Broadcast Detected** | A broadcast exchange was detected in the execution tree | *** ## Recommendation Types Recommendations appear in both static and runtime analysis. ### Query Changes | Type | What Gets Changed | | - | - | | **Query Rewrite** | The SQL itself — for example, reordering joins, replacing OFFSET with keyset pagination, or rewriting a join as a subquery | | **Hint Addition** | A query hint added inline to control join strategy, aggregation behavior, or partition handling | ### Index Additions | Type | Index Added | | - | - | | **Add Inverted Index** | Speeds up equality and IN predicates on the target column | | **Add Range Index** | Speeds up range comparisons on the target column | | **Add Bloom Filter** | Allows fast segment skipping for equality predicates on the target column | | **Add Text Index** | Enables full-text search and speeds up LIKE predicates | | **Add FST Index** | Enables fast regex matching on the target column | | **Add JSON Index** | Speeds up JSON path access on the target column | | **Add Star-Tree Index** | Pre-computes aggregations to potentially eliminate scanning entirely | | **Add Timestamp Index** | Speeds up time-bucketing functions on datetime columns | | **Add Geospatial Index** | Speeds up spatial predicate evaluation | | **Add Sparse Index** | Reduces remote reads for columns in cold/tiered storage | | **Add Composite JSON Index** | Optimizes multiple JSON path accesses together | | **Add Vector Index** | Enables efficient vector similarity search | ### Configuration Changes | Type | What Gets Changed | | - | - | | **Encoding Change** | Column encoding type — for example, switching from raw to dictionary encoding | | **Join Strategy** | How the join is executed — for example, using a lookup join for dimension tables or a colocated join to eliminate data shuffling | | **Aggregation Hint** | Aggregation behavior — for example, enabling group trimming for ORDER BY + LIMIT patterns, or increasing the group count limit | | **Repartition Data** | Set or align `segmentPartitionConfig` and repartition existing data. The config change takes effect on new segments immediately; existing segments require a segment refresh or re-ingest to physically repartition. | | **Table Rebalance** | Trigger a controller-level table rebalance to redistribute segments across servers. No config diff is required — the rebalance operation alone addresses the imbalance. | | **Cluster Scale** | Scale the cluster up or out — add server nodes, increase memory, or add replicas. Recommended only as a last resort for sustained resource saturation (OOM, repeated timeouts) that no query rewrite or configuration change can resolve. | | **Scope Reduction** | Reduce what the query asks for — for example, narrow the time window, add a selective pre-filter, drop high-cardinality GROUP BY columns, or accept approximate aggregates. **This changes the result set by design.** Always a secondary recommendation on resource-exhaustion cases (group-limit hit, OOM, timeout) where the volume of data demanded is itself the problem. Never ranked above an equivalence-preserving fix, and always flagged as result-altering in the suggested change. | ### Sentinel | Type | What It Means | | - | - | | **No Optimization Needed** | The query is already optimal — for example, a trivial `SELECT * FROM t LIMIT n` preview. No suggested change is produced; the recommendation carries only a title and description confirming the query needs no changes. | # Security and Compliance Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/security-and-compliance Security controls, data privacy, and compliance posture for Query Analyzer in StarTree Cloud. Query Analyzer is an opt-in, AI-powered feature in StarTree Cloud. This page describes how your data is handled, which security controls apply, and how Query Analyzer inherits StarTree Cloud's platform compliance posture. > [!NOTE] > > Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment. ## Overview Query Analyzer is **request-scoped**: it processes one analysis per request and does **not** persist query text, metadata, or LLM responses to any database or customer-accessible storage. ## Data Flows ```mermaid theme={null} flowchart LR Portal[Data Portal] Cloud[StarTree Cloud] Pinot[Pinot Cluster] LLM[LLM Provider] Portal -->|"SQL + auth headers"| Cloud Cloud -->|"Metadata, explain, execution stats"| Pinot Cloud -->|"Minimized prompt"| LLM LLM -->|"Analysis response"| Cloud Cloud -->|"Recommendations"| Portal ``` ### Collected from Pinot For normal (non-direct) analysis, Query Analyzer uses your authenticated identity to call **Pinot Controller** admin APIs on your cluster. The following operations run per table referenced in your SQL (except explain and query execution, which run once per analysis request). | Operation | Pinot API | Static Analysis | Runtime Analysis | API reference | | - | - | - | - | - | | Fetch table config and schema | `GET /tableConfigs/{tableName}` | Yes | Yes | [Get table configs](/api-reference/table/get-the-tableconfigs-for-a-given-raw-tablename) | | Infer index configuration | `POST /tables/inferIndexes` | Yes | Yes | StarTree extension API (normalizes index config for analysis) | | Fetch table statistics | `GET /tables/{tableName}/metadata` | Yes | Yes | [Get aggregate table metadata](/api-reference/table/get-the-aggregate-metadata-of-all-segments-for-a-table) | | Run explain plan | `POST /sql` with `EXPLAIN PLAN FOR` and `explainAskingServers=true` | Yes | Yes | Pinot Controller SQL query API | | Execute the SQL query | `POST /sql` with `collectGcStats=true` | No | Yes | Pinot Controller SQL query API | > [!NOTE] > > The `POST /sql` endpoint is the Pinot Controller SQL query API. It is used for explain plans and, in runtime analysis, to execute your query and collect `stageStats`. This is separate from the broker query API (`POST /query/sql`) used for interactive querying in some deployments. Pinot's own RBAC policies govern which tables and operations your identity can access. Runtime analysis **executes your query** against the cluster. Use it only when the query is safe to run in your environment. ### Sent to the LLM #### What is never included in the LLM prompt > [!WARNING] > > | Never included | Detail | > | - | - | > | **Row data and query result values** | Your actual analytics data remains in Pinot | > | **Credentials and secret references** | Masked in table configuration before transmission | > | **Personally identifiable information (PII)** | No user-identifying information is included | > | **Model training** | Prompt data is **not used to train or improve** the underlying LLM | > > Only structural metadata and execution statistics are sent to the LLM provider. Your actual data values are not included. #### What is included in the LLM prompt | Data | Source | Notes | | - | - | - | | SQL query text | User input | The query you submit for analysis | | Table config (reduced) | Pinot | Stripped and masked (see note below) | | Schema | Pinot | Field types, encoding, and index configuration | | Column cardinality estimates | Pinot | Approximate distinct value counts per column | | Explain plan | Pinot | Execution structure only | | Execution statistics | Pinot | Operator timing and memory metrics (runtime analysis only) | > [!NOTE] > > Before transmission, StarTree Cloud **masks** credentials and secret references in table configuration JSON and **strips** non-essential sections (ingestion config, task definitions, tenants, metadata, and routing). ## Data Lifecycle and Retention | Aspect | Behavior | | - | - | | **Persistence** | No query text, metadata, or LLM responses are written to durable storage | | **In-memory scope** | Analysis data exists only for the duration of the HTTP request | | **Application logs** | StarTree Cloud application logs may record SQL text at INFO level; subject to platform log retention and access controls | ## AI Model and Provider | Deployment | AI Provider | Model | | - | - | - | | AWS | Amazon Bedrock | Claude (Anthropic), `us.anthropic.claude-sonnet-4-20250514-v1:0` | | Google Cloud (Coming Soon) | Vertex AI | Not yet available | | Azure (Coming Soon) | Azure AI Foundry | Not yet available | On AWS, prompts are processed by Amazon Bedrock under AWS's data privacy commitments. ## Access Controls Query Analyzer uses the same authentication and authorization model as other StarTree Cloud APIs. See [Manage Security](/corecapabilities/security/manage-security) for OIDC identity provider setup, RBAC policies, and API tokens. | Control | Behavior | | - | - | | **Authentication** | Required for `POST /api/query-analyzer/static-analysis` and `POST /api/query-analyzer/runtime-analysis` | | **Authorization** | Governed by StarTree Cloud RBAC and Pinot cluster policies for the tables in your query | | **Workspace scoping** | Use the `workspace` header to scope requests to a specific workspace | | **Feature availability** | Disabled by default. Contact your StarTree account team to enable. | ## Encryption | Layer | Control | | - | - | | **In transit** | TLS 1.2+ between Data Portal, StarTree Cloud, Pinot, and the LLM provider | | **At rest** | Query Analyzer does not write analysis data to disk; platform policies apply to operational logs | | **Secrets** | LLM API keys are managed through StarTree's secure deployment configuration and are not embedded in customer-facing payloads | ## Platform Compliance Query Analyzer runs within StarTree Cloud infrastructure and inherits platform certifications: * **SOC 2 Type 2** * **ISO 27001** * **HIPAA readiness** For full platform security details, see [Security and Compliance](/getstarted/deployment/security_certifications). ## Related Documentation > **[Query Analyzer Overview](/corecapabilities/ai/query-analyzer/overview)** > > Feature overview, analysis modes, and what gets analyzed. > **[StarTree Cloud Security](/getstarted/deployment/security_certifications)** > > Platform-wide security controls and compliance certifications. # How to Use Query Analyzer Source: https://docs.startree.ai/corecapabilities/ai/query-analyzer/usage > [!NOTE] > > Query Analyzer is disabled by default and available on demand. Contact your StarTree account team to have it enabled for your environment. Query Analyzer is built into the **StarTree Data Portal Query Console**. No setup is required — open the console, write your query, and run an analysis. ## Running Static Analysis Static analysis catches issues **before** your query runs. 1. Navigate to **Data Portal → Query Console** 2. Write or paste your SQL query 3. Select **Analyze → Static Analysis** Query Analyzer fetches table metadata automatically and returns results within seconds. *** ## Running Runtime Analysis Runtime analysis identifies issues **after** your query runs, using real execution statistics. Each risk flag includes the operator metrics and any correctness alerts relevant to that finding. 1. Write your SQL query in the Query Console 2. Select **Analyze → Runtime Analysis** > [!WARNING] > > Runtime analysis executes your query against your Pinot cluster. For very large or expensive queries, run static analysis first to catch structural issues before proceeding. *** ## Acting on Recommendations **Index additions** — Apply through **Data Portal → Table Management**. Index changes require a segment reload to take effect on existing data. **Query rewrites** — Apply directly in the Query Console editor. The recommendation shows a before/after diff. **Query hints** — Added inline to your SQL using `/*+ ... */` syntax. Take effect immediately without schema changes. > [!TIP] > > Start with rank 1 recommendations. If any risk flag carries a correctness alert, address it first — incomplete results affect how you should interpret the rest of the analysis. For a full breakdown of what each output field and recommendation type means, see [Understanding Recommendations](/corecapabilities/ai/query-analyzer/reading-results). *** ## Using the API Both analysis modes are available as REST endpoints on the StarTree Cloud data-manager service. Use these when integrating Query Analyzer into CI pipelines, query validation tooling, or custom dashboards. ### Prerequisites * A valid **StarTree Cloud API token** (Bearer auth) * The **workspace ID** for your environment (obtain from the Data Portal URL or your account team) * Query Analyzer must be enabled for your environment ### Static Analysis via API `POST /api/query-analyzer/static-analysis` Analyzes a SQL query using table metadata and explain plan only — no query execution required. ```bash theme={null} curl -X POST "https:///api/query-analyzer/static-analysis" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -H "workspace: " \ -d '{ "sql": "SELECT COUNT(*) FROM orders WHERE status = '\''shipped'\'' AND ts > 1700000000" }' ``` ### Runtime Analysis via API `POST /api/query-analyzer/runtime-analysis` Executes the query against your Pinot cluster, collects execution statistics, then runs analysis with evidence-backed findings. ```bash theme={null} curl -X POST "https:///api/query-analyzer/runtime-analysis" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -H "workspace: " \ -d '{ "sql": "SELECT status, COUNT(*) FROM orders WHERE ts BETWEEN 1700000000 AND 1710000000 GROUP BY status" }' ``` > [!WARNING] > > Runtime analysis executes your query against the cluster. For expensive queries, run static analysis first. ### Optional Request Parameters Both endpoints accept the following optional fields in the request body: | Parameter | Type | Description | | - | - | - | | `timeout` | string | ISO-8601 duration override for the LLM call (e.g. `"PT60S"`, `"PT5M"`). Minimum: 10 seconds. Maximum: 10 minutes. Default: server-configured value. | | `thinkingMode` | boolean | Enable extended chain-of-thought reasoning in the LLM. Produces deeper analysis at the cost of higher latency. Default: server-configured value. | For the full request and response schemas, see the [API Reference](/corecapabilities/ai/query-analyzer/api-reference). *** ## Tips * **Use real queries**: Query Analyzer uses column cardinality and table size to produce accurate recommendations. Simplified test queries produce weaker results. * **Static first, runtime second**: Static analysis is faster. Use runtime analysis for your most important or problematic queries. * **Read the tradeoffs**: Every recommendation includes tradeoffs, for example, an index that helps one query pattern may not justify the storage cost. # Custom Plugins Source: https://docs.startree.ai/corecapabilities/cluster-operations/custom-plugins Load your own Apache Pinot plugin JARs into StarTree Cloud without a custom image or build. Some workloads depend on custom Apache Pinot plugins — for example, a proprietary decoder for an internal data format — that can't be contributed to open-source Pinot. StarTree Cloud lets you load your own plugin JARs into every Pinot component (broker, server, controller, and minion) at startup, without needing a custom StarTree Cloud image or build. Your plugin JARs are stored in your workspace's deep store S3 bucket. At startup, each Pinot component downloads them and adds them to its runtime classpath — so a plugin registered this way behaves exactly like a built-in Pinot plugin, including for logging, metrics, and tracing. > [!NOTE] > > This is a self-serve, GA feature — no support ticket required. It's supported on the v2 operator (1.10.8+) and Vintner (2.0.8+). ## Prerequisites | Deployment path | Minimum version | | :- | :- | | v2 operator | 1.10.8 | | Vintner | 2.0.8 | > [!WARNING] > > Enabling or changing custom plugins triggers a rolling restart of all Pinot components in the cluster. Plan the change accordingly. ## Setting up a custom plugin **1. Upload your plugin JAR to S3** Place your plugin JAR(s) in your workspace's deep store bucket, under a top-level `custom-plugins` directory: ``` s3:///custom-plugins/ ``` For example: ``` s3://startree-7wzfmix8b-pinot-fs/custom-plugins/my-plugin-1.0.0.jar ``` `` is specific to your workspace — check your workspace's Pinot deep store configuration if you don't already know the bucket name. **2. Name your JAR with a semantic version** JAR filenames must follow `-.jar`, for example: ``` my-plugin-1.0.0.jar ``` If you place more than one version of the same plugin in S3, StarTree Cloud loads **only the latest semantic version** — older versions are left in place but ignored, so you can keep them around for rollback. ## How it works * At startup, each Pinot component checks the `custom-plugins` S3 path and downloads any JARs it finds onto its local classpath. * **If the path is empty or doesn't exist, this is a no-op** — Pinot starts up exactly as it would without the feature enabled. * Custom plugin JARs are given **classloading priority over StarTree Cloud's built-in Pinot libraries**. If one of your classes shares a name with a built-in Pinot class, your version takes precedence — useful if you need to override built-in behavior, but worth knowing if you see unexpected behavior after adding a plugin. * All Pinot components load the same set of plugins. There is currently no way to target a plugin at only specific component types (for example, servers only). ## Failure handling A problem downloading or loading a custom plugin does **not** prevent a Pinot component from starting. The component comes up normally; functionality that specifically depends on the missing or broken plugin may be degraded (for example, a table configured to use a custom decoder may fail to ingest), but the rest of the deployment is unaffected. > [!TIP] > > If a plugin doesn't seem to be loading, check the logs of the **init container** on the affected Pinot pod — this is the container responsible for downloading plugin JARs from S3, and it logs the outcome of that download. ## Updating and rolling back plugins * **Updating a plugin:** Pinot does not pick up new or changed JARs at runtime. After uploading a new version to S3, restart the affected Pinot components to load it. * **Rolling back:** Remove or revert the JAR in S3, then restart the affected Pinot components. ## Responsibility and support * You are responsible for building, testing, and certifying your own plugin JARs. StarTree does not validate, sign, or certify custom plugins through its own CI or certification process — test thoroughly in a non-production environment first. * StarTree Cloud only enforces the semantic-versioning file naming convention described above; it does not perform checksum verification or JAR signing. ## Security Only JARs placed at your workspace's designated `custom-plugins` path in your deep store bucket are ever loaded. Plugins cannot be loaded from any other location. ## Current limitations * **No per-component targeting.** Plugins load on every Pinot component type; you can't restrict a plugin to, say, servers only. * **No hot reload.** Picking up a new or updated plugin requires restarting the affected Pinot components — there's no dynamic reload mechanism. * **No StarTree-side validation.** StarTree does not check, sign, or certify plugin JARs before they're loaded. # Scheduled Server Scaling Source: https://docs.startree.ai/corecapabilities/cluster-operations/scheduled-server-scaling > [!WARNING] > > This feature requires StarTree release 0.15.0 or later. Scheduled Server Scaling lets you automatically scale Apache Pinot server replica groups up or down on a fixed schedule. This is useful when your query load is predictable — for example, running fewer server replica groups overnight or on weekends to save cost, and restoring full capacity before peak hours. Scaling is driven by **cron schedules** defined per Pinot **tenant**. At each scheduled time, the operator asks the Pinot controller which servers to remove (or restore) to reach your target replica-group count, drains queries off the affected servers, and scales their StatefulSets accordingly. > [!NOTE] > > Scaling operates at the granularity of **replica groups**, not individual servers. You specify a target number of replica groups; the Pinot controller computes the exact set of servers to remove or restore to achieve that target. See [Replica Group based Workload Isolation](/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation) for background on replica groups and pools. > [!WARNING] > > Scaling down reduces redundancy for the tenant until the matching scale-up restores it. A schedule that targets `targetReplicaGroups: 1` leaves the tenant with **no replica group failover** for the duration of the scale-down window — if that single replica group has an issue, queries to the tenant's tables will fail. Choose a target that keeps the minimum redundancy your workload needs. ## Prerequisites * Your cluster is on **StarTree release 0.15.0 or later**. * Your cluster is managed by the StarTree Kubernetes operator via the `PinotCluster` custom resource. This feature is configured at the operator level by the StarTree Ops team — see [Enabling it](#enabling-it). * The tenant's tables already use **pool-based, replica-group-aware** instance assignment — see [Controller requirements for scale-down](#controller-requirements-for-scale-down) for the exact conditions the controller checks before it will scale down a tenant. ## How it works ```mermaid theme={null} flowchart TD Cron["Cron time fires\nfor a schedule"] --> Action{"action"} Action -->|SCHEDULED_SCALE_DOWN| AskDown["Ask Pinot controller which\nservers to remove\n(GET /serverReplicaGroupScaleDown)"] AskDown --> Drain["Server marks itself unavailable\nand drains in-flight queries\n(up to 10 minutes)"] Drain --> ScaleDownSS["Scale affected server\nStatefulSets to zero"] Action -->|SCHEDULED_SCALE_UP| AskUp["Determine which previously-removed\nservers must come back"] AskUp --> ScaleUpSS["Restore those server\nStatefulSets"] Missed["Operator was down\nat cron time"] -.->|restarts within\nmissedExecutionWindow| Action Missed -.->|restarts after\nmissedExecutionWindow| Skip["Run is skipped"] ``` 1. Scheduled scaling is enabled in the `PinotCluster` spec under the `server` component, with one or more schedules defined per tenant (see [Enabling it](#enabling-it)). 2. The operator creates and maintains a `ScheduledServerScaling` custom resource for each tenant you configure. 3. At each schedule's cron time: * **Scale down** — the operator calls the Pinot controller to determine which servers can be removed to reach the target replica-group count. Each affected server marks itself unavailable and drains its in-flight queries (up to 10 minutes) before its StatefulSet is scaled to zero. * **Scale up** — the operator determines which previously-removed servers must come back to reach the target, and restores their StatefulSets. 4. If the operator was down when a schedule fired, it still executes the missed run as long as it restarts within the `missedExecutionWindow`. After that window passes, the missed run is skipped until the next occurrence. While a scale operation is in progress, the operator suppresses normal replica reconciliation for the affected servers, so a manually-running cluster reconcile will not fight the schedule. ## Enabling it Scheduled scaling is configured at the Kubernetes operator level, via the `PinotCluster` custom resource. This isn't currently exposed in the Data Portal or any self-serve UI, so **contact the StarTree Ops team** to have it enabled for your cluster. > [!TIP] > > Self-serve configuration of scheduled scaling from the Cloud Console is coming soon. To enable it, share the following with StarTree Ops for each tenant you want to schedule: | Input | Description | | :- | :- | | Tenant name | The Pinot server tenant to schedule (e.g. `DefaultTenant`). | | Schedules | One or more `{name, action, cron, targetReplicaGroups}` combinations — see the [configuration reference](#configuration-reference) below for what each field means. | | Desired replica-group counts | The `targetReplicaGroups` to scale down to (off-peak) and scale back up to (peak). | | Cron times | When each scale-down/scale-up should fire, in **UTC**. | To change or disable a schedule, or to remove scheduled scaling from a tenant altogether, reach out to StarTree Ops with the updated details. When a tenant is removed from the spec, the operator deletes its `ScheduledServerScaling` resource and restores any servers that were left scaled down. ## Configuration reference `scheduledScaling` (under `spec.components.server`): | Field | Type | Description | | :- | :- | :- | | `enabled` | boolean | Master switch for scheduled scaling on this cluster. | | `tenants` | array | One entry per Pinot tenant you want to schedule. Each has `tenant` + `schedules`. | Each entry in `tenants`: | Field | Type | Description | | :- | :- | :- | | `tenant` | string | Pinot server tenant name (e.g. `DefaultTenant`). | | `schedules` | array | One or more scheduled actions for this tenant. | Each entry in `schedules`: | Field | Type | Required | Description | | :- | :- | :- | :- | | `name` | string | yes | A label for the schedule, used in logs and status. | | `action` | enum | yes | `SCHEDULED_SCALE_DOWN` or `SCHEDULED_SCALE_UP`. | | `cron` | string | yes | 5-field Unix cron expression (`minute hour day-of-month month day-of-week`), evaluated in **UTC**. See examples below. | | `targetReplicaGroups` | integer | yes | Desired number of server replica groups for the tenant **after** this action completes. Must be greater than 0. | | `missedExecutionWindow` | string | no | Grace period after the scheduled time during which a missed run still executes if the operator was down (e.g. `30m`, `1h`). Defaults to `15m`. | > [!NOTE] > > Before a server is scaled down, it marks itself unavailable and drains its in-flight queries, for up to **10 minutes**, before its StatefulSet is scaled to zero. This drain timeout is fixed and isn't currently configurable. ### Cron format Schedules use standard **5-field Unix cron** (`minute hour day-of-month month day-of-week`). All times are evaluated in **UTC** — convert your local schedule to UTC before setting `cron`. | Expression | Meaning (UTC) | | :- | :- | | `0 22 * * *` | Every day at 22:00 UTC | | `0 6 * * 1-5` | Weekdays at 06:00 UTC | | `0 0 * * 0` | Every Sunday at midnight UTC | | `30 18 * * 5` | Every Friday at 18:30 UTC | ## Typical pattern: nightly scale-down, morning scale-up Pair a `SCHEDULED_SCALE_DOWN` with a `SCHEDULED_SCALE_UP` to shrink capacity during off-hours and restore it before peak load: * **Scale down** at night to a low `targetReplicaGroups` (e.g. `1`). * **Scale up** in the morning back to your full `targetReplicaGroups` (e.g. `3`). Use the same tenant for both schedules. The scale-up restores exactly the servers that the matching scale-down removed. ## Checking status The operator tracks each tenant's scaling state in the `ScheduledServerScaling` resource: ```bash theme={null} kubectl get scheduledserverscaling -n kubectl get scheduledserverscaling -n -o yaml ``` Key status fields: | Field | Description | | :- | :- | | `status.state` | `AVAILABLE`, `DELETING` (scale-down in progress), `DELETED` (scaled down), `RESTORING` (scale-up in progress), `RESTORED`. | | `status.lastScalingOperation.action` | The most recent action: `SCHEDULED_SCALE_DOWN` or `SCHEDULED_SCALE_UP`. | | `status.lastScalingOperation.lastOccurrence` | Timestamp of the cron fire time that triggered the operation. | | `status.lastScalingOperation.serversAffected` | List of server StatefulSets affected by the operation. | | `status.lastScalingOperation.targetReplicaGroups` | The target replica-group count of the last completed operation. | ## Notes and limitations * `targetReplicaGroups` must be **greater than 0** — a schedule cannot remove every replica group. * If `targetReplicaGroups` equals the tenant's current total replica groups, a scale-down is a no-op (nothing to remove). * Cron times are evaluated in **UTC** — convert your local time before setting `cron`. * Scheduled scaling only affects **server** StatefulSets for the named tenant. It does not change Zookeeper, Controller, Broker, or Minion components. ## Controller requirements for scale-down At each scale-down, the operator asks the Pinot controller (`GET /serverReplicaGroupScaleDown`) which servers can be removed for the tenant. The controller only returns a server list when **all** of the following hold. If any fails, the scale-down is rejected and no servers are removed — fix the underlying condition and the next scheduled run (or a manual retry) will proceed. **Tenant and topology** * The tenant exists and has servers tagged for it (`_OFFLINE` / `_REALTIME`). * **Servers are exclusive to the tenant** — no server tagged for this tenant may also carry tags for another tenant. * Every tenant server has a **pool assignment**, and each pool matches at least one of the tenant's tags. * If a server has both OFFLINE and REALTIME tags, both must point to the **same pool number**. **Target value** * `targetReplicaGroups` must be **≥ 1** and **≤ the current number of replica groups (pools)**. * If `targetReplicaGroups` equals the current replica-group count, the call succeeds but returns **no servers** (nothing to remove). **Table configuration (when the tenant has tables)** * No table may use `instancePartitionsMap` (which bypasses pool-based assignment). * All non-dimension tables must use **pool-based, replica-group-aware** instance assignment. * Each table's configured `numReplicaGroups` (when non-zero) must **equal the current pool count**. **No rebalance in progress** * No table in the tenant may have an active or failed **table rebalance** job. * The tenant may not have an active, aborted, cancelled, or unscheduled **tenant rebalance** job. > [!NOTE] > > The controller evaluates these against a point-in-time snapshot of cluster state. The highest numbered pools are always selected for removal, so a given `(tenant, targetReplicaGroups)` request is deterministic. ## Troubleshooting A scale-down is rejected, with no servers removed, whenever one of the [controller requirements](#controller-requirements-for-scale-down) above isn't met. Fix the underlying condition — the next scheduled run (or a manual retry) will proceed once it's resolved. | Symptom | Likely cause | What to do | | :- | :- | :- | | Scale-down never removes any servers | A server tagged for the tenant is also tagged for another tenant | Make servers exclusive to one tenant, or scope the schedule to a tenant that already has dedicated servers. | | Scale-down never removes any servers | A server is missing a pool assignment, or OFFLINE/REALTIME tags point to different pools | Fix the server's pool assignment so it matches the tenant's tags, and align OFFLINE/REALTIME pool numbers. | | Scale-down never removes any servers | `targetReplicaGroups` is greater than the tenant's current replica-group (pool) count | Lower `targetReplicaGroups` to at most the current pool count. | | Scale-down never removes any servers | A table uses `instancePartitionsMap` | Migrate the table to pool-based instance assignment before scheduling scale-down for its tenant. | | Scale-down never removes any servers | A table's `numReplicaGroups` doesn't match the current pool count | Update the table's replica-group config to match the tenant's current pool count. | | Scale-down never removes any servers | A table or tenant rebalance is active, aborted, cancelled, or unscheduled | Let the rebalance finish (or clear the stuck job), then wait for the next scheduled run or retry manually. | | Scale-up doesn't fully restore capacity | The matching scale-down removed a different set of servers than expected | Check `status.lastScalingOperation.serversAffected` on the `ScheduledServerScaling` resource for both operations to confirm which servers were removed vs. restored. | | A schedule didn't run at all | The operator was down past `missedExecutionWindow` at the cron time | Increase `missedExecutionWindow` if operator restarts routinely take longer than the current setting, or trigger the action manually. | ## Related * [Replica Group based Workload Isolation](/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation) — background on replica groups and pool-based instance assignment. * [Health checks](/corecapabilities/observability/health-checks) — check overall cluster health, including the replication and instance-pool checks relevant to scale-down eligibility. # Troubleshooting resource pressure Source: https://docs.startree.ai/corecapabilities/cluster-operations/troubleshooting Diagnose out-of-memory conditions, high CPU, disk pressure, and degradation after an upgrade or a traffic peak. This guide covers the cluster running out of something: memory, CPU, or disk. These problems announce themselves indirectly — as query timeouts, ingestion lag, or tasks failing in unrelated places — so the first job is recognising that resource pressure is what you are looking at. > [!TIP] > > If StarTree support named an alert, such as `KubernetesPinotContainerOOMKilled` or `PinotHighGcPercent`, look it up in [Platform health alerts](/corecapabilities/observability/alerts/platform-health) first. Disk and key limits are in [Data availability and storage alerts](/corecapabilities/observability/alerts/data-availability-and-storage). ## Recognising it Resource pressure is the likely cause when: * **Several unrelated things degrade at once.** Queries slow, ingestion falls behind, and tasks start failing, all within the same window. A single bad query does not do this. * **Symptoms correlate with load rather than with a specific query or table.** The same query is fast at 03:00 and slow at 14:00. * **Failures cluster in time.** Garbage-collection pauses and restarts produce bursts of timeouts separated by normal periods, rather than a steady failure rate. * **A component restarted.** Anything that was killed and came back was almost certainly killed for consuming too much of something. ## Identify which component and which resource Both halves matter. "The cluster is slow" is not diagnosable; "brokers are fine, servers on one tenant are at 90% heap" is. **1. Narrow to a component** | Component | Symptom when under pressure | | - | - | | **Server** | Query timeouts, slow scans, ingestion lag on tables it hosts, segments going unavailable | | **Broker** | Timeouts affecting all tables equally, planning-phase failures, connection errors | | **Controller** | Table operations hang or fail, segment state stops converging, tasks stop being scheduled | | **Minion** | Batch and ingestion tasks fail in unrelated groups, tasks restart repeatedly | Controllers and minions are easy to overlook because their symptoms surface elsewhere. If tables are healthy but nothing is *changing* — no new segments, no task progress — look at the controller. **2. Narrow to a resource** * **Memory** — restarts, out-of-memory failures, pauses that look like timeouts, degradation that worsens gradually and resets on restart. * **CPU** — latency that scales with load, no restarts, scheduler wait time rising while execution time stays flat. * **Disk** — segment operations failing, ingestion stalling on commit, errors mentioning storage rather than compute. Memory and CPU are frequently confused because both produce slow queries. The distinguishing signal is restarts: a component that keeps restarting is running out of memory. ## Out-of-memory conditions **1. Establish when it started and what changed** Out-of-memory conditions almost always follow a change. Check for: * A version upgrade — memory characteristics change between versions * A new table, especially an upsert or dedup table * Growth in an existing upsert table's primary-key count * A change in query shape — new aggregations, larger result sets, joins that were not there before * A change in data volume or retention **2. Check upsert and dedup memory** Upsert and dedup tables hold per-primary-key state in memory, and that state grows with distinct keys rather than with rows. A table that was comfortable at launch can exhaust memory months later with no change to its config. Pinot exposes an API to estimate memory usage for an upsert table — use it to size before creating one, and to check what an existing one should be consuming against what it is. Consider off-heap options where they suit your workload: see [Off-heap upsert](/corecapabilities/manage-data/offheap-upsert) and [Off-heap dedup](/corecapabilities/manage-data/offheap-dedup). **3. Check whether queries are driving it** Queries with large intermediate results — high-cardinality group-bys, large joins, big `DISTINCT` sets — allocate heavily. In `system_query_log`: ```sql theme={null} SELECT logTimestampMs, tableName, timeUsedMs, offlineTotalMemAllocatedBytes + realtimeTotalMemAllocatedBytes AS memAllocated, numGroupsLimitReached, maxRowsInJoinReached, maxRowsInDistinctReached, query FROM system_query_log WHERE logTimestampMs > now() - 3600000 ORDER BY memAllocated DESC LIMIT 20 ``` Queries hitting the limit flags are being trimmed — they are consuming as much as they are allowed to, which suggests they would consume more if permitted. **4. Consider workload isolation** If analytical queries and latency-sensitive queries share the same servers, the analytical ones will periodically starve the others. StarTree supports isolating workloads so one class cannot exhaust the resources the other depends on — see [Replica group based workload isolation](/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation) and the [memory-throttled scheduler](/corecapabilities/query_data/advanced_operations/workload-isolation/memory-throttled-scheduler). ## High CPU **1. Establish whether it is query-driven** Rank recent queries by CPU: ```sql theme={null} SELECT queryHash, COUNT(*) AS executions, AVG(timeUsedMs) AS avgLatencyMs, SUM(offlineTotalCpuTimeNs + realtimeTotalCpuTimeNs) AS totalCpuNs, ANY_VALUE(query) AS sampleQuery FROM system_query_log WHERE logTimestampMs > now() - 3600000 GROUP BY queryHash ORDER BY totalCpuNs DESC LIMIT 20 ``` Aggregate CPU matters more than per-query CPU. A cheap query running constantly costs more than an expensive one running hourly, and only the grouped view shows that. **2. Check whether the work is necessary** High CPU with high `numEntriesScannedInFilter` means the cluster is scanning data it should be skipping. That is an indexing problem wearing a capacity problem's clothes, and adding servers will not fix it. See [Queries failing or slow](/corecapabilities/query_data/troubleshooting#timeouts) and run [Query Analyzer](/corecapabilities/query_data/query-analyzer) on the worst offenders. **3. Check for background work** Rebalances, segment reloads, merges and purge tasks all consume CPU on the same servers that answer queries. If CPU rose without a change in query volume, check whether a maintenance operation is running. Check `TABLE_SEGMENT_ASSIGNMENT_CHECK` and `TABLE_SEGMENTS_RELOAD_CHECK` on the Health Dashboard. ## Disk and storage * **Check what is actually stored.** The table Storage view in the Data Portal breaks down deep, local and tiered storage. See also [Table storage usage](/corecapabilities/manage-data/table-storage-usage). * **Check retention is running.** `SEGMENT_RETENTION_CHECK` on the Health Dashboard. Data past retention that has not been removed consumes space it should not. * **Check for segments that should not be there.** Failed or interrupted operations can leave segments behind that are no longer referenced but still occupy storage. If your segment count is much higher than your data volume suggests, this is worth raising. * **Check whether you are near a segment-count limit.** [Segment count limits](/corecapabilities/manage-data/recipes/segment-count-limits#monitoring-and-troubleshooting) shows how to count where a table stands and what the cluster-wide picture looks like. * **Check segment sizing.** `SEGMENT_SIZE_CHECK` and `SEGMENT_COUNT_CHECK` flag tables whose segments are too small or too numerous. Many small segments cost more than the same data in fewer, larger ones — in memory, in query fan-out, and in metadata overhead. See [Merge small segments](/recipes/merge-small-segments). * **Consider tiered storage.** If most of your data is queried rarely, moving it to a cheaper tier reduces local pressure. See [Cloud tiered storage](/corecapabilities/manage-data/set-up-tiered-storage/motivation). ## Degradation after an upgrade Several resource problems appear immediately after a version upgrade. When that is the sequence: **1. State the correlation explicitly** "This started after the upgrade" is a diagnosis-shortening fact. Note the previous version, the new version, and the exact time the symptom began. **2. Check the release notes for the versions you crossed** See [StarTree release notes](/reference/startree-release-notes). If you skipped versions, read all of them — behaviour changes accumulate. **3. Check segments that need attention** Upgrades can leave segments requiring reload before they behave as expected, and occasionally leave segments in an error state. Check [`TABLE_SEGMENTS_RELOAD_CHECK`](/corecapabilities/observability/health-checks#table_segments_reload_check) and [`IDEAL_STATE_EV_MISMATCH_CHECK`](/corecapabilities/observability/health-checks#ideal_state_ev_mismatch_check) on the Health Dashboard immediately after any upgrade. **4. Compare query plans, not just latencies** A query that got slower after an upgrade may be planning differently. Compare scan counts before and after — if `numEntriesScannedInFilter` changed materially for the same query, the plan changed, and that is a more specific thing to report than "it got slower." > [!TIP] > > Capture a baseline before your next upgrade: p95 latency and scan counts for a handful of representative queries, current segment and document counts per table, and steady-state resource utilisation. Ten minutes beforehand turns "something feels slower" into a measurement. ## Planning capacity To avoid the reactive version of all of the above: * **Size before you create.** Estimate storage, and for upsert tables estimate primary-key memory, before adding a table to a tenant rather than after. * **Know your headroom before a peak.** If you have a known traffic event, check utilisation against it in advance — scaling ahead of a peak is routine, scaling during one is not. * **Watch upsert key growth.** It grows with distinct keys, not rows, and it does not shrink on its own. * **Revisit sizing after workload changes.** New dashboards, new consumers and new query patterns change resource profiles without changing any config you own. See [Scheduled server scaling](/corecapabilities/cluster-operations/scheduled-server-scaling) for scaling around predictable load, and [Query quotas](/corecapabilities/query_data/advanced_operations/query-quotas) for capping what any single workload can consume. ## Feature-specific guidance For scaling operations that did not take effect, see [Scheduled server scaling → Troubleshooting](/corecapabilities/cluster-operations/scheduled-server-scaling#troubleshooting). External tables have their own operational failure modes: [External table troubleshooting](/corecapabilities/external-table/troubleshooting#queries) covers server OOM and pod restarts under query load, and [Operational guidance](/corecapabilities/external-table/operational-guidance#additional-troubleshooting) covers ZooKeeper packet-size limits as segment count grows. More at [Troubleshooting by feature](/corecapabilities/observability/troubleshooting-by-feature). ## Escalating Include: which component, which resource, when it began, what changed, whether anything restarted, and your current tenant sizing. If queries are implicated, attach the CPU or memory ranking from `system_query_log`. See [what to collect](/corecapabilities/observability/troubleshooting#before-you-open-a-support-ticket). # Best Practices and Configs Source: https://docs.startree.ai/corecapabilities/external-table/best-practices-and-configs The full config reference for External Tables — sync task, tier caching, and per-query options — plus onboarding and performance best practices. > [!WARNING] > > This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it. Best practices for running External Tables, followed by the full reference for every configuration and query option: the table-level sync config, the S3 tier and caching properties, and the per-query `SET` options. > [!NOTE] > > Two 0.16.0 features have their own dedicated config reference: [Large External Tables](./segment-groups) (`segmentGroupConfig`) and [Deletion Vectors](./deletion-vectors) (`enableDeletionVectors` and snapshot readiness). ## Best practices **Onboarding** * Enter the **bucket name only** (no `s3://`) and set the AWS **region in the connection config** — a region in config means you don't depend on the `AWS_REGION` environment variable. * Grant the cluster access to the source bucket with an **assumed IAM role** (`roleArn` + optional `externalId`) or the cluster's node role; verify with `aws s3 ls s3:////` before onboarding. * Keep the source bucket **in the same AWS region as your cluster**. Cross-region reads without an S3 interface endpoint go through the NAT Gateway and can add large AWS charges; see [Networking and data transfer costs](#networking-and-data-transfer-costs). * For very large sources, **start with a smaller prefix subset** and scale the cluster before onboarding the full dataset. * Set `continueOnFileError=true` if a sync run should **skip a file that fails to process** — unreadable, corrupt, or any other per-file failure, not just unreadable ones — instead of failing the whole run (default is to fail). Because it skips on *any* failure, it can silently drop more data than "unreadable file" suggests, so review sync logs after enabling it. This only affects files *during a sync* — it does not protect an already-ingested segment whose file is later deleted or compacted away (see the note below). * **If your source compacts or rewrites files, onboard it through an Iceberg REST catalog, not the raw S3/GCS Data Lake source** — a REST catalog reconciles compaction via snapshot metadata, while raw sources pin segments to physical file paths that break when files are rewritten. See [Segments fail to load after upstream compaction](./troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction). **Schema** * **Keep the data types** the preview step infers; overriding them can break segment generation. * Set a **time column** where the data has one — it enables time-based segment pruning and retention. Recommended but not mandatory; tables without a time column still work. **Indexes** * Add indexes for the columns you **filter and group by** — without them, queries scan remote data. * For dictionary-backed indexes (inverted, FST, IFST), keep the forward index `RAW` and add an explicit **`dictionary` block**. * **Avoid grouping by derived/computed columns** — it defeats segment pruning. **Tiered storage — data & index caching** * Enable the page cache (`enable.prefetch.page.cache`, a per-table tier setting) so index data is served locally. See [Caching and index pinning](#caching-and-index-pinning). * **Pin small, hot indexes** (bloom filter, dictionary) with `preload.index.keys.override`; let the index cache serve the larger/colder ones. * Turn on **index consolidation** to collapse a segment's per-column index files into one mmap, avoiding the OS `max_map_count` limit on wide tables — see [Caching and index pinning](#caching-and-index-pinning). **Operations** * Let the controller watcher schedule syncs; monitor health with the [observability endpoints](./observability). *** ## Networking and data transfer costs External Tables and batch ingestion read data directly from S3, often in large volumes. Where the bucket sits relative to your StarTree cluster decides the network path those reads take, and that path decides what you pay AWS. This matters most for **BYOC** deployments, where the network charges land on your own AWS account. | Bucket location | Network path | What you pay | | - | - | - | | **Same region as the cluster** | S3 gateway endpoint (StarTree creates it by default for BYOC on AWS) | No NAT Gateway or endpoint charges | | **Different region, no endpoint** | Node → NAT Gateway → Internet Gateway → public S3 endpoint | NAT Gateway per-GB processing **plus** inter-region data transfer. This can be expensive for large datasets | | **Different region, cross-region interface endpoint** | AWS PrivateLink to S3 in the source region | Interface endpoint hourly + per-GB processing **plus** inter-region data transfer. Usually much cheaper than NAT | **Recommendations** * **Co-locate the data and the cluster.** Keep source buckets in the same region as your StarTree cluster whenever you can. It is the cheapest and fastest setup. * **Weigh how often the data is read.** For a one-time read, such as a single backfill, moving the data next to the cluster may not save much. Repeated reads of a large dataset are different: External Tables query S3 again and again, and the per-GB charges add up quickly. * **Check the bucket region before onboarding.** Compare it with your cluster's region (`aws s3api get-bucket-location --bucket `). A gateway endpoint only covers buckets in its own region. * **For cross-region buckets, add an S3 interface endpoint before you onboard.** Create a cross-region interface endpoint for S3 in your cluster's VPC so reads skip the NAT Gateway. See the AWS guides for [gateway endpoints for S3](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) and [cross-region PrivateLink support](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html). Contact StarTree support if you need help with the setup. * **Watch NAT Gateway traffic during the first sync.** A spike in the NAT Gateway's CloudWatch byte counters during an initial onboarding usually means S3 reads are leaving the private network. *** ## Caching and index pinning External Table queries read indexes from remote storage, so keeping index data local is what makes them fast. There are two ways to do that — **pinning** and the **index cache** — and they're used together. | | Pinning (preload) | Index cache | | - | - | - | | Enabled by | `preload.index.keys.override` (table setting) | `enable.prefetch.page.cache=true` (table setting) | | What it does | Downloads the listed index files into local mmap at segment load and keeps them **resident** | Serves index byte ranges through an **on-demand, evictable** (LRU) cache | | When it loads | Eagerly, at segment load | Lazily, on first access | | Eviction | Never — always local | Evicted under memory pressure | | Memory cost | Fixed and predictable | Elastic, bounded by the cache size | **Pin the small, always-needed indexes; let the cache handle the rest.** Pinning is resident memory, so reserve it for compact indexes you hit on most queries — **bloom filters, dictionaries**, and small range/inverted indexes. Everything larger or colder (JSON, text, wide columns) should go through the index cache, which fetches on demand and evicts what isn't used. > [!WARNING] > > `preload.enable` is **not a table config key**. Putting it in `tierBackendProperties` is silently accepted and does nothing; StarTree's own onboarding flow currently writes it into new table configs, where it has no effect. Preloading is enabled server-side by StarTree, and `preload.index.keys.override` chooses what to pin. ```json theme={null} "tierBackendProperties": { "preload.index.keys.override": "*.bloom_filter,*.dictionary,*.range_index", "enable.prefetch.page.cache": "true" } ``` Pin keys are `.` (or `*` for all columns), where `indexKey` is the index's file key — e.g. `inverted_index`, `range_index`, `bloom_filter`, or `dictionary`. Examples: `payment_type.inverted_index`, `*.bloom_filter`, `*.dictionary`. **Index consolidation.** On a wide table, each column's index is a separate memory-mapped file, and many segments × many columns can exhaust the OS memory-map limit (`max_map_count`) — surfacing as native-memory / mmap allocation failures and server restarts. Set `preload.enable.index.consolidation=true` to pack a segment's index regions into a single file per segment, drastically cutting the mmap count. Onboarding enables this by default. *** ## Where configs live | Scope | Set in | Applies to | | - | - | - | | **Table config — `ExternalTableSyncTask`** | `task.taskTypeConfigsMap.ExternalTableSyncTask` in the table config | Catalog connection, scheduling, snapshot processing. | | **Tier backend properties** | `tierConfigs[].tierBackendProperties` (S3 tier) | Per-table caching and preload behavior. | | **Query option** | `SET key = 'value'` per query | Per-query cache/prefetch behavior. | *** ## Table config — `ExternalTableSyncTask` ### Catalog & mode | Key | Default | Description | | - | - | - | | `catalogType` | *(required)* | `s3`, `iceberg-rest`, or `gcs-interop`. Selects the catalog provider. | | `executor` | *(none — omitting it silently falls back to the deprecated minion path)* | **Always set `controller`.** The minion executor path is legacy and unsupported for the current feature set — snapshot processing, the observability endpoints, ingestion-time segment grouping, and IAM assume-role auth all require the controller path. There is no default that pushes new tables toward `controller`: a hand-written config that omits this key runs on minion without any error or warning. Onboarding sets it automatically, so this only bites hand-written configs. | | `enabled` | `true` | Enable/disable the per-table sync. | | `forceConfig` | `false` | Bypass external-table config validation **errors** (RAW encoding, unsupported index types, transform configs) and proceed anyway. Use only when you understand why validation is failing. | > [!NOTE] > > External Tables read **Parquet files only**. There is no `inputFormat` config — the sync engine doesn't read one, so setting any such key has no effect. > [!WARNING] > > `glue` is also accepted as a `catalogType` value when a table is created, but sync then fails at runtime — it is not a supported catalog type. Onboard AWS Glue-backed sources with `catalogType=iceberg-rest` and the Glue service type under `catalog.iceberg-rest.*` instead; see [AWS Glue: Onboarding via API](/corecapabilities/external-table/glue/onboarding-api#step-1-validate-and-browse-the-connection). ### Connection keys The catalog connection keys are prefixed `catalog.s3.*` or `catalog.iceberg-rest.*` (REST URI, service type — `glue`, `s3Tables`, `unity`, or `rest` (Nessie and other generic Iceberg REST catalogs) — warehouse/prefix, namespace/table, and `auth.rest.*` / `auth.storage.*` credentials). The full list with examples is in [AWS Glue: Onboarding via API](/corecapabilities/external-table/glue/onboarding-api#step-1-validate-and-browse-the-connection), [Unity Catalog: Onboarding via API](/corecapabilities/external-table/unity/onboarding-api#step-1-validate-and-browse-the-connection) for Unity-specific auth, or [Nessie: Onboarding via API](/corecapabilities/external-table/nessie/onboarding-api#step-1-validate-and-browse-the-connection) for the generic `rest` adapter. Two further top-level keys select source storage independently of `catalogType`: `sourceStorageProvider` (`azure`, for an Azure Blob/ADLS source instead of S3) and `externalTableAccessPolicy` (`unity` for Unity-vended Azure credentials, or `azure-sas` for a configured SAS token via `catalog.{catalogType}.auth.storage.sasToken` / `.credentialPrefix`). Hive Metastore and Unity Catalog / generic REST catalogs accept both; AWS Glue and S3 Tables don't. Onboarded today only through [SQL DDL](./sql-ddl#azure-source-storage) — not the Data Portal or REST onboarding flow. ### Scheduling & task sizing | Key | Default | Description | | - | - | - | | `schedule` | — | Quartz cron for sync frequency, e.g. `0 */5 * * * ?`. | | `desiredSegmentSize` | `500MB` | Reserved for future use — currently has no effect. Each sync run generates one Pinot segment per source Parquet file; there is no size-based segment splitting or merging today. | | `taskMaxNumFiles` | `100` | Max source files per task run (minion mode). | | `taskMaxDataSize` | `1GB` | Max source bytes per task run (minion mode). | | `tableMaxNumTasks` | *(unbounded)* | Not read by the External Table task generator itself; on ad-hoc `/tasks/schedule` triggers the shared task framework applies it, with an effective default of unbounded. (The Data Export Task reads the same key name with a different default — `10000`.) | | `maxParallelOps` | *(CPU cores)* | Parallelism within a single run. | | `continueOnFileError` | `false` | Skip an entire file that fails to process, **for any reason** — not limited to unreadable files — instead of failing the run. Because it's not scoped to a specific failure type, it can silently drop more data than "skip an unreadable file" implies. Applies only *during a sync run* — see the note below. Distinct from `continueOnError`. | | `catalog.s3.startAfter` | *(unset)* | New in 0.16.0. Seeds the lower bound for the **very first** S3 listing call (`ListObjectsV2`'s `startAfter`), as a bucket-relative object key. Only used before any checkpoint exists — once a checkpoint is recorded, it's permanently ignored. Takes precedence over `catalog.s3.startFromLexLatestPartition`. | | `catalog.s3.maxFilesToList` | `1000` | Max objects fetched per listing call for raw S3/GCS Data Lake sources (the per-run batching knob for `catalogType=s3`/`gcs-interop`; the Iceberg path batches by `iceberg.snapshotProcessing.batchSize` instead). | | `catalog.s3.startFromLexLatestPartition` | `false` | Start the first listing from the lexicographically latest partition instead of the beginning of the prefix. Ignored when `catalog.s3.startAfter` is set. | > [!WARNING] > > Don't use `catalog.s3.prefix` as a way to bound where ingestion *starts* within a large bucket — `prefix` is a permanent filter on which objects are ever considered, not a one-time starting cursor. Narrowing it to "skip ahead" permanently excludes everything outside that range, including future files, once the checkpoint reaches the end of the prefix. Use `catalog.s3.startAfter` instead, which only affects the first run. > [!NOTE] > > **`continueOnFileError` vs. `continueOnError` — two different configs.** `continueOnFileError` (this `ExternalTableSyncTask` config) skips an **entire file that fails to process, for any reason** — not just unreadable ones — so one bad file doesn't fail the run. `continueOnError` (an OSS field in the table's `ingestionConfig`, not here) skips **individual bad rows** while reading a file or stream. Both apply only *during ingestion* — neither one protects a segment that is already ingested. In particular, if a source file is deleted or [compacted away upstream](./troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction) after its segment exists, the server fails to serve that segment regardless of either flag; the durable fix for compacting sources is an Iceberg REST catalog. ### Snapshot processing (Iceberg) | Key | Default | Description | | - | - | - | | `iceberg.snapshotProcessing.initialVersionSelector` | `LATEST` | Where to start: `LATEST`, `EARLIEST`, or `SNAPSHOT_ID`. | | `iceberg.snapshotProcessing.initialSnapshotId` | — | Snapshot ID to start from (when selector is `SNAPSHOT_ID`). | | `iceberg.snapshotProcessing.batchSize` | `10000` | Max new files ingested per run. | | `iceberg.snapshotProcessing.maxFilesScannedPerRun` | `100000` | Upper bound on files enumerated per run (≥ `batchSize`). | | `iceberg.snapshotProcessing.maxRetries` | `3` | Consecutive failures before a snapshot is marked `FAILED`. | | `iceberg.snapshotProcessing.retryFailedSnapshot` | `false` | Keep re-arming a `FAILED` snapshot from its last cursor on every cycle instead of abandoning it after `maxRetries`. See [Sync says COMPLETED but no data lands](./troubleshooting#sync-says-completed-but-no-data-lands). | | `iceberg.snapshotProcessing.lagMaxDiffFiles` | `100000` | Cap on the manifest set-diff used by the exact ingestion-lag calculation (`?includeLag=true&pendingExact=true`). Beyond it, lag degrades to summary-based estimates with an explanatory `message`. Only affects the numbers returned by the [lag/status endpoint](./observability#ingestion-lag) — it has no effect on ingestion itself. | ### Checkpointing & schema evolution | Key | Default | Description | | - | - | - | | `enableCheckpointing` | *(always on)* | Declared but not currently read — checkpointing is unconditionally enabled for `ExternalTableSyncTask`; this key has no effect. | | `checkpointTTLSec` | *(unused)* | Declared but not currently read for this task type. | | `schemaEvolution.enabled` | `false` | Refine the Pinot schema to the source schema before each sync. | | `schemaEvolution.failFast` | `false` | Abort (vs. log and continue) when a schema change is rejected. | | `schemaEvolution.typeWidening.enabled` | `false` | Apply a supported source-side type widening (`INT`→`LONG` or `FLOAT`→`DOUBLE`) instead of rejecting it. Requires `schemaEvolution.enabled=true`; takes effect on the periodic-watcher path only (not snapshot-sync). See the note below. | | `storeNestedSchema.enabled` | `true` | Stamp the source's nested-schema metadata onto complex (JSON) columns so field names **inside** `struct` / `array` / `map<_,struct>` values resolve to your logical names. Only matters for Delta UniForm / Unity Catalog tables that use column mapping — see the note below. | > [!NOTE] > > **How schema evolution resolves changes (0.16.0+).** When `schemaEvolution.enabled=true` on a `catalogType=iceberg-rest` source, each sync unions the current schema against the source's Iceberg field-id/alias history instead of matching by column name alone. This means a **source column rename is tracked as an alias** rather than dropping and re-adding a column — the existing Pinot column name is preserved. Resolution only covers **top-level** fields (nested-field renames aren't resolved). A detected **type widening** is rejected by default, unless `schemaEvolution.typeWidening.enabled=true` opts the table into applying it instead — set `schemaEvolution.failFast=true` if you want a rejection (widening or otherwise) to abort the run instead of being logged and skipped. When fail-fast aborts a run, the [status API](./observability#run-status) reports `failurePhase: UPDATING_SCHEMA`. > > **Type widening (opt-in).** `schemaEvolution.typeWidening.enabled=true` lets a table adopt a source-side widening instead of rejecting it — `INT`→`LONG` and `FLOAT`→`DOUBLE` are the supported widenings; anything else is still rejected. It applies only on the **periodic-watcher path** (an `executor=controller, watcher.enabled=true` table) — snapshot-sync ingestion skips schema evolution entirely and keeps the registered schema unchanged. Widening is safe for External Tables specifically: a segment's forward index *is* the backing Parquet file, so adopting the wider type is a metadata-only edit, not a physical rewrite. It defaults to off because applying it opens a transient window, until every segment has reloaded onto the new type, where a query spanning both generations of segments on the widened column can fail (`ORDER BY` / `GROUP BY` / `DISTINCT`) or drop mismatching rows (a plain `SELECT`) — aggregations like `SUM` / `COUNT` are unaffected. Enabling it is a deliberate choice to accept that window during rollout. > > `FLOAT`→`DOUBLE` is lossless but **changes what already-ingested rows print**: the float nearest `0.1` reads back as `0.10000000149011612` once served as a double, because that is the exact value the 4-byte encoding always held — only the rendering changes, and the value narrows back to the same float. An Iceberg-native engine over the same table prints the same digits, since Iceberg performs the identical JVM widening. > > **Decimal precision widening needs no flag.** Iceberg permits only a precision increase at fixed scale, and `BIG_DECIMAL` (Pinot's only decimal type) carries no precision or scale of its own, so a decimal precision change produces an identical registered schema — there's no diff for `schemaEvolution.typeWidening` to force or reject. The physical Parquet encoding still varies by precision (`decimal(9,2)` in an `INT32`, `decimal(18,2)` in an `INT64`, wider in a `FIXED_LEN_BYTE_ARRAY`), and a widened table can hold a mix; each file's own footer resolves the scale on read. > [!NOTE] > > **Nested logical names under column mapping (`storeNestedSchema.enabled`).** Delta UniForm tables created with `delta.columnMapping.mode` (`name` or `id`) don't store your column names in the Parquet files — each physical column is an opaque `col-`, and the logical name plus a stable field-id live only in table metadata. **Top-level** column names are always resolved back to logical names, with or without this flag. Names **inside** a complex value are only resolved when `storeNestedSchema.enabled` is on (the default) for the table's `ExternalTableSyncTask` config; set it to `false` to opt out, and nested keys are then stored under their physical names, so `JSON_EXTRACT_SCALAR(customer, '$.name', ...)` returns `NULL` even though the data is there. Only an explicit, case-insensitive `false` opts out — an unset, blank, or unrecognized value keeps the default on. > > The flag applies to every schema-writing path (onboarding, preview, and the schema-evolution union during sync) and is independent of `schemaEvolution.enabled`. It defaults to on because leaving it off silently materializes the wrong JSON keys for complex columns on column-mapped tables; the only cost of stamping is a larger ZK-persisted schema on tables that don't need it, and it does nothing for native Iceberg or `add_files` tables where physical names already equal logical names. Changing it on an already-onboarded table takes effect only on the **next schema write** (re-preview/re-register, or the next sync if schema evolution is on), followed by a segment reload — resolution happens at read time, so segments don't need regenerating. A table onboarded before this default flipped gets that one-time schema update and segment reload automatically on its first sync after upgrade, if it has complex columns and no explicit `storeNestedSchema.enabled` setting. See [Unity Catalog — schema options for column mapping](./unity/onboarding-api#schema-options-for-column-mapping). *** ## Tier backend properties (caching & preload) Set on the S3 tier in `tierConfigs[].tierBackendProperties`. The onboarding/preview flow sets sensible defaults. > [!NOTE] > > **Two-tier scaffold for `skipSegmentPreprocess=true` (create-time only).** If the table config already sets `indexingConfig.skipSegmentPreprocess=true` when it's created — through the preview API or `CREATE TABLE ... WITH (type='iceberg', ...)` — the enrichment scaffolds **two** tiers instead of the usual single fixed-selector tier: `_preprocessed_tier` (`segmentAge=2h`) and `_s3_tier` (`segmentAge=0s`), both keyed on `creationTime`. They share the same bucket and region but get distinct `pathPrefix` values (the preprocessed tier appends `/preprocessed`), so a segment aging from one tier to the other is a real physical move, and matching `indexingConfig.tierOverwrites` entries are written so segment preprocessing runs on the preprocessed tier and stays skipped on the s3 tier. New segments land on the s3 tier and relocate to the preprocessed tier once they pass the 2-hour threshold. This applies only at create time: adding `skipSegmentPreprocess=true` to an existing table via `PUT /tables//config` does not scaffold the second tier, and turning the flag back off does not remove one already created. ### Common | Key | Default | Description | | - | - | - | | `bucket` / `pathPrefix`² / `region` | *(required)* | S3 location of the tier storage. | | `enable.prefetch.page.cache` | `false`¹ | Route this table's index reads through the page cache. | | `preload.enable.index.consolidation` | `false`¹ | Pack a segment's index regions into one file to reduce mmap count on wide tables. | | `enable.delegate.v2` | `true` | Use the v2 delegate segment format. Already on by default — onboarding also writes this key, but that has no effect since it was already `true`. | | `enable.startree.index` | `false` | Load StarTree index files for this tier. **Off by default** — while it's off, a StarTree index built on an external table is generated at index-build time but never used to serve queries. Turn this on if the table has (or will have) a star-tree index. | ¹ The onboarding/preview flow sets these to `true` on **newly scaffolded** tables (`preload.enable.index.consolidation`, `enable.prefetch.page.cache`). Existing tables are never re-enriched — if a table predates these defaults, add the keys manually. ² `pathPrefix` is not validated. Nothing errors if it's missing or wrong — files simply land at the bucket root instead of under a per-table prefix. Always set it explicitly so each table's files get their own folder. > [!WARNING] > > `preload.enable`, `preload.dir`, `preload.total.size`, and `preload.load.existing.buffers` are **not** tier/table settings, even though they sound like they belong here next to `preload.enable.index.consolidation`. Setting them here does nothing. ### Preload tuning | Key | Default | Description | | - | - | - | | `preload.index.keys.override` | *(empty — nothing is preloaded until you set this)* | Comma-separated index keys to preload for **this table**, e.g. `*.inverted_index,*.range_index`. | > [!NOTE] > > The S3 tier also exposes advanced on-demand buffer, mmap, and read-ahead sub-properties (`ondemand.*`, `mmap.*`, `readAhead.*`). These are internal tuning knobs — leave them at defaults unless StarTree support advises otherwise. *** ## Query options Set per query with `SET "key" = 'value'`. Where an option is also a tier setting, the query option overrides it for that one query. | Query option | Default | Effect | | - | - | - | | `enable.prefetch.page.cache` | `false` | Route index reads through the page cache for this query (overrides read-ahead). | | `prefetch.projection.queue.size` | `10` | Projection blocks prefetched ahead on a scan queue refill (prefetch depth). | | `readAhead.enable` | `false` | Enable read-ahead buffering for index reads on this query. | | `skip.remote.table.cache` | `false` | Bypass the Parquet **page cache** — the cache that holds fetched remote-table data pages in memory/on disk — for both reads and writes. Covers **data pages only**. To measure a genuinely cold query, also set `skip.columnIndex.cache`; index reads otherwise still come from the local cache and the result will look faster than a real cold run. | | `skip.columnIndex.cache` | `false` | Bypass the column index cache for this query. Pair with `skip.remote.table.cache` for a truly cold-cache measurement. | | `snapshotVersionByTable` | — | Time-travel: pin the query to specific Iceberg snapshots, as a JSON map `{"table": snapshotId}`. See [Time travel](#time-travel-query-time-snapshot-pinning). | | `concurrentProjectionPrefetch` | `true` | Concurrent per-column prefetch for wide projections (`SELECT *`). Set `'false'` as a kill switch if a wide scan misbehaves. | | `pinot.parquet.pagereader.max.in.flight.chunk.reads` | `64` | Per-query cap on concurrent chunk reads. Positive values only. | | `pinot.parquet.prefetch.segment.lookahead.enabled` | `false` | Warm upcoming segments' chunks ahead of time for this query. | | `pinot.parquet.prefetch.filter.ahead.enabled` | `false` | Evaluate the query filter ahead of the scanners and warm projection pages for matched rows. Best effort: query execution still applies the authoritative filter. StarTree can disable this cluster-wide, in which case the option has no effect. | | `pinot.parquet.prefetch.filter.ahead.depth` | `8` | Max segments filter-ahead filters concurrently for this query. Can lower, not raise, the cluster-wide ceiling. | | `pinot.parquet.prefetch.filter.ahead.max.outstanding.futures` | `128` | Per-query cap on outstanding filter-ahead prefetch futures. Can lower, not raise, the cluster-wide ceiling. | | `enable.tiered.storage.trace` | `false` | Collect a tiered-storage trace for this query. The trace is written to **server logs only** — set the standard `trace` query option **together with** this one if you also want it back in the query response. | **Example — bypass the cache for one query:** ```sql theme={null} SET "skip.remote.table.cache" = 'true'; SELECT count(*) FROM nyc_taxi_trips WHERE fare > 100; ``` *** ## Time travel (query-time snapshot pinning) By default, a query against an Iceberg external table reads the **newest fully-synced snapshot**, resolved per query at the broker. The `snapshotVersionByTable` query option pins the query to a specific snapshot instead — the broker routes it to exactly that snapshot's file set, so you get the data (and row-level deletes) as of that snapshot: ```sql theme={null} SET "snapshotVersionByTable" = '{"nyc_taxi_trips_OFFLINE": 1234567890123456789}'; SELECT count(*) FROM nyc_taxi_trips; ``` The value is a JSON map of table name → snapshot ID, so a single query can pin several tables (one entry per table). Table names are canonicalized — `nyc_taxi_trips`, `nyc_taxi_trips_OFFLINE`, and `mydb.nyc_taxi_trips` all resolve to the same entry. The option works on both the single-stage and multi-stage query engines. ### Caveats * **Snapshot IDs only, no SQL clause.** There is no `FOR VERSION AS OF` / `FOR TIMESTAMP AS OF` syntax and no timestamp-based variant. Pass the numeric Iceberg snapshot ID (as a JSON number or numeric string) via the query option. * **Bounded time-travel window.** Only snapshots still in the table's **active snapshot list** can be pinned. [Snapshot retention](./snapshot-retention) keeps the newest `iceberg.snapshotProcessing.retention.maxActiveVersions` versions (default **2**). A configured value below **2** is ignored outright and the default of 2 — also the floor — is used instead. This is *not* arbitrary Iceberg-history time travel — it is "pick one of the last N snapshots StarTree has synced and still retains." * **Iceberg catalog sources only.** Supported for Iceberg REST catalog sources (AWS Glue, S3 Tables, Unity Catalog, generic REST). On raw S3/GCS Data Lake tables the option is **silently ignored** — file listings have no snapshot concept. * **Expired or unknown snapshots fail the query.** Pinning a snapshot that is not (or no longer) active returns an error (`Iceberg snapshot is not active for table `) rather than silently falling back to the latest snapshot. Malformed JSON in the option also fails the query. * **No pin = latest ready snapshot.** Without the option, each query reads the newest snapshot that is fully synced and cache-ready at that moment. Two queries issued at different times may therefore read different snapshots; pin a snapshot when you need repeatable reads across queries. * **On some tables the broker sets it automatically.** Every query on a [deletion-vector](./deletion-vectors)-enabled table runs against a pinned snapshot automatically — and so does every query on a table with snapshot consistency enabled, or one using [segment groups](./segment-groups). You only set the option yourself to override the default (latest) choice. * **Not the same as `initialSnapshotId`.** The table-config keys `iceberg.snapshotProcessing.initialVersionSelector` / `initialSnapshotId` (and the DDL `snapshot_mode` / `snapshot_id`) choose where **ingestion starts** when the table is created. They do not affect which snapshot a query reads. * **Finding snapshot IDs.** `GET …/externalTable/status?includeLag=true` returns the newest synced snapshot (`lag.synced.snapshotId`) and the upstream head (`lag.upstream.snapshotId`) — see [Observability](./observability#ingestion-lag). For older retained snapshots, get IDs from the source catalog (e.g. the Iceberg table's `snapshots` metadata); there is currently no API that lists all active snapshot IDs. # Data and Index caching Source: https://docs.startree.ai/corecapabilities/external-table/data-and-index-caching How the Parquet data cache, index cache, and footer cache work for External Tables and tiered-storage tables — sizing, eviction, and restart behavior. ## 1. Overview This document describes how Parquet data cache and Pinot index cache work in case of External tables as well as regular tables configured with remote tiered storage. External tables read their data from object storage on every query. The **Parquet Data Cache** keeps recently-read Parquet pages on the server so subsequent queries do not pay the round-trip. Indexes for both external tables and Pinot tiered storage tables also live on object storage. The **Index Cache** keeps recently-read index byte ranges on the server. | Cache | What it stores | Used by | On-disk location | When is it populated | | - | - | - | - | - | | **Parquet Data Cache** | Compressed Parquet column data and dictionary pages | External tables (Iceberg/S3/GCS) | `{dataDir}/remote_data_cache` | On every Parquet column read (always on) | | **Index Cache** | Raw byte ranges of segment index files (`columns.psf`, `inverted_index`) | External tables and Pinot tiered storage tables | `{dataDir}/index_cache` | On query opt-in | | **Footer Cache** | The Parquet file footer (row-group metadata + schema) for each segment, so it isn't re-fetched on every query. Persisted as part of the Parquet page index. | External tables | `{dataDir}/tieredStorage/segmentCache` (`parquet_page_index.bin`; migrates a legacy standalone `parquet_footer.bin` if present) | At segment load, whenever the persistent disk cache is enabled (on by default — see [Footer Cache](#3-3-footer-cache)) | Both caches share the same underlying machinery and, by default, the same in-memory sizing configuration (it isn't scoped per cache): * **Parquet Data Cache** populates an in-memory **prefetch tier** (decompressed, decoded values on heap, default 1 GB) in front of an mmap-backed **disk tier** (default 15% of disk per dictionary/data pool). * **Index Cache** populates an in-memory raw-bytes tier (the same underlying memory cache used by the Parquet Data Cache — default 256 MB data / 64 MB dictionary, since the memory-size configs aren't per-consumer) in front of its own mmap-backed disk tier. Only the 1 GB prefetch tier is effectively Parquet-Data-Cache-only, since raw index byte ranges have no decoded form to keep there. ![Chat GPT Image May 12, 2026, 04 41 09 PM](https://mintcdn.com/startree/DpCCnx-7zzFNEFCL/images/ChatGPT-Image-May-12,-2026,-04_41_09-PM.png?fit=max\&auto=format\&n=DpCCnx-7zzFNEFCL\&q=85\&s=96fae1acd330535eadcc68a302a921b4) The two caches do not share quota, directory, or eviction. They are sized and tuned independently. Following image depicts the high level query flow and how it uses the different caches. ![Chat GPT Image May 13, 2026, 08 39 05 PM](https://mintcdn.com/startree/3pTZlZuSMzIjC8Ny/images/ChatGPT-Image-May-13,-2026,-08_39_05-PM.png?fit=max\&auto=format\&n=3pTZlZuSMzIjC8Ny\&q=85\&s=6075b015a10ebfbb29944ce68cba7a54) ## 2. What's enabled out of the box ### 2.1 Parquet Data Cache — always on This cache is always on and does not need to be explicitly enabled. The cache is created at server startup and serves every read against a remote-Parquet column. | Tier | Default size | Backing store | | - | - | - | | Prefetch (in-memory) | 1 GB | Java heap (deserialized values) | | Disk | 15% of total disk per pool (dictionary + data) | mmap'd files at `{dataDir}/remote_data_cache` | ### 2.2 Index Cache — opt-in per query The Index Cache is populated only when a query is preceded by ```text theme={null} SET `enable.prefetch.page.cache` = 'true' ``` Byte ranges of segment index files fetched from object storage during that query are persisted on disk, so subsequent queries (with or without the option) get cache hits. This applies to both external tables and Pinot tiered storage tables — any query that needs to read an index buffer from object storage benefits from this cache. | Tier | Default size | Backing store | | - | - | - | | In-memory | 256 MB (data) / 64 MB (dict) — shared with the Parquet Data Cache's memory sizing | Java heap | | Disk | 15% of total disk per pool (dictionary + data) | mmap'd files at `{dataDir}/index_cache` | Without the query option and without StarTree bitmap reads, the Index Cache stays empty and the directory remains untouched. *** ## 3. Advanced operations: Enable, disable, skip ### 3.1 Parquet Data Cache Always on at the server level and cannot be globally disabled. However, you can use a per-query bypass as shown below: ```sql theme={null} SET `skip.remote.table.cache` = 'true'; SELECT ... FROM remoteTable WHERE ...; ``` When set, both tiers are bypassed for that query. Pages are read directly from object storage and not stored. ### 3.2 Index Cache In this case, there is no global enable flag; enablement is a per-query decision. You can enable index cache for a specified query as shown below: | Action | How | | - | - | | Enable for one query | `SET `enable.prefetch.page.cache` = 'true'` | As mentioned previously, once set, subsequent queries can benefit from this index cache as applicable. ### 3.3 Footer Cache The Footer Cache stores the Parquet file footer (row-group metadata and schema) for each segment so it doesn't need to be re-fetched from object storage on every query. It is part of the persistent disk cache, which is on by default, with nothing separate to configure. Metadata caching (including the footer) is always on whenever the disk cache is enabled — there is no separate opt-in flag. The footer is persisted as part of the Parquet page index (`parquet_page_index.bin`) under `/tieredStorage/segmentCache` and always reloaded on server restart. A legacy standalone `parquet_footer.bin` from an older build is migrated into the page index and then removed. Setting `enabled = false` disables the footer cache along with every other managed disk cache, and the footer is then fetched from remote storage on each query. *** ## 4. Behavior on restart Restart reuse is **on by default** (changed in 0.16.x — it used to be opt-in). The disk tier reloads its key index from a snapshot file written during shutdown (and periodically every five minutes) and resumes serving immediately, for both caches identically. The prefetch tier is in-memory — it always starts empty and is not persisted across restarts. *** ## 5. Eviction policies | Tier | Cache | Policy | Defaults | | - | - | - | - | | Prefetch (in-memory) | Parquet Data Cache only | Circular buffer on heap; oldest slot is overwritten on insert | 1 GB capacity, no TTL | | Disk | Both | LRU over fixed-size mmap fragment files. | Parquet Data Cache: 32 MB fragments. Index Cache: 512 MB fragments. Background eviction sweep every 60s (fixed, not configurable). | When the disk tier is full, the oldest fragment file is unmapped and deleted; new pages are written to a fresh fragment. > [!NOTE] > > **TTL-based eviction was removed in 0.16.x.** The disk tier now evicts purely on capacity (LRU) and segment-version invalidation; there is no idle- or write-age TTL to configure, and the old `ttl.access.ms` / `ttl.write.ms` / `eviction.interval.ms` keys are ignored (the sweep interval is a fixed 60s). Entries restored from a restart snapshot are still safe to reuse — entries are keyed by source and segment version, which prevents a stale entry from being served after a same-URI replacement. *** ## 6. Clearing caches ### HTTP endpoints There are certain scenarios in which user may want to clear the caches. StarTree provides Cluster-wide APIs for this (controller fans out to every server): | Endpoint | Effect | | - | - | | `DELETE /pageCache?consumerName=PARQUET_INDEX` | Clear Parquet Data Cache (all tiers) | | `DELETE /pageCache?consumerName=SEGMENT_INDEX` | Clear Index Cache (all tiers) | | `DELETE /pageCache` | Clear both | This can also be done via a per-server API (skip the controller, hit one server directly): ```text theme={null} DELETE http://:/pageCache?consumerName=PARQUET_INDEX ``` Response is JSON listing cleared cache. # Data Export Task: Parquet Writer Source: https://docs.startree.ai/corecapabilities/external-table/data-export-task Export completed Pinot REALTIME segments back to Parquet on a filesystem or an Iceberg REST catalog with the DataExportTask minion task. > [!WARNING] > > This feature requires StarTree Cloud with a minion tier enabled, and must be enabled on demand — contact StarTree support to activate it. `DataExportTask` is a Pinot minion task that exports completed **REALTIME segments** to external storage as Parquet files. It is the complement of `ExternalTableSyncTask`: where that task reads external Parquet into Pinot, `DataExportTask` writes Pinot data back out to your data lake. Common uses include cold-tier archival, feeding downstream Iceberg catalogs, and cross-system data sharing without a separate ETL pipeline. Two destinations are supported: a plain filesystem path (any S3-compatible URI via PinotFS) and an Iceberg REST catalog (AWS Glue, S3 Tables, or any Iceberg REST-compliant catalog). *** ## How it works ```mermaid theme={null} flowchart LR REALTIME["REALTIME Table\n(Pinot)"] --> Generator["Generator\n(Controller)"] Generator --> |"Watermark +\nZK checkpoint"| EligibleSegments["Eligible\nCompleted Segments"] EligibleSegments --> Executor["Executor\n(Minion)"] Executor --> Download["Download segment\nfrom deep store"] Download --> Convert["Convert to\nParquet"] Convert --> Upload["Upload to\ndestination"] Upload --> FilesystemDest["Filesystem\n(S3/GCS/HDFS)"] Upload --> IcebergDest["Iceberg REST\n(Glue / S3 Tables)"] ``` Each cycle, the **Generator** runs on the controller: it loads a time-based watermark from ZooKeeper, filters REALTIME segments that are completed, past the `bufferTimePeriod`, and not already checkpointed or in a running task. It then emits one `PinotTaskConfig` per eligible segment, capped by `tableMaxNumTasks`. The **Executor** runs on a minion: it downloads the segment from Pinot deep store, converts it to Parquet using the configured writer, and uploads the result to the destination. On success, it posts a COMPLETED checkpoint back so the segment is not re-exported. *** ## Supported destinations | Destination | `dataExport.target` | Use with | | - | - | - | | Filesystem (PinotFS) | `filesystem` | Any S3-compatible URI, GCS, HDFS | | Iceberg REST catalog | `iceberg-rest` | AWS Glue, AWS S3 Tables, any Iceberg REST catalog | *** ## Prerequisites * A REALTIME table with completed segments exists in your StarTree cluster. * A minion tier is provisioned in the cluster. * The minion has network access to the destination (S3 bucket, GCS bucket, or Iceberg REST endpoint). * For Iceberg REST: the target table must already exist in the catalog, must not be partitioned, and must use Parquet as its file format. *** ## Configuring the task Add `DataExportTask` to the table's `task.taskTypeConfigsMap`. #### AWS S3 using Basic Authentication ```json theme={null} "task": { "taskTypeConfigsMap": { "DataExportTask": { "schedule": "0 0 * * * ?", "sourceTableName": "my_events_REALTIME", "fileFormat": "parquet", "dataExport.target": "filesystem", "outputDirURI": "s3://my-bucket/exports/my_events/", "output.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "output.fs.prop.region": "", "enableTaskOrchestration": "true", "output.fs.prop.accessKey" : "ACCESS_KEY", "output.fs.prop.secretKey" : "SECRET_KEY" } } } ``` #### AWS S3 using IAM-Based Authentication ```json theme={null} "task": { "taskTypeConfigsMap": { "DataExportTask": { "schedule": "0 0 * * * ?", "sourceTableName": "my_events_REALTIME", "fileFormat": "parquet", "dataExport.target": "filesystem", "enableTaskOrchestration": "true", "outputDirURI": "s3://my-bucket/exports/my_events/", "output.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "output.fs.prop.region": "", "output.fs.prop.roleArn" : "ROLE_ARN", "output.fs.prop.externalId": "env-12abc234xyz" } } } ``` #### GCS Use `GcsPinotFS` with a service account key: ```json theme={null} "task": { "taskTypeConfigsMap": { "DataExportTask": { "schedule": "0 0 * * * ?", "sourceTableName": "my_events_REALTIME", "fileFormat": "parquet", "dataExport.target": "filesystem", "enableTaskOrchestration": "true", "outputDirURI": "gs://my-bucket/exports/my_events/", "output.fs.className": "org.apache.pinot.plugin.filesystem.GcsPinotFS", "output.fs.prop.projectId": "my-gcp-project", "output.fs.prop.jsonKey": "" } } } ``` #### Iceberg REST For an Iceberg REST destination, replace the filesystem keys with the catalog connection: ```json theme={null} "task": { "taskTypeConfigsMap": { "DataExportTask": { "schedule": "0 0 * * * ?", "sourceTableName": "my_events_REALTIME", "fileFormat": "parquet", "dataExport.target": "iceberg-rest", "enableTaskOrchestration": "true", "catalog.iceberg-rest.restUri": "https://glue..amazonaws.com/iceberg", "catalog.iceberg-rest.table.namespace": "my_database", "catalog.iceberg-rest.table.tableName": "my_events_archive", "iceberg.commitThreshold": "250" } } } ``` *** ## Config reference ### Core settings | Key | Default | Description | | - | - | - | | `schedule` | — | Quartz cron expression for how often the Generator runs, e.g. `0 0 * * * ?` (hourly). | | `sourceTableName` | *(required)* | The REALTIME table to export from, with the `_REALTIME` suffix, e.g. `my_events_REALTIME`. | | `fileFormat` | `parquet` | Output file format. Only `parquet` is supported. | | `dataExport.target` | `filesystem` | Destination type. `filesystem` or `iceberg-rest`. | | `bufferTimePeriod` | `1d` | Segments whose end time is within this window of now are excluded — allows segments to fully complete before export. | | `initialWatermarkMs` | `0` | Epoch ms to use as the starting watermark on the first run. Segments older than this value are skipped. | | `tableMaxNumTasks` | `10000` | Maximum number of segments exported per Generator cycle. Limits concurrent minion load. | | `enableTaskOrchestration` | — | Opts into the minion task orchestration framework (MTOF), which auto-progresses across segment batches within a run. Set in all example configs above. | *** ### Filesystem target Set these when `dataExport.target=filesystem`. | Key | Default | Description | | - | - | - | | `outputDirURI` | *(required)* | Destination URI, e.g. `s3://my-bucket/exports/table/` or `gs://my-bucket/exports/table/`. Output files are written as `//.parquet`. | #### AWS S3 using Basic Authentication | Key | Default | Description | | - | - | - | | `output.fs.className` | — | Fully-qualified PinotFS class, e.g. `org.apache.pinot.plugin.filesystem.S3PinotFS`. Defaults to the cluster's configured deep-store FS if unset. | | `output.fs.prop.region` | — | AWS region, e.g. `us-east-1`. | | `output.fs.prop.accessKey` | — | Static AWS access key. | | `output.fs.prop.secretKey` | — | Static AWS secret key. | | `output.fs.prop.endpoint` | — | S3-compatible endpoint override, e.g. for MinIO or other non-AWS S3 implementations. | #### AWS S3 using IAM-Based Authentication | Key | Default | Description | | - | - | - | | `output.fs.className` | — | Fully-qualified PinotFS class, e.g. `org.apache.pinot.plugin.filesystem.S3PinotFS`. Defaults to the cluster's configured deep-store FS if unset. | | `output.fs.prop.region` | — | AWS region, e.g. `us-east-1`. | | `output.fs.prop.roleArn` | — | ARN of the IAM role the minion assumes to access the bucket, in place of static credentials. | | `output.fs.prop.externalId` | — | External ID required by the role's trust policy, if configured. | #### GCS | Key | Default | Description | | - | - | - | | `output.fs.className` | — | Fully-qualified PinotFS class, e.g. `org.apache.pinot.plugin.filesystem.GcsPinotFS`. | | `output.fs.prop.projectId` | — | The GCP project ID that owns the target bucket. | | `output.fs.prop.jsonKey` | — | The string-encoded GCP service account JSON key used to authenticate to the bucket. | *** ### Parquet format These apply to both destinations. | Key | Default | Description | | - | - | - | | `fileFormat.parquet.compressionCodec` | `SNAPPY` | Parquet compression. `SNAPPY`, `GZIP`, `ZSTD`, or `UNCOMPRESSED`. | | `fileFormat.parquet.rowGroupSizeBytes` | `134217728` (128 MB) | Target row group size in bytes. | | `fileFormat.parquet.pageSizeBytes` | `1048576` (1 MB) | Target page size in bytes. | | `fileFormat.parquet.enableDictionary` | `true` | Write Parquet dictionary pages. Disable for high-cardinality columns. | | `fileFormat.parquet.json.maxDepth` | — | Maximum nesting depth when serializing JSON columns to Parquet `group` types. | | `fileFormat.parquet.decimal.precision` | `38` | Override decimal precision for `BIG_DECIMAL` columns. | | `fileFormat.parquet.decimal.scale` | `18` | Override decimal scale for `BIG_DECIMAL` columns. | | `fileFormat.parquet.json.unknownKeysBehavior` | `warn` | What to do when a JSON column in the row data has keys not present in the Iceberg schema: `warn` (log and continue) or `fail` (abort the task). Applies to `iceberg-rest` target only. | *** ### Iceberg REST target Set these when `dataExport.target=iceberg-rest`. | Key | Default | Description | | - | - | - | | `catalog.iceberg-rest.restUri` | *(required)* | Iceberg REST catalog URI, e.g. `https://glue..amazonaws.com/iceberg`. | | `catalog.iceberg-rest.table.namespace` | *(required)* | Catalog namespace (database) containing the target table. | | `catalog.iceberg-rest.table.tableName` | *(required)* | Target table name inside the namespace. | | `catalog.iceberg-rest.token` | — | Static bearer token for catalog authentication. | | `catalog.iceberg-rest.credential` | — | OAuth2 client credential (`clientId:clientSecret`). | | `catalog.iceberg-rest.rest.signing-region` | — | AWS region for SigV4 request signing (e.g. `us-east-1`). | | `catalog.iceberg-rest.s3.access-key-id` | — | Static AWS access key for S3 file I/O (overrides vended credentials). | | `catalog.iceberg-rest.s3.secret-access-key` | — | Static AWS secret key. | > [!NOTE] > > For AWS Glue and S3 Tables, the catalog vends short-lived S3 credentials automatically — do not set static `s3.*` keys unless you want to override them. The minion's IAM role must have `glue:GetTable` and `s3:PutObject` (or equivalent S3 Tables) permissions on the target. *** ### Iceberg batch commits By default, each minion subtask uploads a Parquet file and stages it. The Generator accumulates staged files in ZooKeeper and issues a single Iceberg snapshot commit (`AppendFiles.commit()`) when the batch is ready — one snapshot for many segments instead of one per segment. This reduces Iceberg snapshot churn on high-throughput tables. The Generator runs an **EMIT / WAIT / COMMIT** decision each cycle: | State | Condition | Action | | - | - | - | | `EMIT` | Eligible segments exist and in-flight count is below threshold | Schedule new subtasks | | `WAIT` | Subtasks are still running **and** the uncommitted/staged count has reached `iceberg.commitThreshold` | Skip this cycle; wait for in-flight tasks to drain before committing | | `COMMIT` | No subtasks are running, at least one file is staged, **and** either the staged count has reached `iceberg.commitThreshold` or no eligible segments remain | Flush the staged queue with one `AppendFiles.commit()` | | Key | Default | Description | | - | - | - | | `iceberg.commitThreshold` | `250` | Number of staged files that triggers an immediate batch commit. Tune down for lower latency; tune up to reduce snapshot frequency on high-volume tables. | *** ## Scheduling and monitoring `DataExportTask` uses the standard Pinot minion task scheduler. To trigger a run immediately without waiting for the next `schedule` tick, use the task schedule endpoint: ```bash theme={null} curl -X POST "$BASE_URL/tasks/schedule?taskType=DataExportTask&tableName=my_events_REALTIME" ``` To check task status, use the standard Pinot tasks API — per-table state: ```bash theme={null} curl "$BASE_URL/tasks/DataExportTask/my_events_REALTIME/state" ``` or subtask states across all tables running this task type: ```bash theme={null} curl "$BASE_URL/tasks/DataExportTask/taskstates" ``` For a broader view of sync health and the ingestion checkpoint, see the [Observability](./observability) page. # Data Type Mapping Source: https://docs.startree.ai/corecapabilities/external-table/data-type-mapping How Parquet and Iceberg source types map to Pinot types in an External Table, including time-column detection, granularity, and known limitations. > [!WARNING] > > This feature is available starting in **StarTree release 0.15.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment. When you onboard an External Table, StarTree infers a Pinot schema from the source metadata. The mapping depends on the source: * **Raw S3 Parquet** (S3 Data Lake, `catalogType=s3`) — inferred from the Parquet file's physical and logical types. * **Iceberg** (AWS Glue / S3 Tables, `catalogType=iceberg-rest`) — inferred from the Iceberg table schema. > [!NOTE] > > Numeric columns (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `BIG_DECIMAL`) are inferred as **metrics**; everything else (`BOOLEAN`, `STRING`, `JSON`, `BYTES`, time columns) as **dimensions**. You can adjust this in the schema before creating the table. *** ## Raw S3 Parquet → Pinot For the **S3 Data Lake** source. Types come from the Parquet file metadata. ## Primitive Types | Parquet Type | Pinot Type | Notes | | - | - | - | | `BOOLEAN` | `BOOLEAN` | Optional and required variants both supported. | | `INT32` | `INT` | | | `INT64` | `LONG` | | | `FLOAT` | `FLOAT` | | | `DOUBLE` | `DOUBLE` | | | `BINARY` | `BYTES` | Raw bytes are preserved losslessly rather than decoded as UTF-8. | | `FIXED_LEN_BYTE_ARRAY` | `BYTES` | Fixed-width byte arrays are preserved losslessly rather than decoded as UTF-8. | | `INT96` | `TIMESTAMP` | Deprecated Parquet type. Converted to milliseconds since epoch; no column statistics available. Avoid using as a time column. | *** ## Logical Types ### Integer Subtypes | Parquet Logical Type | Pinot Type | Notes | | - | - | - | | `INT(8, signed)` | `INT` | Range: -128 to 127 | | `INT(16, signed)` | `INT` | Range: -32,768 to 32,767 | | `INT(32, signed)` | `INT` | | | `INT(64, signed)` | `LONG` | | | `INT(8, unsigned)` | `INT` | Range: 0 to 255 | | `INT(16, unsigned)` | `INT` | Range: 0 to 65,535 | | `INT(32, unsigned)` | `LONG` | **Known limitation:** values are stored as signed. Values above `2,147,483,647` may be corrupted. See [Unsigned Integer Limitation](#unsigned-integer-types). | | `INT(64, unsigned)` | `LONG` | **Known limitation:** same signed-storage issue. Values above `Long.MAX_VALUE` are not preserved accurately. | ### String and Binary Types | Parquet Logical Type | Pinot Type | Notes | | - | - | - | | `STRING` | `STRING` | UTF-8 encoded strings. | | `ENUM` | `STRING` | Enum values stored as strings. | | `JSON` | `JSON` | Parsed into Pinot's native `JSON` type, not preserved as a plain string. | | `BSON` | `STRING` | Binary data is decoded as a UTF-8 string, not preserved as a distinct binary type. Non-UTF8-safe byte sequences may not round-trip cleanly — verify BSON columns after ingestion. | | `UUID` | `BYTES` | The raw 16-byte value is preserved losslessly as bytes rather than decoded as a UTF-8 string. | | `INTERVAL` | `STRING` | | | `GEOMETRY`, `GEOGRAPHY` | `BYTES` | Stored as raw bytes; no geospatial functions or H3 index are available on external tables. | ### Temporal Types | Parquet Logical Type | Pinot Type | Notes | | - | - | - | | `DATE` | `LONG` | Stored as epoch days. | | `TIME(MILLIS)` | `LONG` | Milliseconds since midnight; range 0 to 86,399,999. | | `TIME(MICROS)` | `LONG` | Microseconds since midnight; range 0 to 86,399,999,999. | | `TIME(NANOS)` | `LONG` | Nanoseconds since midnight. | | `TIMESTAMP(MILLIS, UTC)` | `TIMESTAMP` | | | `TIMESTAMP(MILLIS, local)` | `TIMESTAMP` | | | `TIMESTAMP(MICROS, UTC)` | `TIMESTAMP` | Native microsecond granularity is preserved (`EPOCH\|MICROSECONDS`) — see [Granularity](#granularity). | | `TIMESTAMP(MICROS, local)` | `TIMESTAMP` | Native microsecond granularity is preserved — see [Granularity](#granularity). | | `TIMESTAMP(NANOS, UTC)` | `TIMESTAMP` | Native nanosecond granularity is preserved (`EPOCH\|NANOSECONDS`) — see [Granularity](#granularity). | | `TIMESTAMP(NANOS, local)` | `TIMESTAMP` | Native nanosecond granularity is preserved — see [Granularity](#granularity). | ### Decimal Types All Parquet `DECIMAL` variants map to Pinot `BIG_DECIMAL`, preserving scale and precision. | Parquet Physical + Logical Type | Pinot Type | Example | | - | - | - | | `INT32` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(9, 2)` | | `INT64` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(18, 4)` | | `FIXED_LEN_BYTE_ARRAY` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(28, 6)` | | `BINARY` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(38, 10)` | *** ## Complex Types | Source type | Pinot type | Notes | | - | - | - | | `LIST` / `ARRAY` of a primitive | Multi-value column of the element type | Element types: `INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`, `BOOLEAN`. A list of timestamps is stored as multi-value `LONG`. | | `LIST` of a complex element | `JSON` | Lists of structs / maps / nested lists are serialized to a JSON string. | | `MAP` | `MAP` | Native map for primitive values (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`). | | `MAP` (other shapes) | `JSON` | Non-string keys or complex values are serialized to JSON. | | `STRUCT` / nested group | `JSON` | Serialized to a JSON string; query with JSON functions. | | `VARIANT` | `OPEN_STRUCT` | Detected from the Parquet VARIANT logical type annotation (or, for writers that predate it, the `metadata` + `value`/`typed_value` group shape the shredding spec fixes). Not the same as `JSON` — a VARIANT's key set and per-key types are data, not schema, and are read per-file from the shredded tree rather than declared up front. | > [!NOTE] > > To use `JSON_MATCH` or `JSON_EXTRACT_SCALAR` on complex columns, you must enable a JSON index on those columns. Without it, queries will fail with a `Cannot apply JSON_MATCH on column without json index` error. Configure the index in the table's index settings. > [!WARNING] > > Multi-value columns are capped at **1,000 entries per row** — the segment metadata is written with a fixed upper bound, and rows whose source array is longer are silently truncated at query time. If your lists can exceed 1,000 elements, keep the column as `JSON` instead. *** ## Iceberg → Pinot For the **AWS Glue** and **AWS S3 Tables** sources. Types come from the Iceberg table schema. | Iceberg type | Pinot type | Notes | | | - | - | - | - | | `boolean` | `BOOLEAN` | | | | `int` | `INT` | | | | `long` | `LONG` | | | | `float` | `FLOAT` | | | | `double` | `DOUBLE` | | | | `decimal(p, s)` | `BIG_DECIMAL` | Precision and scale preserved. | | | `string` | `STRING` | | | | `uuid` | `BYTES` | Preserved losslessly as bytes rather than decoded as a UTF-8 string. | | | `binary`, `fixed` | `BYTES` | Preserved losslessly as bytes rather than decoded as a UTF-8 string. `fixed[N]` length suffix is parsed. | | | `date` | `LONG` | \`EPOCH | DAYS\`. | | `time` | `LONG` | Stored as the raw long; not a time-column candidate. | | | `timestamp`, `timestamptz` | `TIMESTAMP` | Granularity inferred from the type — see note below. | | | `timestamp_ns`, `timestamptz_ns` | `TIMESTAMP` | \`EPOCH | NANOSECONDS\`. | | `struct` | `JSON` | Serialized to a JSON string. | | | `list` | Multi-value column of the element type | `INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`, `BOOLEAN`; `list` becomes multi-value `LONG`. | | | `list` | `JSON` | | | | `map` | `MAP` | | | | `map` (other shapes) | `JSON` | | | | `variant` | `OPEN_STRUCT` | Not the same as `JSON` — a variant's key set and per-key types are data, not schema, and are discovered per-file rather than declared up front. See the raw-Parquet `VARIANT` row above for the equivalent mapping on the S3 Data Lake source. | | | (unrecognized) | `STRING` | Falls back to `STRING`. | | > [!NOTE] > > **Timestamp granularity is inferred from the Iceberg type** (current releases): > > * `timestamp_ns` / `timestamptz_ns` → `EPOCH|NANOSECONDS` > * `timestamp(p)` with an explicit precision → milliseconds for `p ≤ 3`, microseconds for `p ≤ 6`, nanoseconds above that > * plain `timestamp` / `timestamptz` → **`EPOCH|MICROSECONDS`**, matching Iceberg's on-disk microsecond storage > > Older 0.16.0 builds inferred plain `timestamp` as `EPOCH|MILLISECONDS`, which mislabeled microsecond data (values \~1000× off). Tables onboarded on those builds keep their inferred format — set the time column's granularity explicitly if it doesn't match the data. See [Time column](#time-column). *** ## Time column A Pinot table can designate one column as its **primary time column** — a `DateTimeFieldSpec` that records the event time of each row. See the Apache Pinot [DateTimeFieldSpec reference](https://docs.pinot.apache.org/configuration-reference/schema#datetimefieldspec) for the general schema requirement and format syntax. ### How the time column is used in an External Table * **Segment pruning** — a query with a time filter skips segments whose time range can't match, so fewer remote files are read. * **Retention** — if a retention period is configured on the table, segments older than it are dropped. The time column is **optional**: without one, the table is still created and fully queryable, but loses time-based segment pruning and retention. ### How the time column is detected During onboarding, after the schema is inferred, StarTree selects a time column automatically — choosing, in order: 1. The first non-nullable millisecond-granularity datetime column. 2. Otherwise, the first millisecond-granularity datetime column (nullable allowed). 3. Otherwise, the first datetime column of any unit. Only columns that map to a Pinot `DateTimeFieldSpec` are candidates — `TIMESTAMP`, `DATE`, and (raw Parquet) `INT96`. A plain integer column holding an epoch value is **not** auto-detected; set it as the time column manually if needed. If the source has no timestamp/date column, no time column is set; you can assign one in the schema before creating the table. > [!WARNING] > > If auto-detection picks an `INT96` column as the time column, segment generation now **fails by default** with an actionable error naming the column and pointing at the workaround, instead of an opaque decoding error. See [INT96 (Deprecated)](#int96-deprecated) below before onboarding a source whose only datetime column is `INT96`. ### Granularity For **raw Parquet**, the inferred `DateTimeFieldSpec` preserves the source unit: | Source timestamp | Inferred format | Granularity | | - | - | - | | `INT96` (legacy) | `EPOCH\|MILLISECONDS` | `1:MILLISECONDS` | | `TIMESTAMP(millis)` | `EPOCH\|MILLISECONDS` | `1:MILLISECONDS` | | `TIMESTAMP(micros)` | `EPOCH\|MICROSECONDS` | `1:MICROSECONDS` | | `TIMESTAMP(nanos)` | `EPOCH\|NANOSECONDS` | `1:NANOSECONDS` | | `DATE` | `EPOCH\|DAYS` | `1:DAYS` | For **Iceberg**, the granularity follows the type: `timestamp_ns` → nanoseconds, `timestamp(p)` → from the precision, and plain `timestamp` / `timestamptz` → **microseconds** (older 0.16.0 builds inferred milliseconds — see the note in the Iceberg → Pinot section above). > [!NOTE] > > Values are stored in the source unit — there is no implicit downscaling — so account for the unit in time-range queries. If the inferred granularity doesn't match what the underlying Parquet actually stores (for example, a table onboarded on an older build), set the time column's granularity explicitly. *** ## Known Limitations ### Unsigned Integer Types Parquet `UINT32` and `UINT64` (unsigned 32-bit and 64-bit integers) are stored in Pinot as a signed `LONG`. This means: * Values that fit within the signed range are accurate. * Values above `2,147,483,647` (for UINT32) are stored as negative numbers or wrap around, silently corrupting approximately half the data. If your dataset contains unsigned integers larger than `INT32_MAX`, treat the Pinot `LONG` values with caution or apply a transformation at ingestion time. ### Null Handling in Complex Types `MAP` columns that contain null values in keys or values may not propagate null bitmaps correctly to Pinot segments. This can result in nulls being replaced by type defaults (e.g., `0` for integers, empty string for strings). Verify null behavior after ingestion. Current onboarding flows set `nullHandlingEnabled: true` on the generated table config automatically, and when the source schema has nullable fields, the inferred Pinot schema marks them nullable and enables column-based null handling (`enableColumnBasedNullHandling`). Tables created on older builds may have null handling off — see [Advanced Guidance](/corecapabilities/external-table/operational-guidance). ### INT96 (Deprecated) `INT96` has no column statistics in Parquet — its footer min/max are skipped rather than trusted, since the format leaves INT96's sort order undefined (they aren't guaranteed to be the chronological extremes, so pruning on them could drop live rows). This affects min/max pruning during segment generation. Scalar and list `INT96` values still decode correctly for reads. It should not be used as a `timeColumnName`. If it is (including via auto-detection — see [How the time column is detected](#how-the-time-column-is-detected)), segment generation fails by default because no time range can be derived from the skipped statistics, with an error naming the column and the workaround. Set `segmentTimeValueCheck=false` in the table's `ingestionConfig` to ingest without time-range metadata for that column instead. # Iceberg Deletion Vectors Source: https://docs.startree.ai/corecapabilities/external-table/deletion-vectors Apply Iceberg v2 equality/position deletes and v3 Puffin deletion vectors at query time so External Tables reflect source-side row-level deletes and updates, not just appends. > [!WARNING] > > This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it. It is offered as a preview capability; validate it in a non-production environment before relying on it. Iceberg tables record row-level deletes and updates as merge-on-read delete files rather than rewriting data files in place — either native v3 Puffin **deletion vectors**, or standard v2 Parquet **equality and position delete files**. For both REST and Hive Metastore catalogs, the controller's snapshot sync materializes applicable deletes into a per-file delete bitmap at sync time, and servers apply that bitmap at query time, so query results stay correct as source data is updated or deleted — the source doesn't have to be append-only. ## How it works 1. **Sync-time materialization (v2 only)** — for standard Iceberg v2 equality and Parquet position deletes, the controller's snapshot sync reads the applicable delete files using Iceberg's scan-planning rules (partition spec, partition tuple, data sequence number; equality columns resolve by field ID even across renamed/dropped columns) and builds a per-file delete bitmap, written as a Puffin-format blob alongside the snapshot's `dv-index.parquet` and cached separately from native source Puffin files. Native v3 Puffin deletion vectors are already in this form and skip materialization. 2. **Server-side loading** — each server decodes the resulting deletion-vector index (materialized v2 bitmap or native v3 Puffin) for its segments and applies it when answering a query, so deleted/updated rows are excluded from results. 3. **Atomic snapshot readiness** — before a new Iceberg snapshot becomes visible to queries, the controller polls every server until all of them confirm they've loaded the deletion vectors for that snapshot. This avoids a window where some servers see the new snapshot's deletes and others don't. 4. **Broker-side pinning and consistency** — the broker pins every query to a single snapshot per table so all servers involved in a query agree on which snapshot (and therefore which deletes) to use. By default it pins the newest active snapshot automatically and injects the `snapshotVersionByTable` option for the servers; a query can also pin an explicit snapshot (see [Query-time snapshot pinning](#query-time-snapshot-pinning)). An invalid explicitly-pinned snapshot fails the query rather than silently mixing snapshot state. ## Enabling deletion vectors Set on the table's `ExternalTableSyncTask` config: | Key | Default | Description | | :- | :- | :- | | `enableDeletionVectors` | `false` | Enable Iceberg row-delete support (v2 equality/position deletes and v3 Puffin deletion vectors) for this table. Always defaults to `false` and is never auto-enabled by onboarding, even against a catalog capable of deletion vectors — set it explicitly. Setting it to `true` on a source that isn't deletion-vector capable fails DDL compilation or the sync run. | | `deletionVectorSource` | `iceberg` | Source format for deletion vectors. `iceberg` is the only supported value. | Changes to `enableDeletionVectors` and `deletionVectorSource` (and to the table's catalog or storage credentials) take effect on the servers without a restart — the deletion-vector readers are rebuilt when the table config changes. > [!NOTE] > > **Upgrade every controller and server first.** `enableDeletionVectors=true` is only safe once every controller and server in the cluster runs a build that supports derived row-delete blobs. Creating or enabling it earlier can publish artifacts older servers can't read. An existing v3 Puffin table can keep running through a rolling upgrade; don't introduce v2 equality or position delete files to it, and don't flip the flag on an existing table, until the upgrade completes. After enabling the flag on an existing table, sync a fresh Iceberg snapshot before relying on row-delete results — previously published snapshot artifacts are immutable and aren't reprocessed under the same snapshot ID. > [!NOTE] > > **Incompatible with the Star-tree index.** Table-config validation rejects a Star-tree index (explicit `starTreeIndexConfigs` or `enableDefaultStarTree`) on a table with `enableDeletionVectors=true`, and the reverse — enabling deletion vectors on a table that already has a Star-tree index configured. Deletion vectors filter rows per query, and Star-tree's precomputed aggregates can't be reconciled against that. Remove the Star-tree index configuration or disable deletion vectors. See [Supported Indexes](./indexes). > [!NOTE] > > **How Puffin delete files are read is no longer a table setting.** The per-table `iceberg.dv.fetch.mode`, `iceberg.dv.puffinCache.maxBytes`, and `iceberg.dv.puffinCache.downloadTimeoutMs` keys are retired (0.16.x). Puffin files are now cached by the server's persistent disk cache, which StarTree manages; if an old table config still carries these keys, they are ignored. The number of snapshots kept queryable (and therefore pinnable — see [Query-time snapshot pinning](#query-time-snapshot-pinning)) is bounded by snapshot retention, `iceberg.snapshotProcessing.retention.maxActiveVersions` (default `2`). ### Controller: snapshot readiness | Key | Default | Description | | :- | :- | :- | | `iceberg.snapshotProcessing.readiness.maxWaitMs` | `3600000` (1 h) | Max time the controller waits for all servers to confirm a snapshot is ready. | | `iceberg.snapshotProcessing.readiness.pollIntervalMs` | `30000` | Interval between readiness polls. | | `iceberg.snapshotProcessing.readiness.requestTimeoutMs` | `10000` | Per-request timeout for a readiness poll call. | | `iceberg.snapshotProcessing.readiness.stallTimeoutMs` | `600000` (10 min) | Stop the poll early if no server makes load progress for this long, instead of waiting out `maxWaitMs`. Only applies while every not-ready server reports load counts (see the response fields below); a server on an older build falls back to the `maxWaitMs` deadline. | Each **server** exposes a readiness endpoint, which the controller polls: ``` POST /tables/{tableNameWithType}/iceberg/isSnapshotReady?snapshotVersion= ``` Returns `{"ready": true|false}` plus additive load-progress fields the controller's poller uses for stall detection — `loaded`, `total`, `inFlight`, `consecutiveLoadFailures`, and `lastLoadError` (present only when set). Older pollers ignore the extra fields; a server that omits them (older build) is treated as deadline-only, never assumed to be stalled. The check is non-blocking: it answers from the server's in-memory state and, if the deletion vectors aren't loaded yet, kicks off a background load and returns `ready: false` — so the first poll for a new snapshot normally reports not-ready. Returns `412` when the table is not configured for deletion vectors (`enableDeletionVectors=false`) or the deletion-vector index for the requested snapshot is missing or unreadable. ### Controller: IS/EV convergence Before polling for snapshot readiness, the controller waits for every affected segment's Helix ideal-state and external-view to converge: | Key | Default | Description | | :- | :- | :- | | `iceberg.snapshotProcessing.convergence.maxWaitMs` | *(falls back to the effective `readiness.maxWaitMs`)* | Hard deadline for the ideal-state/external-view convergence wait. Unset by default so pre-split deployments keep their tuned readiness timeout. | | `iceberg.snapshotProcessing.convergence.stallTimeoutMs` | `600000` (10 min) | Stop the convergence wait if no mismatched (segment, instance) pair resolves for this long. A pair in Helix `ERROR` on **every** replica stops the run immediately regardless of this window, since Helix never retries an `ERROR` replica on its own; a pair still `ERROR` on some replica when the window ends fails the run the same way. | | `iceberg.snapshotProcessing.convergence.autoReset.maxErrorPairs` | `1000` | During the convergence wait, a segment `ERROR` on one replica while another replica is `ONLINE` is a server problem, not a segment problem — the segment is loadable. Once per run the controller issues a single bulk segment reset (`ERROR` → `OFFLINE` → reload) for every such pair in the table, then keeps waiting; this cap compares against the table-wide count of resettable pairs, and above it the run fails immediately with the count instead of resetting. `0` disables the reset, reproducing pre-auto-reset behavior: any in-scope `ERROR` pair stops the run at once. A pair `ERROR` on every replica always stops the run immediately regardless of this setting. | Broker-side snapshot pruning activates for **OFFLINE** External Tables when any of these is true on the table: `enableDeletionVectors=true`, `enableSnapshotConsistency=true` (a flag the onboarding/sync flow sets automatically for snapshotting catalogs — pins queries to the last completed snapshot so they never see a half-ingested one), or [segment groups](./segment-groups) enabled. It is a zero-cost no-op otherwise. ## Query-time snapshot pinning You do **not** need to pin a snapshot yourself: for a pruning-enabled table the broker automatically resolves each query to the newest active snapshot and injects the server-required `snapshotVersionByTable` option. Set the option explicitly only to pin a query to a *specific* snapshot (time travel / reproducing an earlier state): ```sql theme={null} SET snapshotVersionByTable = '{"nyc_taxi_trips_OFFLINE": 1234567890123456789}'; SELECT count(*) FROM nyc_taxi_trips; ``` If an explicitly pinned snapshot is not in the table's active set (expired, never existed, or not yet confirmed ready on all servers), the query fails with a `QUERY_EXECUTION` error rather than returning results computed from a mix of snapshot states. A query also fails when the table has no active snapshots at all yet (nothing fully synced), or when the broker hasn't finished loading the pinned snapshot's segment state locally (the broker retries in the background). Only snapshots in the active set can be pinned, and the set's size is bounded by snapshot retention (`iceberg.snapshotProcessing.retention.maxActiveVersions`, default `2`) — so time travel reaches back at most that many snapshots. ## Storage format The deletion-vector index itself is stored as a dictionary-encoded **Parquet** file (`dv-index.parquet`), with rows sorted by segment name within each row group and per-row-group bloom filters, which keeps heap usage low even for large delete sets and supports lazy point lookups. On the servers, cached Puffin files live in the persistent disk cache; the cache survives restarts and is cleaned up when the table is deleted. ## Observability | Component | Metric | Meaning | | :- | :- | :- | | Controller | `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIME_MS` (timer) | Time for all servers to converge on a snapshot's deletion-vector state. | | Controller | `EXTERNAL_TABLE_SYNC_SNAPSHOT_READINESS_POLL_TIME_MS` (timer) | Time spent polling servers for readiness. | | Controller | `EXTERNAL_TABLE_SYNC_READINESS_POLL_SUCCESS` / `_ATTEMPTS` / `_FAST_FAIL` / `_TIMEOUT` / `_SKIPPED` (meters) | Outcome counters for the readiness poll loop. `_SKIPPED` fires when `enableDeletionVectors=false` or when the snapshot produced no deletion vectors (nothing to poll for). | | Controller | `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIMEOUT` (meter) | A snapshot's ideal-state / external-view convergence timed out. | | Controller | `EXTERNAL_TABLE_SYNC_UNHEALTHY` (gauge) | `1` when the table's sync health latch is set (set on a `FAILED` run, carried across snapshots and further failures, cleared only when a run completes `DONE`); `0` otherwise. Emitted every tick, including no-work ticks; removed on controller leadership handoff. | | Controller | `EXTERNAL_TABLE_SYNC_CONSECUTIVE_FAILURES` (gauge) | Consecutive failed sync runs for the table. | | Controller | `EXTERNAL_TABLE_SYNC_RUN_FAILED_STALL` / `_ERROR_REPLICA` / `_EXCEPTION` (meters) | Reason a sync run stopped: a wait (convergence or readiness) made no progress for its stall window, a (segment, instance) pair reached Helix `ERROR`, or any other exception. | | Controller | `EXTERNAL_TABLE_SYNC_AUTO_RESET_PAIRS` (meter) | Table-wide `ERROR` (segment, instance) pairs swept by the one auto-reset a convergence wait may issue per run (see `iceberg.snapshotProcessing.convergence.autoReset.maxErrorPairs` above). A steady rate means servers keep flapping between `ERROR` and `ONLINE` and the reset is masking the cause. | | Server | `DELETION_VECTORS_LOADED` (gauge) | Deletion vectors currently resident in memory, per table and snapshot. | | Server | `DELETION_VECTOR_DELETED_ROWS` / `DELETION_VECTOR_LOAD_FAILURES` / `DELETION_VECTOR_INDEX_FAST_FAIL` (meters), `DELETION_VECTOR_LOAD_TIME_MS` (timer) | Deletion-vector load and application activity per server. | | Broker | `BROKER_ICEBERG_DV_ACTIVE_SNAPSHOTS_COUNT`, `BROKER_ICEBERG_DV_LATEST_SNAPSHOT_NOT_LOADED` (gauges), `BROKER_ICEBERG_DV_FILTERED_SEGMENTS_PER_QUERY`, `BROKER_ICEBERG_DV_PARQUET_DOWNLOAD_FAILURE`, `BROKER_ICEBERG_DV_PRUNE_FAILURE`, `BROKER_ICEBERG_DV_PIN_INVALID_SNAPSHOT`, `BROKER_ICEBERG_DV_WARMUP_FAILURES` (meters), `BROKER_ICEBERG_DV_WARMUP_DURATION_MS` (timer) | Broker-side pruning and warmup health. `BROKER_ICEBERG_DV_LATEST_SNAPSHOT_NOT_LOADED` is `1`, per table, while this broker cannot yet serve the newest entry in the table's active-snapshot list (it falls back to the newest cache-ready older snapshot, or fails queries if none is ready) and `0` once it catches up; removed when the table is deleted or stops pruning. | The run-status endpoint's `failurePhase` field (see [Observability](./observability#run-status)) can also report `WAITING_FOR_SEGMENTS_ONLINE` or `WAITING_FOR_SERVERS_READY` when a sync run fails during deletion-vector convergence. ## FAQs ### Do I need to set `snapshotVersionByTable` on every query once deletion vectors are enabled? No — the broker pins each query to the newest active snapshot automatically and injects the option for the servers. Set it explicitly only for time travel to a specific snapshot; an explicit pin that isn't in the active set fails the query rather than returning potentially inconsistent results. ### Does this work for raw S3 Parquet (non-Iceberg) External Tables? No — this is an Iceberg-specific capability (`deletionVectorSource=iceberg`), covering both native v3 Puffin deletion vectors and standard v2 equality/position delete files; raw Parquet listings have no Iceberg delete-file concept to apply. # FAQ Source: https://docs.startree.ai/corecapabilities/external-table/faq Answers to common questions about StarTree External Tables — supported sources, onboarding, schema, indexes, performance, and operations. > [!WARNING] > > This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it. ## General **What is an External Table?** A Pinot table whose data stays in Parquet files in your object store (an Iceberg catalog such as AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris, Snowflake Open Catalog / Horizon, or Nessie, or a raw S3/GCS data lake) instead of being copied into Pinot's own segment format. Pinot reads the remote data at query time, with server-side caching to keep queries fast. **Which sources are supported?** S3 Data Lake (raw Parquet), GCS Data Lake (raw Parquet via S3 interop), AWS Glue (Iceberg REST), AWS S3 Tables (Iceberg REST), Unity Catalog (Iceberg REST, including Delta Lake tables with UniForm enabled), Apache Polaris and Snowflake Open Catalog (Iceberg REST, OAuth2 principal credentials), Snowflake Horizon Catalog (Iceberg REST, PAT), and Nessie (Iceberg REST, via the generic `rest` service type with a branch/ref `prefix`). **Does StarTree support Delta Lake?** Yes, indirectly: a Delta Lake table with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled publishes Iceberg-compatible metadata alongside its Delta log, so registering it in a **Unity Catalog** makes it queryable as an External Table through the same Iceberg REST path used for Glue and S3 Tables. There is no direct Delta-log reader — UniForm is required. Enabling UniForm means setting `'delta.enableIcebergCompatV3' = 'true'` (Databricks Runtime 18 LTS or later; `'delta.enableIcebergCompatV2' = 'true'` on older runtimes) and `'delta.universalFormat.enabledFormats' = 'iceberg'` on the Delta table, plus column mapping (`'delta.columnMapping.mode' = 'id'` recommended for new tables; existing tables can only be upgraded to `'name'`) — see the [Databricks UniForm requirements](https://docs.databricks.com/aws/en/delta/uniform#requirements) for the full prerequisites. Three operational caveats: Databricks generates the Iceberg metadata **asynchronously** after each Delta commit, so freshly written data reaches StarTree only after that conversion plus the next sync; **materialized views and streaming tables** can only be exposed through the pipeline-managed Compatibility V3 path (Databricks Runtime 17.3 or later, plus `delta.enableRowTracking` and `pipelines.externalMetadata.enabled`); and under Compatibility V3 **deletes may be written as deletion vectors** rather than by rewriting data files — StarTree applies Iceberg v3 Puffin position deletes at query time, but that path is a [preview feature](./deletion-vectors) you have to ask StarTree to enable (V2 tables keep copy-on-write deletes, which need no such setup). See [Unity Catalog: Onboarding via API](./unity/onboarding-api). **Can I query Parquet on S3 or GCS without a catalog at all?** Yes — the **S3 Data Lake** and **GCS Data Lake** sources scan a raw bucket prefix directly, with no Iceberg table, Glue database, or catalog service of any kind. See [S3: Onboarding via API](./s3/onboarding-api) or [GCS: Onboarding via API](./gcs/onboarding-api). **How is it different from a regular Pinot table?** A regular table ingests and stores a local copy of the data as Pinot segments. An External Table leaves the data in place and reads it remotely — so there's no data duplication and onboarding is fast, at the cost of relying on the object store plus cache for read performance. **When should I use one — and when not?** Use it when you already have data in S3/Iceberg and want to query it without copying or re-ingesting — large or infrequently queried datasets, or data shared with other engines. Prefer a regular table for the lowest, most predictable query latency, or when you need an unsupported capability such as a sorted index. **Does Pinot copy the Parquet into Pinot, or read it from S3 directly?** It reads directly from the Parquet files in S3 — the data is never duplicated into Pinot's segment format. Servers cache the bytes they read (data, index, and footer caches) so repeat queries are fast. See [Data and Index Caching](./data-and-index-caching). *** ## Onboarding & sync **How do I onboard one?** Two ways: the Data Portal UI or the REST APIs. The Data Portal wizard covers [AWS Glue](/corecapabilities/external-table/glue/onboarding-data-portal), [Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-data-portal), [Unity Catalog](/corecapabilities/external-table/unity/onboarding-data-portal), [Apache Polaris & Snowflake Open Catalog](/corecapabilities/external-table/polaris/onboarding-data-portal), [Snowflake Horizon](/corecapabilities/external-table/horizon/onboarding-data-portal), [S3 Data Lake](/corecapabilities/external-table/s3/onboarding-data-portal), and [GCS Data Lake](/corecapabilities/external-table/gcs/onboarding-data-portal). The API guides cover those plus [Apache Polaris & Snowflake Open Catalog](/corecapabilities/external-table/polaris/onboarding-api), [Snowflake Horizon](/corecapabilities/external-table/horizon/onboarding-api), and [Nessie](/corecapabilities/external-table/nessie/onboarding-api) — Nessie is the only API-only source, with no Data Portal wizard yet. **Do I have to trigger the first sync?** No. After the table is created, the controller's External Table watcher runs the first sync automatically and then re-syncs on the table's schedule. You can trigger a run manually to start sooner — see [Observability](./observability). **How often does it sync? Can I change the schedule?** It syncs on the `schedule` (Quartz cron) in the table's `ExternalTableSyncTask` config. The Data Portal applies a default of every 5 minutes; via the API you set `schedule` yourself. Match it to your source's commit cadence — polling faster than the source changes just adds overhead. **Can I pause and resume syncing?** Yes. In the Data Portal, open the table and select **Pause Sync** — this removes the sync schedule (stashing it so it can be restored) rather than touching the table's data or checkpoint. In-progress runs finish normally and data is preserved; select **Resume Sync** to pick up where it left off. **Connection validation failed — what do I check?** Almost always a permission or region mismatch. Confirm the credentials can read both the catalog and the underlying S3 data, and that the region matches. For AWS Glue, the warehouse must be the numeric AWS account ID. Test Connection now checks storage credentials up front for every catalog type, and — for S3 Data Lake and S3 Tables specifically — the bucket region too; see [Onboarding fails with a region error](/corecapabilities/external-table/troubleshooting#onboarding-fails-with-a-region-error). **Can multiple tables share the same credentials?** Yes — reuse the same catalog connection parameters across as many External Tables as you like. **Do I put `s3://` in the bucket field?** No. The bucket field takes the bucket name only — not an `s3://...` URL — and the prefix should not have a trailing slash. Make sure the AWS region is set; a missing region is a common cause of onboarding errors. See [Troubleshooting](./troubleshooting#onboarding). **How do I grant access to the source bucket?** Three options: an **assumed IAM role** (set `roleArn`, plus `externalId` if required), the cluster's **node IAM role**, or static **AWS access keys**. The role/keys need `s3:GetObject` and `s3:ListBucket` on the source bucket and prefix. Verify access from the cluster with `aws s3 ls s3:////` before onboarding. **Can I onboard many tables at once?** Onboard each table through the Data Portal or the REST APIs ([AWS Glue](/corecapabilities/external-table/glue/onboarding-api) / [Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-api) / [Unity Catalog](/corecapabilities/external-table/unity/onboarding-api)). There is no single bulk table-creation API today; script the per-table API flow to automate it. *** ## Schema & data types **How are source types mapped to Pinot types?** See [Data Type Mapping](./data-type-mapping) for the Parquet (raw S3) and Iceberg type tables. Complex types (struct, map, list) become JSON or multi-value columns. **Can I edit the inferred schema or change a column's data type?** Avoid changing the data types that the preview/onboarding step infers. Forcing a different type (for example, switching a binary/string column to `INT`) breaks segment generation. Adjusting names, the time column, or null handling is fine. **How do I add an inverted index (or another dictionary-backed index)?** Keep the column's forward index `RAW` and add an explicit `dictionary` block alongside the index — dictionary-backed indexes (inverted, FST, IFST) need it on External Tables: ```json theme={null} "indexes": { "forward": { "encodingType": "RAW" }, "dictionary": {}, "inverted": { "disabled": false } } ``` **Can I rename columns on an External Table?** As of 0.16.0, conditionally yes: for `catalogType=iceberg-rest` sources with full Iceberg snapshot metadata access, enable `schemaEvolution.enabled=true` on the table (see [Best Practices & Configs — Checkpointing & schema evolution](./best-practices-and-configs#checkpointing--schema-evolution)). A rename in the source is tracked as an alias against the existing Pinot column rather than creating a new column, so the Pinot column name is preserved. This only resolves **top-level** field renames (not nested-field renames). Raw S3/GCS Parquet sources have no Iceberg field-id history to key off, so renames there still require a manual schema change; confirm with StarTree support whether your specific catalog adapter (Glue, S3 Tables, Unity, generic REST) has the metadata access this needs. **Why do JSON path queries on a struct column return NULL on my Unity Catalog / Delta UniForm table?** The table was likely created with Delta column mapping (`delta.columnMapping.mode` = `name` or `id`), which stores every Parquet column under an opaque physical `col-` name. Top-level column names are always resolved back to your logical names, but names **inside** complex columns (`struct`, `array`, `map<_,struct>`) are only resolved when `storeNestedSchema.enabled` is on (the default) for the table's `ExternalTableSyncTask` config — if it was set to `false`, the nested keys are stored under their physical names and `JSON_EXTRACT_SCALAR(col, '$.yourField', ...)` returns `NULL`. See [Unity Catalog — Schema options for column mapping](./unity/onboarding-api#schema-options-for-column-mapping). **How is the time column chosen?** Automatically, from the first suitable timestamp/date column, preserving the source granularity. See the [time column section](./data-type-mapping#time-column). **Why are my timestamps off by \~1000×?** On current releases, plain Iceberg `timestamp` / `timestamptz` columns are inferred as **microseconds** (matching Iceberg's storage), `timestamp_ns` as nanoseconds, and `timestamp(p)` from its precision. Tables onboarded on older 0.16.0 builds were inferred as milliseconds regardless of the stored unit — that mismatch produces the \~1000× error. Fix it by setting the time column's granularity explicitly to match the data. See [Data Type Mapping](./data-type-mapping#granularity). **Why did I get a "Requires RAW encoding" error?** External Tables require every column to use RAW (no-dictionary) encoding. A column was configured or defaulted to dictionary encoding. The preview/onboarding flow sets RAW automatically. *** ## Indexes **Which indexes are supported?** Most of them — including inverted, range, timestamp, JSON, text, sparse, star-tree, bloom, FST, IFST, and vector (HNSW, with `storeInSegmentFile: true`). Only **sorted** and **geospatial/H3** are not supported. See [Supported Indexes](./indexes). **Why isn't the sorted index supported?** A sorted index is the column's forward data physically stored in sorted order. External Tables read the Parquet files in place and don't own that layout, so the data can't be reordered. **Can I add an index after creating the table?** Yes — update the table config to add a supported index, the same as any Pinot table. The index is built over the existing remote data. *** ## Performance & caching **Where is the data cached?** On each server: a Parquet data cache, an index cache, and a Parquet footer cache. See [Data and Index Caching](./data-and-index-caching). **The first query on a column is slow, then it's fast. Why?** The first read populates the cache from object storage; later reads hit the cache. Enable pre-warm / preload to populate caches at segment load instead of on first query. **How do I make queries faster?** Add supported indexes for your filters, enable the page cache and preload for the table, and pre-warm at segment load. See [Best Practices and Configs](./best-practices-and-configs) for the relevant knobs. **How do I clear a cache?** Use the server page cache endpoint — see [Data and Index Caching](./data-and-index-caching#clearing-caches). *** ## Operations **What happens when the source data changes or files are deleted?** Each scheduled sync advances to the latest Iceberg snapshot (or, for raw S3/GCS, scans for new files) and ingests what's new; the watermark is visible as `checkpointValue` in the [status endpoint](./observability#run-status). **Append-only sources are fully handled.** How deletes and compaction are handled depends on the source type — see the next two questions. **My source compacts or rewrites files — which source type should I use? (Iceberg REST catalog.)** Use an **Iceberg REST catalog** (`catalogType=iceberg-rest` — AWS Glue, S3 Tables, Unity Catalog, Apache Polaris / Snowflake, or Nessie), not the raw S3/GCS Data Lake source. A raw Data Lake source pins each ingested segment to a **physical Parquet file path**; when the source compacts — rewriting small files into larger ones and deleting the originals — those paths break and the server can no longer load or serve the affected segments (you'll see `java.io.FileNotFoundException` in server logs). A REST catalog instead tracks the table's live file set through **snapshot metadata**, so each sync reconciles to the current post-compaction files. Raw S3/GCS is the right fit only for **append-only** sources whose files are never rewritten. See [Segments fail to load after upstream compaction](./troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction). **Does `continueOnFileError` protect against files deleted or compacted after ingestion?** No. `continueOnFileError=true` only lets a **sync run** skip a file it can't read *while it is generating segments* — it has no effect once a segment is already in the table. If that segment's backing file later disappears (e.g. compacted away upstream), the server fails to load or serve it regardless of this setting; the fix is to use an Iceberg REST catalog for compacting sources (above). **What's the difference between `continueOnError` and `continueOnFileError`?** They are two separate configs, set in different places, and easy to confuse. `continueOnError` is an OSS `ingestionConfig` field that skips **individual bad rows** while reading a file or stream. `continueOnFileError` is an `ExternalTableSyncTask` field that skips an **entire unreadable file** so one bad file doesn't fail the whole run. Both apply only during ingestion/sync — neither repairs an already-ingested segment. *** ## Monitoring & observability **How do I check whether my last sync succeeded?** Call the [status endpoint](./observability#run-status) and read `fileOnboardingRun.status` (`COMPLETED` / `RUNNING` / `FAILED` / `IDLE`). On failure, `failurePhase` and `errorMessage` say where and why. See [Observability](./observability). **Does `status: COMPLETED` mean my data is current?** No. Run status only means the last sync run *finished* — the source may have committed more data since. To judge freshness, call `…/externalTable/status?includeLag=true` and read `syncState`, `lag.percentDataIngested`, and `lag.snapshotsBehind`. See [Ingestion lag](./observability#ingestion-lag). **How far behind the source is my data?** With `?includeLag=true`, subtract `lag.synced.committedAt` from `lag.upstream.committedAt` for a time delta, or read `lag.snapshotsBehind` / `lag.percentDataIngested`. There is deliberately no single "lag in seconds" field. **My sync shows `COMPLETED` but the table is empty or stale — why?** A snapshot whose segment generation keeps failing is retried up to `maxRetries` (default 3) runs, then abandoned; a later run with no newer snapshot then reports `COMPLETED` with `segmentsUploaded: 0`, and `fileOnboardingRun` — which only ever describes the run happening now — hides the earlier failures. Compare `filesDiscovered` vs `segmentsUploaded`, check `syncState`, and check `snapshotIngestion.consecutiveFailedRuns` / `lastRunError` for the hidden failure streak. See [Sync says COMPLETED but no data lands](./troubleshooting#sync-says-completed-but-no-data-lands). **Can I monitor sync from dashboards instead of polling the API?** The controller emits per-phase sync timers (`EXTERNAL_TABLE_SYNC_*_TIME_MS`) for latency dashboards, but there is no lag or per-run success/failure metric — alert on freshness and failures through the status API. See [Metrics and alerting](./observability#metrics-and-alerting). # GCS Data Lake: Onboarding via API Source: https://docs.startree.ai/corecapabilities/external-table/gcs/onboarding-api Onboard raw Parquet files on Google Cloud Storage as a StarTree External Table via the REST API, using GCS's S3-compatible interop endpoint and HMAC keys. > [!WARNING] > > This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it. This page shows how to onboard a GCS Data Lake External Table with the StarTree controller REST APIs instead of the [Data Portal UI](./onboarding-data-portal). Use it for automation, infrastructure-as-code, or onboarding many tables at once. GCS Data Lake uses Google Cloud Storage accessed through its **S3-compatible ("interop") endpoint** (`storage.googleapis.com`). The catalog type is `gcs-interop` and it reuses the same S3 wiring as [S3 Data Lake](../s3/onboarding-api) — only the endpoint, addressing style, and credential source differ. ## How it works Onboarding is four calls. Each one feeds the next: ```text theme={null} 1. Validate & browse POST /connections/browse → validate; list files under the GCS prefix 2. Preview POST /tables/preview → infer schema + enriched table config 3. Create schema POST /schemas → register the schema 4. Create table POST /tables → register the External Table ``` The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output. Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding). > [!NOTE] > > All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` — the StarTree Cloud data-plane proxy that fronts the controller. > > If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity. ## Prerequisites * StarTree release with GCS External Table support enabled, and **tiered storage configured** on the cluster with a `GCS_INTEROPERABLE` tier backend. Contact StarTree support if unsure. * Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth. * A GCS bucket with Parquet files and a GCS HMAC key (access ID + secret) with `storage.objects.get` and `storage.objects.list` on the bucket. Generate HMAC keys in the [Google Cloud Console → Cloud Storage → Settings → Interoperability](https://console.cloud.google.com/storage/settings;tab=interoperability). * The GCS bucket name, key prefix, and the project's region (used as a placeholder — GCS ignores it but the field is required). ## Limitations * **HMAC keys are the only credential type** — there's no IAM-role or workload-identity option for GCS Data Lake today. * **`disable.integrity.protections` must be `true`** for every GCS connection — GCS's checksum behavior differs from AWS S3's, and the AWS SDK mis-validates it otherwise. See the Note below. * **No schema evolution** — same limitation as the S3 Data Lake source; a source-side column rename requires a manual schema change. ## Authentication GCS Data Lake authenticates using HMAC keys (Google's S3-interop credentials) — there's no assumed-role equivalent since this path goes through the AWS SDK's S3-compatible client, not native GCP IAM. Choose one of two methods. > [!NOTE] > > `disable.integrity.protections` must be set to `true` for all GCS connections, regardless of method. GCS rejects the AWS SDK's default request checksums on writes and returns whole-object checksums on range GETs that the SDK mis-validates. Setting this flag relaxes both to `WHEN_REQUIRED`, which is required for reads and writes to succeed. ### Inline HMAC keys Quick tests, or when Secret Manager is not available. Generate HMAC keys in the [Google Cloud Console → Cloud Storage → Settings → Interoperability](https://console.cloud.google.com/storage/settings;tab=interoperability). ```json theme={null} { "catalog.gcs-interop.bucketName": "", "catalog.gcs-interop.prefix": "", "catalog.gcs-interop.region": "auto", "catalog.gcs-interop.endpoint": "https://storage.googleapis.com", "catalog.gcs-interop.accessKey": "", "catalog.gcs-interop.secretKey": "", "catalog.gcs-interop.disable.integrity.protections": "true" } ``` ### GCP Secret Manager (recommended for production) `accessKey`/`secretKey` are treated as **secret names** rather than literal values, and resolved from GCP Secret Manager at read time — no HMAC material stored in the table config. ```json theme={null} { "catalog.gcs-interop.bucketName": "", "catalog.gcs-interop.prefix": "", "catalog.gcs-interop.region": "auto", "catalog.gcs-interop.endpoint": "https://storage.googleapis.com", "catalog.gcs-interop.accessKey": "", "catalog.gcs-interop.secretKey": "", "catalog.gcs-interop.keyType": "SECRET", "catalog.gcs-interop.secretmanagertype": "GCS", "catalog.gcs-interop.gcpprojectid": "", "catalog.gcs-interop.gcpkeypath": "", "catalog.gcs-interop.disable.integrity.protections": "true" } ``` The service account at `gcpkeypath` needs Secret Manager read access to the two named secrets. *** ## Step 1: Validate and browse the connection **`POST /connections/browse`** There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your credentials and connectivity. * Set `path` to `""` to browse the **root prefix** — this both validates the connection and lists files/directories. ### Request | Field | Type | Required | Description | | - | - | - | - | | `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. | | `connection.params` | object | Yes | GCS connection settings (see examples below). | | `path` | string | No | Where to browse. `""` or omitted = root prefix. | **Inline HMAC** ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "gcs-interop", "catalog.gcs-interop.bucketName": "", "catalog.gcs-interop.prefix": "", "catalog.gcs-interop.region": "auto", "catalog.gcs-interop.endpoint": "https://storage.googleapis.com", "catalog.gcs-interop.accessKey": "", "catalog.gcs-interop.secretKey": "", "catalog.gcs-interop.disable.integrity.protections": "true" } }, "path": "" }' ``` **GCP Secret Manager** ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "gcs-interop", "catalog.gcs-interop.bucketName": "", "catalog.gcs-interop.prefix": "", "catalog.gcs-interop.region": "auto", "catalog.gcs-interop.endpoint": "https://storage.googleapis.com", "catalog.gcs-interop.accessKey": "", "catalog.gcs-interop.secretKey": "", "catalog.gcs-interop.keyType": "SECRET", "catalog.gcs-interop.secretmanagertype": "GCS", "catalog.gcs-interop.gcpprojectid": "", "catalog.gcs-interop.gcpkeypath": "", "catalog.gcs-interop.disable.integrity.protections": "true" } }, "path": "" }' ``` ### Response ```json theme={null} { "items": [ { "name": "2024/01/", "type": "DIR" }, { "name": "data.parquet", "type": "FILE" } ] } ``` | `items[].type` | Meaning | | - | - | | `DIR` | A directory you can drill into — pass its `name` as `path`. | | `FILE` | A Parquet file under the prefix. | *** ## Step 2: Preview the schema **`POST /tables/preview`** Samples the Parquet files, infers a Pinot schema, and returns an enriched table config plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4. > [!NOTE] > > The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`. ### Request **Top-level fields:** | Field | Type | Required | Description | | - | - | - | - | | `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. | | `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. | | `schema` | object | No | An explicit schema to use instead of inferring one. | **`config.inference` — how the schema is derived:** | Field | Type | Default | Description | | - | - | - | - | | `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Parquet metadata. **Set this to `true` for External Tables.** | | `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. | **`config.sampling` — how rows are sampled:** | Field | Type | Default | Description | | - | - | - | - | | `min` | integer | `1` | Minimum records to sample. | | `max` | integer | `100` | Maximum records to sample. | > [!TIP] > > To list source files without sampling any data, set `config.previewFiles.previewFilesOnly = true`. The response returns matching file URIs in `sourceFiles` and skips schema inference. (OFFLINE only.) **Inline HMAC** ```bash theme={null} curl -X POST "$BASE_URL/tables/preview" \ -H "Content-Type: application/json" \ -d '{ "tableConfig": { "tableName": "my_gcs_table_OFFLINE", "tableType": "OFFLINE", "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "catalogType": "gcs-interop", "executor": "controller", "inputFormat": "parquet", "catalog.gcs-interop.bucketName": "", "catalog.gcs-interop.prefix": "", "catalog.gcs-interop.region": "auto", "catalog.gcs-interop.endpoint": "https://storage.googleapis.com", "catalog.gcs-interop.accessKey": "", "catalog.gcs-interop.secretKey": "", "catalog.gcs-interop.table.namespace": "default", "catalog.gcs-interop.table.tableName": "my_gcs_table", "catalog.gcs-interop.disable.integrity.protections": "true" } } } }, "config": { "inference": { "embeddedSchemaEnabled": true }, "sampling": { "min": 1, "max": 100 } } }' ``` **GCP Secret Manager** ```bash theme={null} curl -X POST "$BASE_URL/tables/preview" \ -H "Content-Type: application/json" \ -d '{ "tableConfig": { "tableName": "my_gcs_table_OFFLINE", "tableType": "OFFLINE", "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "catalogType": "gcs-interop", "executor": "controller", "inputFormat": "parquet", "catalog.gcs-interop.bucketName": "", "catalog.gcs-interop.prefix": "", "catalog.gcs-interop.region": "auto", "catalog.gcs-interop.endpoint": "https://storage.googleapis.com", "catalog.gcs-interop.accessKey": "", "catalog.gcs-interop.secretKey": "", "catalog.gcs-interop.keyType": "SECRET", "catalog.gcs-interop.secretmanagertype": "GCS", "catalog.gcs-interop.gcpprojectid": "", "catalog.gcs-interop.gcpkeypath": "", "catalog.gcs-interop.table.namespace": "default", "catalog.gcs-interop.table.tableName": "my_gcs_table", "catalog.gcs-interop.disable.integrity.protections": "true" } } } }, "config": { "inference": { "embeddedSchemaEnabled": true }, "sampling": { "min": 1, "max": 100 } } }' ``` ### Response | Field | Type | Description | | - | - | - | | `schema` | object | The resolved Pinot schema. **Use this in step 3.** | | `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. | | `rows` | array | Sample records after schema + ingestion transforms. | | `sourceRows` | array | Raw records before transformation. | | `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). | *** ## Step 3: Create the schema **`POST /schemas`** Send the `schema` object from the preview response. Rename its `schemaName` to match your table. ```bash theme={null} curl -X POST "$BASE_URL/schemas" \ -H "Content-Type: application/json" \ -d @schema.json ``` ```json theme={null} { "status": "my_gcs_table successfully added" } ``` *** ## Step 4: Create the table **`POST /tables`** Send the enriched `tableConfigs.offline` object from the preview response. The `ExternalTableSyncTask` block marks the table as external, and the `tierConfigs` array pins it to the `GCS_INTEROPERABLE` tier. ```bash theme={null} curl -X POST "$BASE_URL/tables" \ -H "Content-Type: application/json" \ -d @tableConfig.json ``` ```json theme={null} { "status": "Table my_gcs_table_OFFLINE successfully added" } ``` The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call. > [!NOTE] > > The `tableConfigs.offline` returned by `/tables/preview` includes a `tierConfigs` block pre-configured for `GCS_INTEROPERABLE`. Verify that the tier backend credentials match the source catalog credentials (they can differ if you use separate HMAC keys or Secret Manager secrets for the deep store). > [!TIP] > > The first sync runs on the watcher's next tick. To kick it off immediately: > > ```bash theme={null} > curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=my_gcs_table_OFFLINE" > ``` *** ## Quickstart: onboard a table end-to-end The whole flow as one script (inline HMAC mode). Fill in the variables at the top, run the script, and the first sync starts automatically. ```bash theme={null} #!/usr/bin/env bash set -euo pipefail BASE_URL="https://dp..cp..startree.cloud/api/pinot" AUTH="Authorization: Bearer " BUCKET="" PREFIX="" REGION="" # Required field; GCS ignores the value HMAC_KEY="" HMAC_SECRET="" TABLE="my_gcs_table" # 1. Validate & browse — confirms credentials and lists files. curl -sf -X POST "$BASE_URL/connections/browse" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"connection\": { \"type\": \"CATALOG\", \"params\": { \"catalogType\": \"gcs-interop\", \"catalog.gcs-interop.bucketName\": \"$BUCKET\", \"catalog.gcs-interop.prefix\": \"$PREFIX\", \"catalog.gcs-interop.region\": \"auto\", \"catalog.gcs-interop.endpoint\": \"https://storage.googleapis.com\", \"catalog.gcs-interop.accessKey\": \"$HMAC_KEY\", \"catalog.gcs-interop.secretKey\": \"$HMAC_SECRET\", \"catalog.gcs-interop.disable.integrity.protections\": \"true\" } }, \"path\": \"\" }" | jq . # 2. Preview — infer schema + enriched table config. curl -sf -X POST "$BASE_URL/tables/preview" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"tableConfig\": { \"tableName\": \"${TABLE}_OFFLINE\", \"tableType\": \"OFFLINE\", \"task\": { \"taskTypeConfigsMap\": { \"ExternalTableSyncTask\": { \"catalogType\": \"gcs-interop\", \"executor\": \"controller\", \"inputFormat\": \"parquet\", \"catalog.gcs-interop.bucketName\": \"$BUCKET\", \"catalog.gcs-interop.prefix\": \"$PREFIX\", \"catalog.gcs-interop.region\": \"auto\", \"catalog.gcs-interop.endpoint\": \"https://storage.googleapis.com\", \"catalog.gcs-interop.accessKey\": \"$HMAC_KEY\", \"catalog.gcs-interop.secretKey\": \"$HMAC_SECRET\", \"catalog.gcs-interop.table.namespace\": \"default\", \"catalog.gcs-interop.table.tableName\": \"$TABLE\", \"catalog.gcs-interop.disable.integrity.protections\": \"true\" } } } }, \"config\": { \"inference\": { \"embeddedSchemaEnabled\": true }, \"sampling\": { \"min\": 1, \"max\": 100 } } }" > preview.json # 3. Extract schema and table config from preview response. jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json jq '.tableConfigs.offline' preview.json > tableConfig.json # 4. Create the schema. curl -sf -X POST "$BASE_URL/schemas" \ -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json # 5. Create the table — watcher starts the first sync automatically. curl -sf -X POST "$BASE_URL/tables" \ -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json # 6. (optional) Poll until sync completes. curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \ | jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered' ``` Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable). ## Monitor onboarding Three read-only endpoints report ingestion progress — run status, ingestion checkpoint, and source file count — and require `executor=controller` (set automatically). See [Observability](../observability) for full request and response details. ## Verify it's queryable When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live: ```sql theme={null} SELECT count(*) FROM my_gcs_table; ``` The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting). *** ## GCS-specific config reference All keys go under the `ExternalTableSyncTask` block in `taskTypeConfigsMap`. | Key | Required | Description | | - | - | - | | `catalog.gcs-interop.bucketName` | Yes | GCS bucket name. | | `catalog.gcs-interop.prefix` | Yes | Key prefix for the Parquet data (e.g. `path/to/data/`). | | `catalog.gcs-interop.region` | Yes | Region string. GCS ignores the value but the field must be present with the value "auto". | | `catalog.gcs-interop.endpoint` | Yes | Always `https://storage.googleapis.com`. | | `catalog.gcs-interop.accessKey` | Yes | HMAC access ID, or GCP Secret Manager secret name if `keyType=SECRET`. | | `catalog.gcs-interop.secretKey` | Yes | HMAC secret, or GCP Secret Manager secret name if `keyType=SECRET`. | | `catalog.gcs-interop.disable.integrity.protections` | Yes | Must be `"true"`. Required for GCS range-GET and PUT compatibility with the AWS SDK. | | `catalog.gcs-interop.keyType` | No | Set to `SECRET` to resolve `accessKey`/`secretKey` via GCP Secret Manager. | | `catalog.gcs-interop.secretmanagertype` | If `keyType=SECRET` | Must be `GCS`. | | `catalog.gcs-interop.gcpprojectid` | If `keyType=SECRET` | GCP project ID for Secret Manager lookups. | | `catalog.gcs-interop.gcpkeypath` | If `keyType=SECRET` | Path to a service-account key JSON with Secret Manager read access. | | `catalog.gcs-interop.table.namespace` | No | Logical namespace. Use `default` for a flat prefix. | | `catalog.gcs-interop.table.tableName` | No | Logical table name (any value; used internally). | *** ## What's next Now that the table is created and data is loading, these are the highest-impact follow-up steps: 1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes) 2. **Enable caching and pin hot indexes.** Set `enable.prefetch.page.cache=true` on the S3 tier, and list small hot indexes in `preload.index.keys.override`, so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching) 3. **Monitor ongoing syncs.** Use the observability endpoints to check run status, ingestion checkpoint, and source file count after each scheduled sync. → [Observability](../observability) *** For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting). # GCS Data Lake: Onboarding via Data Portal Source: https://docs.startree.ai/corecapabilities/external-table/gcs/onboarding-data-portal Connect raw Parquet files on GCS to StarTree as an External Table with the point-and-click Data Portal wizard, via the S3-compatible interop endpoint. > [!WARNING] > > This feature is available starting in **StarTree release 0.16.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment. This guide walks through connecting a GCS Data Lake to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface. > Looking for the API-based approach? See [GCS Data Lake: Onboarding via API](./onboarding-api). *** ## Overview **GCS Data Lake** connects to raw Parquet files stored in a Google Cloud Storage bucket, accessed through GCS's S3-compatible ("interop") endpoint (`storage.googleapis.com`). StarTree scans the specified GCS prefix, discovers Parquet files, and makes them queryable through Pinot. Authentication uses **HMAC keys** (Google's S3-interop credentials). Generate HMAC keys in the [Google Cloud Console → Cloud Storage → Settings → Interoperability](https://console.cloud.google.com/storage/settings;tab=interoperability). *** ## Prerequisites Before starting, ensure you have: * **StarTree 0.16.0 or later** with the external table Beta feature enabled and a `GCS_INTEROPERABLE` tier backend configured for your environment. * A GCS HMAC key (access ID + secret) with `storage.objects.get` and `storage.objects.list` permissions on the source bucket. * The GCS bucket name, key prefix, and GCP project region. *** ## Step 1: Open the External Tables 1. Log in to **Data Portal**. 2. In the left navigation, go to **Tables**. 3. Click **+ Connect External Table**. The wizard opens with a connection configuration screen. *** ## Step 2: Select GCS Data Lake Choose **GCS Data Lake** as the catalog type. Fill in the connection details: | Field | Description | | - | - | | **Bucket Name** | Name of the GCS bucket containing your Parquet files. | | **Region** | GCS S3-interoperability region — the default `auto` works for most buckets. | | **Endpoint** | The GCS S3-interop endpoint; keep the default `https://storage.googleapis.com`. | | **Authentication Type** | **Inline HMAC Keys** or **GCP Secret Manager**. | With **Inline HMAC Keys**: | Field | Description | | - | - | | **HMAC Access ID** | The HMAC access ID from the Google Cloud Console Interoperability settings. | | **HMAC Secret** | The HMAC secret corresponding to the access ID. | With **GCP Secret Manager** (credentials resolved from Secret Manager instead of entered inline): | Field | Description | | - | - | | **Access ID Secret Name** | Secret Manager secret holding the HMAC access ID. | | **Secret Key Secret Name** | Secret Manager secret holding the HMAC secret. | | **GCP Project ID** | Project that owns the secrets. | | **Service Account Key Path** | Path to a GCP service-account JSON key used to access Secret Manager. | Click **Validate Connection**. Data Portal confirms credentials and connectivity via the GCS interop endpoint before proceeding. *** ## Step 3: Browse and Select a Table Once the connection is validated, enter the **Prefix** — the key prefix (folder path) pointing to the Parquet data, e.g. `path/to/parquet/data` (no leading slash; the field auto-sanitizes the value). Then: 1. Data Portal lists the available **GCS prefixes** (directories) under your configured prefix. 2. Select a prefix to view the Parquet files it contains. 3. Click the **table** (prefix) you want to onboard. StarTree samples the Parquet files to derive a schema automatically. *** ## Step 4: Review the Schema The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity. Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct. *** ## Step 5: Configure the Table Set the final table options: | Setting | Default | Notes | | - | - | - | | **Table Name** | — | Name of the Pinot table to create. | | **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. | | **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. | The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config. Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required. *** ## Step 6: Monitor Onboarding Once the table is created, onboarding starts automatically. The table detail view shows the status in real time: * **Running** — the task is actively reading Parquet files and building Pinot segments. * **Completed** — onboarding finished successfully. * **Failed** — onboarding encountered an error. The error message and the number of files discovered vs. segments generated are surfaced to help diagnose the issue. For deeper observability — watcher status, checkpoint values, and per-file counts — see the [Observability](../observability) page. *** ## Pausing Onboarding To pause scheduled onboarding from Data Portal: 1. Open the table in the **Tables** view. 2. Click **Pause Sync**. This removes the `schedule` (cron trigger) from the table's `ExternalTableSyncTask` config and stashes its value under `schedule.paused`, so the watcher stops firing new runs. Any run currently in progress completes normally, and existing segments and the last checkpoint are untouched. Click **Resume Sync** to restore the schedule from `schedule.paused` — onboarding resumes from where it left off; it does not restart from scratch. > [!NOTE] > > This is a separate mechanism from the `enabled` key documented in [Best Practices & Configs](../best-practices-and-configs#table-config--externaltablesynctask) — that key can also be set to `false` directly via the table config or API to stop new runs, but the Data Portal's **Pause Sync** button acts on `schedule`/`schedule.paused`, not `enabled`. *** ## Frequently Asked Questions **The Validate step fails — what should I check?** * Confirm the HMAC key has `storage.objects.get` and `storage.objects.list` permissions on the bucket. * Verify the bucket name and prefix are correct — GCS bucket names are globally unique. * Ensure the HMAC key was generated under the correct GCP project. *** **Can I onboard multiple prefixes from the same bucket?** Yes. Each prefix is registered as an independent Pinot table with its own onboarding schedule. Start the wizard again and use the same bucket with a different prefix. *** **The table was created but onboarding hasn't started — what should I check?** Data Portal triggers the first onboarding run automatically after table creation. If onboarding hasn't started, check the table's detail page for an error status and review the error message. You can also trigger a run manually via the [trigger API](../observability#trigger-a-sync-run-optional). # Getting Started: Connect a Data Catalog Source: https://docs.startree.ai/corecapabilities/external-table/getting-started/connect-a-catalog Connect StarTree to an Iceberg-compatible data catalog — AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris, Snowflake Open Catalog / Horizon, or Nessie — and onboard a registered table as an External Table. A data catalog is a central registry that tracks a table's schema, partitions, and file locations, so a query engine can look those up instead of scanning a bucket to rediscover them. If your tables are already registered in one of the catalogs below, StarTree reads that registry directly — you get schema inference, partition awareness, and incremental snapshot sync for free, instead of the raw file-scanning approach in [Query Parquet Directly](./query-without-a-catalog). > [!TIP] > > **A catalog is also the right choice when your source compacts.** Because a REST catalog tracks the table's live file set through snapshot metadata, each sync reconciles to the current files even after the source rewrites and deletes Parquet during compaction. The raw [S3/GCS Data Lake](./query-without-a-catalog) source pins segments to physical file paths, so compaction breaks them — see [Segments fail to load after upstream compaction](../troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction). If your pipeline compacts, connect a catalog rather than pointing at the bucket directly. ## Which catalog do you have? All of them are reached the same way — `catalogType=iceberg-rest` — over the **Iceberg REST protocol**; only the service endpoint and authentication differ. | Catalog | What it is | Onboarding guide | | - | - | - | | **AWS Glue** | AWS's managed Iceberg REST catalog, backed by the Glue Data Catalog | [Onboarding via API](../glue/onboarding-api) · [via Data Portal](../glue/onboarding-data-portal) | | **Amazon S3 Tables** | AWS's managed Iceberg tables, addressed by table-bucket ARN | [Onboarding via API](../s3tables/onboarding-api) · [via Data Portal](../s3tables/onboarding-data-portal) | | **Unity Catalog** | Databricks-managed or OSS Unity Catalog — also covers Delta Lake tables with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled | [Onboarding via API](../unity/onboarding-api) · [via Data Portal](../unity/onboarding-data-portal) | | **Apache Polaris / Snowflake Open Catalog** | Self-managed Apache Polaris or Snowflake Open Catalog, over the generic Iceberg REST adapter with OAuth2 principal credentials | [Onboarding via API](../polaris/onboarding-api) · [via Data Portal](../polaris/onboarding-data-portal) | | **Snowflake Horizon Catalog** | Snowflake-managed Iceberg tables, served under `/polaris/api/catalog` on the account host, authenticated with a PAT | [Onboarding via API](../horizon/onboarding-api) · [via Data Portal](../horizon/onboarding-data-portal) | | **Nessie** | A Git-like, branch/ref-aware Iceberg catalog, self-hosted or managed | [Onboarding via API](../nessie/onboarding-api) | Not sure which authentication method to use for your catalog, or whether a given feature (assumed-role auth, vended credentials, OAuth2) is supported? Each catalog's onboarding guide has an Authentication section with a runnable config sample per method. ## How onboarding works, regardless of catalog Every catalog above follows the same four calls: **validate & browse** the connection, **preview** to infer a schema, **create the schema**, then **create the table** — after which the controller's watcher takes over and syncs on a schedule with no further manual steps. Each catalog's onboarding guide walks through all four with runnable examples. Validate & browse also checks that the storage credentials can actually reach the data bucket, not just the catalog endpoint — for S3 Data Lake and S3 Tables this includes confirming the configured region matches the bucket at that step; for the other catalogs, where no bucket is known until a table is picked, the storage region is confirmed later, once a table is selected in preview or `CREATE TABLE`. See [Support Matrix](../support-matrix#connection-test-storage-validation). ## What's next Once the table is created, add [indexes](../indexes) for the columns you filter and group by, enable [caching](../data-and-index-caching), and track sync health through [Observability](../observability). # Getting Started: Query Parquet Without a Catalog Source: https://docs.startree.ai/corecapabilities/external-table/getting-started/query-without-a-catalog Query raw Parquet files on S3 or GCS as a StarTree External Table with no Iceberg table, Glue database, or catalog service in front of them. **Yes — if your Parquet files just sit under a bucket prefix with nothing else in front of them, StarTree can query them directly as an External Table.** No Iceberg metadata, no Glue database, no catalog REST endpoint, no separate registration step. Point StarTree at the bucket and prefix and it discovers the files itself. ## When this is the right fit * You have a data pipeline (Spark, Flink, a batch job) that lands Parquet files in S3 or GCS, with no catalog layer on top. * You want to try External Tables against a sample dataset before setting up a full Iceberg catalog. * Your data lake uses a catalog for *other* consumers, but you'd rather point StarTree straight at the underlying files. If your data is already registered in a catalog — AWS Glue, Amazon S3 Tables, Databricks/OSS Unity Catalog, or Nessie — skip ahead to [Connect a Catalog](./connect-a-catalog) instead; StarTree can use the catalog's schema and file-listing directly rather than re-discovering it from raw files. > [!WARNING] > > **Raw S3/GCS is for append-only sources.** A Data Lake source pins each ingested segment to a physical Parquet file path, so it works cleanly only when files are added and never rewritten. If your pipeline **compacts** the data — rewriting small files into larger ones and deleting the originals — those paths break and the affected segments can no longer be loaded or served (`java.io.FileNotFoundException`). For a source that compacts, use an [Iceberg REST catalog](./connect-a-catalog) instead: it tracks the live file set through snapshot metadata and reconciles compaction on each sync. See [Segments fail to load after upstream compaction](../troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction). ## Two sources, same shape Both scan a bucket prefix, infer a schema by sampling the Parquet files, and sync on a schedule — the only difference is which object store and credential type you use. | Source | Object store | Credentials | Onboarding guide | | - | - | - | - | | **S3 Data Lake** | Amazon S3 | Assumed IAM role, cluster node role, or static access keys | [S3: Onboarding via API](../s3/onboarding-api) · [via Data Portal](../s3/onboarding-data-portal) | | **GCS Data Lake** | Google Cloud Storage (via its S3-compatible interop endpoint) | HMAC keys (inline or via GCP Secret Manager) | [GCS: Onboarding via API](../gcs/onboarding-api) · [via Data Portal](../gcs/onboarding-data-portal) | Prefer the REST API for automation and scripting multiple tables; use the Data Portal wizard for a one-off table with no JSON to write. > [!TIP] > > Also see the [SQL DDL](../sql-ddl) shortcut — `CREATE TABLE ... WITH (type = 's3', ...)` creates the same table in one statement instead of the 4-step REST flow. ## What's next Once the table is created, add [indexes](../indexes) for the columns you filter and group by, enable [caching](../data-and-index-caching) so repeat queries don't re-fetch from object storage, and track sync health through [Observability](../observability). # Getting Started: Speed Up External Table Queries Source: https://docs.startree.ai/corecapabilities/external-table/getting-started/speed-up-queries Make StarTree External Table queries fast with the right indexes and caching and preload settings. **External Table queries read Parquet from object storage at query time instead of a local segment, so the fastest path to low latency is indexes and caching — not more compute.** Two changes cover almost every slow-query case: ## 1. Add indexes for your filter and group-by columns Without an index, every query does a full remote scan. A range index on time/numeric columns and an inverted index on the low-cardinality columns you filter on most will fix the majority of slow queries. → [Supported Indexes](../indexes) — what's supported, how to add one, and why every column starts out RAW (no dictionary). ## 2. Turn on caching and preload The first read of any column pays the cost of fetching it from object storage; caching keeps that data local so repeat queries don't pay it again. Preload goes further, pulling small hot indexes (bloom filters, dictionaries) into memory eagerly at segment load instead of on first query. → [Data and Index Caching](../data-and-index-caching) — how the Parquet data cache, index cache, and footer cache work. → [Best Practices and Configs — Caching and index pinning](../best-practices-and-configs#caching-and-index-pinning) — which indexes to pin vs. leave to the cache. ## Still slow? If you've done both and a specific query is still slow or times out, check [Troubleshooting — Query times out or returns "servers not responded"](../troubleshooting#query-times-out-or-returns-servers-not-responded) for the full diagnostic order, including under-provisioned servers and derived-column group-bys that defeat pruning. # AWS Glue: Onboarding via API Source: https://docs.startree.ai/corecapabilities/external-table/glue/onboarding-api Onboard an AWS Glue Iceberg table as a StarTree External Table over the REST catalog protocol, with IAM-role, node-role, or static-key auth. > [!WARNING] > > This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it. This page shows how to onboard an **AWS Glue** Iceberg External Table with the StarTree controller REST APIs instead of the [Data Portal UI](./onboarding-data-portal). Use it for automation, infrastructure-as-code, or onboarding many tables at once. Glue tables are accessed over the **Iceberg REST protocol** (`catalogType=iceberg-rest`, `serviceType=glue`) with AWS SigV4 authentication. Data files must be **Parquet**. > Using **Amazon S3 Tables** instead? See [Amazon S3 Tables: Onboarding via API](../s3tables/onboarding-api). > > Using **Nessie** instead? See [Nessie: Onboarding via API](../nessie/onboarding-api). ## How it works Onboarding is four calls. Each one feeds the next: ```text theme={null} 1. Validate & browse POST /connections/browse → validate; list namespaces/tables 2. Preview POST /tables/preview → infer schema + enriched table config 3. Create schema POST /schemas → register the schema 4. Create table POST /tables → register the External Table ``` The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output. Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding). > [!NOTE] > > All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` — the StarTree Cloud data-plane proxy that fronts the controller. > > If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity. ## Prerequisites * StarTree 0.15.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure. * Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth. * AWS credentials (access key + secret key) with read access to the Glue catalog **and** the underlying S3 data. * Your AWS account ID (the catalog warehouse) and the Glue database name. > [!WARNING] > > **Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly. > > If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details. ## Limitations * **Iceberg tables only.** This path reads Glue's Iceberg REST Catalog API — plain Hive/Parquet tables registered in Glue (non-Iceberg) aren't reachable here. A separate native AWS Glue Data Catalog code path exists, but it can only browse databases/tables today, not complete onboarding. * **`catalog.iceberg-rest.auth.rest.service` must be set to `glue` explicitly** or the SigV4 signature is computed for the wrong AWS service — see the Warning under [Catalog (REST) authentication](#catalog-rest-authentication-aws-sigv4). ## Authentication An External Table authenticates in two places, configured independently so the catalog and the data can use different credentials: * **Catalog (REST)** — keys under `catalog.iceberg-rest.auth.rest.*`. Calls to the Glue Iceberg REST endpoint always use AWS SigV4; the signing credentials come from static keys or an assumed IAM role. * **Storage (S3 data files)** — keys under `catalog.iceberg-rest.auth.storage.*`. Choose any one of three methods. The role or keys need `s3:GetObject` and `s3:ListBucket` on the source bucket/prefix, plus Glue read access (`glue:GetDatabase`, `glue:GetTable`, `glue:GetPartitions`) on the catalog. Verify S3 access from the cluster before onboarding: ```bash theme={null} aws s3 ls s3://// ``` ### Catalog (REST) authentication: AWS SigV4 The Glue Iceberg REST endpoint accepts only SigV4-signed requests. The signing credentials can come from **static keys** or an **assumed IAM role** — pick one. #### Static keys `authType` is optional — it's auto-detected as `aws-sigv4` once `accessKeyId`/`secretAccessKey`/`region` are all present. ```json theme={null} { "catalog.iceberg-rest.auth.rest.authType": "aws-sigv4", "catalog.iceberg-rest.auth.rest.accessKeyId": "", "catalog.iceberg-rest.auth.rest.secretAccessKey": "", "catalog.iceberg-rest.auth.rest.region": "", "catalog.iceberg-rest.auth.rest.service": "glue" } ``` #### Assumed IAM role SigV4 signing with role-vended credentials — no static secrets on the catalog surface. `auth.rest.region` is **required** when a catalog-side `roleArn` is set (falls back to `auth.storage.region` if that is set instead); static `auth.rest` keys are ignored once a role is configured; and the `AssumeRole` call authenticates with the cluster's ambient AWS identity (node role / IRSA), which the role's trust policy must trust. ```json theme={null} { "catalog.iceberg-rest.auth.rest.roleArn": "arn:aws:iam::123456789012:role/pinot-glue-catalog-reader", "catalog.iceberg-rest.auth.rest.externalId": "", "catalog.iceberg-rest.auth.rest.region": "", "catalog.iceberg-rest.auth.rest.service": "glue" } ``` > [!NOTE] > > If you set a catalog-side `roleArn` but **no** storage-side `roleArn`, data-file reads through the catalog's default S3 client also assume the catalog role — set `auth.storage.roleArn` explicitly if storage should use a different role. > [!WARNING] > > Always set `catalog.iceberg-rest.auth.rest.service` to `glue` explicitly. If left unset it defaults to `s3tables`, and Glue rejects the resulting SigV4 signature. ### Storage authentication: choose one #### Assumed IAM role (recommended for production) No static secrets in the table config — every S3 read assumes `roleArn` via STS. Three things to know: `auth.storage.region` is **required** whenever a `roleArn` is set (the sync fails fast without it); any static `auth.storage` access keys are **ignored** once a `roleArn` is present; and the `AssumeRole` call itself authenticates with the cluster's ambient AWS identity (node role / IRSA), so the target role's trust policy must trust that identity (plus the `externalId`, if you set one). ```json theme={null} { "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Cluster node role Omit every `auth.storage.*` credential key — only `region` is required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have S3 access. ```json theme={null} { "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Static access keys Quick tests, or when role-based access isn't available. Use the canonical `accessKeyId` / `secretAccessKey` spellings (an optional `sessionToken` is also accepted). Avoid the shorter `accessKey`/`secretKey` aliases: only part of the pipeline honors them (segment generation does, Iceberg manifest and data-file reads do not), so a short-alias config falls back to the cluster's ambient credentials for catalog file reads and can fail in confusing ways. ```json theme={null} { "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` *** ## Step 1: Validate and browse the connection **`POST /connections/browse`** There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your credentials and connectivity. Bad credentials, an unreachable host, or an unknown catalog type return an error. * Set `path` to `""` to browse the **root** — this both validates the connection and lists Glue databases (namespaces). * Set `path` to a database name to list the **tables** inside it. ### Request | Field | Type | Required | Description | | - | - | - | - | | `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. | | `connection.params` | object | Yes | Catalog-specific connection settings (see example below). | | `path` | string | No | Where to browse. `""` or omitted = root. A namespace name = that namespace's tables. | ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "iceberg-rest", "catalog.iceberg-rest.restUri": "https://glue..amazonaws.com", "catalog.iceberg-rest.serviceType": "glue", "catalog.iceberg-rest.warehouse": "123456789012", "catalog.iceberg-rest.auth.rest.authType": "aws-sigv4", "catalog.iceberg-rest.auth.rest.accessKeyId": "", "catalog.iceberg-rest.auth.rest.secretAccessKey": "", "catalog.iceberg-rest.auth.rest.region": "", "catalog.iceberg-rest.auth.rest.service": "glue", "catalog.iceberg-rest.auth.storage.authType": "aws-sigv4", "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } }, "path": "" }' ``` | Key | Value | Description | | - | - | - | | `catalog.iceberg-rest.restUri` | `https://glue..amazonaws.com` | The Glue endpoint for your region. | | `catalog.iceberg-rest.serviceType` | `glue` | Selects the built-in Glue adapter. | | `catalog.iceberg-rest.warehouse` | AWS account ID | The Glue warehouse identifier. | | `catalog.iceberg-rest.auth.rest.service` | `glue` | The SigV4 signing service name. | ### Response ```json theme={null} { "items": [ { "name": "transportation", "type": "NAMESPACE" }, { "name": "nyc_taxi_trips", "type": "TABLE" } ] } ``` | `items[].type` | Meaning | | - | - | | `NAMESPACE` | A Glue database you can drill into — pass its `name` as `path`. | | `TABLE` | A selectable table. | *** ## Step 2: Preview the schema **`POST /tables/preview`** Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4. > [!NOTE] > > The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`. ### Request **Top-level fields:** | Field | Type | Required | Description | | - | - | - | - | | `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. | | `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. | | `schema` | object | No | An explicit schema to use instead of inferring one. | **`config.inference` — how the schema is derived:** | Field | Type | Default | Description | | - | - | - | - | | `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Iceberg metadata. **Set this to `true` for External Tables.** | | `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. | Resolution order: explicit `schema` → embedded schema → inferred schema. **`config.sampling` — how rows are sampled:** | Field | Type | Default | Description | | - | - | - | - | | `min` | integer | `1` | Minimum records to sample. | | `max` | integer | `100` | Maximum records to sample. | > [!TIP] > > To list source files without sampling any data, set `config.previewFiles.previewFilesOnly = true`. The response returns matching file URIs in `sourceFiles` and skips schema inference. (OFFLINE only.) ```bash theme={null} curl -X POST "$BASE_URL/tables/preview" \ -H "Content-Type: application/json" \ -d '{ "tableConfig": { "tableName": "nyc_taxi_trips_OFFLINE", "tableType": "OFFLINE", "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "catalogType": "iceberg-rest", "executor": "controller", "inputFormat": "parquet", "catalog.iceberg-rest.restUri": "https://glue..amazonaws.com", "catalog.iceberg-rest.serviceType": "glue", "catalog.iceberg-rest.warehouse": "123456789012", "catalog.iceberg-rest.table.namespace": "transportation", "catalog.iceberg-rest.table.tableName": "nyc_taxi_trips", "catalog.iceberg-rest.auth.rest.authType": "aws-sigv4", "catalog.iceberg-rest.auth.rest.accessKeyId": "", "catalog.iceberg-rest.auth.rest.secretAccessKey": "", "catalog.iceberg-rest.auth.rest.region": "", "catalog.iceberg-rest.auth.rest.service": "glue", "catalog.iceberg-rest.auth.storage.authType": "aws-sigv4", "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } } } }, "config": { "inference": { "embeddedSchemaEnabled": true } } }' ``` > [!NOTE] > > **Setting the namespace and table.** Take them from the [browse](#step-1-validate-and-browse-the-connection) response — set `catalog.iceberg-rest.table.namespace` / `.tableName` to the `NAMESPACE` (Glue database) and `TABLE` you selected. > > **Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4. ### Response | Field | Type | Description | | - | - | - | | `schema` | object | The resolved Pinot schema. **Use this in step 3.** | | `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. | | `rows` | array | Sample records after schema + ingestion transforms. | | `sourceRows` | array | Raw records before transformation. | | `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). | Adjust the schema (time column, column names, null handling) before moving on. *** ## Step 3: Create the schema **`POST /schemas`** Send the `schema` object from the preview response. Rename its `schemaName` to match your table. ```bash theme={null} curl -X POST "$BASE_URL/schemas" \ -H "Content-Type: application/json" \ -d @schema.json ``` ```json theme={null} { "status": "nyc_taxi_trips successfully added" } ``` *** ## Step 4: Create the table **`POST /tables`** Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external. ```bash theme={null} curl -X POST "$BASE_URL/tables" \ -H "Content-Type: application/json" \ -d @tableConfig.json ``` ```json theme={null} { "status": "Table nyc_taxi_trips_OFFLINE successfully added" } ``` The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call. > [!TIP] > > The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run: > > ```bash theme={null} > curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE" > ``` *** ## Quickstart: onboard a table end-to-end The whole flow as one script. Fill in the variables at the top, run the script, and the first sync starts automatically. ```bash theme={null} #!/usr/bin/env bash set -euo pipefail BASE_URL="https://dp..cp..startree.cloud/api/pinot" AUTH="Authorization: Bearer " GLUE_URI="https://glue..amazonaws.com" WAREHOUSE="123456789012" # AWS account ID NAMESPACE="transportation" # Glue database TABLE="nyc_taxi_trips" REGION="" ACCESS_KEY="" SECRET_KEY="" # 1. Validate & browse root — lists Glue namespaces. curl -sf -X POST "$BASE_URL/connections/browse" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"connection\": { \"type\": \"CATALOG\", \"params\": { \"catalogType\": \"iceberg-rest\", \"catalog.iceberg-rest.restUri\": \"$GLUE_URI\", \"catalog.iceberg-rest.serviceType\": \"glue\", \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\", \"catalog.iceberg-rest.auth.rest.authType\": \"aws-sigv4\", \"catalog.iceberg-rest.auth.rest.accessKeyId\": \"$ACCESS_KEY\", \"catalog.iceberg-rest.auth.rest.secretAccessKey\": \"$SECRET_KEY\", \"catalog.iceberg-rest.auth.rest.region\": \"$REGION\", \"catalog.iceberg-rest.auth.rest.service\": \"glue\", \"catalog.iceberg-rest.auth.storage.authType\": \"aws-sigv4\", \"catalog.iceberg-rest.auth.storage.accessKeyId\": \"$ACCESS_KEY\", \"catalog.iceberg-rest.auth.storage.secretAccessKey\": \"$SECRET_KEY\", \"catalog.iceberg-rest.auth.storage.region\": \"$REGION\" } }, \"path\": \"\" }" | jq . # 2. Preview — infer schema + enriched table config. curl -sf -X POST "$BASE_URL/tables/preview" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"tableConfig\": { \"tableName\": \"${TABLE}_OFFLINE\", \"tableType\": \"OFFLINE\", \"task\": { \"taskTypeConfigsMap\": { \"ExternalTableSyncTask\": { \"catalogType\": \"iceberg-rest\", \"executor\": \"controller\", \"inputFormat\": \"parquet\", \"catalog.iceberg-rest.restUri\": \"$GLUE_URI\", \"catalog.iceberg-rest.serviceType\": \"glue\", \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\", \"catalog.iceberg-rest.table.namespace\": \"$NAMESPACE\", \"catalog.iceberg-rest.table.tableName\": \"$TABLE\", \"catalog.iceberg-rest.auth.rest.authType\": \"aws-sigv4\", \"catalog.iceberg-rest.auth.rest.accessKeyId\": \"$ACCESS_KEY\", \"catalog.iceberg-rest.auth.rest.secretAccessKey\": \"$SECRET_KEY\", \"catalog.iceberg-rest.auth.rest.region\": \"$REGION\", \"catalog.iceberg-rest.auth.rest.service\": \"glue\", \"catalog.iceberg-rest.auth.storage.authType\": \"aws-sigv4\", \"catalog.iceberg-rest.auth.storage.accessKeyId\": \"$ACCESS_KEY\", \"catalog.iceberg-rest.auth.storage.secretAccessKey\": \"$SECRET_KEY\", \"catalog.iceberg-rest.auth.storage.region\": \"$REGION\" } } } }, \"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } } }" > preview.json # 3. Extract schema and table config from preview response. jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json jq '.tableConfigs.offline' preview.json > tableConfig.json # 4. Create the schema. curl -sf -X POST "$BASE_URL/schemas" \ -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json # 5. Create the table — watcher starts the first sync automatically. curl -sf -X POST "$BASE_URL/tables" \ -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json # 6. (optional) Poll until sync completes. curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \ | jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered' ``` Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable). ## Monitor onboarding Three read-only endpoints report ingestion progress — run status, ingestion checkpoint, and source file count — and require `executor=controller` (set automatically). See [Observability](../observability) for full request and response details. ## Verify it's queryable When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips; ``` The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting). *** ## What's next Now that the table is created and data is loading, these are the highest-impact follow-up steps: 1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes) 2. **Enable caching and pin hot indexes.** Set `enable.prefetch.page.cache=true` on the S3 tier, and list small hot indexes in `preload.index.keys.override`, so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching) 3. **Monitor ongoing syncs.** Use the observability endpoints to check run status, ingestion checkpoint, and source file count after each scheduled sync. → [Observability](../observability) *** For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting). # AWS Glue: Onboarding via Data Portal Source: https://docs.startree.ai/corecapabilities/external-table/glue/onboarding-data-portal Connect an AWS Glue Iceberg catalog to StarTree as an External Table with the point-and-click Data Portal wizard — no API calls required. > [!WARNING] > > This feature is available starting in **StarTree release 0.15.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment. This guide walks through connecting an **AWS Glue** Iceberg catalog to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface. Glue tables are accessed over the **Iceberg REST protocol** with AWS SigV4 authentication. > Looking for the API-based approach? See [AWS Glue: Onboarding via API](./onboarding-api). > > Using **Amazon S3 Tables** instead? See [Amazon S3 Tables: Onboarding via Data Portal](../s3tables/onboarding-data-portal). *** ## Prerequisites Before starting, ensure you have: * **StarTree 0.15.0 or later** with the external table Beta feature enabled for your environment. * AWS credentials (access key + secret key) with read permissions on both the Glue catalog service and the underlying S3 data. * Your **AWS account ID** (used as the Glue warehouse identifier). * The target **Glue database** name. > [!WARNING] > > **Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly. > > If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details. *** ## Step 1: Open the External Tables 1. Log in to **Data Portal**. 2. In the left navigation, go to **Tables**. 3. Click **+ Connect External Table**. The wizard opens with a connection configuration screen. *** ## Step 2: Configure the Glue Connection Choose **AWS Glue** as the catalog type. Fill in the credentials and connection details: **Details** | Field | Description | | - | - | | **Connection Name** | A unique name to identify this connection in Data Portal. | **Metastore** — how StarTree authenticates with the AWS Glue catalog API. | Field | Description | | - | - | | **AWS Account ID** | The AWS account that owns the Glue Data Catalog — used as the warehouse identifier. | | **Authentication Type** | **Basic Authentication** (access key + secret key), **IAM Node Role** (the cluster node's own IAM identity — the form displays its ARN under **IAM Role Name** so you can grant it access in your AWS account), or **IAM Role** (a cross-account role StarTree assumes). | | **Access Key** / **Secret Key** | With **Basic Authentication**: static AWS credentials with Glue read access. | | **IAM Role ARN** | With **IAM Role**: the ARN of the role to assume. The form shows **Steps to create IAM Policy** — a ready-made trust policy including the StarTree account and External ID to configure on the role. | | **Region** | AWS region where your Glue catalog is located, e.g. `us-east-1`. | **Storage** — credentials for reading the underlying Parquet data files from S3. | Field | Description | | - | - | | **Reuse metastore credentials** | **Yes** reuses the Metastore authentication for S3 data reads (the same three auth types apply). **No** opens a separate set of storage fields. | | **Region** | With **No**: AWS region where the S3 data files are stored. | | **Authentication Type** | With **No**: **Basic Authentication** (Access Key + Secret Key), **IAM Node Role**, or **IAM Role** (IAM Role ARN + trust-policy steps) — same choices as the Metastore section, configured independently. | Click **Validate Connection**. Data Portal calls the catalog's validate endpoint and confirms credentials and connectivity before proceeding. *** ## Step 3: Browse and Select a Table Once the connection is validated: 1. Data Portal lists the available **Glue databases** (namespaces). 2. Select a database to expand its tables. 3. Click the **table** you want to onboard. Data Portal reads the Iceberg schema and derives a Pinot schema automatically. *** ## Step 4: Review the Schema The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity. Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct. *** ## Step 5: Configure the Table Set the final table options: | Setting | Default | Notes | | - | - | - | | **Table Name** | — | Name of the Pinot table to create. | | **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. | | **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. | The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config. Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required. *** ## Step 6: Monitor Onboarding Once the table is created, onboarding starts automatically. The table detail view shows the status in real time: * **Running** — the task is actively reading Iceberg snapshots and building Pinot segments. * **Completed** — onboarding finished successfully. The last ingested snapshot ID is shown. * **Failed** — onboarding encountered an error. The error message and the number of files discovered vs. segments generated are surfaced to help diagnose the issue. For deeper observability — watcher status, checkpoint values, and per-snapshot file counts — see the [Observability](../observability) page. *** ## Pausing Onboarding To pause scheduled onboarding from Data Portal: 1. Open the table in the **Tables** view. 2. Click **Pause Sync**. This removes the `schedule` (cron trigger) from the table's `ExternalTableSyncTask` config and stashes its value under `schedule.paused`, so the watcher stops firing new runs. Any run currently in progress completes normally, and existing segments and the last checkpoint are untouched. Click **Resume Sync** to restore the schedule from `schedule.paused` — onboarding resumes from where it left off; it does not restart from scratch. > [!NOTE] > > This is a separate mechanism from the `enabled` key documented in [Best Practices & Configs](../best-practices-and-configs#table-config--externaltablesynctask) — that key can also be set to `false` directly via the table config or API to stop new runs, but the Data Portal's **Pause Sync** button acts on `schedule`/`schedule.paused`, not `enabled`. *** ## Frequently Asked Questions **The Validate step fails — what should I check?** * Confirm the access key has `glue:GetCatalog`, `glue:GetDatabase(s)`, `glue:GetTable(s)`, `glue:GetPartition(s)`, `glue:BatchGetPartition`, and `s3:GetObject` permissions (plus `lakeformation:GetDataAccess` if the database is registered with Lake Formation). The **Steps to create IAM Policy** panel in the connection form generates the exact policy for you. * Verify the region matches where your Glue database lives. * Ensure the warehouse value is your numeric AWS account ID, not an account alias. *** **Can I onboard multiple tables from the same catalog?** Yes. After creating the first table, start the wizard again and reuse the same connection credentials. Each table is registered as an independent Pinot table with its own onboarding schedule. *** **The table was created but onboarding hasn't started — what should I check?** Data Portal triggers the first onboarding run automatically after table creation. If onboarding hasn't started, check the table's detail page for an error status and review the error message. You can also trigger a run manually via the [trigger API](../observability#trigger-a-sync-run-optional). # Snowflake Horizon Catalog: Onboarding via API Source: https://docs.startree.ai/corecapabilities/external-table/horizon/onboarding-api Query Snowflake-managed Iceberg tables in StarTree — onboard a Snowflake Horizon Catalog table as an External Table over the Iceberg REST API, authenticating with a Snowflake programmatic access token (PAT). > [!WARNING] > > This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it. This page shows how to onboard a **Snowflake Horizon Catalog** External Table with the StarTree controller REST APIs. Prefer a point-and-click setup? The Data Portal wizard has a dedicated **Snowflake Horizon Catalog** tile — see [Onboarding via Data Portal](./onboarding-data-portal). Snowflake Horizon exposes Snowflake-managed Iceberg tables over the same **Iceberg REST protocol** as Apache Polaris, served on the Snowflake account host under `/polaris/api/catalog`. StarTree reaches it through the generic spec-compliant adapter — `catalogType=iceberg-rest`, `serviceType=rest`. Data files must be **Parquet**. > Running **self-managed Apache Polaris** or **Snowflake Open Catalog** instead? Those use principal `client-id`/`client-secret` credentials — see [Apache Polaris & Snowflake Open Catalog: Onboarding via API](../polaris/onboarding-api). ## How Horizon differs from Apache Polaris Horizon speaks the Polaris REST protocol, but authentication and identifier mapping follow Snowflake's model, not Polaris's: | | Snowflake Horizon Catalog | | - | - | | REST base URI | `https://-.snowflakecomputing.com/polaris/api/catalog` | | Identity | Snowflake **service user** + a **programmatic access token (PAT)** (or key-pair JWT) | | Catalog credential | the **PAT**, passed via `catalog.iceberg-rest.credential` (see [Authentication](#authentication)) | | `oauthScopes` | `session:role:` — Polaris-style `PRINCIPAL_ROLE:ALL` returns `401` | | `warehouse` | the Snowflake **database** name | | Namespace | the Snowflake **schema** (single level; the database is the prefix) | | `prefix` | **required** — set it to the database name (same value as `warehouse`) | | Access control | native Snowflake RBAC (`GRANT SELECT` / `USAGE`); the PAT also needs a **network policy** | > [!NOTE] > > **Why the PAT goes in `catalog.iceberg-rest.credential`.** Horizon's token endpoint expects a `client_credentials` exchange that sends the PAT as `client_secret` with **no** `client_id`. Passing the PAT through `catalog.iceberg-rest.credential` produces exactly that request. (Setting `oauthClientSecret` with an empty `oauthClientId` does **not** currently work — the empty client id causes the credential to be dropped and the catalog call goes out unauthenticated. Use `catalog.iceberg-rest.credential` until first-class secret-only support lands.) ## How it works Onboarding is four calls. Each one feeds the next: ```text theme={null} 1. Validate & browse POST /connections/browse → validate; list namespaces/tables 2. Preview POST /tables/preview → infer schema + enriched table config 3. Create schema POST /schemas → register the schema 4. Create table POST /tables → register the External Table ``` The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output. Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding). > [!NOTE] > > All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` (the StarTree Cloud data-plane proxy). A browse call then posts to `$BASE_URL/connections/browse`. > > If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity. ## Prerequisites * StarTree 0.16.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure. * Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth. * A Snowflake account with **Horizon Catalog** and the target Iceberg table backed by an **external volume** (S3). * A Snowflake **service user** with a **programmatic access token (PAT)**, a role that can read the table, and a **network policy** governing the user (see [Snowflake setup](#snowflake-setup) below). * The Snowflake **database** (used as both `warehouse` and `prefix`), the **schema** (the namespace), and the **table** to onboard. * Read access to the underlying object storage. This connector supports **S3-compatible storage** for the data files. ## Snowflake setup Run these once in Snowflake (or have an ACCOUNTADMIN do so). Replace the placeholders with your own names. ```sql theme={null} -- 1. A role that can read the Iceberg table's database, schema, and external volume. GRANT USAGE ON DATABASE TO ROLE ; GRANT USAGE ON SCHEMA . TO ROLE ; GRANT SELECT ON ALL TABLES IN SCHEMA . TO ROLE ; GRANT USAGE ON EXTERNAL VOLUME TO ROLE ; -- 2. A dedicated service user, and the role granted to it. CREATE USER IF NOT EXISTS DEFAULT_ROLE = ; GRANT ROLE TO USER ; -- 3. A network policy is REQUIRED before a PAT can be created or used. ALTER USER SET NETWORK_POLICY = ; -- 4. The programmatic access token. Copy the secret it prints — it is shown only once. ALTER USER IF EXISTS ADD PAT startree_external_table DAYS_TO_EXPIRY = 90 ROLE_RESTRICTION = '' COMMENT = 'StarTree External Table catalog access'; ``` > [!NOTE] > > The `ROLE_RESTRICTION` role must match the role named in the `session:role:` scope you configure below, and it must hold the grants from step 1. Without a **network policy** in effect, `ADD PAT` fails and the token endpoint later returns `401 unauthorized_client`. ## Authentication An External Table authenticates in two places, configured independently: * **Catalog (REST)** — the Snowflake PAT, supplied via `catalog.iceberg-rest.credential` plus the token URI and scope. * **Storage (S3 data files)** — keys under `catalog.iceberg-rest.auth.storage.*`. Choose one of the methods below. ### Catalog (REST) authentication Pass the **PAT** as the catalog credential. StarTree exchanges it at the Snowflake token endpoint for a short-lived token, which it caches and refreshes automatically. ```json theme={null} { "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens", "catalog.iceberg-rest.auth.rest.oauthScopes": "session:role:", "catalog.iceberg-rest.credential": "" } ``` | Key | Value | Description | | - | - | - | | `catalog.iceberg-rest.credential` | `` | The Snowflake **programmatic access token**. Sent to the token endpoint as `client_secret` with no client id. | | `catalog.iceberg-rest.auth.rest.oauthTokenUri` | `https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens` | Snowflake's OAuth token endpoint (the account host, `/polaris/api/catalog/v1/oauth/tokens`). | | `catalog.iceberg-rest.auth.rest.oauthScopes` | `session:role:` | Activates the Snowflake role for the session. Must match the PAT's `ROLE_RESTRICTION`. | | `catalog.iceberg-rest.auth.rest.authType` | `oauth2` | Optional; auto-detected as `oauth2` when a credential is present. | > [!NOTE] > > The PAT is a **secret** — treat the table config as sensitive and rotate the token on the `DAYS_TO_EXPIRY` cadence. A static PAT is not auto-rotated; when it expires, syncs fail until you update the table config with a new one. Key-pair JWT auth (a signed JWT in place of the PAT) works the same way — put the JWT in `catalog.iceberg-rest.credential`. ### Storage authentication: choose one #### Static access keys What most Snowflake external-volume setups use. Use the canonical `accessKeyId` / `secretAccessKey` spellings (an optional `sessionToken` is also accepted). Avoid the shorter `accessKey`/`secretKey` aliases: only part of the pipeline honors them (segment generation does, Iceberg manifest and data-file reads do not), so a short-alias config falls back to the cluster's ambient credentials for catalog file reads and can fail in confusing ways. ```json theme={null} { "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Assumed IAM role (recommended for production) No static secrets in the table config — every S3 read assumes `roleArn` via STS. Three things to know: `auth.storage.region` is **required** whenever a `roleArn` is set (the sync fails fast without it); any static `auth.storage` access keys are **ignored** once a `roleArn` is present; and the `AssumeRole` call itself authenticates with the cluster's ambient AWS identity (node role / IRSA), so the target role's trust policy must trust that identity (plus the `externalId`, if you set one). ```json theme={null} { "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Cluster node role Omit every `auth.storage.*` credential key — only `region` is required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have S3 access. ```json theme={null} { "catalog.iceberg-rest.auth.storage.region": "" } ``` > [!NOTE] > > Horizon can vend short-lived storage credentials, but the generic `rest` adapter does not consume vended credentials — configure static keys, an assumed role, or the cluster's node role for storage explicitly. ### Selecting the database Set both `catalog.iceberg-rest.warehouse` and `catalog.iceberg-rest.prefix` to the Snowflake **database** name. `prefix` is **required** for Horizon: the generic adapter doesn't send the `warehouse` query parameter that Snowflake's `/v1/config` gates on, so without an explicit prefix, browse falls back to the un-prefixed `/v1/namespaces` path and fails with `404 Unable to find matching target resource method`. ```json theme={null} { "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.prefix": "" } ``` The Snowflake **schema** is the Iceberg namespace — set `catalog.iceberg-rest.table.namespace` to the schema name and `catalog.iceberg-rest.table.tableName` to the table. *** ## Step 1: Validate and browse the connection **`POST /connections/browse`** There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your PAT, scope, and connectivity. * Set `path` to `""` to browse the **root** — this both validates the connection and lists schemas (namespaces). * Set `path` to a schema name to list the **tables** inside it. ### Request | Field | Type | Required | Description | | - | - | - | - | | `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. | | `connection.params` | object | Yes | Catalog-specific connection settings (see example below). | | `path` | string | No | Where to browse. `""` or omitted = root. A schema name = that schema's tables. | ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "iceberg-rest", "catalog.iceberg-rest.serviceType": "rest", "catalog.iceberg-rest.restUri": "https://-.snowflakecomputing.com/polaris/api/catalog", "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.prefix": "", "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens", "catalog.iceberg-rest.auth.rest.oauthScopes": "session:role:", "catalog.iceberg-rest.credential": "", "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } }, "path": "" }' ``` | Key | Value | Description | | - | - | - | | `catalog.iceberg-rest.restUri` | `https://-.snowflakecomputing.com/polaris/api/catalog` | The Snowflake Iceberg REST base URI. | | `catalog.iceberg-rest.serviceType` | `rest` | Selects the generic spec-compliant adapter — there's no Snowflake-specific `serviceType`. | | `catalog.iceberg-rest.warehouse` | `` | The Snowflake database to browse. | | `catalog.iceberg-rest.prefix` | `` | Pins the catalog prefix (the database). Required for Horizon. | ### Response ```json theme={null} { "items": [ { "name": "SCHEMA_ONE", "type": "NAMESPACE" }, { "name": "nyc_taxi_trips", "type": "TABLE" } ] } ``` | `items[].type` | Meaning | | - | - | | `NAMESPACE` | A schema you can drill into — pass its `name` as `path`. | | `TABLE` | A selectable table. | *** ## Step 2: Preview the schema **`POST /tables/preview`** Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4. > [!NOTE] > > The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`. ### Request **Top-level fields:** | Field | Type | Required | Description | | - | - | - | - | | `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. | | `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. | | `schema` | object | No | An explicit schema to use instead of inferring one. | **`config.inference` — how the schema is derived:** | Field | Type | Default | Description | | - | - | - | - | | `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Iceberg metadata. **Set this to `true` for External Tables.** | | `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. | Resolution order: explicit `schema` → embedded schema → inferred schema. **`config.sampling` — how rows are sampled:** | Field | Type | Default | Description | | - | - | - | - | | `min` | integer | `1` | Minimum records to sample. | | `max` | integer | `100` | Maximum records to sample. | ```bash theme={null} curl -X POST "$BASE_URL/tables/preview" \ -H "Content-Type: application/json" \ -d '{ "tableConfig": { "tableName": "nyc_taxi_trips_OFFLINE", "tableType": "OFFLINE", "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "catalogType": "iceberg-rest", "executor": "controller", "inputFormat": "parquet", "catalog.iceberg-rest.serviceType": "rest", "catalog.iceberg-rest.restUri": "https://-.snowflakecomputing.com/polaris/api/catalog", "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.prefix": "", "catalog.iceberg-rest.table.namespace": "", "catalog.iceberg-rest.table.tableName": "nyc_taxi_trips", "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens", "catalog.iceberg-rest.auth.rest.oauthScopes": "session:role:", "catalog.iceberg-rest.credential": "", "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } } } }, "config": { "inference": { "embeddedSchemaEnabled": true } } }' ``` > [!NOTE] > > **Setting the namespace and table.** Take them from the [browse](#step-1-validate-and-browse-the-connection) response — set `catalog.iceberg-rest.table.namespace` (the Snowflake schema) and `.tableName` to the `NAMESPACE` and `TABLE` you selected. > > **Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4. ### Response | Field | Type | Description | | - | - | - | | `schema` | object | The resolved Pinot schema. **Use this in step 3.** | | `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. | | `rows` | array | Sample records after schema + ingestion transforms. | | `sourceRows` | array | Raw records before transformation. | | `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). | Adjust the schema (time column, column names, null handling) before moving on. *** ## Step 3: Create the schema **`POST /schemas`** Send the `schema` object from the preview response. Rename its `schemaName` to match your table. ```bash theme={null} curl -X POST "$BASE_URL/schemas" \ -H "Content-Type: application/json" \ -d @schema.json ``` ```json theme={null} { "status": "nyc_taxi_trips successfully added" } ``` *** ## Step 4: Create the table **`POST /tables`** Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external. ```bash theme={null} curl -X POST "$BASE_URL/tables" \ -H "Content-Type: application/json" \ -d @tableConfig.json ``` ```json theme={null} { "status": "Table nyc_taxi_trips_OFFLINE successfully added" } ``` The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call. > [!TIP] > > The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run: > > ```bash theme={null} > curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE" > ``` *** ## Quickstart: onboard a table end-to-end The whole flow as one script. Fill in the variables at the top, run the script, and the first sync starts automatically. ```bash theme={null} #!/usr/bin/env bash set -euo pipefail BASE_URL="https://dp..cp..startree.cloud/api/pinot" AUTH="Authorization: Bearer " ACCOUNT_HOST="-.snowflakecomputing.com" REST_URI="https://$ACCOUNT_HOST/polaris/api/catalog" TOKEN_URI="https://$ACCOUNT_HOST/polaris/api/catalog/v1/oauth/tokens" DATABASE="" # used as both warehouse and prefix SCHEMA="" # the Iceberg namespace TABLE="nyc_taxi_trips" ROLE="" SCOPES="session:role:$ROLE" PAT="" STORAGE_REGION="" STORAGE_ROLE_ARN="arn:aws:iam::123456789012:role/pinot-external-table-reader" STORAGE_EXTERNAL_ID="" # 1. Validate & browse root — lists schemas in the Snowflake database. curl -sf -X POST "$BASE_URL/connections/browse" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"connection\": { \"type\": \"CATALOG\", \"params\": { \"catalogType\": \"iceberg-rest\", \"catalog.iceberg-rest.serviceType\": \"rest\", \"catalog.iceberg-rest.restUri\": \"$REST_URI\", \"catalog.iceberg-rest.warehouse\": \"$DATABASE\", \"catalog.iceberg-rest.prefix\": \"$DATABASE\", \"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\", \"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\", \"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\", \"catalog.iceberg-rest.credential\": \"$PAT\" } }, \"path\": \"\" }" | jq . # 2. Preview — infer schema + enriched table config. curl -sf -X POST "$BASE_URL/tables/preview" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"tableConfig\": { \"tableName\": \"${TABLE}_OFFLINE\", \"tableType\": \"OFFLINE\", \"task\": { \"taskTypeConfigsMap\": { \"ExternalTableSyncTask\": { \"catalogType\": \"iceberg-rest\", \"executor\": \"controller\", \"inputFormat\": \"parquet\", \"catalog.iceberg-rest.serviceType\": \"rest\", \"catalog.iceberg-rest.restUri\": \"$REST_URI\", \"catalog.iceberg-rest.warehouse\": \"$DATABASE\", \"catalog.iceberg-rest.prefix\": \"$DATABASE\", \"catalog.iceberg-rest.table.namespace\": \"$SCHEMA\", \"catalog.iceberg-rest.table.tableName\": \"$TABLE\", \"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\", \"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\", \"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\", \"catalog.iceberg-rest.credential\": \"$PAT\", \"catalog.iceberg-rest.auth.storage.roleArn\": \"$STORAGE_ROLE_ARN\", \"catalog.iceberg-rest.auth.storage.externalId\": \"$STORAGE_EXTERNAL_ID\", \"catalog.iceberg-rest.auth.storage.region\": \"$STORAGE_REGION\" } } } }, \"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } } }" > preview.json # 3. Extract schema and table config from preview response. jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json jq '.tableConfigs.offline' preview.json > tableConfig.json # 4. Create the schema. curl -sf -X POST "$BASE_URL/schemas" \ -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json # 5. Create the table — watcher starts the first sync automatically. curl -sf -X POST "$BASE_URL/tables" \ -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json # 6. (optional) Poll until sync completes. curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \ | jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered' ``` Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable). ## Monitor onboarding The [status endpoint](../observability#run-status) reports each sync run's progress, and `?includeLag=true` adds how far the table trails the upstream catalog. Both require `executor=controller` (set automatically by this flow). See [Observability](../observability) for full request and response details. ## Verify it's queryable When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips; ``` The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting). *** ## What's next Now that the table is created and data is loading, these are the highest-impact follow-up steps: 1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes) 2. **Enable caching and pin hot indexes.** Set `enable.prefetch.page.cache=true` on the S3 tier, and list small hot indexes in `preload.index.keys.override`, so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching) 3. **Monitor ongoing syncs.** Use the status endpoint (with `includeLag=true` for ingestion lag) after each scheduled sync. → [Observability](../observability) *** For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting). # Snowflake Horizon Catalog: Onboarding via Data Portal Source: https://docs.startree.ai/corecapabilities/external-table/horizon/onboarding-data-portal Connect Snowflake-managed Iceberg tables (Horizon Catalog) to StarTree as an External Table with the point-and-click Data Portal wizard, authenticating with a Snowflake programmatic access token (PAT). > [!WARNING] > > This feature is available starting in **StarTree release 0.16.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment. This guide walks through connecting a **Snowflake Horizon Catalog** to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface. Snowflake Horizon exposes Snowflake-managed Iceberg tables over the standard **Iceberg REST protocol**, served on the Snowflake account host under `/polaris/api/catalog`, authenticated with a **programmatic access token (PAT)**. Data files must be **Parquet**. > Looking for the API-based approach? See [Snowflake Horizon Catalog: Onboarding via API](./onboarding-api). > > Running **self-managed Apache Polaris** or **Snowflake Open Catalog** instead? Those use principal client-id/client-secret credentials — see [Apache Polaris & Snowflake Open Catalog: Onboarding via Data Portal](../polaris/onboarding-data-portal). *** ## Prerequisites Before starting, ensure you have: * **StarTree 0.16.0 or later** with the External Table feature enabled and **tiered storage configured** for your environment. Contact StarTree support if unsure. * A Snowflake account with **Snowflake-managed Iceberg tables** and a **programmatic access token (PAT)** whose role restriction covers a role with read access to them. * The Snowflake **database** backing the Iceberg tables and the schema/table you want to onboard. * Read access to the underlying S3 object storage (the external volume's bucket) — static keys or an IAM role StarTree can assume. See [Onboarding via API — Authentication](./onboarding-api#authentication) for the permissions required. *** ## Step 1: Open the External Tables 1. Log in to **Data Portal**. 2. In the left navigation, go to **Tables**. 3. Click **+ Connect External Table**. The wizard opens with a connection configuration screen. *** ## Step 2: Configure the Horizon Connection Choose **Snowflake Horizon Catalog** as the catalog type — it appears as a top-level tile alongside AWS Glue, Amazon S3 Tables, and Unity Catalog. Fill in the connection details: **Details** | Field | Description | | - | - | | **Connection Name** | A unique name to identify this connection in Data Portal. | **Metastore** — how StarTree authenticates with Horizon's Iceberg REST API. | Field | Description | | - | - | | **Catalog Endpoint URI** | The Horizon Iceberg REST endpoint: `https://-.snowflakecomputing.com/polaris/api/catalog`. | | **Database** | The Snowflake database backing the Iceberg tables. | | **REST Authentication Type** | **OAuth** — the PAT is exchanged through the OAuth2 token endpoint. | | **OAuth Token URI** | The token endpoint on the account host: `https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens`. | | **Personal Access Token (PAT)** | The Snowflake programmatic access token — sent as the OAuth2 client secret. | | **OAuth Scopes** | `session:role:` — must match the PAT's role restriction. | **Storage** — credentials for reading the underlying Parquet data files from S3. | Field | Description | | - | - | | **Region** | AWS region where the S3 data files are stored. Always required. | | **Authentication Type** | **Basic Authentication** (Access Key + Secret Key), **IAM Node Role** (the cluster node's own IAM identity — Data Portal shows the resolved role as a read-only **IAM Role Name** field so you can grant it access in your AWS account), or **IAM Role** (a cross-account role StarTree assumes). | | **Access Key** / **Secret Key** | With **Basic Authentication**: static AWS credentials for S3 data access. | | **IAM Role ARN** | With **IAM Role**: the ARN of the role to assume. The form shows **Steps to create IAM Policy** — a ready-made trust policy including the StarTree account and External ID to configure on the role. | Click **Validate Connection**. Data Portal calls the catalog's validate endpoint and confirms credentials and connectivity before proceeding. *** ## Step 3: Browse and Select a Table Once the connection is validated: 1. Data Portal lists the available **schemas** in the database — the field is labeled **Schema**. 2. Select a schema to expand its tables. 3. Click the **table** you want to onboard. Data Portal reads the Iceberg schema and derives a Pinot schema automatically. *** ## Step 4: Review the Schema The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity. Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct. *** ## Step 5: Configure the Table Set the final table options: | Setting | Default | Notes | | - | - | - | | **Table Name** | — | Name of the Pinot table to create. | | **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. | | **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. | The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config. Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required. *** ## Step 6: Monitor Onboarding The table detail view shows sync status in real time — files discovered, segments generated, last synced snapshot, and ingestion lag. Use **Sync Now** to trigger an immediate sync, or **Pause Sync** / **Resume Sync** from the table menu (while the sync task is paused, the primary button itself reads **Resume Sync**). For API-based monitoring, see [Observability](../observability). # Supported Indexes Source: https://docs.startree.ai/corecapabilities/external-table/indexes Which Pinot indexes External Tables support — inverted, range, JSON, text, bloom, star-tree, and more — with config examples and why columns are RAW. > [!WARNING] > > This feature is available starting in **StarTree release 0.15.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment. External tables support a broad set of Pinot indexes out of the box. Indexes are key to getting low-latency query performance from external tables. Without them, every query performs a full segment scan across data read from object storage. With the right indexes configured for your query patterns, Pinot can skip irrelevant segments and data blocks entirely, reducing query time from minutes/seconds to seconds/milliseconds on large datasets, while also cutting S3 I/O costs. The most impactful indexes to configure based on your workload: * **JSON index** — required for filtering or extracting nested fields from complex columns (`STRUCT`, `LIST`, `MAP`). * **Text index** — enables full-text search on free-text string columns. * **Range index** — speeds up range predicates on numeric or timestamp columns (e.g., `WHERE ts > X AND ts < Y`). * **Inverted index** — efficient equality filtering on low-cardinality columns (e.g., `WHERE status = 'active'`). *** ## Supported | Index | Status | Notes | | - | - | - | | **Forward index (remote)** | ✅ Supported | Reads raw values directly from Parquet on object storage. | | **Inverted index** | ✅ Supported | Dictionary is retained in the index sidecar; see below. | | **Range index** | ✅ Supported | | | **Timestamp index** | ✅ Supported | | | **JSON index** | ✅ Supported | Required for `JSON_MATCH` and `JSON_EXTRACT_SCALAR` on complex columns (`STRUCT`, `LIST`, `MAP`). | | **Composite JSON index** | ✅ Supported | | | **Text index (Lucene)** | ✅ Supported | Enables full-text search on string columns. | | **Sparse index** | ✅ Supported | | | **Min/Max index** | ✅ Supported | No dictionary required. | | **Star-tree index** | ✅ Supported | Pre-aggregates on low-cardinality dimensions. Rejected at table-config validation on a table with Iceberg deletion vectors enabled (`enableDeletionVectors=true`) — deletion vectors filter rows per query, which precomputed aggregates can't honor. See [Deletion Vectors](./deletion-vectors). | | **Bloom filter** | ✅ Supported | | | **FST index** | ✅ Supported | | | **IFST index** | ✅ Supported | | | **Null value vector** | ✅ Supported | Used internally when `nullHandlingEnabled: true` is set on the table. | | **Vector index (HNSW)** | ✅ Supported | The index config must set `"storeInSegmentFile": true` — without it the HNSW index is not loaded and vector queries silently fall back to exact scan (`VECTOR_SIMILARITY_EXACT_SCAN`). | | **Sorted index** | 🚫 Not supported | Requires data to be sorted at ingestion time; not applicable to external tables. | | **Geospatial / H3 index** | 🚫 Not supported | | > [!NOTE] > > Sorted and H3 are the only index *types* the controller rejects on an external table. Two related validations to be aware of: every non-virtual column must have a `fieldConfig` with `encodingType: RAW` (a column without one would default to dictionary encoding and is rejected), and derived columns / `transformConfigs` are not allowed. > [!NOTE] > > For complex (`STRUCT`/nested-JSON) columns, the `open_struct` settings that choose **which keys get materialized** — `maxDenseKeys`, `denseKeys`, `denseKeyMinFillRate`, `maxNestedKeyDepth` — are rejected at table creation on an external table. Those settings are applied by the dense/sparse key splitter while a segment's column data is written; an external table never writes column data, so the split is whatever the source Parquet file's shredding already chose, and the setting can only be ignored. > > Indexes are **not** rejected, in either of the two places an OPEN\_STRUCT (`VARIANT`) column can carry them: per-key `valueFieldConfigs` over a materialized key, and `sparseFieldConfig` (or its older spelling `sparseJsonIndex`) over the shared sparse blob. Both name indexes over columns that already exist rather than a split, and a reload builds them by reading back through the Parquet reader — the same path a per-key index already took. *** ## Why are columns RAW (no dictionary)? In a standard Pinot table, every column is dictionary-encoded by default. Pinot builds a lookup table that maps each unique value to a compact integer ID and stores those IDs in the forward index. This dictionary is created during ingestion, when data is converted into Pinot's native segment format. External tables never go through that ingestion step — they read Parquet files directly from object storage at query time. Because the data is never materialized into Pinot segments, there is no opportunity to build a dictionary, so every column is encoded as `RAW` (no dictionary). You can explicitly see this in the column config generated by the onboarding flow — every column gets RAW encoding and a RAW forward index, and nothing else: ```json theme={null} { "name": "payment_type", "encodingType": "RAW", "indexes": { "forward": { "encodingType": "RAW" } } } ``` This means the forward index stores the actual values from Parquet, not dictionary IDs. **Indexes that need a dictionary — inverted, FST, IFST, star-tree** — still work because Pinot builds a per-segment dictionary sidecar when it writes the index files alongside the Parquet data. The main forward index stays RAW; only the index sidecar carries the dictionary mapping. You opt in by adding a `"dictionary": {}` block to the column's `indexes` config (see examples below). **Sorted index is ruled out entirely** because it requires data to be physically sorted during the ingestion write path. External tables have no write path — they read whatever order the Parquet files are in — so a sorted index cannot be built or maintained. *** ## Adding an index ### Choosing the right index | Query pattern | Use | Cardinality guidance | | - | - | - | | Equality / IN filter | Inverted | Best for low-to-medium cardinality | | Range filter (`BETWEEN`), including time | Range | Any cardinality; ideal for numeric/timestamp | | "Does this value exist?" segment pruning | Bloom filter | Best for high-cardinality equality filters | | Filtering/extracting nested JSON fields | JSON | Works on raw JSON regardless of cardinality | | Full-text / token search | Text | For free-text columns; use `TEXT_MATCH` | | Prefix / fuzzy string match | FST | Dictionary-backed; low-to-medium cardinality | | Repeated aggregations / group-bys | Star-tree | Pre-aggregates on low-cardinality dimensions | | Time-bucketed queries | Timestamp | On the time column | > [!TIP] > > If you're unsure, start with a range index on time and numeric filters and an inverted index on the low-cardinality string columns you filter on most. Add others only when a query needs them. ### Inverted index (low-cardinality equality) Dictionary-backed, so keep the forward index RAW and add a dictionary block: ```json theme={null} { "name": "payment_type", "encodingType": "RAW", "indexes": { "forward": { "encodingType": "RAW" }, "dictionary": {}, "inverted": { "disabled": false } } } ``` Example query: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips WHERE payment_type = 'CASH'; ``` ### Range index (numeric / time ranges) No dictionary needed: ```json theme={null} { "name": "fare_amount", "encodingType": "RAW", "indexes": { "range": {} } } ``` Example query: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips WHERE fare_amount BETWEEN 10 AND 50; ``` ### Bloom filter (high-cardinality equality) ```json theme={null} { "name": "trip_id", "encodingType": "RAW", "indexes": { "bloom": {} } } ``` Example query: ```sql theme={null} SELECT * FROM nyc_taxi_trips WHERE trip_id = 't_8f3a91'; ``` ### JSON index ```json theme={null} { "name": "surcharges", "encodingType": "RAW", "indexes": { "json": {} } } ``` Example query: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips WHERE JSON_MATCH(surcharges, '"$.type" = ''airport'''); ``` ### Text index (full-text search) ```json theme={null} { "name": "notes", "encodingType": "RAW", "indexes": { "text": {} } } ``` Example query: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips WHERE TEXT_MATCH(notes, 'airport AND delay'); ``` > [!NOTE] > > The onboarding/preview flow generates the base `fieldConfigList` with RAW encoding for every column. To add an index, merge the `indexes` block above into the matching column entry — don't change `encodingType`. # Observability Source: https://docs.startree.ai/corecapabilities/external-table/observability REST endpoints to check an External Table sync run's status and ingestion lag against the upstream table, or trigger a run on demand. Once an External Table is created, the controller's External Table watcher ingests data on a schedule. These read-only endpoints let you check each run's status and see how far the queryable data lags behind the upstream table. There is also an endpoint to trigger a run immediately instead of waiting for the next scheduled tick. > [!NOTE] > > The observability endpoints require `executor=controller` in the table's `ExternalTableSyncTask` config (the default for tables onboarded via the Data Portal or the preview API). > > All paths are relative to your data-plane base URL — set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` (the StarTree Cloud data-plane proxy). Add an `Authorization` header if your controller requires auth. > [!NOTE] > > **Run status is not the same as data freshness.** A `fileOnboardingRun.status` of `COMPLETED` only means the last sync run finished — the table can still be behind the source, because the upstream table may have committed more data since. To judge whether your data is current, use [`syncState`](#run-status) (with `lag.percentDataIngested` and `lag.snapshotsBehind` for detail), not the run status. *** ## Common tasks A map from what you want to know to the call that answers it and the field to read. Each row links to the full reference below. | I want to… | Call | Look at | | - | - | - | | Check the last sync succeeded | `GET …/externalTable/status` | [`fileOnboardingRun.status`](#run-status) | | See what an in-progress run is doing right now | `GET …/externalTable/status` | [`fileOnboardingRun.currentPhase`](#run-status) + `currentPhaseStartTime` | | Diagnose a stuck or failed sync | `GET …/externalTable/status` | [`fileOnboardingRun.failurePhase`](#run-status) + `errorMessage`, then [Troubleshooting](./troubleshooting#sync-not-progressing) | | Check my table is caught up with the source | `GET …/externalTable/status?includeLag=true` | [`syncState`](#run-status) / `lag.percentDataIngested` | | Know how far behind my data is, as a duration | `GET …/externalTable/status?includeLag=true` | `lag.upstream.committedAtMs` − `lag.synced.committedAtMs` | | Track progress on a snapshot too large for one run | `GET …/externalTable/status` | [`snapshotIngestion`](#snapshot-ingestion-cross-run-progress) | | See when the next sync is scheduled | `GET …/externalTable/status` | `nextRunTime` | | Check whether the current snapshot is fully ingested | `GET …/externalTable/checkpoint` | [`state`](#ingestion-watermark) | | Trigger a sync now instead of waiting | `POST /tasks/schedule` | [Trigger a sync run](#trigger-a-sync-run-optional) | > [!TIP] > > Prefer clicking to curl? A table's details page in the Data Portal shows the same run status and sync controls — see [Trigger a sync run](#trigger-a-sync-run-optional). *** ## API Endpoints Quick Reference | Method | Endpoint | Purpose | | - | - | - | | `GET` | `/tables/{tableNameWithType}/externalTable/status` | Status of the last (or current) sync run. Add `includeLag=true` for [ingestion lag](#ingestion-lag). | | `GET` | `/tables/{tableNameWithType}/externalTable/checkpoint` | The [ingestion watermark](#ingestion-watermark): which snapshot the table is on and whether it is fully ingested. | | `POST` | `/tasks/schedule` | Optionally trigger a sync run now instead of waiting for the schedule. | | `POST` | `/tables/{tableNameWithType}/iceberg/isSnapshotReady` | Check whether a specific Iceberg snapshot's [deletion vectors](./deletion-vectors) have converged on all servers. | *** ## Run status `GET /tables/{tableNameWithType}/externalTable/status` *Use this to answer "did the last sync run work?" and — with `includeLag=true` — "is my data current?"* Returns the External Table sync watcher's last run status for the table, under a `fileOnboardingRun` object. With `includeLag=true`, the response additionally reports how far the table's queryable data is behind the upstream Iceberg table — see [Ingestion lag](#ingestion-lag). | Query parameter | Default | Description | | - | - | - | | `includeLag` | `false` | Also return ingestion lag as a nested `lag` object (available starting in **StarTree release 0.16.0**), and compute [`syncState`](#run-status). Adds one catalog read; the default call stays cheap (a single ZooKeeper read, no catalog call). | | `pendingExact` | **`true`** | Only with `includeLag=true`: compute pending files and bytes from an exact manifest diff instead of snapshot summary totals. Exact is now the default because summary arithmetic cannot answer the pending question at all once the source has non-append commits (compaction, overwrite, delete) — it yields no estimate rather than a wrong one. Pass `pendingExact=false` to opt out to cheaper summary arithmetic. See [Exact pending mode](#exact-pending-mode). | **Top-level fields:** | Field | Type | Description | | - | - | - | | `tableNameWithType` | string | Table name with the `_OFFLINE` suffix. | | `fileOnboardingRun` | object | The run happening right now — see below. Reports `IDLE` between runs; it never restates a finished run once history has recorded it. | | `snapshotIngestion` | object \| null | Cross-run view of the snapshot currently being ingested — see [Snapshot ingestion](#snapshot-ingestion-cross-run-progress). `null` when the table has neither a watermark nor a finished run. | | `syncState` | string | `CAUGHT_UP` \| `CATCHING_UP` \| `STALLED` \| `UNKNOWN`. One-value ingestion health verdict. Requires `includeLag=true` — without it, always `UNKNOWN` (never assumed healthy). | | `nextRunTime` / `nextRunTimeMs` | string \| null / long \| null | Next scheduled sync, read from the table's live cron trigger. `null` when the table has no cron schedule or cron scheduling is off on this controller. | | `lag` | object \| null | Only with `includeLag=true` (the key is absent otherwise): the [ingestion lag object](#ingestion-lag), or `null` when lag does not apply to the table or could not be computed. | | `lagMessage` | string \| null | Only with `includeLag=true` (the key is absent otherwise): explains why `lag` is `null`; `null` when lag was computed. | **`fileOnboardingRun` fields:** | Field | Type | Description | | - | - | - | | `status` | string | `IDLE` \| `RUNNING` \| `COMPLETED` \| `FAILED`. | | `startTime` | string \| null | Run start, ISO-8601 UTC (e.g. `2026-02-09T18:53:20Z`); `null` if never started. | | `startTimeMs` | long | Run start as raw epoch milliseconds; `0` if never started. | | `endTime` | string \| null | Run end, ISO-8601 UTC; `null` while `RUNNING` or if never finished. | | `endTimeMs` | long | Run end as raw epoch milliseconds; `0` while `RUNNING` or if never finished. | | `filesDiscovered` | int | Source files found in this run. | | `segmentsUploaded` | int | Segments successfully created and uploaded. Stays `0` until the upload phase — a run generates and compresses every segment for its page before uploading them as one batch, so `currentPhase`, not this counter, is the progress signal for an in-progress run. | | `filesFailed` | int \| null | Files this run could not turn into segments and skipped (only possible under `continueOnFileError`). `null` means the run reported no outcome for it yet, which is not the same as `0`. A `COMPLETED` run with a non-zero count is a partial success. | | `currentPhase` | string \| null | Phase the in-flight run is currently executing — same vocabulary as `failurePhase` below. `null` once the run reaches a terminal state. | | `currentPhaseStartTime` / `currentPhaseStartTimeMs` | string \| null / long \| null | When the run entered `currentPhase`; time-in-phase is `now − currentPhaseStartTimeMs`. `null` whenever `currentPhase` is `null` — this is what surfaces a run parked for an hour in `WAITING_FOR_SEGMENTS_ONLINE` instead of just reading `RUNNING`. | | `errorMessage` | string \| null | Populated only when `status=FAILED`. | | `failurePhase` | string \| null | When failed, where it failed: `LISTING_FILES` (credentials/path), `GENERATING_SEGMENTS` (type mismatch), `COMPRESSING_SEGMENTS` (server resources), `UPLOADING_SEGMENTS` (deep-store permissions), `PUBLISHING_SNAPSHOT`, `WAITING_FOR_SEGMENTS_ONLINE`, `WAITING_FOR_SERVERS_READY`, `SAVING_PROGRESS` (ZooKeeper), or `UPDATING_SCHEMA`. `null` on success. See [Troubleshooting](./troubleshooting#sync-not-progressing). | | `checkpointValue` | string \| null | Snapshot ID / watermark after a successful run. | | `requestId` | string \| null | Identifier of the run, for log correlation. | > [!WARNING] > > **`failurePhase` / `currentPhase` values were renamed** to describe what the sync is doing rather than Pinot's internal step names: `FILE_LISTING`→`LISTING_FILES`, `SCHEMA_UPDATE`→`UPDATING_SCHEMA`, `SEGMENT_GENERATION`→`GENERATING_SEGMENTS`, `SEGMENT_COMPRESSION`→`COMPRESSING_SEGMENTS`, `SEGMENT_UPLOAD`→`UPLOADING_SEGMENTS`, `SNAPSHOT_FINALIZE`→`PUBLISHING_SNAPSHOT`, `IS_EV_CONVERGENCE`→`WAITING_FOR_SEGMENTS_ONLINE`, `SNAPSHOT_READINESS_POLL`→`WAITING_FOR_SERVERS_READY`, `CHECKPOINT_SAVE`→`SAVING_PROGRESS`. No alias is kept — update any dashboard, alert, or script matching on the old strings. | `status` | Meaning | | - | - | | `IDLE` | No run yet, or sitting between scheduled runs. | | `RUNNING` | A run is in progress — check `currentPhase` for what it's doing right now. | | `COMPLETED` | Last run succeeded — check `checkpointValue`. | | `FAILED` | Last run failed — check `errorMessage` and `failurePhase`. See [Troubleshooting → Sync not progressing](./troubleshooting#sync-not-progressing) for a diagnosis guide per phase. | ```bash theme={null} curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/status" ``` ```json theme={null} { "tableNameWithType": "nyc_taxi_trips_OFFLINE", "fileOnboardingRun": { "status": "COMPLETED", "startTime": "2026-02-09T18:53:20Z", "startTimeMs": 1770663200000, "endTime": "2026-02-09T18:54:20Z", "endTimeMs": 1770663260000, "filesDiscovered": 15, "segmentsUploaded": 15, "filesFailed": null, "currentPhase": null, "currentPhaseStartTime": null, "currentPhaseStartTimeMs": null, "errorMessage": null, "failurePhase": null, "checkpointValue": "1234567890123456789", "requestId": "a1b2c3d4" }, "snapshotIngestion": { "snapshotId": "1234567890123456789", "state": "DONE", "runs": 1, "totalFiles": 15, "filesDiscovered": 15, "segmentsUploaded": 15, "filesFailed": null, "startedAt": "2026-02-09T18:53:20Z", "startedAtMs": 1770663200000, "updatedAt": "2026-02-09T18:54:20Z", "updatedAtMs": 1770663260000, "consecutiveFailedRuns": 0, "lastRunStatus": "COMPLETED", "lastRunAt": "2026-02-09T18:54:20Z", "lastRunAtMs": 1770663260000, "lastRunError": null }, "nextRunTime": "2026-02-09T19:53:20Z", "nextRunTimeMs": 1770666800000, "syncState": "UNKNOWN" } ``` **Status codes:** | Code | When | | - | - | | `200` + `lag: null` + `lagMessage` | OK, but lag does not apply — the table is not `catalogType=iceberg-rest` with `executor=controller`, or has no sync task config. | | `200` + `lag` with some `null` fields + `lag.message` | OK — a transient catalog/ZooKeeper issue degraded individual lag fields. Lag never returns `500`. | | `404` | Table not found. | | `500` | Internal error (does not apply to lag computation, which degrades to `null` fields instead). | ### Ingestion lag `GET /tables/{tableNameWithType}/externalTable/status?includeLag=true` *Use this when you need to know whether the table is current with the source — and, if not, by how much.* > [!NOTE] > > Ingestion lag reporting is available starting in **StarTree release 0.16.0**. Reports how far the table's queryable data is behind the upstream Iceberg table. Lag is expressed as a snapshot delta rather than a time lag: the last fully synced snapshot versus the upstream head snapshot, plus the files and bytes still pending. Lag applies to `catalogType=iceberg-rest` tables synced with `executor=controller`. For any other table the endpoint still returns `200` with `lag: null` and a `lagMessage` explaining why; the status response is never failed because of lag. Snapshot lag is undefined for raw S3 listings, for example. ```bash theme={null} curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/status?includeLag=true" ``` ```json theme={null} { "tableNameWithType": "nyc_taxi_trips_OFFLINE", "fileOnboardingRun": { "status": "COMPLETED", "startTime": "2026-02-09T18:53:20Z", "startTimeMs": 1770663200000, "endTime": "2026-02-09T18:54:20Z", "endTimeMs": 1770663260000, "filesDiscovered": 15, "segmentsUploaded": 15, "filesFailed": null, "currentPhase": null, "currentPhaseStartTime": null, "currentPhaseStartTimeMs": null, "errorMessage": null, "failurePhase": null, "checkpointValue": "1234567890123456789", "requestId": "a1b2c3d4" }, "snapshotIngestion": { "snapshotId": "1234567890123456789", "state": "DONE", "runs": 1, "totalFiles": 120, "filesDiscovered": 120, "segmentsUploaded": 120, "filesFailed": null, "startedAt": "2026-02-09T18:53:20Z", "startedAtMs": 1770663200000, "updatedAt": "2026-02-09T18:54:20Z", "updatedAtMs": 1770663260000, "consecutiveFailedRuns": 0, "lastRunStatus": "COMPLETED", "lastRunAt": "2026-02-09T18:54:20Z", "lastRunAtMs": 1770663260000, "lastRunError": null }, "nextRunTime": "2026-02-09T19:53:20Z", "nextRunTimeMs": 1770666800000, "lag": { "tableNameWithType": "nyc_taxi_trips_OFFLINE", "synced": { "snapshotId": "1234567890123456789", "sequenceNumber": 41, "committedAtMs": 1770663000000, "committedAt": "2026-02-09T18:50:00Z", "totalFiles": 120, "totalSize": "5.00 GB", "totalBytes": 5368709120, "totalRecords": 48000000 }, "upstream": { "snapshotId": "2345678901234567890", "sequenceNumber": 44, "committedAtMs": 1770663480000, "committedAt": "2026-02-09T18:58:00Z", "totalFiles": 192, "totalSize": "8.00 GB", "totalBytes": 8589934592, "totalRecords": 76800000 }, "snapshotsBehind": 3, "filesPending": 72, "bytesPending": 3221225472, "pendingExact": true, "pendingState": "EXACT", "unavailableReason": null, "percentDataIngested": 62.5, "syncedAt": "2026-02-09T18:54:00Z", "syncedAtMs": 1770663240000, "controllerTimeMs": 1770663300000, "controllerTime": "2026-02-09T18:55:00Z", "message": null }, "lagMessage": null, "syncState": "CATCHING_UP" } ``` > [!WARNING] > > **`lag.caughtUp` was removed.** It was fully derivable from the top-level `syncState`: `true` always meant `CAUGHT_UP`, `false` never did, `null` meant `UNKNOWN`. Read `syncState` instead — it collapses ingestion health to one of `CAUGHT_UP` / `CATCHING_UP` / `STALLED` / `UNKNOWN` so a client never has to infer health from a nullable boolean and a percent. **`lag` object fields:** | Field | Type | Description | | - | - | - | | `tableNameWithType` | string | Echoes the table name with the `_OFFLINE` suffix. | | `synced` | object \| null | The last fully synced snapshot: the newest snapshot whose data is actually queryable. It is recorded only after the ingested segments have converged on all servers. `null` when the table has never synced or when the synced state could not be read; `message` distinguishes the two cases. | | `upstream` | object \| null | The upstream Iceberg head snapshot, read from the catalog. `null` when the catalog is unreachable or the upstream table has no snapshots. | | `snapshotsBehind` | int \| null | How many upstream snapshots sit between the synced snapshot and the head. Best-effort: exact for the linear histories ingestion produces, but branches, tags, WAP snapshots, or retention can skew it. `null` on Iceberg v1 metadata. | | `filesPending` | long \| null | Data files not yet queryable. An exact manifest diff by default (`pendingState=EXACT`), or a net delta of snapshot summary totals with `pendingExact=false` (`pendingState=ESTIMATED`). `null` when it cannot be computed honestly (`pendingState=UNAVAILABLE`; see `unavailableReason`). | | `bytesPending` | long \| null | Bytes not yet queryable, as a raw number, computed the same way as `filesPending`. | | `pendingExact` | boolean | Whether the pending numbers actually came from the exact manifest diff — equivalent to `pendingState == EXACT` (including a caught-up table, where pending is `0` by definition). This is **not** an echo of the `?pendingExact` request flag: a diff that was requested but could not be delivered (cap or concurrency limit hit) reports `false` here. | | `pendingState` | string | `EXACT` \| `ESTIMATED` \| `UNAVAILABLE` — how `filesPending` / `bytesPending` / `percentDataIngested` were arrived at. | | `unavailableReason` | string \| null | Set only when `pendingState=UNAVAILABLE`: `SYNCED_SNAPSHOT_EXPIRED`, `NON_APPEND_HISTORY`, `DIFF_CAP_EXCEEDED`, `CONCURRENCY_LIMIT`, `EXACT_DIFF_FAILED`, `CATALOG_UNREACHABLE`, `UPSTREAM_HAS_NO_SNAPSHOTS`, `SYNCED_STATE_UNREADABLE`, or `SYNCED_AHEAD_OF_HEAD`. `message` carries the human-readable detail. | | `percentDataIngested` | double \| null | Percent of upstream bytes already queryable. `100` when caught up, `0` when never synced, `null` when it cannot be computed. | | `syncedAt` / `syncedAtMs` | string \| null / long \| null | When *this table* committed the synced snapshot (from the internal `ActiveSnapshots` record) — not the upstream commit time. Unlike `synced.committedAt`, this survives the synced snapshot expiring out of source retention, so it stays populated even when `synced.totalFiles` etc. go `null`. | | `controllerTimeMs` | long | Controller clock at computation time, raw epoch milliseconds. | | `controllerTime` | string | Controller clock at computation time, ISO-8601 UTC (display companion of `controllerTimeMs`). | | `message` | string \| null | Explains any degraded or missing fields, for example summary totals being unavailable or the exact-diff cap being hit. `null` when everything was computed cleanly. | **Snapshot fields (`synced` and `upstream`):** | Field | Type | Description | | - | - | - | | `snapshotId` | string \| null | Iceberg snapshot ID, serialized as a JSON **string** — Iceberg ids exceed 2^53, so a JSON number silently loses precision in a JavaScript client. | | `sequenceNumber` | long \| null | Iceberg sequence number. | | `committedAtMs` | long \| null | The upstream Iceberg commit time as raw epoch milliseconds. This is always catalog commit time, never Pinot ingest time. | | `committedAt` | string \| null | The commit time as ISO-8601 UTC, truncated to seconds (display companion of `committedAtMs`). | | `totalFiles` | long \| null | Total data files at this snapshot, from the snapshot summary. `null` when the snapshot has expired from table metadata or the summary keys are absent. | | `totalSize` | string \| null | Total data size at this snapshot, human-readable (e.g. `"5.00 GB"`), from the snapshot summary. Same nullability as `totalFiles`. | | `totalBytes` | long \| null | Raw byte count backing `totalSize` (display companion pattern, like `committedAtMs` for `committedAt`). Same nullability as `totalFiles`. Serialized as a JSON number rather than a string, unlike `snapshotId` — a size can't exceed `2^53` bytes the way a snapshot id's identity-sensitive precision can. | | `totalRecords` | long \| null | Total records at this snapshot, from the snapshot summary. Same nullability as `totalFiles`. | > [!NOTE] > > There is deliberately no `lagMs` field. To express lag as a duration, subtract `lag.synced.committedAtMs` from `lag.upstream.committedAtMs` (raw epoch milliseconds; the ISO-8601 `committedAt` strings are seconds-precision display companions). Both timestamps come from the same upstream catalog clock, so the difference is not skewed by clock drift between the catalog and the Pinot controller. Transient catalog or ZooKeeper failures never turn into a `500`. The affected lag fields are returned as `null` with an explanatory `message`, and a synced state that exists but cannot be read is reported as unavailable rather than as a confident "nothing ingested". #### Exact pending mode `pendingExact` now **defaults to `true`**: the controller diffs the live data-file sets of the head and synced snapshots, giving exact `filesPending` / `bytesPending`. This replaced defaulting to the cheap summary-totals estimate, because netting cumulative summary totals cannot answer the pending question at all once the source has non-append commits (compaction, overwrite, delete) between the synced and head snapshots — it silently reports nothing (`pendingState=UNAVAILABLE`) rather than a wrong number. The exact diff enumerates files and is capped at `iceberg.snapshotProcessing.lagMaxDiffFiles` files per snapshot enumeration (default 100,000) — a per-table key in the `ExternalTableSyncTask` config. It is also bounded by a controller-wide cap on concurrent exact diffs that StarTree manages; either cap degrades the response to the summary estimate and says so in `lag.message`, with `pendingState=UNAVAILABLE` (or `ESTIMATED` if the fallback succeeds) and `pendingExact: false`. Pass `?pendingExact=false` to opt out to the O(1) summary-totals estimate up front. ### Integration notes * **Pending fields can be `null` even when `syncState` is not `CAUGHT_UP`.** A compaction, overwrite, or delete between snapshots can leave both the exact diff and the summary estimate unable to answer (`pendingState=UNAVAILABLE`), so `filesPending`, `bytesPending`, and `percentDataIngested` may come back `null`. Fall back to `snapshotsBehind`; `lag.message` and `lag.unavailableReason` explain why. * **Two different message fields.** Top-level `lagMessage` means `lag` itself is `null` (lag does not apply to the table); without `includeLag=true` neither key appears at all. `lag.message` means `lag` is present but an individual field is degraded. * **Display strings vs numbers.** `startTime`, `endTime`, `committedAt`, `controllerTime`, `syncedAt`, and `totalSize` are display strings. Every timestamp string has a raw epoch-milliseconds companion (`startTimeMs`, `endTimeMs`, `committedAtMs`, `controllerTimeMs`, `syncedAtMs`), and `totalSize` has a raw companion too (`totalBytes`) — use those for arithmetic. `filesPending`, `bytesPending`, `percentDataIngested`, `totalFiles`, `totalRecords`, and `snapshotsBehind` remain numbers. *** ## Snapshot ingestion (cross-run progress) *Use this when a single snapshot is too large to ingest in one run, and you need to know how far the ingestion as a whole has got — not just the last batch.* `fileOnboardingRun` is overwritten with `RUNNING` the moment the next run starts and reports `IDLE` between runs, and each run's own counts (`filesDiscovered`, `segmentsUploaded`) cover only that run's page of at most `iceberg.snapshotProcessing.batchSize` files. Neither one answers "how far through this snapshot are we" for a snapshot ingested over many runs. The `snapshotIngestion` object in the [Run status](#run-status) response does: it accumulates `runs`, `filesDiscovered`, `segmentsUploaded`, and `filesFailed` across every run that has processed the snapshot the table is currently on, alongside the snapshot's own `totalFiles` for context. It also carries the outcome of the **last run that actually finished** — `lastRunStatus`, `lastRunAt`, `lastRunError` — plus `consecutiveFailedRuns`, since a fresh `RUNNING` record can otherwise hide a failure for the length of one cron tick. The failure streak is counted per run, not per snapshot, so a run that re-targets a newer upstream snapshot does not reset it. | Field | Type | Description | | - | - | - | | `snapshotId` | string \| null | The snapshot these totals belong to — compare against the ingestion watermark's `checkpointValue` before trusting the counts as current for a snapshot in progress. | | `state` | string \| null | The ingestion watermark's [state](#ingestion-watermark) (`IN_PROGRESS` \| `DONE` \| `FAILED`) for this snapshot. | | `runs` | int \| null | Runs that have processed this snapshot, successful or not. Well above `ceil(files / batchSize)` means runs are failing and retrying. `null` until the snapshot's first run finishes. | | `totalFiles` | long \| null | Data files in the snapshot from its own metadata summary — its size, not a work estimate (use `lag.filesPending` for what remains). `null` for catalogs without snapshot summaries (`catalogType=s3`) or when the summary could not be read. | | `filesDiscovered` | int \| null | Files picked up for ingestion across those runs (attempts, not distinct files — a retried run's files are counted again). | | `segmentsUploaded` | int \| null | Segments uploaded across those runs, counting only work that stuck. Progress on the snapshot is `segmentsUploaded / lag.filesPending`, since `filesPending` holds still at the snapshot's full new-file count until the snapshot publishes. | | `filesFailed` | int \| null | Files skipped across those runs (only under `continueOnFileError`). `null` means no run has reported a skip outcome yet, not zero. | | `startedAt` / `startedAtMs` | string \| null / long \| null | When the first run for this snapshot started. | | `updatedAt` / `updatedAtMs` | string \| null / long \| null | When the most recent run for this snapshot finished. | | `consecutiveFailedRuns` | int | Consecutive terminal runs that failed, table-scoped. Reset to `0` by a success; not reset by a run re-targeting a newer snapshot. | | `lastRunStatus` | string \| null | `COMPLETED` \| `FAILED` — outcome of the last run to reach a terminal state. `null` until one has. | | `lastRunAt` / `lastRunAtMs` | string \| null / long \| null | When that last completed run finished. | | `lastRunError` | string \| null | Populated only when `lastRunStatus=FAILED`. | `snapshotIngestion` is `null` when the table has neither an ingestion watermark nor a finished run yet (a brand-new table before its first sync). *** ## Ingestion watermark `GET /tables/{tableNameWithType}/externalTable/checkpoint` *Use this to answer "is the snapshot fully ingested, or only partly?"* A large snapshot is ingested over many runs, one page of files at a time. A run that reports `COMPLETED` therefore says nothing about whether the snapshot as a whole is finished — it means that page succeeded. The watermark is what tracks the snapshot itself: which one the table is on, and how far through it ingestion has got. Requires `executor=controller`; legacy minion-executor tables are rejected with `400`, because they never write a watermark and would otherwise be indistinguishable from a table that has not run yet. All value fields are `null` when the table has never synced. The Data Portal does not surface this endpoint — call it directly. | Field | Type | Description | | - | - | - | | `tableNameWithType` | string | Table name with the `_OFFLINE` suffix. | | `checkpointType` | string \| null | Provider type: `iceberg-rest` for every Iceberg catalog, `s3` for bucket-backed tables (including GCS via the S3-interop path). | | `checkpointValue` | string \| null | The Iceberg snapshot ID being ingested, or the S3 sentinel `CURRENT`. | | `state` | string \| null | Ingestion state for that snapshot — see below. | | `updatedAtMs` | long \| null | When the watermark was last written. On `FAILED`, this is the "stalled since" time. | | `consecutiveFailures` | int \| null | Failed runs in a row on this snapshot. Resets to `0` on a successful page; the snapshot goes `FAILED` when it reaches `iceberg.snapshotProcessing.maxRetries`. | | `errorMessage` | string \| null | Error from the most recent failed run on this snapshot, `null` after a success. | | `pageCursor` | object \| null | How far the file enumeration got: `{providerType, manifestIndex, entryPosition}` for Iceberg, `{providerType, lastKey}` for S3. `null` at a fresh start, and on an Iceberg snapshot that is `DONE`. | | `state` | Meaning | | - | - | | `IN_PROGRESS` | The snapshot is partly ingested — more pages remain. Trigger further runs, or wait for the schedule. | | `DONE` | The snapshot was ingested completely. The next run picks up the next upstream snapshot, if there is one. | | `FAILED` | Ingestion stopped after `iceberg.snapshotProcessing.maxRetries` consecutive failures. `errorMessage` on this response says why; `failurePhase` on [Run status](#run-status) says where. The snapshot stays `FAILED` until a newer upstream snapshot arrives, unless `iceberg.snapshotProcessing.retryFailedSnapshot=true`. See [Troubleshooting](./troubleshooting#sync-not-progressing). | | `TIMEDOUT` | Reserved; not currently emitted. | ```bash theme={null} curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/checkpoint" ``` ```json theme={null} { "tableNameWithType": "nyc_taxi_trips_OFFLINE", "checkpointType": "iceberg-rest", "checkpointValue": "1234567890123456789", "state": "IN_PROGRESS", "updatedAtMs": 1770663260000, "consecutiveFailures": 0, "errorMessage": null, "pageCursor": { "providerType": "iceberg-rest", "manifestIndex": 3, "entryPosition": 1200 } } ``` > [!TIP] > > `state: DONE` is the signal that a table is at a snapshot boundary — the point at which it is safe to change how the table ingests, for example when [enabling segment groups](./segment-groups#enabling-grouping-on-a-live-table). *** ## Metrics and alerting The status and lag endpoints answer on-demand questions. For dashboards and continuous monitoring, the controller also emits per-sync metrics (available wherever you scrape StarTree Pinot controller metrics). **Per-phase sync timers** (milliseconds) — how long each stage of a run takes; useful for latency dashboards and for finding where a slow sync spends its time: | Metric | Stage | | - | - | | `EXTERNAL_TABLE_SYNC_FILE_LISTING_TIME_MS` | Listing source files from the catalog | | `EXTERNAL_TABLE_SYNC_SEGMENT_GENERATION_TIME_MS` | Converting Parquet to segments | | `EXTERNAL_TABLE_SYNC_SEGMENT_COMPRESSION_TIME_MS` | Compressing segments | | `EXTERNAL_TABLE_SYNC_SEGMENT_UPLOAD_TIME_MS` | Uploading segments to deep store | | `EXTERNAL_TABLE_SYNC_SNAPSHOT_FINALIZE_TIME_MS` | Finalizing the snapshot | | `EXTERNAL_TABLE_SYNC_CHECKPOINT_SAVE_TIME_MS` | Persisting the checkpoint / watermark | | `EXTERNAL_TABLE_SYNC_OVERALL_TIME_MS` | End-to-end run duration | **Deletion-vector convergence** — only when [deletion vectors](./deletion-vectors) are enabled; covers the server-readiness handshake: | Metric | Meaning | | - | - | | `EXTERNAL_TABLE_SYNC_READINESS_POLL_SUCCESS` / `_TIMEOUT` / `_FAST_FAIL` / `_SKIPPED` / `_ATTEMPTS` | Outcomes of polling servers for snapshot readiness | | `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIMEOUT` | A snapshot's ideal-state / external-view convergence timed out | | `EXTERNAL_TABLE_SYNC_SNAPSHOT_READINESS_POLL_TIME_MS` / `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIME_MS` | Time spent polling for readiness and waiting for convergence | **Sync health** — a per-table latch driven by run outcomes, so a dashboard or alert doesn't have to poll the status API: | Metric | Meaning | | - | - | | `EXTERNAL_TABLE_SYNC_UNHEALTHY` (gauge) | `1` when the table's health latch is set (set on a `FAILED` run, carried across further failures, cleared only when a run completes); `0` otherwise. Emitted every tick, including no-work ticks. | | `EXTERNAL_TABLE_SYNC_CONSECUTIVE_FAILURES` (gauge) | Consecutive failed sync runs for the table. | | `EXTERNAL_TABLE_SYNC_RUN_FAILED_STALL` / `_ERROR_REPLICA` / `_EXCEPTION` (meters) | Reason a sync run stopped: a wait made no progress for its stall window, a (segment, instance) pair reached Helix `ERROR`, or any other exception. | | `EXTERNAL_TABLE_SYNC_AUTO_RESET_PAIRS` (meter) | Table-wide `ERROR` (segment, instance) pairs swept by the one IS/EV convergence auto-reset a run may issue. See [`iceberg.snapshotProcessing.convergence.autoReset.maxErrorPairs`](./deletion-vectors#controller-is-ev-convergence). | | `EXTERNAL_TABLE_SYNC_TASK_GENERATION_FAILURE` (meter, per `
.`) | A sync pass failed for the table, counted once per failed pass regardless of cause — the total to alert on. The `_RUN_FAILED_*` meters above break the controller-executor subset down by reason (summing to at most this total); a failure raised before the run starts (task-config validation, catalog provider construction, an active-task conflict) lands only here. Exported as `pinot_controller_externalTableSyncTaskGenerationFailure_Count`. Previously these failures incremented the shared `FILE_INGESTION_FAILURE` meter, so they paged as `FileIngestionTask` failures — this is now a dedicated series. | > [!WARNING] > > There is **no metric for ingestion lag** today — monitor that through the status API: > > * **Data falling behind** — poll `…?includeLag=true` and alert when `syncState` is `STALLED` (or `CATCHING_UP` for longer than one sync interval), or `lag.percentDataIngested` stays below your threshold. Note `syncState` can flicker `STALLED` → `CATCHING_UP` on a fast-committing source: a permanently broken table's watermark cycles `FAILED` → `IN_PROGRESS` → `FAILED` as it re-targets each new upstream snapshot, so alert on the sustained condition, or on `snapshotIngestion.consecutiveFailedRuns` instead, which does not reset on a re-target. > * **Snapshot silently skipped** — a run that reports `COMPLETED` with `segmentsUploaded: 0` while `syncState` is not `CAUGHT_UP`. See [Sync says COMPLETED but no data lands](./troubleshooting#sync-says-completed-but-no-data-lands). > > Per-run success/failure now has a metric too — `EXTERNAL_TABLE_SYNC_UNHEALTHY` above — but for triage detail (`failurePhase`, `errorMessage`) still poll `…/externalTable/status`. *** ## Trigger a sync run (optional) `POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=` > [!NOTE] > > You normally do **not** need this. After a table is created, the controller's watcher runs the first sync automatically and then re-syncs on the table's schedule. Use this endpoint only to start a run immediately instead of waiting for the next tick. | Query parameter | Required | Description | | - | - | - | | `taskType` | Yes | Always `ExternalTableSyncTask`. | | `tableName` | Yes | Table name **with the `_OFFLINE` suffix**, e.g. `nyc_taxi_trips_OFFLINE`. | ```bash theme={null} curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE" ``` ```json theme={null} { "ExternalTableSyncTask": "Task_ExternalTableSyncTask_3f2a1c9e-4b7d-4a1f-9c3e-2d8f5a6b7c9d_1707500000000" } ``` > [!TIP] > > A table's details page in the Data Portal also exposes **Sync Now**, **Pause Sync**, and **Resume Sync** for the sync task, so you don't need to call these endpoints directly. Pausing removes the table's sync schedule (already-ingested data is untouched); resuming restores it — the primary action button itself becomes **Resume Sync** while the task is paused. See [Task Observability](/corecapabilities/observability/minion-task-runs) for the broader task-monitoring view these controls sit alongside. # Advanced Sizing & Operational Guidance Source: https://docs.startree.ai/corecapabilities/external-table/operational-guidance Capacity guardrails, deployment recommendations, and additional troubleshooting fixes for External Tables, beyond the core Best Practices and Troubleshooting references. > [!WARNING] > > This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it. Supplemental guidance for running External Tables at scale — capacity guardrails, deployment recommendations, and a couple of additional error fixes not covered in [Best Practices and Configs](./best-practices-and-configs) or [Troubleshooting](./troubleshooting). *** ## Why remote Parquet queries are fast Most engines that query Iceberg/Parquet directly (e.g. Trino, Athena) read at the **row-group / column-chunk** level. Pinot reads at the **Parquet page** level — a finer granularity — and uses its own indexes (inverted, range) to skip straight to the pages that matter instead of scanning a whole column chunk. This page-level skip is what makes an indexed External Table competitive with a native Pinot segment despite the data living remotely. External Tables also federate cleanly with regular Pinot tables. A common pattern keeps recent data (e.g. the last 24h) in a REALTIME table fed from Kafka, while the same events land in Iceberg for long-term retention — a single query can span both, giving one query surface across the full time range without copying the archive into Pinot. *** ## Capacity guardrails These aren't hard limits, but the confident, well-tested operating range today: | Guardrail | Guidance | | - | - | | Files per table | Up to **\~350,000** Parquet files. Each source file maps 1:1 to a pseudo-segment today, so file count directly drives segment count. | | Segments per cluster | Stay well inside **\~500,000** total — treat this and the per-table file guardrail as overlapping budgets, not independent ones. | | Segments per server | Rule of thumb: **don't exceed \~10,000 segments per server**. | | File size | **\< 100 MB** is too small — too many files for the benefit; ask the source to compact. **128 MB – 512 MB** is the well-tested sweet spot. Up to **\~2 GB** generally works; beyond that (up to \~4 GB) is less exercised. | > [!NOTE] > > **The per-table guardrail is also enforced.** The controller's segment limit checker — which applies to external tables only — fails ingestion, task generation, and segment upload once a table crosses **350,000** segments or the cluster crosses **1,000,000** (the defaults StarTree Cloud runs with). The \~500,000 cluster figure above remains the recommended comfortable operating range; the enforced ceiling is higher. **Millions of files isn't viable today.** If a source has millions of small files, either compact upstream to raise average file size and cut file count below the guardrail, or onboard a subset first (e.g. the most recent partitions) and expand later rather than pointing at the full dataset on day one. **Full-refresh Iceberg tables** (where a snapshot replaces all existing files rather than appending) are supported, but a full replace can spike the segment count for that cycle. Before onboarding one, check how many files the refreshed snapshot contains, whether old segments need cleanup after each refresh, and whether the file count stays inside the guardrails above. *** ## Deployment * **Use a dedicated tenant** for External Table workloads rather than an existing production tenant — it isolates ingestion and query resource usage from other traffic on the cluster. * **Default replication factor is 1**, chosen to maximize ingestion throughput; pseudo-segments are small (single-digit KB to a few MB before indexes are attached), so the cost of raising it later — e.g. to 2 once onboarding is stable — is low. * Both **SSE and MSE** query execution modes are supported against External Tables. * For **AWS Glue** sources specifically, the cluster's IAM role/credentials also need Glue API access, in addition to the S3 permissions in [Troubleshooting → Access Denied](./troubleshooting#access-denied-http-403-when-creating-the-table): `glue:GetTable`, `glue:GetTables`, `glue:GetCatalog`, `glue:GetDatabase`, `glue:GetDatabases`, `glue:GetPartition`, `glue:GetPartitions`, `glue:BatchGetPartition`, plus `lakeformation:GetDataAccess` if the database is registered with Lake Formation. Scope these to the specific Glue databases/tables the cluster needs, and prefer an assumed IAM role over static access keys. *** ## Ingestion mechanics Each sync run has two phases, which explains a couple of behaviors that are otherwise easy to mistake for a bug: 1. **Segment generation (controller).** Source files are processed in batches (`iceberg.snapshotProcessing.batchSize`, default `1000`, capped by `iceberg.snapshotProcessing.maxFilesScannedPerRun`; the `taskMaxNumFiles` cap applies only to the legacy minion-executor path). The controller reads Parquet footer statistics (row count, min/max) and creates pseudo-segments — metadata and index pointers, not materialized data. 2. **Download and index (servers).** Servers download the segment artifacts and build indexes. **Queries can return partial results while this phase is in flight** — for example, a response noting some segments are unavailable — until all servers finish loading. **Sort or cluster upstream data** on the column most often used in equality filters (in the Iceberg/Parquet writer, before StarTree ever sees it) — this reduces random scans at read time, on top of whatever indexes are added on the Pinot side. **Disable the sync schedule after the initial bootstrap** for a table that only watches an S3 path for a one-time or occasional load (`enabled=false` on `ExternalTableSyncTask`). Leaving it on means every scheduled tick re-scans and bootstraps newly-discovered files, so the segment count keeps growing even when there's no new data to justify it. For catalog-backed incremental sources, rely on checkpointing (see [Best Practices & Configs — Checkpointing & schema evolution](./best-practices-and-configs#checkpointing--schema-evolution)) instead of re-bootstrapping. *** ## Null handling — why it matters Without null handling, Pinot doesn't build null bitmap vectors during segment generation, so columns containing nulls in the source Parquet aren't tracked — `IS NULL`, `IS NOT NULL`, and aggregations that depend on the distinction (`COUNT(col)` vs. `COUNT(*)`) will be wrong on those columns. **Current onboarding flows handle this automatically:** the preview/onboarding step sets `nullHandlingEnabled: true` on every generated table config, and when the source schema has nullable fields the inferred Pinot schema marks those columns nullable and enables column-based null handling. For tables created on older builds (where onboarding left it off), turn it on explicitly: ```json theme={null} "tableIndexConfig": { "nullHandlingEnabled": true } ``` Verify null behavior after ingestion either way — see [Null Handling in Complex Types](/corecapabilities/external-table/data-type-mapping#null-handling-in-complex-types) for a known `MAP`-column caveat. *** ## Missing time-column statistics — tolerated by default on new tables Source Parquet can legitimately carry no min/max footer statistics for the time column — an all-null chunk, a producer-side writer change, or Iceberg schema evolution. Under Pinot's strict default this fails segment generation with `No time statistics available for datetime column ... Cannot determine time range.` **Tables created through the preview API or `CREATE TABLE ... WITH (type = 'iceberg', ...)` are scaffolded with `ingestionConfig.segmentTimeValueCheck: false`**, so files missing time-column stats ingest instead of failing. A segment built from such a file has no derivable time range and is therefore **not pruned by time** — a time-filtered query scans it. Files whose Parquet does carry statistics are unaffected and keep their full time range. Corrupt or out-of-range time values are still rejected regardless of this flag. To restore strict checking, set `"segmentTimeValueCheck": true` under `ingestionConfig` — in the config returned by the preview API before creating the table, or via a table-config edit afterwards for a DDL-created or existing table. **Existing tables, direct `POST /tables` with a hand-written config, and Pinot-managed (non-External) tables are unaffected** — the tolerant default applies only to preview- and DDL-scaffolded External Tables created after upgrading. *** ## Additional troubleshooting ### ZooKeeper packet-size overflow as segment count grows **Symptom:** ``` Caused by: org.apache.zookeeper.KeeperException$... Packet len ... is out of range ``` **Cause:** The serialized ZNode record for the table has grown past ZooKeeper's default 1 MB packet limit — this shows up as segment count increases. **Fix:** Contact StarTree support. Raising the ZooKeeper record size limit is a cluster-level change that needs a controller and server restart. If the table will keep growing, consider [segment groups](./segment-groups), which keep the per-table metadata small. ### 403 on segment upload with 1970-era timestamps **Symptom:** ``` Got error status code: 403 (Forbidden) with reason: "Invalid segment start/end time: 1970-01-21T11:51:21.667Z / ... for segment: ..." ``` **Cause:** Iceberg-generated Parquet files store timestamps in milliseconds; Pinot's segment time-value sanity check misreads them as seconds, producing 1970-era dates and rejecting the segment. **Fix:** Add to the table's `ingestionConfig` (at creation, or via a config update followed by re-triggering ingestion): ```json theme={null} "ingestionConfig": { "segmentTimeValueCheck": false } ``` # External Tables Overview Source: https://docs.startree.ai/corecapabilities/external-table/overview Query Iceberg (AWS Glue, S3 Tables, Unity Catalog, Apache Polaris, Snowflake, Nessie) and raw Parquet (S3, GCS) tables directly from Pinot, with no ingestion or data duplication. > [!WARNING] > > This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it. An External Table is a Pinot table whose data stays in Parquet files in your object store — an Iceberg catalog (AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris, Snowflake Open Catalog / Horizon, Nessie) or an S3/GCS data lake — instead of being copied into Pinot's own segment format. Pinot reads the remote Parquet at query time and exposes it through standard SQL. There is no ETL pipeline, no data duplication, and onboarding takes minutes instead of hours. A watcher on the controller detects source-side changes on a schedule and builds segment index files (bloom filters, inverted indexes, range indexes, etc.) alongside the Parquet. Queries use server-side caches — a Parquet data cache, an index cache, and a footer cache — so repeat reads avoid paying S3 round-trip latency. With the right indexes configured, query times drop from minutes to milliseconds on large datasets. > [!TIP] > > For a single-page view of everything External Tables support — connectors, authentication methods, table formats, and features — see the [Support Matrix](/corecapabilities/external-table/support-matrix). *** ## How it works ```mermaid theme={null} flowchart LR UserSQL["SQL Query"] --> Broker["Pinot Broker"] Broker --> Server["Pinot Server"] Server --> CacheCheck{"Cache hit?"} CacheCheck -->|"Yes"| Result["Return result"] CacheCheck -->|"No"| S3["S3 / Object Store\nParquet files"] S3 --> Server Server --> LocalCache["Local caches\ndata · index · footer"] LocalCache --> Result ``` At query time, the server checks its local caches first. On a miss, it fetches the required Parquet column pages or index byte ranges from object storage and stores them for subsequent queries. Index files (built by the watcher at sync time) live in tiered storage and are also cached locally, so filters and aggregations avoid full column scans. *** ## Supported sources | Source | Protocol | `catalogType` | | - | - | - | | Iceberg (AWS Glue) | Iceberg REST | `iceberg-rest` (`serviceType=glue`) | | Iceberg (AWS S3 Tables) | Iceberg REST | `iceberg-rest` (`serviceType=s3Tables`) | | Unity Catalog | Iceberg REST | `iceberg-rest` (`serviceType=unity`) | | Nessie | Iceberg REST | `iceberg-rest` (`serviceType=rest`, with `prefix` set to a branch/ref) | | Apache Polaris / Snowflake Open Catalog | Iceberg REST | `iceberg-rest` (`serviceType=rest`, with `warehouse` set to the catalog name) | | Snowflake Horizon Catalog | Iceberg REST | `iceberg-rest` (`serviceType=rest`, served under `/polaris/api/catalog` on the Snowflake account host) | | S3 Data Lake | Raw Parquet files under an S3 prefix | `s3` | | GCS Data Lake | Raw Parquet files in GCS via S3 interop | `gcs-interop` | `catalogType=iceberg-rest` works with any Iceberg REST–compliant catalog (built-in service adapters exist for AWS Glue, AWS S3 Tables, Unity Catalog, and generic REST — the last of which also covers Nessie, addressed via its branch/ref as the REST `prefix`). Data files must be Parquet. > [!NOTE] > > There is no dedicated `catalogType` for Delta Lake's native log format. A **Delta Lake table with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled** publishes Iceberg-compatible metadata alongside its Delta log — registering it in a **Unity Catalog** (`serviceType=unity`) makes it queryable as an External Table through the same Iceberg REST path used for Glue and S3 Tables. *** ## Where to start New here? Start with a task instead of a vendor: * **["I just have Parquet files on S3/GCS"](./getting-started/query-without-a-catalog)** — no catalog, no Iceberg, query the bucket directly. * **["I already have a catalog"](./getting-started/connect-a-catalog)** — AWS Glue, Amazon S3 Tables, Unity Catalog, Apache Polaris / Snowflake, or Nessie. * **["My queries are slow"](./getting-started/speed-up-queries)** — indexes, caching, and OOM protection. Otherwise, choose your catalog type and pick your preferred onboarding path: **AWS Glue** * [Onboarding via Data Portal](./glue/onboarding-data-portal) — wizard-based setup for AWS Glue * [Onboarding via API](./glue/onboarding-api) — 4-step REST flow with bash examples and a quickstart script **Amazon S3 Tables** * [Onboarding via Data Portal](./s3tables/onboarding-data-portal) — wizard-based setup for Amazon S3 Tables * [Onboarding via API](./s3tables/onboarding-api) — 4-step REST flow with bash examples and a quickstart script **Unity Catalog** * [Onboarding via Data Portal](./unity/onboarding-data-portal) — wizard-based setup for Databricks-managed and OSS Unity Catalog * [Onboarding via API](./unity/onboarding-api) — 4-step REST flow; supports Databricks-managed and OSS Unity Catalog, including Delta Lake tables with UniForm enabled **Apache Polaris & Snowflake Open Catalog** * [Onboarding via Data Portal](./polaris/onboarding-data-portal) — wizard-based setup with OAuth2 principal credentials * [Onboarding via API](./polaris/onboarding-api) — 4-step REST flow using the generic Iceberg REST adapter with OAuth2 principal credentials **Snowflake Horizon Catalog** * [Onboarding via Data Portal](./horizon/onboarding-data-portal) — wizard-based setup with a Snowflake programmatic access token (PAT) * [Onboarding via API](./horizon/onboarding-api) — 4-step REST flow authenticating with a Snowflake programmatic access token (PAT) **Nessie** * [Onboarding via API](./nessie/onboarding-api) — 4-step REST flow using the generic Iceberg REST adapter with a branch/ref `prefix` * Onboarding via Data Portal is not yet available — the API flow is the only option today **S3 Data Lake** * [Onboarding via Data Portal](./s3/onboarding-data-portal) — wizard-based setup for raw Parquet on S3 * [Onboarding via API](./s3/onboarding-api) — 4-step REST flow with bash examples and a quickstart script **GCS Data Lake** * [Onboarding via Data Portal](./gcs/onboarding-data-portal) — wizard-based setup for Parquet files in GCS * [Onboarding via API](./gcs/onboarding-api) — 4-step REST flow using GCS S3 interop with HMAC keys **After onboarding** * **Monitoring sync progress** → [Observability](./observability) — sync status, checkpoint watermark, and source file count APIs * **Queries are slow** → [Indexes](./indexes) to add the right indexes, then [Best Practices & Configs](./best-practices-and-configs) for caching and tuning * **Something is broken** → [Troubleshooting](./troubleshooting) for symptom-based fixes, or [FAQ](./faq) for common questions **New in 0.16.0** * **[SQL DDL](./sql-ddl)** — create an External Table with a single `CREATE TABLE ... WITH (...)` statement instead of the 4-step REST flow. * **[Large External Tables](./segment-groups)** — segment groups track a batch of small member segments as one unit, keeping cluster metadata manageable for tables backed by millions of source files. * **[Deletion Vectors](./deletion-vectors)** (preview) — apply Iceberg row-level deletes/updates at query time instead of requiring append-only sources. *** ## Page map | Page | What it covers | | - | - | | [Query Parquet Directly](./getting-started/query-without-a-catalog) | Task-based entry point for raw Parquet on S3/GCS with no catalog | | [Connect a Catalog](./getting-started/connect-a-catalog) | Task-based entry point for Glue, S3 Tables, Unity, Polaris/Snowflake, and Nessie | | [Speed Up Queries](./getting-started/speed-up-queries) | Task-based entry point for indexes, caching, and OOM protection | | [AWS Glue: Onboarding via Data Portal](./glue/onboarding-data-portal) | Point-and-click wizard for AWS Glue | | [AWS Glue: Onboarding via API](./glue/onboarding-api) | REST API 4-step flow for AWS Glue (Iceberg REST) | | [Amazon S3 Tables: Onboarding via Data Portal](./s3tables/onboarding-data-portal) | Point-and-click wizard for Amazon S3 Tables | | [Amazon S3 Tables: Onboarding via API](./s3tables/onboarding-api) | REST API 4-step flow for Amazon S3 Tables (Iceberg REST) | | [Unity Catalog: Onboarding via Data Portal](./unity/onboarding-data-portal) | Point-and-click wizard for Unity Catalog, including Delta+UniForm tables | | [Unity Catalog: Onboarding via API](./unity/onboarding-api) | REST API 4-step flow for Unity Catalog (Iceberg REST), including Delta+UniForm tables | | [Apache Polaris & Snowflake Open Catalog: Onboarding via Data Portal](./polaris/onboarding-data-portal) | Point-and-click wizard for Apache Polaris and Snowflake Open Catalog | | [Apache Polaris & Snowflake Open Catalog: Onboarding via API](./polaris/onboarding-api) | REST API 4-step flow with OAuth2 principal credentials (generic `rest` adapter) | | [Snowflake Horizon Catalog: Onboarding via Data Portal](./horizon/onboarding-data-portal) | Point-and-click wizard for Snowflake-managed Iceberg tables | | [Snowflake Horizon Catalog: Onboarding via API](./horizon/onboarding-api) | REST API 4-step flow with a Snowflake programmatic access token (PAT) | | [Nessie: Onboarding via API](./nessie/onboarding-api) | REST API 4-step flow for Nessie (Iceberg REST, generic `rest` adapter with a branch/ref `prefix`) | | [S3: Onboarding via Data Portal](./s3/onboarding-data-portal) | Point-and-click wizard for raw Parquet on S3 | | [S3: Onboarding via API](./s3/onboarding-api) | REST API 4-step flow for S3 Data Lake | | [GCS Data Lake: Onboarding via Data Portal](./gcs/onboarding-data-portal) | Point-and-click wizard for Parquet in GCS via S3 interop | | [GCS Data Lake: Onboarding via API](./gcs/onboarding-api) | REST API 4-step flow using GCS HMAC keys and S3 interop | | [Observability](./observability) | Sync run status, ingestion checkpoint, source file count, and manual trigger APIs | | [Data Type Mapping](./data-type-mapping) | Parquet → Pinot and Iceberg → Pinot type mapping tables, plus time column detection | | [Indexes](./indexes) | Supported indexes, why columns are RAW, and per-index config examples | | [Data and Index Caching](./data-and-index-caching) | Three caches (data, index, footer), eviction, restart behavior, and how to clear them | | [Best Practices & Configs](./best-practices-and-configs) | Full config reference: sync task, tier backend, server/cluster, query options, and OOM protection | | [FAQ](./faq) | Common questions by category: general, onboarding, schema, indexes, performance, operations | | [DataExportTask](./data-export-task) | Export completed REALTIME segments to Parquet on filesystem or Iceberg REST | | [Troubleshooting](./troubleshooting) | Symptom-based diagnostic playbook with exact error strings and escalation guidance | | [SQL DDL](./sql-ddl) | Create an External Table with `CREATE TABLE ... WITH (...)` instead of the REST flow | | [Large External Tables](./segment-groups) | Segment groups: keep cluster metadata manageable for tables with millions of files | | [Deletion Vectors](./deletion-vectors) | Apply Iceberg row-level deletes/updates at query time (preview) | # Apache Polaris & Snowflake Open Catalog: Onboarding via API Source: https://docs.startree.ai/corecapabilities/external-table/polaris/onboarding-api Onboard an Apache Polaris or Snowflake Open Catalog Iceberg table as a StarTree External Table over the Iceberg REST API, authenticating with OAuth2 principal (client-id/client-secret) credentials. > [!WARNING] > > This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it. This page shows how to onboard an **Apache Polaris** or **Snowflake Open Catalog** External Table with the StarTree controller REST APIs. Prefer a point-and-click setup? The Data Portal wizard has dedicated **Apache Polaris** and **Snowflake Open Catalog** tiles — see [Onboarding via Data Portal](./onboarding-data-portal). Polaris tables are accessed over the standard **Iceberg REST protocol**. There's no dedicated `serviceType` for Polaris — use the generic spec-compliant adapter: `catalogType=iceberg-rest`, `serviceType=rest`, with `catalog.iceberg-rest.warehouse` set to the Polaris **catalog name**. Data files must be **Parquet**. > Using **Nessie**? It rides the same generic `rest` adapter with a branch/ref `prefix` — see [Nessie: Onboarding via API](../nessie/onboarding-api). ## Apache Polaris and Snowflake Open Catalog **Snowflake Open Catalog** is Snowflake's managed Apache Polaris service. It speaks the same Iceberg REST protocol and uses the same **principal**-based OAuth2 (`client-id` / `client-secret`, `PRINCIPAL_ROLE:ALL` scope) as self-managed Polaris — only the host and token-endpoint path differ. Self-managed Polaris serves the REST API under `/api/catalog`; Open Catalog serves it under `https://-.snowflakecomputing.com/polaris/api/catalog`. Everything else on this page — principal credentials, scope, catalog-name `warehouse` — applies to both. > [!NOTE] > > Using **Snowflake Horizon Catalog** instead? Snowflake has stopped provisioning new Open Catalog accounts and now routes customers to **Horizon**, which exposes the same Polaris REST API on the Snowflake account host but authenticates differently (a service-user **PAT** passed as the catalog credential, a `session:role:` scope, and a required `prefix`). It has its own page: [Snowflake Horizon Catalog: Onboarding via API](../horizon/onboarding-api). ## How it works Onboarding is four calls. Each one feeds the next: ```text theme={null} 1. Validate & browse POST /connections/browse → validate; list namespaces/tables 2. Preview POST /tables/preview → infer schema + enriched table config 3. Create schema POST /schemas → register the schema 4. Create table POST /tables → register the External Table ``` The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output. Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding). > [!NOTE] > > All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` (the StarTree Cloud data-plane proxy). A browse call then posts to `$BASE_URL/connections/browse`. > > If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity. ## Prerequisites * StarTree 0.16.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure. * Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth. * A running Polaris deployment with its Iceberg REST API reachable: self-managed Apache Polaris serves it under `/api/catalog`; Snowflake Open Catalog serves it under `https://-.snowflakecomputing.com/polaris/api/catalog`. * A Polaris **principal** with client credentials (`client-id` / `client-secret`) whose principal role maps, through a catalog role, to at least read grants on the target catalog (for example `TABLE_READ_DATA`, which implies the list/metadata grants needed for browsing). See the [Polaris access-control docs](https://polaris.apache.org/) for creating principals, principal roles, catalog roles, and grants. * The Polaris **catalog name** (used as the `warehouse`) and the namespace/table to onboard. * Read access to the underlying object storage. This connector supports **S3-compatible storage** for the data files. ## Limitations * **No credential vending.** Polaris can vend short-lived storage credentials to engines that request them, but the generic `rest` adapter does not consume vended credentials — configure static keys, an assumed role, or the cluster's node role for storage explicitly. ## Authentication An External Table authenticates in two places, configured independently: * **Catalog (REST)** — keys under `catalog.iceberg-rest.auth.rest.*`. Polaris always requires authentication; choose one of the methods below. * **Storage (S3 data files)** — keys under `catalog.iceberg-rest.auth.storage.*`. Choose one of the three methods below. ### Catalog (REST) authentication: choose one #### OAuth2 client credentials (recommended) The standard Polaris method: the principal's `client-id` / `client-secret` are exchanged at the Polaris token endpoint for a short-lived token, which StarTree caches and refreshes automatically. `authType` is optional; it's auto-detected as `oauth2` once the three required keys below are all present. Polaris requires a scope naming the principal role to activate — use `PRINCIPAL_ROLE:ALL` to activate all of the principal's roles, or `PRINCIPAL_ROLE:` to restrict the session to one role. ```json theme={null} { "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https:///api/catalog/v1/oauth/tokens", "catalog.iceberg-rest.auth.rest.oauthClientId": "", "catalog.iceberg-rest.auth.rest.oauthClientSecret": "", "catalog.iceberg-rest.auth.rest.oauthScopes": "PRINCIPAL_ROLE:ALL" } ``` > [!NOTE] > > For **Snowflake Open Catalog**, the token endpoint is the same path on the account host: `https://-.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens`. #### OAuth2 against an external identity provider Polaris deployments configured to trust an external IdP (Keycloak, Okta, and so on) accept tokens minted by that IdP instead of the built-in token endpoint. Use the same four keys, pointing `oauthTokenUri` at the IdP's client-credentials token endpoint and setting whatever scopes the IdP requires. ```json theme={null} { "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https:///realms//protocol/openid-connect/token", "catalog.iceberg-rest.auth.rest.oauthClientId": "", "catalog.iceberg-rest.auth.rest.oauthClientSecret": "", "catalog.iceberg-rest.auth.rest.oauthScopes": "" } ``` #### Bearer token If you already hold a valid token (for example, minted out-of-band from your IdP), pass it directly. `authType` doesn't need to be set explicitly — it's auto-detected as token auth when `.token` is present. Note that a static token is **not refreshed**: when it expires, syncs start failing until the table config is updated, so prefer the OAuth2 client-credentials flow for anything long-running. ```json theme={null} { "catalog.iceberg-rest.auth.rest.token": "" } ``` ### Storage authentication: choose one #### Assumed IAM role (recommended for production) No static secrets in the table config — every S3 read assumes `roleArn` via STS. Three things to know: `auth.storage.region` is **required** whenever a `roleArn` is set (the sync fails fast without it); any static `auth.storage` access keys are **ignored** once a `roleArn` is present; and the `AssumeRole` call itself authenticates with the cluster's ambient AWS identity (node role / IRSA), so the target role's trust policy must trust that identity (plus the `externalId`, if you set one). ```json theme={null} { "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Cluster node role Omit every `auth.storage.*` credential key — only `region` is required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have S3 access. ```json theme={null} { "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Static access keys Quick tests, or when role-based access isn't available. Use the canonical `accessKeyId` / `secretAccessKey` spellings (an optional `sessionToken` is also accepted). Avoid the shorter `accessKey`/`secretKey` aliases: only part of the pipeline honors them (segment generation does, Iceberg manifest and data-file reads do not), so a short-alias config falls back to the cluster's ambient credentials for catalog file reads and can fail in confusing ways. ```json theme={null} { "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` ### Selecting the catalog A Polaris server hosts multiple catalogs. Set `catalog.iceberg-rest.warehouse` to the catalog name — the adapter passes it to the server's config endpoint, which resolves the per-catalog prefix automatically. ```json theme={null} { "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.prefix": "" } ``` Also set `catalog.iceberg-rest.prefix` to the catalog name whenever the prefix can't be negotiated automatically. This is **required for Snowflake Open Catalog** and any deployment whose `/v1/config` endpoint gates on the `warehouse` query parameter, or that sits behind a proxy which doesn't forward the config handshake. Without it, browse fails with `404 Unable to find matching target resource method`, because the request falls back to the un-prefixed `/v1/namespaces` path. The examples below set both `warehouse` and `prefix`. *** ## Step 1: Validate and browse the connection **`POST /connections/browse`** There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your credentials, scope, and connectivity. * Set `path` to `""` to browse the **root** — this both validates the connection and lists namespaces. * Set `path` to a namespace name to list the **tables** inside it. ### Request | Field | Type | Required | Description | | - | - | - | - | | `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. | | `connection.params` | object | Yes | Catalog-specific connection settings (see example below). | | `path` | string | No | Where to browse. `""` or omitted = root. A namespace name = that namespace's tables. | ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "iceberg-rest", "catalog.iceberg-rest.serviceType": "rest", "catalog.iceberg-rest.restUri": "https:///api/catalog", "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.prefix": "", "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https:///api/catalog/v1/oauth/tokens", "catalog.iceberg-rest.auth.rest.oauthClientId": "", "catalog.iceberg-rest.auth.rest.oauthClientSecret": "", "catalog.iceberg-rest.auth.rest.oauthScopes": "PRINCIPAL_ROLE:ALL", "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } }, "path": "" }' ``` | Key | Value | Description | | - | - | - | | `catalog.iceberg-rest.restUri` | `https:///api/catalog` | The Polaris Iceberg REST base URI (Snowflake Open Catalog: `https://-.snowflakecomputing.com/polaris/api/catalog`). | | `catalog.iceberg-rest.serviceType` | `rest` | Selects the generic spec-compliant adapter — there's no Polaris-specific `serviceType`. | | `catalog.iceberg-rest.warehouse` | `` | The Polaris catalog to browse; the server resolves the per-catalog prefix from it. | | `catalog.iceberg-rest.prefix` | `` | Pins the catalog prefix explicitly. Required for Snowflake Open Catalog and any proxy that doesn't forward the config handshake. | ### Response ```json theme={null} { "items": [ { "name": "transportation", "type": "NAMESPACE" }, { "name": "nyc_taxi_trips", "type": "TABLE" } ] } ``` | `items[].type` | Meaning | | - | - | | `NAMESPACE` | A namespace you can drill into — pass its `name` as `path`. | | `TABLE` | A selectable table. | *** ## Step 2: Preview the schema **`POST /tables/preview`** Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4. > [!NOTE] > > The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`. ### Request **Top-level fields:** | Field | Type | Required | Description | | - | - | - | - | | `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. | | `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. | | `schema` | object | No | An explicit schema to use instead of inferring one. | **`config.inference` — how the schema is derived:** | Field | Type | Default | Description | | - | - | - | - | | `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Iceberg metadata. **Set this to `true` for External Tables.** | | `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. | Resolution order: explicit `schema` → embedded schema → inferred schema. **`config.sampling` — how rows are sampled:** | Field | Type | Default | Description | | - | - | - | - | | `min` | integer | `1` | Minimum records to sample. | | `max` | integer | `100` | Maximum records to sample. | ```bash theme={null} curl -X POST "$BASE_URL/tables/preview" \ -H "Content-Type: application/json" \ -d '{ "tableConfig": { "tableName": "nyc_taxi_trips_OFFLINE", "tableType": "OFFLINE", "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "catalogType": "iceberg-rest", "executor": "controller", "inputFormat": "parquet", "catalog.iceberg-rest.serviceType": "rest", "catalog.iceberg-rest.restUri": "https:///api/catalog", "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.prefix": "", "catalog.iceberg-rest.table.namespace": "transportation", "catalog.iceberg-rest.table.tableName": "nyc_taxi_trips", "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "https:///api/catalog/v1/oauth/tokens", "catalog.iceberg-rest.auth.rest.oauthClientId": "", "catalog.iceberg-rest.auth.rest.oauthClientSecret": "", "catalog.iceberg-rest.auth.rest.oauthScopes": "PRINCIPAL_ROLE:ALL", "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } } } }, "config": { "inference": { "embeddedSchemaEnabled": true } } }' ``` > [!NOTE] > > **Setting the namespace and table.** Take them from the [browse](#step-1-validate-and-browse-the-connection) response — set `catalog.iceberg-rest.table.namespace` / `.tableName` to the `NAMESPACE` and `TABLE` you selected. > > **Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4. ### Response | Field | Type | Description | | - | - | - | | `schema` | object | The resolved Pinot schema. **Use this in step 3.** | | `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. | | `rows` | array | Sample records after schema + ingestion transforms. | | `sourceRows` | array | Raw records before transformation. | | `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). | Adjust the schema (time column, column names, null handling) before moving on. *** ## Step 3: Create the schema **`POST /schemas`** Send the `schema` object from the preview response. Rename its `schemaName` to match your table. ```bash theme={null} curl -X POST "$BASE_URL/schemas" \ -H "Content-Type: application/json" \ -d @schema.json ``` ```json theme={null} { "status": "nyc_taxi_trips successfully added" } ``` *** ## Step 4: Create the table **`POST /tables`** Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external. ```bash theme={null} curl -X POST "$BASE_URL/tables" \ -H "Content-Type: application/json" \ -d @tableConfig.json ``` ```json theme={null} { "status": "Table nyc_taxi_trips_OFFLINE successfully added" } ``` The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call. > [!TIP] > > The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run: > > ```bash theme={null} > curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE" > ``` *** ## Quickstart: onboard a table end-to-end The whole flow as one script. Fill in the variables at the top, run the script, and the first sync starts automatically. ```bash theme={null} #!/usr/bin/env bash set -euo pipefail BASE_URL="https://dp..cp..startree.cloud/api/pinot" AUTH="Authorization: Bearer " REST_URI="https:///api/catalog" TOKEN_URI="https:///api/catalog/v1/oauth/tokens" WAREHOUSE="" PREFIX="" CLIENT_ID="" CLIENT_SECRET="" SCOPES="PRINCIPAL_ROLE:ALL" NAMESPACE="transportation" TABLE="nyc_taxi_trips" STORAGE_ROLE_ARN="arn:aws:iam::123456789012:role/pinot-external-table-reader" STORAGE_EXTERNAL_ID="" STORAGE_REGION="" # 1. Validate & browse root — lists namespaces in the Polaris catalog. curl -sf -X POST "$BASE_URL/connections/browse" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"connection\": { \"type\": \"CATALOG\", \"params\": { \"catalogType\": \"iceberg-rest\", \"catalog.iceberg-rest.serviceType\": \"rest\", \"catalog.iceberg-rest.restUri\": \"$REST_URI\", \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\", \"catalog.iceberg-rest.prefix\": \"$PREFIX\", \"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\", \"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\", \"catalog.iceberg-rest.auth.rest.oauthClientId\": \"$CLIENT_ID\", \"catalog.iceberg-rest.auth.rest.oauthClientSecret\": \"$CLIENT_SECRET\", \"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\" } }, \"path\": \"\" }" | jq . # 2. Preview — infer schema + enriched table config. curl -sf -X POST "$BASE_URL/tables/preview" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"tableConfig\": { \"tableName\": \"${TABLE}_OFFLINE\", \"tableType\": \"OFFLINE\", \"task\": { \"taskTypeConfigsMap\": { \"ExternalTableSyncTask\": { \"catalogType\": \"iceberg-rest\", \"executor\": \"controller\", \"inputFormat\": \"parquet\", \"catalog.iceberg-rest.serviceType\": \"rest\", \"catalog.iceberg-rest.restUri\": \"$REST_URI\", \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\", \"catalog.iceberg-rest.prefix\": \"$PREFIX\", \"catalog.iceberg-rest.table.namespace\": \"$NAMESPACE\", \"catalog.iceberg-rest.table.tableName\": \"$TABLE\", \"catalog.iceberg-rest.auth.rest.authType\": \"oauth2\", \"catalog.iceberg-rest.auth.rest.oauthTokenUri\": \"$TOKEN_URI\", \"catalog.iceberg-rest.auth.rest.oauthClientId\": \"$CLIENT_ID\", \"catalog.iceberg-rest.auth.rest.oauthClientSecret\": \"$CLIENT_SECRET\", \"catalog.iceberg-rest.auth.rest.oauthScopes\": \"$SCOPES\", \"catalog.iceberg-rest.auth.storage.roleArn\": \"$STORAGE_ROLE_ARN\", \"catalog.iceberg-rest.auth.storage.externalId\": \"$STORAGE_EXTERNAL_ID\", \"catalog.iceberg-rest.auth.storage.region\": \"$STORAGE_REGION\" } } } }, \"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } } }" > preview.json # 3. Extract schema and table config from preview response. jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json jq '.tableConfigs.offline' preview.json > tableConfig.json # 4. Create the schema. curl -sf -X POST "$BASE_URL/schemas" \ -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json # 5. Create the table — watcher starts the first sync automatically. curl -sf -X POST "$BASE_URL/tables" \ -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json # 6. (optional) Poll until sync completes. curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \ | jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered' ``` The script uses an assumed IAM role for storage (`auth.storage.roleArn`); swap in static keys or the node-role default (`region` only) from [Authentication](#authentication) if that fits your setup better. Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable). ## Monitor onboarding The [status endpoint](../observability#run-status) reports each sync run's progress, and `?includeLag=true` adds how far the table trails the upstream catalog. Both require `executor=controller` (set automatically by this flow). See [Observability](../observability) for full request and response details. ## Verify it's queryable When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips; ``` The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting). *** ## What's next Now that the table is created and data is loading, these are the highest-impact follow-up steps: 1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes) 2. **Enable caching and pin hot indexes.** Set `enable.prefetch.page.cache=true` on the S3 tier, and list small hot indexes in `preload.index.keys.override`, so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching) 3. **Monitor ongoing syncs.** Use the status endpoint (with `includeLag=true` for ingestion lag) after each scheduled sync. → [Observability](../observability) *** For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting). # Apache Polaris & Snowflake Open Catalog: Onboarding via Data Portal Source: https://docs.startree.ai/corecapabilities/external-table/polaris/onboarding-data-portal Connect an Apache Polaris or Snowflake Open Catalog Iceberg table to StarTree as an External Table with the point-and-click Data Portal wizard, authenticating with OAuth2 principal credentials. > [!WARNING] > > This feature is available starting in **StarTree release 0.16.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment. This guide walks through connecting an **Apache Polaris** or **Snowflake Open Catalog** table to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface. Both catalogs speak the standard **Iceberg REST protocol** and authenticate with **OAuth2 principal credentials** (client ID + client secret). The wizard has a dedicated tile for each; the two forms are identical apart from defaults, so this page covers both. Data files must be **Parquet**. > Looking for the API-based approach? See [Apache Polaris & Snowflake Open Catalog: Onboarding via API](./onboarding-api). > > Using **Snowflake Horizon Catalog** (Snowflake-managed Iceberg tables, PAT auth) instead? See [Snowflake Horizon: Onboarding via Data Portal](../horizon/onboarding-data-portal). *** ## Prerequisites Before starting, ensure you have: * **StarTree 0.16.0 or later** with the External Table feature enabled and **tiered storage configured** for your environment. Contact StarTree support if unsure. * A running catalog: * **Apache Polaris** — a reachable Polaris server and a principal (client ID + client secret) whose principal role can read the catalog. * **Snowflake Open Catalog** — an Open Catalog account and a service connection's principal credentials. * The **catalog name** (Polaris/Open Catalog catalog backing the Iceberg tables) and the namespace/table you want to onboard. * Read access to the underlying S3 object storage — static keys or an IAM role StarTree can assume. See [Onboarding via API — Authentication](./onboarding-api#authentication) for the permissions required. *** ## Step 1: Open the External Tables 1. Log in to **Data Portal**. 2. In the left navigation, go to **Tables**. 3. Click **+ Connect External Table**. The wizard opens with a connection configuration screen. *** ## Step 2: Configure the Catalog Connection Choose **Apache Polaris** or **Snowflake Open Catalog** as the catalog type — they appear as top-level tiles alongside AWS Glue, Amazon S3 Tables, and Unity Catalog. Fill in the connection details: **Details** | Field | Description | | - | - | | **Connection Name** | A unique name to identify this connection in Data Portal. | **Metastore** — how StarTree authenticates with the Iceberg REST API. | Field | Description | | - | - | | **Catalog Endpoint URI** | The Iceberg REST endpoint — `https:///api/catalog` for Apache Polaris, or `https://.snowflakecomputing.com/polaris/api/catalog` for Snowflake Open Catalog. | | **Catalog Name** | The Polaris / Open Catalog **catalog** backing the Iceberg tables. | | **Prefix** | *(Snowflake Open Catalog only)* Multi-catalog REST prefix; pre-filled from the Catalog Name — leave the default unless your deployment uses a different prefix. | | **REST Authentication Type** | **OAuth** — the only option for these catalogs. | | **OAuth Token URI** | The catalog's OAuth2 token endpoint, e.g. `.../api/catalog/v1/oauth/tokens` (Polaris) or `https://.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens` (Open Catalog). | | **OAuth Client ID** | The principal's client ID. | | **OAuth Client Secret** | The principal's client secret. | | **OAuth Scopes** | The principal-role scope, default `PRINCIPAL_ROLE:ALL`. Narrow it to a specific principal role if your security model requires it. | **Storage** — credentials for reading the underlying Parquet data files from S3. | Field | Description | | - | - | | **Region** | AWS region where the S3 data files are stored. Always required. | | **Authentication Type** | **Basic Authentication** (Access Key + Secret Key), **IAM Node Role** (the cluster node's own IAM identity — Data Portal shows the resolved role as a read-only **IAM Role Name** field so you can grant it access in your AWS account), or **IAM Role** (a cross-account role StarTree assumes). | | **Access Key** / **Secret Key** | With **Basic Authentication**: static AWS credentials for S3 data access. | | **IAM Role ARN** | With **IAM Role**: the ARN of the role to assume. The form shows **Steps to create IAM Policy** — a ready-made trust policy including the StarTree account and External ID to configure on the role. | Click **Validate Connection**. Data Portal calls the catalog's validate endpoint and confirms credentials and connectivity before proceeding. *** ## Step 3: Browse and Select a Table Once the connection is validated: 1. Data Portal lists the available **namespaces** — the field is labeled **Namespace** (Apache Polaris) or **Schema** (Snowflake Open Catalog). 2. Select one to expand its tables. 3. Click the **table** you want to onboard. Data Portal reads the Iceberg schema and derives a Pinot schema automatically. *** ## Step 4: Review the Schema The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity. Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct. *** ## Step 5: Configure the Table Set the final table options: | Setting | Default | Notes | | - | - | - | | **Table Name** | — | Name of the Pinot table to create. | | **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. | | **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. | The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config. Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required. *** ## Step 6: Monitor Onboarding The table detail view shows sync status in real time — files discovered, segments generated, last synced snapshot, and ingestion lag. Use **Sync Now** to trigger an immediate sync, or **Pause Sync** / **Resume Sync** from the table menu (while the sync task is paused, the primary button itself reads **Resume Sync**). For API-based monitoring, see [Observability](../observability). # S3: Onboarding via API Source: https://docs.startree.ai/corecapabilities/external-table/s3/onboarding-api Onboard raw Parquet files under an S3 prefix as a StarTree External Table via the REST API — no Iceberg or catalog service required. > [!WARNING] > > This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it. This page shows how to onboard an S3 Data Lake External Table with the StarTree controller REST APIs instead of the [Data Portal UI](./onboarding-data-portal). Use it for automation, infrastructure-as-code, or onboarding many tables at once. ## How it works Onboarding is four calls. Each one feeds the next: ```text theme={null} 1. Validate & browse POST /connections/browse → validate; list files under the S3 prefix 2. Preview POST /tables/preview → infer schema + enriched table config 3. Create schema POST /schemas → register the schema 4. Create table POST /tables → register the External Table ``` The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output. Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding). > [!NOTE] > > All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` — the StarTree Cloud data-plane proxy that fronts the controller. > > If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity. ## Prerequisites * StarTree 0.15.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure. * Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth. * AWS credentials (access key + secret key) with `s3:GetObject` and `s3:ListBucket` on the source bucket and prefix. * The S3 bucket name, key prefix, and region. > [!WARNING] > > **Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly. > > If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details. ## Limitations * **No catalog metadata** — namespace and table names are values you choose, not read from a catalog; browse only validates the connection and lists files/directories. * **No schema evolution.** Raw Parquet has no Iceberg field-id history to key a rename against, so a source-side column rename always requires a manual schema change (unlike `catalogType=iceberg-rest` sources with `schemaEvolution.enabled=true`). * **`catalog.s3.prefix` is a permanent filter, not a starting cursor** — see the Warning further down this page, under the `catalog.s3.*` reference table, before using it to try to skip ahead in a large bucket. ## Authentication Unlike the Iceberg-catalog sources, there's only one auth surface here — `catalog.s3.*` covers both listing the bucket and reading the Parquet data. Choose one of three credential methods. The role or keys need `s3:GetObject` and `s3:ListBucket` on the source bucket and prefix. Verify from the cluster before onboarding: ```bash theme={null} aws s3 ls s3://// ``` ### Assumed IAM role (recommended for production) No static secrets in the table config — every S3 call assumes `roleArn` via STS. ```json theme={null} { "catalog.s3.bucketName": "", "catalog.s3.prefix": "", "catalog.s3.region": "", "catalog.s3.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.s3.externalId": "" } ``` ### Cluster node role Omit `roleArn`/`accessKey`/`secretKey` entirely — only `bucketName`, `prefix`, and `region` are required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have access to the bucket. ```json theme={null} { "catalog.s3.bucketName": "", "catalog.s3.prefix": "", "catalog.s3.region": "" } ``` ### Static access keys Quick tests, or when role-based access isn't available. ```json theme={null} { "catalog.s3.bucketName": "", "catalog.s3.prefix": "", "catalog.s3.region": "", "catalog.s3.accessKey": "", "catalog.s3.secretKey": "" } ``` ### `catalog.s3.*` reference | Key | Default | Description | | - | - | - | | `catalog.s3.bucketName` | *(required)* | Source bucket name — no `s3://` prefix. | | `catalog.s3.prefix` | *(required)* | Key prefix to scan for source files. | | `catalog.s3.region` | *(required)* | AWS region for the bucket. | | `catalog.s3.roleArn` | — | Assumed IAM role for bucket access (recommended for production). | | `catalog.s3.externalId` | — | External ID for the assumed role, if your role trust policy requires one. | | `catalog.s3.accessKey` / `catalog.s3.secretKey` | — | Static AWS credentials, if not using an assumed role or the node role. | | `catalog.s3.startAfter` | *(unset)* | New in 0.16.0. Seeds the lower bound for the first S3 listing call, as a bucket-relative key. Only takes effect before any checkpoint exists. | > [!WARNING] > > Don't use `catalog.s3.prefix` to bound where ingestion *starts* within a large bucket. `prefix` is a permanent filter on which objects are ever considered — narrowing it to skip ahead permanently excludes everything outside that range, including future files. Use `catalog.s3.startAfter` instead, which only affects the very first sync run. *** ## Step 1: Validate and browse the connection **`POST /connections/browse`** There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your credentials and connectivity. Bad credentials or an unreachable prefix return an error. * Set `path` to `""` to browse the **root prefix** — this both validates the connection and lists files/directories. ### Request | Field | Type | Required | Description | | - | - | - | - | | `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. | | `connection.params` | object | Yes | S3 connection settings (see example below). | | `path` | string | No | Where to browse. `""` or omitted = root prefix. | ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "s3", "catalog.s3.bucketName": "", "catalog.s3.prefix": "", "catalog.s3.region": "", "catalog.s3.accessKey": "", "catalog.s3.secretKey": "" } }, "path": "" }' ``` ### Response ```json theme={null} { "items": [ { "name": "2024/01/", "type": "DIR" }, { "name": "data.parquet", "type": "FILE" } ] } ``` | `items[].type` | Meaning | | - | - | | `DIR` | A directory you can drill into — pass its `name` as `path`. | | `FILE` | A Parquet file under the prefix. | *** ## Step 2: Preview the schema **`POST /tables/preview`** Samples the Parquet files, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4. > [!NOTE] > > The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`. ### Request **Top-level fields:** | Field | Type | Required | Description | | - | - | - | - | | `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. | | `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. | | `schema` | object | No | An explicit schema to use instead of inferring one. | **`config.inference` — how the schema is derived:** | Field | Type | Default | Description | | - | - | - | - | | `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Parquet metadata. **Set this to `true` for External Tables.** | | `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. | **`config.sampling` — how rows are sampled:** | Field | Type | Default | Description | | - | - | - | - | | `min` | integer | `1` | Minimum records to sample. | | `max` | integer | `100` | Maximum records to sample. | > [!TIP] > > To list source files without sampling any data, set `config.previewFiles.previewFilesOnly = true`. The response returns matching file URIs in `sourceFiles` and skips schema inference. (OFFLINE only.) ```bash theme={null} curl -X POST "$BASE_URL/tables/preview" \ -H "Content-Type: application/json" \ -d '{ "tableConfig": { "tableName": "nyc_taxi_trips_OFFLINE", "tableType": "OFFLINE", "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "catalogType": "s3", "executor": "controller", "inputFormat": "parquet", "catalog.s3.bucketName": "", "catalog.s3.prefix": "", "catalog.s3.region": "", "catalog.s3.accessKey": "", "catalog.s3.secretKey": "", "catalog.s3.table.namespace": "default", "catalog.s3.table.tableName": "nyc_taxi_trips" } } } }, "config": { "inference": { "embeddedSchemaEnabled": true }, "sampling": { "min": 1, "max": 100 } } }' ``` > [!NOTE] > > **Setting the namespace and table.** For S3, browse only validates the connection and lists files; the names are values you choose — set `catalog.s3.table.namespace` (use `default` for a flat prefix) and `catalog.s3.table.tableName` (any logical name). > > **Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4. ### Response | Field | Type | Description | | - | - | - | | `schema` | object | The resolved Pinot schema. **Use this in step 3.** | | `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. | | `rows` | array | Sample records after schema + ingestion transforms. | | `sourceRows` | array | Raw records before transformation. | | `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). | Adjust the schema (time column, column names, null handling) before moving on. *** ## Step 3: Create the schema **`POST /schemas`** Send the `schema` object from the preview response. Rename its `schemaName` to match your table. ```bash theme={null} curl -X POST "$BASE_URL/schemas" \ -H "Content-Type: application/json" \ -d @schema.json ``` ```json theme={null} { "status": "nyc_taxi_trips successfully added" } ``` *** ## Step 4: Create the table **`POST /tables`** Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external. ```bash theme={null} curl -X POST "$BASE_URL/tables" \ -H "Content-Type: application/json" \ -d @tableConfig.json ``` ```json theme={null} { "status": "Table nyc_taxi_trips_OFFLINE successfully added" } ``` The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call. > [!TIP] > > The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run: > > ```bash theme={null} > curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE" > ``` *** ## Quickstart: onboard a table end-to-end The whole flow as one script. Fill in the variables at the top, run the script, and the first sync starts automatically. ```bash theme={null} #!/usr/bin/env bash set -euo pipefail BASE_URL="https://dp..cp..startree.cloud/api/pinot" AUTH="Authorization: Bearer " BUCKET="" PREFIX="" REGION="" ACCESS_KEY="" SECRET_KEY="" TABLE="nyc_taxi_trips" # 1. Validate & browse — confirms credentials and lists files. curl -sf -X POST "$BASE_URL/connections/browse" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"connection\": { \"type\": \"CATALOG\", \"params\": { \"catalogType\": \"s3\", \"catalog.s3.bucketName\": \"$BUCKET\", \"catalog.s3.prefix\": \"$PREFIX\", \"catalog.s3.region\": \"$REGION\", \"catalog.s3.accessKey\": \"$ACCESS_KEY\", \"catalog.s3.secretKey\": \"$SECRET_KEY\" } }, \"path\": \"\" }" | jq . # 2. Preview — infer schema + enriched table config. curl -sf -X POST "$BASE_URL/tables/preview" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"tableConfig\": { \"tableName\": \"${TABLE}_OFFLINE\", \"tableType\": \"OFFLINE\", \"task\": { \"taskTypeConfigsMap\": { \"ExternalTableSyncTask\": { \"catalogType\": \"s3\", \"executor\": \"controller\", \"inputFormat\": \"parquet\", \"catalog.s3.bucketName\": \"$BUCKET\", \"catalog.s3.prefix\": \"$PREFIX\", \"catalog.s3.region\": \"$REGION\", \"catalog.s3.accessKey\": \"$ACCESS_KEY\", \"catalog.s3.secretKey\": \"$SECRET_KEY\", \"catalog.s3.table.namespace\": \"default\", \"catalog.s3.table.tableName\": \"$TABLE\" } } } }, \"config\": { \"inference\": { \"embeddedSchemaEnabled\": true }, \"sampling\": { \"min\": 1, \"max\": 100 } } }" > preview.json # 3. Extract schema and table config from preview response. jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json jq '.tableConfigs.offline' preview.json > tableConfig.json # 4. Create the schema. curl -sf -X POST "$BASE_URL/schemas" \ -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json # 5. Create the table — watcher starts the first sync automatically. curl -sf -X POST "$BASE_URL/tables" \ -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json # 6. (optional) Poll until sync completes. curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \ | jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered' ``` Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable). ## Monitor onboarding Three read-only endpoints report ingestion progress — run status, ingestion checkpoint, and source file count — and require `executor=controller` (set automatically). See [Observability](../observability) for full request and response details. ## Verify it's queryable When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips; ``` The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting). *** ## What's next Now that the table is created and data is loading, these are the highest-impact follow-up steps: 1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes) 2. **Enable caching and pin hot indexes.** Set `enable.prefetch.page.cache=true` on the S3 tier, and list small hot indexes in `preload.index.keys.override`, so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching) 3. **Monitor ongoing syncs.** Use the observability endpoints to check run status, ingestion checkpoint, and source file count after each scheduled sync. → [Observability](../observability) *** For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting). # S3: Onboarding via Data Portal Source: https://docs.startree.ai/corecapabilities/external-table/s3/onboarding-data-portal Connect raw Parquet files on S3 to StarTree as an External Table with the point-and-click Data Portal wizard — no catalog service required. > [!WARNING] > > This feature is available starting in **StarTree release 0.15.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment. This guide walks through connecting an S3 Data Lake to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface. > Looking for the API-based approach? See [S3: Onboarding via API](./onboarding-api). *** ## Overview **S3 Data Lake** connects to raw Parquet files stored directly in an S3 bucket — no catalog service required. StarTree scans the specified S3 prefix, discovers Parquet files, and makes them queryable through Pinot. *** ## Prerequisites Before starting, ensure you have: * **StarTree 0.15.0 or later** with the external table Beta feature enabled for your environment. * AWS credentials (access key + secret key) with `s3:GetObject` and `s3:ListBucket` permissions on the source bucket and prefix. * The bucket name, key prefix, and AWS region for your Parquet data. > [!WARNING] > > **Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly. > > If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details. *** ## Step 1: Open the External Tables 1. Log in to **Data Portal**. 2. In the left navigation, go to **Tables**. 3. Click **+ Connect External Table**. The wizard opens with a connection configuration screen. *** ## Step 2: Select S3 Data Lake Choose **S3 Data Lake** as the catalog type. Fill in the connection details: | Field | Description | | - | - | | **Bucket Name** | Name of the S3 bucket containing your Parquet files — no `s3://` prefix. | | **Region** | AWS region where the bucket is located. | | **Authentication Type** | **Basic Authentication** (Access Key + Secret Key), **IAM Node Role** (the cluster node's own IAM identity — Data Portal shows the resolved role as a read-only **IAM Role Name** field so you can grant it access in your AWS account), or **IAM Role** (a cross-account role StarTree assumes). | | **Access Key** / **Secret Key** | With **Basic Authentication**: static AWS credentials with read access to the bucket. | | **IAM Role ARN** | With **IAM Role**: the ARN of the role to assume. The form shows **Steps to create IAM Policy** — a ready-made trust policy including the StarTree account and External ID to configure on the role. | Click **Validate Connection**. Data Portal calls the catalog's validate endpoint and confirms credentials and connectivity before proceeding. *** ## Step 3: Browse and Select a Table Once the connection is validated, enter the **Prefix** — the key prefix (folder path) pointing to the Parquet data, e.g. `path/to/parquet/data` (no leading slash; the field auto-sanitizes the value). It is passed to the S3 `ListObjectsV2` API and matches all objects whose keys start with this string. Then: 1. Data Portal lists the available **S3 prefixes** (directories) under your configured prefix. 2. Select a prefix to view the Parquet files it contains. 3. Click the **table** (prefix) you want to onboard. StarTree samples the Parquet files to derive a schema automatically. *** ## Step 4: Review the Schema The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity. Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct. *** ## Step 5: Configure the Table Set the final table options: | Setting | Default | Notes | | - | - | - | | **Table Name** | — | Name of the Pinot table to create. | | **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. | | **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. | The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config. Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required. *** ## Step 6: Monitor Onboarding Once the table is created, onboarding starts automatically. The table detail view shows the status in real time: * **Running** — the task is actively reading Parquet files and building Pinot segments. * **Completed** — onboarding finished successfully. * **Failed** — onboarding encountered an error. The error message and the number of files discovered vs. segments generated are surfaced to help diagnose the issue. For deeper observability — watcher status, checkpoint values, and per-file counts — see the [Observability](../observability) page. *** ## Pausing Onboarding To pause scheduled onboarding from Data Portal: 1. Open the table in the **Tables** view. 2. Click **Pause Sync**. This removes the `schedule` (cron trigger) from the table's `ExternalTableSyncTask` config and stashes its value under `schedule.paused`, so the watcher stops firing new runs. Any run currently in progress completes normally, and existing segments and the last checkpoint are untouched. Click **Resume Sync** to restore the schedule from `schedule.paused` — onboarding resumes from where it left off; it does not restart from scratch. > [!NOTE] > > This is a separate mechanism from the `enabled` key documented in [Best Practices & Configs](../best-practices-and-configs#table-config--externaltablesynctask) — that key can also be set to `false` directly via the table config or API to stop new runs, but the Data Portal's **Pause Sync** button acts on `schedule`/`schedule.paused`, not `enabled`. *** ## Frequently Asked Questions **The Validate step fails — what should I check?** * Confirm the access key has `s3:GetObject` and `s3:ListBucket` permissions on the bucket and prefix. * Verify the region matches where your S3 bucket is located. Validate now compares the configured region against the bucket directly and reports a region mismatch by name, rather than passing and failing later during preview. * Check that the prefix ends with `/` if it's a folder path. *** **Can I onboard multiple prefixes from the same bucket?** Yes. Each prefix is registered as an independent Pinot table with its own onboarding schedule. Start the wizard again and use the same bucket with a different prefix. *** **The table was created but onboarding hasn't started — what should I check?** Data Portal triggers the first onboarding run automatically after table creation. If onboarding hasn't started, check the table's detail page for an error status and review the error message. You can also trigger a run manually via the [trigger API](../observability#trigger-a-sync-run-optional). # Amazon S3 Tables: Onboarding via API Source: https://docs.startree.ai/corecapabilities/external-table/s3tables/onboarding-api Onboard an Amazon S3 Tables bucket as a StarTree External Table over the REST catalog protocol, with IAM-role, node-role, or static-key auth. > [!WARNING] > > This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it. This page shows how to onboard an **Amazon S3 Tables** External Table with the StarTree controller REST APIs instead of the [Data Portal UI](./onboarding-data-portal). Use it for automation, infrastructure-as-code, or onboarding many tables at once. S3 Tables are Iceberg-compatible tables accessed over the **Iceberg REST protocol** (`catalogType=iceberg-rest`, `serviceType=s3Tables`) with AWS SigV4 authentication. Data files must be **Parquet**. > Using **AWS Glue** instead? See [AWS Glue: Onboarding via API](../glue/onboarding-api). > > Using **Nessie** instead? See [Nessie: Onboarding via API](../nessie/onboarding-api). ## How it works Onboarding is four calls. Each one feeds the next: ```text theme={null} 1. Validate & browse POST /connections/browse → validate; list namespaces/tables 2. Preview POST /tables/preview → infer schema + enriched table config 3. Create schema POST /schemas → register the schema 4. Create table POST /tables → register the External Table ``` The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output. Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding). > [!NOTE] > > All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` — the StarTree Cloud data-plane proxy that fronts the controller. > > If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity. ## Prerequisites * StarTree 0.15.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure. * Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth. * AWS credentials (access key + secret key) with read access to the S3 Tables catalog **and** the underlying S3 data. * The full **table bucket ARN**, e.g. `arn:aws:s3tables:::bucket/`. > [!WARNING] > > **Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly. > > If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details. ## Limitations * **Iceberg tables only** — S3 Tables buckets holding non-Iceberg data aren't reachable through this path. ## Authentication An External Table authenticates in two places, configured independently so the catalog and the data can use different credentials: * **Catalog (REST)** — keys under `catalog.iceberg-rest.auth.rest.*`. Calls to the S3 Tables Iceberg REST endpoint always use AWS SigV4; the signing credentials come from static keys or an assumed IAM role. * **Storage (S3 data files)** — keys under `catalog.iceberg-rest.auth.storage.*`. Choose any one of three methods. The role or keys need `s3:GetObject` and `s3:ListBucket` on the source bucket/prefix, plus S3 Tables read access (`s3tables:GetTable`, `s3tables:GetNamespace`, `s3tables:GetTableData`) on the table bucket. Verify S3 access from the cluster before onboarding: ```bash theme={null} aws s3 ls s3://// ``` ### Catalog (REST) authentication: AWS SigV4 The S3 Tables Iceberg REST endpoint accepts only SigV4-signed requests. The signing credentials can come from **static keys** or an **assumed IAM role** — pick one. #### Static keys `authType` is optional — it's auto-detected as `aws-sigv4` once `accessKeyId`/`secretAccessKey`/`region` are all present. ```json theme={null} { "catalog.iceberg-rest.auth.rest.authType": "aws-sigv4", "catalog.iceberg-rest.auth.rest.accessKeyId": "", "catalog.iceberg-rest.auth.rest.secretAccessKey": "", "catalog.iceberg-rest.auth.rest.region": "", "catalog.iceberg-rest.auth.rest.service": "s3tables" } ``` #### Assumed IAM role SigV4 signing with role-vended credentials — no static secrets on the catalog surface. `auth.rest.region` is **required** when a catalog-side `roleArn` is set (falls back to `auth.storage.region` if that is set instead); static `auth.rest` keys are ignored once a role is configured; and the `AssumeRole` call authenticates with the cluster's ambient AWS identity (node role / IRSA), which the role's trust policy must trust. ```json theme={null} { "catalog.iceberg-rest.auth.rest.roleArn": "arn:aws:iam::123456789012:role/pinot-s3tables-catalog-reader", "catalog.iceberg-rest.auth.rest.externalId": "", "catalog.iceberg-rest.auth.rest.region": "" } ``` > [!NOTE] > > If you set a catalog-side `roleArn` but **no** storage-side `roleArn`, data-file reads through the catalog's default S3 client also assume the catalog role — set `auth.storage.roleArn` explicitly if storage should use a different role. > [!NOTE] > > `catalog.iceberg-rest.auth.rest.service` already defaults to `s3tables` when unset, so it's optional here — unlike Glue, where it must be set explicitly. Setting it anyway keeps the config self-documenting. ### Storage authentication: choose one #### Assumed IAM role (recommended for production) No static secrets in the table config — every S3 read assumes `roleArn` via STS. Three things to know: `auth.storage.region` is **required** whenever a `roleArn` is set (the sync fails fast without it); any static `auth.storage` access keys are **ignored** once a `roleArn` is present; and the `AssumeRole` call itself authenticates with the cluster's ambient AWS identity (node role / IRSA), so the target role's trust policy must trust that identity (plus the `externalId`, if you set one). ```json theme={null} { "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Cluster node role Omit every `auth.storage.*` credential key — only `region` is required. Credential resolution falls back to the AWS SDK default chain (the node's instance profile / IRSA role), which must already have S3 access. ```json theme={null} { "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Static access keys Quick tests, or when role-based access isn't available. Use the canonical `accessKeyId` / `secretAccessKey` spellings (an optional `sessionToken` is also accepted). Avoid the shorter `accessKey`/`secretKey` aliases: only part of the pipeline honors them (segment generation does, Iceberg manifest and data-file reads do not), so a short-alias config falls back to the cluster's ambient credentials for catalog file reads and can fail in confusing ways. ```json theme={null} { "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` *** ## Step 1: Validate and browse the connection **`POST /connections/browse`** There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your credentials and connectivity. Bad credentials, an unreachable host, or an unknown catalog type return an error. * Set `path` to `""` to browse the **root** — this both validates the connection and lists namespaces. * Set `path` to a namespace name to list the **tables** inside it. ### Request | Field | Type | Required | Description | | - | - | - | - | | `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. | | `connection.params` | object | Yes | Catalog-specific connection settings (see example below). | | `path` | string | No | Where to browse. `""` or omitted = root. A namespace name = that namespace's tables. | ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "iceberg-rest", "catalog.iceberg-rest.restUri": "https://s3tables..amazonaws.com/iceberg", "catalog.iceberg-rest.serviceType": "s3Tables", "catalog.iceberg-rest.warehouse": "arn:aws:s3tables::123456789012:bucket/my-table-bucket", "catalog.iceberg-rest.auth.rest.authType": "aws-sigv4", "catalog.iceberg-rest.auth.rest.accessKeyId": "", "catalog.iceberg-rest.auth.rest.secretAccessKey": "", "catalog.iceberg-rest.auth.rest.region": "", "catalog.iceberg-rest.auth.rest.service": "s3tables", "catalog.iceberg-rest.auth.storage.authType": "aws-sigv4", "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } }, "path": "" }' ``` | Key | Value | Description | | - | - | - | | `catalog.iceberg-rest.restUri` | `https://s3tables..amazonaws.com/iceberg` | The S3 Tables Iceberg REST endpoint for your region. | | `catalog.iceberg-rest.serviceType` | `s3Tables` | Selects the built-in S3 Tables adapter. | | `catalog.iceberg-rest.warehouse` | Table bucket ARN | Identifies the S3 Tables bucket. | | `catalog.iceberg-rest.auth.rest.service` | `s3tables` | The SigV4 signing service name. | ### Response ```json theme={null} { "items": [ { "name": "transportation", "type": "NAMESPACE" }, { "name": "nyc_taxi_trips", "type": "TABLE" } ] } ``` | `items[].type` | Meaning | | - | - | | `NAMESPACE` | A namespace you can drill into — pass its `name` as `path`. | | `TABLE` | A selectable table. | *** ## Step 2: Preview the schema **`POST /tables/preview`** Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4. > [!NOTE] > > The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`. ### Request **Top-level fields:** | Field | Type | Required | Description | | - | - | - | - | | `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. | | `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. | | `schema` | object | No | An explicit schema to use instead of inferring one. | **`config.inference` — how the schema is derived:** | Field | Type | Default | Description | | - | - | - | - | | `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Iceberg metadata. **Set this to `true` for External Tables.** | | `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. | Resolution order: explicit `schema` → embedded schema → inferred schema. **`config.sampling` — how rows are sampled:** | Field | Type | Default | Description | | - | - | - | - | | `min` | integer | `1` | Minimum records to sample. | | `max` | integer | `100` | Maximum records to sample. | > [!TIP] > > To list source files without sampling any data, set `config.previewFiles.previewFilesOnly = true`. The response returns matching file URIs in `sourceFiles` and skips schema inference. (OFFLINE only.) ```bash theme={null} curl -X POST "$BASE_URL/tables/preview" \ -H "Content-Type: application/json" \ -d '{ "tableConfig": { "tableName": "nyc_taxi_trips_OFFLINE", "tableType": "OFFLINE", "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "catalogType": "iceberg-rest", "executor": "controller", "inputFormat": "parquet", "catalog.iceberg-rest.restUri": "https://s3tables..amazonaws.com/iceberg", "catalog.iceberg-rest.serviceType": "s3Tables", "catalog.iceberg-rest.warehouse": "arn:aws:s3tables::123456789012:bucket/my-table-bucket", "catalog.iceberg-rest.table.namespace": "transportation", "catalog.iceberg-rest.table.tableName": "nyc_taxi_trips", "catalog.iceberg-rest.auth.rest.authType": "aws-sigv4", "catalog.iceberg-rest.auth.rest.accessKeyId": "", "catalog.iceberg-rest.auth.rest.secretAccessKey": "", "catalog.iceberg-rest.auth.rest.region": "", "catalog.iceberg-rest.auth.rest.service": "s3tables", "catalog.iceberg-rest.auth.storage.authType": "aws-sigv4", "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } } } }, "config": { "inference": { "embeddedSchemaEnabled": true } } }' ``` > [!NOTE] > > **Setting the namespace and table.** Take them from the [browse](#step-1-validate-and-browse-the-connection) response — set `catalog.iceberg-rest.table.namespace` / `.tableName` to the `NAMESPACE` and `TABLE` you selected. > > **Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4. ### Response | Field | Type | Description | | - | - | - | | `schema` | object | The resolved Pinot schema. **Use this in step 3.** | | `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. | | `rows` | array | Sample records after schema + ingestion transforms. | | `sourceRows` | array | Raw records before transformation. | | `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). | Adjust the schema (time column, column names, null handling) before moving on. *** ## Step 3: Create the schema **`POST /schemas`** Send the `schema` object from the preview response. Rename its `schemaName` to match your table. ```bash theme={null} curl -X POST "$BASE_URL/schemas" \ -H "Content-Type: application/json" \ -d @schema.json ``` ```json theme={null} { "status": "nyc_taxi_trips successfully added" } ``` *** ## Step 4: Create the table **`POST /tables`** Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external. ```bash theme={null} curl -X POST "$BASE_URL/tables" \ -H "Content-Type: application/json" \ -d @tableConfig.json ``` ```json theme={null} { "status": "Table nyc_taxi_trips_OFFLINE successfully added" } ``` The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call. > [!TIP] > > The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run: > > ```bash theme={null} > curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE" > ``` *** ## Quickstart: onboard a table end-to-end The whole flow as one script. Fill in the variables at the top, run the script, and the first sync starts automatically. ```bash theme={null} #!/usr/bin/env bash set -euo pipefail BASE_URL="https://dp..cp..startree.cloud/api/pinot" AUTH="Authorization: Bearer " S3TABLES_URI="https://s3tables..amazonaws.com/iceberg" WAREHOUSE="arn:aws:s3tables::123456789012:bucket/my-table-bucket" # table bucket ARN NAMESPACE="transportation" TABLE="nyc_taxi_trips" REGION="" ACCESS_KEY="" SECRET_KEY="" # 1. Validate & browse root — lists namespaces. curl -sf -X POST "$BASE_URL/connections/browse" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"connection\": { \"type\": \"CATALOG\", \"params\": { \"catalogType\": \"iceberg-rest\", \"catalog.iceberg-rest.restUri\": \"$S3TABLES_URI\", \"catalog.iceberg-rest.serviceType\": \"s3Tables\", \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\", \"catalog.iceberg-rest.auth.rest.authType\": \"aws-sigv4\", \"catalog.iceberg-rest.auth.rest.accessKeyId\": \"$ACCESS_KEY\", \"catalog.iceberg-rest.auth.rest.secretAccessKey\": \"$SECRET_KEY\", \"catalog.iceberg-rest.auth.rest.region\": \"$REGION\", \"catalog.iceberg-rest.auth.rest.service\": \"s3tables\", \"catalog.iceberg-rest.auth.storage.authType\": \"aws-sigv4\", \"catalog.iceberg-rest.auth.storage.accessKeyId\": \"$ACCESS_KEY\", \"catalog.iceberg-rest.auth.storage.secretAccessKey\": \"$SECRET_KEY\", \"catalog.iceberg-rest.auth.storage.region\": \"$REGION\" } }, \"path\": \"\" }" | jq . # 2. Preview — infer schema + enriched table config. curl -sf -X POST "$BASE_URL/tables/preview" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"tableConfig\": { \"tableName\": \"${TABLE}_OFFLINE\", \"tableType\": \"OFFLINE\", \"task\": { \"taskTypeConfigsMap\": { \"ExternalTableSyncTask\": { \"catalogType\": \"iceberg-rest\", \"executor\": \"controller\", \"inputFormat\": \"parquet\", \"catalog.iceberg-rest.restUri\": \"$S3TABLES_URI\", \"catalog.iceberg-rest.serviceType\": \"s3Tables\", \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\", \"catalog.iceberg-rest.table.namespace\": \"$NAMESPACE\", \"catalog.iceberg-rest.table.tableName\": \"$TABLE\", \"catalog.iceberg-rest.auth.rest.authType\": \"aws-sigv4\", \"catalog.iceberg-rest.auth.rest.accessKeyId\": \"$ACCESS_KEY\", \"catalog.iceberg-rest.auth.rest.secretAccessKey\": \"$SECRET_KEY\", \"catalog.iceberg-rest.auth.rest.region\": \"$REGION\", \"catalog.iceberg-rest.auth.rest.service\": \"s3tables\", \"catalog.iceberg-rest.auth.storage.authType\": \"aws-sigv4\", \"catalog.iceberg-rest.auth.storage.accessKeyId\": \"$ACCESS_KEY\", \"catalog.iceberg-rest.auth.storage.secretAccessKey\": \"$SECRET_KEY\", \"catalog.iceberg-rest.auth.storage.region\": \"$REGION\" } } } }, \"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } } }" > preview.json # 3. Extract schema and table config from preview response. jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json jq '.tableConfigs.offline' preview.json > tableConfig.json # 4. Create the schema. curl -sf -X POST "$BASE_URL/schemas" \ -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json # 5. Create the table — watcher starts the first sync automatically. curl -sf -X POST "$BASE_URL/tables" \ -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json # 6. (optional) Poll until sync completes. curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \ | jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered' ``` Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable). ## Monitor onboarding Three read-only endpoints report ingestion progress — run status, ingestion checkpoint, and source file count — and require `executor=controller` (set automatically). See [Observability](../observability) for full request and response details. ## Verify it's queryable When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live: ```sql theme={null} SELECT count(*) FROM nyc_taxi_trips; ``` The count will be much larger than the preview's `summary.nSourceRows` (preview only samples up to \~100 rows) — confirm it's non-zero and plausible for your dataset. If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting). *** ## What's next Now that the table is created and data is loading, these are the highest-impact follow-up steps: 1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes) 2. **Enable caching and pin hot indexes.** Set `enable.prefetch.page.cache=true` on the S3 tier, and list small hot indexes in `preload.index.keys.override`, so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching) 3. **Monitor ongoing syncs.** Use the observability endpoints to check run status, ingestion checkpoint, and source file count after each scheduled sync. → [Observability](../observability) *** For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting). # Amazon S3 Tables: Onboarding via Data Portal Source: https://docs.startree.ai/corecapabilities/external-table/s3tables/onboarding-data-portal Connect an Amazon S3 Tables bucket to StarTree as an External Table with the point-and-click Data Portal wizard — no API calls required. > [!WARNING] > > This feature is available starting in **StarTree release 0.15.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment. This guide walks through connecting an **Amazon S3 Tables** bucket to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface. Amazon S3 Tables are Iceberg-compatible tables accessed over the **Iceberg REST protocol** with AWS SigV4 authentication. > Looking for the API-based approach? See [Amazon S3 Tables: Onboarding via API](./onboarding-api). > > Using **AWS Glue** instead? See [AWS Glue: Onboarding via Data Portal](../glue/onboarding-data-portal). *** ## Prerequisites Before starting, ensure you have: * **StarTree 0.15.0 or later** with the external table Beta feature enabled for your environment. * AWS credentials (access key + secret key) with read permissions on both the S3 Tables catalog service and the underlying S3 data. * The full **ARN of your S3 Tables bucket**, e.g. `arn:aws:s3tables:::bucket/`. > [!WARNING] > > **Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly. > > If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details. *** ## Step 1: Open the External Tables 1. Log in to **Data Portal**. 2. In the left navigation, go to **Tables**. 3. Click **+ Connect External Table**. The wizard opens with a connection configuration screen. *** ## Step 2: Configure the S3 Tables Connection Choose **Amazon S3 Tables** as the catalog type. Fill in the credentials and connection details: **Details** | Field | Description | | - | - | | **Connection Name** | A unique name to identify this connection in Data Portal. | **Metastore** — how StarTree authenticates with the S3 Tables REST catalog API. | Field | Description | | - | - | | **Table Bucket ARN** | Full ARN of the S3 Tables bucket, e.g. `arn:aws:s3tables:::bucket/`. | | **Authentication Type** | **Basic Authentication** (access key + secret key), **IAM Node Role** (the cluster node's own IAM identity — Data Portal shows the resolved role as a read-only **IAM Role Name** field so you can grant it access in your AWS account), or **IAM Role** (a cross-account role StarTree assumes). | | **Access Key** / **Secret Key** | With **Basic Authentication**: static AWS credentials with S3 Tables read access. | | **IAM Role ARN** | With **IAM Role**: the ARN of the role to assume. The form shows **Steps to create IAM Policy** — a ready-made trust policy including the StarTree account and External ID to configure on the role. | | **Region** | AWS region where the S3 Tables bucket is located. | **Storage** — credentials for reading the underlying Parquet data files. | Field | Description | | - | - | | **Reuse metastore credentials** | **Yes** reuses the Metastore authentication for S3 data reads (the same three auth types apply). **No** opens a separate set of storage fields. | | **Region** | With **No**: AWS region where the S3 data files are stored. | | **Authentication Type** | With **No**: **Basic Authentication** (Access Key + Secret Key), **IAM Node Role**, or **IAM Role** (IAM Role ARN + trust-policy steps) — same choices as the Metastore section, configured independently. | Click **Validate Connection**. Data Portal calls the catalog's validate endpoint and confirms credentials and connectivity before proceeding. *** ## Step 3: Browse and Select a Table Once the connection is validated: 1. Data Portal lists the available **namespaces** in the S3 Tables bucket. 2. Select a namespace to expand its tables. 3. Click the **table** you want to onboard. Data Portal reads the Iceberg schema and derives a Pinot schema automatically. *** ## Step 4: Review the Schema The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity. Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct. *** ## Step 5: Configure the Table Set the final table options: | Setting | Default | Notes | | - | - | - | | **Table Name** | — | Name of the Pinot table to create. | | **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. | | **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. | The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config. Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required. *** ## Step 6: Monitor Onboarding Once the table is created, onboarding starts automatically. The table detail view shows the status in real time: * **Running** — the task is actively reading Iceberg snapshots and building Pinot segments. * **Completed** — onboarding finished successfully. The last ingested snapshot ID is shown. * **Failed** — onboarding encountered an error. The error message and the number of files discovered vs. segments generated are surfaced to help diagnose the issue. For deeper observability — watcher status, checkpoint values, and per-snapshot file counts — see the [Observability](../observability) page. *** ## Pausing Onboarding To pause scheduled onboarding from Data Portal: 1. Open the table in the **Tables** view. 2. Click **Pause Sync**. This removes the `schedule` (cron trigger) from the table's `ExternalTableSyncTask` config and stashes its value under `schedule.paused`, so the watcher stops firing new runs. Any run currently in progress completes normally, and existing segments and the last checkpoint are untouched. Click **Resume Sync** to restore the schedule from `schedule.paused` — onboarding resumes from where it left off; it does not restart from scratch. > [!NOTE] > > This is a separate mechanism from the `enabled` key documented in [Best Practices & Configs](../best-practices-and-configs#table-config--externaltablesynctask) — that key can also be set to `false` directly via the table config or API to stop new runs, but the Data Portal's **Pause Sync** button acts on `schedule`/`schedule.paused`, not `enabled`. *** ## Frequently Asked Questions **The Validate step fails — what should I check?** * Confirm the access key has `s3tables:GetTableBucket`, `s3tables:ListTableBuckets`, `s3tables:ListNamespaces`, `s3tables:GetNamespace`, `s3tables:ListTables`, `s3tables:GetTable`, `s3tables:GetTableMetadataLocation`, `s3tables:GetTableData`, and `s3:GetObject` permissions (or equivalent). The **Steps to create IAM Policy** panel in the connection form generates the exact policy for you. * Verify the region matches where your S3 Tables bucket lives. Validate parses the region from the Table Bucket ARN and reports a mismatch by name, rather than passing and failing later during preview. * Ensure the Table Bucket ARN is complete and correctly formatted. *** **Can I onboard multiple tables from the same catalog?** Yes. After creating the first table, start the wizard again and reuse the same connection credentials. Each table is registered as an independent Pinot table with its own onboarding schedule. *** **The table was created but onboarding hasn't started — what should I check?** Data Portal triggers the first onboarding run automatically after table creation. If onboarding hasn't started, check the table's detail page for an error status and review the error message. You can also trigger a run manually via the [trigger API](../observability#trigger-a-sync-run-optional). # Large External Tables Source: https://docs.startree.ai/corecapabilities/external-table/segment-groups Use segment groups to keep cluster metadata manageable for External Tables backed by millions of Parquet files. > [!WARNING] > > This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it. A **large External Table** is one backed by a million source files or more. This page explains why tables of that size need a different metadata layout, and how to enable **segment groups** to provide it. An External Table sync run creates one Pinot segment per source data file. A table backed by millions of Parquet files therefore produces millions of segments, and every segment gets its own ZooKeeper znode plus its own entry in the table's ideal state and external view. On tables with millions of segments, such large metadata creates problems: Helix state transitions get slower and controller and broker heap grows. **Segment groups** make the cluster track a batch of files as one unit. A group of member segments gets a single ideal-state entry, a single external-view entry, and a single znode that controllers and brokers read, instead of one of each per file. With the default group size of 25, a one-million-file table stores 40,000 entries instead of 1,000,000. > [!NOTE] > > Query results and latency are unchanged, and servers still scan exactly the segments a query needs — a server expands each group it is routed and prunes member segments individually. See [Query behavior](#query-behavior) for more details. ## When segment groups help Segment groups are built for tables with **a million files or more**, where per-file metadata alone can destabilize the cluster. Consider them from a few hundred thousand files upwards if the table is growing towards that size. Below that, per-file segments are fine: grouping couples member availability (see [Caveats](#caveats)) in exchange for a metadata saving you don't need. ## What's supported | Aspect | Support | | :- | :- | | **Table type** | OFFLINE External Tables only. | | **Sync mode** | Sync must run with `executor=controller` — see [Catalog & mode](./best-practices-and-configs#catalog--mode). The minion-distributed sync path does not support grouping. | | **Cluster gate** | Segment groups must be activated for the cluster before any table can use them. Contact StarTree support. | | **Not supported** | Upsert, dedup, and dimension tables — the controller rejects the config for these. | | **Minion tasks** | A segment-group table's `taskConfig` may declare only `ExternalTableSyncTask` with `executor=controller` (its controller-side ingestion writes segment groups directly). Any other minion task type is rejected at create and update time — including on an existing table, with no grandfathering; drop one side of the config before the update will succeed. | | **Real-time tables** | Not in this release. Grouping for real-time tables is planned for a future release. | | **Existing data** | New segments will be grouped and existing segments won't be grouped. See [Enabling on a table that already has data](#enabling-on-a-table-that-already-has-data). | ## Enable segment groups Set `segmentGroupConfig` as a JSON string under the table's `metadata.customConfigs`: ```json theme={null} { "tableName": "orders_ext_OFFLINE", "tableType": "OFFLINE", "metadata": { "customConfigs": { "segmentGroupConfig": "{\"enabled\":true,\"maxSegmentsPerGroup\":25,\"timeBucket\":\"1d\"}" } } } ``` | Field | Default | Description | | :- | :- | :- | | `enabled` | `false` | Opt this table into segment grouping. Rejected by controller validation unless segment groups are activated for the cluster and the table is not an upsert, dedup, or dimension table. | | `maxSegmentsPerGroup` | `25` | Maximum number of member segments per group. | | `minSegmentsPerGroup` | `25` | Not used by External Table grouping, but validated: must be `>= 1` and `<= maxSegmentsPerGroup`. If you lower `maxSegmentsPerGroup` below 25, lower this too or the table config is rejected. | | `timeBucket` | *(none)* | Period (`"1h"`, `"6h"`, `"1d"`, `"7d"`) that additionally splits members by segment start time, so one group never spans more than one bucket. Improves broker-side time pruning on time-filtered queries. Accepts `d`, `h`, `m`, `s` units and combinations such as `"4h30m"`; `"1w"` is rejected. Tables without a time column bucket by segment creation time instead. | > [!NOTE] > > **A new table is not queryable until its first snapshot completes.** Grouped tables are always queried against a pinned, fully ingested snapshot (as are all Iceberg External Tables, via `enableSnapshotConsistency`). Until the first sync finishes, queries fail with `No active Iceberg snapshots are available for table …`. ### Choosing a group size Bigger groups save more metadata; smaller groups keep the cluster more granular. The trade-offs, in the order they usually matter: * **Metadata saving** scales directly with group size: entries drop by roughly the group size. * **Availability blast radius** grows with group size. A group is queryable only once every member has loaded, so one unreadable file keeps its whole group out of query results rather than just itself. * **Broker pruning** gets coarser. The broker prunes using the group's aggregate time range — the minimum start and maximum end across its members — so a group spanning a wide time range is harder to prune away. Set `timeBucket` to bound the span if your queries filter on time. * **Per-group observability** — some query statistics are reported per group rather than per member (see [Observability](#observability)), so a larger group makes them coarser. Start with the default of 25 and change it only with a specific reason; StarTree support can help size it for your table. ## Enabling on a table that already has data New files form groups; segments that already exist stay standalone and are never regrouped. > [!WARNING] > > **Enable grouping only at a snapshot boundary — never mid-snapshot.** Follow the [enablement sequence](#enabling-grouping-on-a-live-table) below. ### Check whether the table is between snapshots A table is at a snapshot boundary when its last snapshot is **fully** ingested and no run is in flight. Two read-only calls tell you: `GET /tables/{tableNameWithType}/externalTable/checkpoint` — the ingestion watermark. Read `state`: | `state` | Meaning | Safe to enable? | | :- | :- | :- | | `DONE` | The last snapshot was ingested completely; `pageCursor` is `null`. | Yes | | `IN_PROGRESS` | The snapshot is partly ingested; `pageCursor` marks where the last run stopped. | No — finish the snapshot first | | `FAILED` | Ingestion stopped after `iceberg.snapshotProcessing.maxRetries` consecutive failures. The response's own `errorMessage` and `consecutiveFailures` say why; `failurePhase` on `…/externalTable/status` says where. | No — resolve the failure first | | all fields `null` | The table has never synced. | Yes | `GET /tables/{tableNameWithType}/externalTable/status` — `fileOnboardingRun.status` must not be `RUNNING`. > [!WARNING] > > A run reporting `COMPLETED` does **not** mean the snapshot is complete. A run finishes after each batch of files, and a large snapshot takes many runs. `state: DONE` on the checkpoint is the only signal that the snapshot itself is fully ingested. ### Enabling grouping on a live table 1. **Pause the sync schedule.** In the Data Portal, use **Pause Sync** on the table's details page; through the API, remove `schedule` from the table's `ExternalTableSyncTask` config (the Data Portal stashes the old value under `schedule.paused`). This stops only the automatic cadence — on-demand runs still work while paused, and already-ingested data is untouched. 2. **Let the in-flight run finish.** Poll `…/externalTable/status` until `fileOnboardingRun.status` is no longer `RUNNING`. 3. **Get to a snapshot boundary.** Check `…/externalTable/checkpoint`. If `state` is `IN_PROGRESS`, trigger a run and re-check, repeating until `state` is `DONE`: ```bash theme={null} curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=orders_ext_OFFLINE" curl "$BASE_URL/tables/orders_ext_OFFLINE/externalTable/checkpoint" ``` 4. **Enable grouping.** Add `segmentGroupConfig` to the table config as shown [above](#enable-segment-groups). 5. **Restore the schedule.** **Resume Sync** in the Data Portal, or put the `schedule` value back. The next run starts a fresh snapshot with grouping active. ### Grouping data that was already ingested **If you leave it alone.** Every new file on the Iceberg table forms groups, so the grouped share of the table grows with each run. The ungrouped share drains too, but only as the source table loses files: when an upstream delete or compaction removes a file, its standalone segment stops appearing in new snapshots, and snapshot retention deletes it once the last snapshot referencing it is evicted. On a table whose data ages out or gets compacted upstream, the ungrouped remainder therefore disappears on its own over time. **If you need the whole table grouped now**, it has to be ingested again as a new table. ## How groups form Grouping happens during the normal External Table sync run. Within a run, new member segments are bucketed by partition, tier, and (if configured) time bucket. Any bucket that has accumulated a full `maxSegmentsPerGroup` worth of members is committed as a group. Members left over in a bucket are carried forward to the next page of the same snapshot; whatever remains when the snapshot finalizes is committed as one smaller group per bucket. No member ever becomes a standalone segment. Group names are content-addressed (`group_
___`), so a group whose file set changes in a later snapshot appears under a new name and the old one is retired by [snapshot retention](./snapshot-retention). ## Caveats ### A group is available only as a whole Servers load a group's members together, and the group goes ONLINE only when all of them succeed. If one member fails to load, the members already loaded in that attempt are unloaded and the group goes into ERROR. Recovery is to fix the underlying file or segment and reset the group; reset works at group granularity, not per member. ## Query behavior Your queries don't change: same SQL, same schema, same results. What changes underneath is where pruning happens. **Pruning happens at two levels.** The broker prunes whole groups using their aggregate metadata, which is coarser than per-file pruning. The server then expands the groups it was routed and prunes member segments individually before scanning, so a query still only reads the files it needs. ## Observability Grouping changes what shows up where. The control plane reports groups; servers report members. | Where | What you see | | :- | :- | | Ideal state, external view, controller segment listings, the table UI | One entry per group, named with a `group_` prefix so groups are recognizable at a glance. Ungrouped segments (including everything ingested before grouping was enabled) appear as usual. | | A group's ZK metadata | Group-level aggregates — summed document count and size, the widest time range across members — plus `total.segments.count` in its custom map, which is how many members the group has. | | Server metrics | Counted per **member**: `SEGMENT_COUNT` reflects member segments, not groups. | | Query statistics | Mixed. `numSegmentsQueried` counts **groups** (what the broker routed to the server). `numSegmentsPrunedByServer`, `numSegmentsProcessed`, `numSegmentsMatched`, and `totalDocs` count **members**, since the server expands each group before pruning. Pruned and processed counts are therefore legitimately much larger than queried. | | Server segment-metadata endpoints | Given a group name, the response is aggregated across the group's members. | | Member segment names | Not addressable through the controller or through the single-segment metadata endpoint — a member has no standalone metadata entry, so a lookup by member name returns not-found. Query the group instead. | External Table sync status, ingestion lag, and the file-listing endpoints are unaffected — see [Observability](./observability). > [!TIP] > > The mismatch to expect: a table showing 40,000 segments in the UI and `numSegmentsQueried` in the low thousands, while `numSegmentsProcessed` on the same query is in the hundreds of thousands. All three are right — the first two count groups, the last counts member segments. ## Turn it off Setting `segmentGroupConfig.enabled` back to `false` is accepted at any time, and is harmless on a table that has not formed any groups yet — it simply reverts the table to ordinary per-file segments. On a table that **already has groups**, the flip is not a no-op. The flag does not un-group anything; instead, the next sync run re-ingests the whole table: * The ungrouped sync path recognizes already-ingested files by their standalone segment names in the ideal state. Member segments are never in the ideal state, so **every file that was inside a group is treated as new** and gets a fresh standalone segment generated and uploaded. On a million-file table that is a full re-ingest, paged across many runs. * Queries stay pinned to the last completed (grouped) snapshot while this happens, so results remain correct and complete. Once the first ungrouped snapshot completes, the broker routes to the new standalone segments only, and [snapshot retention](./snapshot-retention) deletes the old groups after their last referencing snapshot is evicted. > [!WARNING] > > **Disabling grouping on a grouped table costs a full re-ingest.** Plan it like a new onboarding — the same download, segment-generation, and upload volume as the initial sync — and pause the schedule at a snapshot boundary first, exactly as when [enabling](#enabling-grouping-on-a-live-table). If the table is large, involve StarTree support before flipping the flag. ## FAQs ### Do I need to change how I query the table? No. Segment groups are a metadata-layer optimization: SQL, schema, and results are unchanged. ### I enabled grouping but my existing segments are still ungrouped. Why? Because the table had already completed a sync. Files listed in that snapshot's manifest are recognized as already ingested and keep their existing standalone segments — only files that arrive afterwards form groups. See [Enabling on a table that already has data](#enabling-on-a-table-that-already-has-data). ### My group is in ERROR. What now? A group goes ERROR when one of its members fails to load — the group is never partially served. Fix the underlying file or segment, then reset the group by its group name. Members cannot be reset individually. ### Why was my table rejected when I enabled segment groups? One of three reasons: segment groups are not activated for your cluster; the table is an upsert, dedup, or dimension table — unsupported in this release; or `taskConfig` declares a minion task other than `ExternalTableSyncTask` with `executor=controller`. The validation error names the reason. ### Can I use segment groups on a real-time table? Not in this release. Support for real-time tables is planned for a future release. # Support Matrix Source: https://docs.startree.ai/corecapabilities/external-table/support-matrix What StarTree External Tables support: catalogs and sources, table and file formats, authentication methods for every connector, features, indexes, and limits. > [!WARNING] > > External Tables are available starting in **StarTree release 0.15.0** and must be enabled on demand — contact your StarTree representative to have the feature activated for your environment. Rows marked **0.16.0** require that release or later. This page summarizes what External Tables support today, in one place. Legend: * ✅ Supported — includes capabilities that work under documented conditions or that StarTree enables on request; conditions are noted per row * ⚠️ Preview — early access, not yet generally available * ❌ Not supported by StarTree * — Not applicable: the source, service, or format does not offer this capability, so there is nothing for StarTree to support *** ## Catalogs and sources Every catalog source is reached through a single connector, the Iceberg REST protocol (`catalogType=iceberg-rest`), with a `serviceType` selecting the catalog-specific adapter. Raw-file sources list Parquet files directly from object storage without a catalog. | Source | `catalogType` / `serviceType` | Table format | Storage | Onboarding | Since | | - | - | - | - | - | - | | [AWS Glue](/corecapabilities/external-table/glue/onboarding-data-portal) | `iceberg-rest` / `glue` | Iceberg | Amazon S3 | Data Portal, [API](/corecapabilities/external-table/glue/onboarding-api) | 0.15.0 | | [Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-data-portal) | `iceberg-rest` / `s3Tables` | Iceberg | Amazon S3 | Data Portal, [API](/corecapabilities/external-table/s3tables/onboarding-api) | 0.15.0 | | [Unity Catalog](/corecapabilities/external-table/unity/onboarding-api) (Databricks-managed or OSS) | `iceberg-rest` / `unity` | Iceberg; Delta with [UniForm](#table-and-file-formats) | Amazon S3 (vended credentials); Azure Blob/ADLS via [SQL DDL](./sql-ddl#azure-source-storage) only (vended or configured SAS) | [Data Portal](/corecapabilities/external-table/unity/onboarding-data-portal), [API](/corecapabilities/external-table/unity/onboarding-api) | 0.16.0 | | [Apache Polaris & Snowflake Open Catalog](/corecapabilities/external-table/polaris/onboarding-data-portal) | `iceberg-rest` / `rest` | Iceberg | Amazon S3 | [Data Portal](/corecapabilities/external-table/polaris/onboarding-data-portal), [API](/corecapabilities/external-table/polaris/onboarding-api) | 0.16.0 | | [Snowflake Horizon Catalog](/corecapabilities/external-table/horizon/onboarding-data-portal) | `iceberg-rest` / `rest` | Iceberg (Snowflake-managed) | Amazon S3 | [Data Portal](/corecapabilities/external-table/horizon/onboarding-data-portal), [API](/corecapabilities/external-table/horizon/onboarding-api) | 0.16.0 | | [Generic Iceberg REST catalog](/corecapabilities/external-table/getting-started/connect-a-catalog) (Nessie, Tabular, ...) | `iceberg-rest` / `rest` | Iceberg | S3-compatible | API | 0.16.0 | | [S3 Data Lake](/corecapabilities/external-table/s3/onboarding-data-portal) (no catalog) | `s3` | Raw Parquet files under a prefix | Amazon S3 | Data Portal, [API](/corecapabilities/external-table/s3/onboarding-api) | 0.15.0 | | [GCS Data Lake](/corecapabilities/external-table/gcs/onboarding-api) (no catalog) | `gcs-interop` | Raw Parquet files under a prefix | Google Cloud Storage (S3-interop endpoint) | [Data Portal](/corecapabilities/external-table/gcs/onboarding-data-portal), [API](/corecapabilities/external-table/gcs/onboarding-api) | 0.16.0 | Notes: * For Nessie, address a branch or ref by setting `catalog.iceberg-rest.prefix` (for example `main`); unset means the server default branch. * Azure Blob Storage and ADLS are supported as a source only through [SQL DDL](./sql-ddl#azure-source-storage) (`storage.provider = 'azure'`, with a configured SAS token or — for Unity — vended credentials), for Hive Metastore and Unity Catalog / generic Iceberg REST catalogs. Not yet exposed in the Data Portal or REST onboarding flow. AWS Glue and Amazon S3 Tables remain Amazon S3 only. * `CREATE TABLE` over SQL DDL is available as a preview for Iceberg REST sources (serviceTypes `rest`, `glue`, `s3tables`); contact StarTree to enable it. *** ## Table and file formats | Format | Status | Notes | | - | - | - | | Apache Iceberg v1 | ✅ | Snapshot-lag `snapshotsBehind` is unavailable (`null`) on v1 metadata. | | Apache Iceberg v2 | ✅ | Full snapshot-based ingestion and lag reporting. | | Iceberg v2/v3 row deletes (deletion vectors) | ⚠️ Preview | Row-level deletes and updates reflected at query time — v2 equality and position deletes, and v3 Puffin position deletes. Contact StarTree to enable. | | Delta Lake via UniForm | ✅ | Through [Unity Catalog](/corecapabilities/external-table/unity/onboarding-api) only. The table must have UniForm enabled (`delta.universalFormat.enabledFormats='iceberg'`). Tables with `delta.columnMapping.mode` set to `name` or `id` — including nested struct/map/list columns and renamed time columns — are supported in current releases: physical `col-` Parquet names are resolved through Iceberg field IDs across preview, segment generation, and null-vector building. On older 0.16.0 builds these tables failed segment load — confirm your release with StarTree support if unsure. | | Delta Lake (native log) | ❌ | No direct Delta log reader. | | Apache Hudi | ❌ | | | Apache Paimon | ❌ | | **Data files:** Parquet only, for every source. CSV, JSON, Avro, and ORC data files are not supported. Recommended file size is 128 MB to 512 MB; see [Operational Guidance](/corecapabilities/external-table/operational-guidance). ### Row-level deletes (Iceberg) How each Iceberg delete mechanism is handled: | Delete mechanism | Status | Notes | | - | - | - | | Copy-on-write (deletes rewrite data files) | ✅ | Each sync materializes the new snapshot, so rewritten files are picked up. Append-only sources are fully handled; for heavily mutating or compacted sources, confirm current reconciliation behavior with StarTree. | | Merge-on-read: Iceberg v3 deletion vectors (Puffin) | ⚠️ Preview | Position deletes encoded as deletion vectors are applied at query time. Contact StarTree to enable. | | Merge-on-read: Iceberg v2 position delete files | ⚠️ Preview | Materialized into a per-file delete bitmap during sync and applied at query time, same as v3. Requires `enableDeletionVectors=true` and every controller/server upgraded first. Contact StarTree to enable. | | Merge-on-read: Iceberg v2 equality deletes | ⚠️ Preview | Materialized the same way as v2 position deletes; equality columns resolve by Iceberg field ID, including renamed/dropped columns. Commonly produced by Flink CDC writers. Requires `enableDeletionVectors=true` and every controller/server upgraded first. Contact StarTree to enable. | *** ## Authentication and authorization Iceberg REST sources authenticate on two independent surfaces: the **catalog surface** (the REST API that serves table metadata, keys under `catalog.iceberg-rest.auth.rest.*`) and the **storage surface** (reading Parquet data files from object storage, keys under `catalog.iceberg-rest.auth.storage.*`). Raw S3 and GCS sources have a single combined surface (`catalog.s3.*` / `catalog.gcs-interop.*`). ### Catalog (REST) surface | Method | AWS Glue | S3 Tables | Unity Catalog | Generic REST | | - | - | - | - | - | | AWS SigV4 (access key + secret + region) | ✅ Required¹ | ✅ Required¹ | — | — | | AWS SigV4 via assumed IAM role (`roleArn`) | ✅ | ✅ | — | — | | Bearer token / PAT | — | — | ✅ Databricks PAT | ✅ | | OAuth2 client credentials | — | — | ✅ Recommended (0.16.0) | ✅ | | Unauthenticated | — | — | ✅ OSS Unity default | ✅ Self-hosted default | ¹ SigV4 signing is required by the service; the signing credentials can come from static keys, an assumed IAM role, or the cluster node role. * The dashes reflect the services themselves: AWS Glue and S3 Tables Iceberg REST endpoints accept only SigV4-signed requests, while Unity Catalog and spec-compliant REST catalogs use token or OAuth2 auth and do not speak SigV4. * **Glue:** `catalog.iceberg-rest.auth.rest.service` must be set to `glue` explicitly. It defaults to `s3tables`, which makes Glue reject the SigV4 signature. * **Assumed IAM role on the catalog surface:** set `catalog.iceberg-rest.auth.rest.roleArn` (plus `.externalId` if the trust policy requires it). `catalog.iceberg-rest.auth.rest.region` is **required** whenever a catalog-side `roleArn` is set; when a role is configured, static keys on that surface are ignored. The role is assumed from the cluster's ambient identity (IRSA / instance profile). Available on the controller-executor sync path only. * **OAuth2** uses the RFC 6749 client-credentials grant: `oauthTokenUri`, `oauthClientId`, `oauthClientSecret`, and optional `oauthScopes`. Tokens are cached and refreshed automatically. * Raw S3 and GCS sources have no catalog surface. ### Storage (data file) surface | Method | AWS Glue | S3 Tables | Unity Catalog | Generic REST | S3 Data Lake | GCS Data Lake | | - | - | - | - | - | - | - | | Catalog-vended credentials | ❌ | ❌ | ✅ Default | ❌ | — | — | | Assumed IAM role (`roleArn` + `externalId`) | ✅ Recommended | ✅ Recommended | ✅ Override only | ✅ | ✅ Recommended | — | | Cluster node role (AWS default chain) | ✅ | ✅ | ❌ | ✅ | ✅ | — | | Static access keys | ✅ | ✅ | ✅ Override only | ✅ | ✅ | — | | GCS HMAC keys (inline) | — | — | — | — | — | ✅ Required | | GCS HMAC keys via GCP Secret Manager | — | — | — | — | — | ✅ | * **Assumed IAM role (storage):** `auth.storage.region` is required whenever a storage `roleArn` is set — the sync fails fast without it. Static storage keys are ignored once a role is configured, and the STS `AssumeRole` call authenticates with the cluster's ambient AWS identity (node role / IRSA), which the role's trust policy must trust. * **Unity vends short-lived S3 credentials per table load by default.** Only `catalog.iceberg-rest.auth.storage.region` is needed, and it is required even with vended credentials. Static keys or an assumed role can be configured but are ignored whenever vending succeeds. * **GCS accepts HMAC keys only.** There is no IAM-role or workload-identity option. With `keyType=SECRET`, the `accessKey`/`secretKey` values are Secret Manager secret names, resolved using `secretmanagertype=GCS`, `gcpprojectid`, and `gcpkeypath`. Every GCS connection must also set `disable.integrity.protections="true"` and `endpoint=https://storage.googleapis.com`. * **Cluster node role** means omitting all credential keys so the AWS SDK default chain (instance profile, IRSA) is used. ### Connection-test storage validation Test Connection validates credentials (`sts:GetCallerIdentity`, or `sts:AssumeRole` for an assumed role) for every source below. It also compares the configured storage region against the real bucket region — but only where a bucket is known before a table is selected: | Source | Region checked at Test Connection? | | - | - | | S3 Data Lake | ✅ (`bucketName` is explicit) | | S3 Tables | ✅ (region is parsed from the table-bucket ARN, no AWS call needed) | | AWS Glue, Unity Catalog, Apache Polaris, Snowflake Open Catalog / Horizon, Generic Iceberg REST | Region check runs later, against the selected table's location, at preview or `CREATE TABLE` — no bucket exists to check until a table is picked | | GCS Data Lake | Not checked — region is `auto`; browse already proves bucket access | Set `catalog.{catalogType}.validateStorage=false` to opt a connection out of these checks. ### Configuration keys by method | Method | Keys | | - | - | | AWS SigV4 (catalog) | `catalog.iceberg-rest.auth.rest.authType=aws-sigv4` (auto-detected), `.accessKeyId`, `.secretAccessKey`, `.region`, `.service` (`glue` or `s3tables`), `.sessionToken` (optional) | | Assumed IAM role (catalog, SigV4 sources) | `catalog.iceberg-rest.auth.rest.roleArn`, `.externalId` (optional), `.region` (required with `roleArn`) | | Bearer token (catalog) | `catalog.iceberg-rest.auth.rest.token` | | OAuth2 (catalog) | `catalog.iceberg-rest.auth.rest.authType=oauth2`, `.oauthTokenUri`, `.oauthClientId`, `.oauthClientSecret`, `.oauthScopes` (optional) | | Assumed IAM role (storage) | `catalog.iceberg-rest.auth.storage.roleArn`, `.externalId`, `.region` | | Static keys (storage) | `catalog.iceberg-rest.auth.storage.accessKeyId`, `.secretAccessKey`, `.region` | | S3 Data Lake | `catalog.s3.bucketName`, `.prefix`, `.region`, then one of: `.roleArn` + `.externalId`, or `.accessKey` + `.secretKey`, or nothing (node role) | | GCS Data Lake | `catalog.gcs-interop.bucketName`, `.prefix`, `.region="auto"`, `.endpoint`, `.accessKey`, `.secretKey`, `disable.integrity.protections="true"`; Secret Manager mode adds `.keyType=SECRET`, `.secretmanagertype=GCS`, `.gcpprojectid`, `.gcpkeypath` | ### Minimum permissions per source | Source | Catalog permissions | Storage permissions | | - | - | - | | AWS Glue | `glue:GetCatalog`, `glue:GetDatabase(s)`, `glue:GetTable(s)`, `glue:GetPartition(s)`, `glue:BatchGetPartition`, `lakeformation:GetDataAccess`² | `s3:GetObject`, `s3:ListBucket` | | S3 Tables | `s3tables:GetTableBucket`, `s3tables:ListTableBuckets`, `s3tables:ListNamespaces`, `s3tables:GetNamespace`, `s3tables:ListTables`, `s3tables:GetTable`, `s3tables:GetTableMetadataLocation`, `s3tables:GetTableData` | `s3:GetObject`, `s3:ListBucket` | | Unity Catalog | `EXTERNAL USE SCHEMA` on the schema (plus catalog/schema `USE`) | vended by Unity | | S3 Data Lake | — | `s3:GetObject`, `s3:ListBucket` | | GCS Data Lake | — | `storage.objects.get`, `storage.objects.list` | ² Only needed when the Glue database is registered with Lake Formation permissions; the Data Portal's generated IAM policy always includes it, so it is a no-op otherwise. *** ## Feature support Each supported source has its own column. AWS Glue, S3 Tables, Unity Catalog, and generic Iceberg REST catalogs (the column that also covers Apache Polaris, Snowflake Open Catalog, and Snowflake Horizon) all connect through the same `iceberg-rest` connector, so they share the snapshot-driven capabilities; the Data Lake sources list files directly and have no snapshot machinery. | Feature | AWS Glue | S3 Tables | Unity Catalog | Generic Iceberg REST | S3 Data Lake | GCS Data Lake | Notes | | - | - | - | - | - | - | - | - | | Scheduled incremental sync | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Quartz cron `schedule`; Data Portal default every 5 minutes. | | Checkpointing (resume from watermark) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Watermark visible as `checkpointValue` in the [status API](/corecapabilities/external-table/observability#run-status). | | Snapshot-based ingestion | ✅ | ✅ | ✅ | ✅ | — | — | Ingests snapshot deltas; `initialVersionSelector` = `LATEST`, `EARLIEST`, or `SNAPSHOT_ID`. Raw listings have no snapshot concept. | | Resilience to upstream compaction / file rewrites | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | REST catalogs track the live file set via snapshot metadata, so compaction is reconciled on the next sync. Raw Data Lake sources pin segments to physical file paths — when the source compacts and deletes files, those segments fail to load (`FileNotFoundException`). Use raw sources for append-only data only. See [Troubleshooting](/corecapabilities/external-table/troubleshooting#segments-fail-to-load-or-serve-after-upstream-compaction). | | Backfill start cursor | — | — | — | — | ✅ | ✅ | `catalog.s3.startAfter` seeds the first listing (0.16.0). | | Schema evolution (opt-in) | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | — | — | `schemaEvolution.enabled=true`; unions the source schema using Iceberg field-id history, which raw Parquet files do not carry. | | Nested logical names under Delta column mapping (default on, opt-out) | ✅ | ✅ | ✅ | ✅ | — | — | `storeNestedSchema.enabled` resolves field names inside `struct` / `array` / `map<_,struct>` columns to logical names; set it to `false` to opt out. Relevant only for Delta UniForm tables with `delta.columnMapping.mode` set; top-level names are always resolved. See [Unity Catalog — Schema options for column mapping](/corecapabilities/external-table/unity/onboarding-api#schema-options-for-column-mapping). | | Column renames | ✅ | ✅ | ✅ | ✅ | — | — | Top-level fields only, tracked as aliases so the Pinot column name is preserved; requires schema evolution enabled. Raw sources need a manual schema change. | | Type widening | ⚠️ Opt-in (`INT`→`LONG`, `FLOAT`→`DOUBLE`) | ⚠️ Opt-in (`INT`→`LONG`, `FLOAT`→`DOUBLE`) | ⚠️ Opt-in (`INT`→`LONG`, `FLOAT`→`DOUBLE`) | ⚠️ Opt-in (`INT`→`LONG`, `FLOAT`→`DOUBLE`) | — | — | `schemaEvolution.typeWidening.enabled=true`, plus `schemaEvolution.enabled=true`; periodic-watcher path only (not snapshot-sync). Every other widening is still rejected (a decimal precision change needs no flag — see [Best Practices & Configs](./best-practices-and-configs#checkpointing--schema-evolution)); `schemaEvolution.failFast` controls abort vs log-and-continue. | | Time travel / snapshot pinning | ✅ | ✅ | ✅ | ✅ | — | — | Query option `snapshotVersionByTable`. | | Row-level deletes and updates (deletion vectors) | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | — | — | Iceberg v3 Puffin position deletes and v2 Parquet equality/position deletes, materialized into the same per-file delete index. `enableDeletionVectors` always defaults to `false` and is never auto-enabled by onboarding. Contact StarTree to enable. Incompatible with the Star-tree index — table-config validation rejects the combination either way it's introduced. See [Deletion Vectors](./deletion-vectors). | | Ingestion lag reporting (`includeLag`) | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | ✅ 0.16.0 | — | — | [Status API](/corecapabilities/external-table/observability#ingestion-lag); requires `executor=controller`. Raw sources return `lag: null` (snapshot lag is undefined for listings). | | Snapshot retention (automatic cleanup) | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | ⚠️ Preview | Keeps the newest N snapshot versions and sweeps orphaned artifacts. Contact StarTree to enable. | | Segment groups | ⚠️ Preview (0.16.0) | ⚠️ Preview (0.16.0) | ⚠️ Preview (0.16.0) | ⚠️ Preview (0.16.0) | ❌ | ❌ | Groups many small segments into one logical unit; ingestion-time grouping requires `executor=controller`. Not supported for upsert, dedup, or dimension tables. | | Data and index caching | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [Parquet data cache, index cache, footer cache](/corecapabilities/external-table/data-and-index-caching). | | Pinot indexes | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | See [Supported Indexes](/corecapabilities/external-table/indexes); every column must use `RAW` encoding. Only **sorted** and **geospatial (H3)** indexes are rejected; vector (HNSW) is supported when the index sets `storeInSegmentFile: true`. | | Pause / resume sync | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Data Portal controls or table config; already-ingested data is untouched. | | Manual sync trigger | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | `POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=
_OFFLINE` or Data Portal **Sync Now**. | | Single-stage and multi-stage query engines | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | Federated queries with real-time tables | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | One query spanning Kafka-fed recent data and lake history. | | Writes back to the source | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | External Tables are read-only. Pinot writes to the lake through the separate Data Export Task; see [Writing to the lake](#writing-to-the-lake-data-export-task) below. | *** *** ## Writing to the lake (Data Export Task) External Tables never write to their source. The write path in the other direction is the [Data Export Task](/corecapabilities/external-table/data-export-task), a minion task that exports completed **REALTIME segments** to the lake as Parquet. It is the complement of External Table sync, used for cold-tier archival, feeding downstream Iceberg catalogs, and cross-system sharing. It requires a minion tier and is enabled on demand by StarTree. ### Export destinations | Destination | `dataExport.target` | Status | Notes | | - | - | - | - | | Amazon S3 | `filesystem` | ✅ | `S3PinotFS`; files written as `/
/.parquet`. | | S3-compatible storage (MinIO, ...) | `filesystem` | ✅ | `output.fs.prop.endpoint` override. | | Google Cloud Storage | `filesystem` | ✅ | Native `GcsPinotFS` with a `gs://` URI; no S3-interop needed on the export path. | | HDFS | `filesystem` | ✅ | Via PinotFS. | | AWS Glue (Iceberg REST) | `iceberg-rest` | ✅ | Target table must already exist, must not be partitioned, and must use Parquet. | | Amazon S3 Tables (Iceberg REST) | `iceberg-rest` | ✅ | Same target-table conditions as Glue. | | Generic Iceberg REST catalog | `iceberg-rest` | ✅ | Same target-table conditions. | ### Export authentication | Destination | Methods | | - | - | | S3 filesystem | Static keys (`output.fs.prop.accessKey` / `.secretKey`) or assumed IAM role (`output.fs.prop.roleArn` + `.externalId`). | | GCS filesystem | GCP service account JSON key (`output.fs.prop.projectId`, `.jsonKey`). Unlike ingestion, export to GCS does not use HMAC keys. | | Iceberg REST catalog | Bearer token (`catalog.iceberg-rest.token`), OAuth2 client credential (`catalog.iceberg-rest.credential` as `clientId:clientSecret`), or AWS SigV4 (`catalog.iceberg-rest.rest.signing-region`). AWS Glue and S3 Tables vend short-lived S3 credentials for file I/O automatically; static `s3.*` keys are only for overriding them. | Export facts worth knowing: * **Source: REALTIME tables only.** `sourceTableName` must carry the `_REALTIME` suffix; External (OFFLINE) tables cannot be exported. * **Output format: Parquet only**, with `SNAPPY` (default), `GZIP`, `ZSTD`, or `UNCOMPRESSED` compression. * **Iceberg commits are batched**: staged files are committed as one Iceberg snapshot per batch (`iceberg.commitThreshold`, default 250 files), keeping snapshot churn low on high-throughput tables. *** See [Data Type Mapping](/corecapabilities/external-table/data-type-mapping) for how source types map to Pinot types, [Operational Guidance](/corecapabilities/external-table/operational-guidance) for sizing advice and capacity guardrails, and [Troubleshooting](/corecapabilities/external-table/troubleshooting) for diagnosis guides. # Troubleshooting Source: https://docs.startree.ai/corecapabilities/external-table/troubleshooting Symptom-based fixes for common External Table onboarding, schema, query, and sync errors, indexed by the exact error string to search for. > [!WARNING] > > This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it. Common issues when onboarding and querying External Tables, grouped by symptom. Each entry lists the likely cause and the fix. If an issue isn't covered here, reach out to StarTree support. > [!TIP] > > In the Data Portal, an External Table failure carries a `DPE-####` code, with steps for the type of connection you use. Look it up in [Data Portal error codes](/corecapabilities/observability/error-codes#connections-and-data-sources). This page covers the raw errors that the API and the cluster return. > [!NOTE] > > **Quick lookup — find your error:** > > | Error message or symptom | Jump to | > | - | - | > | AWS region error / `s3://` in bucket field | [Onboarding fails with a region error](#onboarding-fails-with-a-region-error) | > | `Access Denied` / HTTP 403 | [Access Denied when creating the table](#access-denied-http-403-when-creating-the-table) | > | Preview succeeds but no data / wrong files | [Preview can't read the files](#preview-cant-read-the-files--wrong-path) | > | `NumberFormatException: For input string: "Binary{...}"` | [Segment generation fails with NumberFormatException](#segment-generation-fails-with-numberformatexception) | > | `Cannot create inverted index on column ... without dictionary` | [Cannot create inverted index without dictionary](#cannot-create-inverted-index-on-column--without-dictionary) | > | `Failed to create FieldIndexConfigs` | [Failed to create FieldIndexConfigs](#failed-to-create-fieldindexconfigs) | > | `servers not responded` / query timeout | [Query times out or returns servers not responded](#query-times-out-or-returns-servers-not-responded) | > | First query slow, later queries fast | [First query on a column is slow](#first-query-on-a-column-is-slow-later-queries-are-fast) | > | `Native memory allocation (mmap) failed` / pod OOM restarts | [Server OOM / pod restarts](#server-oom--pod-restarts-under-query-load) | > | `Could not acquire distributed lock for orchestrated task type` | [Distributed lock error](#could-not-acquire-distributed-lock-for-orchestrated-task-type--externaltablesynctask) | > | Sync fails on one bad file | [Sync run fails because of one unreadable file](#a-sync-run-fails-because-of-one-unreadable-file) | > | `FileNotFoundException` / missing file after source compaction | [Segments fail to load or serve after upstream compaction](#segments-fail-to-load-or-serve-after-upstream-compaction) | > | Table created, `status` stays `IDLE` | [Table created but no sync ever starts](#table-created-but-no-sync-ever-starts) | > | `FAILED` status — what does `failurePhase` mean? | [Sync not progressing](#sync-not-progressing) | *** ## Onboarding ### Onboarding fails with a region error **Symptom:** Validation or preview fails with an AWS region error, often after entering the bucket path. **Cause:** The AWS region isn't set, or the bucket field contains an `s3://...` URL or a trailing slash. **Fix:** * Enter the **bucket name only** in the bucket field — not `s3://bucket/...`. * Remove any trailing `/` from the prefix. * Set the AWS **region in the connection config** (`catalog.s3.region` / the tier `region`). When the region is in config, the `AWS_REGION` environment variable is only a fallback and isn't required. > [!NOTE] > > For **S3 Data Lake** and **S3 Tables** sources, Test Connection now checks the configured storage region against the bucket's real region and fails immediately with a clear message naming both if they don't match — a region typo no longer passes validation and then surfaces later during preview as an opaque error (for example `Failed to index chunk 0`, the old symptom of an S3 `301 PermanentRedirect`). Test Connection also verifies the storage credentials can authenticate (`sts:GetCallerIdentity`, or `sts:AssumeRole` for an assumed role) for every catalog type. For catalog-backed sources without a bucket at connection time (Glue, Unity, Polaris, Snowflake Open Catalog / Horizon, generic REST), the region check itself still runs later, against the selected table's location, at preview or `CREATE TABLE`. To opt a connection out of the storage checks, set `catalog.{catalogType}.validateStorage=false`. ### `Access Denied` (HTTP 403) when creating the table **Symptom:** ``` software.amazon.awssdk.services.s3.model.AccessDeniedException: Access Denied (Service: S3, Status Code: 403, ...) ``` **Cause:** The cluster cannot read the source bucket. Credentials valid elsewhere are not necessarily the ones the cluster uses. **Fix:** 1. Confirm whatever the table uses for access — an assumed IAM role (`roleArn`/`externalId`), the cluster's node IAM role, or static access keys — has `s3:GetObject` and `s3:ListBucket` on the source bucket and prefix. 2. Verify from inside the cluster (e.g. a debug pod): ```bash theme={null} aws s3 ls s3://// ``` 3. If listing fails there, fix the bucket policy / role before retrying onboarding. ### Preview can't read the files / wrong path **Symptom:** Validation succeeds but preview errors, or it looks for a `default` table. **Cause:** The prefix points at the wrong level — for raw Parquet it should point at the folder that directly contains the `.parquet` files (or the table root for Iceberg). **Fix:** Adjust the prefix to the correct level. Use `aws s3 ls` to confirm the path you give actually lists Parquet files (or an Iceberg `metadata/` + `data/` layout). ### Very large source (hundreds of thousands of files) **Symptom:** Onboarding a path with hundreds of thousands or millions of files is slow or never produces segments. **Cause:** Every file becomes work for the catalog scan and segment generation. **Fix:** For an initial proof-of-concept, point at a smaller sub-prefix (for example, a single month's partition). Scale the cluster for the full dataset. There is a per-table segment threshold; work with StarTree support for very large tables. *** ## Schema & table creation ### Segment generation fails with `NumberFormatException` **Symptom:** ``` java.lang.NumberFormatException: For input string: "Binary{2 reused bytes [48 50]}" ``` **Cause:** A column's data type was changed away from what preview inferred — for example a binary/string column was set to `INT`/`LONG`. The Parquet data no longer matches the declared Pinot type. **Fix:** Use the data types the preview step produced. Don't override inferred types in the schema. See [Data Type Mapping](./data-type-mapping). ### `Cannot create inverted index on column ... without dictionary` **Symptom:** Table creation is rejected for an inverted (or FST/IFST) index on a RAW column. **Cause:** Since StarTree Cloud release 0.15, dictionary-backed indexes require an explicit `dictionary` block — they no longer build an implicit dictionary. **Fix:** Keep the forward index `RAW` and add a `dictionary` block alongside the index: ```json theme={null} "indexes": { "forward": { "encodingType": "RAW" }, "dictionary": {}, "inverted": { "disabled": false } } ``` ### `Failed to create FieldIndexConfigs` **Symptom:** Table creation fails with this generic message. **Cause:** A malformed or conflicting index configuration — often a hand-edited combination of `forward`, `dictionary`, and an index entry. **Fix:** Start from the table config the preview step generates and add only supported indexes. Don't mix incompatible options on one column. ### Inverted index on a multi-value column fails to build **Symptom:** Segment build fails on a multi-value column that has an inverted index (e.g. `Cannot create inverted index for raw index column`, or a "raw inverted index not supported for multi-value columns" message). **Cause:** The inverted index needs a dictionary, and a raw (no-dictionary) inverted index isn't supported — multi-value columns are especially likely to hit this. **Fix:** Add a `dictionary` block to the column (as above). If the build still fails specifically on a multi-value column, remove the inverted index from it and reach out to support. *** ## Queries ### Query times out or returns `servers not responded` **Symptom:** ``` 427: N servers [...] not responded ``` or a group-by/aggregation that never returns within the timeout. **Causes & fixes — check in order:** 1. **Missing or conflicting index config (most common).** Aggregations and filters scanning remote data without the right index are slow. Add a [supported index](./indexes) for your filter/group-by columns, and remove conflicting or leftover index configs. 2. **Caching not enabled.** Turn on the page cache and preload so index data is local. See [Best Practices and Configs](./best-practices-and-configs) (`enable.prefetch.page.cache`, `preload.index.keys.override`). 3. **Group-by on a derived/computed column** or a `$segmentName` filter — these defeat pruning. Group by a real column and drop debugging filters. 4. **Under-provisioned servers.** A single small server against a large dataset will be CPU-bound. Scale out. 5. **Raise the query timeout** while debugging: `SET "timeoutMs" = '60000';`. ### First query on a column is slow, later queries are fast **Symptom:** Cold query is slow; the same query is fast afterward. **Cause:** The first read populates the cache from object storage. **Fix:** Expected behavior. To have hot indexes local from segment load instead, pin them with `preload.index.keys.override`. See [Caching and index pinning](./best-practices-and-configs#caching-and-index-pinning). ### Tuning a large scan For wide scans over big datasets, these query options help (see [Best Practices and Configs](./best-practices-and-configs)): ```sql theme={null} SET "enable.prefetch.page.cache" = 'true'; SET "prefetch.projection.queue.size" = '10'; SET "readAhead.enable" = 'true'; ``` ### Server OOM / pod restarts under query load **Symptom:** A server runs out of native memory or is OOM-killed, often during a large scan (`Native memory allocation (mmap) failed`, or repeated pod restarts). **Causes & fixes:** 1. **Too many memory maps.** Wide tables create one mmap per column index; this can exhaust the OS `max_map_count`. Enable index consolidation (the tier's `preload.enable.index.consolidation=true`) to pack a segment's indexes into one file. 2. **A heavy query.** StarTree Cloud servers kill the most memory-hungry query when heap runs critically low, so one runaway query shouldn't take down a server. If it does, contact StarTree support with the query and the server logs. *** ## Sync & operations ### `Could not acquire distributed lock for orchestrated task type ... ExternalTableSyncTask` **Symptom:** ``` Could not acquire distributed lock for orchestrated task type ExternalTableSyncTask on table _OFFLINE. Another controller may be generating tasks. ``` **Cause:** Another controller is already running a sync for the table. **Fix:** Benign — retry later. The run in progress continues normally. ### A sync run fails because of one unreadable file **Symptom:** A sync run fails, and the failure traces back to a single problematic Parquet file. **Cause:** By default a run fails if any file can't be read, so the whole snapshot is rejected. **Fix:** Set `continueOnFileError=true` in the `ExternalTableSyncTask` config to skip unreadable files and continue. The run still ends as `status=COMPLETED`; compare `filesDiscovered` vs `segmentsUploaded` in the [status endpoint](./observability#run-status) to spot skipped files, then investigate them separately. > [!NOTE] > > `continueOnFileError` only affects what a **sync run** tolerates — it skips a file that can't be read *while segments are being generated*. It does **not** protect a segment that has already been ingested: if that segment's backing Parquet file later disappears (for example, [compacted away upstream](#segments-fail-to-load-or-serve-after-upstream-compaction)), the server fails to load or serve it regardless of this setting. > > Don't confuse it with `continueOnError` (an OSS `ingestionConfig` field): `continueOnError` skips **individual bad rows** while reading a file or stream, whereas `continueOnFileError` skips an **entire unreadable file**. They are separate configs set in different places. ### Segments fail to load or serve after upstream compaction **Symptom:** Queries fail, or a segment reload stalls (`status=IN_PROGRESS` with a rising `estimatedTimeRemainingInMinutes`), and server logs show a missing Parquet file: ``` java.io.FileNotFoundException: No such file or directory: s3a:////data/partition_date=.../.parquet ``` **Cause:** The table uses a **raw S3/GCS Data Lake source** (`catalogType=s3` or `gcs-interop`), which pins each ingested segment to a physical Parquet file path. When the upstream source **compacts** — rewrites many small files into fewer larger ones and deletes the originals — the paths those already-ingested segments point at no longer exist, so the server can't load or serve them. `continueOnFileError` does not help here: it only skips files *during a sync run*, not after a segment is already in the table. **Fix:** * **Use an Iceberg REST catalog for any source that compacts or rewrites files.** A REST catalog (`catalogType=iceberg-rest` — AWS Glue, S3 Tables, Unity Catalog, or generic Nessie/REST) tracks the table's live file set through snapshot metadata, so each sync reconciles to the current post-compaction files instead of leaving segments pointing at deleted paths. Raw S3/GCS is best for **append-only** sources whose files are never rewritten. See [Connect a Catalog](./getting-started/connect-a-catalog). * **On an existing raw source**, the affected segments must be rebuilt against the new files (re-onboard the table, or reload once the files are stable) — there is no config that repoints a segment at a compacted-away file. Setting `continueOnError=true` only lets ongoing ingestion skip rows it can't read; it accepts inconsistent results and does not restore a segment whose file is gone. ### Table created, but no sync ever starts **Symptom:** The table exists, but `status` stays `IDLE` and the observability endpoints return nothing useful. **Cause:** The table isn't on the controller-watcher path — usually `executor=controller` is missing from the `ExternalTableSyncTask` config, or the feature isn't enabled on the cluster. **Fix:** Confirm `executor: controller` is in the task config (the preview/onboarding flow sets it), and that the External Table feature is enabled on the cluster. Then trigger a run to start immediately: `POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=_OFFLINE`. ### Sync not progressing Use the [status endpoint](./observability#run-status) to diagnose: * **`IDLE` and never advancing** — check `executor=controller`, a valid `schedule` cron, and that the feature is enabled (see above). * **`RUNNING` for a long time** — check `currentPhase` + `currentPhaseStartTime` to see which stage it's parked in (a `WAITING_FOR_SEGMENTS_ONLINE` run stuck on Helix convergence looks very different from one stuck generating segments); large source or under-provisioned servers otherwise, so reduce the prefix or scale out. * **`FAILED`** — read `failurePhase`: * `LISTING_FILES` → credentials/path issue * `GENERATING_SEGMENTS` → data-type mismatch (see `NumberFormatException` above; escalate if the config is valid) * `COMPRESSING_SEGMENTS` → server resources * `UPLOADING_SEGMENTS` → deep-store permissions * `SAVING_PROGRESS` → controller/ZooKeeper issue * `PUBLISHING_SNAPSHOT`, `WAITING_FOR_SEGMENTS_ONLINE`, `WAITING_FOR_SERVERS_READY`, `UPDATING_SCHEMA` → later stages of the sync pipeline (snapshot commit, Helix ideal-state/external-view convergence, and schema-evolution update); escalate to StarTree support with the `requestId` if you hit one of these repeatedly > [!NOTE] > > `failurePhase` / `currentPhase` values were renamed (`FILE_LISTING`→`LISTING_FILES`, `SEGMENT_GENERATION`→`GENERATING_SEGMENTS`, `SEGMENT_COMPRESSION`→`COMPRESSING_SEGMENTS`, `SEGMENT_UPLOAD`→`UPLOADING_SEGMENTS`, `SNAPSHOT_FINALIZE`→`PUBLISHING_SNAPSHOT`, `IS_EV_CONVERGENCE`→`WAITING_FOR_SEGMENTS_ONLINE`, `SNAPSHOT_READINESS_POLL`→`WAITING_FOR_SERVERS_READY`, `CHECKPOINT_SAVE`→`SAVING_PROGRESS`, `SCHEMA_UPDATE`→`UPDATING_SCHEMA`). See [Observability](./observability#run-status). ### Sync says `COMPLETED` but no data lands **Symptom:** `fileOnboardingRun.status` is `COMPLETED`, yet the table is empty or missing recent data, and `segmentsUploaded` is `0`. **Cause:** A snapshot whose segment generation keeps failing (for example a persistent `GENERATING_SEGMENTS` error on every file) is retried up to `maxRetries` (default 3) consecutive runs, then marked terminally failed and abandoned. Once it is abandoned, the next run finds no *newer* snapshot to process and reports `COMPLETED` with `filesDiscovered: 0` / `segmentsUploaded: 0`. Because `fileOnboardingRun` reports only the run happening now, the earlier `FAILED` runs are visible only via `snapshotIngestion.lastRunStatus` / `consecutiveFailedRuns` — check those rather than assuming a `COMPLETED` status means all is well. **Diagnose:** * Compare `filesDiscovered` vs `segmentsUploaded` in the [status endpoint](./observability#run-status). A `COMPLETED` run with `0` uploads that is not caught up is the tell. * Add `?includeLag=true`: if `syncState` is not `CAUGHT_UP` (or `lag.percentDataIngested` is well under `100`) while `status` is `COMPLETED`, a snapshot was skipped rather than ingested. * Check `snapshotIngestion.consecutiveFailedRuns` and `lastRunError` for the failure streak the current `COMPLETED` record hides. * Find the underlying error from the runs that actually failed — most often a `GENERATING_SEGMENTS` failure (a data-type or Parquet-read issue). See [Segment generation fails](#segment-generation-fails-with-numberformatexception). **Fix:** Resolve the underlying per-file failure (correct the schema/type mismatch, or set `continueOnFileError=true` to skip a genuinely bad file and continue), then trigger a fresh run: `POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=_OFFLINE`. If `GENERATING_SEGMENTS` keeps failing on a valid config, escalate with the `requestId`. To make a failed snapshot retried indefinitely from its last cursor — instead of being abandoned after `maxRetries` — set `iceberg.snapshotProcessing.retryFailedSnapshot=true` on the task config (default `false`). ### Checking sync health To see why ingestion isn't progressing, use the [observability endpoints](./observability): run status (and `failurePhase` on failure), the ingestion checkpoint, and the source file count. *** ## When to escalate to engineering Most issues are self-serviceable — **auth/region/path, index and dictionary config, cron, and cache/OOM tuning** are all covered above. Escalate to engineering when: * `GENERATING_SEGMENTS` or `SAVING_PROGRESS` keeps failing with a valid config. * Servers still OOM after capping caches and enabling consolidation + OOM protection. * `status=COMPLETED` but query results are wrong or missing. * A specific Parquet type fails to read (e.g. a `getBytes`/decimal error), or you hit a very-large-table segment threshold. # Unity Catalog: Onboarding via API Source: https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-api Onboard a Databricks or OSS Unity Catalog table — including Delta Lake tables with UniForm — as a StarTree External Table, with PAT, OAuth2, or vended-credential auth. > [!WARNING] > > This feature requires StarTree release 0.16.0 or later, and must be enabled on demand — contact StarTree support to activate it. This page shows how to onboard a **Unity Catalog** External Table with the StarTree controller REST APIs. Prefer a point-and-click setup? Unity Catalog also has a [Data Portal wizard](./onboarding-data-portal). Unity Catalog tables are accessed over the **Iceberg REST protocol** (`catalogType=iceberg-rest`, `serviceType=unity`), the same mechanism used for [AWS Glue](../glue/onboarding-api) and [Amazon S3 Tables](../s3tables/onboarding-api) — only the endpoint and auth differ. This works for both OSS Unity Catalog and Databricks-managed Unity Catalog, and for **Delta Lake tables with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled** (UniForm publishes Iceberg-compatible metadata alongside the Delta log, so a Delta+UniForm table registered in Unity Catalog is queryable the same way as a native Iceberg table). Data files must be **Parquet**. > [!NOTE] > > **Delta tables with column mapping are supported in current releases.** Delta's `columnMapping.mode` set to `name` or `id` — mandatory for UniForm's Iceberg reads, which covers most Databricks-managed UniForm tables — is resolved through Iceberg field IDs across preview, segment generation, null-vector building, and renamed time columns. Field names **inside** complex columns (`struct`, `array`, `map<_,struct>`) additionally require the opt-in `storeNestedSchema.enabled=true` — see [Schema options for column mapping](#schema-options-for-column-mapping). On older 0.16.0 builds these tables synced but failed segment load with `org.apache.parquet.io.InvalidRecordException: not found in message spark_schema` — if you see that error, confirm your release with StarTree support. > Using **AWS Glue** instead? See [AWS Glue: Onboarding via API](../glue/onboarding-api). > > Using **Amazon S3 Tables** instead? See [Amazon S3 Tables: Onboarding via API](../s3tables/onboarding-api). > > Using **Nessie** instead? See [Nessie: Onboarding via API](../nessie/onboarding-api). ## How it works Onboarding is four calls. Each one feeds the next: ```text theme={null} 1. Validate & browse POST /connections/browse → validate; list namespaces/tables 2. Preview POST /tables/preview → infer schema + enriched table config 3. Create schema POST /schemas → register the schema 4. Create table POST /tables → register the External Table ``` The preview call (step 2) is the core step: it samples the source, infers a Pinot schema, and returns a ready-to-use table config. Steps 3 and 4 just persist its output. Once the table exists, the controller's watcher runs the first sync and re-syncs on schedule — no manual trigger. Track progress with the [observability endpoints](#monitor-onboarding). > [!NOTE] > > All paths are relative to your data-plane base URL. Set `export BASE_URL=https://dp..cp..startree.cloud/api/pinot` — the StarTree Cloud data-plane proxy that fronts the controller. > > If your controller requires authentication, add an `Authorization` header to every request — e.g. `-H "Authorization: Bearer "`. The examples below omit it for brevity. ## Prerequisites * StarTree 0.16.0 or later with the External Table Beta feature enabled, and **tiered storage configured** on the cluster. Contact StarTree support if unsure. * Network access to the controller REST endpoint, plus an `Authorization` token if your cluster requires auth. * A running Unity Catalog: * **Databricks-managed** — a workspace with Unity Catalog enabled, plus either a **personal access token (PAT)** generated from the workspace (**User Settings → Developer → Access tokens**) with the **`all-apis`** scope, or **OAuth2 service-principal credentials** (client ID + secret) — see [Authentication](#authentication). * **OSS Unity Catalog** — a reachable server. The OSS server is unauthenticated by default; no token is needed. * The Unity Catalog **catalog name** (used as `warehouse` below) and the namespace/table to onboard. * **For Delta Lake tables: UniForm must be enabled.** A Delta table is only exposed over the Iceberg REST endpoint once UniForm's Iceberg support is turned on. For a **new** table, set the properties at creation time — Databricks recommends `id` column mapping for new tables: ```sql theme={null} CREATE TABLE ..
(...) TBLPROPERTIES ( 'delta.columnMapping.mode' = 'id', 'delta.enableIcebergCompatV3' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg' ); ``` For an **existing** table, use `ALTER TABLE` — note that `name` is the only column-mapping mode an existing table can be upgraded to (`id` cannot be applied after creation): ```sql theme={null} ALTER TABLE ..
SET TBLPROPERTIES ( 'delta.columnMapping.mode' = 'name', 'delta.enableIcebergCompatV3' = 'true', 'delta.enableIcebergCompatV2' = 'false', 'delta.universalFormat.enabledFormats' = 'iceberg' ); ``` To check whether an existing table already has these set, run `SHOW TBLPROPERTIES ..
;` and look for the properties above in the output. Iceberg Compatibility V3 requires **Databricks Runtime 18 LTS or later** and applies to Unity Catalog **managed** tables; upgrading a table that already has V2 enabled must set `'delta.enableIcebergCompatV2' = 'false'` in the same `ALTER TABLE`, as shown above. See [Use Apache Iceberg v3 features](https://docs.databricks.com/aws/en/iceberg/iceberg-v3). > [!NOTE] > > **On an older runtime?** Use Iceberg Compatibility V2 instead — set `'delta.enableIcebergCompatV2' = 'true'` in place of the V3 property in either recipe above. V2 requires all writers to run **Databricks Runtime 14.3 LTS or later**, and enabling it on an existing table via `ALTER TABLE` requires **15.4 LTS or later** — see the [Databricks UniForm requirements](https://docs.databricks.com/aws/en/delta/uniform#requirements). StarTree onboards either compatibility level; the difference that matters downstream is how deletes are represented — see the [UniForm caveats](#databricks-uniform-caveats). * Read access to the underlying object storage. Unity Catalog **vends short-lived storage credentials automatically** for tables it manages, so static keys are usually unnecessary — see [Authentication](#authentication). ## Limitations * **Delta tables with column mapping require a current release.** `delta.columnMapping.mode = name` or `id` is supported on current releases; on older 0.16.0 builds it failed segment load with `org.apache.parquet.io.InvalidRecordException`. See the Note at the top of this page. * **`auth.storage.region` is required even when relying on vended credentials** — Unity's short-lived credentials don't carry a region, so the S3 client needs it set explicitly. ### Databricks UniForm caveats These are Databricks-side behaviors of [UniForm](https://docs.databricks.com/aws/en/delta/uniform) that affect a UniForm-backed External Table: * **New data appears only after Iceberg metadata generation.** Databricks converts Delta commits to Iceberg metadata **asynchronously**, batches frequent commits into fewer Iceberg commits, and skips generation while another run is already in progress — so the Iceberg view (and therefore StarTree's sync) can lag the Delta table. End-to-end freshness is the conversion delay plus the External Table sync schedule. Run `MSCK REPAIR TABLE
SYNC METADATA` in Databricks to force a refresh if the Iceberg view is stale. * **Keep `VACUUM` retention longer than your sync interval.** `VACUUM` physically deletes Parquet files that older Iceberg snapshots still reference; a sync run reading a snapshot older than the retention window fails when it fetches those files. Set retention to comfortably exceed the sync interval, plus any period the sync might be paused. * **Materialized views and streaming tables need the pipeline-managed path.** Iceberg Compatibility V2 isn't supported on them at all. With V3, pipeline-managed materialized views and streaming tables can be exposed over the Iceberg REST endpoint on **Databricks Runtime 17.3 or later**, in default publishing mode, and only with `'delta.enableRowTracking' = 'true'` and `'pipelines.externalMetadata.enabled' = 'true'` set alongside the UniForm properties — see [Enable external data access to streaming tables and materialized views](https://docs.databricks.com/aws/en/external-access/external-for-pipelines). Tables with `VOID` columns remain unsupported. * **Enabling UniForm is a one-way door for the Delta table.** It permanently upgrades the table's protocol (`minReaderVersion` ≥ 2, `minWriterVersion` ≥ 7) and switches new files from Snappy to Zstandard compression. StarTree reads both fine, but older external Delta clients reading the same table may break — check the table's other consumers before enabling. * **Deletes can produce Iceberg deletion vectors under Compatibility V3.** Unlike V2, Iceberg Compatibility V3 permits deletion vectors, and Databricks enables them by default on new Iceberg v3 tables — so updates and deletes may be written as Puffin delete files rather than by rewriting data files. StarTree applies Iceberg v3 Puffin **position** deletes at query time, but that path is a **preview** feature that has to be turned on: see [Deletion Vectors](../deletion-vectors) and contact StarTree to enable it. Equality deletes are still not applied. On a table left at Iceberg Compatibility V2, deletion vectors are disallowed, so deletes stay copy-on-write and every Iceberg snapshot is self-contained. ## Schema options for column mapping UniForm tables created with column mapping (`delta.columnMapping.mode` = `name` or `id`) don't store your column names in the Parquet data files — each physical column is an opaque `col-`; the logical name and a stable field-id live only in table metadata. StarTree always resolves **top-level** column names back to your logical names. Two `ExternalTableSyncTask` settings control what happens beyond that: `storeNestedSchema.enabled` defaults to `true` (opt out with `false`); `schemaEvolution.enabled` defaults to `false` (opt in). **`storeNestedSchema.enabled`** — resolves field names **inside** complex JSON columns (`struct`, `array`, `map<_,struct>`) to your logical names. It's on by default; set it to `false` to opt out. When it's off, nested keys are stored under their physical `col-` names, so JSON path queries on logical names return `NULL`: ```text theme={null} customer struct, value {name: "Alice", tier: "gold"}: OFF: stored {"col-5e6f…": "Alice", "col-7a8b…": "gold"} JSON_EXTRACT_SCALAR(customer, '$.name', 'STRING', 'null') → NULL ON: stored {"name": "Alice", "tier": "gold"} JSON_EXTRACT_SCALAR(customer, '$.name', 'STRING', 'null') → "Alice" ``` Not needed for flat tables, or for native Iceberg / `add_files` tables where physical names already equal logical names. **`schemaEvolution.enabled`** (plus `schemaEvolution.failFast`) — before each snapshot's segments, unions the registered Pinot schema with the source schema by field-id: new source columns are added automatically, and renamed columns keep resolving (the existing Pinot column name is preserved, with the source's names tracked as aliases). `failFast=true` aborts the run on an incompatible change (reported as `failurePhase: SCHEMA_UPDATE` in the [status API](../observability#run-status)); `false` logs it and continues on the previous schema. Leave off to pin the Pinot schema and manage changes manually. See [Best Practices & Configs](../best-practices-and-configs#checkpointing--schema-evolution) for the resolution rules and limitations. `storeNestedSchema.enabled` is on by default, so onboarding needs no setting for it. To opt out, or to set schema evolution, use the `ExternalTableSyncTask` block of the preview request (Step 2): ```json theme={null} "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "storeNestedSchema.enabled": "false", "schemaEvolution.enabled": "true", "schemaEvolution.failFast": "false" }}} ``` > [!NOTE] > > Changing `storeNestedSchema.enabled` on an **already-onboarded** table takes effect on the next schema write (re-preview/re-register, or the next sync when schema evolution is on), followed by a segment reload — nested-name resolution happens at read time, so segments don't need regenerating. A table onboarded before this setting defaulted to on gets that schema update and reload automatically, once, on its first sync after upgrade, if it has complex columns and no explicit setting of its own. To opt out cleanly, set `storeNestedSchema.enabled=false` before onboarding. ## Authentication An External Table authenticates in two places, configured independently: * **Catalog (REST)** — keys under `catalog.iceberg-rest.auth.rest.*`. Choose one of three methods below, depending on your Unity Catalog deployment. * **Storage (S3 data files)** — keys under `catalog.iceberg-rest.auth.storage.*`. Unity Catalog vends short-lived S3 credentials per table load, and those take priority over anything set here — see [Storage authentication](#storage-authentication) below. ### Catalog (REST) authentication: choose one #### Databricks: personal access token (PAT) `authType` doesn't need to be set explicitly — it's auto-detected as token auth when `.token` is present. ```json theme={null} { "catalog.iceberg-rest.auth.rest.token": "" } ``` > [!NOTE] > > **The PAT must have the `all-apis` scope.** Every call StarTree makes — browse, preview, and each sync run — goes through the Unity Catalog Iceberg REST API, so a token issued with a narrower scope is rejected (typically a `401`/`403` on `POST /connections/browse`) even though the token itself is valid and unexpired. If your workspace offers scope selection when generating a token, pick **all-apis**; tokens created without an explicit scope get it by default. The same requirement applies to the [Data Portal wizard](./onboarding-data-portal#step-2-configure-the-unity-catalog-connection). #### Databricks: OAuth2 client-credentials (recommended, new in 0.16.0) For service-principal / machine-to-machine access — the preferred method over a long-lived PAT for automated pipelines. `authType` is optional; it's auto-detected as `oauth2` once the three required keys below are all present. ```json theme={null} { "catalog.iceberg-rest.auth.rest.authType": "oauth2", "catalog.iceberg-rest.auth.rest.oauthTokenUri": "", "catalog.iceberg-rest.auth.rest.oauthClientId": "", "catalog.iceberg-rest.auth.rest.oauthClientSecret": "", "catalog.iceberg-rest.auth.rest.oauthScopes": "" } ``` | Key | Required | Description | | - | - | - | | `oauthTokenUri` | Yes | OAuth2 token endpoint. | | `oauthClientId` | Yes | Service-principal client ID. | | `oauthClientSecret` | Yes | Service-principal client secret. | | `oauthScopes` | No | Space-separated OAuth2 scopes, if your Unity Catalog deployment requires them. | The client-credentials grant (RFC 6749) is exchanged for a short-lived token that's cached and refreshed automatically ahead of expiry — you don't need to rotate or refresh anything yourself. Note the [SQL DDL](../sql-ddl) shortcut for this same setup uses `auth_type = 'oauth'`, which normalizes to this OAuth2 configuration. #### OSS Unity Catalog (unauthenticated) The OSS server is unauthenticated by default — omit `auth.rest.*` entirely. ```json theme={null} {} ``` ### Storage authentication #### Vended credentials (default — no keys needed) Unity Catalog vends short-lived S3 credentials scoped to the table's location on every load; StarTree uses those automatically. On Databricks, credential vending has two workspace-side prerequisites: **external data access must be enabled on the metastore**, and the principal behind the PAT or OAuth2 credentials needs the **`EXTERNAL USE SCHEMA`** privilege on the schema — see [Enable external data access to Unity Catalog](https://docs.databricks.com/aws/en/external-access/admin). Without them, catalog browsing works but table loads fail. Only `region` is required on the StarTree side, so the S3 client knows which region to sign requests for: ```json theme={null} { "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Override with static access keys Set these only if you need to bypass the vended credentials — for example, a fixed IAM identity for the tier's deep-store access, separate from the source read path. If present, these values are ignored whenever Unity successfully vends credentials for the load. ```json theme={null} { "catalog.iceberg-rest.auth.storage.accessKeyId": "", "catalog.iceberg-rest.auth.storage.secretAccessKey": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` #### Override with an assumed IAM role Same override caveat as above — same STS assumed-role mechanism used for [Glue](../glue/onboarding-api#storage-authentication-choose-one) and [S3 Tables](../s3tables/onboarding-api#storage-authentication-choose-one). ```json theme={null} { "catalog.iceberg-rest.auth.storage.roleArn": "arn:aws:iam::123456789012:role/pinot-external-table-reader", "catalog.iceberg-rest.auth.storage.externalId": "", "catalog.iceberg-rest.auth.storage.region": "" } ``` ## `restUri` formats | Deployment | Example `restUri` | | - | - | | Databricks-managed Unity Catalog | `https:///api/2.1/unity-catalog/iceberg-rest` | | OSS Unity Catalog | `http://:8080/api/2.1/unity-catalog/iceberg` | Both suffixes are exactly what you provide — StarTree doesn't rewrite the host or path, it only appends the standard Iceberg REST `/v1/...` routes. *** ## Step 1: Validate and browse the connection **`POST /connections/browse`** There is no separate "validate" endpoint. Browsing the catalog *is* the validation step: a `200` with an `items` list (even an empty one) confirms your credentials and connectivity. * Set `path` to `""` to browse the **root** — this both validates the connection and lists namespaces. * Set `path` to a namespace name to list the **tables** inside it. ### Request | Field | Type | Required | Description | | - | - | - | - | | `connection.type` | string | Yes | Use `CATALOG` for all External Table sources. | | `connection.params` | object | Yes | Catalog-specific connection settings (see example below). | | `path` | string | No | Where to browse. `""` or omitted = root. A namespace name = that namespace's tables. | **Databricks-managed** ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "iceberg-rest", "catalog.iceberg-rest.serviceType": "unity", "catalog.iceberg-rest.restUri": "https:///api/2.1/unity-catalog/iceberg-rest", "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.auth.rest.token": "" } }, "path": "" }' ``` **OSS Unity Catalog** ```bash theme={null} curl -X POST "$BASE_URL/connections/browse" \ -H "Content-Type: application/json" \ -d '{ "connection": { "type": "CATALOG", "params": { "catalogType": "iceberg-rest", "catalog.iceberg-rest.serviceType": "unity", "catalog.iceberg-rest.restUri": "http://:8080/api/2.1/unity-catalog/iceberg", "catalog.iceberg-rest.warehouse": "" } }, "path": "" }' ``` ### Response ```json theme={null} { "items": [ { "name": "default", "type": "NAMESPACE" }, { "name": "countries", "type": "TABLE" } ] } ``` | `items[].type` | Meaning | | - | - | | `NAMESPACE` | A namespace you can drill into — pass its `name` as `path`. | | `TABLE` | A selectable table. | *** ## Step 2: Preview the schema **`POST /tables/preview`** Samples the source, infers a Pinot schema, and returns an enriched table config (S3 tier, raw field configs, time column) plus sample rows. Review it, tweak if needed, then carry the schema and config forward to steps 3 and 4. > [!NOTE] > > The request and response share the same JSON shape. You send a `tableConfig` describing the source; the response fills in `schema`, the enriched `tableConfigs.offline`, sampled `rows`, and a `summary`. ### Request **Top-level fields:** | Field | Type | Required | Description | | - | - | - | - | | `tableConfig` | object | Yes | A single OFFLINE table config whose `ExternalTableSyncTask` block describes the source. | | `config` | object | No | Sampling and inference controls (see below). Defaults are applied if omitted. | | `schema` | object | No | An explicit schema to use instead of inferring one. | **`config.inference` — how the schema is derived:** | Field | Type | Default | Description | | - | - | - | - | | `embeddedSchemaEnabled` | boolean | `false` | Read the schema embedded in the Iceberg metadata. **Set this to `true` for External Tables.** | | `schemaInferenceEnabled` | boolean | `false` | Infer the schema from sampled rows. Fallback when no embedded schema exists. | Resolution order: explicit `schema` → embedded schema → inferred schema. **`config.sampling` — how rows are sampled:** | Field | Type | Default | Description | | - | - | - | - | | `min` | integer | `1` | Minimum records to sample. | | `max` | integer | `100` | Maximum records to sample. | > [!TIP] > > To list source files without sampling any data, set `config.previewFiles.previewFilesOnly = true`. The response returns matching file URIs in `sourceFiles` and skips schema inference. (OFFLINE only.) ```bash theme={null} curl -X POST "$BASE_URL/tables/preview" \ -H "Content-Type: application/json" \ -d '{ "tableConfig": { "tableName": "countries_OFFLINE", "tableType": "OFFLINE", "task": { "taskTypeConfigsMap": { "ExternalTableSyncTask": { "catalogType": "iceberg-rest", "executor": "controller", "inputFormat": "parquet", "catalog.iceberg-rest.serviceType": "unity", "catalog.iceberg-rest.restUri": "https:///api/2.1/unity-catalog/iceberg-rest", "catalog.iceberg-rest.warehouse": "", "catalog.iceberg-rest.table.namespace": "default", "catalog.iceberg-rest.table.tableName": "countries", "catalog.iceberg-rest.auth.rest.token": "", "catalog.iceberg-rest.auth.storage.region": "" } } } }, "config": { "inference": { "embeddedSchemaEnabled": true } } }' ``` > [!NOTE] > > **Setting the namespace and table.** Take them from the [browse](#step-1-validate-and-browse-the-connection) response — set `catalog.iceberg-rest.table.namespace` / `.tableName` to the `NAMESPACE` and `TABLE` you selected. > > **`auth.storage.region`** is required even when relying on Unity's vended storage credentials (see [Authentication](#authentication)) — it tells the S3 client which region to sign requests for. > > **Use `executor: controller`** — it's required for the controller-watcher flow and the [observability endpoints](../observability). The input `tableConfig` is intentionally minimal; `/tables/preview` returns the complete `tableConfigs.offline` you persist in Step 4. ### Response | Field | Type | Description | | - | - | - | | `schema` | object | The resolved Pinot schema. **Use this in step 3.** | | `tableConfigs.offline` | object | The enriched OFFLINE table config, ready to persist. **Use this in step 4.** Secrets are masked as `*****`. | | `rows` | array | Sample records after schema + ingestion transforms. | | `sourceRows` | array | Raw records before transformation. | | `summary` | object | Run summary: `nSourceRows`, `nRows`, `nColumns`, and `summary.batch.nMatchingFiles` (files found). | Adjust the schema (time column, column names, null handling) before moving on. *** ## Step 3: Create the schema **`POST /schemas`** Send the `schema` object from the preview response. Rename its `schemaName` to match your table. ```bash theme={null} curl -X POST "$BASE_URL/schemas" \ -H "Content-Type: application/json" \ -d @schema.json ``` ```json theme={null} { "status": "countries successfully added" } ``` *** ## Step 4: Create the table **`POST /tables`** Send the enriched `tableConfigs.offline` object from the preview response. Its `ExternalTableSyncTask` block is what marks the table as external. ```bash theme={null} curl -X POST "$BASE_URL/tables" \ -H "Content-Type: application/json" \ -d @tableConfig.json ``` ```json theme={null} { "status": "Table countries_OFFLINE successfully added" } ``` The controller's External Table watcher then discovers the table, runs the first sync, and re-syncs at the `schedule` (cron) interval. There is no separate start call. > [!TIP] > > The first sync runs on the watcher's next tick. To kick it off immediately instead of waiting, you can manually trigger a run: > > ```bash theme={null} > curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=countries_OFFLINE" > ``` *** ## Quickstart: onboard a table end-to-end The whole flow as one script (Databricks-managed Unity Catalog). Fill in the variables at the top, run the script, and the first sync starts automatically. `schema.json` and `tableConfig.json` — referenced in steps 3 and 4 above — are extracted here from the preview response with `jq`. ```bash theme={null} #!/usr/bin/env bash set -euo pipefail BASE_URL="https://dp..cp..startree.cloud/api/pinot" AUTH="Authorization: Bearer " REST_URI="https:///api/2.1/unity-catalog/iceberg-rest" WAREHOUSE="" NAMESPACE="default" TABLE="countries" PAT="" STORAGE_REGION="" # 1. Validate & browse root — lists Unity Catalog namespaces. curl -sf -X POST "$BASE_URL/connections/browse" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"connection\": { \"type\": \"CATALOG\", \"params\": { \"catalogType\": \"iceberg-rest\", \"catalog.iceberg-rest.serviceType\": \"unity\", \"catalog.iceberg-rest.restUri\": \"$REST_URI\", \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\", \"catalog.iceberg-rest.auth.rest.token\": \"$PAT\" } }, \"path\": \"\" }" | jq . # 2. Preview — infer schema + enriched table config. curl -sf -X POST "$BASE_URL/tables/preview" \ -H "$AUTH" -H 'Content-Type: application/json' \ --data-raw "{ \"tableConfig\": { \"tableName\": \"${TABLE}_OFFLINE\", \"tableType\": \"OFFLINE\", \"task\": { \"taskTypeConfigsMap\": { \"ExternalTableSyncTask\": { \"catalogType\": \"iceberg-rest\", \"executor\": \"controller\", \"inputFormat\": \"parquet\", \"catalog.iceberg-rest.serviceType\": \"unity\", \"catalog.iceberg-rest.restUri\": \"$REST_URI\", \"catalog.iceberg-rest.warehouse\": \"$WAREHOUSE\", \"catalog.iceberg-rest.table.namespace\": \"$NAMESPACE\", \"catalog.iceberg-rest.table.tableName\": \"$TABLE\", \"catalog.iceberg-rest.auth.rest.token\": \"$PAT\", \"catalog.iceberg-rest.auth.storage.region\": \"$STORAGE_REGION\" } } } }, \"config\": { \"inference\": { \"embeddedSchemaEnabled\": true } } }" > preview.json # 3. Extract schema and table config from preview response. jq --arg t "$TABLE" '.schema | .schemaName = $t' preview.json > schema.json jq '.tableConfigs.offline' preview.json > tableConfig.json # 4. Create the schema. curl -sf -X POST "$BASE_URL/schemas" \ -H "$AUTH" -H 'Content-Type: application/json' -d @schema.json # 5. Create the table — watcher starts the first sync automatically. curl -sf -X POST "$BASE_URL/tables" \ -H "$AUTH" -H 'Content-Type: application/json' -d @tableConfig.json # 6. (optional) Poll until sync completes. curl -sf "$BASE_URL/tables/${TABLE}_OFFLINE/externalTable/status" -H "$AUTH" \ | jq -e '.fileOnboardingRun.status == "COMPLETED" and .fileOnboardingRun.segmentsUploaded == .fileOnboardingRun.filesDiscovered' ``` For OSS Unity Catalog, drop the `PAT` variable and the `auth.rest.token` key from both requests, and point `REST_URI` at your server (e.g. `http://:8080/api/2.1/unity-catalog/iceberg`). Once step 6 exits `0`, [verify it's queryable](#verify-its-queryable). ## Monitor onboarding Three read-only endpoints report ingestion progress — run status, ingestion checkpoint, and source file count — and require `executor=controller` (set automatically). See [Observability](../observability) for full request and response details. ## Verify it's queryable When `fileOnboardingRun.status` is `COMPLETED` and `segmentsUploaded` matches `filesDiscovered`, run a query against the broker (or the Data Portal query console) to confirm the data is live: ```sql theme={null} SELECT count(*) FROM countries; ``` If the run status is `COMPLETED` but the count is `0`, give segments a moment to load on the servers, then recheck; if it persists, see [Troubleshooting](../troubleshooting). *** ## What's next Now that the table is created and data is loading, these are the highest-impact follow-up steps: 1. **Add indexes for your query patterns.** Without indexes, every query scans all remote Parquet data. Add a range index on time/numeric columns, an inverted index on low-cardinality filter columns, and a bloom filter on high-cardinality ID columns. → [Indexes](../indexes) 2. **Enable caching and pin hot indexes.** Set `enable.prefetch.page.cache=true` on the S3 tier, and list small hot indexes in `preload.index.keys.override`, so index data is served from local disk on repeated queries instead of re-fetched from S3. → [Data and Index Caching](../data-and-index-caching) 3. **Monitor ongoing syncs.** Use the observability endpoints to check run status, ingestion checkpoint, and source file count after each scheduled sync. → [Observability](../observability) *** For common questions and failures, see the [FAQ](../faq) and [Troubleshooting](../troubleshooting). # Unity Catalog: Onboarding via Data Portal Source: https://docs.startree.ai/corecapabilities/external-table/unity/onboarding-data-portal Connect a Databricks or OSS Unity Catalog — including Delta Lake tables with UniForm — to StarTree as an External Table with the point-and-click Data Portal wizard, no API calls required. > [!WARNING] > > This feature is available starting in **StarTree release 0.16.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment. This guide walks through connecting a **Unity Catalog** to StarTree using the **Data Portal UI**. No API calls or JSON configuration required — Data Portal guides you through catalog connection, table selection, and onboarding setup through a point-and-click interface. Unity Catalog tables are accessed over the **Iceberg REST protocol**, and this path also covers **Delta Lake tables with [UniForm](https://docs.delta.io/latest/delta-uniform.html) enabled** registered in the catalog. This works for both Databricks-managed and OSS Unity Catalog. Data files must be **Parquet**. > [!WARNING] > > **Delta tables with column mapping — supported in current releases.** Delta tables whose `delta.columnMapping.mode` is `name` or `id` (which covers most Databricks-managed UniForm tables) are supported: the physical `col-` Parquet column names are resolved through Iceberg field IDs during preview and segment generation. Resolving field names **inside** complex columns (`struct`, `array`, `map<_,struct>`) additionally uses the `storeNestedSchema.enabled` task config, on by default — see [Schema options for column mapping](./onboarding-api#schema-options-for-column-mapping). On older 0.16.0 builds these tables failed segment load — if you hit `InvalidRecordException` at load, confirm your release with StarTree support. See [Onboarding via API — Limitations](./onboarding-api#limitations). > Looking for the API-based approach? See [Unity Catalog: Onboarding via API](./onboarding-api). > > Using **AWS Glue** instead? See [AWS Glue: Onboarding via Data Portal](../glue/onboarding-data-portal). > > Using **Amazon S3 Tables** instead? See [Amazon S3 Tables: Onboarding via Data Portal](../s3tables/onboarding-data-portal). *** ## Prerequisites Before starting, ensure you have: * **StarTree 0.16.0 or later** with the external table Beta feature enabled and **tiered storage configured** for your environment. Contact StarTree support if unsure. * A running Unity Catalog: * **Databricks-managed** — a workspace with Unity Catalog enabled, plus either a **personal access token (PAT)** or **OAuth2 service-principal credentials** (client ID + secret) to authenticate with. A PAT must carry the **`all-apis`** scope; a token issued with a narrower scope is rejected by the Unity Catalog Iceberg REST endpoint. * **OSS Unity Catalog** — a reachable server. The OSS server is unauthenticated by default, so no token is needed. * The Unity Catalog **catalog name** (used as the warehouse identifier) and the namespace/table you want to onboard. * **For Delta Lake tables: UniForm must be enabled.** A Delta table is only exposed over the Iceberg REST endpoint once UniForm's Iceberg support is turned on. For a **new** table, set the properties at creation time — Databricks recommends `id` column mapping for new tables: ```sql theme={null} CREATE TABLE ..
(...) TBLPROPERTIES ( 'delta.columnMapping.mode' = 'id', 'delta.enableIcebergCompatV3' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg' ); ``` For an **existing** table, use `ALTER TABLE` — note that `name` is the only column-mapping mode an existing table can be upgraded to (`id` cannot be applied after creation): ```sql theme={null} ALTER TABLE ..
SET TBLPROPERTIES ( 'delta.columnMapping.mode' = 'name', 'delta.enableIcebergCompatV3' = 'true', 'delta.enableIcebergCompatV2' = 'false', 'delta.universalFormat.enabledFormats' = 'iceberg' ); ``` To check whether an existing table already has these set, run `SHOW TBLPROPERTIES ..
;` and look for the properties above in the output. Iceberg Compatibility V3 requires **Databricks Runtime 18 LTS or later** and applies to Unity Catalog **managed** tables; upgrading a table that already has V2 enabled must set `'delta.enableIcebergCompatV2' = 'false'` in the same `ALTER TABLE`, as shown above. See [Use Apache Iceberg v3 features](https://docs.databricks.com/aws/en/iceberg/iceberg-v3). > [!NOTE] > > **On an older runtime?** Use Iceberg Compatibility V2 instead — set `'delta.enableIcebergCompatV2' = 'true'` in place of the V3 property in either recipe above. V2 requires all writers to run **Databricks Runtime 14.3 LTS or later**, and enabling it on an existing table via `ALTER TABLE` requires **15.4 LTS or later** — see the [Databricks UniForm requirements](https://docs.databricks.com/aws/en/delta/uniform#requirements). StarTree onboards either compatibility level; the difference that matters downstream is how deletes are represented — see the [UniForm caveats](#databricks-uniform-caveats) below. * Read access to the underlying S3 object storage. Unity Catalog **vends short-lived storage credentials automatically** for tables it manages, so static keys are usually unnecessary — but the storage **region is always required**. *** ## Step 1: Open the External Tables 1. Log in to **Data Portal**. 2. In the left navigation, go to **Tables**. 3. Click **+ Connect External Table**. The wizard opens with a connection configuration screen. *** ## Step 2: Configure the Unity Catalog Connection Choose **Unity Catalog** as the catalog type. Fill in the credentials and connection details: **Details** | Field | Description | | - | - | | **Connection Name** | A unique name to identify this connection in Data Portal. | **Metastore** — how Data Portal authenticates with the Unity Catalog Iceberg REST API. | Field | Description | | - | - | | **Catalog Endpoint URI** | The Unity Catalog Iceberg REST endpoint — e.g. `https:///api/2.1/unity-catalog/iceberg-rest` (Databricks) or `http://:8080/api/2.1/unity-catalog/iceberg` (OSS). | | **Catalog Name** | The Unity Catalog catalog name, e.g. `main`. Required. | | **REST Authentication Type** | **Personal Access Token** or **OAuth**. | For **Personal Access Token**: | Field | Description | | - | - | | **Personal Access Token** | Databricks personal access token, generated from **User Settings → Developer → Access tokens**. Must be issued with the **`all-apis`** scope. | > [!NOTE] > > **The PAT needs the `all-apis` scope.** Data Portal reaches Unity Catalog over the Iceberg REST API, so a token restricted to a narrower scope fails at **Validate Connection** even though the credentials themselves are valid. If your workspace offers scope selection when generating a token, pick **all-apis**; tokens created without an explicit scope get it by default. For **OAuth** (Databricks service principal — recommended over a long-lived PAT for automated pipelines, new in 0.16.0): | Field | Description | | - | - | | **OAuth Token URI** | OAuth2 token endpoint. | | **OAuth Client ID** | Service-principal client ID. | | **OAuth Client Secret** | Service-principal client secret. | **Storage** — credentials for reading the underlying Parquet data files. Unity Catalog vends short-lived storage credentials automatically for most tables — whatever you configure here acts as the fallback/override for tables where vending doesn't apply. **Region** is always required, even with vended credentials, because Unity's short-lived credentials don't carry one. | Field | Description | | - | - | | **Region** | AWS region where the S3 data files are stored. Always required. | | **Authentication Type** | **Basic Authentication** (Access Key + Secret Key), **IAM Node Role** (the cluster node's own IAM identity — Data Portal shows the resolved role as a read-only **IAM Role Name** field so you can grant it access in your AWS account), or **IAM Role** (a cross-account role StarTree assumes). | | **Access Key** / **Secret Key** | With **Basic Authentication**: static AWS credentials for S3 data access. | | **IAM Role ARN** | With **IAM Role**: the ARN of the role to assume. The form shows **Steps to create IAM Policy** — a ready-made trust policy including the StarTree account and External ID to configure on the role. | > [!NOTE] > > The External ID used in the cross-account trust policy is **provisioned by StarTree** and filled into the connection automatically — the trust-policy steps shown in the form include it, so you never enter it by hand. Click **Validate Connection**. Data Portal calls the catalog's validate endpoint and confirms credentials and connectivity before proceeding. *** ## Step 3: Browse and Select a Table Once the connection is validated: 1. Data Portal lists the available **schemas** (the field is labeled **Database**) in the Unity Catalog. 2. Select a schema to expand its tables. 3. Click the **table** you want to onboard. Data Portal reads the Iceberg schema and derives a Pinot schema automatically. *** ## Step 4: Review the Schema The inferred Pinot schema is displayed for review. For External Tables this step is **mostly read-only** — columns can't be added, removed, or renamed, and the inferred data types should be kept (overriding them can break segment generation). One thing **is** editable inline: the **Pinot Field Type** column lets you switch a column between **Dimension**, **Metric**, and **Date-Time** where its data type allows (for example, a `LONG` can be any of the three; a `TIMESTAMP` can only be Date-Time). Selecting **Date-Time** opens a dialog to set the value format and granularity. Multi-value columns are flagged in a dedicated column. Click **Next** when the schema looks correct. *** ## Step 5: Configure the Table Set the final table options: | Setting | Default | Notes | | - | - | - | | **Table Name** | — | Name of the Pinot table to create. | | **Serving Replicas** | `1` | Replication factor for the table's (metadata-only) segments. `1` maximizes onboarding throughput; it can be raised later. | | **Time Column** | *(first candidate)* | Shown only when the schema has date-time columns — pick the primary time column used for segment pruning and retention, or leave the table without one. | The sync schedule is fixed at **every 5 minutes** at creation (change it afterwards via the table config, or use **Pause Sync** / **Resume Sync**), and **null handling is enabled automatically** on the generated table config. Click **Create Table** to register the schema and table with Pinot. Data Portal automatically triggers the first onboarding run immediately after creation — no manual step required. *** ## Step 6: Monitor Onboarding Once the table is created, onboarding starts automatically. The table detail view shows the status in real time: * **Running** — the task is actively reading Iceberg snapshots and building Pinot segments. * **Completed** — onboarding finished successfully. The last ingested snapshot ID is shown. * **Failed** — onboarding encountered an error. The error message and the number of files discovered vs. segments generated are surfaced to help diagnose the issue. For deeper observability — watcher status, checkpoint values, and per-snapshot file counts — see the [Observability](../observability) page. *** ## Pausing Onboarding To pause scheduled onboarding from Data Portal: 1. Open the table in the **Tables** view. 2. Click **Pause Sync**. This removes the `schedule` (cron trigger) from the table's `ExternalTableSyncTask` config and stashes its value under `schedule.paused`, so the watcher stops firing new runs. Any run currently in progress completes normally, and existing segments and the last checkpoint are untouched. Click **Resume Sync** to restore the schedule from `schedule.paused` — onboarding resumes from where it left off; it does not restart from scratch. > [!NOTE] > > This is a separate mechanism from the `enabled` key documented in [Best Practices & Configs](../best-practices-and-configs#table-config--externaltablesynctask) — that key can also be set to `false` directly via the table config or API to stop new runs, but the Data Portal's **Pause Sync** button acts on `schedule`/`schedule.paused`, not `enabled`. *** ## Databricks UniForm caveats These are Databricks-side behaviors of [UniForm](https://docs.databricks.com/aws/en/delta/uniform) that affect a UniForm-backed External Table onboarded through this wizard. * **Deletes can produce Iceberg deletion vectors under Compatibility V3.** Unlike V2, Iceberg Compatibility V3 permits deletion vectors, and Databricks enables them by default on new Iceberg v3 tables — so updates and deletes may be written as Puffin delete files rather than by rewriting data files. StarTree applies Iceberg v3 Puffin **position** deletes at query time, but that path is a **preview** feature that has to be turned on: see [Deletion Vectors](../deletion-vectors) and contact StarTree to enable it. Equality deletes are still not applied. On a table left at Iceberg Compatibility V2, deletion vectors are disallowed, so deletes stay copy-on-write and every Iceberg snapshot is self-contained. * **Materialized views and streaming tables need the pipeline-managed path.** Iceberg Compatibility V2 isn't supported on them at all. With V3, pipeline-managed materialized views and streaming tables can be exposed over the Iceberg REST endpoint on **Databricks Runtime 17.3 or later**, in default publishing mode, and only with `'delta.enableRowTracking' = 'true'` and `'pipelines.externalMetadata.enabled' = 'true'` set alongside the UniForm properties — see [Enable external data access to streaming tables and materialized views](https://docs.databricks.com/aws/en/external-access/external-for-pipelines). Tables with `VOID` columns remain unsupported. * **New data appears only after Iceberg metadata generation.** Databricks converts Delta commits to Iceberg metadata **asynchronously**, so the Iceberg view — and therefore the next sync run — can lag the Delta table. End-to-end freshness is the conversion delay plus the sync schedule you set in Step 5. * **Keep `VACUUM` retention longer than your sync interval.** `VACUUM` physically deletes Parquet files that older Iceberg snapshots still reference; a sync run reading a snapshot older than the retention window fails when it fetches those files. * **Enabling UniForm is a one-way door for the Delta table.** It permanently upgrades the table's protocol and switches new files from Snappy to Zstandard compression. StarTree reads both fine, but older external Delta clients reading the same table may break. See [Onboarding via API — Databricks UniForm caveats](./onboarding-api#databricks-uniform-caveats) for the full detail on each. *** ## Frequently Asked Questions **The Validate step fails — what should I check?** * For Databricks with **PAT**: confirm the token is valid, hasn't expired, and was issued with the **`all-apis`** scope, and that the workspace has Unity Catalog enabled. A scope-restricted token is the most common cause of a validate failure that looks like a bad credential. * For Databricks with **OAuth2**: confirm the token URI, client ID, and client secret are correct, and that the service principal has access to the catalog. * For **OSS Unity Catalog**: confirm the server is reachable from the cluster and the REST URI includes the correct path suffix (`/api/2.1/unity-catalog/iceberg`). * Verify the warehouse value matches the Unity Catalog catalog name exactly. *** **Table load or sync fails even though browsing worked — what should I check?** * Confirm the storage **Region** is set — it's required even when relying on Unity's vended credentials, which don't carry a region. * On Databricks, credential vending requires **external data access enabled on the metastore** and the **`EXTERNAL USE SCHEMA`** privilege on the schema for the principal behind your PAT or OAuth2 credentials. Without them, catalog browsing works but table loads fail. See [Onboarding via API — Storage authentication](./onboarding-api#storage-authentication). *** **Can I onboard multiple tables from the same catalog?** Yes. After creating the first table, start the wizard again and reuse the same connection credentials. Each table is registered as an independent Pinot table with its own onboarding schedule. *** **The table was created but onboarding hasn't started — what should I check?** Data Portal triggers the first onboarding run automatically after table creation. If onboarding hasn't started, check the table's detail page for an error status and review the error message. You can also trigger a run manually via the [trigger API](../observability#trigger-a-sync-run-optional). # Atomic Ingestion Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/atomic-sync-consistent-push > [!NOTE] > > The terms *Atomicity* and *Consistent Push* are often used interchangeably in the context of Apache Pinot and StarTree. Consistent Push ensures that the data in the Pinot table remains either unchanged or fully updated, with all files ingested and new segments activated. Once the ingestion task is completed, the table seamlessly switches to the new segments, and the old segments are purged to maintain data consistency. ## Core Concepts ### consistentPush vs. syncMode `consistentPushEnabled` and `syncMode` are independent concepts and should not be confused: * **`syncMode`** controls whether there is a 1:1 mapping between source files and Pinot segments. When enabled, each source file maps to exactly one segment. This is needed for incremental ingestion where older files are deleted in Source and needs to be purged from Pinot as well. * **`consistentPushEnabled`** controls atomicity at ingestion time. It ensures that the data in the Pinot table remains either unchanged or fully updated. Once the ingestion task is completed, the table seamlessly switches to the new segments, and the old segments are purged to maintain data consistency. You can use `consistentPushEnabled` with or without `syncMode`. The two settings serve different purposes. ### consistentPushSwapEnabled — Full Table Refresh `consistentPushSwapEnabled` is a special variant of consistent push that replaces the **entire table** on every ingestion run, rather than applying incremental changes. Key differences: * **`consistentPushEnabled`** applies atomic transitions to the incremental set of new files only. * **`consistentPushSwapEnabled`** atomically swaps all data in the table with the current state of the source files. Any source file that has been deleted will also have its corresponding segment removed from Pinot. Critically, **`consistentPushSwapEnabled` is independent of `syncMode`**. A 1:1 file-to-segment mapping is not required for a full table refresh. If your use case is to periodically replace the entire table, enable `consistentPushSwapEnabled` and leave `syncMode` configuration aside. ## Configuration Parameters | Parameter | Type | Default | Purpose | Behavior/When to Use | | :- | - | - | :- | - | | consistentPush SwapEnabled | Mandatory (Boolean) | false | Enables a full table refresh to atomically replace the entire table with new files. | Overrides consistentPushEnabled. Deletes data in Pinot if corresponding source files are deleted. | | consistentPush Enabled | Mandatory (Boolean) | false | Allows atomic ingestion of a new set of files without full table replacement. | Manages atomic transitions. Retains original behavior for sync or append operations. | | clearConsistent PushState | Optional (Boolean) | false | Clears the metadata and progress state associated with the consistent push session in ZK. | Use for debugging or resetting corrupted metadata. Default retains 2 ingestion sessions for troubleshooting. | | consistentPushMax Retries | Optional (Integer, max: INT\_MAX) | 0 | Configures the max retries for tasks if file ingestion fails. | Useful for large ingestions to avoid reprocessing successful files when retrying after failures. | | retryCount (read-only) | Integer | | Tracks the number of retries for a task. | Stored in ZK at `/pinot/PROPERTYSTORE/ MINION_TASK_METADATA/` `/FileIngestionTask/ consistentPush/ ` | | state (read-only) | Enum (INIT, IN\_PROGRESS, SWITCH, DONE) | | Tracks the current status of an ingestion session. | INIT: Session starts. IN\_PROGRESS: Ingestion ongoing. SWITCH: Preparing final ingestion stage. DONE: All data ingested and queryable. Retries required if state isn't DONE. | Below is a sample task configuration. ```json theme={null} "task": { "taskTypeConfigsMap": { "FileIngestionTask": { "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "...", "input.fs.prop.secretKey": "...", "input.fs.prop.accessKey": "...", "inputDirURI": "s3://.../", "inputFormat": "json", "includeFileNamePattern": "glob:**/*.json.gz", "consistentPushEnabled": "true", "schedule": "" } } }, ``` ## Consistent Push with Retry on failure The retry mechanism is implemented to reuse partial progress and avoid re-reading successful files. It ensures data consistency and completeness in file ingestion tasks. Consistent Push handles failures effectively, ensuring all files are ingested without compromising consistency. The goal of this mechanism is to ingest all or none of the files, maintaining atomicity across retries. ## Why Retry is Needed In typical file ingestion, if any subtask fails, the files linked to those tasks remain un-ingested. Consistent push ensures no partial or inconsistent ingestion. Retrying ingestion is vital to: 1. Ensure atomicity (all-or-none ingestion) in consistent push flows. 2. Prevent the loss of previously ingested data. 3. Optimize ingestion tasks by reusing the progress of successful sub-tasks in the previous session. ## Consistent Push Retry Handling To avoid restarting ingestion from scratch due to a few failed subtasks, Pinot enables extending a consistent push session across multiple task triggers: 1. On Failure: Failed subtasks do not cause progress loss. Subsequent task triggers are aware of previous progress and only generate subtasks for unread files. 2. Retry Cycle: This process continues until all files are ingested or retry limits are met. 3. Session Completion: Upon successful ingestion, segments become available for querying, and the next trigger initiates a new session. 4. Retry Exhaustion: If retries are exhausted without successful ingestion, Pinot clears session progress and begins a new ingestion attempt from scratch, that ingests all files again. ## Key Points to Note 1. The default nature of consistent push is 0 retries. This means it will throw away the successfully read files, clean up the earlier session completely, and re-ingest everything. 2. Consistent Push can be used with both Append Mode and Sync Mode. 3. Retry Tracking: Each retry count is stored within consistent push metadata. 4. Manual/Cron Triggers: Retries do not auto-trigger to allow issue resolution, it has to be configured. By default, the retry will happen with the next schedule. 5. Data Consistency During Session: Files processed in prior retries are not reprocessed if they’re modified. Only unprocessed files are picked up in subsequent retries, ensuring data integrity across retries. 6. Single Session Limit: Only one ingestion session per table can run when consistent push is enabled, with subsequent triggers paused until the current session completes. 7. Backward Compatibility: For sync mode compatibility, enableAtomicUpload functions as previously configured. 8. Since segments aren’t available for querying until the last segment is fully uploaded in Consistent Push, it may appear that more space is being used than expected during the ingestion process. This retry mechanism ensures robust and consistent data ingestion in scenarios where atomicity is critical, helping maintain data integrity while maximizing ingestion efficiency. ## Example A File Ingestion Task is scheduled to run every 4 hours, the total number of retry attempts configured is 3 with the below config: * consistentPushEnabled = true * consistentPushMaxRetries = 3 #### Scene 1: Initial Ingestion Attempt: The File Ingestion Task(FIT\_1) is scheduled for 12 PM. * 5 new files found F1, F2, F3, F4, F5 * F1 and F2 are ingested successfully, but F3, F4, and F5 fail. * As a result of failure, due to Consistent Push, F1 and F2 remain non-queryable, and no partial data is available for querying. #### Scene 2 - At 3 PM, after the initial failure, a new File, F6 is added * FIT\_Retry\_1 will run at 4 PM because the cron scheduled for FIT\_1 is every 4 hours. * FIT\_Retry\_1 will retain the original session of FIT\_1 and will retry the remaining F3, F4, and F5, and includes F6 as well. #### Scene 3 - At 3 PM, A failed file F3 is rewritten after the initial failure, let’s call the file F3.1 * FIT\_Retry\_1 will run at 4 PM because the cron scheduled for FIT\_1 is every 4 hours. * FIT\_Retry\_1 will retain the original session of FIT\_1 and now recognizes this updated version, F3.1, and ingests F3.1 along with F4 and F5. #### Scene 4 - At 3 PM, A successful file F1 is rewritten after the initial failure, let’s call the file F1.1 * FIT\_Retry\_1 will run at 4 PM because the cron scheduled for FIT\_1 is every 4 hours. * FIT\_Retry\_1 will retain the original session of FIT\_1 however, it now doesn’t recognize this updated version, F1.1, and ingests only F3, F4, and F5. This Retry succeeds. * The next scheduled run will be at 8 PM because the cron scheduled for FIT is every 4 hours. * The new FIT\_2 will not retain any context from the earlier session of FIT\_1 and will process F1.1 along with other files, if any. #### Scene 5 - At 3 PM, A successfully ingested file, F1 is deleted from the source. * FIT\_Retry\_1 will retain the original session of FIT\_1, run at 4 PM, and it will process the remaining files from the failed ingestion session: F3, F4, and F5. As F1 has been removed from the source, the segment corresponding to F1 is also removed during this retry, ensuring that the data set reflects the current state of the source files. #### Scene 6 - At 3 PM, A failed file, F3 is deleted from the source. * When FIT\_Retry\_1 runs at 4 PM, it processes F4 and F5. Since F3 no longer exists, FIT\_Retry\_1 has no context within this retry. This means that F3 will be omitted, and no segment is created for it, ensuring consistency between the source data and the ingestion process. # Delta Lake Ingestion Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/delta-lake-connector The Delta Lake connector allows ingestion from a Delta Lake data platform hosted in S3 and GCS into Pinot. The ingestion is done as a periodic pull based system that is executed by the Pinot Minion framework. You specify the period that you want the ingestion task to follow and on each trigger Minion will check if there has been a change to the Delta Lake since the last execution and, if there has, it will ingest the new data. StarTree's Delta Lake connector supports both the older Standalone Delta Lake API(v3.3.2) and the new Delta Kernel API(v4.0.0). With the Delta Kernel API, the StarTree Delta Lake connector also supports the use of [Deletion Vectors](https://docs.delta.io/latest/delta-deletion-vectors.html). ## Prerequisites * Delta Lake ingestion can only be enabled for offline Pinot tables * This connector supports Delta tables hosted on Amazon S3 and Google Cloud Storage(GCS). * This connector supports only sync mode, meaning each file in Delta Lake maps to a single segment in Pinot. Therefore, it is recommended to size the files appropriately to avoid creating an excessive number of segments. Find the [official documentation](https://docs.databricks.com/en/delta/tune-file-size.html#set-a-target-file-size) on how to tune the size of files in your Delta table. ## Limitations * Support for ADLS is not yet available. * Delta tasks do not utilize checkpoints. Consequently, no table entry is created at the ZK path PROPERTYSTORE→MINION\_TASK\_METADATA. This limitation applies to both Delta Standalone mode and Delta Kernel mode. Tasks will fail if not completed within one day, as Delta does not track progress. Subsequent triggers will generate new tasks to resume processing. ## Pinot Table Configuration The offline table task configs can be manually set with the following connection and ingestion parameters. ## Connection Parameters All the following properties are required to establish connection with Delta Tables. | Property Name | Required | Description | | - | - | - | | delta.ingestion.table.fs | Yes | Must be set to "S3" or "GCS" | | delta.ingestion.table.uri | Yes | It is the source location of the Delta Lake table. For Amazon S3, the URI should start with: `s3a:///path/to/delta-table`. For GCS, the URI should start with: `gs:///path/to/delta-table`. | | delta.ingestion.useDeltaKernel | No | When set to `"true"` this will make the Delta Lake connector use the Delta Kernel API. When not provided this will default to `false`. It should be set to `true`, when delta table is hosted on GCS | | delta.ingestion.table.kernel.maxFilesPerRun | No | Set this to limit files processed per ingestion run. The default value is -1, which means unlimited or all files. | | delta.ingestion.max.parallel.ops | No | Control parallel operations, the default value is 1 | | delta.ingestion.preserveNullValues | No | Preserve null values in complex data types(arrays, maps, structs). The default value is false. The supported values are true, and false. | | tableMaxNumTasks | No | This is used to determine how to partition ingestion jobs across Minion workers | | schedule | Yes | This sets the schedule for execution and uses the [Quartz schedule format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/tutorial-lesson-06.html) for setting the schedule | | segmentIngestionType | Yes | It should be set to REFRESH. As the complete segment is replaced every time during ingestion for a particular file. This configuration is part of `ingestionConfig.batchIngestionConfig` | | batch.segment.upload | No | Enable batch segment uploads. This is kept enabled by default. The supported values are true, and false. When enabled it requires push.mode to be set as METADATA, which is set as default. | ## Authentication Parameters ### Delta Tables hosted on S3 via Basic Authentication Use the following JSON configuration when Delta is set up on S3 with Basic Authentication ```json theme={null} "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "REFRESH" } } "task": { "taskTypeConfigsMap": { "DeltaTableIngestionTask": { "delta.ingestion.table.fs": "S3", "delta.ingestion.useDeltaKernel": "true", "delta.ingestion.table.uri": "s3a:///", "delta.ingestion.table.s3.accessKey": "", "delta.ingestion.table.s3.secretKey": "", "delta.ingestion.table.s3.region": "", "schedule": "", "tableMaxNumTasks": "" } } } ``` #### Property Descriptions | Property Name | Required | Description | | - | - | - | | delta.ingestion.table.s3.region | Yes | The AWS region that houses the Delta Lake. | | delta.ingestion.table.s3.accessKey | No | If you are using Access Key based authorization then this is for the AWS access key. You will also need the AWS Secret Key below. | | delta.ingestion.table.s3.secretKey | No | The AWS Secret Key. | ### Delta Tables hosted on S3 via IAM based Authentication ```json theme={null} "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "REFRESH" } } "task": { "taskTypeConfigsMap": { "DeltaTableIngestionTask": { "tableMaxNumTasks": "", "schedule": "", "delta.ingestion.useDeltaKernel": "true", "delta.ingestion.table.fs": "S3", "delta.ingestion.table.uri": "s3a:///", "delta.ingestion.table.s3.role.arn": "arn:aws:iam:::role/", "delta.ingestion.table.s3.region": "", "fs.s3a.assumed.role.arn": "arn:aws:iam:::role/", "fs.s3a.assumed.role.sts.endpoint": "https://sts..amazonaws.com", "fs.s3a.assumed.role.sts.endpoint.region": "", "fs.s3a.aws.credentials.provider": "ai.startree.connectors.auth.AssumedRoleCredentialsProvider", "fs.s3a.assumed.role.credentials.provider": "com.amazonaws.auth.InstanceProfileCredentialsProvider" } } } ``` #### Property Descriptions If you are using IAM to managed access to your S3 bucket then, use the following parameters: | Property Name | Required | Description | | - | - | - | | delta.ingestion.table.s3.region | Yes | The AWS region that houses the Delta Lake. | | delta.ingestion.table.s3.role.arn | Yes | `"arn:aws:iam::REPLACE_WITH_AWS_ACCOUNT_ID_WHERE_DATA_RESIDES:role/DataManagerDeltaLakeS3IAMRole"` | | fs.s3a.aws.credentials.provider | Yes | Must be `"org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider"` on StarTree Pinot versions 1.2.0\* and `"ai.startree.connectors.auth.AssumedRoleCredentialsProvider"` on StarTree Pinot versions 1.3.0\*. | | fs.s3a.assumed.role.sts.endpoint | Yes | The STS endpoint for your account (e.g., `https://sts.us-west-2.amazonaws.com`) | | fs.s3a.assumed.role.sts.endpoint.region | Yes | The Region for your account (e.g, `us-west-2`) | | fs.s3a.assumed.role.arn | Yes | The ARN For the Role that will be assumed to get access (`"arn:aws:iam::REPLACE_WITH_AWS_ACCOUNT_ID_WHERE_DATA_RESIDES:role/DataManagerDeltaLakeS3IAMRole"`) | | fs.s3a.assumed.role.credentials.provider | Yes | Must be `com.amazonaws.auth.InstanceProfileCredentialsProvider`. Used to gain a set of temporary security credentials to access an ARN that is in an account outside of the StarTree cluster. | ### Delta Tables hosted on GCS Use the following JSON configuration when Delta is set up on GCS ```json theme={null} "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "REFRESH" } } "task": { "taskTypeConfigsMap": { "DeltaTableIngestionTask": { "delta.ingestion.table.fs": "GCS", "delta.ingestion.useDeltaKernel": "true", "delta.ingestion.table.uri": "gs:///", "delta.ingestion.table.gcs.projectId": "", "delta.ingestion.table.gcs.keyContentBase64": "", "schedule": "", "tableMaxNumTasks": "" } } } ``` #### Property Descriptions | Property Name | Required | Description | | - | - | - | | delta.ingestion.table.gcs.projectId | Yes | It is the GCP project ID associated with the bucket that stores Delta Lake data. | | delta.ingestion.table.gcs.keyContentBase64 | No | It is used to specify the Google Cloud Service Account key in Base64-encoded format for authentication when accessing Delta Lake data stored in GCS. `delta.ingestion.table.gcs.keyFile` can be alternatively used instead of this parameter. See [Google documentation](https://cloud.google.com/iam/docs/keys-create-delete#creating) on how to create a service account key. | | delta.ingestion.table.gcs.keyFile | No | It specifies the path to the Google Cloud Service Account key file used for authentication when accessing Delta Lake data stored in a GCS bucket. It should be available locally in all minions. `delta.ingestion.table.gcs.keyContentBase64` can be alternatively used instead of this parameter. | ## Frequently Asked Questions > **What happens if the same row is updated multiple times in my Delta table (e.g., order status updates)?** > > StarTree uses Delta Kernel Library to get the updated list of files and uses the Minion framework to manage data segments. Every time a row is added, updated, or deleted in Delta, the underlying files change, which updates the table version. Pinot detects these changes and updates its corresponding table segments accordingly — ensuring the latest state is always reflected. > > If there are 1M rows in a table and only one row is updated. This will result into update in one segment only since file and segment has 1:1 mapping. In other words, > > * If all 1M rows are stored in a single file and hence one single segment, the entire segment will be replaced even if just one row changes. > * If the 1M rows are split across 10 files, only the relevant segment (1 out of 10) mapped to the particular file containing the updated row will be replaced. > > This ensures Pinot always reflects the latest state while maintaining efficient resource usage, depending on how the segments are structured. > **Can I configure Pinot to refresh every minute like my Delta table?** > > Yes, you can. Pinot supports high-frequency scheduling, including a 1-minute refresh. This will create one ingestion task per minute. However, if you’re targeting sub-minute or near real-time updates, a streaming ingestion approach may be more efficient. > **Does StarTree support Unity Catalog (UC) for column-level access control?** > > Not at the moment. Unity Catalog support is currently not available in the Delta Kernel libraries, so Pinot doesn’t support it either. As a workaround, you can filter out restricted columns while creating the Pinot table. > **Does Pinot always poll the S3 bucket to list files for ingestion?** > > No. Pinot uses an internal watermark system via Zookeeper to avoid scanning all files every time. It only checks for new files added after the watermark. DeltaIngestionTask (Delta ingestion) relies on Delta’s transaction logs. Starting from version 1.3.0, it no longer uses a watermark — it simply applies the differences between versions. > **Does StarTree support Delta Column Mapping?** > > Yes, We now support column mapping with both NAME and ID mode. > **Why does Pinot have more rows than my Databricks table?** > > If the source is a Databricks streaming table built by a CDC pipeline (for example a Lakeflow Connect Salesforce table), its Parquet files keep superseded row versions that Databricks hides at query time. Pinot reads the physical files and ingests those rows too. See [Fix Row Count Mismatches for Databricks CDC Delta Tables](/corecapabilities/ingestdata/recipes/delta-databricks-cdc-row-counts) for the ingestion filter that removes them. > **Why does the segment count stay higher than the number of files in my Delta table?** > > Each ingestion run that replaces segments records a lineage entry, and the replaced segments stay in the table until the hourly retention job deletes them. How long they're kept is set by `segmentsConfig.replacedSegmentsRetentionPeriod`. When it's unset, StarTree Cloud 0.16 and later default it to 1 hour; earlier releases keep replaced segments for 24 hours on `REFRESH` tables. With a frequent schedule, such as every 15 minutes, a day of replaced segments can pile up. Set `replacedSegmentsRetentionPeriod` explicitly (for example `"1h"`) to clean them up sooner. Queries aren't affected, because only the latest segments are served. `SELECT COUNT(*)` reflects the live data. # File Ingestion Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/file-ingestion-task The `FileIngestionTask` is responsible for converting files into segments. These files can originate from various sources, such as Amazon S3, Google Cloud Storage (GCS), Azure Data Lake Storage (ADLS), or a local file system. Additionally, other file systems are supported by implementing the PinotFS interface. By default, `FileIngestionTask` is designed for OFFLINE tables during the ingestion stage. It can also be configured to backfill data into REALTIME tables, but treat that as a workaround rather than the recommended approach. For certified backfill of REALTIME tables, use the [Segment Backfill Task](/corecapabilities/manage-data/segment-backfill-task) (`SegmentBackfillTask`) instead. It provides selective, atomic backfill with stronger operational guarantees. `SegmentBackfillTask` does not currently support upsert-enabled tables; support is under active development. This task can be customized to suit various use cases. For instance, you can configure it to allow the addition of new files or enable updates to existing files, depending on your specific requirements. > [!NOTE] > > Keep an eye on the data retention value set in table config when ingesting old datasets. > If the value is set too low segments will be removed by the data retention task soon after they are ingested. ## Prerequisites Given a folder of many many files, ingest all of them into segments with indices and sizes as configured. During Proof of Concepts, it's common to bootstrap Pinot tables with very large datasets. For those cases we can assume the input files won't change during ingestion. The task configurations are described below. #### Access The Input Folder | Property Name | Required | Description | | - | - | - | | inputDirURI | Yes | The input folder. | | inputFormat | Yes | The input file format. | | includeFileNamePattern | No | To filter the target input files out, e.g. to exlude mark file \_SUCCESS. It's empty by default to allow all files. The syntax is defined by [FileSystem.getPathMatcher](https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystem.html#getPathMatcher\(java.lang.String\)). | | input.fs.className | No | The className used to ingest the files. It's inferred per URI. | | input.fs.prop.\<> | No | Those props are based on which fs.className is picked. | | ~~hasDirectories~~ | No | Enable a sanity check on whether a file URI is directory. DEPRECATED, as the sanity check got optimized a lot, and it's enabled all the time. | | skipFileIfFailToInitReader | No | Skip input file if record reader fails to init for it. Enabled by default, the names and count of skipping files are tracked in task metadata for debugging. | | push.mode | No | Tar (default) or Metadata . Use Metadata push mode if controller becomes a bottlelneck when tasks upload segments. | > [!NOTE] > > The [OSS Pinot docs](https://docs.pinot.apache.org/basics/data-import/batch-ingestion#segment-push-job-type) have explained the push modes clearly. In short, with Tar mode, the segments are uploaded from tasks to controller directly; with Metadata mode, the tasks extract metadata from segments, upload segments to deep store and segment metadata to controller separately. If only a few minion tasks run in parallel, Tar mode should just work fine. If many tens or hundreds of tasks run in parallel, then better use Metadata mode. #### Control The Ingestion Parallelism | Property Name | Required | Description | | - | - | - | | tableMaxNumTasks | No | The max number of parallel tasks a table can run at any time. It's 10,000 by default. But if set to -1 explicitly, tasks are generated as many as needed to ingest all files in one round. | | taskMaxDataSize | No | The max number of bytes one task can process, to spread workload among parallel tasks. It's 1G by default. If set, taskMaxNumFiles is ignored. | | taskMaxNumFiles | No | The max number of files one task can process, to spread workload among parallel tasks. It's 32 by default. This config is ignored if taskMaxDataSize is set explicitly. | | maxNumRecordsPerSegment | No | The number of records to put in output segment, to control size of output segments. It's 5M by default. | | desiredSegmentSize | No | The segment size desired (Default is 500M. K for kilobyte, M megabyte, G for gigabyte). With this configuration, we do not need both maxNumRecordsPerSegment and taskMaxDataSize properties to be configured.. | | schedule | No | CRON per [Quartz cron syntax](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) for when the job will be routinely triggered. If not set, the task is not cron scheduled but can still be triggered via endpoint /tasks/schedule. | > [!NOTE] > > If set `tableMaxNumTasks` to -1, tasks are generated as many as needed to ingest all files in one round, with `taskMaxDataSize` to control the workload for each subtask. In this one-shot manner, user should still check subtask completion states to decide if another round is needed to retry failed subtasks, by simply triggering another task generation. > [!NOTE] > > Running the task every 10min+ is a good starting point. > You can then adjust the schedule and task parallelism to finish ingestion faster. > We don't recommend running tasks very frequently (for example, every few seconds) given schedules of tasks across all tables are handled by a single cron scheduler in the Pinot Controller, and frequent schedules may delay other tasks. #### A sample task configuration for S3 ```json theme={null} "task": { "taskTypeConfigsMap": { "FileIngestionTask": { "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "...", "input.fs.prop.secretKey": "...", "input.fs.prop.accessKey": "...", "inputDirURI": "s3://..../", "inputFormat": "parquet", "includeFileNamePattern": "glob:**/*.parquet", "taskMaxNumFiles": "32", "tableMaxNumTasks": "", "maxNumRecordsPerSegment": "5000000", "schedule": "" } } }, "tableIndexConfig": { "segmentNameGeneratorType": "normalizedDate" } ``` > [!NOTE] > > The [segmentNameGeneratorType](https://docs.pinot.apache.org/configuration-reference/job-specification#segment-name-generator-spec) parameter is useful when the time column has date formatted string values instead of numeric epoch values. > By default 'simple' generator is used and it only works with numeric epoch values. #### A sample task configuration for GCS ```json theme={null} "task": { "taskTypeConfigsMap": { "FileIngestionTask": { "input.fs.className": "org.apache.pinot.plugin.filesystem.GcsPinotFS", "input.fs.prop.projectId": "", "input.fs.prop.gcpKey": "/home/pinot/gcp/credentials.json", "inputDirURI": "gs:////", "includeFileNamePattern": "glob:**/*.parquet", "schedule": "", "inputFormat": "parquet", "tableMaxNumTasks": "", "taskMaxDataSize": "1G", "maxNumRecordsPerSegment": "2147483647", "push.mode": "metadata" } } }, ``` In the following subsections, we'll discuss some advanced features and configs when processing the input data via FileIngestionTask. #### Derive Columns from The Source File Paths This is useful when the source file directory is partitioned on some dimensions present in the file path(eg: time with day as the smallest bucket). Those dimensions can be extracted as Pinot table columns. | Property Name | Required | Description | | - | - | - | | pathColumn.``.name | Yes | The name of the value that is extracted from the file path or transformed from other values. the name should be unique in the list If `pathColumn..output` (the next property) is true, the name must match one name defined in the schema, and the value must match the `dataType` and `format` defined in the schema | | pathColumn.``.output | No | Whether the value should be output to the Pinot table. | | pathColumn.``.pathComponentStartIndex | No | The start path component index used to extract a sub part of the file path. The scheme part is omitted when calculating the number of path components. For example, `s3://bucket-name/year=2022/month=01/day=01/city=sunnyvale/file.csv` has 6 components The index is inclusive If the value is non-negative, it is 0 based The value can also be negative, where the last path component has index -1, and the first path component has index "-(the total number of path component)" If it is not specified, it is 0 | | pathColumn.``.numPathComponentsToExtract | No | The number of path components to extract starting from `pathColumn..pathComponentStartIndex`. If it is not specified, it will extract path components from `pathColumn..pathComponentStartIndex` to the last one | | pathColumn.``.charStartIndex | No | The start character index used to further extract a sub part from the part determined by `pathColumn..pathComponentStartIndex` & `pathColumn..numPathComponentsToExtract`. The index is inclusive If the value is non-negative, it is 0 based The value can also be negative, where the last path component has index -1, and the first path component has index "-(the total number of characters)" If it is not specified, it is 0 | | pathColumn.``.numCharsToExtract | No | The number of chars to extract starting from `pathColumn..charStartIndex`. If it is not specified, we will extract chars from `pathColumn..charStartIndex` to the last one | | pathColumn.``.regex | No | A regex to further extract one or more groups from the part determined by `pathColumn..pathComponentStartIndex` & `pathColumn..numPathComponentsToExtract` & `pathColumn..charStartIndex` & `pathColumn..charEndIndex` | | pathColumn.``.transform | No | A function or an expression used to transform values referred by the names to a new value. Currently, only string concatenation is supported. Values are referred by their names (the first property), the format is `${another value’s name}` This field cannot be used together with the 3rd \~ 5th properties | Note that `` is an identifier, properties with the same `` form a group and are used together to extract or transform a value. To help readers understand how it works, here is an example. Say the file path is `s3://bucket/year=2022/month=05/day=20/file.csv`, and a table has the following schema and configurations ``` { "schemaName": "...", "dateTimeFieldSpec": [ { "name": "date", "dataType": "STRING", "format": "1:DAYS:SIMPLE_DATE_FORMAT:yyyy/MM/dd" } ], ... } ``` ``` { ... "task": { "taskTypeConfigsMap": { "FileIngestionTask": { ... "pathColumn.0.name": "year", "pathColumn.0.output": "false", "pathColumn.0.pathComponentStartIndex": "1", "pathColumn.0.numPathComponentsToExtract": "1", "pathColumn.0.charStartIndex": "5", "pathColumn.0.numCharsToExtract": "4", "pathColumn.1.name": "month", "pathColumn.1.output": "false", "pathColumn.1.pathComponentStartIndex": "2", "pathColumn.1.numPathComponentsToExtract": "1", "pathColumn.1.regex": "^month=(.*)$") "pathColumn.2.name": "day", "pathColumn.2.output": "false", "pathColumn.2.pathComponentStartIndex": "3", "pathColumn.2.numPathComponentsToExtract": "1", "pathColumn.2.charStartIndex": "4", "pathColumn.2.numCharsToExtract": "2" "pathColumn.3.name": "date", "pathColumn.3.output": "true", "pathColumn.3.transform": "${year}/${month}/${day}", ... } } }, ``` Those 4 groups (indicated by 0 \~ 3) of `pathColumn` work as following: 1. The first group extracts 2022 as "year". 2. The second group extracts 01 as "month". 3. The third group extracts 01 as "day". 4. The fourth group combines "year", "month" and "day" as "date", and mark "date" as a column in the Pinot table. #### Partition Data by Sub-directory Level in The Source File Paths This feature is useful to group data from different folders and files into the same segment or set of segments. | Property Name | Description | | - | - | | subDirLevelForDataPartition | Treated `inputDirURI` as the root directory (`the level of sub-directory` = 0), `subDirLevelForDataPartition` defines the level of sub-directory used to group/partition data. Files within the same sub-directory defined by `subDirLevelForDataPartition` are grouped into the same segment or set of segments As a special case, files with sub-directory levels less than `subDirLevelForDataPartition` are treated as a special group, and also are grouped into the same segment or set of segments | Note that the configuration `taskMaxNumFiles` is still respected when `subDirLevelForDataPartition` is configured. To help readers understand how it works, here is an exmaple. Given the following list of files and the following configurations ``` s3://bucket/a/c/file1.csv s3://bucket/a/c/file2.csv s3://bucket/a/file1.csv s3://bucket/a/file2.csv s3://bucket/b/c/file1.csv s3://bucket/b/c/file2.csv s3://bucket/b/c/file3.csv s3://bucket/b/c/file4.csv s3://bucket/b/file1.csv s3://bucket/b/file2.csv ``` ``` { ... "task": { "taskTypeConfigsMap": { "FileIngestionTask": { ... "inputDirURI": "s3://bucket", "subDirLevelForDataPartition": "2", "taskMaxNumFiles": "3", ... } } }, ``` 1. Files "s3://bucket/a/c/file1.csv" and "s3://bucket/a/c/file2.csv" will be grouped together. 2. Files "s3://bucket/a/file1.csv", "s3://bucket/a/file2.csv" and "s3://bucket/b/file1.csv" will be grouped together. 3. Files "s3://bucket/b/c/file1.csv", "s3://bucket/b/c/file2.csv" and "s3://bucket/b/c/file3.csv" will be grouped together. 4. File "s3://bucket/b/file2.csv" is a group. 5. File "s3://bucket/b/c/file4.csv" is a group. > [!NOTE] > > Combined together with the feature `Derive Columns from The Source File Paths`, one can achieve the goal of partitionning data by columns extracted from file paths. #### Other Implicit Configurations Data records can be partitioned and sorted based on configs in [tableIndexConfig](https://docs.pinot.apache.org/configuration-reference/table#table-index-config) and transformed based on configs in [ingestionConfig](https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations). The FileIngestionTask uses the same segment processing library used by [MergeRollupTask](https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task#config) and [RealtimeToOfflineTask](https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows#config), which means it can merge/rollup the data records when generating the segments in similar way. Those configs to tweak the segment processing library can be set in task configs e.g. `mergeType`, `.aggregationType`, `windowStartMs`, `windowEndMs`, etc. You can find some examples in the Pinot documentation for MergeRollupTask and RealtimeToOfflineTask. ## Sync mode This is for cases where the existing files are updated regularly and users want to keep the segments and files in sync. Most of the task configs are the same as those used to Bootstrap, with a few exceptions like setting `enableSync`. The `segmentNameGeneratorType` should be `fixed` in sync mode, but it is set automatically if omit. Below is a sample task configuration: ```json theme={null} "task": { "taskTypeConfigsMap": { "FileIngestionTask": { "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "...", "input.fs.prop.secretKey": "...", "input.fs.prop.accessKey": "...", "inputDirURI": "s3://.../", "inputFormat": "json", "includeFileNamePattern": "glob:**/*.json.gz", "tableMaxNumTasks": "", "schedule": "", "enableSync": "true" } } }, "tableIndexConfig": { "segmentNameGeneratorType": "fixed", } ``` > [!NOTE] > > 1. One file is mapped to one segment. So the file is better to be well prepared ahead, like the size. > 2. File update is detected by file modification time alone. > 3. The number of files is should be less than 30K, because one file generates one segment and it's better to limit the total number of segments for a table. > However, this is a soft limit and can be tuned. ## Sync mode with atomic switch A sync mode with atomic switch ensures that data in the Pinot table is consistent both during and after file ingestion. The table remains unchanged (nothing ingested) or successfully updated (all files ingested, and new segments made available). After the ingestion task is complete, the table atomically switches to the new set of segments, and the old segments are purged. If you want to handle deletes, use this in combination with Sync mode. It means setting both `enableSync` and `consistentPushEnabled` to true. Below is a sample task configuration using `consistentPushEnabled` flag : ```json theme={null} "task": { "taskTypeConfigsMap": { "FileIngestionTask": { "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "...", "input.fs.prop.secretKey": "...", "input.fs.prop.accessKey": "...", "inputDirURI": "s3://.../", "inputFormat": "json", "includeFileNamePattern": "glob:**/*.json.gz", "tableMaxNumTasks": "", "schedule": "", "enableSync": "true", "consistentPushEnabled": "true" } } }, "tableIndexConfig": { "segmentNameGeneratorType": "fixed", } ``` ## Incremental ingestion This is for cases where the new files can be added during ingestion, and you want the tasks to dectect and ingest the new files automatically. The sync mode can handle some of those cases subject to the aforementioned limitations. To be more scalable, you can enable incremental ingestion like below. The input folders must be date partitioned. ``` e.g. /YYYY/MM/DD/ /2022/05/11/000000.csv /2022/05/11/000001.csv /2022/05/11/000002.csv /2022/05/12/000000.csv /2022/05/12/000001.csv e.g. /year=YYYY/month=MM/day=DD/ /year=2022/month=05/day=11/000000.csv /year=2022/month=05/day=11/000001.csv /year=2022/month=05/day=11/000002.csv /year=2022/month=05/day=12/000000.csv /year=2022/month=05/day=12/000001.csv ``` Input folders should be added chronologically, and files should be added alphabetically. For example: `/2022/05/11/` should be added before `/2022/05/12/`, and `/2022/05/11/000000.csv` should be added before `/2022/05/11/000001.csv` > [!NOTE] > > There are a few requirements on the date values in the file paths: > > 1. The year, month and day are numbers. Year has four digits; month and day have two digits, so you'll need to pad with zeroes e.g. `05`. > 2. It’s in year, month, day order. It doesn’t matter if you use `/` or `-` as separator, or if you have a prefix like year=2022/month=05. A sample task configuration is shown below. In addition to the task configs explained for Bootstrap above, you need to add `checkpointTTLSec` to enable incremental ingestion. With this config, the checkpoints are kept around for as long as TTL so that files added within this time window can be detected. ```json theme={null} "task": { "taskTypeConfigsMap": { "FileIngestionTask": { "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "...", "input.fs.prop.secretKey": "...", "input.fs.prop.accessKey": "...", "inputDirURI": "s3://..../", "inputFormat": "csv", "includeFileNamePattern": "glob:**/*.csv", "checkpointTTLSec": "259200", "taskMaxNumFiles": "32", "tableMaxNumTasks": "", "maxNumRecordsPerSegment": "5000000", "schedule": "" } } }, "tableIndexConfig": { "segmentNameGeneratorType": "normalizedDate" } ``` > [!NOTE] > > More about checkpointTTLSec: > > 1. It's 0 by default, to remove checkpoints once the input files are confirmed as ingested and bump the watermark ahead, used by the Bootstrap use case. > FYI, watermark is an internal track of the last file that's got ingested. > Files before it are ingested and files are scanned in alphabetical order. > 2. It can be -1, to keep all checkpoints and disable watermark, so that files can be added in any daily subfolders and be detected and ingested. > This way results in many checkpoints in ZooKeeper and should work fine only for small input folder. > 3. It can be a positive values like 259200 (3 days) to allow new files to be added in most recent few daily subfolders. > [!NOTE] > > The incremental ingestion only detects whether new files are added. This is different from the sync mode where both new files and updates made on existing files are detected. ### User driven ingestion There are cases where the automation aforementioned doesn't work with the existing input folders. This could be because the folder is not date partitioned as assumed by the automation. For these cases, APIs are provided for you to fully control the ingestion process. #### Start ingestion ```bash theme={null} POST /tasks/execute with request body like below to config the ingestion job { "taskType": "FileIngestionTask", "tableName": "
", "taskName": "myTask-0", "taskConfigs": { "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "...", "input.fs.prop.secretKey": "...", "input.fs.prop.accessKey": "...", "inputDirURI": "s3://..../", "inputFormat": "parquet", "includeFileNamePattern": "glob:**/*.parquet", "taskMaxNumFiles": "32", "tableMaxNumTasks": "", "maxNumRecordsPerSegment": "5000000" } } ``` > [!NOTE] > > 1. The taskName can be set explicitly or auto generated if omit. > 2. There is no need to set the cron schedule, as this POST call triggers the ingestion. > 3. Can set a large value for `tableMaxNumTasks` so that enough tasks are generated to ingest all input files in the folder. > But no problem if more tasks are needed as we can call this endpoint again and again based on the ingestion progress. #### Poll progress ```bash theme={null} GET /tasks/subtask/{taskName}/state with the taskName used to start ingestion ``` This returns if all subtasks are COMPLETED or any is failed with ERROR. If there are failed subtasks, call `/tasks/execute` again with same task configs but a different taskName. Don't worry about duplicating or missing any files, as there is internal progress metadata to make sure files are ingested exactly once even though the same input folder is ingested again and again. If all subtasks are completed, also call `/tasks/execute` again with same task configs but a different taskName just to make sure no files are left behind. The response will say that no subtasks are generated when all files in the folder are ingested. #### Reset ingestion Before ingesting a new folder, the progress metadata in ZooKeeper has to be reset. The following request deletes the progress metadata kept in ZK. ```bash theme={null} DELETE /tasks/{taskType}/{tableNameWithType}/metadata ``` #### Remove the schedule/cron parameter from the table's task configuration Scheduled and user-driven task triggers should not co-exist. This step is to remove the scheduled trigger, so that the scheduled task no longer runs. In the table config, look for the following and remove the **schedule** parameter. ```bash theme={null} "task": { "taskTypeConfigsMap": { "FileIngestionTask": { "input.fs.prop.secretKey": "", "**schedule**": "", ``` ## Monitoring There are metrics as listed in [OSS Pinot docs](https://docs.pinot.apache.org/basics/components/minion#task-related-metrics) to be collected and plotted via tools like Prometheus and Grafana. It's also very handy to use Pinot UI to check task runtime status and completion results [for example](https://docs.pinot.apache.org/basics/components/minion#task-manager-ui). # Minion Task Orchestration Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration Automatically chain multi-batch Minion tasks to completion using task plans, without manual re-triggers or a fixed schedule. > [!NOTE] > > Minion Task Orchestration is available starting in **StarTree release 0.15.0**. Cluster-level orchestration is **enabled by default as of release 0.16.0**; in 0.15.0 it is off by default and must be turned on explicitly with the `controller.startree.task.manager.enableTaskOrchestration` property. ## Overview Some Minion tasks — such as large ingestion jobs, purges, or exports — process too much data to finish in a single run. They're broken up into multiple **batches**, and today each batch normally requires a separate trigger: either you call the task API again, or you wait for the next cron run. **Minion Task Orchestration** removes that manual step. When enabled for a task, StarTree Cloud tracks the multi-batch job as a **task plan**: a single trigger creates the plan, and the controller automatically generates and submits each subsequent batch as soon as the previous one finishes, until there's no more work left to do. A task plan is a long-running orchestration for one **table + task type** combination. Only one active plan is allowed per table and task type at a time, and the plan tracks overall status, its batches, and progress until it reaches a terminal state (`COMPLETED`, `CANCELLED`, or `FAILED`). Task orchestration is supported for both the ad hoc [Execute API](/api-reference/task/execute-a-task-on-minion) and scheduled (cron) task triggers. ## How is this different from a normal task run? | Aspect | Normal task run | Orchestrated task run | | :- | :- | :- | | **Trigger** | Each run is independent (ad hoc API call or cron). | One trigger creates a **plan**; later batches are driven automatically by batch completion. | | **Scope** | One invocation generates one batch of subtasks. | One plan spans multiple batches over time. | | **Next batch** | You must trigger again, or wait for the next cron run. | The controller automatically submits the next batch when the current one finishes. | | **Progress tracking** | Only the Helix task state is available. | A task plan tracks status, batch history, and progress, queryable via API. | | **Concurrency** | Multiple runs for the same table/task can overlap. | At most one active plan per table and task type; new triggers are rejected or skipped while a plan is active. | In short: a normal run produces one batch per trigger, while an orchestrated run produces a chain of batches from a single trigger, running until the job is done. ## Enabling task orchestration Task orchestration requires the feature to be enabled at the cluster level, and then opted into per task. ### Cluster-level control As of **release 0.16.0**, task orchestration is **enabled by default** at the cluster level. (In 0.15.0, where the feature was introduced, it is off by default and must be turned on explicitly.) It can be turned off entirely with the controller configuration property below, which acts as a cluster-wide kill switch — when disabled, no task plans are created or progressed, regardless of any per-task setting. | Property | Default | Description | | :- | :- | :- | | `controller.startree.task.manager.enableTaskOrchestration` | `true` | Enables the orchestration infrastructure cluster-wide. | Enabling this property does not by itself orchestrate any task — each task must still opt in individually, as described below. ### Enabling for an ad hoc trigger Add `enableTaskOrchestration` to the task configuration when calling the [Execute API](/api-reference/task/execute-a-task-on-minion): ```json theme={null} { "taskType": "SegmentPurgeTask", "tableName": "
_OFFLINE", "taskConfigs": { "enableTaskOrchestration": "true" } } ``` * Only [task types that support orchestration](#supported-task-types) will use this path; other task types ignore the flag and run as before. * If a plan is already active for that table and task type, the ad hoc request is **rejected** until the existing plan completes or is aborted. ### Enabling for a scheduled (cron) trigger Add `enableTaskOrchestration` alongside the `schedule` key in the table configuration: ```json theme={null} { "task": { "taskTypeConfigsMap": { "SegmentPurgeTask": { "schedule": "", "enableTaskOrchestration": "true" } } } } ``` When a cron trigger fires and orchestration is enabled, the controller creates a task plan from the table configuration and begins multi-batch orchestration automatically. > [!NOTE] > > If a scheduled trigger fires while a plan is already active for that table and task type, the trigger is **skipped** (not rejected) and the existing plan continues unaffected. This is different from the ad hoc path, which rejects the request outright. If the task's generator doesn't support orchestration, the trigger automatically falls back to the normal, one-shot batch generation. ## Supported task types Orchestration is available only for task types whose generator implements batch-by-batch generation. Currently supported task types: | Task type | Description | | :- | :- | | [**File Ingestion Task**](/corecapabilities/ingestdata/adv-concepts/batch/file-ingestion-task) | Supports all ingestion modes (sync/append, with or without consistent push) with automatic batch chaining and retries. Also supports consistent push with full-swap mode, completing a swap reliably across retries while avoiding accidental duplicate swaps. | | [**Segment Purge Task**](/corecapabilities/manage-data/purge-task) | Chains batches automatically for large purge jobs that span more segments than a single batch can process. | | [**Data Export Task**](/corecapabilities/external-table/data-export-task) | Exports completed segments from a source real-time table to an external target. Segments beyond the per-batch limit, and any pending commits, automatically flow into subsequent batches until the export queue is fully drained. | For any other task type, enabling `enableTaskOrchestration` has no effect — the task always uses the standard, one-shot generation flow. ## Monitoring task plans via API When available, the controller exposes REST endpoints for inspecting and managing task plans. All endpoints require the same authentication and table-level authorization as other task APIs. > [!TIP] > > If you'd rather not query the API directly, [Task Observability](/corecapabilities/observability/minion-task-runs) surfaces this same task plan data — list, drill-down, abort, and schedule-now — directly in the Data Portal UI. | Method | Path | Description | | :- | :- | :- | | `GET` | `/tasks/taskPlans/isActive/{taskPlanId}` | Check whether a task plan is currently active. | | `GET` | `/tasks/taskPlans/{planId}` | Get the full details of a task plan by ID. | | `GET` | `/tasks/taskPlans` | List all task plan IDs for a table and task type. Requires `tableNameWithType` and `taskType` query parameters. | | `GET` | `/tasks/taskPlans/active` | List all active task plans, optionally filtered by `tableNameWithType` and/or `taskType`. | | `GET` | `/tasks/taskPlans/active/ids` | List active task plan IDs only, with the same optional filters. | | `DELETE` | `/tasks/taskPlans/{planId}` | Abort a task plan. Sets its status to `ABORTING`; no new batches are generated, though any already-running batch may still complete. Poll the `isActive` endpoint to confirm the plan has fully stopped. | ### Task plan data model Each task plan returned by the API contains the following fields: | Field | Type | Description | | :- | :- | :- | | `planId` | String | Unique ID, formatted as `____`. | | `tableNameWithType` | String | The table this plan belongs to (e.g. `myTable_OFFLINE`). | | `taskType` | String | The task type this plan orchestrates (e.g. `SegmentPurgeTask`). | | `status` | Enum | One of `ACTIVE`, `COMPLETED`, `CANCELLED`, `ABORTING`, `FAILED`. | | `source` | String | How the plan was created: `ADHOC_CONFIG` or `TABLE_CONFIG`. | | `properties` | Map\ | Plan-level configuration used to generate each batch. | | `batches` | List | Ordered list of batch records submitted so far (see below). | | `inputsToProcess` | Long | Total number of input units (e.g. segments) to process. | | `inputsBeingProcessed` | Long | Number of input units currently in flight. | | `inputUnit` | String | The unit the counts above are measured in (e.g. `"segments"`). | | `statusMessage` | String | Human-readable status, such as the reason for an abort or completion. | | `customStats` | Map\ | Optional, task-type-specific statistics. | Each entry in `batches` includes: | Field | Type | Description | | :- | :- | :- | | `batchSequence` | Integer | 0-based sequence number of the batch. | | `submittedAtMs` | Long | Time (epoch ms) the batch was submitted. | | `submittedTaskName` | String | Name of the parent task submitted for this batch. | | `taskState` | Enum | Current state of that batch's parent task (e.g. `NOT_STARTED`, `IN_PROGRESS`, `COMPLETED`). | ## Task plan cleanup Task plans are cleaned up automatically when their table is dropped, so deleted tables don't leave orphaned plans behind: * Dropping a table synchronously removes all of its task plans (across every task type) before the table's metadata is torn down. * A periodic background sweep also reaps any plans whose table no longer exists, catching cases the synchronous cleanup may have missed. This sweep runs on an interval controlled by: | Property | Default | Description | | :- | :- | :- | | `controller.startree.task.manager.orphanedTaskPlanCleanupIntervalInSeconds` | `28800` (8 hours) | How often the controller sweeps for and removes plans belonging to deleted tables. | ## Observability and metrics Task orchestration emits controller-side metrics scoped to `tableNameWithType` and `taskType`, so you can monitor and alert on orchestration health. Global gauges are emitted per controller — since only the controller that leads a table progresses its plans, aggregate global metrics across all controllers when building dashboards. ### Meters | Metric | Meaning | | :- | :- | | `taskPlanAborted` | A plan was aborted, either due to a failure-threshold breach or a generator-driven abort. | | `orchestrationCycleFailure` | An exception occurred while progressing a plan. | | `taskPlanProgressionBlocked` | Plan progression was skipped because the task queue is paused, or resource-utilization limits were hit. Stays set while blocked and clears once progression can resume. | | `scheduledTriggerActivePlanConflict` | A scheduled trigger was skipped because a plan was already active for that table and task type. | | `taskGenerationFailureCount` | Task generation failed for the table/task type. | ### Gauges | Metric | Meaning | | :- | :- | | `taskPlanInputsToProcess`, `taskPlanInputsBeingProcessed`, `taskPlanInputsProcessed` | Plan progress: total, in-flight, and cumulative-completed input units. | | `activeTaskPlansCount` | Number of `ACTIVE` plans this controller is currently progressing. | | `taskPlansAbortingCount` | Number of plans stuck in `ABORTING`, waiting for in-flight subtasks to terminate. | | `orchestrationTimeSinceLastPollMs` | Time since the plan-completion polling job last ran. Rising continuously would indicate the polling job has stalled. | ## FAQs ### Do I need to change anything for task types that don't support orchestration? No. Setting `enableTaskOrchestration` on an unsupported task type has no effect — it continues to use the standard one-shot generation flow. ### What happens if I trigger an ad hoc run while a plan is already active? The request is rejected with an error. Wait for the active plan to complete, or abort it using the `DELETE /tasks/taskPlans/{planId}` endpoint, before triggering again. ### What happens if a scheduled (cron) trigger fires while a plan is active? Unlike the ad hoc path, the scheduled trigger is silently skipped rather than rejected, and the `scheduledTriggerActivePlanConflict` metric is incremented. The existing plan is unaffected and continues to progress. ### How do I stop an in-progress plan? Call `DELETE /tasks/taskPlans/{planId}`. This moves the plan to `ABORTING`; any batch already running is allowed to finish, but no new batch is generated. Poll `GET /tasks/taskPlans/isActive/{planId}` until it reports the plan is no longer active. ### How can I tell whether a task type supports orchestration? Check the [supported task types](#supported-task-types) table. If a task type isn't listed there, enabling `enableTaskOrchestration` for it has no effect. # Offline Upserts Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/offline-upserts ## Introduction Offline upserts allow an OFFLINE Pinot table to upsert records by primary key across multiple segments, presenting query results as if only the most-recent version of each record exists. In a standard OFFLINE table every segment is independent — pushing two segments that each contain a row for the same primary key means queries will return both rows. Offline upserts fix this: when a new segment is pushed, the server resolves conflicts with all previously loaded segments and surfaces only the winning record per primary key. ### Record precedence ordering The row with the greater comparison value wins. The comparison value comes from `upsertConfig.comparisonColumns` when it is set, and from the table's `timeColumnName` otherwise. A table needs one of the two, or the server refuses to load it. Segment push time does not pick the winner. It only breaks a tie between two rows with the same comparison value, and then the row from the segment pushed most recently wins. If you want the most recent push to win regardless of the data, add a column that increases with every push, such as a batch timestamp, and set it as the comparison column. The full set of tie breakers is described under [Record Comparison and Tie Breakers](/corecapabilities/manage-data/offheap-upsert#record-comparison-and-tie-breakers). *** ## Use Cases | Scenario | Why offline upserts help | | - | - | | Incremental corrections | A small fraction of rows in the table has changed (e.g., late-arriving corrections, status updates) and you want to push only the changed rows as a new segment without re-ingesting the whole table. | | Snapshot uploads with overlap | An upstream job exports daily snapshots that may overlap (the same primary key appears in yesterday's and today's export). Offline upserts deduplicate automatically. | Offline upserts are a good fit when the **number of rows being updated is a small-to-moderate fraction of the total table size** (roughly less than \~40%). For larger fractions — where you are effectively replacing the whole table — consider Atomic Sync (see below). *** ## How to Configure Offline Upserts Enabling offline upserts requires changes to three places: the schema, the table config, and (optionally) the ingestion task config. ### 1. Schema — declare primary key columns ```json theme={null} { "schemaName": "
", "primaryKeyColumns": ["userId"], "dimensionFieldSpecs": [ { "name": "userId", "dataType": "INT" }, { "name": "status", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "score", "dataType": "FLOAT" } ], "dateTimeFieldSpecs": [ { "name": "eventTime", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` `primaryKeyColumns` can contain multiple columns — Pinot hashes them together as a composite key. ### 2. Table config Three sections of the table config must be set together: `upsertConfig`, `segmentPartitionConfig`, and `routing`. ```json theme={null} { "tableName": "
", "tableType": "OFFLINE", "segmentsConfig": { "timeColumnName": "eventTime" }, "upsertConfig": { "mode": "FULL", "metadataManagerClass": "ai.startree.pinot.upsert.rocksdb.RocksDBTableUpsertMetadataManager" }, "indexingConfig": { "segmentPartitionConfig": { "columnPartitionMap": { "userId": { "functionName": "Murmur", "numPartitions": 4 } } } }, "routing": { "instanceSelectorType": "strictReplicaGroup" } } ``` **Key points:** * `upsertConfig.mode` must be `"FULL"`. Partial upsert mode is not supported for offline tables (see [Limitations](#limitations)). * `upsertConfig.metadataManagerClass` must point to the StarTree RocksDB implementation. If omitted the in-heap OSS default is used, which does not persist metadata across server restarts and is not recommended for production. * `segmentPartitionConfig` on the **primary key column** is recommended. The partition function and count define how the data is partitioned and also governs the overall scalability. Every segment for a given primary key must land on the same server — without this the server cannot see all versions of a key and deduplication will be incorrect. In general, the recommendation is to use a high partition count (eg: 128) to account for organic growth. Note that you cannot change this post table creation. * `instanceSelectorType: strictReplicaGroup` ensures that a query is routed to exactly one replica group. This is required so that the server's local RocksDB view (which is per-partition) is authoritative for the keys it owns. **Optional: explicit comparison column** ```json theme={null} "upsertConfig": { "mode": "FULL", "metadataManagerClass": "ai.startree.pinot.upsert.rocksdb.RocksDBTableUpsertMetadataManager", "comparisonColumns": ["eventTime"] } ``` When `comparisonColumns` is set, the row with the highest value in that column wins. When it is omitted, the table's `timeColumnName` is the comparison column. Segment push time only breaks ties between rows with the same comparison value. ### 3. Partitioning upstream data with FileIngestionTask For offline upserts to work correctly, **every segment must be partitioned on the primary key**: all rows with the same primary key value must reside in the same server. `FileIngestionTask` handles repartitioning automatically, even when the upstream S3 data is not pre-partitioned by primary key. On each execution the task executor reads `segmentPartitionConfig` from the table config and passes it to `SegmentProcessorFramework`, which physically sorts all ingested rows into per-partition buckets by applying the configured partition function (e.g., Murmur) across the configured number of partitions. Each output segment contains only rows that hash to a single partition bucket, satisfying the upsert requirement without any extra config. The `segmentPartitionConfig` in the table config acts as the contract: the task executor reads it to know how many partitions to create and which hash function to use, producing pre-partitioned segments that satisfy the upsert requirement automatically. *** ## Offline Upserts vs. Atomic Sync StarTree also offers **Atomic Sync** (via `consistentPushSwapEnabled=true` on `FileIngestionTask`), which replaces the entire table atomically. Both features update an offline table from S3, but they are designed for different scenarios. | Dimension | Offline Upserts | Atomic Sync | | - | - | - | | **What it does** | Merges new/updated rows into existing segments. Only changed rows need to be pushed. | Rebuilds and atomically swaps **all** segments in the table from the input source. | | **Granularity** | Row-level upsert/deduplication by primary key. | File / Partition / Table-level replacement. | | **Data volume per push** | Small-to-moderate fraction of the table (works well under \~40% of total rows being upserted). | Optimised for full partition or table refreshes (essentially 100% of rows). | | **Query consistency during push** | New segments become visible as they are pushed; there is a brief window where both old and new versions of a row are in the table and the upsert filter resolves them. | Old and new sets of segments are swapped atomically — queries see either the old or the new version of the table, never a mix. | | **Index support** | No StarTree index support (see [Limitations](#limitations)). | StarTree indexes are supported which significantly improves query performance | | **Partial upsert** | Not supported (FULL mode only). | Not applicable — full re-ingestion replaces all data. | | **When to choose** | You are pushing incremental corrections, late data, or a small update batch. The primary key is known and stable. | You are regenerating the entire table (e.g., reprocessing pipeline, full restatement), or you require atomic cutover. | **Rule of thumb:** if less than \~40% of the table's rows are changing in a given push cycle, offline upserts are the better fit — they avoid re-ingesting unchanged data and are cheaper to operate. Above \~40%, or when you need atomic visibility guarantees, use Atomic Sync. *** ## Limitations **No partial upsert support.** Only `mode: FULL` is supported for offline tables. FULL mode means the entire winning row replaces the entire losing row. Column-level merge strategies available in PARTIAL mode for realtime upsert tables are not available. Note that this is generally ok since column level update pattern is typically found in CDC/streaming use cases not batch. **No StarTree index.** Columns in an offline upsert table cannot have a StarTree (multi-dimensional pre-aggregation) index configured. The controller will reject such a table config at creation time. **Segment push time breaks ties, and push order cannot be undone.** When two rows for the same primary key have the same comparison value, the row from the segment pushed most recently wins. Pushing an older segment after a newer one makes the older data win for those tied rows. Use a comparison column that reflects the real order of your data, such as a business event timestamp, if push order cannot be guaranteed. # Batch Ingestion in Apache Pinot with Minions Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/overview Batch ingestion in Apache Pinot is the process of importing large volumes of static or historical data from sources like cloud storage (S3, GCS, ADLS), distributed file systems (HDFS), or SQL databases(Snowflake, BigQuery) into Pinot’s offline tables. These tables are optimized for high-performance analytical queries. Traditionally in Apache Pinot, batch ingestion relied on external orchestration frameworks such as Apache Spark or Hadoop. However, with the introduction of Pinot Minions, this process can now be handled natively within the Pinot cluster itself. Minions allow ingestion workflows and post-processing tasks to be automated, distributed, and managed internally without requiring external job schedulers. ## Pinot Minions: Cluster-Native Ingestion & Processing Pinot Minions are lightweight, background workers that execute asynchronous tasks on Pinot segments. They are orchestrated by the Pinot Controller and coordinated using Apache Helix, which handles task assignments, execution tracking, and failure recovery. Minions enable Pinot to run ingestion and data lifecycle operations, such as segment generation, merging, transformation, and movement, without interrupting query-serving processes. They provide a scalable and fault-tolerant mechanism to handle ingestion and processing tasks in a distributed manner. ## Common Pinot Minion Ingestion Tasks The **File Ingestion Task** offers a simpler way to ingest files from cloud storage(S3, GCS, ADLS) directly into Pinot without external orchestration. It reads input data from directories in almost all formats and not limited to JSON, CSV, Avro, or Parquet. It transforms them into segments ready for ingestion. This task is often used for small to large scale ingestion jobs where lightweight, frequent ingestion is necessary—for example, when periodically ingesting new logs or exports dropped into a cloud storage bucket or one time ingestion of historical data. More details on the File Ingestion Task can be found [here](/corecapabilities/ingestdata/adv-concepts/batch/file-ingestion-task). The **Delta Ingestion Task** is particularly useful to ingest data from Delta Lake. Especially when only a subset of data needs to be ingested, such as newly arrived records for a given time range. Instead of reprocessing an entire dataset, this task ingests only the delta—defined by a start and end timestamp—thereby improving efficiency and reducing processing time. It’s well suited for append-only datasets like event logs, transactions, or metric streams that arrive periodically. More details on the Delta Ingestion Task can be found [here](/corecapabilities/ingestdata/adv-concepts/batch/delta-lake-connector). Another critical task is the **SQL Connector Batch Push Task**, which allows Pinot to directly fetch data from relational databases such as Snowflake or Google BigQuery using SQL queries. With this connector, you can write a SQL query that selects a subset of rows from a source table, and Pinot will use that result set to generate and ingest offline segments. This provides a seamless pipeline for importing analytical data from structured databases into Pinot without intermediate ETL systems. More details on the SQL Connector Batch Push Task can be found [here](/corecapabilities/ingestdata/adv-concepts/batch/sql-connector). ### How to Schedule or Trigger Minion Tasks Minion tasks can be scheduled automatically or triggered manually. For scheduled execution, tasks are defined in the Pinot table configuration using the tasks section. Each task type can be associated with a frequency—for example, a File Ingestion Task might run every 12 hours. These frequencies are parsed and enforced by the Pinot Controller, which ensures that tasks are launched at regular intervals and assigned to available Minion instances. For manual execution, Pinot exposes REST endpoints that can be used to trigger tasks on demand. By sending a POST request to the Controller’s */tasks/schedule* endpoint and specifying the task type, operators can initiate ingestion or transformation processes at any time. # SQL Ingestion Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/sql-connector The SQL connector allows ingestion from a SQL table (E.g. Snowflake or BigQuery) into Pinot. This is done through the Pinot Minion framework, which will ingest data on a scheduled basis. The connector follows a pull model which will ingest data over a given time period. Currently, SQL ingestion from Snowflake and BigQuery is supported. A templatized SQL query must provided that contains a time filter using the `BETWEEN` operator like the following: ```sql theme={null} SELECT col1, col2 FROM myTable WHERE timeColumn BETWEEN $START AND $END ``` > [!NOTE] > > The filter `BETWEEN $START AND $END` must exist somewhere in the query. > Joins in the templatized SQL are not supported at the moment. The connector will continually ingest data from a user provided start time until the current time. The connector will attempt to process time periods since the last execution time. ## Prerequisites * Data is assumed to be immutable after ingestion, since the connector uses watermark and segment time boundary metadata for one time ingestion. * SQL ingestion can only be enabled for offline Pinot tables. * The SQL source table should have a time column and the user must know the format of the time column (millisecondsSinceEpoch, secondsSinceEpoch, hoursSinceEpoch, or ISO format e.g. `yyyy-MM-dd`). ## Pinot Table Configuration The offline table task configs can be manually set with the following connection and ingestion parameters. ### Snowflake connection parameters All the following properties are required to establish connection with Snowflake. | Property Name | Required | Description | | - | - | - | | sql.snowflake.user | Yes | Snowflake username | | sql.snowflake.password | Yes | Snowflake password | | sql.snowflake.account | Yes | Snowflake account name | | sql.snowflake.db | Yes | Snowflake database | | sql.snowflake.schema | Yes | Snowflake schema (collection of tables/views) | ### BigQuery connection parameters All the following properties are required to establish connection with BigQuery. | Property Name | Required | Description | | - | - | - | | sql.bigquery.projectid | Yes | BigQuery project ID | In addition, a Google service account key must be exposed through the environment variable GOOGLE\_APPLICATION\_CREDENTIALS. For instructions on how to create a sercice account key, reference the [Google cloud documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key). ### Ingestion parameters These ingestion parameters apply to all SQL connectors. | Property Name | Required | Description | | - | - | - | | sql.queryTemplate | Yes | Templatized query to pull data from Snowflake and pull into Pinot | | sql.timeColumnFormat | Yes | The format of the time column in their templatized query. This is needed to understand how to break the query into batches. | | sql.timeColumnName | Yes | The time column name provided in the templatized query. | | sql.startTime | Yes | Marks the beginning time of the Snowflake table for which to ingest. This is used to identify time buckets. Should be in the format of their Snowflake table time column. | | sql.endTime | No | Provide if a one time bootstrap job is desired. Continuous ingestion up to the current time will be disabled if provided. | | sql.bucketTimePeriod | No | If you want to override the segment time periods in case you are noticing that the segments created are too small or too large. This parameter will determine the time buckets for the Pinot segments in relation to the sql.startTime value. All segments will be bucketed by this length. E.g. A 5 day bucket period would be expressed as “5d”. A 1 hour bucket period would be expressed as “1h”. Valid period units are d (days), (h) hours, (m) minutes, and (s) seconds. | | sql.delayTimeLength | No | This will be a time value that will determine the latest time value for which data is ingested from Snowflake. \$END = CURRENT\_TIME - DELAY. If omitted, defaults to 1. E.g. A 5 day delay from the current time of execution would be expressed as “5d”. A 1 hour delay would be expressed as “1h”. Valid time delay length units are d (days), (h) hours, (m) minutes, and (s) seconds. | | schedule | No | CRON expression for when the job will be routinely triggered. Scheduling support is natively built into Minion jobs (see PinotTaskManager). If not provided, the ingestion task can be manually triggered via the Pinot task API | Below is a sample task configuration for Snowflake ingestion: ```json theme={null} "task": { "taskTypeConfigsMap": { "SqlConnectorBatchPushTask": { "sql.snowflake.user": "", "sql.startTime": "1644321600000", "sql.timeColumnName": "timeColumn", "sql.snowflake.password": "", "sql.snowflake.account": "", "sql.delayTimeLength": "0d", "schedule": "", "sql.timeColumnFormat": "millisecondsSinceEpoch", "sql.bucketTimePeriod": "1d", "sql.snowflake.db": "", "sql.queryTemplate": "SELECT * FROM myTable WHERE timeColumn BETWEEN $START AND $END", "sql.className": "ai.startree.connectors.sql.snowflake.SnowflakeConnector", "sql.snowflake.schema": "" } } } ``` # AZ Aware Kafka ingestion in Pinot Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/az-aware-kafka Guide for setting up AZ aware Kafka ingestion ## **Cross AZ data transfer cost** In a StarTree Pinot cluster, Pinot servers utilize low-level Kafka consumers to retrieve data from Kafka brokers. When a Pinot consumer operates in a different Availability Zone than the Kafka broker hosting the required partition, each fetch request generates cross-AZ network traffic. Cross-AZ traffic for Kafka consumers creates several challenges: * Increased costs: Cross-AZ data transfer incurs additional charges * Higher latency: Network requests across zones introduce additional delay * Reduced reliability: Cross-zone communication increases potential failure points Implementing AZ-aware consumption in StarTree pinot provides: * Improved application performance through reduced latency * Significant cost savings on data transfer fees * Enhanced system reliability and fault tolerance ![Az Aware 2 Pn](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/az_aware_2.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=cb44ebcd88680e3c97e444d48640b91b) ## **Solution Architecture** The optimization strategy centers on implementing AZ-aware Kafka consumers using the Kafka RackAwareReplicaSelector. This approach ensures that Pinot servers preferentially consume from Kafka brokers within the same Availability Zone. ![Az Aware 1 Pn](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/az_aware_1.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=8c1a82fc7350457d84ac35b8a0111620) Here are the key steps in achieving this * Step 1: Implement AZ-Aware Instance Assignment (Recommended but optional) Configure the instance assignment strategy to consider Availability Zone placement when distributing workloads across the cluster. * Step 2: Configure AZ-Aware Table Settings ## **Implementation details** ### **Make instance assignment AZ-aware** > [!NOTE] > > Note: This step is recommended but not necessary. Even if the Pinot servers are not perfectly provisioned in the same zones as Kafka cluster, we can still get partial benefits (best effort). First thing to do is setup pool-based instance assignment, wherein we tag servers in the same AZ with the same name (eg CLOUD\_AZ\_POOL\_REALTIME). For example, set servers in aps1-az1 with value 0, aps1-az2 with value 1, etc. ``` {   "listFields": {     "TAG_LIST": {       "CLOUD_AZ_POOL_REALTIME"     }   },   "mapFields": {     "pool": {       "CLOUD_AZ_POOL_REALTIME": 0   }, } ``` For pool-based instance assignment, you need to configure CONSUMING with tag CLOUD\_AZ\_POOL\_REALTIME and poolBased in instanceAssignmentConfigMap:\ Example config: ``` "instanceAssignmentConfigMap": {   "CONSUMING": {     "tagPoolConfig": {       "tag": "CLOUD_AZ_POOL_REALTIME",       "poolBased": true     },     "replicaGroupPartitionConfig": {       "replicaGroupBased": true,       "numInstances": 0,       "numReplicaGroups": 2,       "numInstancesPerReplicaGroup": 0,       "numPartitions": 0,       "numInstancesPerPartition": 1     },     "partitionSelector": "INSTANCE_REPLICA_GROUP_PARTITION_SELECTOR"   } } ``` ### **Make table configuration AZ-aware** When we create realtime table, configure client.rack ``` "client.rack": "${CLOUD_AZ}" ``` This environment variable *CLOUD\_AZ* is automatically set on the servers and includes the coprresponding cloud zone information. ## **Summary** This guide addresses the critical issue of cross-Availability Zone network traffic in StarTree Pinot clusters. This can be enabled by configuring pool based instance assignment and setting *client.rack* property of kafka consumer to the right value. Results demonstrate substantial optimization with same-AZ traffic increasing from 50% to 96-98% across all tested zones, resulting in significant cost savings and improved system performance. # Avro Message Decoder Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/decoders/avro Avro message decoder is used for **real-time ingestion** in Pinot when consuming **Avro-encoded messages** from Kafka topics that use **Confluent Schema Registry**. ### Class `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` ## When to Use Use this decoder when: * Kafka messages are serialized using **Avro format**. * Kafka messages must be Avro-encoded binary payloads and the schema must be registered in Confluent Schema Registry. Pinot uses the registry to deserialize the messages correctly. * A **Confluent Schema Registry** is managing your Avro schemas. ### Configuration Example Include it in your `streamConfig` section: ```json theme={null} "streamConfig": { "streamType": "kafka", "stream.kafka.topic.name": "orders_topic", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "stream.kafka.broker.list": "broker:9092", "stream.kafka.decoder.prop.schema.registry.url": "" } ``` ## Decoder Behavior * Automatically fetches and caches Avro schemas from the registry. * Deserializes messages based on the schema ID embedded in the message. * Converts Avro records into Pinot-compatible rows. * Drops fields not present in the Pinot schema. * Avoid nested records - flatten them before ingestion if needed. * Use nullHandlingEnabled if fields might be missing or optional. # Debezium Message Decoder Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/decoders/debezium Debezium provides a unified format for Change Data Capture (CDC) data from various database sources like MySQL, MongoDB, Postgres etc. The generated CDC data is written to a streaming system like Kafka and Kinesis and, thereafter, made available in real-time for downstream applications. Native support for the Debezium data format in Pinot allows users to consume CDC data in real time from traditional OLTP / NoSQL stores in a source-agnostic manner. As long as the data is available in any of Pinot's supported streaming connectors, it can be ingested into a Pinot table. > [!NOTE] > > Debezium covers more sources than this decoder does. `DebeziumMessageDecoder` parses MySQL and Postgres > envelopes only — `dbz.source` accepts no other value. To ingest Debezium's > MongoDB output, use `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder` and pull fields out of the > envelope with ingestion transforms such as `jsonPathString(after, '$.status')`. Debezium's MongoDB connector > encodes the changed document in `after` as a JSON string rather than a nested object. See > [Connect MongoDB](/corecapabilities/ingestdata/mongodb) for a worked example and the other MongoDB ingestion paths. ## Debezium Message Decoder Configurations In order to configure a Pinot table to use a Debezium formatted streaming source, Pinot provides a decoder - `ai.startree.pinot.plugin.inputformat.debezium.DebeziumMessageDecoder`. The properties of this decoder are listed below: | Configuration Key | Description | | :- | :- | | `dbz.source` | Specifies the source database used for CDC. Allowed values: `mysql`, `postgres` | | `dbz.envelope.decoder.class.name` | Specifies the envelope decoder class name to decode the debezium payload in the stream into a pinot record. Allowed values - any of the supported `StreamMessageDecoder` classes in Pinot | | `dbz.envelope.decoder.prop.` | Prefix to be used for any of the properties associated with the envelope decoder class specified by `dbz.envelope.decoder.class.name` | | `dbz.source.timeColumnName` | (Optional) If the source DB's event timestamp has to be extracted as a pinot table column, the column name should be specified here. This is useful when using CDC data with upsert-enabled pinot table. This timestamp can be used as the table's default time column, if the payload itself doesn't have a time column. | ## Configuration Example When ingesting a Debezium formatted payload from a stream, the decoder used for the stream must be `ai.startree.pinot.plugin.inputformat.debezium.DebeziumMessageDecoder`. The following is an example stream config where the Pinot table is consuming from a JSON-encoded Kafka topic containing Debezium CDC payload from a MySQL source DB. ```json theme={null} { "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "simple", "stream.kafka.topic.name": "test.world.city", "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.debezium.DebeziumMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.zk.broker.url": "localhost:2181", "stream.kafka.broker.list": "localhost:9092", "realtime.segment.flush.threshold.time": "1h", "realtime.segment.flush.threshold.size": "100", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.decoder.prop.dbz.source": "mysql", "stream.kafka.decoder.prop.cdc.timeColumn": "created_at_timestamp", "stream.kafka.decoder.prop.dbz.envelope.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder" } } ``` In the above sample, the Kafka consumer factory used is `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory` and the decoder associated with this stream is `ai.startree.pinot.plugin.inputformat.debezium.DebeziumMessageDecoder`. Since it is a JSON-encoded debezium payload, the `dbz.enveloper.decoder.class.name` is configured as `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder`. Properties related to a stream's decoder are prefixed with `stream.$streamType.decoder.prop.`. For the Debezium message decoder all properties begin with `stream.kafka.decoder.prop`, as seen above in the `stream.kafka.decoder.prop.dbz.source`, `stream.kafka.decoder.prop.cdc.timeColumn` and `stream.kafka.decoder.prop.dbz.envelope.decoder.class.name` properties. # Amazon DynamoDB Message Decoder Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/decoders/dynamodb DynamoDB provides Change Data Capture (CDC) capabilities through DynamoDB Streams, which capture data modifications in DynamoDB tables. The generated CDC data is written to streaming systems like Kafka and made available in real-time for downstream applications. Native support for the DynamoDB data format in Pinot allows users to consume CDC data in real-time from DynamoDB tables without complex transformations. As long as the data is available in any of Pinot's supported streaming connectors, it can be ingested into a Pinot table. ## DynamoDB Message Decoder Configurations To configure a Pinot table to use a DynamoDB formatted streaming source, Pinot provides a decoder - `ai.startree.pinot.plugin.inputformat.dynamodb.DynamoDbMessageDecoder`. The properties of this decoder are listed below: | Configuration Key | Description | | :- | :- | | `decoder.class.name` | Specifies the primary decoder for DynamoDB messages. Set this to `ai.startree.pinot.plugin.inputformat.dynamodb.DynamoDbMessageDecoder` to enable DynamoDB CDC ingestion. | | `dynamodb.timeColumnName` | The column name where the `ApproximateCreationDateTime` from the DynamoDB JSON record should be stored. This timestamp can be used as the table's default time column. | | `dynamodb.deleteColumnName` | The column name that will be set to `true` when a `REMOVE` record is received from DynamoDB, and `false` otherwise. This helps track deletion events in your Pinot table. | | `dynamodb.envelope.decoder.class.name` | Specifies the underlying decoder used to parse the message format. Since DynamoDB messages are in JSON format, this should typically be set to `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder`. | | `dynamodb.envelope.decoder.prop.` | Prefix to be used for any properties associated with the envelope decoder class. | ## Configuration Example When ingesting a DynamoDB formatted payload from a stream, the decoder used for the stream must be `ai.startree.pinot.plugin.inputformat.dynamodb.DynamoDbMessageDecoder`. The following is an example stream config where the Pinot table is consuming from a JSON-encoded Kafka topic containing DynamoDB CDC payload: ```json theme={null} { "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "simple", "stream.kafka.topic.name": "dynamodb-cdc-topic", "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.dynamodb.DynamoDbMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.zk.broker.url": "localhost:2181", "stream.kafka.broker.list": "localhost:9092", "realtime.segment.flush.threshold.time": "1h", "realtime.segment.flush.threshold.size": "100", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.decoder.prop.dynamodb.timeColumnName": "created_at_timestamp", "stream.kafka.decoder.prop.dynamodb.deleteColumnName": "is_deleted", "stream.kafka.decoder.prop.dynamodb.envelope.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder" } } ``` In the above sample, the Kafka consumer factory used is `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory` and the decoder associated with this stream is `ai.startree.pinot.plugin.inputformat.dynamodb.DynamoDbMessageDecoder`. The configuration uses several key components: * The primary decoder `ai.startree.pinot.plugin.inputformat.dynamodb.DynamoDbMessageDecoder` handles the DynamoDB-specific message format * The `dynamodb.timeColumnName` is populated with the `ApproximateCreationDateTime` from the DynamoDB JSON record * The `dynamodb.deleteColumnName` is set to `true` when `REMOVE` records are received from DynamoDB * The `dynamodb.envelope.decoder.class.name` is set to `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder` since the underlying DynamoDB messages are in JSON format ## Simplified Configuration The DynamoDB Message Decoder significantly simplifies the configuration needed for ingesting DynamoDB CDC data. Without this decoder, you would need to specify multiple JSON path transformations for each column. Here's a comparison: ### Without DynamoDB Decoder ```json theme={null} { "ingestionConfig": { "transformConfigs": [ { "columnName": "rsvp_count", "transformFunction": "JSONPATHLONG(dynamodb, '$.NewImage.rsvp_count.N')" }, { "columnName": "venue_name", "transformFunction": "JSONPATHSTRING(dynamodb, '$.NewImage.venue_name.S')" }, { "columnName": "group_city", "transformFunction": "JSONPATHSTRING(dynamodb, '$.NewImage.group_city.S')" } // ... more transform configs for each column ] } } ``` ### With DynamoDB Decoder ```json theme={null} { "streamConfigs": { "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.dynamodb.DynamoDbMessageDecoder", "stream.kafka.decoder.prop.dynamodb.timeColumnName": "created_at_timestamp", "stream.kafka.decoder.prop.dynamodb.deleteColumnName": "is_deleted", "stream.kafka.decoder.prop.dynamodb.envelope.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder" } } ``` The decoder automatically handles the complex DynamoDB JSON structure, including the type indicators (S, N, B) and nested structure, making it much easier to ingest DynamoDB CDC data into Pinot. Properties related to a stream's decoder are prefixed with `stream.$streamType.decoder.prop.`. For the Dynamodb message decoder all properties begin with `stream.kafka.decoder.prop`, as seen above in the `stream.kafka.decoder.prop.dynamodb.source`, `stream.kafka.decoder.prop.dynamodb.timeColumn` and `stream.kafka.decoder.prop.dynamodb.envelope.decoder.class.name` properties. # AWS Glue Schema Registry Avro Decoder Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/decoders/glue The Glue decoder is used for **real-time ingestion** in Pinot when consuming **Avro-encoded messages** from Kafka (including Amazon MSK) topics whose schemas are managed by the **AWS Glue Schema Registry**. Unlike the [Confluent Avro decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/avro), Glue Schema Registry wraps each message with its own binary header rather than Confluent's magic-byte + schema-ID format, so a dedicated decoder is required. ### Class `ai.startree.pinot.plugin.inputformat.glue.avro.GlueSchemaRegistryAvroMessageDecoder` ## When to Use Use this decoder when: * Kafka (or MSK) messages are serialized using **Avro** and wrapped with the **AWS Glue Schema Registry** wire format. * Schemas are registered and versioned in **AWS Glue Schema Registry** rather than Confluent Schema Registry. * The topic's producers may write with **different schema versions over time** (schema evolution) and messages from any version must decode correctly. ## Operating Modes The decoder supports two mutually exclusive modes, selected automatically by whether the `schema` config property is set. | Mode | Trigger | Behavior | | - | - | - | | **Legacy mode** | `schema` is provided | Uses the inline schema you supply as both the writer and reader schema. No Glue API calls are made. This is the original decoder behavior. | | **Registry mode** | `schema` is absent, `region` is provided | Resolves the actual writer schema per message from AWS Glue Schema Registry using the schema-version UUID embedded in the message header. Handles schema evolution transparently. | > [!WARNING] > > Legacy mode requires the inline schema to **exactly match** the writer schema used to produce every message on the topic. If the topic's schema evolves (for example, a producer starts writing with 13 fields instead of 10), messages written with a different version than the inline schema fail to decode — this can surface as `Sampled: 0 Expected min: 1` from the table preview API, or as silent row loss during ingestion. Switch to registry mode to support schema evolution. ## How Registry Mode Works Each Glue Schema Registry message is wrapped with an 18-byte header: ``` [1 byte header version] [1 byte compression] [16 bytes schema-version UUID] [Avro payload...] ``` * **Header version** must be `0x03` — the current AWS Glue Schema Registry wire format. Messages with any other version byte are skipped. * **Compression** is either `0x00` (none) or `0x05` (ZLIB). Any other value is skipped. ZLIB-compressed payloads are inflated up to the `maxDecompressedBytes` limit before decoding; payloads that would exceed it are dropped and logged. * **Schema-version UUID** identifies the exact writer schema used for that message. For each message, the decoder: 1. **Resolves the writer schema per message** — extracts the UUID from the header and looks up the matching Avro schema from its in-memory cache, or fetches it from Glue on a cache miss. 2. **Pre-warms the cache at init**, if `registry.name` and `schemaName` are both set — all known schema versions for that Glue schema are paginated via `listSchemaVersions` and fetched via `getSchemaVersion` before the first message arrives, avoiding a Glue round-trip on cold start. 3. **Falls back on demand** for UUIDs not already cached (whether pre-warming was configured or not) — the schema is fetched from Glue and cached for subsequent messages with the same UUID. 4. **Does not cache failed fetches** — if a Glue API call fails (throttling, transient network error, permissions), the UUID is not cached as a failure, so the next message with that UUID retries the fetch rather than being permanently unable to decode. 5. **Bounds the cache with LRU eviction** — a single cache holds the writer schema, its Avro `DatumReader`, and a reusable `GenericData.Record` per schema-version UUID, bounded by `glue.maxCachedSchemas` (default `1000`). Because writer schema, reader, and reused record live in one entry, eviction is atomic — there's no risk of one map evicting a UUID while a companion map still holds stale state for it. > [!NOTE] > > Each stream partition consumer creates its own decoder instance, and therefore its own Glue client and schema cache. `glue.maxCachedSchemas` bounds the cache **per partition consumer**, not per table — size it with the number of partitions in mind if a table has many. ## Configuration Properties All properties are set via `stream.kafka.decoder.prop.*` (or the equivalent prefix for other stream types). | Property | Required | Default | Description | | - | - | - | - | | `schema` | Mutually exclusive with registry mode | — | Inline Avro schema JSON. If present, the decoder runs in **legacy mode** and makes no Glue API calls. | | `region` | Required for registry mode | — | AWS region for the Glue client, e.g. `ap-south-1`. Required when `schema` is absent. | | `registry.name` | Optional | — | Glue registry name. Combine with `schemaName` to pre-warm the schema cache at init. | | `schemaName` | Optional | — | Glue schema name. Combine with `registry.name` to pre-warm the schema cache at init. | | `glue.endpoint` | Optional | — | Custom Glue API endpoint URL — for example, a VPC endpoint or a cross-region override. | | `glue.accessKeyId` | Optional (paired) | — | Static AWS access key ID for the Glue client. Requires `glue.secretAccessKey`. | | `glue.secretAccessKey` | Optional (paired) | — | Static AWS secret access key for the Glue client. Requires `glue.accessKeyId`. | | `glue.roleArn` | Optional | — | IAM role ARN to assume for Glue API calls — useful for cross-account access or scoping down permissions. | | `glue.roleSessionName` | Optional | `pinot-glue-sr` | Session name used when assuming `glue.roleArn`. | | `glue.roleExternalId` | Optional | — | External ID required by the assumed role's trust policy, for cross-account setups. | | `glue.maxCachedSchemas` | Optional | `1000` | Maximum writer schemas cached per decoder instance, with LRU eviction once exceeded. | | `maxDecompressedBytes` | Optional | `16777216` (16 MiB) | Upper bound on ZLIB-decompressed payload size; larger payloads are dropped rather than fully inflated. | ### Glue Credential Resolution Order The Glue client's credentials are resolved independently of your Kafka/MSK credentials (for example, MSK IAM auth configured via `sasl.jaas.config`) — they follow their own chain: 1. If `glue.accessKeyId` and `glue.secretAccessKey` are both set → static credentials. 2. If `glue.roleArn` is also set → assume that role via STS, using the static credentials from step 1 as the base identity if present, otherwise the default credential chain. 3. Otherwise → the AWS SDK v2 default credential chain (environment variables, system properties, instance profile, IRSA, etc.). ## Configuration Examples **Legacy Mode** Inline schema, no Glue API calls — identical to the decoder's original behavior: ```json theme={null} "streamConfig": { "streamType": "kafka", "stream.kafka.topic.name": "orders_topic", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.glue.avro.GlueSchemaRegistryAvroMessageDecoder", "stream.kafka.broker.list": "broker:9092", "stream.kafka.decoder.prop.schema": "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[...]}" } ``` **Registry Mode — Same Account** Pinot's ambient IAM role or instance profile already has Glue read access, so no explicit credentials are needed. `registry.name` + `schemaName` pre-warm the cache at startup: ```json theme={null} "streamConfig": { "streamType": "kafka", "stream.kafka.topic.name": "orders_topic", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.glue.avro.GlueSchemaRegistryAvroMessageDecoder", "stream.kafka.broker.list": "broker:9092", "stream.kafka.decoder.prop.region": "", "stream.kafka.decoder.prop.registry.name": "MySchemaRegistry", "stream.kafka.decoder.prop.schemaName": "ORDER_EVENT" } ``` **Registry Mode — Cross-Account Keys** Glue Schema Registry lives in a different AWS account, accessed with a dedicated IAM user's access keys: ```json theme={null} "streamConfig": { "streamType": "kafka", "stream.kafka.topic.name": "orders_topic", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.glue.avro.GlueSchemaRegistryAvroMessageDecoder", "stream.kafka.broker.list": "broker:9092", "stream.kafka.decoder.prop.region": "", "stream.kafka.decoder.prop.registry.name": "MySchemaRegistry", "stream.kafka.decoder.prop.schemaName": "ORDER_EVENT", "stream.kafka.decoder.prop.glue.accessKeyId": "", "stream.kafka.decoder.prop.glue.secretAccessKey": "" } ``` **Registry Mode — Assume Role** Pinot assumes an IAM role in the target account that has Glue permissions: ```json theme={null} "streamConfig": { "streamType": "kafka", "stream.kafka.topic.name": "orders_topic", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.glue.avro.GlueSchemaRegistryAvroMessageDecoder", "stream.kafka.broker.list": "broker:9092", "stream.kafka.decoder.prop.region": "", "stream.kafka.decoder.prop.registry.name": "MySchemaRegistry", "stream.kafka.decoder.prop.schemaName": "ORDER_EVENT", "stream.kafka.decoder.prop.glue.roleArn": "arn:aws:iam:::role/", "stream.kafka.decoder.prop.glue.roleSessionName": "pinot-glue-sr", "stream.kafka.decoder.prop.glue.roleExternalId": "" } ``` ## Decoder Behavior * Converts Avro records into Pinot-compatible rows, dropping fields not present in the Pinot schema. * Reuses the underlying Avro `GenericData.Record` across decode calls (one instance in legacy mode, one per cached schema-version UUID in registry mode) to reduce per-message allocations on high-throughput topics. * Avoid nested records — flatten them before ingestion if needed. * Use `nullHandlingEnabled` if fields might be missing or optional. * Malformed or unrecognized payloads (bad header version, unsupported compression byte, truncated payload, decode failure) are logged and skipped rather than failing the consumer. ## Troubleshooting > **Preview shows "Sampled: 0 Expected min: 1", or rows silently go missing** > > This is the classic symptom of legacy mode's inline schema no longer matching every message on the topic — typically because the Glue schema evolved (fields added, types changed) after the inline `schema` was captured. Switch to registry mode: remove `schema` and set `region` (plus `registry.name` + `schemaName` if you want pre-warming), and each message decodes with its own correct writer schema. > **Init fails with "'region' is required when no inline 'schema' is provided"** > > Registry mode requires `region`. If you intended legacy mode instead, set `schema` to the inline Avro schema JSON. > **Init fails with a credential pairing error** > > `glue.accessKeyId` and `glue.secretAccessKey` must be set together — supplying only one throws at init. Either provide both, or omit both and rely on `glue.roleArn` or the default AWS credential chain. > **Decoding is slow right after a deploy or partition reassignment** > > Set `registry.name` and `schemaName` so all known schema versions are pre-warmed into the cache at decoder init, avoiding a per-partition Glue round-trip on the first message of each previously-unseen schema version. # JSON Message Decoder Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/decoders/json The JSON message decoder is used in **real-time ingestion** to **decode JSON-formatted messages** from Kafka topic and convert them into Pinot rows. ### Class `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder` ## Configuration Example Define it in your `streamConfig` within the table config: ```json theme={null} "streamConfig": { "streamType": "kafka", "stream.kafka.topic.name": "events_topic", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kafka.broker.list": "broker:9092" } ``` ## Decoder Behavior * Parses JSON strings using a fast JSON library (e.g., Jackson/Gson). * Converts each key-value pair into Pinot column values. * Logs and skips invalid or malformed messages. # Message Decoders Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/decoders/overview In **Apache Pinot**, message decoders play a crucial role in the real-time ingestion pipeline. They are responsible for transforming raw byte-stream messages from streaming systems like Apache Kafka, AWS Kinesis, or Pulsar into Pinot-compatible row objects (GenericRow) that can be indexed and stored. When data is consumed from a stream, it often comes in a variety of formats—such as JSON, Avro, ORC or Thrift—and Pinot needs to understand how to deserialize that data into something it can work with. That’s where message decoders come in. Each decoder implements a strategy to handle a specific serialization format and convert it into a structured row according to the schema defined in Pinot. Message decoders are configured per real-time table, under the streamConfigs section, and allow you to define how incoming messages are interpreted and parsed. ## Supported Message Decoders in Pinot Apache Pinot supports a range of message decoders: > **[JSON Decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/json)** > > Used to Decode JSON-formatted messages from Kafka topics > **[Avro Decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/avro)** > > Used when consuming Avro-encoded messages from Kafka topics > **[AWS Glue Schema Registry Decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/glue)** > > Used to decode Avro messages wrapped with the AWS Glue Schema Registry wire format, with support for schema evolution > **[Protobuf Decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/protobuf)** > > Used to decode Protobuf-encoded messages from Kafka topics > **[Debezium Decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/debezium)** > > Used to configure a Pinot table to use a Debezium formatted streaming source > **[DynamoDB Decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/dynamodb)** > > * Used to configure a Pinot table to use a DynamoDB formatted streaming source > **[Prometheus Decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/prometheus)** > > Used to provide support for ingesting Prometheus-formatted metrics data # Prometheus Message Decoder Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/decoders/prometheus **Release Note:** The Prometheus Message Decoder feature is available in StarTree Platform releases after 0.9.0. Prometheus is a widely-used open-source monitoring and alerting toolkit that collects and stores time series data from various sources. The Prometheus Message Decoder in Pinot provides native support for ingesting Prometheus-formatted metrics data into pinot tables. ## Prometheus Message Format ``` {=,=,...} ``` This format encapsulates the four main components of our schema: * `` corresponds to the metric field (STRING) * `{=,=,...}` represents the labels field (JSON) * `` maps to the value field (DOUBLE) * `` corresponds to the ts field (TIMESTAMP) ## Table schema | Column | Type | Field type | Description | | - | - | - | - | | metric | STRING | Dimension | The name of the Prometheus metric | | labels | JSON | Dimension | A JSON object containing key-value pairs of Prometheus labels | | value | DOUBLE | Metric | The numeric value of the metric | | ts | TIMESTAMP | Date-Time | The timestamp when the metric was recorded, in milliseconds since the Unix epoch | * The schema created for ingesting Prometheus metrics MUST adhere to the format specified above, particularly with respect to the field names and data types. * If you require different field names in your Pinot table, you can add transform functions to alias the names. * However, the incoming data must match this specified schema for the Prometheus Message Decoder to function correctly. ## Schema Configuration ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [ { "name": "metric", "dataType": "STRING" }, { "name": "labels", "dataType": "JSON" } ], "metricFieldSpecs": [ { "name": "value", "dataType": "DOUBLE" } ], "dateTimeFieldSpecs": [{ "name": "ts", "dataType": "TIMESTAMP", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` ## Configuration Example When ingesting a Prometheus Message formatted payload from a stream, the decoder used for the stream must be ai.startree.pinot.plugin.inputformat.prometheus.PrometheusMessageDecoder ```json theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.topic.name": "events", "stream.kafka.decoder.class.name": "ai.startree.pinot.plugin.inputformat.prometheus.PrometheusMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.broker.list": "localhost:9092", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "50M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } } } ``` In the above sample, the Kafka consumer factory used is `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory` and the decoder associated with this stream is `ai.startree.pinot.plugin.inputformat.prometheus.PrometheusMessageDecoder"`. # Protobuf Message Decoder Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/decoders/protobuf Protobuf message decoder is used in **real-time ingestion** to handle **Protobuf-encoded messages** from Kafka topics using the **Confluent Schema Registry** for schema management. ### Class `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` *** ## When to Use Use this decoder when: * The Kafka messages are serialized using **Protocol Buffers (Protobuf)**. * The schema must be registered in the Confluent Schema Registry. * You need to ingest structured Protobuf data into a Pinot real-time table. *** ## Configuration Example Specify this decoder in the `streamConfig` of your table config: ```json theme={null} "streamConfig": { "streamType": "kafka", "stream.kafka.topic.name": "metrics_topic", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder", "stream.kafka.broker.list": "broker:9092", "stream.kafka.decoder.prop.schema.registry.url": "" } ``` ## Decoder Behavior * Retrieves and caches Protobuf schemas from the Schema Registry. * Uses the schema to parse Protobuf messages into column-value pairs. * The decoder fetches the schema dynamically using the schema ID in the message payload. * Ignores fields not defined in the Pinot schema. * Supports nested fields if flattened before ingestion. * Enable nullHandlingEnabled for optional fields. # MongoDB CDC Ingestion Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/mongodb-cdc Consume MongoDB Change Streams directly into a realtime Pinot table. MongoDB CDC ingestion lets a StarTree Pinot realtime table consume MongoDB Change Streams directly, without first copying changes through Kafka, Kinesis, Pulsar, or another durable stream. If you have not chosen an ingestion path yet, start at [Connect MongoDB](/corecapabilities/ingestdata/mongodb), which compares this connector against the Kafka and `mongodump` routes. > [!NOTE] > > This connector is intended for advanced realtime ingestion designs where the MongoDB source namespaces, partitioning, credentials, and replay window are explicitly managed. If you only need to consume Debezium-formatted MongoDB events from Kafka, use the [Debezium decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/debezium) instead. ## How It Works The connector maps each configured MongoDB source partition to one Pinot partition group: ```text theme={null} app.orders_00 -> Pinot partition group 0 app.orders_01 -> Pinot partition group 1 app.orders_02 -> Pinot partition group 2 ``` Use the StarTree MongoDB CDC consumer factory: ```text theme={null} ai.startree.pinot.plugin.stream.mongodb.MongoCdcConsumerFactory ``` Pinot controllers resolve source-partition metadata and checkpoints. Pinot servers open and resume MongoDB Change Stream cursors for the partitions they own. Pinot replication creates another cursor per partition replica, so plan source-side Change Stream load as: ```text theme={null} source partitions x Pinot replicas ``` For sharded MongoDB deployments, a cursor opened through `mongos` can fan out to every shard. Plan closer to: ```text theme={null} source partitions x Pinot replicas x MongoDB shards ``` ## Source-Partition Contract MongoDB Change Streams do not expose Kafka-style durable partitions. The connector therefore requires a manifest of physical, disjoint MongoDB namespaces. Follow these rules: * Partition IDs are dense, stable integers: `0`, `1`, `2`, and so on. * Every descriptor has a globally unique and immutable `sourceId`, such as `atlas-prod-a/app/orders-00`. * `_id` values must be globally unique across the configured namespaces. * Every insert, update, replace, and delete for one primary key must stay in the same source partition for that key's lifetime. * Do not remove, renumber, rebind, or repartition an active source partition. Create a new Pinot table, snapshot/backfill, catch up CDC, and cut over queries instead. * Do not use `hash(_id) % currentPartitionCount` for upstream routing. Increasing the partition count would move existing keys and break upsert partition affinity. A common routing pattern is to encode the source partition into the immutable `_id`: ```text theme={null} _id = "02:ord_123" -> app.orders_02 _id = "07:ord_456" -> app.orders_07 ``` New partitions can receive only newly created IDs. Existing IDs must continue to route to their original namespace. ## Prerequisites * MongoDB Atlas, a replica set, or a sharded cluster. Standalone `mongod` processes do not support Change Streams. * MongoDB credentials available to Pinot controllers and servers. * `find` and `changeStream` permissions on every source namespace. * If you use registry mode, `find` on the manifest collection and `listCollections` on source databases. * If you use `outputMode=full_document`, MongoDB Server 6.0 or later with Change Stream pre/post-images enabled. * An oplog and post-image retention window longer than the maximum Pinot outage and recovery window. ## Static Manifest Mode Use static mode when the source partition list is fixed. Omit `partitionDiscoveryMode` and set `sourcePartitions` to a JSON array encoded as a string. ```json theme={null} { "streamType": "mongodb", "stream.mongodb.topic.name": "mongo-orders", "stream.mongodb.consumer.factory.class.name": "ai.startree.pinot.plugin.stream.mongodb.MongoCdcConsumerFactory", "stream.mongodb.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.mongodb.consumer.prop.auto.offset.reset": "largest", "sourcePartitions": "[{\"partitionId\":0,\"epoch\":1,\"sourceId\":\"atlas-prod-a/app/orders-00\",\"connectionStringEnvironmentVariable\":\"MONGODB_ATLAS_URI\",\"database\":\"\",\"collection\":\"\"},{\"partitionId\":1,\"epoch\":1,\"sourceId\":\"atlas-prod-a/app/orders-01\",\"connectionStringEnvironmentVariable\":\"MONGODB_ATLAS_URI\",\"database\":\"\",\"collection\":\"\"}]", "fullDocument": "required", "outputMode": "full_document", "cursorBatchSize": "100", "maxBatchMessages": "500", "maxBatchBytes": "4194304", "maxAwaitTimeMs": "500", "stream.mongodb.idle.timeout.millis": "180000" } ``` In this mode, every descriptor requires: | Field | Description | | - | - | | `partitionId` | Dense Pinot partition ID. | | `epoch` | Positive operator-controlled lineage epoch. Treat it as immutable for an active table. | | `sourceId` | Globally unique source identity for the physical namespace lineage. | | `database` | MongoDB database name. | | `collection` | MongoDB collection name. | | `connectionRef`, `connectionString`, or `connectionStringEnvironmentVariable` | Exactly one connection source. Prefer environment variables or connection refs for production. | | `initialOffset` | Required when appending capacity to an existing table. Optional for initial creation with `auto.offset.reset=largest`. | ## Registry Manifest Mode Use `partitionDiscoveryMode=mongodb_manifest` when an upstream routing service can append new physical bucket collections. The table config allowlists connection references and points Pinot at one authoritative registry document. ```json theme={null} { "streamType": "mongodb", "stream.mongodb.topic.name": "mongo-orders", "stream.mongodb.consumer.factory.class.name": "ai.startree.pinot.plugin.stream.mongodb.MongoCdcConsumerFactory", "stream.mongodb.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.mongodb.consumer.prop.auto.offset.reset": "largest", "partitionDiscoveryMode": "mongodb_manifest", "connectionRefs": "{\"atlas-prod-a\":{\"connectionStringEnvironmentVariable\":\"MONGODB_ATLAS_URI\"}}", "partitionManifest.connectionRef": "atlas-prod-a", "partitionManifest.database": "pinot_cdc_control", "partitionManifest.collection": "partition_manifests", "partitionManifest.id": "mongo-orders", "maxPartitionCount": "256", "fullDocument": "required", "outputMode": "full_document", "cursorBatchSize": "100", "maxBatchMessages": "500", "maxBatchBytes": "4194304", "maxAwaitTimeMs": "500" } ``` Example registry document: ```json theme={null} { "_id": "mongo-orders", "schemaVersion": 1, "generation": {"$numberLong": "3"}, "partitions": [ { "partitionId": 0, "epoch": {"$numberLong": "1"}, "sourceId": "atlas-prod-a/app/orders-00", "connectionRef": "atlas-prod-a", "database": "", "collection": "", "collectionUuid": {"$binary": {"base64": "AAAAAAAAAAAAAAAAAAAAAA==", "subType": "04"}}, "initialOffset": { "resumeToken": {"_data": ""}, "logicalTime": {"$timestamp": {"t": 1810000000, "i": 1}} } } ] } ``` Registry mode fails closed if an existing descriptor changes, a partition is removed, a partition ID is skipped, a namespace UUID changes, or a later generation mutates an old prefix. `maxPartitionCount` is a safety limit that prevents a malformed registry from creating unbounded Pinot segments or MongoDB cursors. ## Output Modes | Mode | Use Case | Notes | | - | - | - | | `change_stream` | Preserve the MongoDB Change Stream envelope. | Default. Leaves `fullDocument` nested and defaults to `fullDocument=default`. | | `full_document` | Maintain a current-state Pinot upsert table. | Emits document fields at the top level and adds reserved `__mongodb_cdc_*` metadata. Requires `fullDocument=required` and MongoDB pre/post-images. | For `full_document`, enable post-images on every source collection: ```javascript theme={null} db.runCommand({ collMod: "orders_00", changeStreamPreAndPostImages: {enabled: true} }) ``` Source documents must not contain any of the connector's reserved fields: ```text theme={null} __mongodb_cdc_document_id __mongodb_cdc_document_key __mongodb_cdc_operation_type __mongodb_cdc_is_deleted __mongodb_cdc_event_time_ms __mongodb_cdc_cluster_time_seconds __mongodb_cdc_cluster_time_increment __mongodb_cdc_comparison_value __mongodb_cdc_resume_token __mongodb_cdc_partition_id __mongodb_cdc_partition_epoch ``` ## Tuning Options and Defaults Every option below is optional. The values shown in the examples on this page are the defaults unless noted. | Option | Default | Notes | | - | - | - | | `cursorBatchSize` | `100` | MongoDB Change Stream cursor batch size. | | `maxBatchMessages` | `500` | Maximum events returned to Pinot per fetch. | | `maxBatchBytes` | `4194304` (4 MiB) | Maximum bytes returned per fetch. | | `maxAwaitTimeMs` | `500` | Must be strictly less than Pinot's stream fetch timeout. MongoDB server-selection, connect, socket-read, and operation timeouts are all capped below this budget. | | `maxPartitionCount` | `1024` | Registry mode only. A safety bound, not a target — it caps how many Pinot segments and MongoDB cursors a malformed or compromised registry can create. | Size the MongoDB URI's `maxPoolSize` for the maximum number of MongoDB CDC partitions that can be active in one Pinot JVM across all tables, plus controller metadata requests. Partitions sharing an identical connection string and timeout settings share one client and pool; distinct credentials, endpoints, or timeouts get their own. ### Rejected Options The connector fails closed rather than accept these: | Option | Why | | - | - | | `pipeline` | A generic source predicate cannot prove exhaustive, non-overlapping primary-key routing, and can hide deletes or lifecycle events. A top-level `pipeline` is rejected as a legacy singleton key, and the error suggests moving it into each `sourcePartitions` descriptor — but descriptors do not accept it either. There is no supported place for it. | | `showExpandedEvents=true` | Expanded DDL and split large events need a broader checkpoint and reassembly contract. | | Pinot realtime offset auto-reset | The connector never silently skips to latest. A lost resume position is a permanent failure, not something to paper over. | | `fullDocument=updateLookup` or `whenAvailable` | Rejected in every output mode, not only `full_document`: `fullDocument` must be `default` or `required` so replayed rows stay replica-deterministic. Independent Pinot replicas could otherwise observe different post-images or nullability during replay. | ## Current-State Upsert Example Schema: ```json theme={null} { "schemaName": "mongo_orders", "primaryKeyColumns": ["__mongodb_cdc_document_id"], "dimensionFieldSpecs": [ {"name": "__mongodb_cdc_document_id", "dataType": "STRING"}, {"name": "__mongodb_cdc_document_key", "dataType": "STRING"}, {"name": "__mongodb_cdc_comparison_value", "dataType": "STRING"}, {"name": "__mongodb_cdc_operation_type", "dataType": "STRING"}, {"name": "__mongodb_cdc_is_deleted", "dataType": "BOOLEAN"}, {"name": "__mongodb_cdc_partition_id", "dataType": "INT"}, {"name": "__mongodb_cdc_partition_epoch", "dataType": "LONG"}, {"name": "status", "dataType": "STRING"}, {"name": "region", "dataType": "STRING"} ], "metricFieldSpecs": [ {"name": "amount", "dataType": "DOUBLE"} ], "dateTimeFieldSpecs": [ { "name": "__mongodb_cdc_event_time_ms", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Realtime upsert config: ```json theme={null} { "upsertConfig": { "mode": "FULL", "comparisonColumns": ["__mongodb_cdc_comparison_value"], "deleteRecordColumn": "__mongodb_cdc_is_deleted", "snapshot": "ENABLE", "preload": "ENABLE" }, "routing": { "instanceSelectorType": "strictReplicaGroup" } } ``` `comparisonColumns` uses the connector-generated comparison value so MongoDB event order is preserved. `strictReplicaGroup` is required so a query sees a consistent upsert view for each partition. ## Capturing Initial Offsets When creating a table with `auto.offset.reset=largest`, the controller captures a starting boundary for each partition that does not already provide `initialOffset`. To include historical documents: 1. Capture an exact resume boundary for each MongoDB namespace. 2. Snapshot/backfill the historical data up to that boundary. 3. Configure each descriptor with the captured `initialOffset`. 4. Start realtime CDC consumption from those offsets. For step 1, the connector JAR exposes a bootstrap API that prints the descriptor-ready Extended JSON. Run it from a short Java program with the shaded connector JAR on the classpath, using the same `fullDocument` and `maxAwaitTimeMs` values as the target table: ```java theme={null} import ai.startree.pinot.plugin.stream.mongodb.MongoCdcInitialOffsetCapture; import ai.startree.pinot.plugin.stream.mongodb.MongoCdcCollectionMetadata; import com.mongodb.client.MongoClient; import com.mongodb.client.MongoClients; import com.mongodb.client.model.changestream.FullDocument; import org.bson.BsonDocument; try (MongoClient client = MongoClients.create(System.getenv("MONGODB_ATLAS_URI"))) { System.out.println(new BsonDocument("collectionUuid", MongoCdcCollectionMetadata.getCollectionUuid(client, "", "", 5_000)).toJson()); BsonDocument initialOffset = MongoCdcInitialOffsetCapture.capture( client, "", "", FullDocument.REQUIRED, 500, 5_000); System.out.println(initialOffset.toJson()); } ``` Capture each namespace independently, paste the printed document in as that descriptor's `initialOffset`, then snapshot that namespace. Registry mode also needs the printed `collectionUuid`. Boundary capture is per namespace and is not a transactionally atomic cut across collections. Descriptor example: ```json theme={null} { "partitionId": 2, "epoch": 1, "sourceId": "atlas-prod-a/app/orders-02", "connectionStringEnvironmentVariable": "MONGODB_ATLAS_URI", "database": "", "collection": "", "initialOffset": { "resumeToken": {"_data": ""}, "logicalTime": {"$timestamp": {"t": 1810000000, "i": 1}} } } ``` Do not substitute a bare `clusterTime`; the offset must include the resume token and its safe logical time. ## Adding Capacity To append a partition safely: 1. Create a new, quiescent physical namespace. 2. Capture the namespace UUID and exact initial offset. 3. Append only the next highest dense `partitionId`. 4. Wait until Pinot reports the new partition group consuming on the required replicas. 5. Route only new keys to the new namespace. Do not configure a primary-key `segmentPartitionConfig` on a table whose MongoDB source-partition count can grow. A growing partition count changes key ownership for older rows. ## Troubleshooting | Symptom | Likely Cause | Fix | | - | - | - | | Connector fails before opening a cursor | Descriptor identity or fingerprint mismatch. | Check `sourceId`, `epoch`, database, collection, UUID, output mode, and watch options. | | Resume fails after an outage | MongoDB oplog or post-image history expired. | Create a new table from a snapshot and matching Change Stream boundaries. | | Deletes are not reflected in Pinot | `full_document` mode is not configured with delete handling. | Use the connector-generated `__mongodb_cdc_is_deleted` as `deleteRecordColumn`. | | Source load is higher than expected | Cursor count scales with partitions, Pinot replicas, and MongoDB shards. | Reduce partitions or replicas, or isolate source partitions on separate replica sets/Atlas deployments. | | New partitions never start consuming | Registry generation is invalid or descriptor lacks an exact initial offset. | Ensure the registry appends only dense new partitions and includes `initialOffset`. | # Streaming Ingestion in Apache Pinot Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/overview Streaming ingestion in Apache Pinot enables real-time data to be ingested, indexed, and queried within seconds of its arrival. This capability forms the backbone of low-latency analytics use cases such as anomaly detection, dashboards, personalization engines, and monitoring platforms. Unlike batch ingestion, which processes static historical datasets, streaming ingestion is designed for continuously evolving data, ingesting millions of events per second from modern event platforms like Apache Kafka, Amazon Kinesis, and Pulsar. Streaming ingestion populates REALTIME tables in Pinot, which are optimized for handling append-only data streams. These tables are divided into consuming segments that ingest data in-memory, and then seal and persist those segments periodically based on time or size thresholds. This architecture allows Pinot to serve low-latency queries over streaming data while seamlessly managing freshness and durability. ## How Streaming Ingestion Works in Pinot The ingestion process starts with configuring a REALTIME table to connect to a streaming source. Pinot fetches and processes records from the stream. Each partition of a Kafka topic (or equivalent in other platforms) maps to a separate segment in Pinot, giving fine-grained control over offset tracking and recovery. As events flow in, Pinot uses a stream ingestion plugin that parses, transforms, and indexes records according to the table’s schema and ingestion config. Events are stored in in-memory consuming segments, which are periodically flushed to disk when thresholds (like segmentFlushSizeRows or flushThresholdTime) are met. This segment is then sealed, and a new consuming segment is created for continued ingestion. Sealed segments are pushed to deep storage and become part of the table’s queryable data. ### Supported Stream Sources Apache Pinot natively supports several popular streaming platforms for real-time ingestion: * Apache Kafka: The most widely used source, with mature integration and offset tracking. * Amazon Kinesis: Supported via plugins; suitable for AWS-native architectures. * Apache Pulsar: Increasingly popular for event-driven systems with built-in multitenancy. ### Streaming Table Configuration A streaming table in Pinot is defined by a REALTIME table config, which includes ingestion-specific details like stream type, topic name, consumer type, and decoder class. For example, a Kafka-backed REALTIME table will specify the Kafka broker list, topic, and any stream-level properties such as auto offset reset, max records per fetch, and watermarking options. ### Segment Lifecycle and Stream Consumption Each streaming segment goes through a defined lifecycle. When a segment starts consuming data, it is in the CONSUMING state. After reaching a size or time threshold, it is sealed and moved to ONLINE state after being persisted. The system then creates a new consuming segment for the same stream partition. In the event of a failure or restart, Pinot can recover using committed offsets, ensuring exactly-once or at-least-once semantics depending on configuration. Helix manages segment assignments, transitions, and failure recovery across the cluster, ensuring high availability and continuity. ## Configuring Realtime Ingestion Below is a sample configuration for a Pinot REALTIME table that consumes data from a Kafka topic named events\_topic. ```json theme={null} "ingestionConfig": { "streamIngestionConfig": { "streamConfigMaps": [ { "streamType": "kafka", "stream.kafka.topic.name": "events_topic", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.broker.list": "kafka-broker:9092", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } ] } } ``` ## Support for Upsert StarTree supports [full upserts and partial upserts](/corecapabilities/manage-data/offheap-upsert) on realtime tables. This enables record deduplication and fine-grained mutation of streaming records using a primary key. ### Full Upsert When using full upsert, the latest record (by comparison column, usually timestamp) replaces the older one entirely for a given key. ```json theme={null} "upsertConfig": { "mode": "FULL", "comparisonColumn": "event_time" } ``` ### Partial Upsert Partial upserts allow merging only certain columns based on a strategy (like INCREMENT, APPEND, OVERWRITE), while others remain unchanged. Partial upsert is commonly used in metrics systems or user state tracking, where some fields are additive and others replace previous values. ```json theme={null} "upsertConfig": { "mode": "PARTIAL", "partialUpsertStrategies": { "view_count": "INCREMENT", "tags": "APPEND", "last_updated": "OVERWRITE" }, "comparisonColumn": "event_time" } ``` # Pauseless Consumption Source: https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption # Introduction Pauseless consumption enhances real-time analytics by minimizing ingestion delays. By allowing data ingestion to continue while the previous segment is being built and uploaded, it significantly reduces the latency gap between data arrival and query availability. ## How to Enable Pauseless Consumption Enabling this feature involves two major phases: preparing the cluster/table and updating the configurations. ### Step 1: Preparation (Existing Tables Only) If you are creating a new table from scratch, you may skip to Step 2. This step is mandatory for existing tables that currently consume data from a stream. #### Pause Ingestion Stop data ingestion to ensure all current segments are committed across all servers/replicas. **API**: `POST /tables/{tableName}/pauseConsumption` #### Verify Inactive State Confirm that no consuming segments are active before proceeding. * Ideal State: Verify that all segments are marked as ONLINE. * Consuming Info: Use the API below to check for active segments. **API**: `GET /tables/{tableName}/consumingSegmentsInfo` \ **Success Criteria**: The field `_segmentToConsumingInfoMap` in the response must be empty. ### Step 2: Cluster & Controller Configuration These changes are applied at the system level and require a restart of the Servers and Controllers. a. **Enable Peer Download**: Ensure peer segment download is enabled on your cluster. Follow this documentation to do so. \ b. **Update Cluster Configuration**: Add the following parameters to your cluster configuration. **API**: `POST /cluster/configs` \ Use the payload below to the API: ```json theme={null} { "pinot.server.instance.segment.upload.to.deep.store": "true", "controller.segment.level.validation.intervalPeriod" : "7200s" } ``` | Parameter | Value | Reason | | :- | :- | :- | | pinot.server.instance.segment.upload.to.deep.store | true | Servers upload completed segments directly to the deep store on commit. | | controller.segment.level.validation.intervalPeriod | 7200s | Controls how often the segment-level validation runs, which retries deep store uploads for committed segments that are missing from the deep store. The default is 24 hours; reduce it to 2 hours for pauseless. | > [!NOTE] > > The following configs were previously required but are now applied automatically and no longer need to be set: > > * `controller.realtime.segment.deepStoreUploadRetryEnabled`: deep store upload retry always runs for tables with pauseless consumption enabled, regardless of this flag. > * `pinot.controller.segment.completion.fsm.scheme.pauseless`: the pauseless segment completion FSM is registered by default and selected automatically when `pauselessConsumptionEnabled` is `true`. ### Step 3: Table Configuration Update the table configuration to enable the pauseless flag and set the peer segment download scheme. **Important**: The pauselessConsumptionEnabled flag must be placed outside of the streamConfigMaps array, directly under streamIngestionConfig. ```json theme={null} "segmentsConfig": { "peerSegmentDownloadScheme": "https" }, "ingestionConfig": { "streamIngestionConfig": { "pauselessConsumptionEnabled": true, "streamConfigMaps": [ { "streamType": "kafka", "stream.kafka.topic.name": "githubEvents" } ] } } ``` > [!NOTE] > > `peerSegmentDownloadScheme` is mandatory for tables with replication greater than 1 — table creation/update fails without it when pauseless consumption is enabled. It allows replicas to download a committed segment from a peer server if the segment is not yet available in the deep store. ### Step 4: Resume Ingestion Once the configurations are applied and the system is restarted (if applicable), activate pauseless ingestion by resuming the table. Use **API**: `POST /tables/{tableName}/resumeConsumption` ## How to Enable Pauseless Consumption for Dedup-enabled Table Deduplication (Dedup) ensures data correctness by removing duplicate records based on primary keys during ingestion. When combined with Pauseless Consumption, users achieve both low-latency ingestion and data integrity. Pauseless mode normally runs background tasks (build/commit) parallel to ingestion. However, because Dedup relies on knowing the exact state of previous data to identify duplicates, strict enforcement of ordering is required across replicas. No additional cluster configuration is required beyond Step 2 above. #### Table Configuration To enable dedup with pauseless, you need to add the following configs in addition to the above mentioned configuration: ```json theme={null} { "routing": { "instanceSelectorType": "strictReplicaGroup" }, "dedupConfig": { "dedupEnabled": true }, "task": { "taskTypeConfigsMap": { "DedupSnapshotCreationTask": { "schedule": "0 0 12 * * ?" } } }, "ingestionConfig": { "streamIngestionConfig": { "pauselessConsumptionEnabled": true, "enforceConsumptionInOrder": true } } } ``` ### Configuration Parameters | Parameter | Value | Reason | | :- | :- | :- | | instanceSelectorType | strictReplicaGroup | Required for all dedup tables. Routes queries to a single replica group so each query sees a consistent dedup state. | | dedupEnabled | true | Activates record-level deduplication. | | enforceConsumptionInOrder | true | Ensures Segment N+1 cannot start until Segment N finishes. Prevents data gaps if a server lags. | > [!NOTE] > > The following settings are applied automatically for pauseless dedup tables and no longer need to be set explicitly: > > * `controller.segment.error.autoReset` (controller config): defaults to `true`, and error-segment repair always runs for pauseless tables regardless of this flag. > * `parallelSegmentConsumptionPolicy`: when unset, pauseless dedup tables automatically resolve to `ALLOW_DURING_BUILD_ONLY`, which releases the consumption semaphore only during the build phase (while the server still holds its exact local copy of the data) to ensure correctness. > * `allowDedupConsumptionDuringCommit`: deprecated in favor of `parallelSegmentConsumptionPolicy`; the default `false` is the required value. > * `useIdealStateToCalculatePreviousSegment`: defaults to `false`, which is the recommended value (avoids extra Zookeeper load). ## Observability | Metric Name | Type | Unit | Description | Usage | | :- | :- | :- | :- | :- | | **Controller Metrics** | | | | | | `PAUSELESS_CONSUMPTION_ENABLED` | Gauge (per-table) | Boolean (0/1) | Indicates whether pauseless consumption is enabled for the table. | Quick verification if pauseless mode is active for a table. | | `PAUSELESS_SEGMENTS_IN_ERROR_COUNT` | Gauge (per-table) | Count | Number of segments where ALL replicas are in ERROR state. | Critical metric to detect when segments are completely failed across all replicas. Non-zero value indicates immediate attention needed. | | `PAUSELESS_SEGMENTS_IN_UNRECOVERABLE_ERROR_COUNT` | Gauge (per-table) | Count | Number of segments in COMMITTING state with all replicas in ERROR that cannot be auto-repaired. | Tracks segments that need manual recovery, typically for tables with dedup/partial upsert where auto-repair is disabled. | | **Server Metrics** | | | | | | `PAUSELESS_CONSUMPTION_ENABLED` | Gauge (per-table) | Boolean (0/1) | Server-side indicator of pauseless consumption status. | Verify server and controller are in sync about pauseless mode. | ### Measuring Ingestion Lag To assess the impact of pauseless consumption, monitor the offset lag during ingestion using Grafana. Update the query for the metric, “Realtime Ingestion Delay in Ms (Avg)”, as follows: #### Ingestion Lag Query (Grafana) ```sql theme={null} avg by (${GroupBy:csv}) (pinot_server_realtimeIngestionOffsetLag_Value{kubernetes_pod_name=~"$Server$" , table=~"$Tables$"}) ``` ### Debugging APIs #### 1. Get Pauseless Debug Information **Description:** Retrieves diagnostic information to identify segments that are stuck in COMMITTING or are in ERROR state. **Usage:** * **Error detection:** Identify which servers have segments in ERROR state. * **Commit tracking:** Monitor segments that are stuck in the COMMITTING state. * **Debugging hangs:** If commits appear stalled, check whether segments remain in the `committingSegments` list for extended periods. **API**: `GET /tables/{tableName}/pauselessDebugInfo` **Response** ```json theme={null} { "instanceToErrorSegmentsMap": { "Server_host1_8098": ["segment1", "segment2"], "Server_host2_8098": ["segment3"] }, "committingSegments": [ "tableName__0__123__20231201T1000Z", "tableName__1__456__20231201T1005Z" ] } ``` #### 2. Get Re-ingestion Jobs **Description:** Returns list of all currently running re-ingestion jobs. \ **Usage:** Useful for monitoring auto-repair progress. \ **API:** `GET /reingestSegment/jobs` #### 3. Pauseless Debug Metadata Path **Description:** Tracks all segments currently in COMMITTING state for debugging purposes. **Usage:** * Maintains a persisted list of segments that are currently in the commit process. * Used by the controller to track the lifecycle of segment commits. * Entries are automatically cleaned up once segments complete the commit. * Useful for understanding which segments were in flight during failures and can be accessed using the API listed above. **API:** `GET /PAUSELESS_DEBUG_METADATA/{tableNameWithType}` **Response:** ```json theme={null} { "id": "tableName_REALTIME", "listFields": { "committingSegments": [ "tableName__0__123__20231201T1000Z", "tableName__1__456__20231201T1005Z" ] } } ``` ## Operational Troubleshooting ### Ordering Enforcement Pinot enforces segment ordering using a `ConsumerCoordinator` that serializes Helix `OFFLINE → CONSUMING` state transitions. Consumers wait for prior segments to either finish consumption or be fully loaded before starting to consume new segments. This guarantees strict ordering and prevents gaps in data ingestion. The enforcement strategy depends on the `useIdealStateToCalculatePreviousSegment` flag. It is strongly recommended to keep this flag set to `false` to avoid additional load on Zookeeper. Pinot includes a built-in safety mechanism that handles ordering without relying on IdealState. ### Segment Build Failures (Rollback) In rare scenarios, such as a server crash before a segment is uploaded, it may be necessary to roll back ingestion to the last successfully committed segment. To perform a rollback: 1. Pause ingestion for the table. 2. Identify the last valid committed segment. 3. Execute the delete operation using the Delete from Sequence Number API. **API**: `DELETE /deleteSegmentsFromSequenceNum/{tableNameWithType}?segments=segmentA,segmentB` This operation deletes all segments with a sequence ID greater than or equal to the specified segment. 4\. Resume ingestion for the table. ### Server Restart Hangs If a server restarts and continues consuming data but does not transition to the `READY` state, follow these steps: 1. Pause ingestion for the table. 2. Wait for the server to transition to `READY`. This typically takes a few minutes. 3. Resume ingestion. Once resumed, the server will catch up quickly. ### Ingestion Lag Spikes When enabling pauseless consumption, ingestion must be paused first. Resuming ingestion forces a commit of all active segments simultaneously. During this transition, expect a temporary spike in controller load and ingestion lag. This behavior is expected and limited to the enablement phase. ### COMMITTING vs ERROR State If a segment is in the `COMMITTING` state in Zookeeper but appears as `ERROR` or `OFFLINE` in the External View, corrective validation is required. Run the Realtime Segment Validation job manually, or wait for the scheduled execution. The schedule is controlled by `controller.realtime.segment.validation.frequencyPeriod`, which defaults to 1 hour. ### Key Logging Patterns for Debugging #### Success Path Logs * "Starting to commit changes to ZK and ideal state for the segment: during pauseless ingestion" * "Successfully updated segment metadata for segment: " * "Removing segment: from committing segment list" #### Warning Logs * "Committing segment: was not uploaded to deep store" * "Found segments with at least one replica in ERROR state" * "Segment: in table: is COMMITTING with all replicas in ERROR state" #### Error Logs * "Failed to update committing segments list for table: " * "No alive server found to re-ingest segment: in table: " # Additional Configuration Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/additional-config-create-table Configure advanced settings for the table in the Data Portal, including ingestion behavior and time partitioning for Apache Pinot tables. These are crucial for correct table functioning and data integrity in Apache Pinot. This section is a key part of the table creation process in **Data Portal**, appearing after the **Data Modeling** step. Open the StarTree Cloud console and navigate to the Data Portal section. ## Step 1: Table Details In this step, users are required to provide core table-level metadata that governs how the table will behave in Pinot. ### Table Name Enter a unique and meaningful name for the table. This name will be used to reference the table within Pinot and downstream tools. ### Replication Factor Specify the number of replicas for each segment. A higher replication factor improves fault tolerance and availability but consumes more resources. ### Table Type Select whether the table is a: * **Fact Table**: Used for large-scale event or transactional data. Requires a **Time Column** to be selected, which helps with time-based retention, partitioning, and querying. * **Dimension Table**: Typically used for lookups or joins. Requires the user to define a list of **Primary Key Columns**, ordered as per their significance, to uniquely identify each record. After filling in these fields, click **Save** to confirm and move to the next step. ## Step 2: Ingestion Type Configuration This step appears only if the data source is a streaming platform such as **Kafka**, **Kinesis**, etc. Users must define how Pinot should handle incoming records—whether to **append**, **update**, or **deduplicate** them. ### Choose Ingestion Type You have three options: * **Standard Table**: Default ingestion mode where records are simply appended as they arrive. No deduplication or update logic is applied. * **Upsert Table**: Enables update capabilities where new events can replace older ones based on the primary key. Suitable for late-arriving data and event corrections. * **Deduplication Table**: Ensures that only the most recent record for a given key is retained and drops duplicate records during ingestion. Useful in scenarios where duplicates are common in the event stream. *** #### If “Upsert Table” is selected: You will be required to configure the following: * **Upsert Type**: Select from the dropdown options (e.g., *Full* or *Partial*) depending on whether you want the entire row to be replaced or only specific fields updated. * **Partial Upsert Strategy** (only if Partial Upsert is selected): Define how each upsertable column should be handled—common strategies include *overwrite*, *increment*, or *ignore*. * **Primary Key**: Choose one or more columns that uniquely identify each row. These will be used by Pinot to determine whether incoming data should update an existing record. *** #### If “Deduplication Table” is selected: You will be required to configure the **Dedupe Primary Key**. Select the column(s) from the dropdown that should be used to identify and eliminate duplicate records in the stream. ## Step 3: Create Table Review the overall table configuration and create the table in the last step. In the final step of ingestion using the Data Portal, the table schema that was configured in the previous steps is displayed. Review the configuration details. You can edit the configurations and check the sample data to see if your edits produce the required output. Once the schema is ready, click **Create Table**. # Connect to Azure Data Lake Storage (ADLS) Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/batch/adls Create a connection to ingest files from Azure Data Lake Storage. **1. In the Data Portal, click Tables and then click Create Table.** **2. Select ADLS as the Data Source.** **3. Create a New Connection.** Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**. Enter a **Source Name** for the new connection. Select the **Authentication Type** from the drop-down list. **4. Configure Connection Parameters** ## Connecting to ADLS Using the Access Key Use the following JSON configuration when ADLS is set up with **basic authentication** using an access key. ```json theme={null} { "input.fs.className": "org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS", "input.fs.prop.accountName": "", "authenticationType": "ACCESS_KEY", "input.fs.prop.accessKey": "", "input.fs.prop.fileSystemName": "", "inputDirURI": "abfs://@.dfs.core.windows.net//" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `input.fs.className` | Yes | The class name of the ADLS Gen2 Pinot file system. | | `input.fs.prop.accountName` | Yes | The Azure Storage account name used for ADLS Gen2 integration. | | `authenticationType` | Optional | Set as `ACCESS_KEY` to use an access key for authentication. | | `input.fs.prop.accessKey` | Yes | The access key used for authentication when `authenticationType` is `ACCESS_KEY`. | | `input.fs.prop.fileSystemName` | Yes | The name of the file system (container) in the Azure Storage account. | | `inputDirURI` | Yes | The URI of the input directory in ADLS Gen2 where Pinot reads data. | ## Connecting to ADLS Using Azure Active Directory Use the following JSON configuration when ADLS is set up with **Azure Active Directory authentication**. ```json theme={null} { "input.fs.className": "org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS", "input.fs.prop.accountName": "", "authenticationType": "AZURE_AD", "input.fs.prop.clientId": "", "input.fs.prop.clientSecret": "", "input.fs.prop.tenantId": "", "input.fs.prop.fileSystemName": "", "inputDirURI": "abfs://@.dfs.core.windows.net//" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `input.fs.className` | Yes | The class name of the ADLS Gen2 Pinot file system. Must be set to `org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS` for ADLS Gen2 integration. | | `input.fs.prop.accountName` | Yes | The name of the Azure Storage account to be used for ADLS Gen2 integration. | | `authenticationType` | Optional | The type of authentication used to access the ADLS storage. Set as `AZURE_AD` to use Azure Active Directory (AAD) Service Principal for authentication. | | `input.fs.prop.clientId` | Yes | The Client ID of the Azure Service Principal used for authentication. | | `input.fs.prop.clientSecret` | Yes | The Client Secret key for the Azure Service Principal. | | `input.fs.prop.tenantId` | Yes | The Tenant ID associated with the Azure Active Directory (AAD). | | `input.fs.prop.fileSystemName` | Yes | The name of the file system (container) in the Azure Storage account. | | `inputDirURI` | Yes | The URI for the input directory in ADLS Gen2 where Pinot reads data. It is generally in the format: `abfs://my-container-name@myazure-storage-account.dfs.core.windows.net/data-directory/`. | ## Connecting to ADLS Using Azure Active Directory with Proxy Use the following JSON configuration when ADLS is set up with **Azure Active Directory authentication via a proxy server**. ```json theme={null} { "input.fs.className": "org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS", "input.fs.prop.accountName": "", "authenticationType": "AZURE_AD_WITH_PROXY", "input.fs.prop.clientId": "", "input.fs.prop.clientSecret": "", "input.fs.prop.tenantId": "", "input.fs.prop.proxyHost": "", "input.fs.prop.proxyPort": "", "input.fs.prop.proxyUsername": "", "input.fs.prop.proxyPassword": "", "input.fs.prop.fileSystemName": "", "inputDirURI": "abfs://@.dfs.core.windows.net//" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `input.fs.className` | Yes | The class name of the ADLS Gen2 Pinot file system. Must be set to `org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS` for ADLS Gen2 integration. | | `input.fs.prop.accountName` | Yes | The name of the Azure Storage account to be used for ADLS Gen2 integration. | | `authenticationType` | Optional | The type of authentication used to access the ADLS storage. Set as `AZURE_AD_WITH_PROXY` to use Azure Active Directory (AAD) Service Principal for authentication via a proxy server. | | `input.fs.prop.clientId` | Yes | The Client ID of the Azure Service Principal used for authentication. | | `input.fs.prop.clientSecret` | Yes | The Client Secret key for the Azure Service Principal. | | `input.fs.prop.tenantId` | Yes | The Tenant ID associated with the Azure Active Directory (AAD). | | `input.fs.prop.proxyHost` | Yes | The hostname of the proxy server used to connect to ADLS. | | `input.fs.prop.proxyPort` | Yes | The port number of the proxy server. | | `input.fs.prop.proxyUsername` | Yes | The username for proxy authentication. | | `input.fs.prop.proxyPassword` | Yes | The password for proxy authentication. | | `input.fs.prop.fileSystemName` | Yes | The name of the file system (container) in the Azure Storage account. | | `inputDirURI` | Yes | The URI for the input directory in ADLS Gen2 where Pinot reads data. It is generally in the format: `abfs://my-container-name@myazure-storage-account.dfs.core.windows.net/data-directory/`. | **5. Test the Connection and Configure Data Ingestion** After you have configured the connection properties, test the connection to ensure it is working. ## Configure Data Ingestion When the connection is successful, use the following JSON to configure additional data settings: ```json theme={null} { "inputFormat": "", "includeFileNamePattern": "" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `inputFormat` | Yes | The format of the input files. Supported values include **csv, json, avro, parquet**, etc. | | `includeFileNamePattern` | Yes | The glob pattern to identify which files to include for ingestion. This is useful when the input directory contains a mix of files, and only specific files should be ingested. | | `excludeFileNamePattern` | Optional | The glob pattern to identify which files to exclude from ingestion. This is useful when the input directory contains a mix of files, and only specific files should not be ingested. | ## Configure Record Reader Configure the record reader to customize how the file format is read during ingestion. > **CSV** > > The `CSVRecordReaderConfig` is used for handling CSV files with the following customizable options: > > * *header*: Provide a header when the input file has no headers. > * *skipHeader*: Provide an alternate header when the input file has a corrupt header. > * *delimiter*: Use an alternate delimiter when fields are not separated by the default delimiter comma. > * *skipUnParseableLines*: Skip records that are not parseable instead of failing ingestion. > > **Example:** Provide a header when the input file has no headers. > > ```bash theme={null} > { > "header": "colA,colB,colC" > } > ``` > > **Example:** Provide an alternate header when the input file has a corrupt header. > > ```bash theme={null} > { > "header": "colA,colB,colC", > "skipHeader": "true" > } > ``` > > **Example:** Use an alternate delimiter when fields are not separated by the default delimiter comma. > > ```bash theme={null} > { > "delimiter": ";" > } > ``` > > OR > > ```bash theme={null} > { > "delimiter": "\\t" > } > ``` > > **Example:** Skipping records that are not parseable instead of failing ingestion. This option to be used with caution as it can lead to data loss. > > ```bash theme={null} > { > skipUnParseableLines: "true" > } > ``` > > **Example:** Handling CSV files with no header, tab-separated fields, empty lines, and unparsable records. > > ```bash theme={null} > { > "header": "colA,colB,colC", > "delimiter": "\\t", > "ignoreEmptyLines": "true", > "skipUnParseableLines": "true" > } > ``` > > For a comprehensive list of available CSV record reader configurations, see the Pinot [CSV documentation](https://docs.pinot.apache.org/basics/data-import/pinot-input-formats#csv). > **AVRO** > > One configuration option `AvroRecordReaderConfig` is supported. > > * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS. > > ```bash theme={null} > { > "enableLogicalTypes": "true" > } > ``` > > For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start. > **Parquet** > > For Parquet files, Data Manager provides the ParquetRecordReaderConfig with customizable configurations in Data Manager. > > Use Parquet Avro Record Reader: > > ```bash theme={null} > { > "useParquetAvroRecordReader" : "true" > } > ``` > > When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetAvroRecordReader` > > Use Parquet Native Record Reader: > > ```bash theme={null} > { > "useParquetNativeRecordReader" : "true" > } > ``` > > When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetNativeRecordReader` **6. Preview the Data** Click Show Sample Data to preview the source data before finalizing the configuration. > **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)** > > Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview). # Connect to Google BigQuery Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/batch/bigquery Create a connection to load data from Google BigQuery tables and views. **1. In the Data Portal, click Tables and then click Create Table.** **2. Select BigQuery as the Data Source.** **3. Create a New Connection.** Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**. Enter a **Source Name** for the new connection. **4. Configure Connection Parameters** ## Service Account Authentication Use the following JSON configuration: ```json theme={null} { "sql.bigquery.projectid": "", "sql.bigquery.jsonkey": "" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `sql.bigquery.projectId` | Yes | The GCP project ID. Find your project ID in the Google Cloud console. | | `sql.bigquery.jsonKey` | Yes | The string-encoded Google service account key. Ensure the correct JSON key format. See Google's documentation for creating a service account key. | **5. Test the Connection and Configure Data Ingestion** After you have configured the connection properties, test the connection to ensure it is working. ## Configure Data Ingestion When the connection is successful, use the following JSON to configure additional data settings: ```json theme={null} { "sql.queryTemplate": "SELECT * FROM
WHERE BETWEEN $START AND $END", "sql.timeColumnName": "TIME_COLUMN", "sql.timeColumnFormat": "yyyy-MM-dd", "sql.startTime": "", "sql.endTime": "" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `sql.queryTemplate` | Yes | The SQL query template for fetching data from BigQuery. | | `sql.timeColumnName` | Yes | The time column used for filtering data while ingesting. | | `sql.timeColumnFormat` | Yes | The datetime format of the time column used for filtering. | | `sql.startTime` | Yes | The start time for data extraction. This value is dynamically used in the `$START` placeholder of `sql.queryTemplate`. It should match the specified time column format. | | `sql.endTime` | Yes | The end time for data extraction. This value is dynamically used in the `$END` placeholder of `sql.queryTemplate`. It should match the specified time column format. | **6. Preview the Data** Click Show Sample Data to preview the source data before finalizing the configuration. > **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)** > > Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview). # Connect to Delta Lake Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/batch/deltalake The Delta Lake connector allows ingestion from a Delta Lake data platform hosted in S3 and GCS into Pinot. A native integration with Delta Lake is not supported currently, however a **Custom Connection** can be used to configure the source. **1. In the Data Portal, click Tables and then click Create Table.** **2. Select Custom Connection as the Data Source.** **3. Select Data Lake as the Category.** **4. Configure the Task Configs.** Delta Lake ingestion is driven by the offline table task configs. Refer to the [Delta Lake](/corecapabilities/ingestdata/adv-concepts/batch/delta-lake-connector) connector documentation to determine the connection, authentication, and ingestion parameters for your setup, and set them in the task configs. > **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)** > > Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview). # Connect to Google Cloud Storage (GCS) Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/batch/gcs Create a connection to ingest files from Google Cloud Storage. **1. In the Data Portal, click Tables and then click Create Table.** **2. Select GCS as the Data Source.** **3. Create a New Connection.** Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**. Enter a **Source Name** for the new connection. **4. Configure Connection Parameters.** ## Service Account Authentication Use the following JSON script to set up your service account: ```json theme={null} { "inputDirURI": "gs:///", "input.fs.prop.projectId": "", "input.fs.prop.jsonKey": "", "input.fs.className": "org.apache.pinot.plugin.filesystem.GcsPinotFS" } ``` ### Property Descriptions The following table outlines the required properties for configuring a Google Cloud Storage (GCS) connection. | Property | Required | Description | | - | - | - | | `input.fs.prop.projectId` | Yes | The GCP project ID. Find your project ID in the Google Cloud console. | | `input.fs.prop.jsonKey` | Yes | The string-encoded Google service account key. Include the appropriate format for your selected JSON key. See Google documentation on how to create a service account key. | | `inputDirURI` | Yes | The path to input file(s). | | `input.fs.className` | Yes | The class name (`org.apache.pinot.plugin.filesystem.GcsPinotFS`) for the Pinot file system. | **5. Test the Connection and Configure Data Settings** After you have configured the connection properties, test the connection to ensure it is working. ## Configure Data Ingestion When the connection is successful, use the following JSON to configure additional data settings: ```json theme={null} { "inputFormat": "", "includeFileNamePattern": "gs://*.csv" } ``` ### Property Descriptions The following table outlines the required and optional properties for configuring data ingestion in Google Cloud Storage (GCS). | Property | Required | Description | | - | - | - | | `inputFormat` | Yes | The input file format. Supported values include **csv, json, avro, parquet**, etc. | | `includeFileNamePattern` | Optional | The glob pattern to identify which files to include for ingestion. This parameter fetches data only from the files in the `inputDirURI` path that match this pattern. | | `excludeFilePatternMatch` | Optional | The glob pattern to identify which files to exclude from ingestion. This parameter restricts fetching data from the files in the `inputDirURI` path that match this pattern. | ## Configure Record Reader Configure the record reader to customize how the file format is read during ingestion. > **CSV** > > The `CSVRecordReaderConfig` is used for handling CSV files with the following customizable options: > > * *header*: Provide a header when the input file has no headers. > * *skipHeader*: Provide an alternate header when the input file has a corrupt header. > * *delimiter*: Use an alternate delimiter when fields are not separated by the default delimiter comma. > * *skipUnParseableLines*: Skip records that are not parseable instead of failing ingestion. > > **Example:** Provide a header when the input file has no headers. > > ```bash theme={null} > { > "header": "colA,colB,colC" > } > ``` > > **Example:** Provide an alternate header when the input file has a corrupt header. > > ```bash theme={null} > { > "header": "colA,colB,colC", > "skipHeader": "true" > } > ``` > > **Example:** Use an alternate delimiter when fields are not separated by the default delimiter comma. > > ```bash theme={null} > { > "delimiter": ";" > } > ``` > > OR > > ```bash theme={null} > { > "delimiter": "\\t" > } > ``` > > **Example:** Skipping records that are not parseable instead of failing ingestion. This option to be used with caution as it can lead to data loss. > > ```bash theme={null} > { > skipUnParseableLines: "true" > } > ``` > > **Example:** Handling CSV files with no header, tab-separated fields, empty lines, and unparsable records. > > ```bash theme={null} > { > "header": "colA,colB,colC", > "delimiter": "\\t", > "ignoreEmptyLines": "true", > "skipUnParseableLines": "true" > } > ``` > > For a comprehensive list of available CSV record reader configurations, see the Pinot [CSV documentation](https://docs.pinot.apache.org/basics/data-import/pinot-input-formats#csv). > **AVRO** > > One configuration option `AvroRecordReaderConfig` is supported. > > * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS. > > ```bash theme={null} > { > "enableLogicalTypes": "true" > } > ``` > > For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start. > **Parquet** > > For Parquet files, Data Manager provides the ParquetRecordReaderConfig with customizable configurations in Data Manager. > > Use Parquet Avro Record Reader: > > ```bash theme={null} > { > "useParquetAvroRecordReader" : "true" > } > ``` > > When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetAvroRecordReader` > > Use Parquet Native Record Reader: > > ```bash theme={null} > { > "useParquetNativeRecordReader" : "true" > } > ``` > > When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetNativeRecordReader` **6. Sample Data** Click **Show Sample Data** to see a preview of the source data. > **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)** > > Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview). # Connect to Amazon S3 Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/batch/s3 Create a connection to ingest files stored in Amazon S3 buckets. > [!WARNING] > > **Keep your S3 data in the same AWS region as your StarTree cluster.** For BYOC deployments, StarTree creates an S3 gateway endpoint in your cluster's region, so reads from buckets in that region stay on the AWS private network. A gateway endpoint does not cover buckets in other regions. Reads from those buckets go through your VPC's NAT Gateway (node → NAT Gateway → Internet Gateway → public S3 endpoint). For large datasets, the NAT Gateway processing charges show up on your AWS bill and can be significant. A one-time read may not justify moving the data, but the charges from repeatedly reading a large dataset add up quickly. > > If the bucket has to stay in a different region, set up a [cross-region S3 interface endpoint (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-cross-region-privatelink-support.html) before you onboard the data. See [Networking and data transfer costs](/corecapabilities/external-table/best-practices-and-configs#networking-and-data-transfer-costs) for details. **1. In the Data Portal, click Tables and then click Create Table.** **2. Select S3 as the Data Source.** **3. Create a New Connection.** Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**. Enter a **Source Name** for the new connection. Select the **Authentication Type** from the drop-down list. **4. Configure Connection Parameters.** ## Connecting to S3 Using Basic Authentication Use the following JSON configuration when S3 is set up with **basic authentication** using an **access key** and **secret key**. ```json theme={null} { "inputDirURI": "s3:////", "input.fs.prop.region": "", "input.fs.prop.accessKey": "", "input.fs.prop.secretKey": "" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `inputDirURI` | Yes | URI of the input directory/files for ingestion. It tells Pinot where the data resides. | | `input.fs.prop.region` | Yes | Region of the file system (e.g., `us-east-1`). | | `input.fs.prop.accessKey` | Yes | Access key for authentication to the file system. | | `input.fs.prop.secretKey` | Yes | Secret key for authentication (paired with access key). | ## Connecting to S3 Using IAM-Based Authentication Please follow [these steps](/corecapabilities/ingestdata/recipes/iam-role-s3) to create IAM role for S3 setup. Use the following JSON configuration when S3 is set up with IAM-based authentication by assuming an IAM Role for secure access. ```json theme={null} { "inputDirURI": "s3:////", "input.fs.prop.roleArn": "arn:aws:iam:::role/", "input.fs.prop.region": "", "input.fs.prop.externalId": "", "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `inputDirURI` | Yes | URI of the input directory/files for ingestion. It tells Pinot where the data resides. | | `input.fs.prop.region` | Yes | Region of the file system (e.g., `us-east-1`). | | `input.fs.prop.externalId` | Yes | External ID of the AWS Account of your StarTree Cloud. | | `input.fs.prop.roleArn` | Yes | The Amazon Resource Name (ARN) of an AWS IAM role to assume for accessing AWS S3. Allows Pinot to securely access resources in different AWS accounts. **Example Use Case**: If Pinot is running in Account A but the S3 bucket is in Account B, you can assume a role in Account B that grants access to the bucket. The role must have permissions like "s3:List\*" and "s3:GetObject" for proper access. | | `input.fs.className` | Yes | The file system implementation class to access the input directory. **Examples:** - `org.apache.pinot.plugin.filesystem.S3PinotFS` (for Amazon S3) - `org.apache.pinot.plugin.filesystem.LocalPinotFS` (for local file systems) - `org.apache.pinot.plugin.filesystem.HadoopPinotFS` (for HDFS) | **5. Test the Connection and Configure Data Ingestion.** After you have configured the connection properties, test the connection to ensure it is working. ## Configure Data Ingestion When the connection is successful, use the following JSON to configure additional data settings: ```json theme={null} { "inputFormat": "", "includeFileNamePattern": "", "excludeFileNamePattern": "" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `inputFormat` | Yes | The format of the input files. Supported values include **csv, json, avro, parquet**, etc. | | `includeFileNamePattern` | Yes | The glob pattern to filter which files to include for ingestion. Used when the input directory contains mixed files and only specific files should be ingested. | | `excludeFileNamePattern` | No | The glob pattern to filter which files to exclude from ingestion. Used when the input directory contains mixed files and only specific files should not be ingested. | ## Configure Record Reader Configure the record reader to customize how the file format is read during ingestion. > **CSV** > > The `CSVRecordReaderConfig` is used for handling CSV files with the following customizable options: > > * *header*: Provide a header when the input file has no headers. > * *skipHeader*: Provide an alternate header when the input file has a corrupt header. > * *delimiter*: Use an alternate delimiter when fields are not separated by the default delimiter comma. > * *skipUnParseableLines*: Skip records that are not parseable instead of failing ingestion. > > **Example:** Provide a header when the input file has no headers. > > ```bash theme={null} > { > "header": "colA,colB,colC" > } > ``` > > **Example:** Provide an alternate header when the input file has a corrupt header. > > ```bash theme={null} > { > "header": "colA,colB,colC", > "skipHeader": "true" > } > ``` > > **Example**: Use an alternate delimiter when fields are not separated by the default delimiter comma. > > ```bash theme={null} > { > "delimiter": ";" > } > ``` > > OR > > ```bash theme={null} > { > "delimiter": "\\t" > } > ``` > > **Example:** Skipping records that are not parseable instead of failing ingestion. This option to be used with caution as it can lead to data loss. > > ```bash theme={null} > { > skipUnParseableLines: "true" > } > ``` > > **Example:** Handling CSV files with no header, tab-separated fields, empty lines, and unparsable records. > > ```bash theme={null} > { > "header": "colA,colB,colC", > "delimiter": "\\t", > "ignoreEmptyLines": "true", > "skipUnParseableLines": "true" > } > ``` > > For a comprehensive list of available CSV record reader configurations, see the Pinot [CSV documentation](https://docs.pinot.apache.org/basics/data-import/pinot-input-formats#csv). > **AVRO** > > One configuration option `AvroRecordReaderConfig` is supported. > > * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS. > > ```bash theme={null} > { > "enableLogicalTypes": "true" > } > ``` > > For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start. > **Parquet** > > For Parquet files, Data Manager provides the ParquetRecordReaderConfig with customizable configurations in Data Manager. > > Use Parquet Avro Record Reader: > > ```bash theme={null} > { > "useParquetAvroRecordReader" : "true" > } > ``` > > When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetAvroRecordReader` > > Use Parquet Native Record Reader: > > ```bash theme={null} > { > "useParquetNativeRecordReader" : "true" > } > ``` > > When this config is used the parquet record reader used is: `org.apache.pinot.plugin.inputformat.parquet.ParquetNativeRecordReader` **6. Preview the Data** Click Show Sample Data to preview the source data before finalizing the configuration. > **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)** > > Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview). # Connect to Snowflake Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/batch/snowflake Create a connection to load batch data from Snowflake into StarTree Cloud. **1. In the Data Portal, click Tables and then click Create Table.** **2. Select Snowflake as the Data Source.** **3. Create a New Connection.** Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**. Enter a **Source Name** for the new connection. **4. Configure Connection Parameters** ## Connecting to Snowflake Using Basic Authentication Use the following JSON configuration when Snowflake is set up with **basic authentication** using **user** and **password**. ```json theme={null} { "sql.snowflake.account": "", "sql.snowflake.db": "", "sql.snowflake.schema": "", "authentication.type": "basic", "sql.snowflake.user": "", "sql.snowflake.password": "" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `sql.snowflake.account` | Yes | The name of the Snowflake account. Found in your Snowflake URL (e.g., `xyz.snowflakecomputing.com`). | | `sql.snowflake.db` | Yes | The name of the database where the data resides. | | `sql.snowflake.schema` | Yes | The name of the schema where the data resides. | | `sql.snowflake.user` | Yes | The username for authenticating with Snowflake. Ensure the user has appropriate read access. | | `sql.snowflake.password` | Yes | The password for authenticating with Snowflake. | ## Connecting to Snowflake Using Key Pair Authentication Use the following JSON configuration when Snowflake is set up with Key Pair based authentication. ```json theme={null} { "sql.snowflake.account": "", "sql.snowflake.db": "", "sql.snowflake.schema": "", "sql.snowflake.user": "", "authentication.type": "key_pair", "sql.snowflake.privateKey": "", "sql.snowflake.privateKeyPassword": "" } ``` ### Property Descriptions | Parameter | Required | Description | | - | - | - | | `sql.snowflake.account` | Yes | The name of the Snowflake account. Found in your Snowflake URL (e.g., `xyz.snowflakecomputing.com`). | | `sql.snowflake.db` | Yes | The name of the database where the data resides. | | `sql.snowflake.schema` | Yes | The name of the schema where the data resides. | | `sql.snowflake.user` | Yes | The username for authenticating with Snowflake. Ensure the user has appropriate read access. | | `sql.snowflake.privateKey` | Yes | Make sure the private key is Base64-encoded for secure authentication. | | `sql.snowflake.privateKeyPassword` | Yes | Make sure the password for the private key is Base64-encoded for secure authentication. | For more details, please refer to the [Snowflake official documentation here](https://docs.snowflake.com/en/developer-guide/jdbc/jdbc-configure#using-key-pair-authentication-and-key-rotation) **5. Test the Connection and Configure Data Ingestion** After you have configured the connection properties, test the connection to ensure it is working. ## Configure Data Ingestion When the connection is successful, use the following JSON to configure additional data settings: ```json theme={null} { "sql.queryTemplate": "SELECT * FROM
WHERE BETWEEN $START AND $END", "sql.timeColumnName": "TIME_COLUMN", "sql.timeColumnFormat": "yyyy-MM-dd", "sql.startTime": "", "sql.endTime": "" } ``` ### Property Descriptions | Property | Required | Description | | - | - | - | | `sql.queryTemplate` | Yes | The SQL query template for fetching data from Snowflake. | | `sql.timeColumnName` | Yes | The time column used for filtering data while ingesting. | | `sql.timeColumnFormat` | Yes | The datetime format of the time column used for filtering. | | `sql.startTime` | Yes | The start time for data extraction. Dynamically used in the `$START` placeholder of `sql.queryTemplate`. It should match the mentioned time column format. | | `sql.endTime` | Yes | The end time for data extraction. Dynamically used in the `$END` placeholder of `sql.queryTemplate`. It should match the mentioned time column format. | ## Additional Optional Parameters Use the following JSON for optional configurations: ```json theme={null} { "sql.bufferTimePeriod": "0d", "sql.numBucketsToLookBack": "0", "sql.bucketTimePeriod": "1d" } ``` ### Optional Configuration Properties | Property | Required | Description | | - | - | - | | `sql.bufferTimePeriod` | No | The buffer time period before fetching data. Default value: `0d`. Defines additional time to consider before fetching data. | | `sql.bucketTimePeriod` | No | The time period granularity for each data bucket. Default value: `1d`. | | `sql.numBucketsToLookBack` | No | The number of time buckets to look back when fetching data. Default value: `0`. Used for incremental data fetching. | **6. Preview the Data** Click Show Sample Data to preview the source data before finalizing the configuration. > **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)** > > Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview). # Upload any file Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/batch/upload Upload a file up to 30MB in AVRO, CSV, JSON, Parquet, ProtoBuf, ORC. **1. In the Data Portal, click Tables and then click Create Table.** **2. Select Upload File as the Data Source.** **3. Configure File to ingest** 1. Drag and drop your file or click to use your computer's file manager to locate and select the file. 2. Select the file format if not auto inferred. You will be given additional configuration for tuning it as per your needs. Find more details on these configuration here at OSS Pinot documentation for [ supported file formats](https://docs.pinot.apache.org/manage-data/data-import/pinot-input-formats#supported-input-formats). **4. Preview the Data** Click Show Sample Data to preview the source data before finalizing the configuration. # Connect to Any Data Source Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/custom-connection Connect to any data source using the custom connector. Configure the connector to use a batch or streaming source. ## Step 1: Select the Data Source Category To begin the connection setup, select the **appropriate data source category** that best represents the type of system you are connecting to. This helps in identifying the **ingestion mode** and ensures the correct configurations are applied. ### **Batch Sources** * Select this option for **file-based systems**, such as: * **AWS S3** * **Google Cloud Storage (GCS)** * **Azure Blob Storage** * **HDFS** * **Local file systems** * These sources typically provide **static data** that can be **ingested periodically**. ### **Streaming Sources** * Choose this category for **real-time streaming data sources**, such as: * **Apache Kafka** * **Amazon Kinesis** * **Apache Pulsar** * These sources **continuously generate data**, requiring Pinot to process events in **real time**. ### **SQL-Based Sources** * Use this option when connecting to **traditional relational databases**, such as: * **Snowflake** * **Google BigQuery** * This setup allows Pinot to **fetch data using SQL queries**. *** ## Step 2: Configure the Data Source Connection Once you have selected the appropriate **data source category**, enter the required **connection configuration details**. These settings will depend on the type of source you are connecting to and should align with **Apache Pinot’s ingestion configuration**. After entering the configuration details correctly, proceed with **validating the connection**. *** ## Step 3: Preview the Data Click **Show Sample Data** to preview the source data before finalizing the configuration. > **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)** > > Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview). # Examples Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/data-modeling/concepts Derive new columns or modify existing columns during ingestion by applying expressions, functions, or logic. Derived columns are useful when: * You want to compute columns (examples: dayOfWeek, hour, lat\_lon\_hash). * You need to format or clean up incoming data (trimming strings, parsing timestamps). * You want to simplify querying by precomputing fields. Transformations are applied before data is persisted, meaning the transformed fields are stored as part of the segment. ### How it Works Pinot supports transformations through a section in the table config called `ingestionConfig`. The key part of this is the `transformConfigs`, where each transform is defined as a mapping between a column name and an expression. Here’s a basic structure: ```json theme={null} "ingestionConfig": { "transformConfigs": [ { "columnName": "eventDate", "transformFunction": "DATETIMECONVERT(event_timestamp, '1:MILLISECONDS:EPOCH', '1:DAYS:SIMPLE_DATE_FORMAT:yyyy-MM-dd')" }, { "columnName": "isLargeTransaction", "transformFunction": "amount > 1000" } ] } ``` #### Components * `columnName`: The name of the column to be created. * `transformFunction`: The expression or function that generates the column’s value. Transformation functions in JSON have the following 3 components: * transforms * timeTransforms * filter ```json theme={null} { "timeTransforms": [], "transforms": [], "filter": null } ``` ## Column operations ### Add new columns You can add new Dimension, Metric or Datetime columns. Introducing new columns to your Pinot table schema is a seamless process. Whether it's adding dimensions or metrics, mandatory mapping functions can be assigned to these newly created columns. Moreover, DateTime columns can be generated from an existing column, simplifying the management of date formats or varying levels of granularity within your datasets. This feature empowers you to enhance the richness of their data by incorporating new elements efficiently. #### Dimension column ```json theme={null} { "name": "student_name", "fieldType": "dimension", "dataType": "string" } ``` #### Metric column ```json theme={null} { "name": "studentID", "fieldType": "metric", "dataType": "long" } ``` ### Customize columns When organizing your data in Pinot tables, consider the column data type, for example, as a Dimension, Metric, or DateTime, you can adjust these attributes to suit the nature of the data fetched. Additionally, if your data is structured in arrays, it enables you to configure columns capable of handling multiple values. Moreover, you can set default values for columns in instances where the value is absent. This customization lets you align your data more effectively with your analytical needs. #### Single value dimension column ```json theme={null} { "name": "gender", "fieldType": "dimension", "dataType": "string" } ``` #### Multi-value dimension column For columns with multiple values (like `courses`), set the type as `dimension` with `mv(string)` for handling arrays. ```json theme={null} { "name": "courses", "fieldType": "dimension", "dataType": "mv(string)" } ``` #### Set values To set values for default, customize the max length for string columns, and set the column as mandatory. ```json theme={null} { "name": "description", "maxLength": "5122", "notNull": false, "dataType": "STRING", "defaultNullValue": "null", "singleValueField": true, "fieldType": "DIMENSION", } ``` ### Delete columns Keeping your Pinot table schema concise and relevant is crucial for efficient data management. Pinot lets you remove unnecessary columns by using the delete function in the schema interface. Retain only pertinent data columns in your dataset structures. ## Transform data with functions Apache Pinot offers a flexible and powerful mechanism for transforming data at ingestion time using transform functions. This allows users to shape and enrich data before it is indexed, which can simplify queries and improve performance. Modify columns through a variety of transformation functions to manipulate and refine your data. Test transformations to ensure they meet your expectations before applying changes. ### Unnesting Columns With JSON unnesting, you can flatten nested structures—including arrays and objects—directly during ingestion. This allows each nested element to be transformed into individual rows or columns, making the data significantly easier to query and analyze. Key Capabilities: 1. Flatten Arrays: Unnest array elements into separate rows, enabling row-level access to each item. 2. Extract Nested Fields: Promote nested JSON fields to top-level columns for direct querying. For more information, see [JSON Unnesting](./unnesting-json). ### Datetime-related examples #### Timestamp column with epoch format Create timestamp columns like `created_ts` with the format `EPOCH` and granularity `SECONDS`. ```json theme={null} { "name": "created_ts", "fieldType": "datetime", "dataType": "long", "format": "EPOCH|SECONDS|1", "granularity": "SECONDS|1" } ``` #### Transforming epoch seconds to epoch milliseconds Easily convert timestamps from seconds to milliseconds using transformation functions. ```json theme={null} { "timeTransforms": [ { "inputFieldName": "timestamp", "inputFormat": "EPOCH", "inputGranularity": "SECONDS", "outputFieldName": "created_ts", "outputFormat": "EPOCH", "outputGranularity": "MILLISECONDS" } } ``` ### String transformation example #### Creating a new string column by concatenation Combine columns (like `firstName` and `lastName`) into a new column called `name`. ```json theme={null} { "transforms": [ { "outputFieldName": "name", "transformFunction": "concat("firstName", "lastName")" } } ``` ## Supported Functions Pinot supports a rich set of built-in functions for transformations, including: * Math: ADD(), SUB(), MULT(), DIV() * String: LOWER(), UPPER(), CONCAT(), REGEXP\_EXTRACT() * Date/Time: DATETIMECONVERT(), DATE\_TRUNC(), EPOCH() * Logical: IF(), CASE WHEN, AND, OR, NOT * Geospatial: ST\_DISTANCE(), GEO\_TO\_H3() ### Further Reading For a complete and up-to-date guide to Pinot’s transformation functions and configuration please read the official documentation on [Apache Pinot Transform Functions](https://docs.pinot.apache.org/users/user-guide-query/query-syntax/supported-transformations) # Map type Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/data-modeling/maptype Use the **MAP** data type to efficiently store and query semi-structured data where the number of keys is large or unpredictable (e.g., Metric labels, cloud tags, or application metadata). ## **Overview and Purpose** The MAP data type  bridges the gap between flexible JSON blobs and rigid dedicated columns. It is designed for observability use cases where data is semi-structured and sparse. Unlike a standard JSON string column, the MAP type uses a specialized **Forward Index** that intelligently adapts to your data: * **Dense Keys:** Frequently occurring keys (e.g., host, region) are automatically detected and stored as efficient, column-like structures (Dense Storage) for fast **O(1)** retrieval. * **Sparse Keys:** Rare keys (e.g., error\_id) are stored in a compact Key-Value format (Sparse Storage) to save disk space. This hybrid approach allows you to maintain schema flexibility without sacrificing query performance or storage efficiency. ## **Configuration** To use the MAP data type, you must update both your **Schema** and **Table Configuration**. ### **1. Schema Definition** Define the column using complexFieldSpecs. You must specify the dataType as MAP and provide childFieldSpecs for both the key and the value. * **Key:** Must be of type STRING. * **Value:** Can be strongly typed (e.g., INT, STRING, LONG, FLOAT, DOUBLE). [https://docs.pinot.apache.org/configuration-reference/schema#complexfieldspec](https://docs.pinot.apache.org/configuration-reference/schema#complexfieldspec)  **Example Schema:** ``` {   "schemaName": "appLogs",   "complexFieldSpecs": [     {       "name": "attributes",       "dataType": "MAP",       "fieldType": "COMPLEX",       "childFieldSpecs": {         "key": {           "name": "key",           "dataType": "STRING",           "fieldType": "DIMENSION"         },         "value": {           "name": "value",           "dataType": "STRING",           "fieldType": "DIMENSION"         }       }     }   ] } ``` ### **2. Table Configuration** In the table configuration, you can tune the behavior of the Forward Index (to control Dense vs. Sparse behavior) and enable the JSON Index (for fast filtering). Configure the forward index properties under the indexes section. | **Parameter** | **Default** | **Description** | | :- | :- | :- | | maxDenseKeys | 32 | The maximum number of keys to promote to "Dense" (columnar) storage. | | denseKeyThresholdPct | 70.0 | The frequency threshold (0-100). A key must appear in this percentage of rows to be promoted to Dense storage. | | dynamicallyCreateDenseKeys | true | If true, the engine automatically identifies dense keys during segment creation. | ## Index Support on MAP type columns Currently Json and Composite Json Index are supported for MAP type columns. Enable the json index to support high-performance filtering (Inverted, Range, and Text indexing) on your Map keys. [https://docs.startree.ai/corecapabilities/manage-data/indexes/composite-json-index](https://docs.startree.ai/corecapabilities/manage-data/indexes/composite-json-index)  [https://docs.startree.ai/corecapabilities/manage-data/indexes/json-index#json-index](https://docs.startree.ai/corecapabilities/manage-data/indexes/json-index#json-index) **Example Table Config:** ``` {   "tableIndexConfig": {     "indexes": {       "forward": {         "configs": {           "maxDenseKeys": 50,           "denseKeyThresholdPct": 50.0,           "dynamicallyCreateDenseKeys": true         }       },       "json": {         "enabled": true,         "maxLevels": 2,         "excludeArray": false       }     }   } } ``` ## Queries  Querying MAP columns uses standard SQL syntax with the \*\*bracket \['key'] \*\*notation. ### **Selection** Select specific keys directly. If a key does not exist for a row, it returns null. ``` SELECT attributes['host'] AS host, attributes['region'] AS region FROM appLogs LIMIT 10 ``` ### **Filtering** Filter records based on map values. If the JSON index is enabled, these lookups use the inverted index for sub-second latency. ``` -- Equality Check SELECT count(\*) FROM appLogs  WHERE attributes['status'] = 'failed' -- Set Membership SELECT \* FROM appLogs  WHERE attributes['tier'] IN ('gold', 'platinum') ``` ### **Aggregation and Grouping** You can aggregate or group by specific map keys just like regular columns. ``` -- Group by a map key SELECT attributes['team'], COUNT(\*)  FROM appLogs  GROUP BY attributes['team'] -- Aggregate a map value (e.g., if Map value type is INT/DOUBLE) SELECT MAX(metricsMap['latency_ms'])  FROM appLogs ``` ## **Limitations** * **Nesting:** Currently, only one level of nesting is supported (Key -> Value). You cannot store a Map inside a Map. * **Multi-Value:** Multi-value Map columns are not supported. * **Key Type:** The Map key must always be a STRING. # Overview Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/data-modeling/overview Data modeling helps you organize and structure the data fetched from the data source into Pinot tables. Use the Data Portal to customize your columns, transform your data, and add or remove columns. The Data Portal provides a UI-based method to customize columns and to transform data. ![Dp Data Modeling Pn](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/dp_data_modeling.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=d0f267ec00a104aa71d282120aa13ff2) ## Column operations ### **Adding a column** 1. Click **Create column**. 2. Select the type: **Dimension**, **Metric**, or **Datetime**. 3. Specify the **New Column Name**. 4. Select a **Data Type** from the drop-down list. 5. Specify the **Max Length**. 6. Enter a **Default Value**. 7. *For Dimension columns:* To allow for multiple values, click the **Multi Value** toggle button. 8. *For Datetime columns:* Select the **Datetime Format** and specify the **Datetime Format Value**. 9. Specify the **Transformation** function to be applied when deriving the new column. 10. Click **Fetch sample data** to check if the data returned is as expected. 11. Click **Create column**. ### Customizing a column You can customize columns by changing any of the column properties. * To change the **Field Type** or **Data Type**, select from the drop-down lists. * To change the **Default Value** or **Max Length**, click on the value to update it. * To enable multiple values or to specify a mandatory column, click the toggle buttons for **Multi Value** and **Not Null**. ### Deleting a column To remove a column from the schema, click the 'trash' icon alongside the row for that column. > **Next Steps** > > For more information and examples for data modeling operations, see [Data Modeling Examples](/corecapabilities/ingestdata/dataportal/data-modeling/concepts). After completing Data Modeling, proceed to [Additional Configurations](/corecapabilities/ingestdata/dataportal/additional-config-create-table). # Dealing with TimeZones Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/data-modeling/timezones In this guide we'll learn how to import fields that have DateTime strings that contain timezones using StarTree Data Manager. # Importing DateTimes that contain timezones In this guide we'll learn how to import fields that have DateTime strings in various formats, including one with timezones. We'll start with the following file, `events.json`, that contains JSON documents: ```json theme={null} {"ts": "1659949952791", "ts1": "Mon 08 Aug 2022 10:12:32 +0100", "ts2": "2022-08-08 10:12:32", "ts3": "1659949952", "ts4": "2022-08-08T10:12:32.795Z"} {"ts": "1659949952797", "ts1": "Mon 08 Aug 2022 10:12:32 +0100", "ts2": "2022-08-08 10:12:32", "ts3": "1659949952", "ts4": "2022-08-08T10:12:32.802Z"} {"ts": "1659949952805", "ts1": "Mon 08 Aug 2022 10:12:32 +0100", "ts2": "2022-08-08 10:12:32", "ts3": "1659949952", "ts4": "2022-08-08T10:12:32.809Z"} ``` *events.json* This file contains timestamps in different formats: * `ts` - epoch milliseconds. * `ts1` - DateTime string of the format `EEE dd MMM YYYY HH:mm:ss Z`. * `ts2` - DateTime string of the format `YYYY-MM-dd HH:mm:ss`. * `ts3` - epoch seconds. * `ts4` - DateTime string of the ISO DateTime format (`yyyy-MM-dd'T'HH:mm:ss.SSS'Z'`) Upload that file following the instructions in the [upload file](/corecapabilities/ingestdata/upload-file) guide. Once you've selected the file and clicked through to the next screen, you'll see the following: ![Data Modeling](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/timezones-data-modeling.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=3f32a1f7f9ddc15d738490dc6ab5cb90) Let's change the field type for each column to `DATETIME`: ![DateTime Fields](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/timezones-datetime-fields.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=2e1f0323bfecfcba539964a092f744c3) The `ts` column is already in the right format, but we'll need to apply transformation functions to the `ts1`, `ts2`, and `ts3` columns. If you click on the cog icon next to each row, you'll see a modal window like this: ![Adding a transformation function](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/timezones-modal.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=203997270c9afc36a1fa075a7b7cd7d3) *Use 1:MICROSECONDS:EPOCH in case of the data is stored in seconds instead of millis* We'll need to change the column name (you can't use the same field name when applying a transformation function) and paste the transformation function into `Mapping Function` based on the table below: | Current column name | New column name | Transformation function | | - | - | - | | ts1 | timestamp1 | 
  `  FromDateTime(ts1, 'EEE dd MMM YYYY HH:mm:ss Z')`
| | ts2 | timestamp2 | 
  `  FromDateTime(ts2, 'YYYY-MM-dd HH:mm:ss')`
| | ts3 | timestamp3 | 
  `  "ts3" * 1000`
| *Transformation functions to apply* After you've updated each column, click **Apply** to apply your changes. For the field `ts4` we need to change Date Time Format to `ISO DATETIME`. Making this change will change the field type to `STRING`. ### Using native DateTime formats in Pinot The above section is helpful when you want to transform incoming time formats into Milliseconds Epoch which is generally more performant. Alternatively, you could also use Pinot's built in DateTime format support as follows: | Current column name | DateTime Format | | - | - | | ts2 | 
  `  SIMPLE_DATE_FORMAT\|yyyy-MM-dd HH:mm:ss`
| | ts3 | 
  `  EPOCH\|SECONDS\|1`
| | ts4 | 
  `  SIMPLE_DATE_FORMAT\|yyyy-MM-dd'T'HH:mm:ss.SSSZ`
| **`Note: ts1 - DateTime string of the format EEE dd MMM YYYY HH:mm:ss Z is not natively supported in Pinot. This is because there is no way to sort this string on lexicographical and datetime order`** Here's how the corresponding Pinot schema would look like: 

Here are some other commonly used examples:

| Sample | DateTime Format |
| - | - |
| 2025-04-01T14:31:16.551-05:00 | 
  `  SIMPLE_DATE_FORMAT\|yyyy-MM-dd'T'HH:mm:ss.SSSZZ`    
 |

For a full list of supported DateTime formats in Pinot please refer to [this link](https://docs.pinot.apache.org/configuration-reference/schema#new-datetime-formats).

Now click on **Pinot Data** and you should see the following:

![Pinot Data](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/pinot-data.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=3f7f0269b639f19518ce5594c00a090a)

Before you click through to the next screen, make sure that the Pinot Time Column is set to `ts`.

![Pinot Time Column](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/pinot-time-column.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=0bf1a37c149d0ccd63be54e205b8fe6e)

You can now click through the rest of the steps to add this dataset.

If you click on **query console** from the final screen and then select this table, you'll see the results of a query that gets the first 10 rows:

![Timestamps in Pinot](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/timestamps-pinot-ui.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=bcbdef2f35c6751a3f4f5d4d38490c7d)


# JSON Unnesting
Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/data-modeling/unnesting-json

JSON unnesting makes it easy to work with deeply nested and semi-structured data. Extract nested fields from JSON objects and arrays into flat, queryable formats during ingestion.

### Why JSON Unnesting?

* **Simplifies Queries**: Flatten nested data for easier filtering, aggregations, and joins.
* **Improves Performance**: Avoids expensive JSON path evaluations at query time.
* **Streamlines Data Modeling**: Convert complex structures into relational form at ingestion.

Whether you’re dealing with clickstream data, telemetry logs, or API responses, unnesting simplifies data modeling and enhances query performance.

### Enabling JSON Unnesting in StarTree

JSON unnesting can be configured via:

* The **StarTree Data Portal UI**, or
* The **ingestion config JSON** using `transformConfigs` and `unnestConfig`.

## Examples

Let's go through a few examples to unnest a JSON Object,

### Example 1: Unnesting a JSON Object

**Input JSON:**

```json theme={null}
{
  "user": {
    "id": 123,
    "name": "Alice"
  },
  "event": "click"
}
```

**Unnesting Config:**

```json theme={null}
"ingestionConfig": {
    "transformConfigs": [
      {
        "columnName": "user_id",
        "transformFunction": "JSONPATHLONG(user, '$.id')"
      },
      {
        "columnName": "user_name",
        "transformFunction": "JSONPATHSTRING(user, '$.name')"
      }
    ]
}
```

**Resulting Schema:**

| user\_id | user\_name | event |
| - | - | - |
| 123 | Alice | click |

### Example 2: Unnesting an Array into Multiple Rows

**Input JSON:**

```json theme={null}
{
  "orderId": "A001",
  "items": [
    { "sku": "X1", "qty": 2 },
    { "sku": "X2", "qty": 1 }
  ]
}
```

**Unnesting Config:**

```json theme={null}
{
 "ingestionConfig": {
   "complexTypeConfig": {
     "delimiter": "_",
      "fieldsToUnnest": ["items"],
      "collectionNotUnnestedToJson": "NON_PRIMITIVE"
   }
 }
}
```

**Resulting Schema:**

| orderId | items\_sku | items\_qty |
| - | - | - |
| A001 | X1 | 2 |
| A001 | X2 | 1 |


# Connect to Aiven Kafka
Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/streaming/aiven

Create a connection to ingest data from the managed Kafka service by Aiven.

**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select Kafka as the Data Source.**

**3. Create a New Connection.**

Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**.

Enter a **Source Name** for the new connection and specify the **Broker URL**.

Select the **Security Protocol** to be used for authentication, from the drop-down list.

**4. Configure Connection Parameters**

Log in to the Aiven Console, and then select your Apache Kafka service. In the Overview section, find the Connection information. Add the connection parameters according to your authentication method.

## Connecting to Kafka with SASL\_PLAINTEXT or SASL\_SSL Security Protocol

Use the following connection JSON when Kafka is configured with the SASL\_PLAINTEXT or SASL\_SSL security protocol enabled.

```json theme={null}
{
   "streamType": "kafka",
   "stream.kafka.broker.list": "",
   "security.protocol": "",
   "sasl.mechanism": "",
   "stream.kafka.password": "",
   "stream.kafka.username": "",
   "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required \n username=\"\" \n password=\"\";"
}
```

### Property Descriptions

| Property Name | Required | Description |
| - | - | - |
| stream.kafka.broker.list | Yes | List of Kafka brokers to connect to, typically in the format host1:port1,host2:port2. |
| security.protocol | Yes | The security protocol used to connect to Kafka (for example, SASL\_SSL, PLAINTEXT). This parameter is mandatory when the Kafka cluster uses SSL or SASL. |
| sasl.mechanism | Yes | The SASL mechanism for authentication (for example, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512). This parameter is required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT. |
| stream.kafka.username | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT | The user name for authenticating with Kafka. |
| stream.kafka.password | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT | The password for authenticating with Kafka. |
| sasl.jaas.config | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT | The JAAS configuration string for SASL authentication.  When the PLAIN SASL Mechanism is used: "org.apache.kafka.common.security.plain.PlainLoginModule required \n username="" \n password="";" When the SCRAM SASL Mechanism is used: "org.apache.kafka.common.security.scram.ScramLoginModule required \n username="USERNAME" \n password="PASSWORD";" |

## Connecting to Kafka with SSL Security Protocol

See [prerequisites for SSL](/corecapabilities/ingestdata/dataportal/streaming/kafka-ssl-prerequisites) for steps to obtain the bootstrap server endpoint, the server certificate, the client certificate, and the client key, and to update an SSL certificate.

### One-way SSL (Server Authentication)

In one-way SSL, only the server presents its SSL certificate to the client to prove its identity. The client verifies the server's certificate using a trusted Certificate Authority (CA) before establishing an encrypted connection.
Use the following connection JSON when Kafka is configured with the SSL security protocol enabled:

```json theme={null}
{
  "stream.kafka.broker.list": "",
  "security.protocol": "SSL",

  "ssl.truststore.location": "",
  "ssl.truststore.password": "",

  "stream.kafka.ssl.server.certificate": ""
}
```

### Two-way SSL (Mutual Authentication)

In two-way SSL, both the client and the server present certificates to authenticate each other. The server verifies the client's certificate, and the client verifies the server's certificate. This setup provides a higher level of security and trust, commonly used.
Use the following connection JSON when Kafka is configured with the SSL security protocol enabled:

```json theme={null}
{
  "stream.kafka.broker.list": "",
  "security.protocol": "SSL",

  "ssl.truststore.location": "",
  "ssl.truststore.password": "",

  "ssl.keystore.location":"",
  "ssl.keystore.password":"",
  "ssl.key.password":"",

  "stream.kafka.ssl.server.certificate": "",
  "stream.kafka.ssl.client.certificate": "",
  "stream.kafka.ssl.client.key": ""
}
```

> [!NOTE]
>
> The values for the ssl properties (server/client certificate, key) need to be base64 encoded.

#### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.broker.list` | Yes | List of Kafka brokers to connect to, typically in the format `host1:port1,host2:port2`. |
| `security.protocol` | Yes | Defines the security protocol used to connect to Kafka (e.g., `SASL_SSL`, `SSL`, `PLAINTEXT`). This parameter is mandatory when the Kafka cluster uses SSL or SASL. |
| `ssl.truststore.location` | Yes | Absolute path to the truststore file that contains trusted CA certificates (used to verify Kafka broker certificates). |
| `ssl.truststore.password` | Yes | Password to open the truststore file. |
| `stream.kafka.ssl.server.certificate` | Required in one-way SSL | Server Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL`. It needs to be Base64 encoded. |
| `stream.kafka.ssl.client.certificate` | Required in two-way SSL | Client Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL`. It needs to be Base64 encoded. |
| `stream.kafka.ssl.client.key` | Required in two-way SSL | Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL` and `stream.kafka.ssl.client.certificate` is also set. It needs to be Base64 encoded. |
| `ssl.keystore.location` | Required in two-way SSL | Path to the keystore file containing the client's certificate and private key. |
| `ssl.keystore.password` | Required in two-way SSL | Password to open the keystore file. |
| `ssl.key.password` | Required in two-way SSL | Password protecting the private key inside the keystore. |

**5. Test the Connection and Configure Data Ingestion**

## Configure Data Ingestion

After you have configured the connection properties, test the connection to ensure it is working. When the connection is successful, use the following JSON to configure the source's Kafka topic and data format:

```json theme={null}
{
    "stream.kafka.decoder.prop.format": "",
    "stream.kafka.decoder.class.name": "",
    "stream.kafka.topic.name": "",
    "stream.kafka.consumer.type": "lowlevel",
    "stream.kafka.consumer.factory.class.name": "ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory",
    "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
    "stream.kafka.decoder.prop.schema": "",
    "stream.kafka.decoder.prop.schema.registry.rest.url": "",
    "stream.kafka.decoder.prop.schema.registry.schema.name": "",
    "stream.kafka.decoder.prop.schema.registry.key": "",
    "stream.kafka.decoder.prop.schema.registry.secret": ""
}
```

#### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.decoder.prop.format` | Yes | Format of the messages in Kafka. Supported values include `json`, `avro`, `proto` etc. |
| `stream.kafka.decoder.class.name` | Yes | The class name of the decoder used for Kafka message parsing. It is set based on the message format and schema.    **Examples:**    \* For JSON Messages use `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder`     \* For AVRO Messages use `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` \* For PROTO Messages use `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` See [Message Decoders](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/overview) for more information. |
| `stream.kafka.topic.name` | Yes | The name of the Kafka topic from which Pinot will consume data. |
| `stream.kafka.consumer.type` | No | The type of Kafka consumer used in Apache Pinot. Use `lowlevel` for granular control of partitions and offsets. |
| `stream.kafka.consumer.factory.class.name` | Yes | The Kafka consumer factory class to use. The specified class `ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory` is the default Kafka consumer factory when using Kafka 2.0+ or Confluent Kafka clients.      Use `ai.startree.pinot.plugin.stream.kafka10.ConfluentKafkaConsumerFactory` for legacy Kafka (1.x) environments with Confluent client integration. |
| `stream.kafka.consumer.prop.auto.offset.reset` | Yes | The behavior when no committed offset exists or offsets are invalid (`smallest`, `latest`, `none`). Use `smallest` for backfilling or consuming all historical data. Use `latest` for real-time streaming. |
| `stream.kafka.decoder.prop.schema` | No | The schema definition that Pinot can use to deserialize and interpret Kafka messages during ingestion. |
| `stream.kafka.decoder.prop.schema.registry.rest.url` | No | The URL endpoint of the Schema Registry service. This is required if the data type of message is either AVRO or PROTO |
| `stream.kafka.decoder.prop.schema.registry.schema.name` | No | The name of the schema to use for decoding Kafka messages. |
| `stream.kafka.decoder.prop.schema.registry.key` | No | The API key or identifier for authenticating with the Schema Registry. |
| `stream.kafka.decoder.prop.schema.registry.secret` | No | The secret (password) paired with the API key for Schema Registry authentication. |

### Additional Configuration

The parameters mentioned above are the **minimum-required parameters** to establish the connection. Apache Pinot also provides additional configuration options to fine-tune and control the ingestion process. For a complete list of configuration parameters, refer to the [Apache Pinot documentation](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#consume-transactionally-committed-messages). In addition, [Kafka Consumer properties](https://kafka.apache.org/documentation/#consumerconfigs) can also be used.

## Configure Record Reader

> **AVRO**
>
> One configuration option `AvroRecordReaderConfig` is supported.
>
> * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS.
>
> ```bash theme={null}
> {
>   "enableLogicalTypes": "true"
> }
> ```
>
> For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start.

> **PROTO**
>
> `ProtoBufRecordReaderConfig` exists in Pinot and the following configuration is possible.
>
> ```bash theme={null}
> {
>   "descriptorFile": "/path/to/descriptor/file"
> }
> ```

**6. Sample Data**

Click **Show Sample Data** to see a preview of the source data.

> **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)**
>
> Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview).


# Connect to MSK - Managed Apache Kafka by AWS
Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/streaming/aws_msk



**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select Custom as the Data Source.**

Native support for AWS MSK is coming soon.

**3. Create a New Connection.**

Select "Streaming" as the Data Source Category.

## Connect to AWS MSK Using IAM-Based SASL Authentication

Use the following JSON configuration when MSK is set up with IAM-based SASL authentication (AWS\_MSK\_IAM) and TLS (SASL\_SSL).

```json theme={null}
{
  "authentication.type": "SASL",
  "security.protocol": "SASL_SSL",
  "sasl.mechanism": "AWS_MSK_IAM",
  "sasl.jaas.config": "org.apache.pinot.shaded.software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn=\"arn:aws:iam:::role/\";",
  "sasl.client.callback.handler.class": "org.apache.pinot.shaded.software.amazon.msk.auth.iam.IAMClientCallbackHandler",
  "stream.kafka.broker.list": "",
  "stream.kafka.topic.name": "",
  "stream.kafka.consumer.type": "lowlevel",
  "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
  "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
  "stream.kafka.decoder.prop.format": "JSON",
  "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
  "realtime.segment.flush.threshold.rows": "0",
  "realtime.segment.flush.threshold.segment.size": "200M",
  "realtime.segment.flush.threshold.time": "24h"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `authentication.type` | Yes | Authentication mode for Kafka/MSK. Set to `SASL` to enable SASL-based authentication. |
| `security.protocol` | Yes | Kafka security protocol. Use `SASL_SSL` to enable TLS + SASL for MSK IAM authentication. |
| `sasl.mechanism` | Yes | SASL mechanism. For AWS MSK IAM-based auth, set to `AWS_MSK_IAM`. |
| `sasl.jaas.config` | Yes | JAAS configuration string for the IAM login module. Must use `org.apache.pinot.shaded.software.amazon.msk.auth.iam.IAMLoginModule` and specify `awsRoleArn` (or equivalent credentials strategy). |
| `sasl.client.callback.handler.class` | Yes | Callback handler implementation for MSK IAM authentication. Use `org.apache.pinot.shaded.software.amazon.msk.auth.iam.IAMClientCallbackHandler`. |
| `stream.kafka.broker.list` | Yes | Comma-separated list of MSK bootstrap brokers with ports. Example: `b-1.example.amazonaws.com:9098,b-2.example.amazonaws.com:9098`. Should match the Broker URL / bootstrap servers configured in the UI. |
| `stream.kafka.topic.name` | Yes | Name of the MSK topic to consume from. Example: `sample_topic`. |
| `stream.kafka.consumer.type` | No | Kafka consumer mode. `lowlevel` uses Pinot's low level consumer for fine-grained partition/offset control. |
| `stream.kafka.consumer.factory.class.name` | Yes | Kafka consumer factory implementation. For Kafka 2.0+ clusters (including MSK), use `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory`. |
| `stream.kafka.decoder.class.name` | Yes | The class name of the decoder used for Kafka message parsing. It is set based on the message format and schema.    **Examples:** \* For JSON Messages use `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder`     \* For AVRO Messages use `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` \* For PROTO Messages use `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` See [Message Decoders](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/overview) for more information. |
| `stream.kafka.decoder.prop.format` | Yes | Format of the messages in Kafka. Supported values include `json`, `avro`, `proto` etc. |
| `stream.kafka.consumer.prop.auto.offset.reset` | No | Defines starting offset if there is no committed one. `smallest` (Kafka 0.8-style) = earliest offset. |
| `realtime.segment.flush.threshold.rows` | Yes\* | Row-based flush threshold. `0` disables row-based flushing and makes size/time the only triggers. |
| `realtime.segment.flush.threshold.segment.size` | Yes | Target segment size before flush (e.g., `200M`). Controls memory usage and segment count. |
| `realtime.segment.flush.threshold.time` | Yes | Maximum time before a segment flush, even if row/size thresholds aren't met. Example: `24h`. |

**4. Preview the Sample Data**

Click **Show Sample Data** to see a preview of the source data.

> **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)**
>
> Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview).


# Connect to Confluent Cloud
Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/streaming/confluent

Create a connection to ingest from fully managed Kafka in Confluent Cloud.

**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select Confluent Cloud as the Data Source.**

**3. Create a New Connection.**

Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**.

Enter a **Source Name** for the new connection and specify the **Broker URL**.

**4. Configure Connection Parameters**

## Confluent API Key Authentication

Use the following JSON configuration:

```json theme={null}
{
  "stream.kafka.broker.list": "",
  "security.protocol": "SASL_SSL",
  "sasl.mechanism": "PLAIN",
  "stream.confluent.key": "",
  "stream.confluent.secret": "",
  "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required \n username=\"\" \n password=\"\";"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.broker.list` | Yes | The list of Kafka brokers to connect to, typically in the format `host1:port1,host2:port2`. Required to establish a connection to Confluent Kafka. |
| `security.protocol` | Yes | The security protocol used to connect to Kafka (e.g., `SASL_SSL`). Mandatory when the Confluent Kafka cluster uses SASL. |
| `sasl.mechanism` | Yes | The SASL mechanism used for authentication. This value should be `PLAIN`. |
| `stream.confluent.key` | Yes | The Confluent key for authentication. |
| `stream.confluent.secret` | Yes | The Confluent secret for authentication. |
| `sasl.jaas.config` | Yes | The JAAS configuration string for SASL authentication. |

**5. Test the Connection and Configure Data Ingestion.**

After you have configured the connection properties, **test the connection** to ensure it functions properly.

## Configure Data Ingestion

Once the connection is successful, configure the additional data settings using the following JSON format:

```json theme={null}
{
    "stream.kafka.topic.name": "",
    "stream.kafka.decoder.prop.format": "",
    "stream.kafka.decoder.class.name": "",
    "stream.kafka.consumer.type": "lowlevel",
    "stream.kafka.consumer.factory.class.name": "ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory",
    "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
    "stream.kafka.decoder.prop.schema": "",
    "stream.kafka.decoder.prop.schema.registry.rest.url": "",
    "stream.kafka.decoder.prop.schema.registry.schema.name": "",
    "stream.kafka.decoder.prop.schema.registry.key": "",
    "stream.confluent.schema.registry.secret": ""
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.topic.name` | Yes | The name of the Kafka topic from which Pinot will consume data. |
| `stream.kafka.decoder.prop.format` | Yes | The format of the input data. Supported values include **json, avro, proto**, etc. |
| `stream.kafka.decoder.class.name` | Yes | The class name of the decoder used for Kafka message parsing. Set based on the message format and schema. **Examples:** \* For JSON Messages: `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder` \* For AVRO Messages: `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` \* For PROTO Messages: `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` See [Message Decoders](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/overview) for more information. |
| `stream.kafka.consumer.type` | No | The type of Kafka consumer used in Apache Pinot. Use `lowlevel` for granular control of partitions and offsets. |
| `stream.kafka.consumer.factory.class.name` | Yes | The Kafka consumer factory class to use. The default class is `ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory` for Kafka 2.0+ or Confluent Kafka clients.     Use `ai.startree.pinot.plugin.stream.kafka10.ConfluentKafkaConsumerFactory` for legacy Kafka (1.x) environments with Confluent client integration. |
| `stream.kafka.consumer.prop.auto.offset.reset` | Yes | Defines behavior when no committed offset exists or offsets are invalid.    **Options:** `smallest` (for backfilling or consuming all historical data), `latest` (for real-time streaming), or `none`. |
| `stream.kafka.decoder.prop.schema` | No | The schema definition that Pinot can use to deserialize and interpret Kafka messages during ingestion. |
| `stream.kafka.decoder.prop.schema.registry.rest.url` | No | The URL endpoint of the Schema Registry service. This is required if the data type of message is either AVRO or PROTO |
| `stream.kafka.decoder.prop.schema.registry.schema.name` | No | The name of the schema to use for decoding Kafka messages. |
| `stream.kafka.decoder.prop.schema.registry.key` | No | The API key or identifier for authenticating with the Schema Registry. |
| `stream.confluent.schema.registry.secret` | No | The secret (password) paired with the API key for Schema Registry authentication. |

**6. Preview the Sample Data**

Click Show Sample Data to preview the source data before finalizing the configuration.

> **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)**
>
> Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview).


# Connect to Apache Kafka
Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/streaming/kafka

Create a connection to stream real-time events using Apache Kafka.

**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select Kafka as the Data Source.**

**3. Create a New Connection.**

Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**.

Enter a **Source Name** for the new connection and specify the **Broker URL**.

Select the **Security Protocol** to be used for authentication, from the drop-down list.

**4. Configure Connection Parameters**

Add the connection parameters according to your authentication method.

## Connecting to Kafka with No Authentication

Use the following connection JSON when connecting to Kafka without authentication.

```json theme={null}
{
   "stream.kafka.broker.list": "",
   "security.protocol": "PLAIN_TEXT"
}
```

### Property Descriptions

| Property Name | Required | Description |
| - | - | - |
| stream.kafka.broker.list | Yes | List of Kafka brokers to connect to, typically in the format host1:port1,host2:port2. |
| security.protocol | Yes | Defines the security protocol used to connect to Kafka (e.g., PLAIN\_TEXT). |

## Connecting to Kafka with SASL\_PLAINTEXT or SASL\_SSL Security Protocol

Use the following connection JSON when Kafka is configured with the SASL\_PLAINTEXT or SASL\_SSL security protocol enabled.

```json theme={null}
{
   "streamType": "kafka",
   "stream.kafka.broker.list": "",
   "security.protocol": "",
   "sasl.mechanism": "",
   "stream.kafka.password": "",
   "stream.kafka.username": "",
   "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required \n username=\"\" \n password=\"\";"
}
```

### Property Descriptions

| Property Name | Required | Description |
| - | - | - |
| stream.kafka.broker.list | Yes | List of Kafka brokers to connect to, typically in the format host1:port1,host2:port2. |
| security.protocol | Yes | The security protocol used to connect to Kafka (for example, SASL\_SSL, PLAINTEXT). This parameter is mandatory when the Kafka cluster uses SASL. |
| sasl.mechanism | Yes | The SASL mechanism for authentication (for example, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512). This parameter is required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT. |
| stream.kafka.username | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT | The user name for authenticating with Kafka. |
| stream.kafka.password | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT | The password for authenticating with Kafka. |
| sasl.jaas.config | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT | The JAAS configuration string for SASL authentication.  When the PLAIN SASL Mechanism is used: "org.apache.kafka.common.security.plain.PlainLoginModule required \n username="" \n password="";"  When the SCRAM SASL Mechanism is used: "org.apache.kafka.common.security.scram.ScramLoginModule required \n username="USERNAME" \n password="PASSWORD";" |

## Connecting to Kafka with SSL Security Protocol

See [prerequisites for SSL](/corecapabilities/ingestdata/dataportal/streaming/kafka-ssl-prerequisites) for steps to obtain the bootstrap server endpoint, the server certificate, the client certificate, and the client key, and to update an SSL certificate.

### One-way SSL (Server Authentication)

In one-way SSL, only the server presents its SSL certificate to the client to prove its identity. The client verifies the server's certificate using a trusted Certificate Authority (CA) before establishing an encrypted connection.
Use the following connection JSON when Kafka is configured with the SSL security protocol enabled:

```json theme={null}
{
  "stream.kafka.broker.list": "",
  "security.protocol": "SSL",

  "ssl.truststore.location": "",
  "ssl.truststore.password": "",

  "stream.kafka.ssl.server.certificate": ""
}
```

### Two-way SSL (Mutual Authentication)

In two-way SSL, both the client and the server present certificates to authenticate each other. The server verifies the client's certificate, and the client verifies the server's certificate. This setup provides a higher level of security and trust, commonly used.
Use the following connection JSON when Kafka is configured with the SSL security protocol enabled:

```json theme={null}
{
  "stream.kafka.broker.list": "",
  "security.protocol": "SSL",

  "ssl.truststore.location": "",
  "ssl.truststore.password": "",

  "ssl.keystore.location":"",
  "ssl.keystore.password":"",
  "ssl.key.password":"",

  "stream.kafka.ssl.server.certificate": "",
  "stream.kafka.ssl.client.certificate": "",
  "stream.kafka.ssl.client.key": ""
}
```

> [!NOTE]
>
> The values for the ssl properties (server/client certificate, key) need to be base64 encoded.

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.broker.list` | Yes | List of Kafka brokers to connect to, typically in the format `host1:port1,host2:port2`. |
| `security.protocol` | Yes | Defines the security protocol used to connect to Kafka (e.g., `SASL_SSL`, `SSL`, `PLAINTEXT`). This parameter is mandatory when the Kafka cluster uses SSL or SASL. |
| `ssl.truststore.location` | Yes | Absolute path to the truststore file that contains trusted CA certificates (used to verify Kafka broker certificates). |
| `ssl.truststore.password` | Yes | Password to open the truststore file. |
| `stream.kafka.ssl.server.certificate` | Required in one-way SSL | Server Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL`. It needs to be Base64 encoded. |
| `stream.kafka.ssl.client.certificate` | Required in two-way SSL | Client Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL`. It needs to be Base64 encoded. |
| `stream.kafka.ssl.client.key` | Required in two-way SSL | Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL` and `stream.kafka.ssl.client.certificate` is also set. It needs to be Base64 encoded. |
| `ssl.keystore.location` | Required in two-way SSL | Path to the keystore file containing the client's certificate and private key. |
| `ssl.keystore.password` | Required in two-way SSL | Password to open the keystore file. |
| `ssl.key.password` | Required in two-way SSL | Password protecting the private key inside the keystore. |

**5. Test the Connection and Configure Data Ingestion**

After you have configured the connection properties, test the connection to ensure it is working.

## Configure Data Ingestion

When the connection is successful, use the following JSON to configure the source's Kafka topic and data format:

```json theme={null}
{
  "stream.kafka.decoder.prop.format": "",
  "stream.kafka.decoder.class.name": "",
  "stream.kafka.topic.name": "",
  "stream.kafka.consumer.type": "lowlevel",
  "stream.kafka.consumer.factory.class.name": "ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory",
  "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.decoder.prop.format` | Yes | Format of the messages in Kafka. Supported values include `json`, `avro`, `proto` etc. |
| `stream.kafka.decoder.class.name` | Yes | The class name of the decoder used for Kafka message parsing. It is set based on the message format and schema.    **Examples:** \* For JSON Messages use `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder`     \* For AVRO Messages use `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` \* For PROTO Messages use `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` See [Message Decoders](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/overview) for more information. |
| `stream.kafka.topic.name` | Yes | The name of the Kafka topic from which Pinot will consume data. |
| `stream.kafka.consumer.type` | No | The type of Kafka consumer used in Apache Pinot. Use `lowlevel` for granular control of partitions and offsets. |
| `stream.kafka.consumer.factory.class.name` | Yes | The Kafka consumer factory class to use. The specified class `ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory` is the default Kafka consumer factory when using Kafka 2.0+ or Confluent Kafka clients.      Use `ai.startree.pinot.plugin.stream.kafka10.ConfluentKafkaConsumerFactory` for legacy Kafka (1.x) environments with Confluent client integration. |
| `stream.kafka.consumer.prop.auto.offset.reset` | Yes | The behavior when no committed offset exists or offsets are invalid (`smallest`, `latest`, `none`). Use `smallest` for backfilling or consuming all historical data. Use `latest` for real-time streaming. |

### Additional Configuration

The parameters mentioned above are the **minimum-required parameters** to establish the connection. Apache Pinot also provides additional configuration options to fine-tune and control the ingestion process. For a complete list of configuration parameters, refer to the [Apache Pinot documentation](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#consume-transactionally-committed-messages). In addition, [Kafka Consumer properties](https://kafka.apache.org/documentation/#consumerconfigs) can also be used.

## Configure Record Reader

> **AVRO**
>
> One configuration option `AvroRecordReaderConfig` is supported.
>
> * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS.
>
> ```bash theme={null}
> {
>   "enableLogicalTypes": "true"
> }
> ```
>
> For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start.

> **PROTO**
>
> `ProtoBufRecordReaderConfig` exists in Pinot and the following configuration is possible.
>
> ```bash theme={null}
> {
>   "descriptorFile": "/path/to/descriptor/file"
> }
> ```

**6. Preview the Sample Data**

Click **Show Sample Data** to see a preview of the source data.

> **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)**
>
> Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview).


# Connect to Amazon Kinesis
Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/streaming/kinesis

Create a connection to ingest real-time data from Amazon Kinesis.

**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select Kinesis as the Data Source.**

**3. Create a New Connection.**

Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**.

Enter a **Source Name** for the new connection and specify the **Broker URL**.

**4. Configure Connection Parameters**

## Connect Kinesis Using Basic Authentication

Use the following JSON configuration when Kinesis is set up with **basic authentication** using an **access key** and **secret key**.

```json theme={null}
{
  "region": "",
  "accessKey": "",
  "secretKey": ""
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `region` | Yes | The AWS region (e.g., `us-east-1`) where the Kinesis stream is located. |
| `accessKey` | Yes | The AWS access key used for authentication. |
| `secretKey` | Yes | The AWS secret key used for authentication (paired with the access key). |

## Connecting to Kinesis Using IAM-Based Authentication

Please follow [these steps](/corecapabilities/ingestdata/recipes/iam-role-kinesis) to create IAM role for Kinesis setup.

Use the following JSON configuration when Kinesis is set up with IAM-based authentication by assuming an IAM Role for secure access.

```json theme={null}
{
  "region": "",
  "iamRoleBasedAccessEnabled" : "true",
  "roleArn": "arn:aws:iam:::role/",
  "roleSessionName": "pinot_session"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `region` | Yes | The AWS region (e.g., `us-east-1`) where the Kinesis stream is located. |
| `iamRoleBasedAccessEnabled` | Yes | Mark it as true, when IAM role based authentication is used. |
| `roleArn` | Yes | ARN of the IAM role Pinot will assume to access Kinesis |
| `roleSessionName` | Yes | An identifier used when assuming an IAM role via AWS Security Token Service to access the Kinesis stream securely. It can help track usage in AWS CloudTrail logs, metrics, etc. |

**5. Test the Connection and Configure Data Ingestion**

## Configure Data Ingestion

After configuring the connection properties, test the connection to ensure it functions properly. Once the connection is successful, configure the additional data settings using the following JSON format:

```json theme={null}
{
  "shardIteratorType": "TRIM_HORIZON",
  "stream.kinesis.topic.name": "",
  "stream.kinesis.decoder.prop.format": "Message-Format-Type",
  "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
  "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `shardIteratorType` | No | The shardIteratorType defines where consumption starts. Use LATEST to read only new records, or TRIM\_HORIZON to start from the earliest. Default is TRIM\_HORIZON. |
| `stream.kinesis.topic.name` | Yes | The name of the Kinesis topic from which Pinot will consume data. |
| `stream.kinesis.decoder.prop.format` | Yes | The format of the Kinesis messages. Supported values include **csv, json, avro, protobuf, parquet**, etc. |
| `stream.kinesis.decoder.class.name` | Yes | The class name of the decoder used for Kinesis message parsing. It is set based on the message format and schema.     **Examples:** \* For JSON Messages: `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder`     \* For AVRO Messages: `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` \* For PROTO Messages: `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` See [Message Decoders](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/overview) for more information. |
| `stream.kinesis.consumer.factory.class.name` | Yes | The Kinesis consumer factory class to use. The default class is `org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory` for AWS Kinesis. |

## Configure Record Reader

> **AVRO**
>
> One configuration option `AvroRecordReaderConfig` is supported.
>
> * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS.
>
> ```bash theme={null}
> {
>   "enableLogicalTypes": "true"
> }
> ```
>
> For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start.

**6. Preview the Data**

Click Show Sample Data to preview the source data before finalizing the configuration.

> **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)**
>
> Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview).

## FAQs

## Why should I use **Provisioned** mode instead of **On-Demand** for Kinesis when integrating with StarTree?

We recommend configuring Kinesis in **Provisioned** mode, rather than **On-Demand** mode, when using it as a source for StarTree ingestion pipelines. Here's why:

## On-Demand Mode

* **Automatic Scaling**: Manages shards dynamically to meet demand—no manual provisioning needed.
* **Pay-as-you-go**: Charges are based on actual throughput (reads and writes).
* **Best for**: Applications with unpredictable or spiky traffic patterns.
* **Auto-Scaling Limit**: Can scale up to **2×** the peak write throughput observed over the past 30 days.

> While On-Demand mode is flexible, it can make capacity planning opaque and may lead to surprises in performance or billing when dealing with sustained, high-throughput ingestion.

***

## Provisioned Mode (Recommended)

* **Manual Shard Management**: You define the number of shards explicitly.
* **Fixed Capacity and Cost**: Billed based on shard count, regardless of actual usage.
* **Best for**: Workloads with predictable, steady traffic where throughput can be estimated.
* **Manual Scaling**: You are responsible for increasing/decreasing shard count as needed.

> Provisioned mode offers **greater predictability, control, and stability**, which is important when tuning for ingestion throughput, latency, and cost efficiency in a real-time analytics platform like StarTree.


# Connect to Redpanda
Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/streaming/redpanda

Create a connection for high-performance streaming ingestion with Redpanda.

**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select Redpanda as the Data Source.**

**3. Create a New Connection.**

Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**.

Enter a **Source Name** for the new connection and specify the **Broker URL**.

Select the **Security Protocol** to be used for authentication, from the drop-down list.

**4. Configure Connection Parameters**

Add the connection parameters according to your authentication method.

## Connecting to Redpanda with No Authentication

Use the following connection JSON when connecting to Redpanda without authentication.

```json theme={null}
{
   "stream.kafka.broker.list": "",
   "security.protocol": "PLAIN_TEXT"
}
```

### Property Descriptions

| Property Name | Required | Description |
| :- | :- | :- |
| stream.kafka.broker.list | Yes | List of Kafka brokers to connect to, typically in the format host1:port1,host2:port2. |
| security.protocol | Yes | Defines the security protocol used to connect to Kafka (e.g., PLAIN\_TEXT). |

## Connecting to Redpanda with SASL\_PLAINTEXT or SASL\_SSL Security Protocol

Use the following connection JSON when Redpanda is configured with the SASL\_PLAINTEXT or SASL\_SSL security protocol enabled.

```json theme={null}
{
   "streamType": "kafka",
   "stream.kafka.broker.list": "",
   "security.protocol": "",
   "sasl.mechanism": "",
   "stream.kafka.password": "",
   "stream.kafka.username": "",
   "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required \n username=\"\" \n password=\"\";"
}
```

### Property Descriptions

| Property Name | Required | Description |
| :- | :- | :- |
| stream.kafka.broker.list | Yes | List of Kafka brokers to connect to, typically in the format host1:port1,host2:port2. |
| security.protocol | Yes | The security protocol used to connect to Kafka (for example, SASL\_SSL, PLAINTEXT). This parameter is mandatory when the Kafka cluster uses SSL or SASL. |
| sasl.mechanism | Yes | The SASL mechanism for authentication (for example, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512). This parameter is required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT. |
| stream.kafka.username | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT. | The user name for authenticating with Kafka. |
| stream.kafka.password | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT. | The password for authenticating with Kafka. |
| sasl.jaas.config | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT. | The JAAS configuration string for SASL authentication. The JAAS configuration string for SASL authentication.  When the PLAIN SASL Mechanism is used: "org.apache.kafka.common.security.plain.PlainLoginModule required \n username="" \n password="";"  When the SCRAM SASL Mechanism is used: "org.apache.kafka.common.security.scram.ScramLoginModule required \n username="USERNAME" \n password="PASSWORD";" |

## Connecting to Kafka with SSL Security Protocol

See [prerequisites for SSL](/corecapabilities/ingestdata/dataportal/streaming/kafka-ssl-prerequisites) for steps to obtain the bootstrap server endpoint, the server certificate, the client certificate, and the client key, and to update an SSL certificate.

### One-way SSL (Server Authentication)

In one-way SSL, only the server presents its SSL certificate to the client to prove its identity. The client verifies the server's certificate using a trusted Certificate Authority (CA) before establishing an encrypted connection.
Use the following connection JSON when Kafka is configured with the SSL security protocol enabled:

```json theme={null}
{
  "stream.kafka.broker.list": "",
  "security.protocol": "SSL",

  "ssl.truststore.location": "",
  "ssl.truststore.password": "",

  "stream.kafka.ssl.server.certificate": ""
}
```

### Two-way SSL (Mutual Authentication)

In two-way SSL, both the client and the server present certificates to authenticate each other. The server verifies the client's certificate, and the client verifies the server's certificate. This setup provides a higher level of security and trust, commonly used.
Use the following connection JSON when Kafka is configured with the SSL security protocol enabled:

```json theme={null}
{
  "stream.kafka.broker.list": "",
  "security.protocol": "SSL",

  "ssl.truststore.location": "",
  "ssl.truststore.password": "",

  "ssl.keystore.location":"",
  "ssl.keystore.password":"",
  "ssl.key.password":"",

  "stream.kafka.ssl.server.certificate": "",
  "stream.kafka.ssl.client.certificate": "",
  "stream.kafka.ssl.client.key": ""
}
```

> [!NOTE]
>
> The values for the ssl properties (server/client certificate, key) need to be base64 encoded.

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.broker.list` | Yes | List of Kafka brokers to connect to, typically in the format `host1:port1,host2:port2`. |
| `security.protocol` | Yes | Defines the security protocol used to connect to Kafka (e.g., `SASL_SSL`, `SSL`, `PLAINTEXT`). This parameter is mandatory when the Kafka cluster uses SSL or SASL. |
| `ssl.truststore.location` | Yes | Absolute path to the truststore file that contains trusted CA certificates (used to verify Kafka broker certificates). |
| `ssl.truststore.password` | Yes | Password to open the truststore file. |
| `stream.kafka.ssl.server.certificate` | Required in one-way SSL | Server Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL`. It needs to be Base64 encoded. |
| `stream.kafka.ssl.client.certificate` | Required in two-way SSL | Client Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL`. It needs to be Base64 encoded. |
| `stream.kafka.ssl.client.key` | Required in two-way SSL | Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL` and `stream.kafka.ssl.client.certificate` is also set. It needs to be Base64 encoded. |
| `ssl.keystore.location` | Required in two-way SSL | Path to the keystore file containing the client's certificate and private key. |
| `ssl.keystore.password` | Required in two-way SSL | Password to open the keystore file. |
| `ssl.key.password` | Required in two-way SSL | Password protecting the private key inside the keystore. |

**5. Test the Connection and Configure Data Ingestion**

After you have configured the connection properties, test the connection to ensure it is working.

## Configure Data Ingestion

When the connection is successful, use the following JSON to configure the source's Kafka topic and data format:

```json theme={null}
{
    "stream.kafka.decoder.prop.format": "",
    "stream.kafka.decoder.class.name": "",
    "stream.kafka.topic.name": "",
    "stream.kafka.consumer.type": "lowlevel",
    "stream.kafka.consumer.factory.class.name": "ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory",
    "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
    "stream.kafka.decoder.prop.schema": "",
    "stream.kafka.decoder.prop.schema.registry.rest.url": "",
    "stream.kafka.decoder.prop.schema.registry.schema.name": "",
    "stream.kafka.decoder.prop.schema.registry.key": "",
    "stream.kafka.decoder.prop.schema.registry.secret": ""
}
```

### Property Descriptions

| Property | Required | Description |
| :- | :- | :- |
| `stream.kafka.decoder.prop.format` | Yes | Format of the messages in Kafka. Supported values include `json`, `avro`, `proto` etc. |
| `stream.kafka.decoder.class.name` | Yes | The class name of the decoder used for Kafka message parsing. It is set based on the message format and schema.    **Examples:** \* For JSON Messages use `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder`     \* For AVRO Messages use `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` \* For PROTO Messages use `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` See [Message Decoders](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/overview) for more information. |
| `stream.kafka.topic.name` | Yes | The name of the Kafka topic from which Pinot will consume data. |
| `stream.kafka.consumer.type` | No | The type of Kafka consumer used in Apache Pinot. Use `lowlevel` for granular control of partitions and offsets. |
| `stream.kafka.consumer.factory.class.name` | Yes | The Kafka consumer factory class to use. The specified class `ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory` is the default Kafka consumer factory when using Kafka 2.0+ or Confluent Kafka clients.      Use `ai.startree.pinot.plugin.stream.kafka10.ConfluentKafkaConsumerFactory` for legacy Kafka (1.x) environments with Confluent client integration. |
| `stream.kafka.consumer.prop.auto.offset.reset` | Yes | The behavior when no committed offset exists or offsets are invalid (`smallest`, `latest`, `none`). Use `smallest` for backfilling or consuming all historical data. Use `latest` for real-time streaming. |
| `stream.kafka.decoder.prop.schema` | No | The schema definition that Pinot can use to deserialize and interpret Kafka messages during ingestion. |
| `stream.kafka.decoder.prop.schema.registry.rest.url` | No | The URL endpoint of the Schema Registry service. This is required if the data type of message is either AVRO or PROTO |
| `stream.kafka.decoder.prop.schema.registry.schema.name` | No | The name of the schema to use for decoding Kafka messages. |
| `stream.kafka.decoder.prop.schema.registry.key` | No | The API key or identifier for authenticating with the Schema Registry. |
| `stream.kafka.decoder.prop.schema.registry.secret` | No | The secret (password) paired with the API key for Schema Registry authentication. |

## Configure Record Reader

> **AVRO**
>
> One configuration option `AvroRecordReaderConfig` is supported.
>
> * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS.
>
> ```bash theme={null}
> {
>   "enableLogicalTypes": "true"
> }
> ```
>
> For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start.

> **PROTO**
>
> `ProtoBufRecordReaderConfig` exists in Pinot and the following configuration is possible.
>
> ```bash theme={null}
> {
>   "descriptorFile": "/path/to/descriptor/file"
> }
> ```

### Additional Configuration

The parameters mentioned above are the **minimum-required parameters** to establish the connection. Apache Pinot also provides additional configuration options to fine-tune and control the ingestion process. For a complete list of configuration parameters, refer to the [Apache Pinot documentation](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#consume-transactionally-committed-messages). In addition, [Kafka Consumer properties](https://kafka.apache.org/documentation/#consumerconfigs) can also be used.

**6. Sample Data**

Click **Show Sample Data** to see a preview of the source data.

> **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)**
>
> Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview).


# Connect to StreamNative Kafka
Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/streaming/streamnative

Create a connection to ingest from fully managed Kafka in StreamNative.

**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select Kafka as the Data Source.**

**3. Create a New Connection.**

Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**.

Enter a **Source Name** for the new connection and specify the **Broker URL**.

**4. Configure Connection Parameters.**

## SASL\_SSL with PLAIN Authentication

StreamNative supports `SASL_SSL` as the security protocol with `PLAIN` as the SASL mechanism. Use the following JSON configuration:

```json theme={null}
{
  "stream.kafka.broker.list": ":9093",
  "security.protocol": "SASL_SSL",
  "sasl.mechanism": "PLAIN",
  "stream.kafka.username": "",
  "stream.kafka.password": "",
  "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required \n username=\"\" \n password=\"\";"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.broker.list` | Yes | The list of Kafka brokers to connect to, typically in the format `host1:port1,host2:port2`. Required to establish a connection to the StreamNative cluster. |
| `security.protocol` | Yes | The security protocol used to connect to Kafka. StreamNative only supports `SASL_SSL`. |
| `sasl.mechanism` | Yes | The SASL mechanism used for authentication. StreamNative only supports `PLAIN`. |
| `stream.kafka.username` | Yes | The StreamNative service account token name used for authentication. |
| `stream.kafka.password` | Yes | The StreamNative service account token secret used for authentication. |
| `sasl.jaas.config` | Yes | The JAAS configuration string for SASL authentication. |

**5. Test the Connection and Configure Data Ingestion.**

After you have configured the connection properties, **test the connection** to ensure it functions properly.

## Configure Data Ingestion

Once the connection is successful, configure the additional data settings using the following JSON format:

```json theme={null}
{
  "stream.kafka.topic.name": "",
  "stream.kafka.decoder.prop.format": "",
  "stream.kafka.decoder.class.name": "",
  "stream.kafka.consumer.type": "lowlevel",
  "stream.kafka.consumer.factory.class.name": "ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory",
  "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.topic.name` | Yes | The name of the Kafka topic from which Pinot will consume data. |
| `stream.kafka.decoder.prop.format` | Yes | The format of the input data. Supported values include **json, avro, proto**, etc. |
| `stream.kafka.decoder.class.name` | Yes | The class name of the decoder used for Kafka message parsing. Set based on the message format and schema. **Examples:** \* For JSON Messages: `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder` \* For AVRO Messages: `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` \* For PROTO Messages: `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` See [Message Decoders](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/overview) for more information. |
| `stream.kafka.consumer.type` | No | The type of Kafka consumer used in Apache Pinot. Use `lowlevel` for granular control of partitions and offsets. |
| `stream.kafka.consumer.factory.class.name` | Yes | The Kafka consumer factory class to use. The default class is `ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory` for Kafka 2.0+ or Confluent Kafka clients.  Use `ai.startree.pinot.plugin.stream.kafka10.ConfluentKafkaConsumerFactory` for legacy Kafka (1.x) environments with Confluent client integration. |
| `stream.kafka.consumer.prop.auto.offset.reset` | Yes | Defines behavior when no committed offset exists or offsets are invalid. **Options:** `smallest` (for backfilling or consuming all historical data), `latest` (for real-time streaming), or `none`. |

**6. Preview the Sample Data.**

Click **Show Sample Data** to preview the source data before finalizing the configuration.

> **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)**
>
> Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview).


# Connect to WarpStream
Source: https://docs.startree.ai/corecapabilities/ingestdata/dataportal/streaming/warpstream

Create a connection to stream data using WarpStream's Kafka-compatible API.

**1. In the Data Portal, click Tables and then click Create Table.**

**2. Select Kafka as the Data Source.**

**3. Create a New Connection.**

Click **New Connection**. If you want to use an existing connection, select the connection from the list and proceed to **Step 5**.

Enter a **Source Name** for the new connection and specify the **Broker URL**.

Select the **Security Protocol** to be used for authentication, from the drop-down list.

**4. Configure Connection Parameters**

Add the connection parameters according to your authentication method.

## Connecting to Kafka with SASL\_PLAIN or SASL\_SCRAM

Use the following connection JSON when Kafka is configured with the SASL\_PLAINTEXT or SASL\_SSL security protocol enabled.

```json theme={null}
{
   "streamType": "kafka",
   "stream.kafka.broker.list": "",
   "security.protocol": "",
   "sasl.mechanism": "",
   "stream.kafka.password": "",
   "stream.kafka.username": "",
   "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required \n username=\"\" \n password=\"\";"
}
```

### Property Descriptions

| Property Name | Required | Description |
| - | - | - |
| stream.kafka.broker.list | Yes | List of Kafka brokers to connect to, typically in the format host1:port1,host2:port2. |
| security.protocol | Yes | The security protocol used to connect to Kafka (for example, SASL\_SSL, PLAINTEXT). This parameter is mandatory when the Kafka cluster uses SASL. |
| sasl.mechanism | Yes | The SASL mechanism for authentication (for example, PLAIN, SCRAM-SHA-512). This parameter is required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT. |
| stream.kafka.username | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT | The user name for authenticating with Kafka. |
| stream.kafka.password | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT | The password for authenticating with Kafka. |
| sasl.jaas.config | Required when security.protocol is set to SASL\_SSL or SASL\_PLAINTEXT | The JAAS configuration string for SASL authentication.  When the PLAIN SASL Mechanism is used: "org.apache.kafka.common.security.plain.PlainLoginModule required \n username="" \n password="";"  When the SCRAM SASL Mechanism is used: "org.apache.kafka.common.security.scram.ScramLoginModule required \n username="USERNAME" \n password="PASSWORD";" |

## Connecting to Kafka with SSL Security Protocol

See [prerequisites for SSL](/corecapabilities/ingestdata/dataportal/streaming/kafka-ssl-prerequisites) for steps to obtain the bootstrap server endpoint, the server certificate, the client certificate, and the client key, and to update an SSL certificate.

### One-way SSL (Server Authentication)

In one-way SSL, only the server presents its SSL certificate to the client to prove its identity. The client verifies the server's certificate using a trusted Certificate Authority (CA) before establishing an encrypted connection.
Use the following connection JSON when Kafka is configured with the SSL security protocol enabled:

```json theme={null}
{
  "stream.kafka.broker.list": "",
  "security.protocol": "SSL",

  "ssl.truststore.location": "",
  "ssl.truststore.password": "",

  "stream.kafka.ssl.server.certificate": ""
}
```

### Two-way SSL (Mutual Authentication)

In two-way SSL, both the client and the server present certificates to authenticate each other. The server verifies the client's certificate, and the client verifies the server's certificate. This setup provides a higher level of security and trust, commonly used.
Use the following connection JSON when Kafka is configured with the SSL security protocol enabled:

```json theme={null}
{
  "stream.kafka.broker.list": "",
  "security.protocol": "SSL",

  "ssl.truststore.location": "",
  "ssl.truststore.password": "",

  "ssl.keystore.location":"",
  "ssl.keystore.password":"",
  "ssl.key.password":"",

  "stream.kafka.ssl.server.certificate": "",
  "stream.kafka.ssl.client.certificate": "",
  "stream.kafka.ssl.client.key": ""
}
```

> [!NOTE]
>
> The values for the ssl properties (server/client certificate, key) need to be base64 encoded.

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.broker.list` | Yes | List of Kafka brokers to connect to, typically in the format `host1:port1,host2:port2`. |
| `security.protocol` | Yes | Defines the security protocol used to connect to Kafka (e.g., `SASL_SSL`, `SSL`, `PLAINTEXT`). This parameter is mandatory when the Kafka cluster uses SSL or SASL. |
| `ssl.truststore.location` | Yes | Absolute path to the truststore file that contains trusted CA certificates (used to verify Kafka broker certificates). |
| `ssl.truststore.password` | Yes | Password to open the truststore file. |
| `stream.kafka.ssl.server.certificate` | Required in one-way SSL | Server Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL`. It needs to be Base64 encoded. |
| `stream.kafka.ssl.client.certificate` | Required in two-way SSL | Client Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL`. It needs to be Base64 encoded. |
| `stream.kafka.ssl.client.key` | Required in two-way SSL | Certificate for connecting to a Kafka cluster secured with SSL. Required when `security.protocol` is set to `SSL` and `stream.kafka.ssl.client.certificate` is also set. It needs to be Base64 encoded. |
| `ssl.keystore.location` | Required in two-way SSL | Path to the keystore file containing the client's certificate and private key. |
| `ssl.keystore.password` | Required in two-way SSL | Password to open the keystore file. |
| `ssl.key.password` | Required in two-way SSL | Password protecting the private key inside the keystore. |

**5. Test the Connection and Configure Data Ingestion**

After you have configured the connection properties, test the connection to ensure it is working.

## Configure Data Ingestion

When the connection is successful, use the following JSON to configure the source's Kafka topic and data format:

```json theme={null}
{
  "stream.kafka.decoder.prop.format": "",
  "stream.kafka.decoder.class.name": "",
  "stream.kafka.topic.name": "",
  "stream.kafka.consumer.type": "lowlevel",
  "stream.kafka.consumer.factory.class.name": "ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory",
  "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
}
```

### Property Descriptions

| Property | Required | Description |
| - | - | - |
| `stream.kafka.decoder.prop.format` | Yes | Format of the messages in Kafka. Supported values include `json`, `avro`, `proto` etc. |
| `stream.kafka.decoder.class.name` | Yes | The class name of the decoder used for Kafka message parsing. It is set based on the message format and schema.    **Examples:**    \* For JSON Messages use `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder`     \* For AVRO Messages use `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` \* For PROTO Messages use `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` See [Message Decoders](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/overview) for more information. |
| `stream.kafka.topic.name` | Yes | The name of the Kafka topic from which Pinot will consume data. |
| `stream.kafka.consumer.type` | No | The type of Kafka consumer used in Apache Pinot. Use `lowlevel` for granular control of partitions and offsets. |
| `stream.kafka.consumer.factory.class.name` | Yes | The Kafka consumer factory class to use. The specified class `ai.startree.pinot.plugin.stream.kafka20.ConfluentKafkaConsumerFactory` is the default Kafka consumer factory when using Kafka 2.0+ or Confluent Kafka clients.      Use `ai.startree.pinot.plugin.stream.kafka10.ConfluentKafkaConsumerFactory` for legacy Kafka (1.x) environments with Confluent client integration. |
| `stream.kafka.consumer.prop.auto.offset.reset` | Yes | The behavior when no committed offset exists or offsets are invalid (`smallest`, `latest`, `none`). Use `smallest` for backfilling or consuming all historical data. Use `latest` for real-time streaming. |

### Additional Configuration

The parameters mentioned above are the **minimum-required parameters** to establish the connection. Apache Pinot also provides additional configuration options to fine-tune and control the ingestion process. For a complete list of configuration parameters, refer to the [Apache Pinot documentation](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#consume-transactionally-committed-messages). In addition, [Kafka Consumer properties](https://kafka.apache.org/documentation/#consumerconfigs) can also be used.

## Configure Record Reader

> **AVRO**
>
> One configuration option `AvroRecordReaderConfig` is supported.
>
> * *enableLogicalTypes*: Enable logical type conversions for specific Avro logical types, such as DECIMAL, UUID, DATE, TIME\_MILLIS, TIME\_MICROS, TIMESTAMP\_MILLIS, and TIMESTAMP\_MICROS.
>
> ```bash theme={null}
> {
>   "enableLogicalTypes": "true"
> }
> ```
>
> For example, if the schema type is INT, logical type is DATE, the conversion applied is a TimeConversion, and the value is V; then a date is generated V days from epoch start.

> **PROTO**
>
> `ProtoBufRecordReaderConfig` exists in Pinot and the following configuration is possible.
>
> ```bash theme={null}
> {
>   "descriptorFile": "/path/to/descriptor/file"
> }
> ```

### Additional Configuration

The parameters mentioned above are the **minimum-required parameters** to establish the connection. Apache Pinot also provides additional configuration options to fine-tune and control the ingestion process. For a complete list of configuration parameters, refer to the [Apache Pinot documentation](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#consume-transactionally-committed-messages). In addition, [Kafka Consumer properties](https://kafka.apache.org/documentation/#consumerconfigs) can also be used.

**6. Sample Data**

Click **Show Sample Data** to see a preview of the source data.

> **[Next Step](/corecapabilities/ingestdata/dataportal/data-modeling/overview)**
>
> Proceed with [Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview).


# Ingest Data
Source: https://docs.startree.ai/corecapabilities/ingestdata/ingest_data_overview

Ingest data from various streaming and batch sources using connectors. The StarTree Data Portal makes it easy to ingest and transform data.

## Overview

StarTree Cloud provides a streamlined approach to data ingestion, eliminating the need to build complex ingestion architecture before using your data. The platform delivers pre-built, scalable ingestion mechanisms that seamlessly connect to your data sources, supporting petabyte-scale analytics with minimal setup.

![Ingestion Pn](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/images/ingestion.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=da72a5597c0363030ea23f68579228d3)

Key ingestion capabilities include:

* Real-time streaming with sub-second latency
* Scalable batch processing for large historical datasets
* Built-in connectors for popular data sources
* Automatic schema detection and optimization
* Support for custom transformations during ingestion

## Ingestion Types

### Real-Time Ingestion

StarTree Cloud enables streaming data ingestion from sources like Kafka, allowing you to query data within seconds of it being generated. This capability supports use cases requiring immediate insights, such as dashboards, monitoring, and real-time analytics.

### Batch Ingestion

For historical or large datasets stored in file systems like S3 or cloud data warehouses, StarTree Cloud provides efficient batch ingestion. This approach is optimized for loading large volumes of data while maintaining query performance.

> [!NOTE]
>
> StarTree Cloud also supports hybrid tables, which combine both real-time and batch data into a single table view. This configuration provides the benefits of both ingestion methods - real-time data access plus historical data completeness. Hybrid tables must be configured using Controller APIs. For detailed instructions, please refer to the [Hybrid Tables](/corecapabilities/manage-data/hybrid-tables) documentation.

## Table Creation Flow

Creating a table in StarTree Cloud involves the following key steps, they can be seamlessly done using [Data Portal](/corecapabilities/ingestdata/ingest_data_overview#data-portal) or using the [Controller APIs](/api-reference/table/add-the-tableconfigs-using-the-tableconfigsstr-json).

![Table Creation Flow](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/ingestdata/images/table_creation_flow.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=86cb13bfc29373a11afb0d22bc1724ca)

**1. Connection and Dataset**

* Create a reusable connection to your data source (S3, Kafka, Confluent, etc.)
* Test the connection to ensure it works properly
* Specify which topic or folders to ingest from
* Preview sample data to verify correct source selection

**2. Data Modeling**

* Review and edit schema details
* Adjust column properties to ensure correct data formats
* Configure any needed preload transformations
* Verify proper column type identification
* Preview updated sample data with applied changes

**3. Additional Configuration**

* Specify table type and time column for partitioning
* Configure real-time data update handling through upserts
* Define primary key for deduplication if needed

**4. Table Configuration**

* Apply additional configurations like indexing
* Review final table configuration and schema
* Create the table once all settings are confirmed

**5. Table Created**

## Data Portal

The StarTree Data Portal makes it easy to ingest data into Pinot tables stored in StarTree Cloud. The Data Portal has a visual interface, which lets you ingest data from a variety of streaming and batch sources. Perform various transformations with minimal complexity, minimizing potential errors. Save time by catching issues like data format incompatibility, poor data quality, and connectivity issues.

The Data Portal automatically generates certain indexes based on the Pinot schema and data characteristics which are done transparently to the user. You can tune certain column indexes or add new indexes such as StarTree (which enables users to generate highly optimized materialized views), to suit your specific use case.

Connect to your data sources quickly using our growing library of pre-built connectors.

## Streaming Sources

> **[Apache Kafka](/corecapabilities/ingestdata/dataportal/streaming/kafka)**
>
> Stream real-time events using Apache Kafka

> **[Amazon Kinesis](/corecapabilities/ingestdata/dataportal/streaming/kinesis)**
>
> Ingest real-time data from Amazon Kinesis

> **[Confluent Cloud](/corecapabilities/ingestdata/dataportal/streaming/confluent)**
>
> Ingest from fully managed Kafka in Confluent Cloud

> **[Redpanda](/corecapabilities/ingestdata/dataportal/streaming/redpanda)**
>
> High-performance streaming ingestion with Redpanda

> **[Aiven Kafka](/corecapabilities/ingestdata/dataportal/streaming/aiven)**
>
> Connect to the managed Kafka service by Aiven

> **[WarpStream](/corecapabilities/ingestdata/dataportal/streaming/warpstream)**
>
> Stream data using WarpStream's Kafka-compatible API

## Batch Sources

> **[Amazon S3](/corecapabilities/ingestdata/dataportal/batch/s3)**
>
> Batch ingest files stored in Amazon S3 buckets

> **[Snowflake](/corecapabilities/ingestdata/dataportal/batch/snowflake)**
>
> Load batch data from Snowflake into StarTree Cloud

> **[Google BigQuery](/corecapabilities/ingestdata/dataportal/batch/bigquery)**
>
> Load data from Google BigQuery tables and views

> **[Google Cloud Storage](/corecapabilities/ingestdata/dataportal/batch/gcs)**
>
> Batch ingest files from Google Cloud Storage

> **[Azure Data Lake Storage](/corecapabilities/ingestdata/dataportal/batch/adls)**
>
> Batch ingest files from Azure Data Lake Storage


# Connect MongoDB
Source: https://docs.startree.ai/corecapabilities/ingestdata/mongodb

Choose and configure a path for getting MongoDB data into StarTree Cloud.

Getting MongoDB data into StarTree means picking one of three ingestion paths and writing the table or job configuration yourself.

This page helps you pick, then walks through each one.

## Pick a path

| Path | Use when | Freshness | Effort |
| - | - | - | - |
| [Direct Change Streams](#direct-change-streams-cdc) | You want a live, current-state table and you do not already run Kafka for this data. | Seconds | High — you own a manifest of physical MongoDB collections |
| [Debezium into Kafka](#debezium-into-kafka) | You already publish MongoDB CDC to Kafka. | Seconds | Medium — reuse your existing pipeline, add ingestion transforms |
| [`mongodump` BSON files](#mongodump-bson-files) | You need a one-time load, a periodic refresh, or a backfill. | Batch | Low |

> [!NOTE]
>
> A frequent wrong turn: the [Debezium message decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/debezium) does **not** parse MongoDB envelopes. Its `dbz.source` accepts only `mysql` and `postgres`. Ingest Debezium's MongoDB output with the plain JSON decoder instead — see [Debezium into Kafka](#debezium-into-kafka).

## Direct Change Streams (CDC)

The StarTree MongoDB CDC connector opens MongoDB Change Streams straight from Pinot servers. No Kafka, Kinesis, or Pulsar sits in between.

### What you need first

* **MongoDB Atlas, a replica set, or a sharded cluster.** Standalone `mongod` processes do not serve Change Streams.
* **MongoDB Server 6.0 or later** if you want a current-state upsert table, because that mode requires Change Stream post-images.
* **Network reachability and credentials on both Pinot controllers and servers.** The identity needs `find` and `changeStream` on every source namespace.
* **An oplog retention window longer than your worst-case Pinot outage.** The oplog is the only replay buffer. If it ages out, recovery means rebuilding the table from a snapshot, not resuming.
* **A partition plan.** This is the part teams underestimate — read the next section before anything else.

### The partition plan is the real design work

MongoDB Change Streams have no Kafka-style durable partitions, so the connector cannot discover them. You supply a manifest of physical, disjoint collections, and each one becomes one Pinot partition group:

```text theme={null}
app.orders_00  ->  Pinot partition group 0
app.orders_01  ->  Pinot partition group 1
app.orders_02  ->  Pinot partition group 2
```

Two consequences drive most of the design:

**Every key must stay in its partition forever.** Every insert, update, replace, and delete for one `_id` has to land in the same collection for that key's lifetime. A common way to guarantee that is to encode the partition into the immutable `_id` at creation time:

```text theme={null}
_id = "02:ord_123"  ->  app.orders_02
_id = "07:ord_456"  ->  app.orders_07
```

Do not route with `hash(_id) % currentPartitionCount`. Growing the count moves existing keys and breaks upsert partition affinity.

**Cursor load multiplies.** Each Pinot replica opens its own cursor per partition, so plan for `source partitions x Pinot replicas`. On a sharded deployment, a cursor opened through `mongos` fans out to every shard regardless of shard key, so plan closer to `source partitions x Pinot replicas x MongoDB shards`.

### Minimum stream config

```json theme={null}
{
  "streamType": "mongodb",
  "stream.mongodb.topic.name": "mongo-orders",
  "stream.mongodb.consumer.factory.class.name": "ai.startree.pinot.plugin.stream.mongodb.MongoCdcConsumerFactory",
  "stream.mongodb.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
  "stream.mongodb.consumer.prop.auto.offset.reset": "largest",
  "sourcePartitions": "[{\"partitionId\":0,\"epoch\":1,\"sourceId\":\"atlas-prod-a/app/orders-00\",\"connectionStringEnvironmentVariable\":\"MONGODB_ATLAS_URI\",\"database\":\"\",\"collection\":\"\"}]",
  "outputMode": "full_document",
  "fullDocument": "required",
  "maxAwaitTimeMs": "500"
}
```

`stream.mongodb.topic.name` is only a logical stream name — it does not identify a MongoDB namespace. Prefer `connectionStringEnvironmentVariable` or a `connectionRef` over an inline `connectionString`, because table configs can be logged or exposed.

> [!WARNING]
>
> This config is for an **empty or brand-new** source collection. With `auto.offset.reset=largest` the controller captures a Change Stream boundary at table creation and consumes only changes after it — documents already sitting in the collection are never emitted, and any that are never updated again stay permanently absent from the table.
>
> For a collection that already holds data, do not use this config as-is. Capture an exact boundary per namespace first, snapshot/backfill up to it, then start CDC from that same boundary supplied as each descriptor's `initialOffset`. See [Capturing Initial Offsets](/corecapabilities/ingestdata/adv-concepts/realtime/mongodb-cdc#capturing-initial-offsets).

Pick an output mode:

* `change_stream` (default) keeps the lossless MongoDB event envelope with `fullDocument` nested inside it.
* `full_document` lifts document fields to the top level and adds reserved `__mongodb_cdc_*` metadata columns. Use this for a current-state upsert table. It needs `fullDocument=required` and post-images enabled on every source collection:

  ```javascript theme={null}
  db.runCommand({
    collMod: "orders_00",
    changeStreamPreAndPostImages: {enabled: true}
  })
  ```

For the full option set — registry-based partition discovery, capturing exact initial offsets, the upsert schema, and adding capacity to a live table — see [MongoDB CDC Ingestion](/corecapabilities/ingestdata/adv-concepts/realtime/mongodb-cdc).

## Debezium into Kafka

If Debezium already streams your MongoDB collections into Kafka, ingest that topic as a normal Kafka stream. The one thing to get right is the decoder.

The StarTree [Debezium message decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/debezium) handles MySQL and Postgres only. For MongoDB, use `JSONMessageDecoder` and unpack the envelope with ingestion transforms.

Debezium's MongoDB connector puts the changed document in `after` as a **JSON string**, not a nested object, so read fields out of it with `jsonPathString`:

```json theme={null}
{
  "streamConfigs": {
    "streamType": "kafka",
    "stream.kafka.topic.name": "",
    "stream.kafka.broker.list": "",
    "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
    "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
    "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
  },
  "ingestionConfig": {
    "transformConfigs": [
      {"columnName": "orderId", "transformFunction": "jsonPathString(after, '$._id')"},
      {"columnName": "status", "transformFunction": "jsonPathString(after, '$.status')"},
      {"columnName": "amount", "transformFunction": "jsonPathDouble(after, '$.amount')"},
      {"columnName": "opType", "transformFunction": "op"},
      {"columnName": "eventTimeMs", "transformFunction": "ts_ms"}
    ]
  }
}
```

> [!WARNING]
>
> The transforms above alone cannot back an upsert table. On a delete, Debezium sets `after` to null and carries the deleted document's `_id` only in the **Kafka record key**. `JSONMessageDecoder` decodes the record value, so every `jsonPathString(after, ...)` transform yields null and the row has no primary key — Pinot cannot match the existing row to remove it.

For an upsert table, unwrap the envelope in Debezium instead of in Pinot. The MongoDB `ExtractNewDocumentState` SMT lifts the document fields to the top level of the value, but it only preserves the deleted document's `_id` if you ask it to — both of the `delete.tombstone.handling.mode` settings below are required:

```properties theme={null}
transforms=unwrap
transforms.unwrap.type=io.debezium.connector.mongodb.transforms.ExtractNewDocumentState
transforms.unwrap.delete.tombstone.handling.mode=rewrite
transforms.unwrap.delete.tombstone.handling.mode.rewrite-with-id=true
transforms.unwrap.add.fields=op,ts_ms
```

`rewrite` keeps deletes as real records carrying a `__deleted` boolean instead of emitting null-valued tombstones, and `rewrite-with-id` copies `id` from the record key into the payload as `_id`. Without both, a delete still reaches Pinot with no primary key and cannot remove the existing row.

Records then arrive as flat documents, so `JSONMessageDecoder` needs no `jsonPath` transforms at all: `_id` becomes a normal column you can name in `primaryKeyColumns`, `__deleted` is the `deleteRecordColumn`, and `__ts_ms` serves as the upsert comparison column. `add.fields` prefixes what it adds with `__` by default, configurable via `add.fields.prefix`.

Also set the connector's `capture.mode` to a full-document mode such as `change_streams_update_full`, so updates carry the whole document rather than a partial update description. Without it, updates arrive as patches that Pinot cannot merge.

## `mongodump` BSON files

Apache Pinot reads BSON natively, so `mongodump` output loads without conversion. Use this for backfills, periodic snapshots, or any table that does not need to be live.

Dump a collection and stage the files where your ingestion job can read them:

```bash theme={null}
mongodump --uri "$MONGODB_URI" --db app --collection orders --out ./dump
aws s3 cp ./dump/app/orders.bson s3:///mongo/orders/orders.bson
```

Then set the input format in the ingestion job spec:

```yaml theme={null}
executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'

jobType: SegmentCreationAndTarPush
inputDirURI: 's3:///mongo/orders/'
includeFileNamePattern: 'glob:**/*.bson'
outputDirURI: 's3:///segments/orders/'
overwriteOutput: true

pinotFSSpecs:
  - scheme: s3
    className: org.apache.pinot.plugin.filesystem.S3PinotFS
    configs:
      region: ''

recordReaderSpec:
  dataFormat: 'bson'
  className: 'org.apache.pinot.plugin.inputformat.bson.BSONRecordReader'

tableSpec:
  tableName: 'orders'
  schemaURI: '/tables/orders/schema'
  tableConfigURI: '/tables/orders'

pinotClusterSpecs:
  - controllerURI: ''
```

`className` is optional — Pinot resolves `dataFormat: 'bson'` to `BSONRecordReader` on its own — but naming it explicitly makes the spec self-documenting.

### How BSON types land in Pinot

| BSON type | Pinot-side value |
| - | - |
| `Double`, `Int32`, `Int64`, `Boolean`, `String` | Same type, passed through |
| `ObjectId` | `STRING` — the 24-character hex form |
| `DateTime` | Timestamp, millisecond precision |
| `Timestamp` (internal replication type) | Timestamp at **second** granularity; the within-second ordinal is dropped |
| `Decimal128` | `BigDecimal`. `NaN` and `Infinity` become null |
| `Binary` (including UUID subtypes) | Raw `byte[]`, not a `java.util.UUID` |
| Embedded document | Map — flatten it with [complex type handling](/corecapabilities/ingestdata/dataportal/data-modeling/unnesting-json) |
| Array | Multi-value column |
| Everything else | The driver's `toString()` rendering |

Because `Timestamp` loses sub-second ordering, do not use an oplog `ts` field as an upsert comparison column.

## Troubleshooting

| Symptom | Cause | Fix |
| - | - | - |
| `dbz.source` rejects `mongodb` | The Debezium decoder supports MySQL and Postgres only. | Use `JSONMessageDecoder` with ingestion transforms. |
| Change Streams fail to open | Source is a standalone `mongod`. | Change Streams need a replica set, sharded cluster, or Atlas. |
| CDC resume fails after an outage | Oplog or post-image history expired. | Rebuild the table from a snapshot plus matching Change Stream boundaries. |
| Deletes never reach the table | No delete column is wired up. | Use `__mongodb_cdc_is_deleted` (CDC connector) or derive one from `op` (Debezium). |
| MongoDB load higher than expected | Cursors scale with partitions x replicas x shards. | Reduce partitions or replicas, or isolate sources on separate deployments. |
| Updates arrive as partial patches | Debezium is not capturing post-images. | Set the connector's capture mode to a full-document mode. |


# Fix Row Count Mismatches for Databricks CDC Delta Tables
Source: https://docs.startree.ai/corecapabilities/ingestdata/recipes/delta-databricks-cdc-row-counts

Filter superseded row versions out of Databricks streaming tables during Delta Lake ingestion, so Pinot row counts match Unity Catalog.

Use this recipe when a Delta table ingested with the [Delta Lake connector](/corecapabilities/ingestdata/adv-concepts/batch/delta-lake-connector) has **more rows in Pinot than Databricks reports**, and the table is a Databricks streaming table maintained by a CDC pipeline. Lakeflow Connect ingestion pipelines, such as the Salesforce connector, produce tables like this.

## Why the counts differ

Databricks keeps older versions of updated rows inside the table's Parquet files and marks them with hidden columns. The one that matters here is `__DeleteVersion`. When you query the table through Databricks or Unity Catalog, Databricks hides those superseded rows for you.

The Delta Lake connector reads the physical Delta files, and it ingests every row in them. Nothing filters on the hidden columns, so each superseded version lands in Pinot as an extra row. Running `OPTIMIZE` on the source table doesn't help, because the superseded rows survive compaction.

How to tell this is your problem:

* `SELECT COUNT(*)` in Pinot is higher than the same count in Databricks, and the gap grows as the source table takes more updates.
* The extra rows have distinct primary keys and aren't flagged as deleted in your business columns, so they don't look like duplicates.
* The gap remains on files that have no deletion vector, so deletion vectors aren't the cause.

You can confirm it in Databricks by reading the physical files (for example with `boto3` or a Parquet reader) and counting the rows where `__DeleteVersion` is set. That number should account for the gap. Direct Spark reads of the backing files may be blocked by your workspace's security settings.

## Fix: filter on `__DeleteVersion` at ingestion

**1. Add \_\_DeleteVersion to the Pinot schema**

Add the column as a `STRING` dimension and turn on column-based null handling:

```json theme={null}
{
  "schemaName": "",
  "enableColumnBasedNullHandling": true,
  "dimensionFieldSpecs": [
    {
      "name": "__DeleteVersion",
      "dataType": "STRING"
    }
  ]
}
```

Keep your existing columns. The snippet shows only what to add.

**2. Check which value marks a live row**

Before you add the filter, ingest a sample with the new column (a test table is fine) and look at its values:

```sql theme={null}
SELECT __DeleteVersion, COUNT(*) AS rows
FROM 
GROUP BY __DeleteVersion
ORDER BY rows DESC
LIMIT 10
```

In the tables we've seen, live rows come through as `0` and superseded rows carry a non-zero value. The count for `0` should match the row count in Databricks. If your live rows show a different marker, use that value in the next step.

**3. Add the ingestion filter**

Add a `filterConfig` to the table's `ingestionConfig`:

```json theme={null}
"ingestionConfig": {
  "filterConfig": {
    "filterFunction": "__DeleteVersion != '0'"
  },
  "batchIngestionConfig": {
    "segmentIngestionType": "REFRESH"
  }
}
```

Pinot **drops** every row where the filter function returns `true`, so the expression has to describe the rows you want to discard. `__DeleteVersion = 'null'` looks reasonable but does the opposite of what you want.

**4. Re-ingest the table**

The filter applies only to segments built after the change. Segments that already exist still contain the superseded rows. Drop and recreate the table, or delete its segments, and then trigger the `DeltaTableIngestionTask` again. See [Managing Adhoc Minion Tasks](/corecapabilities/manage-data/recipes/adhoc-task-trigger) for how to trigger it without waiting for the schedule.

**5. Compare counts**

Once the task finishes, compare `SELECT COUNT(*)` in Pinot with the count in Databricks. Pinot can briefly lag behind if the source table changed after the task read its snapshot. The next run catches up.

## Limitations

* `__DeleteVersion` is an internal Databricks column. It isn't a documented interface, and its name, type, or values could change in a later Databricks release. Recheck the counts after Databricks upgrades your pipeline.
* The filter removes superseded row versions. It doesn't recover deletes that Databricks itself never applied. Lakeflow Connect doesn't propagate Salesforce hard deletes automatically, and it can miss soft deletes that were purged before the next sync. Those need a full refresh of the Databricks table. See [Databricks Salesforce connector limitations](https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/salesforce-limits#pipelines).
* If you can't change the Pinot table, you can instead publish a filtered copy from Databricks (`WHERE __DeleteVersion IS NULL`) to a separate Delta location and point `delta.ingestion.table.uri` at it. It costs an extra pipeline step, but Pinot no longer depends on Databricks internals.


# IAM-Role Based Access for Kinesis
Source: https://docs.startree.ai/corecapabilities/ingestdata/recipes/iam-role-kinesis



This guide details the process of setting up cross-account IAM roles to allow your **StarTree Cloud Data Plane** to access **Kinesis streams** residing in a separate AWS account.

### 1. Source Account (Account A) Configuration

This section covers the necessary configurations within the AWS account that owns the Kinesis stream (**Source Account**).

#### 1.1. Create an IAM Policy for Kinesis Access

Create a new **IAM policy** that grants the necessary permissions to read from your Kinesis stream.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords",
        "kinesis:ListShards",
        "kinesis:DescribeStreamSummary",
        "kinesis:RegisterStreamConsumer",
        "kinesis:SubscribeToShard",
        "kinesis:DescribeStreamConsumer"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:kinesis:::stream/",
        "arn:aws:kinesis:::stream//*"
      ]
    }
  ],

}
```

##### Replace the placeholders:

1. REGION: The AWS region where your Kinesis stream is located (e.g., us-east-1).
2. SOURCE\_AWS\_ACCOUNT\_ID: Your AWS account ID where the Kinesis stream resides.
3. STREAM\_NAME: The name of your Kinesis stream.

#### 1.2. Create an IAM Role (e.g., KA-Source-Stream-Role)

Create a new IAM role in your Source Account. After creation, attach the IAM policy you created in the previous step to this new role.

#### 1.3. Update Trust Relationships for the IAM Role

Modify the trust policy of the newly created IAM role (KA-Source-Stream-Role) to allow the StarTree Cloud Data Plane (acting as a "Sink AWS Principal") to assume this role. This establishes the cross-account trust.

```json theme={null}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": ""
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
```

##### Replace the placeholder:

1. SINK\_AWS\_PRINCIPAL\_ARN: The ARN of the AWS Principal in the Sink Account (Account B) that will be assuming this role. This can be:

* The Root User: arn:aws:iam::SINK\_AWS\_ACCOUNT\_ID:root
* A Service Role: arn:aws:iam::SINK\_AWS\_ACCOUNT\_ID:role/service-role/SERVICE\_ROLE\_NAME

### 2. Sink Account (Data Plane - Account B) Configuration

This section details the configuration required within the AWS account where your StarTree Data Plane is deployed (Sink Account).

#### 2.1. Identify the Data Plane IAM Role ARN

Obtain the ARN of the EC2 Instance Profile Role attached to the EC2 instances running your StarTree Data Plane. This role is automatically created by StarTree during environment deployment.

#### 2.2. Attach an Assume Role Policy to the Data Plane IAM Role

Create and attach an IAM policy to the Data Plane IAM Role (identified in the previous step). This policy grants the Data Plane permission to assume the KA-Source-Stream-Role in the Source Account.

```json theme={null}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AssumeRoleInSourceAccount",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam:::role/KA-Source-Stream-Role"
        }
    ]
}
```

##### Replace the placeholder:

1. SOURCE\_AWS\_ACCOUNT\_ID: Your AWS account ID where the Kinesis stream and KA-Source-Stream-Role reside.

### 3. Verify Access

Once the IAM roles and policies are configured, you can verify access by creating a table in StarTree Data Portal. [More documentation here](/corecapabilities/ingestdata/dataportal/streaming/kinesis).


# IAM-Role Based Access for S3
Source: https://docs.startree.ai/corecapabilities/ingestdata/recipes/iam-role-s3



This document guides you through setting up an **AWS IAM role** that allows your StarTree Cloud environment to securely access data stored in your Amazon S3 buckets. This is crucial for enabling StarTree to ingest data directly from S3.

## Setting Up Your AWS IAM Role for S3 Access

Before you begin creating the role, please collect the following vital pieces of information:

* **StarTree Account ID** and **External ID**: You can find this when you initiate the process of creating an S3 connection within the StarTree Data Portal.
* **StarTree Environment Name**: This is visible in your StarTree Cloud Portal.

### Step 1: Verify AWS STS Configuration

To ensure cross-account access works correctly, the AWS Security Token Service (STS) must be active in the region where your S3 bucket is located.

1. Open the [AWS IAM console](https://console.aws.amazon.com/iam) and click on **Account settings** in the left navigation pane.
2. Locate the **Security Token Service (STS)** section. Make sure its status is **Active** for the AWS region where your S3 bucket resides. If it's not, please activate it.

### Step 2: Create the IAM Role

Now, let's create the IAM role that StarTree will assume to access your S3 bucket.

1. In the AWS IAM console, navigate to **Roles** and then click the **Create role** button.
2. Under **Trusted entity type**, select **AWS account**.
   * **If StarTree and S3 are in the same AWS account**: Select **This Account**.
   * **If StarTree and S3 are in different AWS accounts**:
     * From the StarTree Data Portal, copy the **StarTree AWS Account ID** and paste it into the **Another AWS Account** field.
     * Next, copy the **External ID** from the StarTree Data Portal and paste it into the **External ID** field.
3. **Attach Permissions**: Search for and select the IAM policy you've already created that grants access to your S3 bucket (e.g., `s3-my-bucket-policy` as used in our example). Click **Next**.
4. **Name Your Role**:
   * If your **External ID** has the `env-*` prefix (which is common for newer environments), you can skip this step, and the role will be named automatically based on the External ID.
   * Otherwise, provide a clear **Role name** (e.g., `startree-my-stream-role`) and consider adding descriptive **Tags** (key-value pairs) for organization.
5. Finally, click **Create role**.

### Step 3: Provide the Role ARN to StarTree

After the role is successfully created:

1. Go to the role's Summary page in the AWS IAM console.
2. Copy the **Role ARN** (Amazon Resource Name).
3. Paste this ARN into the **IAM Role ARN** field within the StarTree Data Portal when setting up your S3 connection. This tells StarTree which role to assume for accessing your S3 data. [More documentation here](/corecapabilities/ingestdata/dataportal/batch/s3).


# Troubleshooting ingestion
Source: https://docs.startree.ai/corecapabilities/ingestdata/troubleshooting

Diagnose real-time ingestion that has stopped or fallen behind, batch and minion tasks that failed, and tasks that were never scheduled.

This guide covers data that has stopped arriving, data arriving later than it should, and the tasks responsible for moving it.

> [!TIP]
>
> If StarTree support named an alert, such as `RealtimeIngestionStopped` or `FailedToGenerateFileIngestionTask`, look it up in [Real-time ingestion alerts](/corecapabilities/observability/alerts/real-time-ingestion) or [Batch ingestion and minion task alerts](/corecapabilities/observability/alerts/batch-and-minion-tasks) first. Each entry says what fired and what to check.

## First: lag, stopped, or never started?

These three look similar from the outside and have different causes. Establish which one you have before going further.

**1. Check the ingestion health check**

On the Data Portal **Health Dashboard**, open the table and expand `TABLE_INGESTION_CHECK`. It reports:

| Field | What it tells you |
| - | - |
| `partitionsNotConsuming` | How many stream partitions have stopped |
| `totalPartitions` | How many there should be |
| `consumingSegmentsCount` | How many segments are actively consuming |
| `maxIngestionDelayMs` | Worst-case lag between the source and the table |
| `maxEndToEndIngestionDelayMs` | Worst-case lag including the producer side |

Read it as:

* **`partitionsNotConsuming` is 0 and lag is high** → you are behind, but still moving. See [Ingestion is lagging](#ingestion-is-lagging).
* **`partitionsNotConsuming` is non-zero** → some or all consumption has stopped. See [Ingestion has stopped](#ingestion-has-stopped).
* **`partitionsNotConsuming` equals `totalPartitions`** → nothing is consuming at all. Check first whether consumption was paused deliberately.
* **`totalPartitions` does not match your source** → your stream topology changed. See [The source changed](#the-source-changed).

**2. Confirm the data really is missing**

Query the table directly for the time range in question, filtering on the table's time column rather than relying on a dashboard that may cache. On a hybrid table, query the `_REALTIME` and `_OFFLINE` halves separately — data that looks missing is sometimes present on the other side of the time boundary. See [Hybrid tables](/corecapabilities/manage-data/hybrid-tables).

## Ingestion is lagging

Data is still arriving, just late. The question is whether the gap is growing or recovering.

**1. Determine the direction of travel**

Check `maxIngestionDelayMs` several times a few minutes apart.

* **Growing** — consumption is slower than production. This is a capacity or throughput problem.
* **Flat and high** — you fell behind once and are keeping pace without catching up. Same causes, less urgent.
* **Shrinking** — you are recovering. Confirm the rate is fast enough to matter, then find what caused the original gap.

**2. Check whether production rate changed**

A lag that starts without any change on the StarTree side is usually a change on the producer side — a traffic peak, a backfill, or a replay. Compare your current message rate against a normal day.

Lag during a known peak with recovery afterwards may not need any action beyond planning for the next one.

**3. Check whether the consumers have room**

If production is unchanged, the consuming side has slowed. Look for memory or CPU pressure on the servers hosting the table — see [Resource pressure](/corecapabilities/cluster-operations/troubleshooting). Servers under pressure consume more slowly before they fail outright, so growing lag is often the first visible symptom of a capacity problem rather than a problem in its own right.

**4. Check partition balance**

If some partitions are far behind and others are current, the problem is skew rather than capacity — the partitioning key is concentrating traffic. `SEGMENT_SKEW_HEALTH_CHECK` on the Health Dashboard surfaces the segment-level version of this.

## Ingestion has stopped

One or more partitions are no longer consuming.

> [!WARNING]
>
> Check first whether consumption was paused — deliberately, or by an operation that pauses it as a side effect. A paused table looks identical to a broken one on most surfaces. See [Pause and resume consumption](/recipes/pause-resume-consumption).

Once you have ruled out a pause, work through these:

1. **Did the upstream schema change?** A producer that starts emitting a new schema version can stop consumption cleanly, with no error on the StarTree side beyond the halt itself. This is one of the most common causes and one of the least obvious. Check when your producer last deployed against when consumption stopped.

2. **Is the problem in decoding rather than consuming?** If messages are arriving but failing to parse, the decoder is at fault rather than the stream. Each decoder documents its own failure modes — see [Troubleshooting by feature](/corecapabilities/observability/troubleshooting-by-feature#streaming-ingestion).

3. **Did credentials or permissions change?** Rotated keys, an expired token, or a changed IAM role stop consumption at the next fetch. If the stop is abrupt and total, this is a strong candidate.

4. **Is the source reachable and does the offset still exist?** If retention on the source expired past the offset the table was reading, consumption cannot resume from where it left off. This is more likely the longer a table has been stopped, and it changes the remedy — you are choosing where to restart, not resuming.

5. **Did the partition count change?** See [The source changed](#the-source-changed).

6. **Are segments stuck in a bad state?** Check `TABLE_SEGMENT_AVAILABILITY_CHECK` and `IDEAL_STATE_EV_MISMATCH_CHECK`. A consuming segment that cannot commit will block the partition behind it. [Force commit](/recipes/force-commit) is sometimes the way forward, but understand why the segment is stuck before forcing it.

> [!NOTE]
>
> For tables using [pauseless consumption](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption), the commit path differs and consumption is not interrupted by segment commit. That page's [Operational Troubleshooting](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#operational-troubleshooting) section covers ordering enforcement and stalled transitions, and its [Debugging APIs](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#debugging-apis) section covers inspecting consuming-segment state directly.

## The source changed

Changes to your stream's topology are not always visible from the table until they cause a symptom.

* **Partition count increased.** New partitions need to be picked up. Until they are, you are silently missing whatever routes to them — the table looks healthy and the data is incomplete, which is worse than an outright failure. Compare `totalPartitions` on the ingestion check against your source.

* **Partition count decreased, or brokers were removed.** Consumption may continue against partitions that no longer exist, or fail in ways that look like connectivity problems.

* **The topic was recreated with the same name.** Offsets no longer mean what the table thinks they mean.

* **Schema version changed.** See [Ingestion has stopped](#ingestion-has-stopped), item 1.

> [!TIP]
>
> A change to partition count on a partitioned table also affects query pruning, so the symptom may show up first as slower queries rather than as an ingestion problem.

## Batch and minion task failures

Tasks that move data in batches — file ingestion, segment refresh, purge, merge, external table sync — fail differently from streaming ingestion.

**1. Find the task and its state**

In the Data Portal, open **Tasks** and locate the task plan. See [Task Observability](/corecapabilities/observability/minion-task-runs) for how to read plan, task and subtask state, and for the progress logs each subtask emits.

**2. Distinguish 'failed' from 'never scheduled'**

These are different problems with the same symptom — no new data.

A task that **ran and failed** has a failure message and a subtask to inspect.

A task that was **never generated** has nothing to inspect, because nothing ran. The usual cause is that the generator evaluated its inputs and matched nothing: a path or prefix that resolves to no files, a source with no new data since the last run, or a filter that excluded everything. Verify independently that the files or rows you expect actually exist at the location the task is configured to read.

**3. Read the innermost error**

Task failures often surface a generic outer message with the real cause nested inside. Storage access problems in particular — wrong region, missing permission, moved object — produce inner exceptions that name the problem precisely. Always expand to the innermost cause before concluding anything.

**4. Check whether the failure is resource-related**

Minions that run out of memory take their tasks down with them, and a minion that restarts repeatedly produces a pattern of unrelated tasks failing together. If several different task types started failing at the same time, suspect the minion rather than the tasks. See [Resource pressure](/corecapabilities/cluster-operations/troubleshooting).

**5. Check for a stuck queue**

A task queue that has stopped progressing holds up everything behind it. If tasks are pending rather than failing, and the oldest pending task is not moving, that is a queue problem rather than a task problem.

## Data arrived but looks wrong

If rows are present but the values or counts are not what you expect:

* **Compare row counts against the source** for a bounded time window, rather than comparing totals. A discrepancy in one window is far easier to trace than a discrepancy in the whole table.

* **On hybrid tables, check the time boundary.** Rows can appear missing or duplicated depending on where the boundary sits relative to the data. See [Hybrid tables](/corecapabilities/manage-data/hybrid-tables).

* **On upsert tables, check the primary key and comparison column.** Unexpected values usually mean records are being resolved against a different previous record than you assume. See the [Upsert operations guide](/corecapabilities/manage-data/upsert-operations-guide).

* **For duplicates or counts running ahead of the source**, [Diagnosing duplicates and count mismatches](/corecapabilities/manage-data/offheap-upsert#diagnosing-duplicates-and-count-mismatches) gives the concrete techniques: querying with `skipUpsert` to see un-masked rows, projecting the `$segmentName` / `$docId` / `$hostName` virtual columns to locate a duplicate, and comparing strict-replica-group results to detect cross-replica divergence.

* **Check whether retention removed the data.** `SEGMENT_RETENTION_CHECK` on the Health Dashboard shows retention state. Data older than your configured retention is supposed to disappear.

* **Check transformations.** A transformation or filter function that silently drops records will show up as a row-count shortfall with no error anywhere. If counts are short and nothing failed, review the ingestion transform config against a sample of the records you expect to be present.

## Feature-specific guidance

For task types with their own documented failure modes — purge, backfill, the real-time to offline flow, minion orchestration — see [Troubleshooting by feature](/corecapabilities/observability/troubleshooting-by-feature#batch-ingestion-and-minion-tasks).

**External table sync?** [External table troubleshooting](/corecapabilities/external-table/troubleshooting), *Sync & operations* section, covers a sync that never starts, a run failing on a single unreadable file, distributed-lock contention on `ExternalTableSyncTask`, and segments failing to load after upstream compaction.

## Escalating

Include: how far behind, whether the gap is growing, which partitions are affected, what changed upstream and when, and for task failures the task type, plan or task ID, and the full nested error. See [what to collect](/corecapabilities/observability/troubleshooting#before-you-open-a-support-ticket).


# Alter Table Task
Source: https://docs.startree.ai/corecapabilities/manage-data/alter-table-task



The `StarTreeAlterTableTask` keeps segments in sync with the latest Pinot table configuration by detecting inconsistencies, rebuilding affected segments, and replacing outdated segments atomically. It is used for changes such as re-sorting, re-partitioning, adding indexes, resizing segments, and syncing deep store segments with the table configuration.

## How is the Alter Table Task different from the Segment Refresh Task?

The StarTree Alter Table Task is an improved alternative to the [Segment Refresh Task](/corecapabilities/manage-data/segment-refresh-task), and is designed for more efficient, batch-wise segment refreshes.

The Segment Refresh Task performs the map and segment generation phases locally, which can create redundant segment files and extra merge work. The StarTree Alter Table Task writes mapper output files to shared storage, then generates replacement segments directly from that output.

The Alter Table Task follows a structured Map, Reduce, and Upload workflow. This gives better resource utilization and fault tolerance for large table refreshes. Built-in retries adjust memory and segment-generation parameters when subtasks fail.

## When is the Alter Table Task used?

* Re-partitioning in case of a change in the partition columns.
* Re-sorting by changing the sorted column.
* Merge or split segments to keep segment sizes healthy.
* Change a column data type.
* Change a column encoding.
* Adding new indexes.
* Other table config changes that cannot be applied on the server side with segment reload.

## How does the Alter Table Task work?

The Alter Table Task is a [Minion](/corecapabilities/ingestdata/adv-concepts/batch/overview#pinot-minions%3A-cluster-native-ingestion-%26-processing) task in Apache Pinot, designed to run asynchronously without affecting real-time or offline ingestion.

The Alter Table Task refreshes table segments in batches. In each batch, the task selects a subset of segments, processes them, and replaces them in the original table. The process repeats until all segments are updated.

> [!NOTE]
>
> If new segments requiring a refresh arrive during an ongoing batch, they are deferred until the current batch completes.

### Example workflow

Suppose we need to refresh a 10TB table, with the following table configuration:

* **tableMaxNumTasks**: 100 (the default for non-upsert Alter Table Task runs)
* **desiredSegmentSize**: 500MB

The Alter Table Task workflow splits the table into batches based on this formula:

* Each map task processes data up to `taskMaxDataSize`, defaulting to 5GB, or `maxNumRecordsPerTask`, defaulting to 3 times the computed `maxNumRecordsPerSegment`.
* With `tableMaxNumTasks` set to 100 and `taskMaxDataSize` at 5GB, each batch can process up to about 500GB.
* Therefore, it will take approximately 20 batches (10TB / 500GB = 20) to process a 10TB table.

### Batch Processing Stages

Each batch has these stages:

* The **Map Phase** reads input segments and generates intermediate files. Each subtask partitions the data (`partition_timeBucket`) and uploads the files to deep storage.
* The **Reduce Phase** downloads the necessary intermediate files, processes them to generate new segments and metadata, and then uploads both to deep storage.
* The **Upload Phase** retrieves the list of `fromSegments` from metadata, determines `toSegments` from deep storage, and performs batch segment replacement. By default, batch upload is used as it is the most efficient mode.

All stages are associated with the same **session ID**, ensuring continuity and data integrity.

## Enabling the Alter Table Task

To enable the Alter Table Task, simply configure it in your table configuration as follows:

1. In `task.taskTypeConfigsMap`, add an object called `StarTreeAlterTableTask`.
2. Add the following configuration:

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "StarTreeAlterTableTask": {
        "bucketTimePeriod": "1d",
        "desiredSegmentSize": "500MB",
        "schedule": "0 */10 * ? * * *"
      }
    }
  }
}
```

3. Run the task immediately using the `POST /tasks/schedule?taskType=StarTreeAlterTableTask&tableName=` API, or wait for the cron job to trigger it. In the example above, the task is scheduled every 10 minutes.

## Configuration Parameters

The following parameters are configured under `task.taskTypeConfigsMap.StarTreeAlterTableTask`.

### Scheduling and mode

| Parameter | Required | Default | Applies to | Description |
| - | - | - | - | - |
| `schedule` | No | None | Scheduled runs | Quartz cron expression for recurring task generation. If omitted, trigger the task manually with the task scheduling API. |
| `bucketTimePeriod` | Yes, unless `reloadOnly=true` | None | Non-upsert Map/Reduce/Upload runs | Time bucket for output segments. A generated segment should not span more than one bucket. |
| `reloadOnly` | No | `false` | Deep store sync | When `true`, ATT runs only the reload phase and refreshes stale deep-store segments in place. This is the mode used by [Deep Store Segment Sync](/corecapabilities/manage-data/deep-store-sync). |
| `forceReload` | No | `false` | Deep store sync | Used only when `reloadOnly=true`. Reloads all selected segments without relying on the table-hash freshness check. |
| `minionInstanceTag` | No | Untagged Minion instance tag | All runs | Restricts generated ATT subtasks to Minions with the configured Helix instance tag. |

> [!NOTE]
>
> Upsert tables use the upsert refresh path. In that path, `validDocIds*`, `invalidRecordsThresholdPercent`, and `maxSegmentsPerTask` apply; `reloadOnly` and `forceReload` are deep-store-sync controls and do not change upsert compaction selection.

### Segment selection and task sizing

| Parameter | Required | Default | Applies to | Description |
| - | - | - | - | - |
| `tableMaxNumTasks` | No | `100` for non-upsert ATT; `1000` for upsert tables | All runs | Maximum subtasks generated per table per scheduling round. Lower this on large tables to reduce upload, replacement, and disk pressure. |
| `desiredSegmentSize` | No | `500MB` | All runs | Desired output segment size. When set, ATT computes `maxNumRecordsPerSegment` from table statistics. This takes precedence over `maxNumRecordsPerSegment`. |
| `maxNumRecordsPerSegment` | No | Computed from `desiredSegmentSize`; fallback code default is `5000000` | All runs | Maximum rows per output segment when `desiredSegmentSize` is not set or cannot be computed. |
| `maxNumRecordsPerTask` | No | `3 * maxNumRecordsPerSegment` | Non-upsert Map/Reduce/Upload runs | Maximum records processed by a map task. |
| `taskMaxDataSize` | No | `5GB` | Non-upsert Map/Reduce/Upload runs | Maximum input bytes processed by a map task. |
| `maxNumRowsSegmentDiffPercentThreshold` | No | `20` when `desiredSegmentSize` is used; otherwise `0` | Non-upsert Map/Reduce/Upload runs | Skips resizing when an existing segment's row count is within this percentage of the target. |
| `segmentSelector` | No | None | All runs | Expression used to filter selected segments by segment metadata, for example `segment.creation.time < 1780617600000`. |
| `segmentExpirationBufferPeriod` | No | `0` | All runs | Skips segments that are already expired or will expire within this extra retention buffer. Set a negative value to disable expiration filtering. |
| `bufferTimePeriod` | No | `3d` | Realtime tables | Skips recently completed realtime segments and the latest committed segment, so ATT does not interfere with ingestion offsets. |

### Output segment and rollup options

| Parameter | Required | Default | Applies to | Description |
| - | - | - | - | - |
| `generateSegmentOnlyAtPartitionTimeBucketBoundary` | No | `true` | Non-upsert Reduce phase | Keeps generated output segments from crossing the configured partition time bucket boundary. |
| `mergeType` | No | Pinot segment-generation default | Non-upsert Reduce phase | Advanced merge/rollup mode forwarded to Pinot segment generation. Omit this for ordinary ATT rebuilds. |
| `roundBucketTimePeriod` | No | None | Non-upsert rollup runs | Optional time rounding period forwarded to Pinot segment generation when rollup-style output is required. |
| `.aggregationType` | No | None | Non-upsert rollup runs | Per-column aggregation function forwarded to Pinot segment generation, for example `views.aggregationType`. |

### Upsert table options

| Parameter | Required | Default | Description |
| - | - | - | - |
| `validDocIdsType` | No | `SNAPSHOT`, or `SNAPSHOT_WITH_DELETE` when `upsertConfig.deleteRecordColumn` is set | Source used to fetch valid-doc-ids from servers. Supported values are `SNAPSHOT`, `IN_MEMORY`, `SNAPSHOT_WITH_DELETE`, and `IN_MEMORY_WITH_DELETE`. Snapshot modes require snapshots to be enabled in `upsertConfig`. |
| `validDocIdsConsensusMode` | No | `EQUAL` | Replica consensus policy when fetching valid-doc-ids for the deep-store segment CRC. Supported values are `UNSAFE`, `EQUAL`, and `MOST_VALID_DOCS`. Leave the default unless directed by StarTree support. |
| `validDocIdsFetchTimeoutMs` | No | `5000` | Timeout for fetching valid-doc-ids metadata from servers. Increase this for large tables or slow servers. |
| `validDocIdsFetchBatchSize` | No | `500` | Number of segments requested per server call when fetching valid-doc-ids metadata. Lower this if server requests are too heavy. |
| `invalidRecordsThresholdPercent` | No | `20` | A segment is eligible for refresh only when invalid records exceed this percentage. Use `0` to refresh any segment with invalid records. |
| `minPrematureBatchSize` | No | `0` | Minimum segment count required before flushing a partial batch when a segment is skipped during upsert batching. |
| `maxSegmentsPerTask` | No | `100` | Maximum input segments assigned to a single upsert refresh subtask. Lower this to reduce per-task memory, valid-doc-ids fetch size, and temporary disk growth. |
| `mergeMode` | No | `carryOverLineage` | Upsert segment lineage mode. Supported internal values include `carryOverLineage`, `noCarryOverLineage`, and `reProcessSegments`; use non-default modes only with StarTree support. |
| `segmentMapperFileSizeThresholdInBytes` | No | Pinot segment processor default | Upsert executor only. Optional intermediate mapper file size threshold passed to the upsert segment processor. |

### Large upsert table guidance

> [!WARNING]
>
> On large upsert tables, high ATT parallelism can temporarily increase table storage because new segments are uploaded before old segments are fully replaced and deleted. If `validDocIdsType` uses snapshot mode and snapshot generation lags behind ATT, old segments can remain visible to cleanup logic for longer than expected.

For large upsert maintenance runs:

* Prefer conservative `tableMaxNumTasks` and `maxSegmentsPerTask` values.
* Use `segmentSelector` to process older or narrower segment ranges first.
* Use `bufferTimePeriod` to avoid segments close to active ingestion.
* Use `validDocIdsType: "IN_MEMORY"` or `IN_MEMORY_WITH_DELETE` for one-off maintenance when valid-doc-ids snapshots are stale or snapshot generation takes too long. Snapshot mode is still preferred for routine runs when snapshots are fresh.
* Increase `validDocIdsFetchTimeoutMs` and reduce `validDocIdsFetchBatchSize` when valid-doc-ids requests time out or overload servers.

Example large upsert run:

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "StarTreeAlterTableTask": {
        "bufferTimePeriod": "1d",
        "maxSegmentsPerTask": "3",
        "tableMaxNumTasks": "100",
        "validDocIdsFetchTimeoutMs": "60000",
        "validDocIdsFetchBatchSize": "100",
        "invalidRecordsThresholdPercent": "0",
        "segmentSelector": "segment.creation.time < 1780617600000",
        "validDocIdsType": "IN_MEMORY"
      }
    }
  }
}
```

### Segment freshness checks

| Parameter | Required | Default | Applies to | Description |
| - | - | - | - | - |
| `skipSegmentIndexCheck` | No | `false` | Non-upsert Map/Reduce/Upload runs | Skips the metadata compatibility check that detects segments whose indexes, schema, sorting, partitioning, or column metadata do not match the current table config. |
| `useServerApiForSegmentStaleCheck` | No | `true` | Non-upsert Map/Reduce/Upload runs | Uses the server stale-segment API instead of controller-side batched metadata reads. Set to `false` to use the older batched metadata path. |
| `segmentMetadataFetchBatchSize` | No | `10000` | Non-upsert Map/Reduce/Upload runs with `useServerApiForSegmentStaleCheck=false` | Number of segment metadata records fetched per batch during compatibility checks. |

### Deep store, push, tiered storage, and cleanup

| Parameter | Required | Default | Description |
| - | - | - | - |
| `output.segment.dir.uri` | No | Table deep-store directory | Shared storage location for ATT intermediate files and generated segment artifacts. Override only for advanced deployments or tests. |
| `overwriteOutput` | No | `false`; `true` for `reloadOnly` metadata writes when absent | Allows ATT to overwrite existing files in the output directory. |
| `push.mode` | No | `METADATA` in StarTree Minion ATT push config | Segment push mode. ATT normally uses metadata push; use other modes only for advanced deployments. |
| `batchSegmentUpload` | No | `true` | Enables batch segment upload when `push.mode` is `METADATA`. |
| `push.segmentUriPrefix` | No | None | Optional URI prefix passed through to Pinot segment push. |
| `push.segmentUriSuffix` | No | None | Optional URI suffix passed through to Pinot segment push. |
| `push.attempts` | No | Pinot batch push default | Number of segment push attempts. |
| `push.retry.interval.millis` | No | Pinot batch push default | Delay between segment push retries. |
| `directUploadToTier` | No | Minion config `pinot.minion.direct.tier.upload.enable`, default `true` | For tiered storage tables, upload generated segments directly to the target tier when applicable. |
| `cleanupDanglingIntermediateFiles` | No | `true` | Cleans historical dangling ATT intermediate files after a session cleanup. |
| `bufferDaysToPurgeOutputSegments` | No | `3` | Extra retention buffer used when deleting old segment-list files from deep store. |
| `skipDeepStoreCleanup` | No | `false` | Skips deletion of ATT intermediate files and session cleanup artifacts. Use only for troubleshooting. |
| `endReplaceSegmentsTimeoutMs` | No | `3600000` | Timeout for the final `endReplaceSegments` controller call in the Upload phase. |
| `numMetadataFileDownloadThreads` | No | `4` | Parallelism for Reduce-phase metadata file downloads. Increase if metadata file download times out. |

### Execution and memory tuning

| Parameter | Required | Default | Description |
| - | - | - | - |
| `intermediateFileFormat` | No | `ARROW` | Intermediate mapper output format. Supported values are `ARROW`, `PARQUET`, and `GENERIC_ROW`. |
| `numTaskExecutorServiceThreads` | No | Available processors | Thread count for Map and Upload phase executor services. |
| `numSegmentGenerationThreads` | No | `2` | Thread count for Reduce phase segment generation. Retries can lower this value automatically. |
| `maxSubTaskRetryCount` | No | `5` | Maximum retries per failed subtask before ATT cleans up the session. |
| `offHeapMemoryAllocationForArrow` | No | Minimum of half MaxDirectMemory and 4GB; fallback 2GB | Total Arrow off-heap memory allocation before division by concurrent task count. |
| `upperCapForOffHeapMemoryAllocationForArrow` | No | `4294967296` | Upper cap, in bytes, used when deriving the default Arrow off-heap allocation. |
| `offHeapMemoryToMaxBatchBytesFactor` | No | `2` | Divides Arrow off-heap allocation to derive `maxBatchBytes` when `maxBatchBytes` is absent. |
| `maxBatchBytes` | No | Derived from Arrow off-heap memory | Maximum Arrow batch bytes before flushing mapper output. Retries lower this value down to a 200MB floor. |
| `maxBatchRows` | No | `1000000` | Maximum Arrow batch rows before flushing mapper output. |
| `rowsPerLoadForSortedCase` | No | `5000` | Rows loaded at a time for sorted-column reduce processing. |
| `fileWriterBufferSizeInBytes` | No | `1048576` | File writer buffer size for intermediate output. |
| `useFixedWindowScan` | No | `false` | Uses fixed-window reads for Arrow sorted-case processing. Use only when directed by StarTree support. |
| `use.InMemorySegmentProcessorFramework` | No | `false` | Experimental in-memory segment processor path. Use only when directed by StarTree support. |

### Ingestion transform controls

By default, ATT rebuild subtasks skip user-defined ingestion filters and transforms to avoid reapplying non-idempotent ingestion logic to already-built segments. Timestamp-index-generated transform configs and safety transformers still apply. Enable these only when you know the transform should be re-evaluated during rebuild.

| Parameter | Required | Default | Description |
| - | - | - | - |
| `skipIngestionFilters` | No | `true` | Skips inherited `ingestionConfig.filterConfig`. |
| `usePreComplexTypeTransformer` | No | `false` | Reuses inherited pre-complex enrichment configs. |
| `useComplexTypeTransformer` | No | `false` | Reuses inherited `complexTypeConfig`. |
| `usePostComplexTypeTransformer` | No | `false` | Reuses inherited post-complex enrichment configs, user transform configs, field transform functions, and schema-conforming transformer config unless overridden by the more specific flags below. |
| `useIngestionTransformFunctions` | No | Same as `usePostComplexTypeTransformer` | Reuses user-authored `transformConfigs` and non-virtual schema `transformFunction` values. |
| `useSchemaConformingTransformer` | No | Same as `usePostComplexTypeTransformer` | Reuses inherited `schemaConformingTransformerConfig`. |
| `ingestionTransformPipelineConfig` | No | None | JSON object that replaces the whole user-defined transform pipeline for the task. When set, it takes precedence over the shortcut flags below. |
| `ingestionFilterConfig` | No | None | JSON task-level replacement for `ingestionConfig.filterConfig`. |
| `preComplexTypeTransformerConfigs` | No | None | JSON list of pre-complex `EnrichmentConfig` replacements. |
| `complexTypeTransformerConfig` | No | None | JSON task-level replacement for `complexTypeConfig`. |
| `postComplexTypeTransformerConfigs` | No | None | JSON list of post-complex `EnrichmentConfig` replacements. |
| `ingestionTransformConfigs` | No | None | JSON list of task-level `TransformConfig` replacements. |
| `fieldTransformFunctions` | No | None | JSON object mapping schema column names to task-level field transform functions. |
| `schemaConformingTransformerConfig` | No | None | JSON task-level replacement for `schemaConformingTransformerConfig`. |

### Internal phase properties

ATT generates additional subtask properties such as `taskSessionId`, `isReloadMode`, `isMapMode`, `isReduceMode`, `isUploadMode`, `segmentsToRefresh`, `toSegmentsList`, `subTaskIdentifier`, `retryCount`, `numConcurrentTasks`, `partitionId`, `sequenceId`, `segmentName`, `downloadURL`, `uploadURL`, `enableReplaceSegments`, `tableName`, `dataChunksRowCount`, `sortColumnUrls`, `nonSortColumnUrls`, `partitionBucketTimePeriod`, `segmentNamePrefix`, `taskNamePrefix`, `taskNameSuffix`, `deepStoreSegmentsDir`, `mapperDir`, `dataDir`, `metadata`, and `bytesPerRow`. These are generated and managed by ATT. Do not set them in the table config unless StarTree support specifically asks you to recover or debug a task session.

## Monitoring

You can monitor the StarTree Alter Table Task in two levels of granularity:

* **Batch-Level**: Monitoring individual task batches.
* **Cycle-Level**: Tracking overall task cycles.

### Batch-Level Monitoring

Batch-level monitoring can be done using the Minion Task Manager or Minion Metadata.

* **Name-Based Task Monitoring**: Each task name contains two key details:
  * **Session ID**: Identifies a specific batch within a cycle.
  * **Task Phase**: Indicates whether the task is in the Map, Reduce, or Upload phase.
  * Since multiple batches can run within a cycle, they are linked together using the session ID.
* **Minion Metadata-Based Monitoring**: Minion Metadata provides real-time batch tracking through Zookeeper:
  * **Path**: Zookeeper Browser -> MINION\_TASK\_METADATA -> TableName -> StarTreeAlterTableTask.
  * Each batch has a fixed list of input segments, which remains unchanged during processing.

### Cycle-Level Monitoring

Cycle-level monitoring helps track the overall progress of the Alter Table Task across multiple batches.

**Progress Tracking & Estimation API:**

* **Endpoint**: `/alterTable/tableNameWithType/estimatedTime`
* **Method**: GET

**Sample Response**:

```json theme={null}
{
  "estimatedCompletionTimeMs": "123456789",
  "percentProgress": "50.37"
}
```

### New Timer Metrics for Observability

Additional metrics have been introduced for better tracking of each task phase:

* **ALTER\_TABLE\_MAP\_PHASE\_DURATION\_MS**: Time taken for the **Map** phase.
* **ALTER\_TABLE\_REDUCE\_PHASE\_DURATION\_MS**: Time taken for the **Reduce** phase.
* **ALTER\_TABLE\_UPLOAD\_PHASE\_DURATION\_MS**: Time taken for the **Upload** phase.

These metrics help monitor task performance and identifying potential bottlenecks during execution.

## Migrating from the Segment Refresh Task

> [!NOTE]
>
> To migrate, simply change the task name from SegmentRefreshTask to StarTreeAlterTableTask.

The Alter Table Task configuration is backward compatible with the Segment Refresh Task. Only one of these tasks is needed, so there is no need to run both.

> [!TIP]
>
> The Alter Table Task will eventually replace the Segment Refresh Task as the default.

## Handling Task Failures and Retries

Minion subtasks can fail due to various reasons. When this happens, tasks are automatically retried with adjusted configurations to improve success rates:

* **Map Phase**: maxBatchBytes = maxBatchBytes/(retryCount + 1).
* **Reduce Phase**: numSegmentGenerationThreads = numSegmentGenerationThreads/2, with a minimum value of 1.

By handling failures efficiently, the system ensures seamless segment refreshes with minimal disruptions.


# Custom Partition Function
Source: https://docs.startree.ai/corecapabilities/manage-data/custom-partition-function

Derive segment partition IDs from a Pinot scalar expression so StarTree Cloud can prune segments on transformed column values.

## Overview and Purpose

The `Custom` partition function computes a column's partition ID by evaluating a Pinot scalar expression against the column value, instead of applying one of the built-in partition functions such as `Murmur` or `Modulo`.

Partitioning writes a partition ID into each segment's metadata. At query time, StarTree Cloud evaluates the same partition function on the value in an equality predicate and skips every segment whose metadata says it cannot hold that partition. Built-in partition functions hash the raw column value, which works only when the value itself is what you partitioned on. The `Custom` function removes that restriction: the partition ID can come from any deterministic scalar expression over the value, such as decoding a hex string or hashing a string ID after converting it to bytes.

Use the `Custom` partition function when:

* The partition ID is derived from a transformed column value rather than the raw value.
* An upstream system already partitions data with a specific hash, and StarTree Cloud needs to reproduce that same partition ID to keep segment metadata aligned.
* You want segment pruning on a string ID column, hashed through a scalar function chain such as `fnv1a_hash_32_utf8(md5(toUtf8(v)))`.
* A built-in partition function exists but uses the wrong normalization for your partition count.

> [!NOTE]
>
> The `Custom` partition function drives two different things depending on how segments are built.
>
> It always writes partition metadata into each segment, which is what enables pruning. In addition, ingestion flows that run the `SegmentProcessorFramework` — `FileIngestionTask`, merge/rollup, and realtime-to-offline tasks — read `segmentPartitionConfig` and bucket rows by the configured partition function, so the same expression also decides which output segment each row lands in. See [Partitioning upstream data with FileIngestionTask](/corecapabilities/ingestdata/adv-concepts/batch/offline-upserts#3-partitioning-upstream-data-with-fileingestiontask).
>
> In those repartitioning flows, every value that evaluates to `-1` is grouped into a single output bucket. Validate the expression against sample values before pointing one of these tasks at a `Custom`-partitioned column.

## Availability

The `Custom` partition function is available in StarTree release 0.15 and later. The corresponding STP release branch is `release/1.6.0-STP-2.164.x`.

## How It Works

The function is configured per column under `indexingConfig.segmentPartitionConfig.columnPartitionMap`. For each value it processes:

1. The column value is bound to the reserved identifier `v` in the configured `partitionExpression`.
2. The expression is evaluated as a Pinot scalar expression.
3. The numeric result is mapped into `[0, numPartitions)` by the configured `partitionIdNormalizer`.
4. If the normalized value falls inside `[0, numPartitions)`, it becomes the partition ID.

If any of those steps cannot produce a valid partition ID, the function returns `-1`.

Pruning is a set-membership check: segment build records both the partition IDs a segment contains and the partition function that produced them. At query time StarTree Cloud evaluates that recorded function on the literal in an equality predicate and skips segments whose recorded set does not contain the resulting ID. Because the same expression runs on both sides, a `-1` is consistent and results stay correct. What you lose is selectivity — every value that fails to evaluate collapses into the same `-1` bucket, which most segments will contain, so those queries prune nothing.

The function returns `-1` when:

* No `partitionExpression` is configured, so the raw string value is used and is not numeric.
* Expression evaluation throws at runtime, for example `hex_decimal_to_long(v)` on the value `not-a-hex-value`.
* The expression returns a non-numeric result, for example `md5(v)`, which returns bytes.
* The normalized value falls outside `[0, numPartitions)`, which is possible with the `NO_OP` normalizer.

## Requirements

* The partition column must exist in the table schema.
* The partition column must be single-value.
* `numPartitions` must be greater than `0`.
* Every scalar function used in the expression must be registered in the Pinot function registry.

## Configuration

Configure the function through the standard `ColumnPartitionConfig` entry for the column.

```json theme={null}
{
  "tableName": "events_OFFLINE",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "schemaName": "events"
  },
  "indexingConfig": {
    "segmentPartitionConfig": {
      "columnPartitionMap": {
        "correlationId": {
          "functionName": "Custom",
          "numPartitions": 16,
          "functionConfig": {
            "partitionExpression": "fnv1a_hash_32_utf8(md5(toUtf8(v)))",
            "partitionIdNormalizer": "ABS"
          }
        }
      }
    }
  },
  "routing": {
    "segmentPrunerTypes": [
      "partition"
    ]
  }
}
```

### Enabling Segment Pruning

Partition metadata is consumed by two independent pruning layers, and they need different setup:

* **Server-side pruning works with no extra configuration.** `ColumnValueSegmentPruner` is part of the default server pruner list, so each server skips its own non-matching segments as soon as the request lands.
* **Broker-side pruning requires opting in.** Add `partition` to `routing.segmentPrunerTypes`, as in the example above. Without it, the broker fans the query out to every server hosting the table and pruning happens only after dispatch. With it, the broker drops non-matching segments during routing, which also reduces the number of servers contacted.

Both layers evaluate the partition function recorded in each segment's own metadata, not the function currently in the table config.

### Configuration Parameters

| Parameter | Required | Default | Description |
| - | - | - | - |
| `functionName` | Yes | — | Must be `Custom`. |
| `numPartitions` | Yes | — | Number of logical partitions for the column. Must be greater than `0`. |
| `functionConfig.partitionExpression` | No | None | Pinot scalar-function expression evaluated for the column value. Without it, every value resolves to partition `-1`. |
| `functionConfig.partitionIdNormalizer` | No | `POSITIVE_MODULO` | Name of the `PartitionIdNormalizer` used to map the raw expression result into `[0, numPartitions)`. Blank values fall back to the default. |

### Expression Rules

The partition column value is exposed to the expression as the reserved identifier `v`.

* `v` is the **only** allowed argument name. `hex_decimal_to_long(partitionValue)` and `concat(v, otherColumn)` are both rejected.
* The expression must reference `v` at least once. A constant expression such as `plus(1, 0)` is rejected.
* `v` may appear more than once, and every occurrence is bound to the same column value. For example, `fnv1a_hash_32_utf8(md5(toUtf8(concat(v, v))))` is valid.
* The expression should return a number. `Integer` and `Long` results are normalized directly; any other `Number` is converted with its long value first, so `floor(v)` on `"11.9"` produces `11`.
* The expression must be deterministic. A non-deterministic scalar function produces segment metadata that does not match the partition ID computed at query time, which silently breaks pruning.

> [!WARNING]
>
> Use the key `partitionExpression`. Other spellings, including `functionExpr`, are ignored, which leaves the function without an expression and sends every value to partition `-1`.

### Partition ID Normalizers

The normalizer maps the raw expression result to a partition ID. The default is `POSITIVE_MODULO`.

| Normalizer | Behavior | When to use |
| - | - | - |
| `POSITIVE_MODULO` | `value % numPartitions`, then adds `numPartitions` if the remainder is negative. | The default. Safe for any signed result. |
| `ABS` | Absolute value of `value % numPartitions`. | Hash functions that can return negative integers, when you want to match this normalization. |
| `MASK` | Clears the sign bit, then applies modulo. | Partition counts that are compatible with bit masking. |
| `PRE_MODULO_ABS` | `abs(value) % numPartitions`, mapping `Integer.MIN_VALUE` and `Long.MIN_VALUE` to `0`. | Matching the legacy semantics of `HashCodePartitionFunction` and `ByteArrayPartitionFunction`. |
| `NO_OP` | Uses the expression result as the partition ID unchanged. | The expression already returns a valid ID in `[0, numPartitions)`. Out-of-range results become `-1`. |

Normalizer names are case-insensitive.

## Examples

### Hash a String ID

Convert a string column to UTF-8 bytes, hash it with `md5`, then fold the hash into an integer. This is the common shape for pruning on a high-cardinality string ID.

```json theme={null}
{
  "correlationId": {
    "functionName": "Custom",
    "numPartitions": 16,
    "functionConfig": {
      "partitionExpression": "fnv1a_hash_32_utf8(md5(toUtf8(v)))",
      "partitionIdNormalizer": "ABS"
    }
  }
}
```

Queries with an equality predicate on the column can then use the partition metadata for pruning:

```sql theme={null}
SELECT count(*)
FROM events
WHERE correlationId = 'correlation-123';
```

### Convert a Hex String to a Partition ID

Decode a hexadecimal string into a long and let the default `POSITIVE_MODULO` normalizer map it into the partition count.

```json theme={null}
{
  "hexUserId": {
    "functionName": "Custom",
    "numPartitions": 128,
    "functionConfig": {
      "partitionExpression": "hex_decimal_to_long(v)"
    }
  }
}
```

With `numPartitions` set to `128`:

| Value | Raw result | Partition |
| - | - | - |
| `ff` | `255` | `127` |
| `80` | `128` | `0` |
| `11` | `17` | `17` |
| `not-a-hex-value` | Evaluation failure | `-1` |

### Use an Expression Result Directly

`NO_OP` skips normalization. Use it only when the expression already returns a valid partition ID.

```json theme={null}
{
  "partitionIdText": {
    "functionName": "Custom",
    "numPartitions": 4,
    "functionConfig": {
      "partitionExpression": "hex_decimal_to_long(v)",
      "partitionIdNormalizer": "NO_OP"
    }
  }
}
```

With `numPartitions` set to `4`:

| Value | Raw result | Partition |
| - | - | - |
| `0` | `0` | `0` |
| `3` | `3` | `3` |
| `63` | `99` | `-1` (out of range) |

### Reference the Value More Than Once

Repeat `v` when the expression needs the same value in several positions. No column name is configured anywhere in the expression.

```json theme={null}
{
  "userId": {
    "functionName": "Custom",
    "numPartitions": 32,
    "functionConfig": {
      "partitionExpression": "fnv1a_hash_32_utf8(md5(toUtf8(concat(v, v))))",
      "partitionIdNormalizer": "ABS"
    }
  }
}
```

## Invalid Configurations

| Expression | Result | Reason |
| - | - | - |
| `hex_decimal_to_long(partitionValue)` | Rejected | `partitionValue` is not the reserved identifier `v`. |
| `concat(v, otherColumn)` | Rejected | `otherColumn` is a second argument name. |
| `plus(1, 0)` | Rejected | The expression never references `v`. |
| `md5(v)` | Partition `-1` | Accepted by the evaluator, but returns bytes rather than a number. |
| `v` | Partition `-1` | Returns the raw string value, which is not numeric. |

Rejected expressions throw when the partition function is first constructed. Table config validation does not build the partition function, so the failure surfaces during segment build or query-time pruning rather than when you apply the config. Validate the expression on sample values before pushing it to a production table.

## Operational Considerations

1. **Changing the config does not rewrite existing segments.** Partition metadata is written at segment build, and both pruners evaluate the function recorded in each segment's own metadata rather than the current table config. Old segments therefore keep pruning correctly against the expression they were built with, and a config change does not produce wrong results. What you are left with is a table whose segments are partitioned inconsistently, which matters wherever partitioning also drives segment-to-server assignment — upsert, dedup, and `strictReplicaGroup` routing. StarTree Cloud rejects a `numPartitions` change on an already-partitioned column of an upsert or dedup table for exactly this reason. Treat these fields as fixed for the life of the table, and rebuild segments if you have to change them.
2. **Pruning applies to equality predicates.** Both layers prune on `=` against the partition column, and the broker pruner also handles `IN`. Range predicates, `LIKE`, and expressions wrapped around the column in the `WHERE` clause do not benefit.
3. **Watch for `-1` in segment metadata.** If a column's recorded partitions include `-1`, the expression is missing, failing at runtime, or returning a non-numeric result for some values. Check the segment metadata after the first segments are built rather than waiting to notice that pruning never engaged.
4. **Prefer a built-in function when one fits.** `Murmur`, `Murmur3`, `Modulo`, `HashCode`, `Fnv`, and `ByteArray` avoid per-value expression evaluation. Reach for `Custom` only when none of them produce the partition ID you need.


# Deep Store Segment Sync
Source: https://docs.startree.ai/corecapabilities/manage-data/deep-store-sync



When a user makes changes to a Pinot table configuration or schema and those updates require a segment rebuild, the servers reload the local segments and keep them up to date with the table configuration and schema.

But the deep store segment is not included in the sync and this causes an increase in startup time because a server needs to download the segment from the deep store and rebuild it again locally. To avoid this, the [Alter Table Task](/corecapabilities/manage-data/alter-table-task) has been enhanced to offload the segment reload to minions and to keep the deep store in sync with the latest table configuration and schema, reducing server load.

## How does deep store segment sync work?

When the alter table task is used in the `reloadOnly` mode.

1. The process fetches stale segments on the servers for the table.
2. Generates a minion task that rebuilds the segment with the latest table configuration and schema. Then uploads the segment to the deep store with the same name.
3. This triggers a segment refresh message. After that, all servers hosting the segment will download the segment and load it locally.
4. After the reload through the alter table task, the segment metadata will include a table hash value. This value is used by subsequent task generations to skip these segments.

## Configuration

Add the following configuration parameters to the alter table task:

```json theme={null}
"task": {
  "taskTypeConfigsMap": {
    "StarTreeAlterTableTask": {
      "reloadOnly": "true",
      "forceReload": "true"
    }
  }
}
```

> [!WARNING]
>
> Experimental feature, use with caution.

**reloadOnly**

Set this flag to **true** to use the alter table task for deep store sync. 

General recommendations

* Make sure there is no concurrent server reload operation as that would lead to ATT skipping those segments from reload on deep store.
* Set **skipSegmentPreprocess** on the table so that segments are not reloaded on server restart.
* Run this task periodically to reduce the probability of server side reload through user triggers.
* reloadOnly takes precedence if forceReload is also configured.

**forceReload**

This parameter uses the table hash value added by the alter table task in the reloaded segments. The process does not check for segment metadata on servers. Use this flag for the first iteration of deep store sync to bootstrap all the historical segments and to add the table hash. This ensures that all the existing segments are in sync with the table configuration. After the first iteration, this parameter can be removed. Please note that reloadOnly takes precedence if both reloadOnly and forceReload are also configured.

## Scenarios

The table shows the state of deep store sync and the effect of the configuration parameters for different scenarios:

| ***Scenario*** | | | | *Action by the Alter Table Task on the Segment* | |
| :- | :- | :-: | :-: | :-: | :-: |
| | **Case** | **Deep Store** | **Local Server** | `reloadOnly` | `reloadOnly`**with** `forceReload` |
| **Existing segment before task introduction** | Table not updated since segment creation |  *(In Sync)* | |  *(Not Refreshed)* |  *(Refreshed Once)* |
| | Table updated and segment reloaded on server |  *(Not In Sync)* | | | |
| | Table updated but segment not reloaded on server | | | | |
| **Segment generated post task introduction** | Table not updated or segment is already processed by task | | | | |
| | Table updated and segment reloaded on server | | | | |
| | Table updated and task is yet to run | | | | |

## Limitations

* In the background, the servers still load segments but the loading process is faster.
* The initial run will involve a process overhead to reload all segments. Consecutive runs will be incremental.
* Server-side reloads are not automatically blocked. Maintain a frequent cron on the task to make sure segments are reloaded by the alter table task and to reduce the probability of server side reload through user triggers.


# Hybrid Tables
Source: https://docs.startree.ai/corecapabilities/manage-data/hybrid-tables



## Overview

Hybrid tables in StarTree Cloud combine the benefits of both real-time and offline ingestion in a single logical table. This powerful configuration allows you to query across both streaming and batch data seamlessly, without having to specify which data source you're accessing.

![Hybridtables Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/images/hybridtables.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=d70fcbf95c9c0e36506052c03d8f1899)

## How Hybrid Tables Work

A hybrid table consists of two physical tables that share the same name:

* A real-time table ingesting data from streaming sources (e.g., Kafka)
* An offline table containing historical data loaded from batch sources

The query broker intelligently routes queries to the appropriate segments based on time boundaries, providing a unified view of your data. When an offline segment is pushed to cover a time period that overlaps with real-time data, the broker automatically prioritizes the offline segments for that period.

## Key Benefits

* **Complete Data View**: Access both real-time and historical data through a single table
* **Optimized Storage**: Keep long-term historical data in offline segments while maintaining a shorter retention for real-time data
* **Data Correction**: Replace real-time data with corrected/deduplicated offline data as it becomes available
* **Seamless Querying**: Users query a single table without needing to understand the underlying table types

## Common Use Cases

* Daily ETL processes that push cleaned, deduplicated data to offline segments while continuously ingesting real-time data
* Maintaining years of historical data in offline segments while keeping only recent data in real-time segments
* Providing immediate visibility into streaming data while ensuring consistency with batch-processed data

## Configuration

Hybrid tables must be configured using Controller APIs. A typical configuration involves:

1. Creating both real-time and offline table configurations
2. Setting appropriate retention periods for each (longer for offline, shorter for real-time)
3. Configuring time boundaries to manage query routing

## Managed Offline Flow

StarTree Cloud offers a "Managed Offline Flow" that can automatically move data from real-time to offline segments, read more about it [here](/corecapabilities/manage-data/segment-import-task).

> [!NOTE]
>
> Hybrid tables configuration requires using Controller APIs as this setup is not yet available through the Data Portal interface. For detailed configuration instructions and examples, refer [here](https://docs.pinot.apache.org/basics/concepts/components/table)


# Bloom Filter Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/bloom-filter

The Bloom Filter index enables fast filtering of high-cardinality columns by reducing unnecessary data scans.

## Overview and Purpose

A Bloom filter index is a space-efficient probabilistic data structure that enables fast segment pruning for equality-based queries. When applied to a column, it allows the system to quickly determine if a segment does not contain any records matching an equality (=) or IN predicate, thereby skipping unnecessary segment processing.

In StarTree Cloud (powered by Apache Pinot), Bloom filter indexes are particularly valuable for:

* Accelerating point queries that filter on high-cardinality columns
* Improving performance of queries with equality conditions
* Reducing processing overhead for IN clause queries (with up to 10 values)
* Optimizing segment pruning in distributed query execution
* Any query pattern where quickly eliminating irrelevant segments is beneficial

> [!NOTE]
>
> Bloom filters are designed to provide a definitive "no" but only a probable "yes" when checking for value existence. They never produce false negatives (missing values that exist) but may produce false positives (indicating values might exist when they don't).

## How the Index Works

### Core Concepts

Traditional query processing must examine every segment that might contain matching records, even if many segments ultimately contain no matches. This approach leads to unnecessary I/O and computation.

The Bloom Filter index provides:

1. **Probabilistic Hash Structure**: During indexing, each unique value in the column is hashed and added to a bit array.
2. **Segment-Level Filtering**: One Bloom filter is created per segment for each configured column.
3. **Fast Negative Answers**: When executing a query, the system checks the Bloom filter first to determine if a segment cannot possibly contain the requested value.
4. **Mathematical Trade-offs**: The filter's size, false positive rate, and the column's cardinality are mathematically related, allowing for tunable accuracy versus memory trade-offs.

### Example Illustration

For a query filtering on `playerId = 12345`:

1. **Without Bloom Filter Index**: The system must scan every segment, reading and checking values to find matches.
2. **With Bloom Filter Index**: Before scanning each segment:
   * The system checks the segment for `playerId` value 12345
   * If the filter returns "definitely not present," the entire segment is skipped
   * If the filter returns "possibly present," the segment is scanned normally
   * This eliminates unnecessary work for segments that don't contain the value

For a table with 100 segments where only 5 segments contain records with `playerId = 12345`, the Bloom filter could potentially eliminate processing for 95 segments, dramatically reducing query time.

## Configuration

### Enabling Bloom Filter Index

To enable a Bloom filter index on a column in your StarTree Cloud table, add the following configuration to your table definition:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "playerId",
      "indexes": {
        "bloom": {}
      }
    }
  ]
}
```

This enables a Bloom filter index with default settings.

### Customizing Bloom Filter Index Parameters

To customize the Bloom filter index configuration:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "playerId",
      "indexes": {
        "bloom": {
          "fpp": 0.01,
          "maxSizeInBytes": 1000000,
          "loadOnHeap": true
        }
      }
    }
  ]
}
```

### Alternative Configuration (Legacy Method)

You can also use the older configuration format with default settings:

```json theme={null}
{
  "tableIndexConfig": {
    "bloomFilterColumns": [
      "playerId"
    ]
  }
}
```

Or with customized parameters:

```json theme={null}
{
  "tableIndexConfig": {
    "bloomFilterConfigs": {
      "playerId": {
        "fpp": 0.01,
        "maxSizeInBytes": 1000000,
        "loadOnHeap": true
      }
    }
  }
}
```

### Configuration Parameters

1. **fpp**: False positive probability of the Bloom filter
   * Default: 0.05 (5%)
   * Range: 0 to 1
   * Impact: Lower values increase accuracy but also increase memory usage
   * Recommended values: 0.01 to 0.1 depending on accuracy requirements
2. **maxSizeInBytes**: Maximum size of the Bloom filter
   * Default: 0 (unlimited)
   * Impact: Limits memory usage by potentially increasing false positive rate
   * Takes precedence over fpp if both are specified
3. **loadOnHeap**: Memory allocation type
   * Default: false (use off-heap memory)
   * When true: Use Java heap memory
   * When false: Use off-heap memory
   * Impact: Affects memory management and garbage collection
4. **disabled**: Explicitly disable the Bloom filter
   * Default: false
   * When true: Disables the Bloom filter even if other parameters are specified

## Performance Considerations

1. **False Positive Rate vs Memory Usage**: Lower fpp values increase accuracy but require more memory. Choose a value that balances your accuracy needs with memory constraints.
2. **Column Characteristics**: Bloom filters are most effective for columns with:
   * High cardinality (many unique values)
   * Selective equality predicates (filtering for specific values)
   * Values distributed across segments (not all values in all segments)
3. **Query Patterns**: Bloom filters accelerate queries with:
   * Equality conditions (`column = value`)
   * IN predicates with up to 10 values (`column IN (value1, value2, ...)`)
   * They do not help with range queries, LIKE, or other non-equality filters
4. **Segment Distribution**: The benefit increases with the number of segments that can be pruned. If all segments contain all values, the benefit will be minimal.
5. **Memory Management**: Use loadOnHeap=true with caution, as it impacts Java garbage collection. For most cases, the default off-heap setting is recommended.


# Composite JSON Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/composite-json-index

Use the Composite JSON index to accelerate complex JSON queries by enabling efficient filtering and retrieval of nested fields.

## Overview

The Composite JSON index is an enhanced version of the [JSON index](/corecapabilities/manage-data/indexes/json-index). It indexes flattened JSON records and can add specialized per-path indexes for common `JSON_MATCH` predicates.

> [!NOTE]
>
> The Composite JSON index is available from StarTree version 0.11.0. Compact doc ID mapping and the related `JSON_MATCH` query-path performance improvements are available after the StarTree 0.15 release.

Use the Composite JSON index when you need to query nested JSON paths with one or more of the following patterns:

* Equality, `IN`, `NOT IN`, `IS NULL`, or `IS NOT NULL` filters on selected JSON paths.
* Range filters on numeric JSON fields.
* Text search with `TEXT_MATCH` on selected JSON paths.
* Regex or prefix-style filters with `REGEXP_LIKE` on selected paths accelerated by an FST index.
* Count filters over the number of matching flattened records in a JSON document.

The Composite JSON index can also reduce index size by limiting the paths included in the inverted index, disabling array-position indexing, and, for eligible array-free columns, storing a compact doc ID mapping.

> [!WARNING]
>
> Creating both a JSON index and a Composite JSON index on the same column is not recommended. When evaluating `JSON_MATCH`, Pinot checks for the regular JSON index first and ignores the Composite JSON index.

## Configure the Index

Add the Composite JSON index under the column's `fieldConfigList[].indexes` entry.

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "json",
      "encodingType": "RAW",
      "indexes": {
        "compositeJson": {
          "flushThreshold": 10000,
          "enablePositionalIndexing": true,
          "invertedIndexConfigs": [
            {
              "path": "$.id"
            }
          ],
          "rangeIndexConfigs": [
            {
              "path": "$[*].timestamp",
              "name": "eventTimestamp",
              "dataType": "LONG",
              "createDictionary": "true",
              "dictionaryType": "fixedLength",
              "defaultValue": "0"
            }
          ],
          "textIndexConfigs": [
            {
              "path": "$[*].message",
              "name": "messageText"
            }
          ],
          "fstIndexConfigs": [
            {
              "path": "$.email",
              "name": "emailFst"
            }
          ]
        }
      }
    }
  ]
}
```

To index every path in the inverted index, use `includeAllPaths`:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "json",
      "encodingType": "RAW",
      "indexes": {
        "compositeJson": {
          "invertedIndexConfigs": [
            {
              "includeAllPaths": "true"
            }
          ]
        }
      }
    }
  ]
}
```

## Configuration Parameters

The Composite JSON index supports the JSON index options `maxLevels`, `excludeArray`, `disableCrossArrayUnnest`, `includePaths`, `excludePaths`, `excludeFields`, `indexPaths`, `maxValueLength`, and `skipInvalidJson`.

It also supports these Composite JSON options:

| Parameter | Default | Description |
| - | - | - |
| `flushThreshold` | unset | Flushes off-heap posting lists every N documents to limit temporary memory usage during index creation. |
| `enablePositionalIndexing` | `true` | Includes array positions such as `$[0]` in the inverted index. Set to `false` to reduce index size when queries only use wildcard paths such as `$[*]`. |
| `invertedIndexConfigs` | empty | Lists paths to add to the inverted index. Use objects with `path`, or one object with `includeAllPaths: "true"` to include all paths. |
| `invertedIndexPartitionCount` | `1` | Splits the inverted index into multiple path partitions. Partitioning can reduce large index overhead, but it affects compact doc ID mapping eligibility and FST availability. |
| `rangeIndexConfigs` | empty | Lists paths to index with an internal range index. |
| `textIndexConfigs` | empty | Lists paths to index with an internal Lucene text index. |
| `fstIndexConfigs` | empty | Lists paths to index with an FST for faster `REGEXP_LIKE` filters. |
| `enableCompactDocIdMapping` | `false` | Allows eligible new immutable segments to use a compact doc ID mapping instead of the legacy per-document `int[]` mapping. Enable only after all nodes that read the index support the new format. |

## Range Indexes

Use `rangeIndexConfigs` for JSON paths that appear in range predicates.

```json theme={null}
{
  "path": "$[*].timestamp",
  "name": "eventTimestamp",
  "dataType": "LONG",
  "createDictionary": "true",
  "dictionaryType": "fixedLength",
  "defaultValue": "0"
}
```

Range index fields:

| Field | Required | Description |
| - | - | - |
| `path` | yes | JSON path to index. |
| `name` | yes | Unique index name. Use only letters, digits, and underscores. |
| `dataType` | yes | JSON field type. With `createDictionary: "true"`, supported types are `INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`, and `BIG_DECIMAL`. Without a dictionary, supported types are `INT`, `LONG`, `FLOAT`, and `DOUBLE`. |
| `createDictionary` | no | Set to `"true"` to build a dictionary-encoded range index. Default is `"false"`. |
| `dictionaryType` | no | `fixedLength` or `variableLength`. Default is `variableLength`. `BIG_DECIMAL` supports only `variableLength`. |
| `defaultValue` | no | Value used for flattened records that do not contain the field or contain an unparseable value. |

The range index has one value for each flattened record. If a value is missing or unparseable, Pinot uses the configured `defaultValue`, or the type default when `defaultValue` is omitted. This can affect unbounded range queries. For example, if the default value is `0`, this query can include documents without `$.value`:

```sql theme={null}
SELECT count(*)
FROM users
WHERE JSON_MATCH(json, '"$.value" < 3');
```

## Text Indexes

Use `textIndexConfigs` for JSON paths queried with `TEXT_MATCH`.

```json theme={null}
{
  "path": "$[*].message",
  "name": "messageText"
}
```

The `name` must be unique. Text index configs also accept the same fields supported by a regular text index, including:

```text theme={null}
rawValue, queryCache, useANDForMultiTermQueries, stopWordsInclude, stopWordsExclude,
luceneUseCompoundFile, luceneMaxBufferSizeMB, luceneAnalyzerClass,
luceneAnalyzerClassArgs, luceneAnalyzerClassArgTypes, luceneQueryParserClass,
enablePrefixSuffixMatchingInPhraseQueries, reuseMutableIndex,
luceneNRTCachingDirectoryMaxBufferSizeMB
```

For more information, see the [text index documentation](/corecapabilities/manage-data/indexes/text-index).

## FST Indexes

> [!NOTE]
>
> FST indexes within Composite JSON were added in StarTree 0.12.

Use `fstIndexConfigs` to accelerate `REGEXP_LIKE` predicates on selected JSON paths.

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "json",
      "encodingType": "RAW",
      "indexes": {
        "compositeJson": {
          "invertedIndexConfigs": [
            {
              "path": "$.email"
            }
          ],
          "fstIndexConfigs": [
            {
              "path": "$.email",
              "name": "emailFst"
            }
          ]
        }
      }
    }
  ]
}
```

FST indexes require the path to also be available in the Composite JSON inverted index. FST acceleration is not used when `invertedIndexPartitionCount` is greater than `1`; queries fall back to dictionary scanning.

## Compact Doc ID Mapping

> [!NOTE]
>
> Compact doc ID mapping is available after the StarTree 0.15 release.

Composite JSON indexes store matches over flattened JSON records and then translate them back to table document IDs. Historically, each immutable segment stored a full `(numDocs + 1) * 4` byte `int[]` mapping for this translation, even when the JSON column was array-free and each document produced at most one flattened record.

For array-free columns, the mapping can dominate index size and query latency. On an 11.3M-row segment with about 8% empty JSON documents, the full mapping was about 45 MB. With compact doc ID mapping, the same shape can store a small null-docs bitmap instead, around 125 KB in that measured case.

Enable compact mapping with `enableCompactDocIdMapping`:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "tagsJSON",
      "encodingType": "RAW",
      "indexes": {
        "compositeJson": {
          "enableCompactDocIdMapping": true,
          "invertedIndexConfigs": [
            {
              "path": "$.dir"
            }
          ]
        }
      }
    }
  ]
}
```

> [!WARNING]
>
> `enableCompactDocIdMapping` writes a newer Composite JSON on-disk format for eligible segments. Keep it disabled during rolling upgrades. Enable it only after every server, peer-download path, and validation path that may read the index has been upgraded to a version that supports compact Composite JSON doc ID mappings.

### Eligibility

The flag allows compact mapping, but the index creator still chooses the smallest safe representation for each new immutable segment.

| Segment shape with `enableCompactDocIdMapping: true` | Mapping written |
| - | - |
| Array-free, no empty JSON documents | No mapping. Flattened IDs already equal real document IDs. |
| Array-free, has empty documents, no range or text index, and `invertedIndexPartitionCount: 1` | Compact null-docs bitmap. Inverted-index postings are relabeled to real document IDs at segment seal time. |
| Array-free, has empty documents, and has a range index, text index, or `invertedIndexPartitionCount > 1` | Legacy full `int[]` mapping. |
| Contains array values | Legacy full `int[]` mapping. |
| Empty segment | Legacy full `int[]` mapping. |

With `enableCompactDocIdMapping` omitted or set to `false`, all new segments keep the legacy full `int[]` mapping.

### Rollout and Rollback

`enableCompactDocIdMapping` is read only while creating new immutable segments. Changing the setting does not rewrite existing segments by itself.

Use this rollout order:

1. Upgrade every component that may read Composite JSON indexes.
2. Set `enableCompactDocIdMapping: true` on the target column.
3. Rebuild, refresh, or re-commit the affected segments so new segments are written with the compact representation when eligible.

To roll back to a version that does not support compact mappings:

1. Set `enableCompactDocIdMapping: false`.
2. Refresh or re-commit any affected compact segments so they are written with the legacy mapping.
3. Downgrade only after no compact-mapping segments remain.

## Query Examples

### Equality

```sql theme={null}
SELECT count(*)
FROM users
WHERE JSON_MATCH(json, '"$.dir" = ''upstream''');
```

### Range Query

```sql theme={null}
SELECT count(*)
FROM users
WHERE JSON_MATCH(json, '"$[*].timestamp" BETWEEN 1577836800000 AND 1735689599999');
```

### Text Search

```sql theme={null}
SELECT count(*)
FROM users
WHERE JSON_MATCH(json, 'TEXT_MATCH("$[*].message", ''timeout OR exception'')');
```

### Regex Search with FST

```sql theme={null}
SELECT count(*)
FROM users
WHERE JSON_MATCH(json, 'REGEXP_LIKE("$.email", ''.*@example[.]com'')');
```

### Count Filter

Use the optional third argument to filter by the number of matching flattened records per document.

```sql theme={null}
SELECT count(*)
FROM users
WHERE JSON_MATCH(json, '"$[*].id" = 10', '"cnt" >= 5');
```

For `IS NULL`, a count filter can distinguish empty JSON documents from present documents where the path is absent:

```sql theme={null}
SELECT count(*)
FROM users
WHERE JSON_MATCH(json, '"$.dir" IS NULL', '"cnt" >= 1');
```

## Array Position Indexing

Queries that specify a concrete array position require `enablePositionalIndexing: true`, even when the same path is also range-indexed.

```sql theme={null}
SELECT count(*)
FROM users
WHERE JSON_MATCH(json, '"$.grades[1].value" BETWEEN 1 AND 10');
```

When `enablePositionalIndexing` is `false`, use wildcard paths such as `$.grades[*].value` instead.

## Range Queries on Real-Time Tables

For committed immutable segments, range predicates can use the internal Composite JSON range index. Mutable consuming segments rely on the inverted index path.

When you use Composite JSON range predicates on a real-time or hybrid table, include the same JSON path in `invertedIndexConfigs`. Otherwise, mutable consuming segments can return no rows for that predicate until they commit.

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "json",
      "encodingType": "RAW",
      "indexes": {
        "compositeJson": {
          "invertedIndexConfigs": [
            {
              "path": "$.grades[*].value"
            }
          ],
          "rangeIndexConfigs": [
            {
              "path": "$.grades[*].value",
              "name": "gradeValue",
              "dataType": "LONG",
              "createDictionary": "true",
              "dictionaryType": "fixedLength",
              "defaultValue": "0"
            }
          ]
        }
      }
    }
  ]
}
```

## Performance Notes

The following comparisons were measured on internal Composite JSON benchmarks for the improvements available after the StarTree 0.15 release.

| Scenario | Before | After | Impact |
| - | -: | -: | - |
| Array-free, inverted-index-only segment with compact mapping enabled: non-selective equality `JSON_MATCH` on 5M documents | 93.2 ms | 0.21 ms | About 450x faster |
| Same shape: Composite JSON index size | 25.7 MB | 9.6 MB | About 63% smaller |
| Same shape: doc ID mapping size | 19 MB | 0.6 MB | About 97% smaller |
| Real 11.3M-row `tagsJSON` segment: doc ID mapping size | 45 MB | 125 KB | About 99.7% smaller |
| Identity-mapped segment: reader-level `getMatchingDocIds` on a 359K-match posting | 0.052 ms | 0.003 ms | About 17x faster |
| Legacy full-mapping segment: dense non-selective remap on 2M rows | 33 ms | 11 ms | About 3x faster |
| Legacy full-mapping segment: sparse selective remap on 2M rows | 1.4 ms | 0.9 ms | About 36% faster |

Recent Composite JSON query-path improvements also avoid cloning read-only postings for compact identity segments and speed up the legacy remap path for array and full-mapping segments. Those improvements require no table-config changes.


# Dictionary Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/dictionary-index

The Dictionary index enhances storage efficiency and query performance by encoding repetitive column values into compact dictionary IDs, making it ideal for columns with moderate cardinality and supporting other indexes like inverted and FST indexes.

## Overview and Purpose

A dictionary index is a fundamental indexing structure that improves storage efficiency and query performance by encoding repetitive data. It replaces actual column values with compact dictionary IDs, which is particularly beneficial for columns with a moderate number of unique values repeated across many rows.

In StarTree Cloud (powered by Apache Pinot), dictionary indexes serve as both an encoding mechanism and an indexing structure, enabling:

* Efficient storage of repetitive data through value deduplication
* Faster query processing through compact integer representations
* Reduced memory footprint for columns with moderate cardinality
* Optimized performance for string and other variable-length data types
* Support for other index types that rely on dictionary encoding

> [!NOTE]
>
> Dictionary indexes are enabled by default in StarTree Cloud for most columns, as they typically provide storage and performance benefits. However, they may be less efficient for columns with very high cardinality (where unique values approach the number of rows).

## How the Index Works

### Core Concepts

When data contains repetitive values across many rows, storing each value individually wastes space and reduces query efficiency. Dictionary encoding addresses this challenge through value deduplication.

The dictionary index in StarTree Cloud:

1. Creates a **value mapping** table (dictionary) that assigns a unique integer ID to each unique value in the column.
2. Performs **ID substitution**, replacing actual values in the data with their corresponding dictionary IDs, which are typically much smaller.
3. Offers **bidirectional lookup** by maintaining structures for both:
   * Forward lookup: ID → Value (for retrieving original values)
   * Reverse lookup: Value → ID (for encoding and filtering)
4. Serves a **dual purpose**, by functioning as both a storage optimization technique and an indexing structure that enables efficient value lookups.

### Example Illustration

For a column storing countries with many repeated values:

1. **Original Data**:

   ```
   Row 1: "United States"
   Row 2: "Canada"
   Row 3: "United States"
   Row 4: "United Kingdom"
   Row 5: "Canada"
   ...
   ```
2. **Dictionary Approach**:
   * Create a dictionary mapping unique values to IDs:

     ```
     ID 0: "United States"
     ID 1: "Canada"
     ID 2: "United Kingdom"
     ...
     ```
   * Store the column using the more compact IDs:

     ```
     Row 1: 0
     Row 2: 1
     Row 3: 0
     Row 4: 2
     Row 5: 1
     ...
     ```

This approach significantly reduces storage requirements, especially for strings and other variable-length data types, while enabling efficient lookups and transformations.

## Relationship with Other Indexes

Dictionary indexes influence the behavior and implementation of many other index types:

| Index Type | Relationship | Description |
| :- | :- | :- |
| Forward Index | Conditional | Implementation depends on whether dictionary is enabled |
| Range Index | Conditional | Implementation depends on whether dictionary is enabled |
| Inverted Index | Required | Requires dictionary index to be enabled |
| JSON Index | May disable | When optimizeDictionary is enabled, dictionary may be disabled |
| Text Index | May disable | When optimizeDictionary is enabled, dictionary may be disabled |
| FST Index | Required | Requires dictionary index to be enabled |
| H3/Geospatial Index | Incompatible | Cannot use dictionary index |

## Configuration

### Default Behavior

Dictionary indexes are enabled by default for all columns, based on the assumption that the number of unique values will be significantly lower than the number of rows.

### Explicitly Disabling Dictionary Index

To disable a dictionary for a column with high cardinality, use one of these approaches:

**Method 1**: Using the `dictionary` property in `indexes`:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "highCardinalityColumn",
      "indexes": {
        "dictionary": {
          "disabled": true
        }
      }
    }
  ]
}
```

**Method 2**: Setting `encodingType` to `RAW`:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "highCardinalityColumn",
      "encodingType": "RAW"
    }
  ]
}
```

**Method 3**: Using the `noDictionaryColumns` table configuration:

```
  "tableIndexConfig": {
    "noDictionaryColumns": ["column1", "column2"]
  },
```

### Heuristic Dictionary Configuration

For columns where the decision to use a dictionary isn't clear, you can enable heuristic-based dictionaries at the table level:

```json theme={null}
{
  "tableIndexConfig": {
    "optimizeDictionary": true,
    "noDictionarySizeRatioThreshold": 0.85
  }
}
```

### Heuristic Parameters

1. **optimizeDictionary**: When set to true, enables heuristic dictionary creation for all eligible columns.
   * Default: false
2. **optimizeDictionaryForMetrics**: When set to true, enables heuristic dictionary creation only for metric columns.
   * Default: false
   * Note: optimizeDictionary takes precedence if both are specified
3. **noDictionarySizeRatioThreshold**: The ratio threshold used to determine if a dictionary should be created.
   * Default: 0.85
   * Meaning: If the ratio of (dictionary-encoded size)/(raw-encoded size) is lower than this threshold, the dictionary is created

### Advanced Configuration Parameters

You can use advanced configuration parameters for fine-tuning the dictionary behavior.

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "productName",
      "encodingType": "DICTIONARY",
      "indexes": {
        "dictionary": {
          "onHeap": false,
          "useVarLengthDictionary": false,
          "intern": {
            "capacity": 32000
          }
        }
      }
    }
  ]
}
```

1. **onHeap**: Determines whether the dictionary is stored in Java heap memory.
   * Default: false (off-heap)
   * When true: Dictionary is loaded into Java heap memory
   * When false: Dictionary is kept in off-heap memory (recommended)
   * Impact: May slightly improve performance, but increases heap usage and GC pressure
2. **useVarLengthDictionary**: Controls storage approach for variable-length data (strings, bytes, etc.).
   * Default: false
   * When false: All values are padded to the length of the longest value
   * When true: Values are stored with their actual length plus an offset
   * Recommended: Set to true when the length of values varies significantly
3. **intern**: Controls string interning for on-heap dictionaries.
   * Only applies to string and byte columns with onHeap=true
   * Purpose: Reduces memory usage when the same values appear across segments
4. **intern.capacity**: Maximum number of distinct values to intern.
   * Default: null (no interning)
   * Recommended: Set based on the estimated unique values across segments

## Performance Considerations

1. **Cardinality Trade-offs**:
   * Low cardinality (few unique values): Dictionary provides significant benefits
   * High cardinality (many unique values): Raw encoding may be more efficient
   * Rule of thumb: If the ratio of unique values is greater than 30% of total rows, consider disabling dictionary
2. **Variable Length Data**:
   * For string columns with widely varying lengths, set useVarLengthDictionary to true
   * For string columns with similar lengths, keep the default false for better performance
3. **Memory Management**:
   * Keep dictionaries off-heap (default) in most cases
   * Use on-heap dictionaries with caution, only for low-cardinality columns
   * When using on-heap dictionaries, consider enabling interning to reduce the memory footprint
4. **Segment Considerations**:
   * Dictionary scope is per segment
   * Very large tables may have many segments, each with its own dictionary
   * This can impact memory usage, especially with on-heap dictionaries


# Forward Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/forward-index

The Forward index maps document IDs to column values, enabling efficient data retrieval for queries involving selection, aggregation, and sorting. There are three implementations: dictionary-encoded, sorted, and raw value, optimized for different data characteristics.

## Overview and Purpose

A forward index is the fundamental data structure that maps document IDs (row indices) to their actual column values in StarTree Cloud. It serves as the primary mechanism for retrieving column values when processing queries that select, aggregate, or sort data.

Unlike specialized indexes that accelerate specific query patterns, the forward index is the default storage mechanism that enables basic data access. It maintains a direct mapping from each row to its corresponding values, allowing the system to efficiently retrieve data for any query operation.

Forward indexes are particularly important for:

* SELECT clause operations that retrieve actual values
* Aggregation functions that compute over column values
* GROUP BY and ORDER BY operations that organize results
* Any query pattern that requires access to the raw data values

> [!NOTE]
>
> Forward indexes are enabled by default for all columns. They can be selectively disabled in specific scenarios to save storage space when other indexes sufficiently support your query patterns.

## How the Index Works

### Core Concepts

At its core, a forward index implements a straightforward mapping:

```
Document ID (Row Number) → Column Value
```

StarTree Cloud offers multiple implementations of forward indexes, optimized for different data characteristics:

1. **Dictionary-Encoded Forward Index**: Maps row IDs to dictionary IDs rather than actual values, providing space efficiency for columns with low to medium cardinality.
2. **Sorted Forward Index**: Uses run-length encoding for sorted columns, storing only the start and end document IDs for each unique value, offering significant compression benefits.
3. **Raw Value Forward Index**: Stores actual values directly, eliminating dictionary lookup overhead and improving performance for high-cardinality columns.

The implementation choice affects both storage requirements and query performance, with each approach offering different trade-offs.

### Example Illustration

Consider a column storing product categories:

**Dictionary-Encoded Approach**:

1. Create a dictionary mapping unique values to IDs:

   ```
   "Electronics" → 1
   "Clothing" → 2
   "Home" → 3
   "Beauty" → 4
   ```
2. Store these IDs in the forward index instead of the actual values:

   ```
   Row 1 → 1 (representing "Electronics")
   Row 2 → 2 (representing "Clothing")
   Row 3 → 1 (representing "Electronics")
   Row 4 → 3 (representing "Home")
   ```

**Sorted Forward Index Approach** (if the column is sorted): Instead of storing an ID for each row, store ranges:

```
"Clothing" → Rows 1-2
"Electronics" → Rows 3-5
"Home" → Rows 6-8
```

**Raw Value Approach**: Store the actual values directly:

```
Row 1 → "Electronics"
Row 2 → "Clothing"
Row 3 → "Electronics"
Row 4 → "Home"
```

## Types of Forward Indexes

### Dictionary-Encoded Forward Index (Default)

The dictionary-encoded forward index uses bit compression to efficiently store dictionary IDs instead of actual values.

**When to Use:**

* For columns with low to medium cardinality (few unique values)
* When compression is important to reduce storage requirements
* For columns that may benefit from inverted indexes

**Benefits:**

* Significant space savings for columns with duplicate values
* Enables efficient inverted index creation
* Supports compression techniques for further space optimization

**Trade-offs:**

* Adds one level of indirection (dictionary lookup) when retrieving values
* Less efficient for columns with high cardinality
* For string columns, may add padding overhead in the dictionary

### Sorted Forward Index

When a column is physically sorted, StarTree Cloud can use a specialized sorted forward index that combines run-length encoding with dictionary encoding.

**When to Use:**

* For columns that are used as a sort key for the segment
* When the column has an enabled dictionary
* For columns frequently used in range filters and sorting operations

**Benefits:**

* Maximum compression efficiency
* Improved data locality (related values are stored together)
* Automatically serves as both a forward and inverted index
* Enables efficient binary search for range queries

**Requirements:**

* The segment must be sorted by the column
* Dictionary encoding must be enabled for the column

### Raw Value Forward Index

The raw value forward index stores actual values directly instead of dictionary IDs, eliminating dictionary lookup overhead.

**When to Use:**

* For columns with high cardinality (many unique values)
* When query performance is more important than storage efficiency
* For columns rarely used in equality filters

**Benefits:**

* Eliminates dictionary lookup overhead
* Better performance for sequential value scanning
* No padding overhead for variable-length data

**Trade-offs:**

* Typically requires more storage space
* Does not support inverted indexes
* Less efficient for random access patterns

## Configuration

### Dictionary-Encoded Forward Index Configuration

Dictionary-encoded forward indexes are the default. To explicitly configure one:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "productCategory",
      "encodingType": "DICTIONARY",
      "indexes": {
        "forward": {
          "compressionCodec": "LZ4"
        }
      }
    }
  ]
}
```

For multi-value columns with repeated entries, you can enable MV\_ENTRY\_DICT compression:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "productTags",
      "encodingType": "DICTIONARY",
      "indexes": {
        "forward": {
          "compressionCodec": "LZ4",
          "compressionType": "MV_ENTRY_DICT"
        }
      }
    }
  ]
}
```

### Sorted Forward Index Configuration

To enable a sorted forward index for real-time tables, specify the sorted column in the table configuration:

```json theme={null}
{
  "tableIndexConfig": {
    "sortedColumn": [
      "timestamp"
    ]
  }
}
```

**Note:**

> [!WARNING]
>
> * For offline tables, data must be pre-sorted before ingestion. The `sortedColumn` property is ignored for offline tables.
> * For online tables, only one column should be included. Using an array with more than one column will not result in segments being sorted by all the columns listed in the array.

### Raw Value Forward Index Configuration

To enable a raw value forward index, set the encoding type to RAW:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "userId",
      "encodingType": "RAW",
      "indexes": {
        "forward": {
          "compressionCodec": "PASS_THROUGH", 
          "deriveNumDocsPerChunk": false,
          "rawIndexWriterVersion": 4,
          "targetDocsPerChunk": 2048,
          "targetMaxChunkSize": 2097152
        }
      }
    }
  ]
}
```

### Compression Options

The following compression options are available:

* **PASS\_THROUGH**: No compression
* **SNAPPY**: Balanced compression ratio and speed
* **ZSTANDARD**: High compression ratio but slower
* **LZ4**: Fast compression with moderate ratio
* **GZIP**: High compression ratio but slower

### Raw Index Configuration Parameters

1. **compressionCodec**: Defines the compression algorithm to use.
2. **deriveNumDocsPerChunk**: For variable-length data, determines whether to derive chunk size dynamically (true) or use a fixed value (false).
3. **rawIndexWriterVersion**: The on-disk format version used to write the raw index. The default is 6 on clusters created on StarTree Cloud 0.17.0 or later, and 4 on older clusters. See [Raw forward index format V6](#raw-forward-index-format-v6).
4. **targetDocsPerChunk**: Sets the target number of documents per chunk.
5. **targetMaxChunkSize**: Sets the maximum size of a chunk in bytes.

### Raw forward index format V6

V6 is a raw forward index format that makes variable-length columns smaller on disk. Starting with StarTree Cloud 0.17.0, raw forward indexes default to V6 with `ZSTANDARD` compression on new clusters. Clusters created before 0.17.0 keep writing V4 until StarTree switches their default; contact StarTree support to move a cluster to V6.

Each chunk of a raw index starts with a header that locates every value in the chunk. V4 writes cumulative byte offsets in that header, which change on almost every row and compress poorly. V6 writes the size of each value instead. Sizes repeat a lot, so the header compresses well, and the server rebuilds the offsets when it reads the chunk. The values themselves are stored the same way in both versions.

**Columns it applies to:**

* Single-value raw `STRING`, `BYTES`, `BIG_DECIMAL`, `JSON`, and `MAP` columns
* All multi-value raw columns

Fixed-width single-value columns (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `BOOLEAN`, `TIMESTAMP`) and dictionary-encoded columns don't change. Columns using the `PASS_THROUGH` codec don't benefit, because nothing is compressed.

**What to expect:** In internal tests on realistic string data, both using `ZSTANDARD`, V6 forward indexes were about 28% smaller than V4, and server CPU dropped by roughly 4–8% on most queries. Combined with the move from `LZ4` to `ZSTANDARD`, raw forward indexes were about 56% smaller. Data with little repetition, such as random strings, shrinks much less. Query results are the same in both formats.

#### Default compression codec

From 0.17.0, raw forward indexes that don't set `compressionCodec` use `ZSTANDARD` instead of `LZ4`, on new and existing clusters. `ZSTANDARD` compresses better but costs more CPU to decompress, which shows most on [tiered storage](/corecapabilities/manage-data/set-up-tiered-storage/setup), where every chunk read is a fetch plus a decompress. Raw metric columns stay uncompressed, and `MAP` columns stay on `LZ4`.

#### Which segments change

The defaults apply only to segments built after the change, by ingestion, a segment push, or a minion task. Existing segments keep their format and codec, and restarting or reloading a server doesn't rewrite them. A table can hold V4 and V6 segments, and `LZ4` and `ZSTANDARD` segments, at the same time.

A column that sets `rawIndexWriterVersion` or `compressionCodec` in the table config keeps that value. Remove the setting from the column's forward index config to pick up the cluster default.

#### Set the version for a column

To use V6 on a cluster that still defaults to V4, set `rawIndexWriterVersion` to 6 on the column:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "eventPayload",
      "encodingType": "RAW",
      "indexes": {
        "forward": {
          "compressionCodec": "ZSTANDARD",
          "rawIndexWriterVersion": 6
        }
      }
    }
  ]
}
```

To keep a column on V4, set `rawIndexWriterVersion` to 4 the same way. Only new segments go back to V4; existing V6 segments stay in V6 and keep loading normally.

### Disabling Forward Index

To disable a forward index for storage optimization:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "filterOnlyColumn",
      "indexes": {
        "forward": {
          "disabled": true
        }
      }
    }
  ]
}
```

## Important Considerations When Disabling Forward Index

Disabling a forward index has the following limitations:

1. **Supported only for offline segments**: Not applicable to real-time segments.
2. **Range index requirements**: If the column has a range index, it must be single-valued and use range index version 2.
3. **Dictionary and inverted index**: To support regeneration on reload, both dictionary and inverted index must be enabled.
4. **Multi-value columns**: Ordering and duplicate entries in multi-value columns will not be preserved if forward index is regenerated after being disabled.
5. **Query limitations**: Columns with disabled forward indexes cannot be used in:
   * SELECT clause
   * GROUP BY or ORDER BY clauses
   * Some aggregation functions (SUM, AVG, etc.)
   * DISTINCT queries
   * Range queries without a version 2 range index


# FST Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/fst-index

The FST (Finite State Transducer) index accelerates regex and prefix text queries by compactly encoding dictionary-encoded columns.

## Overview and Purpose

A Finite State Transducer (FST) index is a specialized data structure that accelerates regular expression queries on text columns while dramatically reducing the on-disk index size. It provides efficient pattern matching capabilities while consuming 4-6 times less storage space compared to traditional indexing approaches.

In StarTree Cloud (powered by Apache Pinot), FST indexes are particularly valuable for:

* Supporting regular expression (regex) queries on text columns
* Optimizing prefix match operations (`LIKE 'prefix%'`)
* Reducing index storage requirements for text search functionality
* Pattern matching across large text datasets with storage efficiency
* Applications requiring wildcard searches with storage constraints

> [!NOTE]
>
> FST indexes are optimized for regex operations and prefix queries. They complement text indexes by providing a more storage-efficient option for specific text search patterns.

## How the Index Works

### Core Concepts

Traditional text indexing approaches can require significant storage space to enable pattern matching across text data. The FST index takes a different approach, using finite state transducers - a specialized data structure from computational linguistics - to represent patterns efficiently.

The FST index in StarTree Cloud:

1. **Compact Representation**: Represents all possible text patterns in the column as a directed graph structure that recognizes valid strings.
2. **Prefix Optimization**: Particularly efficient for prefix searches, as the FST can quickly traverse the initial states of the automaton.
3. **Storage Efficiency**: Achieves 4-6x reduction in on-disk index size compared to other text indexing approaches.
4. **Dictionary Integration**: Leverages dictionary encoding to further optimize performance and storage.

### Example Illustration

For a column storing product names:

1. **Traditional Text Index**: Stores tokenized terms and their positions, consuming significant storage space.
2. **FST Index Approach**:
   * Builds a finite state automaton representing all product names
   * Enables regex queries like `product LIKE 'Smart%'` to quickly find all products starting with "Smart"
   * Consumes 4-6x less storage space compared to text index

When processing a query like `WHERE product LIKE 'Phone%'`, FST can efficiently traverse the initial states of the automaton that correspond to the prefix "Phone", then identify all possible completions that exist in the data.

## Configuration

### Enabling FST Index

To enable an FST index on a column in your StarTree Cloud table, add the following configuration to your table definition:

```
{
  "fieldConfigList": [
    {
      "name": "productName",
      "encodingType": "DICTIONARY",
      "indexes": {
        "fst": {}
      }
    }
  ]
}
```

### Alternative Configuration (Legacy Method)

You can also use the older configuration format:

```
{
  "fieldConfigList": [
    {
      "name": "productName",
      "encodingType": "DICTIONARY",
      "indexTypes": ["FST"]
    }
  ]
}
```

### Important Configuration Requirements

1. **Dictionary Encoding**: The column must be dictionary-encoded (not RAW encoded).
   * Set `encodingType` to `DICTIONARY` (this is the default if left unspecified)
2. **Segment Types**: FST indexes are only supported on:
   * Completed segments (offline mode)
   * Committed segments (real-time mode)
   * Not supported on consuming segments (real-time in-progress segments)
3. **Complementary Indexing**: FST indexes work well alongside inverted indexes:
   * If an inverted index is also enabled on the same column, the FST index can leverage it for additional performance improvements

## Performance Considerations

1. **Prefix Query Efficiency**: FST indexes perform best with prefix queries (patterns starting with a fixed string followed by wildcards) rather than patterns with leading wildcards.
2. **Storage Benefits**: The primary advantage is the 4-6x reduction in on-disk index size compared to other text index types.
3. **Case Sensitivity**: Remember that the index is case-sensitive, which might require adjustments to query patterns.
4. **Dictionary Requirement**: The requirement for dictionary encoding means FST indexes are best suited for columns with manageable cardinality.
5. **Complementary Use**: Consider using FST indexes alongside other index types:
   * FST for regex and prefix queries
   * Text index for more complex text search needs
   * Inverted index for exact match operations


# GeoSpatial Index Configuration
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/geospatial-configuration

H3 resolution selection, schema setup, and index optimization for maximum spatial query performance.

This guide covers everything you need to know about configuring geospatial indexes in Apache Pinot for optimal spatial query performance.

## Overview

Apache Pinot uses **H3 (Hierarchical Hexagons)** indexing system for geospatial operations. H3 divides the Earth's surface into hexagonal cells at multiple resolution levels, enabling fast spatial filtering before applying precise geospatial calculations.

> **H3 Indexing Benefits**
>
> * **Fast spatial filtering** using hexagonal grids
> * **Configurable precision** with H3 resolution levels
> * **Hierarchical structure** for efficient range queries
> * **10-100x performance improvement** over non-indexed queries

> **Configuration Steps**
>
> 1. Define geospatial column as BYTES in schema
> 2. Add transform function for data conversion
> 3. Configure H3 index with appropriate resolutions
> 4. Disable dictionary encoding (use RAW)

## H3 Resolution Levels

Understanding H3 resolution levels is crucial for optimal configuration:

### **Resolution to Precision Mapping**

| Resolution | Hex Edge Length | Hex Area | Use Case |
| - | - | - | - |
| 0 | \~1,107 km | \~4,250,547 km² | Continental analysis |
| 1 | \~418 km | \~607,220 km² | Country-level analysis |
| 2 | \~158 km | \~86,745 km² | State/province level |
| 3 | \~59 km | \~12,392 km² | Regional analysis |
| 4 | \~22 km | \~1,770 km² | Metropolitan areas |
| 5 | \~8.2 km | \~252.9 km² | City-wide analysis |
| 6 | \~3.1 km | \~36.1 km² | Urban districts |
| 7 | \~1.2 km | \~5.2 km² | Neighborhood level |
| 8 | \~461 m | \~0.73 km² | Block-level precision |
| 9 | \~174 m | \~0.10 km² | Building-level precision |
| 10 | \~65 m | \~0.015 km² | Property-level precision |
| 11 | \~25 m | \~0.002 km² | Room-level precision |
| 12 | \~9.4 m | \~0.0003 km² | Meter-level precision |
| 13 | \~3.5 m | \~0.00004 km² | Sub-meter precision |
| 14 | \~1.3 m | \~0.000006 km² | Centimeter precision |
| 15 | \~0.5 m | \~0.0000009 km² | Millimeter precision |

### **Resolution Selection Guidelines**

**Use Case Based**

**Global Applications (0-4)**

* Cross-country logistics
* International commerce
* Climate/weather analysis

**Regional Applications (5-8)**

* City-wide services
* Delivery optimization
* Urban planning

**Local Applications (9-12)**

* Store locators
* Asset tracking
* Navigation systems

**Precision Applications (13-15)**

* Indoor positioning
* IoT sensors
* Robotics/automation

**Query Pattern Based**

**Large Radius Queries (>10km)**

* Use resolutions 3-7
* Example: "Find all stores within 50km"

**Medium Radius Queries (1-10km)**

* Use resolutions 6-10
* Example: "Find restaurants within 5km"

**Small Radius Queries (\<1km)**

* Use resolutions 9-13
* Example: "Find nearby parking spots"

**Precision Queries (\<100m)**

* Use resolutions 11-15
* Example: "Find exact building entrance"

## Basic Configuration

### **Step 1: Schema Configuration**

Define your geospatial column in the table schema:

```json theme={null}
{
  "schemaName": "spatial_data",
  "dimensionFieldSpecs": [
    {
      "name": "location",
      "dataType": "BYTES",
      "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "timestamp",
      "dataType": "TIMESTAMP", 
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]
}
```

### **Step 2: Table Configuration**

Configure the H3 index in your table config:

```json theme={null}
{
  "tableName": "spatial_data",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "timestamp",
    "schemaName": "spatial_data",
    "replication": "1"
  },
  "fieldConfigList": [
    {
      "name": "location",
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [8]
        }
      }
    }
  ],
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "noDictionaryColumns": ["location"]
  }
}
```

## Advanced Configuration Examples

### **Single Resolution Strategy**

Configure a single H3 resolution appropriate for your query patterns:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "location",
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [8]
        }
      }
    }
  ]
}
```

**Why Single Resolution?**
The H3 index operators (`H3IndexFilterOperator` and `H3InclusionIndexFilterOperator`) use only the **lowest (most coarse-grained) resolution** from your configuration. Multiple resolutions provide no performance benefit and may increase storage overhead unnecessarily.

**Distance vs Resolution Matching**:
Choose your resolution based on typical query distances. If your query distance is >100x the hexagon edge length, the index will automatically revert to full scan.

Example:

* **Resolution 8** (hex edge \~461m): Good for queries up to \~46km
* **Resolution 5** (hex edge \~8.2km): Good for queries up to \~820km
* **Resolution 11** (hex edge \~65m): Good for queries up to \~6.5km

### **Legacy Configuration Format**

The older configuration format is still supported:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "location",
      "encodingType": "RAW",
      "indexTypes": ["H3"],
      "properties": {
        "resolutions": "8"
      }
    }
  ]
}
```

> [!NOTE]
>
> **Note**: In legacy format, use "resolutions" as a string. In the current format, use "resolution" as an array.

## Transform Function Options

### **Basic Point Creation**

```json theme={null}
{
  "name": "location",
  "dataType": "BYTES",
  "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
}
```

### **From Existing WKT Data**

```json theme={null}
{
  "name": "geom_location", 
  "dataType": "BYTES",
  "transformFunction": "ST_GeomFromText(wkt_column)"
}
```

### **From GeoJSON Data**

```json theme={null}
{
  "name": "geojson_location",
  "dataType": "BYTES", 
  "transformFunction": "ST_GeogFromGeoJson(geojson_column)"
}
```

### **Conditional Transform Functions**

```json theme={null}
{
  "name": "processed_location",
  "dataType": "BYTES",
  "transformFunction": "CASE WHEN longitude IS NOT NULL THEN toSphericalGeography(stPoint(longitude, latitude)) ELSE ST_GeogFromText(wkt_backup) END"
}
```

## Real-World Configuration Examples

### **Ride-Sharing Service**

```json theme={null}
{
  "schemaName": "ride_requests",
  "dimensionFieldSpecs": [
    {
      "name": "pickup_location",
      "dataType": "BYTES", 
      "transformFunction": "toSphericalGeography(stPoint(pickup_lon, pickup_lat))"
    },
    {
      "name": "dropoff_location",
      "dataType": "BYTES",
      "transformFunction": "toSphericalGeography(stPoint(dropoff_lon, dropoff_lat))"
    }
  ]
}
```

**Table Config:**

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "pickup_location",
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [8]
        }
      }
    },
    {
      "name": "dropoff_location", 
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [8]
        }
      }
    }
  ]
}
```

### **Retail Store Locator**

```json theme={null}
{
  "schemaName": "stores",
  "dimensionFieldSpecs": [
    {
      "name": "store_location",
      "dataType": "BYTES",
      "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
    },
    {
      "name": "delivery_zone",
      "dataType": "BYTES", 
      "transformFunction": "ST_GeogFromText(delivery_polygon_wkt)"
    }  
  ]
}
```

**Table Config:**

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "store_location",
      "encodingType": "RAW", 
      "indexes": {
        "h3": {
          "resolution": [9]
        }
      }
    },
    {
      "name": "delivery_zone",
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [7]
        }
      }
    }
  ]
}
```

### **IoT Sensor Network**

```json theme={null}
{
  "schemaName": "sensor_readings",
  "dimensionFieldSpecs": [
    {
      "name": "sensor_location", 
      "dataType": "BYTES",
      "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
    }
  ]
}
```

**Table Config:**

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "sensor_location",
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [11]
        }
      }
    }
  ]
}
```

## Configuration Best Practices

### **Required Configuration**

1. **Disable dictionary encoding** with `"encodingType": "RAW"` (required for H3 index to work)
2. **Add to noDictionaryColumns** in tableIndexConfig (achieves the same as above - can use both for safety)
3. **Use BYTES data type** for geospatial columns in schema
4. **Choose single resolution** appropriate for your query patterns (only lowest resolution is used by index operators)

### **Common Configuration Errors**

1. **Using "resolutions" instead of "resolution"** in current config format - will cause indexer to fail
2. **Forgetting RAW encoding** - index won't work with dictionary encoding
3. **Using multiple resolutions** - only the lowest resolution is used, others waste storage
4. **Wrong data type** - must use BYTES, not STRING for geospatial columns
5. **Resolution too high for query distances** - when query distance is >100x the hexagon size, index is bypassed and reverts to full scan

## Validation and Testing

### **Verify Index Creation**

Check if your index was created successfully:

```sql theme={null}
-- Verify index usage with EXPLAIN PLAN
EXPLAIN PLAN FOR 
SELECT * FROM your_table 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;
```

Look for these operators in the query plan:

* `FILTER_H3_INDEX` - Index is being used for ST\_Distance
* `INCLUSION_FILTER_H3_INDEX` - Index is being used for ST\_Within/ST\_Contains

### **Test Query Performance**

```sql theme={null}
-- Test different query ranges
SELECT COUNT(*) FROM your_table 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 1000;   -- Should use high resolution

SELECT COUNT(*) FROM your_table  
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 50000;  -- Should use low resolution
```

### **Monitor Index Effectiveness**

```sql theme={null}
-- Compare performance with and without index
-- (Create a duplicate table without index for comparison)

-- Indexed query
SELECT COUNT(*) FROM spatial_data_indexed 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;

-- Non-indexed query  
SELECT COUNT(*) FROM spatial_data_no_index
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;
```

## Migration from Non-Indexed Tables

### **Adding Index to Existing Table**

1. **Update table configuration**:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "existing_location_column",
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [8]
        }
      }
    }
  ]
}
```

2. **Reload table configuration**:

```bash theme={null}
# Use Pinot admin tools or REST API
curl -X PUT "http://controller:9000/tables/tableName" \
  -H "Content-Type: application/json" \
  -d @updated_table_config.json
```

3. **Refresh/rebuild segments** to apply the new index

```bash theme={null}
# Use Pinot admin tools or REST API
curl -X POST "http://controller:9000/segments/tableName/reload?type=OFFLINE&forceDownload=false"
```


# GeoSpatial Data Types & Formats
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/geospatial-data-types

Geometry vs geography types, format conversions, and choosing the right spatial data representation.

Apache Pinot supports multiple geospatial data types and formats, enabling you to work with location data in the most appropriate format for your use case.

## Data Type Overview

### **Geometry vs Geography**

Understanding the difference between geometry and geography types is crucial for choosing the right approach for your spatial data:

> **Geometry Type**
>
> **Cartesian Plane Calculations**
>
> * Treats coordinates as X/Y values on a flat surface
> * Distance measurements in coordinate units (often degrees)
> * Best for smaller areas where Earth's curvature is negligible
> * Faster calculations but less accurate over large distances

> **Geography Type**
>
> **Spherical Earth Calculations**
>
> * Accounts for Earth's curvature in all calculations
> * Distance measurements always in meters
> * More accurate for large distances and areas near poles
> * Slightly slower but geographically correct

### **When to Use Each Type**

| Use Case | Recommended Type | Reason |
| - | - | - |
| City-wide analysis | Geometry | Fast, sufficient accuracy |
| Global applications | Geography | Accurate across all locations |
| Small regions (\< 100km) | Either | Difference is minimal |
| Navigation/GPS data | Geography | Requires precise distances |
| Engineering/CAD data | Geometry | Working in projected coordinates |

## Supported Data Formats

Apache Pinot supports three industry-standard geospatial formats:

### **1. Well-Known Text (WKT)**

Human-readable text format that's easy to understand and debug.

**Supported WKT Types:**

```sql theme={null}
-- Basic shapes
POINT (longitude latitude)
LINESTRING (x1 y1, x2 y2, x3 y3)
POLYGON ((x1 y1, x2 y2, x3 y3, x1 y1))

-- Multi-geometries  
MULTIPOINT ((x1 y1), (x2 y2))
MULTILINESTRING ((x1 y1, x2 y2), (x3 y3, x4 y4))
MULTIPOLYGON (((x1 y1, x2 y2, x3 y3, x1 y1)), ((x4 y4, x5 y5, x6 y6, x4 y4)))

-- Complex collections
GEOMETRYCOLLECTION(POINT(x1 y1), POLYGON((x2 y2, x3 y3, x4 y4, x2 y2)))
```

**Examples:**

```sql theme={null}
-- San Francisco coordinates
POINT (-122.4194 37.7749)

-- Simple route
LINESTRING (-122.4194 37.7749, -122.4094 37.7849, -122.3994 37.7949)

-- Golden Gate Park boundary (simplified)
POLYGON ((-122.5137 37.7694, -122.4529 37.7694, -122.4529 37.7849, -122.5137 37.7849, -122.5137 37.7694))
```

### **2. Well-Known Binary (WKB)**

Compact binary format optimized for storage and transmission.

```sql theme={null}
-- Create from WKB
SELECT ST_GeomFromWKB(wkb_column) FROM spatial_data;

-- Convert to WKB for storage
SELECT ST_AsBinary(ST_Point(-122.4194, 37.7749));
```

### **3. GeoJSON Format**

**NEW**: Full GeoJSON support for modern web applications and APIs.

**Supported GeoJSON Types:**

* Point, LineString, Polygon
* MultiPoint, MultiLineString, MultiPolygon
* GeometryCollection
* Feature, FeatureCollection

#### **GeoJSON Functions**

| Function | Purpose | Example |
| - | - | - |
| `ST_GeomFromGeoJson(string)` | Create geometry from GeoJSON | Convert GeoJSON to internal format |
| `ST_GeogFromGeoJson(string)` | Create geography from GeoJSON | Same but with spherical calculations |
| `ST_AsGeoJson(binary)` | Convert to GeoJSON | Export data for web applications |

#### **GeoJSON Examples**

**Point Geometry:**

```sql theme={null}
-- Convert GeoJSON to geometry
SELECT ST_AsText(ST_GeomFromGeoJson('{"type":"Point","coordinates":[-122.4194,37.7749]}'))
-- Returns: POINT (-122.4194 37.7749)

-- Convert WKT to GeoJSON  
SELECT ST_AsGeoJson(ST_GeomFromText('POINT (-122.4194 37.7749)'))
-- Returns: {"type":"Point","coordinates":[-122.4194,37.7749],"crs":{"type":"name","properties":{"name":"EPSG:0"}}}
```

**Point Geography:**

```sql theme={null}
-- Create geography from GeoJSON
SELECT ST_AsGeoJson(ST_GeogFromGeoJson('{"type":"Point","coordinates":[-122.4194,37.7749]}'))
-- Returns: {"type":"Point","coordinates":[-122.4194,37.7749],"crs":{"type":"name","properties":{"name":"EPSG:4326"}}}
```

**Complex Geometries:**

```sql theme={null}
-- MultiLineString example
SELECT ST_AsGeoJson(ST_GeomFromText('MULTILINESTRING ((1 1, 5 1), (2 4, 4 4))'))
-- Returns:
{
  "type":"MultiLineString",
  "coordinates":[[[1,1],[5,1]],[[2,4],[4,4]]],
  "crs":{"type":"name","properties":{"name":"EPSG:0"}}
}

-- Convert back from GeoJSON
SELECT ST_AsText(ST_GeomFromGeoJson('{"type":"MultiLineString","coordinates":[[[1,1],[5,1]],[[2,4],[4,4]]]}'))
-- Returns: MULTILINESTRING ((1 1, 5 1), (2 4, 4 4))
```

**Feature Collections:**

```sql theme={null}
-- Working with GeoJSON Features
SELECT ST_GeomFromGeoJson('{
  "type": "Feature",
  "geometry": {
    "type": "Point", 
    "coordinates": [-122.4194, 37.7749]
  },
  "properties": {
    "name": "San Francisco",
    "population": 875000
  }
}')
```

## Data Type Conversion

### **Between Geometry and Geography**

```sql theme={null}
-- Convert geometry to geography (for spherical calculations)
SELECT toSphericalGeography(ST_GeomFromText('POINT(-122.4194 37.7749)'))

-- Convert geography back to geometry  
SELECT toGeometry(geography_column) FROM my_table
```

### **Format Conversions**

```sql theme={null}
-- WKT ↔ WKB
SELECT ST_AsText(ST_GeomFromWKB(wkb_data))        -- WKB to WKT
SELECT ST_AsBinary(ST_GeomFromText(wkt_string))   -- WKT to WKB

-- WKT ↔ GeoJSON  
SELECT ST_AsGeoJson(ST_GeomFromText(wkt_string))  -- WKT to GeoJSON
SELECT ST_AsText(ST_GeomFromGeoJson(geojson))     -- GeoJSON to WKT

-- WKB ↔ GeoJSON
SELECT ST_AsGeoJson(ST_GeomFromWKB(wkb_data))     -- WKB to GeoJSON  
SELECT ST_AsBinary(ST_GeomFromGeoJson(geojson))   -- GeoJSON to WKB
```

## Schema Configuration

### **Basic Geospatial Column**

```json theme={null}
{
  "schemaName": "spatial_data",
  "dimensionFieldSpecs": [
    {
      "name": "location",
      "dataType": "BYTES",
      "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
    }
  ]
}
```

### **Transform Function Examples**

Choose the appropriate transform function based on your data source and application requirements:

**For Global Applications** (prefer geography):

```json theme={null}
{
  "name": "location",
  "dataType": "BYTES",
  "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
}
```

**For Regional Analysis** (geometry may be sufficient):

```json theme={null}
{
  "name": "local_coords", 
  "dataType": "BYTES",
  "transformFunction": "stPoint(x_coordinate, y_coordinate)"
}
```

**For Existing GeoJSON Data**:

```json theme={null}
{
  "name": "geojson_geom",
  "dataType": "BYTES",
  "transformFunction": "ST_GeogFromGeoJson(geojson_field)"
}
```

## Best Practices

### **Coordinate Order**

> [!WARNING]
>
> **Important**: Always use longitude, latitude order (X, Y) in functions, which matches GeoJSON standard but differs from some other systems that use latitude, longitude.

```sql theme={null}
-- Correct: longitude first, latitude second
ST_Point(-122.4194, 37.7749)  -- San Francisco

-- Incorrect: would place point in Antarctica  
ST_Point(37.7749, -122.4194)
```

### **Performance Considerations**

* **Geography calculations** are \~10-20% slower than geometry but more accurate
* **ST\_Distance with geography** only supports point geometries (use geometry types for linestrings/polygons)
* **GeoJSON parsing** has slight overhead compared to WKT
* **WKB format** is most compact for storage
* **Multiple formats** in the same table are supported but increase storage


# GeoSpatial Examples & Use Cases
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/geospatial-examples

Industry-specific implementations from retail to IoT, with complete schemas, configurations, and optimized queries.

Explore practical implementations of geospatial analytics with Apache Pinot through real-world examples and industry-specific use cases.

> **[Location-Based Services](#location-based-services)**
>
> Store locators, service areas, and proximity analysis

> **[Transportation & Logistics](#transportation--logistics)**
>
> Fleet tracking, route optimization, and delivery management

> **[Real Estate & Urban Planning](#real-estate--urban-planning)**
>
> Property analysis, market research, and urban analytics

> **[IoT & Sensor Networks](#iot--sensor-networks)**
>
> Environmental monitoring, smart cities, and sensor analytics

## Quick Start Example

Here's a step-by-step example to get you started with GeoSpatial queries:

### **1. Schema Definition**

Define your geospatial column in the table schema:

```json theme={null}
{
  "schemaName": "stores",
  "dimensionFieldSpecs": [
    {
      "name": "store_name",
      "dataType": "STRING"
    },
    {
      "name": "address",
      "dataType": "STRING"
    },
    {
      "name": "location",
      "dataType": "BYTES",
      "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
    }
  ]
}
```

### **2. Index Configuration**

Configure the H3 index in your table configuration:

```json theme={null}
{
  "tableName": "stores",
  "tableType": "REALTIME",
  "fieldConfigList": [
    {
      "name": "location",
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [5, 8, 11]
        }
      }
    }
  ],
  "tableIndexConfig": {
    "noDictionaryColumns": ["location"]
  }
}
```

### **3. Sample Queries**

**Find nearby stores:**

```sql theme={null}
SELECT store_name, address, 
       ST_Distance(location, ST_Point(-122.4194, 37.7749)) as distance_meters
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000
ORDER BY distance_meters
LIMIT 10;
```

**Count stores by distance range:**

```sql theme={null}
SELECT 
  CASE 
    WHEN ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 1000 THEN 'Within 1km'
    WHEN ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000 THEN '1-5km away'
    ELSE 'More than 5km'
  END as distance_range,
  COUNT(*) as store_count
FROM stores
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 10000
GROUP BY distance_range;
```

## Complete Implementation Examples

### **Store Locator Application**

**Scenario**: Build a store locator that finds nearby locations with performance optimizations.

#### **Schema & Configuration**

```json theme={null}
{
  "schemaName": "store_locations",
  "dimensionFieldSpecs": [
    {
      "name": "store_id",
      "dataType": "STRING"
    },
    {
      "name": "store_name", 
      "dataType": "STRING"
    },
    {
      "name": "address",
      "dataType": "STRING"
    },
    {
      "name": "city",
      "dataType": "STRING"
    },
    {
      "name": "category",
      "dataType": "STRING"
    },
    {
      "name": "location",
      "dataType": "BYTES",
      "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
    },
    {
      "name": "service_area",
      "dataType": "BYTES", 
      "transformFunction": "ST_GeogFromText(service_polygon_wkt)"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "rating",
      "dataType": "DOUBLE"
    },
    {
      "name": "review_count",
      "dataType": "INT"
    }
  ]
}
```

**Table Configuration:**

```json theme={null}
{
  "tableName": "store_locations",
  "tableType": "REALTIME",
  "fieldConfigList": [
    {
      "name": "location",
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [7, 10]
        }
      }
    },
    {
      "name": "service_area", 
      "encodingType": "RAW",
      "indexes": {
        "h3": {
          "resolution": [5, 8]
        }
      }
    }
  ],
  "tableIndexConfig": {
    "noDictionaryColumns": ["location", "service_area"]
  }
}
```

#### **Common Queries**

**Find Nearest Stores:**

```sql theme={null}
SELECT 
  store_name,
  address, 
  category,
  rating,
  ST_Distance(location, ST_Point(-122.4194, 37.7749)) as distance_meters
FROM store_locations 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000
  AND rating >= 4.0
ORDER BY distance_meters
LIMIT 10;
```

**Store Density Analysis:**

```sql theme={null}
SELECT 
  city,
  category,
  COUNT(*) as store_count,
  AVG(rating) as avg_rating,
  AVG(ST_Distance(location, ST_Point(-122.4194, 37.7749))) as avg_distance_from_center
FROM store_locations 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 25000
GROUP BY city, category
ORDER BY store_count DESC;
```

**Service Area Coverage:**

```sql theme={null}
-- Find which stores can serve a specific location
SELECT 
  store_name,
  address,
  'delivers_to_location' as service_type
FROM store_locations 
WHERE ST_Contains(service_area, ST_Point(-122.4094, 37.7849))
UNION ALL
-- Find stores within walking distance
SELECT 
  store_name,
  address, 
  'walking_distance' as service_type
FROM store_locations
WHERE ST_Distance(location, ST_Point(-122.4094, 37.7849)) < 800;
```

***

## Location-Based Services

### **Food Delivery Platform**

**Use Case**: Real-time delivery optimization and customer matching.

#### **Dynamic Delivery Radius**

```sql theme={null}
-- Find available restaurants based on delivery capacity and distance
WITH restaurant_capacity AS (
  SELECT 
    restaurant_id,
    location,
    CASE 
      WHEN active_orders < 10 THEN 5000  -- Extended radius when less busy
      WHEN active_orders < 20 THEN 3000  -- Standard radius
      ELSE 1500                          -- Reduced radius when busy
    END as delivery_radius
  FROM restaurants 
  WHERE is_open = true
)
SELECT 
  r.restaurant_name,
  r.cuisine_type,
  r.avg_delivery_time,
  ST_Distance(r.location, ST_Point(-122.4194, 37.7749)) as distance_meters
FROM restaurant_capacity rc
JOIN restaurants r ON rc.restaurant_id = r.restaurant_id
WHERE ST_Distance(rc.location, ST_Point(-122.4194, 37.7749)) < rc.delivery_radius
ORDER BY distance_meters, r.avg_delivery_time;
```

#### **Delivery Driver Optimization**

```sql theme={null}
-- Find optimal driver assignments based on location and capacity
SELECT 
  d.driver_id,
  d.driver_name,
  COUNT(o.order_id) as current_orders,
  ST_Distance(d.current_location, ST_Point(-122.4194, 37.7749)) as distance_to_pickup,
  AVG(ST_Distance(o.delivery_location, ST_Point(-122.4194, 37.7749))) as avg_delivery_distance
FROM drivers d
LEFT JOIN active_orders o ON d.driver_id = o.assigned_driver
WHERE d.is_available = true
  AND ST_Distance(d.current_location, ST_Point(-122.4194, 37.7749)) < 3000
GROUP BY d.driver_id, d.driver_name, d.current_location
HAVING COUNT(o.order_id) < 3
ORDER BY distance_to_pickup, current_orders;
```

### **Healthcare Facility Finder**

**Use Case**: Emergency services and healthcare accessibility analysis.

```sql theme={null}
-- Find nearest hospitals with specific capabilities
SELECT 
  h.hospital_name,
  h.hospital_type,
  h.capabilities,
  h.emergency_services,
  ST_Distance(h.location, ST_Point(-122.4194, 37.7749)) as distance_meters,
  h.wait_time_minutes
FROM hospitals h
WHERE ST_Distance(h.location, ST_Point(-122.4194, 37.7749)) < 15000
  AND h.emergency_services = true
  AND h.capabilities LIKE '%cardiology%'
  AND h.current_capacity > 0
ORDER BY 
  CASE 
    WHEN h.hospital_type = 'Level 1 Trauma' THEN 1
    WHEN h.hospital_type = 'Level 2 Trauma' THEN 2  
    ELSE 3
  END,
  distance_meters;
```

**Healthcare Accessibility Analysis:**

```sql theme={null}
-- Analyze healthcare coverage by area
WITH coverage_areas AS (
  SELECT 
    ST_Point(longitude, latitude) as area_center,
    neighborhood_name,
    population
  FROM census_blocks
),
nearest_hospital AS (
  SELECT 
    ca.neighborhood_name,
    ca.population,
    MIN(ST_Distance(ca.area_center, h.location)) as distance_to_nearest_hospital,
    COUNT(CASE WHEN ST_Distance(ca.area_center, h.location) < 5000 THEN 1 END) as hospitals_within_5km
  FROM coverage_areas ca
  CROSS JOIN hospitals h
  GROUP BY ca.neighborhood_name, ca.population
)
SELECT 
  neighborhood_name,
  population,
  distance_to_nearest_hospital,
  hospitals_within_5km,
  CASE 
    WHEN distance_to_nearest_hospital > 10000 THEN 'Underserved'
    WHEN distance_to_nearest_hospital > 5000 THEN 'Limited Access'
    ELSE 'Good Access'
  END as accessibility_level
FROM nearest_hospital
ORDER BY distance_to_nearest_hospital DESC;
```

***

## Transportation & Logistics

### **Fleet Management System**

**Schema for Vehicle Tracking:**

```json theme={null}
{
  "schemaName": "vehicle_tracking",
  "dimensionFieldSpecs": [
    {
      "name": "vehicle_id",
      "dataType": "STRING"
    },
    {
      "name": "driver_id", 
      "dataType": "STRING"
    },
    {
      "name": "location",
      "dataType": "BYTES",
      "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
    },
    {
      "name": "route_polygon",
      "dataType": "BYTES",
      "transformFunction": "ST_GeogFromText(route_wkt)"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "speed_kmh",
      "dataType": "DOUBLE"
    },
    {
      "name": "fuel_level",
      "dataType": "DOUBLE"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "timestamp",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]
}
```

#### **Route Deviation Detection**

```sql theme={null}
-- Identify vehicles that have deviated from their assigned routes
SELECT 
  v.vehicle_id,
  v.driver_id,
  v.timestamp,
  ST_AsText(v.location) as current_position,
  ST_Distance(v.location, 
    (SELECT location FROM route_waypoints rw 
     WHERE rw.route_id = v.assigned_route_id 
     ORDER BY ST_Distance(rw.location, v.location) 
     LIMIT 1)
  ) as deviation_distance_meters
FROM vehicle_tracking v
WHERE ST_Within(v.location, v.route_polygon) = false
  AND v.timestamp >= now() - INTERVAL '1' HOUR
  AND v.speed_kmh > 5  -- Exclude parked vehicles
ORDER BY deviation_distance_meters DESC;
```

#### **Geofencing Alerts**

```sql theme={null}
-- Monitor vehicles entering/exiting specific zones
WITH zone_events AS (
  SELECT 
    v.vehicle_id,
    v.timestamp,
    z.zone_name,
    z.zone_type,
    CASE 
      WHEN ST_Within(v.location, z.zone_polygon) THEN 'ENTERED'
      WHEN ST_Distance(v.location, z.zone_polygon) < 100 THEN 'APPROACHING'
      ELSE 'OUTSIDE'
    END as zone_status
  FROM vehicle_tracking v
  CROSS JOIN geofence_zones z
  WHERE v.timestamp >= now() - INTERVAL '5' MINUTES
    AND (ST_Within(v.location, z.zone_polygon) 
         OR ST_Distance(v.location, z.zone_polygon) < 500)
)
SELECT 
  vehicle_id,
  zone_name,
  zone_type,
  zone_status,
  COUNT(*) as status_changes
FROM zone_events
WHERE zone_type IN ('RESTRICTED', 'CUSTOMER_SITE', 'DEPOT')
GROUP BY vehicle_id, zone_name, zone_type, zone_status
HAVING COUNT(*) > 1  -- Detect multiple status changes (potential issues)
ORDER BY status_changes DESC;
```

### **Airport Operations Analytics**

#### **Aircraft Positioning & Runway Management**

```sql theme={null}
-- Monitor aircraft positions and runway conflicts
SELECT 
  a.flight_id,
  a.aircraft_type,
  a.status,
  ST_AsText(a.current_position) as position,
  r.runway_id,
  ST_Distance(a.current_position, r.runway_centerline) as distance_to_runway,
  CASE 
    WHEN ST_Within(a.current_position, r.runway_safety_zone) THEN 'IN_SAFETY_ZONE'
    WHEN ST_Distance(a.current_position, r.runway_centerline) < 100 THEN 'APPROACHING'
    ELSE 'CLEAR'
  END as runway_proximity
FROM aircraft_positions a
CROSS JOIN airport_runways r
WHERE a.airport_code = 'SFO'
  AND a.timestamp >= now() - INTERVAL '10' MINUTES
  AND (a.status IN ('LANDING', 'TAKEOFF', 'TAXIING') 
       OR ST_Distance(a.current_position, r.runway_centerline) < 500)
ORDER BY distance_to_runway;
```

***

## Real Estate & Urban Planning

### **Property Market Analysis**

#### **Comprehensive Property Valuation**

```sql theme={null}
-- Analyze property values based on location and nearby amenities
WITH property_amenities AS (
  SELECT 
    p.property_id,
    p.address,
    p.price,
    p.square_feet,
    p.location,
    COUNT(CASE WHEN a.type = 'school' 
               AND ST_Distance(p.location, a.location) < 1000 THEN 1 END) as schools_nearby,
    COUNT(CASE WHEN a.type = 'transit' 
               AND ST_Distance(p.location, a.location) < 500 THEN 1 END) as transit_nearby,
    COUNT(CASE WHEN a.type = 'park' 
               AND ST_Distance(p.location, a.location) < 1000 THEN 1 END) as parks_nearby,
    MIN(CASE WHEN a.type = 'shopping' 
             THEN ST_Distance(p.location, a.location) END) as distance_to_shopping
  FROM properties p
  CROSS JOIN amenities a
  WHERE ST_Distance(p.location, a.location) < 2000
  GROUP BY p.property_id, p.address, p.price, p.square_feet, p.location
)
SELECT 
  property_id,
  address,
  price,
  price / square_feet as price_per_sqft,
  schools_nearby,
  transit_nearby, 
  parks_nearby,
  distance_to_shopping,
  -- Walkability score based on amenities
  (schools_nearby * 2 + transit_nearby * 3 + parks_nearby + 
   CASE WHEN distance_to_shopping < 800 THEN 3 ELSE 0 END) as walkability_score
FROM property_amenities
ORDER BY walkability_score DESC, price_per_sqft;
```

#### **Market Trend Analysis by Neighborhood**

```sql theme={null}
-- Analyze price trends and market dynamics by geographic area
WITH neighborhood_analysis AS (
  SELECT 
    n.neighborhood_name,
    ST_Centroid(n.boundary_polygon) as neighborhood_center,
    COUNT(p.property_id) as total_properties,
    AVG(p.price) as avg_price,
    AVG(p.price / p.square_feet) as avg_price_per_sqft,
    PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY p.price) as median_price,
    COUNT(CASE WHEN p.sale_date >= now() - INTERVAL '90' DAY THEN 1 END) as recent_sales
  FROM neighborhoods n
  JOIN properties p ON ST_Within(p.location, n.boundary_polygon)
  WHERE p.sale_date >= now() - INTERVAL '1' YEAR
  GROUP BY n.neighborhood_name, n.boundary_polygon
)
SELECT 
  na.neighborhood_name,
  na.total_properties,
  na.avg_price,
  na.avg_price_per_sqft,
  na.median_price,
  na.recent_sales,
  -- Calculate nearby neighborhood competition
  AVG(nearby.avg_price_per_sqft) as nearby_avg_price_per_sqft,
  na.avg_price_per_sqft - AVG(nearby.avg_price_per_sqft) as price_premium
FROM neighborhood_analysis na
JOIN neighborhood_analysis nearby 
  ON ST_Distance(na.neighborhood_center, nearby.neighborhood_center) < 5000
  AND na.neighborhood_name != nearby.neighborhood_name
GROUP BY na.neighborhood_name, na.total_properties, na.avg_price, 
         na.avg_price_per_sqft, na.median_price, na.recent_sales
ORDER BY price_premium DESC;
```

### **Urban Planning & Development**

#### **Population Density & Infrastructure Analysis**

```sql theme={null}
-- Analyze population density and infrastructure adequacy
SELECT 
  cb.census_block_id,
  cb.population,
  ST_Area(cb.boundary_polygon) / 1000000 as area_km2,  -- Convert to km²
  cb.population / (ST_Area(cb.boundary_polygon) / 1000000) as population_density_per_km2,
  COUNT(CASE WHEN i.type = 'hospital' 
             AND ST_Distance(ST_Centroid(cb.boundary_polygon), i.location) < 5000 
             THEN 1 END) as hospitals_within_5km,
  COUNT(CASE WHEN i.type = 'school' 
             AND ST_Distance(ST_Centroid(cb.boundary_polygon), i.location) < 2000 
             THEN 1 END) as schools_within_2km,
  COUNT(CASE WHEN i.type = 'fire_station' 
             AND ST_Distance(ST_Centroid(cb.boundary_polygon), i.location) < 3000 
             THEN 1 END) as fire_stations_within_3km,
  -- Infrastructure adequacy score
  CASE 
    WHEN cb.population / (ST_Area(cb.boundary_polygon) / 1000000) > 10000 THEN 'High Density'
    WHEN cb.population / (ST_Area(cb.boundary_polygon) / 1000000) > 5000 THEN 'Medium Density'
    ELSE 'Low Density'
  END as density_category
FROM census_blocks cb
CROSS JOIN infrastructure i
WHERE ST_Distance(ST_Centroid(cb.boundary_polygon), i.location) < 10000
GROUP BY cb.census_block_id, cb.population, cb.boundary_polygon
ORDER BY population_density_per_km2 DESC;
```

***

## IoT & Sensor Networks

### **Environmental Monitoring System**

#### **Air Quality Analysis**

```json theme={null}
{
  "schemaName": "air_quality_sensors",
  "dimensionFieldSpecs": [
    {
      "name": "sensor_id",
      "dataType": "STRING"
    },
    {
      "name": "sensor_location", 
      "dataType": "BYTES",
      "transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"
    },
    {
      "name": "region",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "pm25_level",
      "dataType": "DOUBLE"
    },
    {
      "name": "pm10_level", 
      "dataType": "DOUBLE"
    },
    {
      "name": "ozone_level",
      "dataType": "DOUBLE"
    },
    {
      "name": "temperature_celsius",
      "dataType": "DOUBLE"
    }
  ]
}
```

**Air Quality Analysis Queries:**

```sql theme={null}
-- Find air quality patterns and pollution hotspots  
WITH air_quality_analysis AS (
  SELECT 
    sensor_id,
    sensor_location,
    region,
    AVG(pm25_level) as avg_pm25,
    AVG(pm10_level) as avg_pm10,
    AVG(ozone_level) as avg_ozone,
    COUNT(*) as reading_count,
    CASE 
      WHEN AVG(pm25_level) > 35 THEN 'Unhealthy'
      WHEN AVG(pm25_level) > 12 THEN 'Moderate'
      ELSE 'Good'
    END as air_quality_category
  FROM air_quality_sensors 
  WHERE timestamp >= now() - INTERVAL '24' HOUR
  GROUP BY sensor_id, sensor_location, region
),
pollution_clusters AS (
  SELECT 
    aqa.sensor_id,
    aqa.region,
    aqa.avg_pm25,
    aqa.air_quality_category,
    COUNT(nearby.sensor_id) as nearby_unhealthy_sensors
  FROM air_quality_analysis aqa
  LEFT JOIN air_quality_analysis nearby 
    ON ST_Distance(aqa.sensor_location, nearby.sensor_location) < 2000
    AND nearby.air_quality_category = 'Unhealthy'
    AND aqa.sensor_id != nearby.sensor_id
  GROUP BY aqa.sensor_id, aqa.region, aqa.avg_pm25, aqa.air_quality_category
)
SELECT 
  region,
  air_quality_category,
  COUNT(*) as sensor_count,
  AVG(avg_pm25) as regional_avg_pm25,
  MAX(nearby_unhealthy_sensors) as max_pollution_cluster_size
FROM pollution_clusters
GROUP BY region, air_quality_category
ORDER BY regional_avg_pm25 DESC;
```

#### **Smart City Traffic Optimization**

```sql theme={null}
-- Analyze traffic patterns and optimize signal timing
WITH traffic_analysis AS (
  SELECT 
    ts.intersection_id,
    ts.sensor_location,
    DATE_TRUNC('hour', ts.timestamp) as hour_window,
    AVG(ts.vehicle_count) as avg_vehicles_per_hour,
    AVG(ts.average_speed_kmh) as avg_speed,
    COUNT(CASE WHEN ts.congestion_level = 'HIGH' THEN 1 END) as high_congestion_periods
  FROM traffic_sensors ts
  WHERE ts.timestamp >= now() - INTERVAL '7' DAY
  GROUP BY ts.intersection_id, ts.sensor_location, DATE_TRUNC('hour', ts.timestamp)
),
congestion_impact AS (
  SELECT 
    ta.intersection_id,
    ta.hour_window,
    ta.avg_vehicles_per_hour,
    ta.avg_speed,
    -- Find nearby intersections affected by congestion
    COUNT(nearby.intersection_id) as affected_nearby_intersections,
    AVG(nearby.avg_speed) as nearby_avg_speed
  FROM traffic_analysis ta
  LEFT JOIN traffic_analysis nearby 
    ON ST_Distance(ta.sensor_location, nearby.sensor_location) < 1000
    AND ta.hour_window = nearby.hour_window
    AND ta.intersection_id != nearby.intersection_id
  WHERE ta.high_congestion_periods > 0
  GROUP BY ta.intersection_id, ta.hour_window, ta.avg_vehicles_per_hour, ta.avg_speed
)
SELECT 
  intersection_id,
  EXTRACT(hour FROM hour_window) as hour_of_day,
  AVG(avg_vehicles_per_hour) as peak_vehicle_count,
  AVG(avg_speed) as intersection_speed,
  AVG(nearby_avg_speed) as area_speed,
  AVG(affected_nearby_intersections) as congestion_spread,
  -- Recommend signal timing adjustments
  CASE 
    WHEN AVG(avg_speed) < 20 AND AVG(affected_nearby_intersections) > 3 THEN 'Increase green time'
    WHEN AVG(avg_speed) > 50 AND AVG(avg_vehicles_per_hour) < 100 THEN 'Reduce green time'
    ELSE 'Current timing OK'
  END as recommendation
FROM congestion_impact
GROUP BY intersection_id, EXTRACT(hour FROM hour_window)
ORDER BY intersection_id, hour_of_day;
```

## Advanced Spatial Analytics Patterns

### **Spatial Clustering Analysis**

**Find Business Clusters:**

```sql theme={null}
-- Identify business clustering patterns using spatial density
WITH business_density AS (
  SELECT 
    b1.business_id,
    b1.business_name,
    b1.category,
    b1.location,
    COUNT(b2.business_id) as nearby_businesses,
    COUNT(CASE WHEN b2.category = b1.category THEN 1 END) as same_category_nearby
  FROM businesses b1 
  LEFT JOIN businesses b2 
    ON ST_Distance(b1.location, b2.location) < 500
    AND b1.business_id != b2.business_id
  GROUP BY b1.business_id, b1.business_name, b1.category, b1.location
),
cluster_analysis AS (
  SELECT 
    category,
    AVG(nearby_businesses) as avg_business_density,
    AVG(same_category_nearby) as avg_category_clustering,
    COUNT(*) as total_businesses,
    STDDEV(nearby_businesses) as density_variation
  FROM business_density
  GROUP BY category
)
SELECT 
  category,
  total_businesses,
  ROUND(avg_business_density, 2) as avg_nearby_businesses,
  ROUND(avg_category_clustering, 2) as category_clustering_index,
  ROUND(density_variation, 2) as clustering_consistency,
  CASE 
    WHEN avg_category_clustering > avg_business_density * 0.5 THEN 'Highly Clustered'
    WHEN avg_category_clustering > avg_business_density * 0.3 THEN 'Moderately Clustered'
    ELSE 'Dispersed'
  END as clustering_pattern
FROM cluster_analysis
ORDER BY category_clustering_index DESC;
```

### **Geospatial Time Series Analysis**

**Temporal-Spatial Pattern Detection:**

```sql theme={null}
-- Analyze how spatial patterns change over time
WITH hourly_patterns AS (
  SELECT 
    EXTRACT(hour FROM timestamp) as hour_of_day,
    EXTRACT(dow FROM timestamp) as day_of_week,
    ST_Distance(event_location, ST_Point(-122.4194, 37.7749)) as distance_from_center,
    event_type,
    COUNT(*) as event_count
  FROM events 
  WHERE timestamp >= now() - INTERVAL '30' DAY
  GROUP BY 
    EXTRACT(hour FROM timestamp),
    EXTRACT(dow FROM timestamp), 
    ST_Distance(event_location, ST_Point(-122.4194, 37.7749)),
    event_type
),
pattern_analysis AS (
  SELECT 
    hour_of_day,
    day_of_week,
    event_type,
    AVG(distance_from_center) as avg_distance_from_center,
    SUM(event_count) as total_events,
    STDDEV(distance_from_center) as spatial_spread
  FROM hourly_patterns
  GROUP BY hour_of_day, day_of_week, event_type
)
SELECT 
  event_type,
  hour_of_day,
  CASE 
    WHEN day_of_week IN (0, 6) THEN 'Weekend'
    ELSE 'Weekday'
  END as day_type,
  total_events,
  ROUND(avg_distance_from_center, 0) as avg_distance_from_center,
  ROUND(spatial_spread, 0) as spatial_dispersion,
  CASE 
    WHEN avg_distance_from_center < 2000 THEN 'Central'
    WHEN avg_distance_from_center < 5000 THEN 'Urban'
    ELSE 'Suburban'
  END as spatial_pattern
FROM pattern_analysis
WHERE total_events > 10
ORDER BY event_type, day_type, hour_of_day;
```

## Performance Optimization Examples

### **Query Optimization Patterns**

**Efficient Multi-Location Analysis:**

```sql theme={null}
-- Optimized approach for analyzing multiple locations
WITH locations_of_interest AS (
  SELECT * FROM (VALUES 
    ('Downtown SF', -122.4194, 37.7749),
    ('Mission District', -122.4194, 37.7589),
    ('SOMA', -122.3997, 37.7879)
  ) AS locations(area_name, longitude, latitude)
),
location_analysis AS (
  SELECT 
    loi.area_name,
    s.store_name,
    s.category,
    ST_Distance(s.location, ST_Point(loi.longitude, loi.latitude)) as distance
  FROM locations_of_interest loi
  CROSS JOIN stores s
  WHERE ST_Distance(s.location, ST_Point(loi.longitude, loi.latitude)) < 2000
)
SELECT 
  area_name,
  category,
  COUNT(*) as store_count,
  AVG(distance) as avg_distance,
  MIN(distance) as nearest_store_distance
FROM location_analysis
GROUP BY area_name, category
ORDER BY area_name, store_count DESC;
```

**Hierarchical Spatial Filtering:**

```sql theme={null}
-- Use coarse-to-fine filtering for large datasets
WITH coarse_filter AS (
  -- First pass: larger radius for initial filtering
  SELECT store_id, location, category 
  FROM stores 
  WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 10000
),
fine_filter AS (
  -- Second pass: precise filtering on reduced dataset
  SELECT 
    store_id,
    location,
    category,
    ST_Distance(location, ST_Point(-122.4194, 37.7749)) as exact_distance
  FROM coarse_filter
  WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000
)
SELECT 
  category,
  COUNT(*) as matching_stores,
  AVG(exact_distance) as avg_distance,
  MIN(exact_distance) as nearest_distance
FROM fine_filter
GROUP BY category
ORDER BY nearest_distance;
```

## Next Steps & Integration

### **API Integration Patterns**

**GeoJSON Output for Web Applications:**

```sql theme={null}
-- Format results for web mapping applications
SELECT json_object(
  'type', 'FeatureCollection',
  'features', json_arrayagg(
    json_object(
      'type', 'Feature',
      'geometry', json(ST_AsGeoJson(location)),
      'properties', json_object(
        'store_name', store_name,
        'category', category,
        'rating', rating,
        'distance', ST_Distance(location, ST_Point(-122.4194, 37.7749))
      )
    )
  )
) as geojson_result
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000
  AND rating >= 4.0;
```

### **Implementation Guide**

1. **Choose Your Use Case**: Start with the example most similar to your requirements
2. **Adapt Schema**: Modify the schema to match your data structure
3. **Configure Indexes**: Set appropriate H3 resolutions based on query patterns
4. **Test Queries**: Validate performance using EXPLAIN PLAN
5. **Monitor & Optimize**: Use performance monitoring to fine-tune configuration


# GeoSpatial Functions Reference
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/geospatial-functions

SQL/MM compliant spatial functions for creating, measuring, and analyzing geographic data with practical examples.

Apache Pinot provides a comprehensive set of geospatial functions that follow the SQL/MM specification with the `ST_` prefix. These functions enable you to create, manipulate, measure, and analyze spatial data.

## Function Categories

> **[Constructors](#constructors)**
>
> Create geospatial objects from coordinates, WKT, WKB, and GeoJSON

> **[Measurements](#measurements)**
>
> Calculate distances, areas, and geometric properties

> **[Relationships](#relationships)**
>
> Test spatial relationships like containment and intersection

> **[Outputs](#outputs)**
>
> Convert geospatial objects to various formats

> **[Conversions](#conversions)**
>
> Transform between geometry and geography types

> **[Aggregations](#aggregations)**
>
> Combine multiple geometries into collections

***

## Constructors

Functions for creating geospatial objects from various input formats.

### **ST\_Point**

Creates a point geometry from X,Y coordinates.

```sql theme={null}
ST_Point(double x, double y) → Point
```

**Examples:**

```sql theme={null}
-- Create San Francisco point
SELECT ST_Point(-122.4194, 37.7749) as sf_point;

-- Use in query to find nearby locations
SELECT store_name 
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 1000;
```

### **ST\_GeomFromText**

Creates geometry from Well-Known Text (WKT) representation.

```sql theme={null}
ST_GeomFromText(String wkt) → Geometry
```

**Examples:**

```sql theme={null}
-- Create various geometry types
SELECT ST_GeomFromText('POINT(-122.4194 37.7749)') as point_geom;
SELECT ST_GeomFromText('LINESTRING(-122.4194 37.7749, -122.4094 37.7849)') as line_geom;
SELECT ST_GeomFromText('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))') as poly_geom;
```

### **ST\_GeomFromWKB**

Creates geometry from Well-Known Binary (WKB) representation.

```sql theme={null}
ST_GeomFromWKB(bytes wkb) → Geometry
```

**Examples:**

```sql theme={null}
-- Create geometry from stored WKB data
SELECT ST_GeomFromWKB(wkb_column) FROM spatial_table;

-- Round-trip conversion
SELECT ST_GeomFromWKB(ST_AsBinary(ST_Point(-122.4194, 37.7749)));
```

### **ST\_Polygon**

Creates a polygon geometry from WKT representation.

```sql theme={null}
ST_Polygon(String wkt) → Polygon
```

**Examples:**

```sql theme={null}
-- Create a simple rectangle
SELECT ST_Polygon('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))');

-- Use in spatial queries
SELECT * FROM locations 
WHERE ST_Within(point_column, ST_Polygon('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))'));
```

### **ST\_GeogFromText**

Creates geography from Well-Known Text with spherical calculations.

```sql theme={null}
ST_GeogFromText(String wkt) → Geography
```

**Examples:**

```sql theme={null}
-- Create geography for accurate distance calculations
SELECT ST_GeogFromText('POINT(-122.4194 37.7749)') as sf_geography;

-- Use with ST_Distance for meter-based results
SELECT ST_Distance(
  ST_GeogFromText('POINT(-122.4194 37.7749)'),
  ST_GeogFromText('POINT(-74.0060 40.7128)')
) as nyc_to_sf_meters;
```

### **ST\_GeogFromWKB**

Creates geography from Well-Known Binary representation.

```sql theme={null}
ST_GeogFromWKB(bytes wkb) → Geography
```

**Examples:**

```sql theme={null}
-- Create geography from WKB data column
SELECT ST_GeogFromWKB(wkb_geography_column) FROM geo_table;

-- Convert point to WKB and back to geography
SELECT ST_Point(-122, 37) AS point,
       ST_GeogFromWKB(
         ST_AsBinary(ST_Point(-122, 37))
       ) AS geography_from_wkb
FROM ignoreMe;
```

### **ST\_GeomFromGeoJson**

Creates geometry from GeoJSON string.

```sql theme={null}
ST_GeomFromGeoJson(string geojson) → Geometry
```

**Examples:**

```sql theme={null}
-- Create point from GeoJSON
SELECT ST_GeomFromGeoJson('{"type":"Point","coordinates":[-122.4194,37.7749]}');

-- Create polygon from GeoJSON
SELECT ST_GeomFromGeoJson('{
  "type":"Polygon",
  "coordinates":[[
    [-122.5,37.7],[-122.4,37.7],[-122.4,37.8],[-122.5,37.8],[-122.5,37.7]
  ]]
}');

-- Handle Feature objects
SELECT ST_GeomFromGeoJson('{"type":"Feature","geometry":{"type":"Point","coordinates":[-122.4194,37.7749]}}');
```

### **ST\_GeogFromGeoJson**

Creates geography from GeoJSON string.

```sql theme={null}
ST_GeogFromGeoJson(string geojson) → Geography
```

**Examples:**

```sql theme={null}
-- Create geography for accurate calculations
SELECT ST_GeogFromGeoJson('{"type":"Point","coordinates":[-122.4194,37.7749]}');

-- Use in distance calculations
SELECT ST_Distance(
  location_geography,
  ST_GeogFromGeoJson('{"type":"Point","coordinates":[-122.4194,37.7749]}')
) as distance_meters FROM stores;
```

***

## Measurements

Functions for calculating spatial measurements and properties.

### **ST\_Distance**

Calculates distance between two geometries or geographies.

```sql theme={null}
ST_Distance(Geometry/Geography g1, Geometry/Geography g2) → double
```

**For Geometry**: Returns 2D Cartesian distance in coordinate units\
**For Geography**: Returns great-circle distance in meters

> [!WARNING]
>
> **Geography Limitation**: When using geography types, `ST_Distance` only accepts **POINT** geometries. For other geometry types (LINESTRING, POLYGON), use geometry types instead.

**Examples:**

```sql theme={null}
-- Distance with geometry types (any geometry supported)
SELECT ST_Distance(
  ST_GeomFromText('LINESTRING(-122.4194 37.7749, -122.4094 37.7849)'),
  ST_GeomFromText('POINT(-74.0060 40.7128)')
) as coordinate_distance;

-- Distance with geography types (points only)
SELECT ST_Distance(
  ST_GeogFromText('POINT(-122.4194 37.7749)'),
  ST_GeogFromText('POINT(-74.0060 40.7128)')
) as meters_distance;

-- Find nearby stores (indexed query)
SELECT store_name, address
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000
ORDER BY ST_Distance(location, ST_Point(-122.4194, 37.7749));
```

### **ST\_Area**

Calculates the area of a polygon geometry or geography.

```sql theme={null}
ST_Area(Geometry/Geography g) → double
```

**For Geometry**: Returns 2D Euclidean area in coordinate units²\
**For Geography**: Returns area in square meters using spherical model

**Examples:**

```sql theme={null}
-- Area of a polygon in coordinate units²
SELECT ST_Area(ST_GeomFromText('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))'));

-- Area in square meters using geography
SELECT ST_Area(ST_GeogFromText('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))'));

-- Calculate area of all regions
SELECT region_name, ST_Area(boundary_geography) as area_sqm 
FROM geographic_regions 
ORDER BY area_sqm DESC;
```

### **ST\_GeometryType**

Returns the type of a geometry or geography as a string.

```sql theme={null}
ST_GeometryType(Geometry/Geography g) → String
```

**Examples:**

```sql theme={null}
-- Works with geometry types
SELECT ST_GeometryType(ST_Point(-122.4194, 37.7749));  -- Returns: ST_Point
SELECT ST_GeometryType(ST_GeomFromText('LINESTRING(0 0, 1 1)'));  -- Returns: ST_LineString
SELECT ST_GeometryType(ST_GeomFromText('POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))'));  -- Returns: ST_Polygon

-- Works with geography types  
SELECT ST_GeometryType(ST_GeogFromText('POINT(-122.4194 37.7749)'));  -- Returns: ST_Point

-- Filter by geometry type
SELECT * FROM mixed_geometries 
WHERE ST_GeometryType(geom_column) = 'ST_Point';
```

***

## Relationships

Functions for testing spatial relationships between geometries.

### **ST\_Contains**

Tests if the first geometry completely contains the second.

```sql theme={null}
ST_Contains(Geometry/Geography g1, Geometry/Geography g2) → boolean
```

Returns true if no points of g2 lie outside g1, and at least one interior point of g1 lies in the interior of g2.

> [!WARNING]
>
> **Note**: ST\_Contains on Geography types provides close approximation, not exact results.

**Examples:**

```sql theme={null}
-- Check if polygon contains points
SELECT COUNT(*) as points_in_region
FROM locations 
WHERE ST_Contains(
  ST_GeomFromText('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))'),
  location_point
);

-- Boundary vs Interior distinction
-- Returns true: linestring goes from boundary to interior
SELECT ST_Contains(
  ST_GeogFromText('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))'),
  ST_GeogFromText('LINESTRING(-122.5 37.7, -122.45 37.75)')
); -- Returns: 1 (true)

-- Returns false: linestring lies exactly on boundary 
SELECT ST_Contains(
  ST_GeogFromText('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))'),
  ST_GeogFromText('LINESTRING(-122.5 37.7, -122.4 37.7)')
); -- Returns: 0 (false)

-- Find all stores within a delivery zone
SELECT store_name, address
FROM stores 
WHERE ST_Contains(delivery_zone_polygon, store_location);

-- **Index Usage**: First argument must be literal, second must be column
WHERE ST_Contains('', point_column)  -- Uses index
```

### **ST\_Within**

Tests if the first geometry is completely within the second.

```sql theme={null}
ST_Within(Geometry/Geography g1, Geometry/Geography g2) → boolean
```

Returns true if every point of g1 is a point of g2, and the interiors of the two geometries have at least one point in common.

**Examples:**

```sql theme={null}
-- Find points within a region
SELECT location_name 
FROM locations 
WHERE ST_Within(
  location_point,
  ST_GeomFromText('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))')
);

-- Check if delivery addresses are within service area
SELECT customer_id, address
FROM customers 
WHERE ST_Within(delivery_address, service_boundary);

-- **Index Usage**: First argument must be column, second must be literal
WHERE ST_Within(point_column, '')  -- Uses index
```

### **ST\_Equals**

Tests if two geometries or geographies represent the same geometry.

```sql theme={null}
ST_Equals(Geometry/Geography g1, Geometry/Geography g2) → boolean
```

Returns true if g1 and g2 have at least one point in common, and no point of either lies in the exterior of the other.

**Examples:**

```sql theme={null}
-- Check if two geometries are identical
SELECT ST_Equals(
  ST_GeomFromText('POINT(-122.4194 37.7749)'),
  ST_Point(-122.4194, 37.7749)
);  -- Returns: true

-- Works with geography types
SELECT ST_Equals(
  ST_GeogFromText('POINT(-122.4194 37.7749)'),
  ST_GeogFromText('POINT(-122.4194 37.7749)')
);  -- Returns: true

-- Find duplicate locations
SELECT COUNT(*) as duplicate_count
FROM locations l1, locations l2 
WHERE l1.id < l2.id 
  AND ST_Equals(l1.location, l2.location);
```

***

## Outputs

Functions for converting geospatial objects to various output formats.

### **ST\_AsText**

Converts geometry or geography to Well-Known Text (WKT).

```sql theme={null}
ST_AsText(Geometry/Geography g) → string
```

**Examples:**

```sql theme={null}
-- Convert geometry to readable text
SELECT ST_AsText(location_column) as wkt_location FROM spatial_data;

-- Debug spatial queries
SELECT store_name, ST_AsText(location) as location_wkt
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 1000;
```

### **ST\_AsBinary**

Converts geometry or geography to Well-Known Binary (WKB).

```sql theme={null}
ST_AsBinary(Geometry/Geography g) → bytes
```

**Examples:**

```sql theme={null}
-- Convert to binary format for storage
SELECT ST_AsBinary(location_geom) as wkb_data FROM locations;

-- Export binary data
SELECT id, ST_AsBinary(boundary) as boundary_wkb 
FROM regions 
WHERE region_type = 'administrative';
```

### **ST\_AsGeoJson**

Converts geometry or geography to GeoJSON format.

```sql theme={null}
ST_AsGeoJson(Geometry/Geography g) → string
```

**Examples:**

```sql theme={null}
-- Convert point to GeoJSON
SELECT ST_AsGeoJson(ST_Point(-122.4194, 37.7749));
-- Returns: {"type":"Point","coordinates":[-122.4194,37.7749],"crs":{"type":"name","properties":{"name":"EPSG:0"}}}

-- Export data for web applications
SELECT 
  store_name,
  ST_AsGeoJson(location) as geojson_location,
  category
FROM stores 
WHERE city = 'San Francisco';

-- Create GeoJSON FeatureCollection-ready output
SELECT json_object(
  'type', 'Feature',
  'geometry', json(ST_AsGeoJson(location)),
  'properties', json_object('name', store_name, 'category', category)
) as geojson_feature
FROM stores;
```

***

## Conversions

Functions for converting between geometry and geography types.

### **toSphericalGeography**

Converts a Geometry object to spherical geography.

```sql theme={null}
toSphericalGeography(Geometry g) → Geography
```

**Examples:**

```sql theme={null}
-- Convert geometry to geography for accurate distance calculations
SELECT toSphericalGeography(ST_GeomFromText('POINT(-122.4194 37.7749)'));

-- Use in transform functions
"transformFunction": "toSphericalGeography(stPoint(longitude, latitude))"

-- Convert existing geometry column
SELECT 
  location_name,
  ST_Distance(
    toSphericalGeography(geometry_location),
    toSphericalGeography(ST_Point(-122.4194, 37.7749))
  ) as distance_meters
FROM locations;
```

### **toGeometry**

Converts a spherical geography object to geometry.

```sql theme={null}
toGeometry(Geography g) → Geometry
```

**Examples:**

```sql theme={null}
-- Convert geography back to geometry
SELECT toGeometry(geography_column) FROM spatial_data;

-- Use when you need faster calculations over small areas
SELECT location_name
FROM locations 
WHERE ST_Distance(
  toGeometry(geography_location),
  toGeometry(reference_geography)
) < 0.01;  -- Approximate distance in degrees
```

***

## Aggregations

Functions for combining multiple geometries.

### **ST\_Union**

Combines multiple geometries into a single MULTI geometry.

```sql theme={null}
ST_Union(geometry[] g1_array) → Geometry
```

This aggregate function returns a MULTI geometry or NON-MULTI geometry from a set of geometries. It ignores NULL geometries.

**Examples:**

```sql theme={null}
-- Combine all store locations into a MultiPoint
SELECT ST_Union(ARRAY_AGG(location)) as all_store_locations
FROM stores 
WHERE city = 'San Francisco';

-- Create union of all delivery zones
SELECT region, ST_Union(ARRAY_AGG(delivery_polygon)) as combined_delivery_area
FROM delivery_zones 
GROUP BY region;

-- Combine geometries by category
SELECT 
  store_category,
  ST_AsText(ST_Union(ARRAY_AGG(location))) as combined_locations
FROM stores 
GROUP BY store_category;
```

***

## Function Usage with Indexes

### **Functions that Use H3 Index**

These functions automatically leverage geospatial indexes when properly configured:

| Function | Index Usage | Requirements |
| - | - | - |
| `ST_Distance` | FILTER\_H3\_INDEX | Column as first arg, literal as second, used in range condition |
| `ST_Within` | INCLUSION\_FILTER\_H3\_INDEX | Column as first arg, literal as second |
| `ST_Contains` | INCLUSION\_FILTER\_H3\_INDEX | Literal as first arg, column as second |

**Index-Optimized Query Patterns:**

```sql theme={null}
-- These queries use the H3 index:
WHERE ST_Distance(location_column, ST_Point(-122, 37)) < 5000
WHERE ST_Within(location_column, ST_GeomFromText('POLYGON(...)'))  
WHERE ST_Contains(ST_GeomFromText('POLYGON(...)'), location_column)

-- These queries do NOT use the index:
WHERE ST_Distance(ST_Point(-122, 37), location_column) < 5000  -- Arguments reversed
WHERE ST_Within(ST_GeomFromText('POLYGON(...)'), location_column)  -- Arguments reversed
```

### **Checking Index Usage**

Use `EXPLAIN PLAN FOR` to verify index usage:

```sql theme={null}
EXPLAIN PLAN FOR 
SELECT * FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;
```

Look for these operators in the query plan:

* `FILTER_H3_INDEX` - ST\_Distance queries using index
* `INCLUSION_FILTER_H3_INDEX` - ST\_Within/ST\_Contains queries using index

***

## Best Practices

### **Performance Tips**

* **Use geography types** for global applications requiring meter-based distances
* **Use geometry types** for regional analysis where coordinate units are acceptable
* **Index your spatial columns** with appropriate H3 resolutions
* **Structure queries** to leverage indexes (proper argument order)

### **Function Selection Guide**

```sql theme={null}
-- For distance-based queries
ST_Distance + geography types = Most accurate (points only)
ST_Distance + geometry types = Faster, accepts any geometry type

-- For containment queries  
ST_Within = Point-in-polygon testing
ST_Contains = Polygon-contains-point testing (reverse of ST_Within)

-- For data export
ST_AsGeoJson = Modern web applications
ST_AsText = Debugging and human-readable output
ST_AsBinary = Compact storage and transmission
```


# GeoSpatial Overview
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/geospatial-index

H3 hexagonal indexing, multiple data formats, and spatial functions for real-time location analytics.

Build location-based analytics with geospatial support powered by Apache Pinot's spatial indexing capabilities. Query geographic data at scale with sub-second performance using H3 hexagonal indexing.

## What is GeoSpatial Support?

GeoSpatial support enables you to:

* **Store and query location data** efficiently using industry-standard formats
* **Perform spatial operations** like distance calculations, containment checks, and area measurements
* **Accelerate spatial queries** using H3 hexagonal indexing for sub-second performance
* **Analyze spatiotemporal data** by combining location and time-series queries

## Key Capabilities

### **GeoSpatial Data Types**

* **Geometry**: For Cartesian plane calculations (meters, feet, etc.)
* **Geography**: For spherical Earth calculations (longitude/latitude)
* **Format Support**: WKT (Well-Known Text), WKB (Well-Known Binary), and **GeoJSON**

### **Spatial Operations**

* Distance calculations (`ST_Distance`)
* Containment relationships (`ST_Within`, `ST_Contains`)
* Area and measurement functions (`ST_Area`)
* Geometric constructors and converters

### **H3 Hexagonal Indexing**

* Hierarchical hexagonal grid system
* Multiple resolution levels for different precision needs
* Dramatically improves query performance
* Industry-standard spatial indexing approach

## Sample Query

Here's a simple example of a geospatial query to find nearby stores:

```sql theme={null}
SELECT store_name, address, 
       ST_Distance(location, ST_Point(-122.4194, 37.7749)) as distance_meters
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000
ORDER BY distance_meters
LIMIT 10;
```

This query finds all stores within 5km of San Francisco city center, using the H3 index for fast filtering.

## Common Use Cases

### **Location-Based Services**

* Find nearby restaurants, stores, or services
* Delivery radius calculations
* Service area analysis

### **Fleet & Asset Tracking**

* Vehicle location monitoring
* Route optimization
* Geofencing alerts

### **Real Estate & Urban Planning**

* Property search by location
* Market analysis by geographic region
* Demographic studies

### **IoT & Sensor Data**

* Environmental monitoring by location
* Smart city analytics
* Weather and climate analysis

## Performance Benefits

With proper geospatial indexing, you can expect:

* **10-100x faster** spatial queries compared to non-indexed queries
* **Sub-second response times** for complex spatial operations on large datasets
* **Scalable performance** across distributed Pinot clusters
* **Efficient memory usage** through H3's hierarchical structure

## Explore GeoSpatial Documentation

> **[Data Types & Formats](/corecapabilities/manage-data/indexes/geospatial-data-types)**
>
> Learn about geometry vs geography types, WKT, WKB, and GeoJSON format support

> **[Functions Reference](/corecapabilities/manage-data/indexes/geospatial-functions)**
>
> Complete catalog of geospatial functions including constructors, measurements, and relationships

> **[Index Configuration](/corecapabilities/manage-data/indexes/geospatial-configuration)**
>
> H3 index setup, resolution selection, and performance configuration

> **[Performance & Optimization](/corecapabilities/manage-data/indexes/geospatial-performance)**
>
> Query optimization, performance tuning, and best practices

> **[Examples & Use Cases](/corecapabilities/manage-data/indexes/geospatial-examples)**
>
> Real-world examples, common patterns, and complete implementations


# GeoSpatial Performance & Optimization
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/geospatial-performance

Query plan analysis, H3 index tuning, and optimization techniques for high-performance spatial queries.

Optimize geospatial query performance with Apache Pinot. This guide covers everything from basic performance principles to tuning techniques.

> **[Query Plan Analysis](#query-plan-analysis)**
>
> Use EXPLAIN PLAN to understand and verify index usage

> **[Index Optimization](#index-optimization)**
>
> Configure H3 resolutions for optimal performance

> **[Query Patterns](#optimized-query-patterns)**
>
> Structure queries to use geospatial indexes effectively

> **[Performance Monitoring](#performance-monitoring)**
>
> Monitor and measure geospatial query performance

## Performance Fundamentals

### **How H3 Index Acceleration Works**

Understanding the two-phase optimization approach is key to writing efficient queries:

**1. Coarse Filtering (H3 Index)**

The H3 index quickly identifies candidate records within nearby hexagons, eliminating the vast majority of data from consideration.

**2. Precise Filtering (Geospatial Functions)**

Only records in candidate hexagons undergo expensive precise geospatial calculations (ST\_Distance, ST\_Within, etc.).

**Example: Finding locations within 5km**

* **Without Index**: Calculate exact distance for ALL records (expensive)
* **With H3 Index**:
  1. Find hexagon containing given point
  2. Find hexagons fully contained within the searched area
  3. Include all points associated with those hexagons
  4. Find hexagons overlapping searched area
  5. Calculate exact distance for records in those hexagons

**Important**: If query distance is >100x the hexagon size, the index is automatically bypassed and falls back to full scan for performance reasons.

## Query Plan Analysis

### **Using EXPLAIN PLAN FOR**

The most important tool for geospatial performance optimization:
Single-stage query engine (SSQE):

```sql theme={null}
EXPLAIN PLAN FOR
SELECT store_name, address, 
       ST_Distance(location, ST_Point(-122.4194, 37.7749)) as distance
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000
ORDER BY distance;
```

Multi-stage query engine MSQE:

```sql theme={null}
set explainAskingServers=true;
EXPLAIN PLAN FOR
SELECT store_name, address,
       ST_Distance(location, ST_Point(-122.4194, 37.7749)) as distance
FROM stores
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000
ORDER BY distance;
```

Setting `explainAskingServers` query option is necessary for MSQE to show usage of segment-level indexes.

### **Index Usage Operators**

Look for these operators in your query plans:

**FILTER\_H3\_INDEX**

Shows as `FilterH3Index` in MSQE.
**When**: ST\_Distance with range conditions

```sql theme={null}
WHERE ST_Distance(location_column, literal_point) < distance
WHERE ST_Distance(location_column, literal_point) > distance  
WHERE ST_Distance(location_column, literal_point) BETWEEN min_dist AND max_dist
```

**Query Plan Example**:
SSQE:

```
BROKER_REDUCE(limit:1000)
└── COMBINE_SELECT
    └── SELECT(selectList:[store_name, address, distance])
        └── TRANSFORM_PASSTHROUGH(distance:ST_DISTANCE(...))
            └── FILTER_H3_INDEX(predicate:ST_DISTANCE(...) < 5000)
                └── SEGMENT_SCAN(table:stores)
```

MSQE:

```
PinotLogicalExchange(distribution=[broadcast])
  LeafStageCombineOperator(table=[stores])
    StreamingInstanceResponse
      StreamingCombineSelect
        SelectStreaming(segment=[stores_OFFLINE_0], table=[starbucksStores])
          Project(columns=[[store_name, address, distance]])
            DocIdSet(maxDocs=[10000])
              FilterH3Index(predicate=[stdistance(...) < '5000.0'], indexLookUp=[h3_index], operator=[RANGE])
```

**INCLUSION\_FILTER\_H3\_INDEX**

Shows as InclusionFilterH3Index in MSQE.
**When**: ST\_Within and ST\_Contains functions

```sql theme={null}
WHERE ST_Within(location_column, literal_polygon)
WHERE ST_Contains(literal_polygon, location_column)
```

**Query Plan Example**:
SSQE:

```
BROKER_REDUCE(limit:1000)
└── COMBINE_SELECT
    └── SELECT(selectList:[store_name, address])
        └── INCLUSION_FILTER_H3_INDEX(predicate:ST_WITHIN(...))
            └── SEGMENT_SCAN(table:stores)
```

MSQE:

```
PinotLogicalExchange(distribution=[broadcast])
  LeafStageCombineOperator(table=[stores])
    StreamingInstanceResponse
      StreamingCombineSelect
        SelectStreaming(segment=[stores_OFFLINE_0], table=[stores])
          Project(columns=[[store_name, address]])
            DocIdSet(maxDocs=[10000])
              InclusionFilterH3Index(predicate=[stcontains(...) = '1'], operator=[EQ])
```

**No Index Usage**

**When**: Index not being used (performance problem!)

```
BROKER_REDUCE(limit:1000)
└── COMBINE_SELECT
    └── SELECT(selectList:[store_name, address])
        └── FILTER(predicate:ST_DISTANCE(...) < 5000)  -- No H3!
            └── SEGMENT_SCAN(table:stores)
```

**Common Causes**:

* Arguments in wrong order
* Missing H3 index configuration
* Dictionary encoding not disabled

### **EXPLAIN PLAN Examples**

Here are real examples of what you should see when H3 indexes are working:

**FILTER\_H3\_INDEX Example (ST\_Distance)**:

```sql theme={null}
EXPLAIN PLAN FOR
SELECT * FROM starbucksStores
WHERE ST_Distance(location_st_point, ST_Point(-122.4194, 37.7749)) < 5000;
```

**Output**:

```
BROKER_REDUCE(limit:10)
COMBINE_SELECT
PLAN_START(numSegmentsForThisPlan:1)
SELECT(selectList:address, lat, location_st_point, lon, name)
PROJECT(name, lon, address, location_st_point, lat)
DOC_ID_SET
FILTER_H3_INDEX
```

**INCLUSION\_FILTER\_H3\_INDEX Example (ST\_Contains)**:

```sql theme={null}
EXPLAIN PLAN FOR
SELECT * FROM starbucksStores
WHERE ST_Contains(ST_GeomFromText('POLYGON((0 0, 4 0, 4 4, 0 4, 0 0))'), location_st_point);
```

**Output**:

```
BROKER_REDUCE(limit:10)
COMBINE_SELECT
PLAN_START(numSegmentsForThisPlan:1)
SELECT(selectList:address, lat, location_st_point, lon, name)
PROJECT(name, lon, address, location_st_point, lat)
DOC_ID_SET
INCLUSION_FILTER_H3_INDEX
```

**Key Indicators**:

* `FILTER_H3_INDEX` for distance-based queries
* `INCLUSION_FILTER_H3_INDEX` for containment queries
* `DOC_ID_SET` showing efficient document filtering
* No `FILTER` operator (which would indicate full scan)

## Index Optimization

### **Resolution Selection Strategy**

Choose H3 resolutions based on your query patterns and performance requirements:

#### **Single Resolution Approach**

**When to use**: Uniform query patterns with consistent search radius

```json theme={null}
{
  "indexes": {
    "h3": {
      "resolution": [8]  // ~500m precision, good for city-wide queries
    }
  }
}
```

**Query Performance**:

* Excellent for queries around the target resolution
* Suboptimal for very different search radii

#### **Multi-Resolution Approach** (Recommended)

**When to use**: Varied query patterns with different search radii

```json theme={null}
{
  "indexes": {
    "h3": {
      "resolution": [5, 8, 11]  // Coarse, medium, fine
    }
  }
}
```

**Resolution Selection Algorithm**:

* **Resolution 5** (8km): For queries >20km radius
* **Resolution 8** (500m): For queries 1-20km radius
* **Resolution 11** (25m): For queries \<1km radius

### **Performance vs Storage Analysis**

| Resolutions | Query Performance | Storage Overhead | Memory Usage | Use Case |
| - | - | - | - | - |
| \[8] | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ | Low | Uniform patterns |
| \[5, 8] | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | Medium | Most applications |
| \[5, 8, 11] | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | Medium-High | Varied patterns |
| \[3, 6, 9, 12] | ⭐⭐⭐⭐⭐ | ⭐⭐ | High | Complex analytics |
| \[1, 4, 7, 10, 13] | ⭐⭐⭐⭐ | ⭐ | Very High | Over-optimization |

### **Resolution Tuning Examples**

**Retail Store Locator**:

```json theme={null}
{
  "resolution": [7, 10]  // 1.2km and 65m
}
```

* Resolution 7: "Stores within 5km" queries
* Resolution 10: "Nearest store" precision queries

**Logistics & Delivery**:

```json theme={null}
{
  "resolution": [4, 7, 10]  // 22km, 1.2km, 65m
}
```

* Resolution 4: Regional route planning
* Resolution 7: City-wide delivery zones
* Resolution 10: Precise delivery locations

**Global IoT Network**:

```json theme={null}
{
  "resolution": [2, 5, 8, 11]  // 158km, 8km, 500m, 25m
}
```

* Resolution 2: Country/region analysis
* Resolution 5: City-level aggregation
* Resolution 8: Local area monitoring
* Resolution 11: Device-level precision

## Optimized Query Patterns

### **Distance-Based Queries**

**Index-Optimized Patterns**:

```sql theme={null}
-- Correct argument order (column first, literal second)
WHERE ST_Distance(location_column, ST_Point(-122.4194, 37.7749)) < 5000

-- Range queries work well
WHERE ST_Distance(location_column, ST_Point(-122.4194, 37.7749)) BETWEEN 1000 AND 5000

-- Use with ORDER BY for nearest neighbor
SELECT * FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 10000
ORDER BY ST_Distance(location, ST_Point(-122.4194, 37.7749))
LIMIT 10;
```

**Non-Optimized Patterns**:

```sql theme={null}
-- Wrong argument order (literal first)
WHERE ST_Distance(ST_Point(-122.4194, 37.7749), location_column) < 5000

-- Both arguments are columns (can't use index)
WHERE ST_Distance(pickup_location, dropoff_location) < 5000

-- Complex expressions in spatial function
WHERE ST_Distance(location, ST_Point(lon_column, lat_column)) < 5000
```

### **Containment Queries**

**ST\_Within Optimization**:

```sql theme={null}
-- Optimized: column first, literal second  
SELECT * FROM events
WHERE ST_Within(
  event_location,
  ST_GeomFromText('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))')
);

-- Not optimized: literal first
WHERE ST_Within(
  ST_GeomFromText('POLYGON(...)'),
  event_location
);
```

**ST\_Contains Optimization**:

```sql theme={null}
-- Optimized: literal first, column second
SELECT * FROM delivery_addresses
WHERE ST_Contains(
  ST_GeomFromText('POLYGON((-122.5 37.7, -122.4 37.7, -122.4 37.8, -122.5 37.8, -122.5 37.7))'),
  address_location
);

-- Not optimized: column first  
WHERE ST_Contains(service_area, fixed_point);
```

### **Advanced Query Optimization**

**Multiple Spatial Conditions**:

```sql theme={null}
-- Combine spatial and non-spatial filters efficiently
SELECT store_name, category
FROM stores 
WHERE category = 'restaurant'  -- Fast filter first
  AND ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 2000  -- Spatial filter
  AND rating > 4.0;  -- Additional filters after spatial
```

**Subquery Optimization**:

```sql theme={null}
-- Use spatial filtering in subquery to reduce dataset
SELECT s.store_name, s.address, r.avg_rating
FROM (
  SELECT store_id, store_name, address 
  FROM stores 
  WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000
) s
JOIN ratings r ON s.store_id = r.store_id;
```

**Spatial Aggregations**:

```sql theme={null}
-- Efficient spatial grouping
SELECT 
  region,
  COUNT(*) as store_count,
  AVG(ST_Distance(location, ST_Point(-122.4194, 37.7749))) as avg_distance
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 20000
GROUP BY region
ORDER BY avg_distance;
```

## Performance Monitoring

### **Key Performance Metrics**

**Query Execution Time**

**Monitor**:

* Overall query response time
* Spatial filter execution time
* Index effectiveness ratio

**Tools**:

```sql theme={null}
-- Enable query execution metrics
SET enableQueryExecutionMetrics = true;

-- Run your spatial query and check execution time
SELECT store_name FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;
```

**Index Effectiveness**

**Measure**: Ratio of filtered docs to total docs

```sql theme={null}
EXPLAIN PLAN FOR 
SELECT COUNT(*) FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;
```

**Good Ratios**:

* **>90% filtered**: Excellent index effectiveness
* **50-90% filtered**: Good performance
* **\<50% filtered**: Consider different resolutions

**Resolution Utilization**

**Check which resolutions are being used**:

Test different query radii:

```sql theme={null}
-- Small radius - should use high resolution
EXPLAIN PLAN FOR SELECT COUNT(*) FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 500;

-- Large radius - should use low resolution  
EXPLAIN PLAN FOR SELECT COUNT(*) FROM stores
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 50000;
```

### **Performance Benchmarking**

**Create Test Scenarios**:

```sql theme={null}
-- Benchmark different query patterns
-- Small radius (high precision)
SELECT 'Small Radius' as test_type, COUNT(*) as result_count
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 1000;

-- Medium radius (typical use case)
SELECT 'Medium Radius' as test_type, COUNT(*) as result_count  
FROM stores
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;

-- Large radius (coarse filtering)
SELECT 'Large Radius' as test_type, COUNT(*) as result_count
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 25000;
```

**Performance Comparison**:

```sql theme={null}
-- Compare indexed vs non-indexed performance
-- (Temporarily disable index for comparison)

-- Time indexed query
SELECT store_name FROM stores_indexed
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;

-- Time non-indexed query  
SELECT store_name FROM stores_no_index
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;
```

## Advanced Optimization Techniques

### **Query Rewriting for Performance**

**Distance Approximation**:

```sql theme={null}
-- For very large datasets, use approximate filtering first
WITH nearby_candidates AS (
  SELECT store_id, location
  FROM stores 
  WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 6000  -- Slightly larger radius
)
SELECT s.store_name, s.address
FROM nearby_candidates nc
JOIN stores s ON nc.store_id = s.store_id
WHERE ST_Distance(nc.location, ST_Point(-122.4194, 37.7749)) < 5000;  -- Exact filter
```

**Batch Spatial Operations**:

```sql theme={null}
-- Process multiple locations efficiently
WITH locations_of_interest(location_name, lon, lat) AS (
  VALUES 
    ('SF', -122.4194, 37.7749),
    ('Oakland', -122.2711, 37.8044),
    ('San Jose', -121.8863, 37.3382)
)
SELECT 
  loi.location_name,
  COUNT(*) as nearby_stores
FROM locations_of_interest loi
CROSS JOIN stores s
WHERE ST_Distance(s.location, ST_Point(loi.lon, loi.lat)) < 5000
GROUP BY loi.location_name;
```

### **Memory Optimization**

**Column Selection**:

```sql theme={null}
-- Select only needed columns
SELECT store_name, address 
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;

-- Avoid SELECT * with spatial data
SELECT * FROM stores  -- Includes large spatial columns
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 5000;
```

**Result Set Limiting**:

```sql theme={null}
-- Use LIMIT for interactive queries
SELECT store_name, 
       ST_Distance(location, ST_Point(-122.4194, 37.7749)) as distance
FROM stores 
WHERE ST_Distance(location, ST_Point(-122.4194, 37.7749)) < 10000
ORDER BY distance
LIMIT 20;  -- Only get top 20 results
```

## Troubleshooting Performance Issues

### **Common Performance Problems**

> **Slow Query Performance**
>
> **Symptoms**: Queries taking >1 second, high CPU usage
>
> **Diagnosis**:
>
> ```sql theme={null}
> EXPLAIN PLAN FOR your_slow_query;
> ```
>
> **Solutions**:
>
> * Verify H3 index operators are present
> * Check argument order in spatial functions
> * Ensure resolutions match query patterns
> * Add additional non-spatial filters

> **Index Not Being Used**
>
> **Symptoms**: EXPLAIN PLAN shows FILTER instead of FILTER\_H3\_INDEX
>
> **Diagnosis Checklist**:
>
> * [ ] Column configured with `encodingType: "RAW"`
> * [ ] Column in `noDictionaryColumns` array
> * [ ] H3 index configured with appropriate resolution
> * [ ] Query uses correct argument order
> * [ ] Spatial function is supported (ST\_Distance, ST\_Within, ST\_Contains)
> * [ ] Query distance is not >100x the hexagon size (auto-reverts to full scan)
>
> **Distance vs Resolution Check**:
> If your query distance is much larger than hexagon size, the H3IndexFilterOperator automatically reverts to non-indexed expression for performance reasons.
>
> **Fix Example**:
>
> ```json theme={null}
> {
>   "fieldConfigList": [{
>     "name": "location",
>     "encodingType": "RAW",  // Must be RAW
>     "indexes": {
>       "h3": {
>         "resolution": [8]  // Single resolution only
>       }
>     }
>   }]
> }
> ```

> **High Memory Usage**
>
> **Symptoms**: Out of memory errors, slow ingestion
>
> **Solutions**:
>
> * Reduce number of H3 resolutions (keep 2-3 max)
> * Remove very high resolutions (13-15) if not needed
> * Optimize query to select fewer columns
> * Use LIMIT in interactive queries
>
> **Memory-Optimized Config**:
>
> ```json theme={null}
> {
>   "indexes": {
>     "h3": {
>       "resolution": [6, 9]  // Reduced from [3, 6, 9, 12, 15]
>     }
>   }
> }
> ```

> **Poor Index Effectiveness**
>
> **Symptoms**: High filteredDocs/totalDocs ratio in query plans
>
> **Analysis**:
>
> ```sql theme={null}
> -- Check filtering effectiveness
> EXPLAIN PLAN FOR 
> SELECT COUNT(*) FROM your_table 
> WHERE ST_Distance(location, ST_Point(-122.4, 37.7)) < your_radius;
> ```
>
> **Solutions**:
>
> * Adjust H3 resolutions to better match query radius
> * Add complementary non-spatial filters
> * Consider table partitioning strategies

### **Performance Tuning Workflow**

**1. Baseline Measurement**

Measure current performance and identify bottlenecks using EXPLAIN PLAN and query timing.

**2. Index Configuration**

Optimize H3 resolutions based on query patterns and test different configurations.

**3. Query Optimization**

Rewrite queries to use indexes effectively and add non-spatial filters where possible.

**4. Validation**

Verify improvements using query plans and performance benchmarks.

**5. Monitoring**

Set up ongoing monitoring of query performance and index effectiveness.

## Best Practices Summary

### **Configuration Best Practices**

* Use 2-3 H3 resolutions for most applications
* Choose resolutions based on typical query radius
* Always disable dictionary encoding (`encodingType: "RAW"`)
* Test resolution configurations with real query patterns

### **Query Best Practices**

* Use correct argument order in spatial functions
* Combine spatial and non-spatial filters efficiently
* Use EXPLAIN PLAN to verify index usage
* Limit result sets in interactive applications

### **Performance Monitoring**

* Monitor query execution times regularly
* Check index effectiveness ratios
* Benchmark different query patterns
* Set up alerts for performance degradation


# Inverted Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/inverted-index

The Inverted index maps column values to row locations through bitmap or sorted structures, enabling rapid equality, range, and membership filtering.

## Overview and Purpose

An inverted index is a data structure that maps content (like words or values) to their locations in a database table. Think of it as a book's index that tells you exactly which pages contain specific words - but for your data.

The primary purpose of an inverted index is to dramatically speed up query performance when you're filtering data. Without an index, the system would need to scan every row to find matching values. With an inverted index, it can jump directly to the relevant rows.

Inverted indexes are especially valuable for:

* Equality filters (column = value)
* Membership checks (column IN \[value1, value2])
* Range queries (column > value)
* Any scenario where you frequently filter on specific columns

## How the Index Works

### Core Concepts

In traditional "forward" indexes, the system maps row IDs to their corresponding values:

```
Row 1 → Values (A, B, C)
Row 2 → Values (B, D, E)
Row 3 → Values (A, E, F)
```

An inverted index flips this relationship, mapping values to the rows containing them:

```
Value A → Rows [1, 3]
Value B → Rows [1, 2]
Value C → Rows [1]
Value D → Rows [2]
Value E → Rows [2, 3]
Value F → Rows [3]
```

StarTree Cloud (powered by Apache Pinot) supports two types of inverted indexes:

1. **Bitmap Inverted Index**: Creates a bitmap (a compact way to represent a set of rows) for each unique value in a column. This makes value lookups extremely fast with constant time complexity.
2. **Sorted Inverted Index**: Creates an index leveraging sorted column data structure for efficient binary search operations. This offers logarithmic lookup time and benefits of data locality.

Both types require dictionary encoding to be enabled for the column (which creates a mapping between actual values and encoded IDs).

### Example Illustration

For a `productCategory` column with values:

```
Row 1: "Electronics"
Row 2: "Clothing"
Row 3: "Electronics"
Row 4: "Home"
Row 5: "Clothing"
```

A bitmap inverted index would create:

```
"Electronics" → Bitmap [1,0,1,0,0]  (Rows 1 and 3)
"Clothing" → Bitmap [0,1,0,0,1]     (Rows 2 and 5)
"Home" → Bitmap [0,0,0,1,0]         (Row 4)
```

When filtering for "Electronics", the system immediately knows to retrieve rows 1 and 3 without scanning the entire dataset.

## Types of Inverted Indexes

### Bitmap Inverted Index

A bitmap inverted index maintains a mapping from each value to a bitmap of rows. This design makes value lookups extremely fast with constant time complexity.

**When to Use:**

* For columns with low to medium cardinality (number of unique values)
* When the column is frequently used in filter conditions
* When the column is not sorted

**Benefits:**

* Constant time lookup performance
* Works well with equality filters and IN clauses
* Can be created on multi-value columns

### Sorted Inverted Index

When a column is both sorted and has a dictionary enabled, StarTree Cloud automatically leverages the sorted structure to implement an efficient inverted index.

**When to Use:**

* For columns that are already sorted (like timestamps or sequential IDs)
* When the column is frequently used in filter conditions

**Benefits:**

* Logarithmic lookup time (log(n))
* Better data locality (related values are stored close together)
* Generally better performance than bitmap indexes
* No additional storage overhead (uses the same structure as the forward index)

## Configuration

### Configuring a Bitmap Inverted Index

To enable a bitmap inverted index on a column in your StarTree Cloud table, add the following configuration to your table definition:

```
{
  "fieldConfigList": [
    {
      "name": "yourColumnName",
      "indexes": {
        "inverted": {}
      }
    }
  ]
}
```

### Alternative Configuration (Legacy Method)

You can also use the older configuration format, though it's not recommended for new deployments:

```
{
  "tableIndexConfig": {
    "invertedIndexColumns": [
      "yourColumnName"
    ]
  }
}
```

### Configuring a Sorted Inverted Index

To set up a sorted inverted index:

1. First, configure the column as a sorted column in your table configuration:

```
{
  "tableIndexConfig": {
    "sortedColumn": [
      "yourColumnName"
    ]
  }
}
```

1. Ensure dictionary encoding is enabled for this column (it is enabled by default)

The sorted inverted index will be automatically available without additional configuration when these conditions are met.

### Important Considerations

1. **When the Index is Created**: By default, bitmap inverted indexes are created when segments are loaded by Pinot, not during initial segment generation. If you want to create them during segment generation, set `indexingConfig.createInvertedIndexDuringSegmentGeneration` to `true` in your table configuration.
2. **Dictionary Requirement**: Both types of inverted indexes require dictionary encoding to be enabled for the column.
3. **Multi-Value Support**: You can create a bitmap inverted index on a multi-value column.

If your queries frequently filter on a specific column and performance is critical, consider using a sorted column with its implicit sorted inverted index for maximum efficiency.


# JSON Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/json-index

The JSON index flattens and indexes JSON structures to allow fast, path-based filtering without the need for full document parsing.

## Overview and Purpose

A JSON index is a specialized data structure that accelerates filtering operations on JSON string columns. It enables efficient value lookups within JSON documents, without requiring the system to parse and analyze the entire JSON at query time.

JSON strings provide flexibility for representing complex data structures such as arrays, maps, and nested fields, without enforcing a rigid schema. However, without proper indexing, queries that filter on JSON content can be computationally expensive as they require scanning and reconstructing the entire JSON object for each record.

JSON indexes are particularly valuable for:

* Applications using JSON to store semi-structured or schema-flexible data
* Queries that filter on specific fields or paths within JSON documents
* Nested data structures where some fields are frequently used for filtering
* Use cases requiring fast lookups within arrays or nested objects

> [!NOTE]
>
> The JSON index optimizes query performance by pre-processing JSON documents during indexing, creating a flattened and searchable representation that dramatically reduces query-time computation.

## How the Index Works

### Core Concepts

Traditional approaches to JSON filtering require the system to:

1. Scan every record in the table
2. Parse each JSON string into a structured object
3. Extract the required value based on the specified path
4. Apply the filter condition to determine matches

This process is computationally expensive, especially for large datasets or complex JSON structures.

The JSON index in StarTree Cloud optimizes this process, by:

1. **Flattening**: During indexing, the system "flattens" the JSON structure, extracting key-value pairs using paths that identify each value's location in the original structure.
2. **Path Indexing**: The flattened key-value pairs are indexed, allowing fast lookups based on specific JSON paths.
3. **Array Handling**: Arrays are processed by either:
   * Creating separate entries for each array element
   * Combining elements from multiple arrays using cross-array unnesting
4. **Query Optimization**: At query time, the system can directly access the indexed paths without parsing the entire JSON structure.

### Example Illustration

Consider a JSON document representing a person:

```json theme={null}
{
  "name": "adam",
  "age": 30,
  "addresses": [
    {
      "number": 112,
      "street": "main st",
      "country": "us"
    },
    {
      "number": 2,
      "street": "second st",
      "country": "ca"
    }
  ]
}
```

Without a JSON index, to find all persons with addresses in "ca", the system would need to parse and analyze every JSON document.

With a JSON index, the document is flattened during indexing into entries like:

* `name`: "adam"
* `age`: 30
* `addresses[0].number`: 112
* `addresses[0].street`: "main st"
* `addresses[0].country`: "us"
* `addresses[1].number`: 2
* `addresses[1].street`: "second st"
* `addresses[1].country`: "ca"

At query time, the system can directly check the indexed values for `addresses[*].country` = "ca" without parsing the entire JSON structure.

## Configuration

### Enabling JSON Index

To enable a JSON index on a column in your StarTree Cloud table, add the following configuration to your table definition:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "jsonColumn",
      "indexes": {
        "json": {
          "maxLevels": 5,
          "excludeArray": false,
          "disableCrossArrayUnnest": false,
          "includePaths": null,
          "excludePaths": null,
          "excludeFields": null,
          "indexPaths": null,
          "maxValueLength": 0,
          "skipInvalidJson": false
        }
      }
    }
  ]
}
```

For a simple configuration with default values, you can use:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "jsonColumn",
      "indexes": {
        "json": {}
      }
    }
  ]
}
```

### Configuration Parameters

1. **maxLevels**: Maximum levels to flatten in the JSON object (arrays count as one level)
   * Default: 5
   * Higher values allow indexing deeper nested structures
   * Lower values reduce index size but limit deep path accessibility
2. **excludeArray**: Whether to exclude arrays when flattening the object
   * Default: false
   * When true, array elements are not indexed
   * Useful to reduce index size when arrays aren't used in filters
3. **disableCrossArrayUnnest**: Controls array unnesting behavior
   * Default: false (calculate unique combinations of all elements)
   * When true, prevents cross-product expansion of multiple arrays
   * Recommended for documents with multiple large arrays to avoid combinatorial explosion
4. **includePaths**: Only include the specified JSON paths (mutually exclusive with excludePaths)
   * Example: \["$.name", "$.addresses\[\*].country"]
   * Paths under included paths are automatically included
   * Useful to precisely control which paths are indexed
5. **excludePaths**: Exclude the specified JSON paths (mutually exclusive with includePaths)
   * Example: \["$.age", "$.addresses\[\*].number"]
   * Paths under excluded paths are automatically excluded
   * Useful to omit rarely filtered paths
6. **excludeFields**: Exclude specific field names regardless of path
   * Example: \["street", "zip"]
   * Applies even if the field is under included paths
   * Useful for excluding specific fields across all objects
7. **indexPaths**: Control indexed paths using glob patterns
   * Example: \["*.*", "addresses.\*\*"]
   * More flexible than includePaths/excludePaths
   * Can be combined with other configuration options
8. **maxValueLength**: Maximum length for indexed values
   * Values longer than this are replaced with "$SKIPPED$"
   * Useful for limiting index size when some values are very large
9. **skipInvalidJson**: Handle invalid JSON documents gracefully
   * When true, replaces invalid JSON with empty key/path and "$SKIPPED$" value
   * When false (default), invalid JSON causes errors during indexing

### Alternative Configuration Methods

While the above method is recommended, you can also configure JSON indexes using legacy approaches:

```json theme={null}
{
  "tableIndexConfig": {
    "jsonIndexConfigs": {
      "jsonColumn": {
        "maxLevels": 5,
        "excludeArray": false
      }
    }
  }
}
```

Or more simply:

```json theme={null}
{
  "tableIndexConfig": {
    "jsonIndexColumns": [
      "jsonColumn"
    ]
  }
}
```

### Important Configuration Considerations

1. **Column Type**: JSON indexes can only be applied to columns with STRING or JSON data types that contain valid JSON strings.
2. **Dictionary Encoding**: To reduce storage overhead, consider [disabling the dictionary indexes](../indexes/dictionary-index#explicitly-disabling-dictionary-index) on  JSON-indexed columns.
3. **Array Handling**: For documents with multiple large arrays, setting disableCrossArrayUnnest to true can prevent the "Got too many combinations" error, which occurs when more than 100,000 flattened documents would be created.
4. **Index Size**: The index size grows with JSON complexity, depth, and the number of paths indexed. Use configuration parameters to control this.


# Min/Max Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/minmax-index

Use the Min/Max index to prune sorted raw forward-index chunks for equality, IN, and range filters.

## Overview and Purpose

The MinMax index is a lightweight, coarse-grained index that divides a sorted column into fixed-size chunks of documents and records the minimum and maximum value of each chunk. At query time it uses binary search over these boundaries to skip chunks that cannot match a filter, then runs an exact forward-index scan only on the surviving ("candidate") chunks.

## **When to use it**

Use the MinMax index for large sorted raw columns where you frequently filter with equality or range predicates and want to reduce the number of documents scanned. It is especially useful for wide variable-width columns (`STRING`, `BYTES`, `BIG_DECIMAL`) where scanning every value is expensive. The index only helps with `EQ`, `IN`, and `RANGE` predicates; other predicate types are not routed through it.

The index is especially useful for:

* Time or sequence columns that are sorted during ingestion.
* High-cardinality columns where a full inverted index would be too large.
* Equality, `IN`, and range predicates on sorted raw columns.
* Object-store backed segments where pruning chunks also reduces remote reads.

> [!NOTE]
>
> This page covers the StarTree `minmax` index configured on a field. It is different from Pinot's segment-level `columnMinMaxValueGeneratorMode`, which writes whole-segment min/max metadata for segment pruning.

## Availability

The Min/Max index is available in StarTree release 0.15 and later. The corresponding STP release branch is `release/1.6.0-STP-2.164.x`.

## How the Index Works

For each configured column, StarTree Cloud divides the sorted raw forward index into document chunks. The Min/Max index records the minimum and maximum column value in each chunk.

When a query has a predicate on that column, the server checks the predicate against the chunk boundary:

1. If the predicate cannot match any value in the chunk, the chunk is skipped.
2. If the predicate might match the chunk, StarTree Cloud reads and evaluates the candidate rows.
3. Final predicate evaluation still runs on candidate rows, so pruning does not change query results.

For example, if an `eventTimeMillis` chunk covers values from `1764556800000` to `1764557399999`, a query for `eventTimeMillis >= 1764560400000` can skip that chunk immediately.

## Requirements

* The column must be single-value.
* The column must be sorted in each segment.
* The column forward index must be enabled.
* The column must use `RAW` encoding.
* `chunkSize`, when configured, must be a positive power of two.
* Supported data types are `INT`, `LONG`, `FLOAT`, `DOUBLE`, `BIG_DECIMAL`, `TIMESTAMP`, `STRING`, and `BYTES`. `BOOLEAN`, `JSON`, and `MAP` are not supported.

## Configuration

To enable the Min/Max index, add a `minmax` index to the column's `fieldConfigList` entry.

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "eventTimeMillis",
      "encodingType": "RAW",
      "indexes": {
        "forward": {},
        "minmax": {
          "chunkSize": 2048
        }
      }
    }
  ],
  "tableIndexConfig": {
    "sortedColumn": [
      "eventTimeMillis"
    ]
  }
}
```

* `chunkSize`: number of documents per chunk. If omitted, it defaults to `1024`. It must be a **positive power of 2**.

For offline tables, sort input records by the same column before segment generation. For real-time tables, configure `tableIndexConfig.sortedColumn` so consuming segments are built in sorted order.

### Configuration Parameters

| Parameter | Required | Default Value | Description |
| - | - | - | - |
| `chunkSize` | No | 1024 | Number of documents covered by each min/max boundary. Smaller chunks improve pruning precision but increase index size. The value must be a positive power of two. |

### **Choosing** `chunkSize`

There is no automatic per-segment derivation of chunk size — the optimal value depends on data type, cardinality, and query shape. Smaller chunks give finer pruning (fewer rows scanned per surviving chunk) but a larger index; larger chunks give a smaller index but coarser pruning. Variable-width columns (`STRING`/`BYTES`/`BIG_DECIMAL`) scan more bytes per chunk, so they may benefit from smaller chunks.

## Query Examples

The Min/Max index can prune chunks for range predicates:

```sql theme={null}
SELECT count(*)
FROM clickstream
WHERE eventTimeMillis BETWEEN 1764556800000 AND 1764560400000;
```

It can also prune for `IN` predicates:

```sql theme={null}
SELECT userId, eventType, eventTimeMillis
FROM clickstream
WHERE eventTimeMillis IN (1764556800000, 1764556860000, 1764556920000)
LIMIT 100;
```

## Performance Considerations

1. **Sort quality matters**: The index is only useful when values are sorted or highly clustered. Unsorted data produces wide chunk boundaries and weak pruning.
2. **Chunk size is the main tuning knob**: Smaller chunks usually reduce scanned rows, while larger chunks reduce index size.
3. **Use with raw forward indexes**: The index targets raw forward-index reads. Use a range or inverted index instead for dictionary-encoded columns when those indexes fit your workload better.
4. **Reload existing segments**: After adding the index to an existing table, reload segments so StarTree Cloud builds the new index files.


# Overview
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/overview

Boost query performance using indexing strategies, including inverted, range, sorted, and star-tree indexes for efficient data management.

Apache Pinot is built for scale, effortlessly handling massive datasets and high query throughput. At the heart of its exceptional performance and flexibility are Apache Pinot's advanced indexing capabilities, enabling users to execute ultra-fast analytics even at petabyte scale. With a comprehensive set of indexing techniques, Pinot empowers users to confidently select the indexes best suited for their unique data characteristics and evolving query patterns.

## Why Use Indexes?

* **Accelerated Query Performance**: Indexes drastically enhance query speed, efficiently pinpointing relevant data segments even at massive scale.
* **Optimized Resource Usage**: Strategic indexing reduces unnecessary data scans, effectively lowering resource consumption and operational costs.
* **Flexible Analytics**: A variety of index types allows Pinot to accommodate diverse analytical workloads, ranging from straightforward lookups to complex analytics and sophisticated similarity searches.

## Supported Index Types

Apache Pinot supports a wide range of indexes tailored to optimize various query scenarios:

> **[Inverted Index](/corecapabilities/manage-data/indexes/inverted-index)**
>
> Maps each value directly to its rows for fast lookups.

> **[Star-tree Index](/corecapabilities/manage-data/indexes/startree-index)**
>
> Delivers superior aggregation performance on large, high-cardinality datasets.

> **[Range Index](/corecapabilities/manage-data/indexes/range-index)**
>
> Handles numeric range queries efficiently without requiring data sorting.

> **[Forward Index](/corecapabilities/manage-data/indexes/forward-index)**
>
> Enhances range queries by maintaining data in a sorted sequence. Types: Dictionary-Encoded, Sorted, and Raw Value.

> **[JSON Index](/corecapabilities/manage-data/indexes/json-index)**
>
> * Enables fast queries on JSON-structured data.

> **[Geospatial Index](/corecapabilities/manage-data/indexes/geospatial-index)**
>
> Powers geographic queries, enabling proximity searches and spatial analytics.

> **[Text Index (Lucene)](/corecapabilities/manage-data/indexes/text-index)**
>
> Provides rapid search capabilities for unstructured text fields through full-text indexing.

> **[Text Index (Native)](/corecapabilities/manage-data/indexes/native-text-index)**
>
> Provides rapid search capabilities for unstructured text fields through full-text indexing.

> **[Timestamp Index](/corecapabilities/manage-data/indexes/timestamp-index)**
>
> Enables fast filtering on timestamp columns by indexing at a defined time granularity.

> **[Vector Index](/corecapabilities/manage-data/indexes/vector-index)**
>
> Supports fast similarity searches on vector embeddings, ideal for Gen AI and recommendation workloads.

> **[Sparse Index](/corecapabilities/manage-data/indexes/sparse-index)**
>
> Optimizes high-cardinality equality filters using chunked partitioning.

> **[Bloom Filter](/corecapabilities/manage-data/indexes/bloom-filter)**
>
> Fast segment pruning for equality queries with minimal memory.

> **[FST Index](/corecapabilities/manage-data/indexes/fst-index)**
>
> Compact regex search on dictionary-encoded text columns.

> **[Dictionary Index](/corecapabilities/manage-data/indexes/dictionary-index)**
>
> Replaces repeated values with integer IDs for storage efficiency.

> **[Composite JSON Index](/corecapabilities/manage-data/indexes/composite-json-index)**
>
> An enhanced version of the [JSON Index](/corecapabilities/manage-data/indexes/json-index) to reduce index size and improve performance.

When selecting the right index for your use case, consider the following:

* **Query Patterns**: Assess the types of queries you run—point lookups, range queries, aggregations, or similarity searches.
* **Data Type and Cardinality**: Evaluate column uniqueness, data distribution, and characteristics.
* **Performance vs. Storage Trade-offs**: Understand that some indexes enhance performance substantially but may require additional storage.

By strategically selecting indexes based on your data and query requirements, Apache Pinot empowers you to deliver blazing-fast analytics at any scale, making complex data exploration powerful and seamless.

## Updating Indexes

Updating indexes involves the following abstracted steps:

1. **Assess the Right Indexes**: Determine the appropriate indexes based on your query needs and data characteristics.
2. **Apply Index Configurations**: Configure indexes in your table configuration, referring to each index's dedicated documentation page for specific configuration options.
3. **Apply Changes and Reload**: Invoke the table reload using the reload API. This process occurs seamlessly, without downtime, and remains completely transparent to active queries.


# Range Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/range-index

Use the Range index to optimize range-based queries. The index enhances performance for inequality filters and BETWEEN operations on high-cardinality numeric or timestamp columns by mapping value ranges to corresponding rows.

## Overview and Purpose

A range index is a specialized data structure that accelerates queries involving range-based filtering operations. While traditional inverted indexes excel at equality-based filters (column = value), range indexes are optimized for comparison operations like greater than, less than, or between.

Range indexes transform range-based filtering from full table scans to targeted lookups, significantly improving query performance when filtering data across numeric ranges or time intervals.

Range indexes are particularly valuable for:

* Queries with inequality filters (`>`, `<`, `>=`, `<=`)
* Between operations (`column BETWEEN x AND y`)
* Numeric or timestamp columns with high cardinality
* Analytical workloads that frequently filter metric values by ranges

> [!NOTE]
>
> Range indexes are especially beneficial for high-cardinality columns (columns with many unique values) where traditional inverted indexes would be inefficient due to their large size.

## How the Index Works

### Core Concepts

Traditional inverted indexes create mappings from individual values to the rows containing them. This approach works well for equality checks but becomes inefficient for range operations, especially with high-cardinality columns.

A range index in StarTree Cloud complements inverted indexes by utilizing:

1. **Range-Based Mapping**: Instead of mapping individual values to rows, a range index maps ranges of values to rows that contain values within those ranges.
2. **Efficient Range Evaluation**: When a query includes a range predicate (e.g., `column > 100`), the index can quickly identify the exact set of rows that meet this condition without scanning the entire dataset.

### Example Illustration

For a column storing sales amounts with values ranging from \$0 to \$1,000:

A traditional approach would need to scan every row and check each value against the filter condition.

With a range index, the system:

1. Divides the value range into segments (e.g., \$0-\$100, \$100-\$200, etc.)
2. Maps each segment to the rows containing values in that range
3. For a query like `sales_amount > 500`, quickly identifies only the relevant segments and returns the corresponding rows

This dramatically reduces the number of rows that need to be examined, resulting in faster query performance.

## Configuration

### Enabling Range Index

To enable a range index on a column in your StarTree Cloud table, add the following configuration to your table definition:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "sales_amount",
      "indexes": {
        "range": {}
      }
    }
  ]
}
```

### Alternative Configuration (Legacy Method)

You can also use the older configuration format, though it's not recommended for new deployments:

```json theme={null}
{
  "tableIndexConfig": {
    "rangeIndexColumns": [
      "sales_amount"
    ]
  }
}
```

### Important Configuration Considerations

1. **Column Requirements**: Range indexes can be created on:
   * Dictionary-encoded columns of any data type
   * Raw-encoded columns of numeric types (INT, LONG, FLOAT, DOUBLE)
   * Dictionary-encoded time columns using STRING type (if the datetime format maintains lexicographical order)
2. **Cardinality Recommendation**: Range indexes are best suited for columns with high cardinality (many unique values), where inverted indexes would be inefficient.
3. **Storage and Memory**: Range indexes are more compact than inverted indexes for high-cardinality columns, resulting in better storage efficiency and memory usage.
4. **Query Pattern**: Use range indexes on columns that are frequently filtered with range predicates (`>`, `<`, `>=`, `<=`, `BETWEEN`).


# Sparse Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/sparse-index

The Sparse index improves equality-based query performance on high-cardinality columns. It is a hybrid between inverted indexes and Bloom filters, particularly beneficial for tiered storage environments.

## Overview and Purpose

A sparse index is a specialized data structure that accelerates equality-based filter operations on columns with extremely high cardinality. It represents a hybrid approach between inverted indexes and Bloom filters, designed specifically for scenarios where traditional indexes become inefficient due to the sheer number of unique values.

Sparse indexes are particularly valuable for:

* Columns storing hundreds of thousands of unique values
* Random identifiers like UUIDs or session IDs
* High-cardinality fields such as public IP addresses
* Any query pattern involving equality filters (`column = value` or `column IN (...)`) on high-cardinality columns
* Deployments using tiered storage with data in cloud object stores

> [!NOTE]
>
> Sparse index is a StarTree extension that shows particular performance benefits with tiered storage because it can significantly reduce data transferred from cloud storage during query execution.

## How the Index Works

### Core Concepts

Traditional inverted indexes create a direct mapping between each unique value and the rows containing that value. While effective for low- to medium-cardinality columns, this approach becomes inefficient as cardinality increases, leading to large index sizes and degraded performance.

The sparse index in StarTree Cloud takes a different approach:

1. **Chunking**: Rows are grouped into contiguous chunks of a predefined size.
2. **Partitioning**: Column values are classified into a fixed number of partitions using one or more hash functions (mappers).
3. **Pseudo-Inverted Index**: Instead of mapping values directly to rows, the index maps partitions to chunks that contain values from that partition.

### Example Illustration

For a column storing device IDs with millions of unique values:

1. **Traditional Approach**: An inverted index would map each unique device ID to its corresponding rows, potentially creating an enormous index.
2. **Sparse Index Approach**:
   * Rows are grouped into chunks
   * Device IDs are hashed into partitions using multiple hash functions
   * The index maps each partition to the chunks that contain values from that partition

When processing a query like `WHERE deviceId = 'abc123'`:

1. The system determines which partition contains 'abc123'
2. It identifies the chunks associated with that partition
3. Only those chunks are scanned, dramatically reducing I/O operations

Sparse indexes are probabilistic at chunk level. They reduce scan work but may include extra chunks that are removed by final predicate evaluation.

This approach is particularly beneficial with tiered storage, as it minimizes the amount of data downloaded from cloud storage.

## Configuration

### Enabling Sparse Index at Server Level

To use sparse indexes, you must first enable them at the server level:

```
pinot.server.instance.index.sparse.enabled=true
```

This can be set via the Swagger UI using the `/cluster/configs` API, or through your server configuration files.

**Note:** After adding this configuration, restart the servers for the changes to take effect.

### Configuring Sparse Index for a Column

To enable a sparse index on a column in your StarTree Cloud table, add the following configuration to your table definition:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "deviceId",
      "indexes": {
        "sparse": {
          "chunkSize": 1024,
          "partitions": 10000,
          "hashFunctionCount": 3
        }
      }
    }
  ]
}
```

### Configuration Parameters

1. **chunkSize**: The number of documents per chunk.
   * Must be a power of 2
   * Default: 1024
   * Recommended range: 128 to 8192
   * Impact: Smaller chunks typically reduce false positives and `numEntriesScannedInFilter`, but can increase index size
   * Usually the strongest tuning knob for scan reduction
2. **partitions**: The number of partitions used for classifying values.
   * Default: 10000
   * Recommended: Depends on cardinality and target false-positive rate
   * Impact: Higher values improve selectivity and generally increase index size
   * Size impact is often sub-linear when partitions are already large
3. **hashFunctionCount**: The number of partition mapping functions used.
   * Default: 3
   * Recommended range: 1 to 8
   * Impact: Higher values increase selectivity and index lookups
   * Usually increases index size close to linearly

> [!NOTE]
>
> Sparse index cannot be created on a column that also has an inverted index enabled.

> [!NOTE]
>
> Sparse index is not supported on MAP columns.

### Advanced Configuration Parameters

These parameters are rarely needed and should only be used in specific scenarios:

1. **seedGenerator**: An integer seed for generating hash functions.
   * Only necessary if hash collisions are unusually frequent
2. **mapperId**: Specifies the hash function type.
   * Currently, only "murmur" is supported

## False-Positive and Size Formulas

The following formulas are practical approximations for tuning.\
Assume:

* $R$: rows in segment
* $c$: chunkSize
* $m$: partitions
* $k$: hashFunctionCount
* $v$: average values per row for the indexed column ($v = 1$ for single-value)
* $N_c = \lceil R / c \rceil$ and $n = c \cdot v$

Expected unique partitions touched in one chunk for one hash:

$$
u = m \left(1 - e^{-n/m}\right)
$$

Chunk-level false-positive rate for non-existing keys:

$$
p_{\mathrm{fp},\mathrm{chunk}} \approx \left(\frac{u}{m}\right)^k \approx \left(1 - e^{-n/m}\right)^k
$$

Expected false-positive chunks per segment:

$$
E_{\mathrm{fp},\mathrm{chunks}} \approx N_c \cdot p_{\mathrm{fp},\mathrm{chunk}}
$$

Candidate-docs proxy:

$$
E_{\mathrm{candidateDocs}} \approx c \cdot E_{\mathrm{fp},\mathrm{chunks}}
$$

Index-size proxy:

$$
E_{\mathrm{entries}} \approx N_c \cdot k \cdot u
$$

$$
\mathrm{SizeBytes} \approx b \cdot E_{\mathrm{entries}}
$$

Where $b$ is calibrated bytes per entry for your dataset.

For single-value columns ($v = 1$) and fixed $c, m, k$, you can derive:

$$
S_{\mathrm{required}} \approx \left(\frac{b \cdot R \cdot k \cdot m}{c}\right)
\cdot p_{\mathrm{fp},\mathrm{chunk,target}}^{1/k}
$$

This gives required index size for a target chunk-level false-positive rate.

## Optimization Workflow

Use this workflow to tune sparse index safely:

1. **Pick the right columns**
   * High-cardinality columns queried with `=` or `IN`.
   * Especially useful when an inverted index would be too large.
2. **Benchmark baseline**
   * Run representative queries and capture:
     * `timeUsedMs`
     * `numEntriesScannedInFilter`
     * `numDocsScanned`
3. **Tune in this order**
   * `chunkSize` first (often largest impact on scan reduction)
   * `partitions` second (improve selectivity with moderate size growth)
   * `hashFunctionCount` third (precision gains with near-linear size growth)
4. **Evaluate each candidate**
   * Compare p50/p95 latency, `numEntriesScannedInFilter`, and index size.
   * Choose the smallest index that meets your latency and scan targets.
5. **Rebuild/reload segments after config updates**
   * Ensure segments are reloaded so new index settings take effect.

## Performance Considerations

1. **Tiered Storage Benefits**: Sparse indexes provide the greatest performance improvements with tiered storage because they reduce object-store reads.
2. **Cardinality Trade-offs**: For low-cardinality columns, traditional inverted indexes are often more efficient.
3. **Predicate Coverage**: Sparse indexes are designed for equality/IN filter acceleration. Final predicate evaluation still runs on candidate rows.
4. **Storage Impact**:
   * `hashFunctionCount` often increases index size close to linearly.
   * `partitions` increases size, but not always linearly in practice.
   * `chunkSize` trades off index size versus scan precision.


# Star-tree Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/startree-index

The Star-tree index boosts aggregation and group-by query performance. It pre-aggregates data across multiple dimensions to reduce query latency and storage usage.

## Overview and Purpose

A star-tree index is an advanced multi-dimensional indexing structure that significantly accelerates aggregation and group-by queries by using pre-computed aggregation results. Unlike single-column indexes, star-tree indexes work across multiple dimensions to dramatically reduce the number of records that need to be processed at query time.

One of the biggest challenges in real-time analytics is maintaining low query latencies on large datasets while efficiently using storage resources. Traditional indexes improve query performance but are still limited by the number of records that must be processed. Pure pre-aggregation strategies guarantee fast query responses but can lead to storage explosion.

Star-tree indexes in StarTree Cloud provide an optimal balance between these approaches by selectively pre-aggregating data based on common query patterns, offering:

* Predictable, low query latencies for aggregation operations
* Efficient use of storage space compared to full pre-aggregation
* Significant performance improvements for multi-dimensional queries
* Configurable trade-offs between query speed and storage requirements

> [!NOTE]
>
> Star-tree indexes are particularly valuable for analytical workloads with high query volumes on large datasets where aggregation queries are common.

## How the Index Works

### Core Concepts

The star-tree index creates a hierarchical tree structure that organizes data based on multiple dimensions. The key elements of this structure include:

1. **Dimension-based Organization**: Data is organized hierarchically based on an ordered list of dimensions, with each level in the tree representing a particular dimension.
2. **Pre-aggregation**: For each node in the tree, metrics are pre-aggregated, allowing the system to directly use these pre-computed results when possible.
3. **Star Nodes**: Special nodes that contain pre-aggregated results after removing a specific dimension, enabling efficient handling of queries that don't filter on certain dimensions.
4. **Configurable Depth**: The tree depth and node size can be configured to balance between storage requirements and query performance.

### Example Illustration

Consider a dataset with dimensions Country, Browser, and Locale, and a metric Impressions:

1. **Traditional Approach**: To find the sum of Impressions for USA, Chrome across all Locales, the system would need to scan all matching records and compute the sum at query time.
2. **Star-tree Approach**:
   * The data is organized hierarchically by Country, then Browser, then Locale
   * At the Country=USA, Browser=Chrome level, a star node pre-computes the sum across all Locales
   * The query directly uses this pre-computed result without scanning individual records

This approach significantly reduces query time by eliminating the need to process individual records when aggregate values can be used instead.

## Tree Structure and Components

### Node Types

1. **Root Node**: The single starting point of the tree from which all other nodes can be traversed.
2. **Leaf Node**: Contains at most T records (where T is configurable), representing the most granular level of data.
3. **Non-leaf Node**: Contains more than T records and is further split into child nodes based on the next dimension in the split order.
4. **Star Node**: A special child node that contains pre-aggregated records after removing the dimension on which data was split at that level.

### Node Properties

Each node in the star-tree contains:

* **Dimension**: The dimension the node is split on
* **Document ID Range**: The range of documents this node encompasses
* **Aggregated Document ID**: A single document containing the pre-aggregated results for all documents in this node

## Configuration

### Enabling Star-tree Index

To enable a star-tree index in your StarTree Cloud table, add the following configuration to your table definition:

```json theme={null}
{
  "tableIndexConfig": {
    "starTreeIndexConfigs": [
      {
        "dimensionsSplitOrder": [
          "country",
          "browser",
          "locale"
        ],
        "skipStarNodeCreationForDimensions": [],
        "functionColumnPairs": [
          "SUM__impressions",
          "COUNT__*"
        ],
        "maxLeafRecords": 10000
      }
    ]
  }
}
```

### Alternative Configuration Using Aggregation Configs

You can also use the more detailed `aggregationConfigs` approach with compression settings:

```json theme={null}
{
  "tableIndexConfig": {
    "starTreeIndexConfigs": [
      {
        "dimensionsSplitOrder": [
          "country",
          "browser",
          "locale"
        ],
        "skipStarNodeCreationForDimensions": [],
        "aggregationConfigs": [
          {
            "columnName": "impressions",
            "aggregationFunction": "SUM",
            "compressionCodec": "LZ4"
          },
          {
            "columnName": "*",
            "aggregationFunction": "COUNT"
          }
        ],
        "maxLeafRecords": 10000
      }
    ]
  }
}
```

### Default Star-tree Index

For a simpler approach, you can enable the default star-tree index:

```json theme={null}
{
  "tableIndexConfig": {
    "enableDefaultStarTree": true
  }
}
```

The default configuration:

* Includes all single-value dictionary-encoded dimensions with cardinality ≤ 10,000
* Sorts dimensions by cardinality in descending order
* Appends time columns at the end of the dimension split order
* Includes COUNT(\*) and SUM for all numeric metrics
* Uses a default maxLeafRecords value of 10,000

### Configuration Parameters

1. **dimensionsSplitOrder**: An ordered list of dimension names that determines the split order in the tree.
   * Only these dimensions are included in the star-tree index
   * The order is important as it determines the hierarchy of the tree
   * All columns in the filter and group-by clauses should be included for the index to be used
2. **skipStarNodeCreationForDimensions**: (Optional) Dimensions for which star nodes should not be created.
3. **functionColumnPairs** / **aggregationConfigs**: The metrics and aggregation functions to pre-compute.
   * Format for functionColumnPairs: FUNCTION\_\_COLUMN (e.g., SUM\_\_impressions)
   * All aggregations in a query must be included here for the star-tree to be used
4. **maxLeafRecords**: (Optional, default 10,000) The threshold for splitting nodes further.
5. **compressionCodec**: (Optional, via aggregationConfigs) The compression type for aggregated values.

### Supported Aggregation Functions

The star-tree index supports aggregation functions that have bounded-sized intermediate results:

* **SUM**, **MIN**, **MAX**, **COUNT**
* **AVG** (implemented as SUM/COUNT)
* **DISTINCTCOUNT** variants with bounded intermediate size:
  * DISTINCTCOUNT\_HLL
  * DISTINCTCOUNT\_THETA\_SKETCH
  * DISTINCTCOUNT\_CPC\_SKETCH
  * DISTINCTCOUNT\_TUPLE\_SKETCH
* **PERCENTILE** implementations with bounded size:
  * PERCENTILE\_TDIGEST

Unsupported functions include:

* Basic DISTINCTCOUNT (unbounded Set)
* PERCENTILE (unbounded List)

## Query Behavior and Optimization

### When Star-tree Index Is Used

The star-tree index is automatically used when:

1. All predicates in the query are supported by the star-tree index
2. All group-by columns are included in the dimensionsSplitOrder
3. All aggregation functions are included in the index configuration

### How Queries Are Optimized

When a query is executed with a star-tree index:

1. The system first checks if the query can be answered using the star-tree
2. It traverses the tree to identify nodes that satisfy the query predicates
3. For each level in the tree:
   * If no predicate or group-by exists on the split dimension, the star node is selected if available
   * If predicates exist on the split dimension, only matching child nodes are selected
   * If no predicate but a group-by exists on the split dimension, all child nodes except the star node are selected
4. The system collects documents from selected nodes:
   * If all predicates and group-bys are satisfied, it uses the single pre-aggregated document from each node
   * Otherwise, it processes all documents in the range

### Supported Predicates

* **Equality**: column = value
* **Range**: >, >=, \<, \<=, BETWEEN
* **NOT**: Applied to simple predicates (added in 1.2.0)
* **OR**: Only for predicates on the same dimension

### Unsupported Predicates

* **REGEXP\_LIKE**: Not supported as it requires scanning the entire dictionary
* **IS NULL/IS NOT NULL**: Limited support (use column = default or column != default as workaround)
* **OR across dimensions**: Cannot be used due to double-counting risks
* **Complex NOT**: NOT on top of AND/OR is not supported


# Text Index (Lucene)
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/text-index

The Text index (Lucene) enhances complex text search capabilities by enabling efficient phrase matching, regex, and prefix searches on large text fields.

## Overview and Purpose

Pinot supports super-fast query processing through its indexes on non-BLOB like columns. Queries with exact match filters are run efficiently through a combination of dictionary encoding, inverted index, and sorted index.

This is useful for a query like the following, which looks for exact matches on two columns of type STRING and INT respectively:

```javascript theme={null}
SELECT COUNT(*) 
FROM Foo 
WHERE STRING_COL = 'ABCDCD' 
AND INT_COL > 2000
```

For arbitrary text data that falls into the BLOB/CLOB territory, we need more than exact matches. This often involves using regex, phrase, fuzzy queries on BLOB like data. Text indexes can efficiently perform arbitrary search on STRING columns where each column value is a large BLOB of text using the `TEXT_MATCH` function, like this:

```javascript theme={null}
SELECT COUNT(*) 
FROM Foo 
WHERE TEXT_MATCH (, '')
```

where `` is the column text index is created on and `` conforms to one of the following:

| **Search Expression Type** | **Example** |
| - | - |
| Phrase query | TEXT\_MATCH (\, '"distributed system"') |
| Term Query | TEXT\_MATCH (\, 'Java') |
| Boolean Query | TEXT\_MATCH (\, 'Java AND c++') |
| Prefix Query | TEXT\_MATCH (\, 'stream\*') |
| Regex Query | TEXT\_MATCH (\, '/Exception.\*/') |
| Not Query | TEXT\_MATCH (\, '*\*:\** NOT c%')                                                    NOT TEXT\_MATCH (\, 'c%') |

A text index is a specialized data structure that enables efficient text search operations on large text data stored in STRING columns. Unlike traditional indexes that support exact matches or range queries, text indexes facilitate complex text search capabilities including phrase matching, term queries, regex patterns, and prefix searches.

In StarTree Cloud (powered by Apache Pinot), the text index bridges the gap between structured query processing and text search functionality. It allows users to perform flexible search operations on text data that would otherwise require expensive full table scans and string manipulations.

Text indexes are particularly valuable for:

* Log analysis where each log entry contains large chunks of text
* Document search across free-form text fields
* Search across JSON blobs stored as strings
* Any use case involving queries on large text values where exact matching is insufficient

> [!NOTE]
>
> Text indexes provide significant performance benefits for search operations on STRING columns containing large text values (BLOB/CLOB-like data) that don't fit the pattern of dictionary-encoded exact matches.

## How the Index Works

### Core Concepts

Traditional indexing approaches in Pinot (dictionary encoding, inverted index, sorted index) excel at exact match and range queries but are inefficient for complex text search operations on large text values.

The text index in StarTree Cloud:

1. **Text Processing Pipeline**: During indexing, each text value is processed through a pipeline that includes:
   * Tokenization: Breaking the text into individual terms
   * Stop word removal: Filtering out common words with low search value
   * Term normalization: Converting terms to a standard form
2. **Inverted Index Creation**: An inverted index is built that maps terms to the documents containing them, enabling fast lookup.
3. **Query Processing**: At query time, search expressions are analyzed and used to efficiently locate matching documents without full table scans.

### Example Illustration

Consider a column containing Apache access logs where each row stores an entire log entry:

```
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
```

Without a text index:

* To find all log entries containing "Firefox", the system would need to scan every row and check if the string "Firefox" appears in the text.

With a text index:

* During indexing, the log entry is tokenized into terms including "Firefox"
* The term "Firefox" is linked to the documents containing it
* At query time, the system can directly retrieve the rows containing "Firefox" without scanning all data

This approach significantly improves query performance for text search operations, especially on large datasets.

## **Enable a per-column text index**

Enable a text index on a column in the [table configuration](https://docs.pinot.apache.org/configuration-reference/table) by adding a new section with the name "fieldConfigList".

```json theme={null}
"fieldConfigList":[
  {
     "name":"text_col_1",
     "encodingType":"RAW",
     "indexTypes":["TEXT"]
  },
  {
     "name":"text_col_2",
     "encodingType":"RAW",
     "indexTypes":["TEXT"]
  }
]
```

Each column that has a text index should also be specified as `noDictionaryColumns` in `tableIndexConfig`:

```json theme={null}
"tableIndexConfig": {
   "noDictionaryColumns": [
     "text_col_1",
     "text_col_2"
 ]}
```

You can configure text indexes in the following scenarios:

* Adding a new table with text index enabled on one or more columns.
* Adding a new column with text index enabled to an existing table.
* Enabling a text index on an existing column.

### Important Configuration Considerations

1. **Column Requirements**:
   * The column must be of type STRING
   * The column must be single-valued (not multi-valued)
   * The column should use RAW encoding (no dictionary)
2. **noDictionaryColumns**: Text-indexed columns should be added to the noDictionaryColumns list in tableIndexConfig to reduce storage overhead.
3. **Index Coexistence**: Using a text index in coexistence with other Pinot indexes on the same column is not supported.

### Customizing Stop Words

You can customize the stop words used during indexing:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "log_content",
      "encodingType": "RAW",
      "indexes": {
        "text": {}
      },
      "properties": {
        "stopWordInclude": "incl1, incl2, incl3",
        "stopWordExclude": "it"
      }
    }
  ]
}
```

* **stopWordInclude**: Comma-separated list of words to include as stop words (in addition to defaults)
* **stopWordExclude**: Comma-separated list of words to exclude from the default stop words list

### Enabling Prefix/Suffix Matching in Phrase Queries

For more flexible phrase matching across term boundaries:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "log_content",
      "encodingType": "RAW",
      "indexes": {
        "text": {}
      },
      "properties": {
        "enablePrefixSuffixMatchingInPhraseQueries": "true"
      }
    }
  ]
}
```

### Tuning Lucene Index Creation

For optimizing index creation performance:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "log_content",
      "encodingType": "RAW",
      "indexes": {
        "text": {}
      },
      "properties": {
        "luceneUseCompoundFile": "false",
        "luceneMaxBufferSizeMB": "128"
      }
    }
  ]
}
```

## **TEXT\_MATCH Query Options**

The `TEXT_MATCH` function supports an optional third parameter for specifying Lucene query parser options at query time. This allows for flexible and advanced text search without changing table configuration.

**Function Signature:**

```
TEXT_MATCH(text_column_name, search_expression [, options])
```

* `text_column_name`: Name of the column to perform text search on.
* `search_expression`: The query string for text search.
* `options` (optional): Comma-separated string of key-value pairs to control query parsing and search behavior.

**Available Options:**

| **Option** | **Values** | **Description** |
| :- | :- | :- |
| `parser` | `CLASSIC`, `STANDARD`, `COMPLEX, MATCHPHRASE, MATCH` | Selects the Lucene query parser to use. Default is `CLASSIC`. |
| `allowLeadingWildcard` | `true`, `false` | Allows queries to start with a wildcard (e.g., `*term`). Default is `false`. |
| `defaultOperator` | `AND`, `OR` | Sets the default boolean operator for multi-term queries. Default is `OR`. |

**Query Parsers Summary**

| **Parser** | **Best For** |
| :- | :- |
| `CLASSIC (Default)` | Standard Lucene syntax with wildcards and boolean operators. |
| `STANDARD` | Advanced regex pattern matching and field-specific queries. |
| `COMPLEX` | Enhanced phrase matching that allows wildcards inside quotes. |
| `MATCHPHRASE` | Autocomplete-style phrase search with prefix matching. |
| `MATCH` | Queries requiring a specific number of term matches (minimum\_should\_match). |

### **Parser-Specific Options**

**1. CLASSIC Parser (Default)**

The standard Lucene query parser. It is robust and handles most boolean and wildcard requirements.

**Options Supported:**

* allowLeadingWildcard: (Boolean) Enable to allow queries like \*term. Default: false.
* defaultOperator: (String) Logic between terms if no operator is provided. Values: OR (default), AND.
* analyzeWildcard: (Boolean) Whether to analyze terms within wildcard queries. Default: false.
* fuzzyMinSim: (Float) Minimum similarity for fuzzy matching (\~). Default: 2.0 (edit distance).

**2. STANDARD Parser**

Based on the StandardQueryParser, this is a more flexible and modern parser that supports advanced expressions like regular expressions.

* **Options Supported:**
  * allowLeadingWildcard: (Boolean) Default: false.
  * defaultOperator: (String) Values: OR, AND.

**3. COMPLEX Parser**

Designed for phrase queries that require internal wildcards (e.g., "standard \*index").

* **Options Supported:**
  * allowLeadingWildcard: (Boolean) Default: false.
  * defaultOperator: (String) Values: OR, AND.
  * inOrder: (Boolean) If true, terms in a phrase must appear in the exact order specified. Default: true.

**4. MATCHPHRASE Parser**

A specialized parser for "starts-with" behavior on phrases, often used for search-as-you-type features.

* **Options Supported:**
  * enablePrefixMatch: (Boolean) When true, the last term of the phrase is treated as a prefix. Default: true.
  * slop: (Integer) The number of "jumps" allowed between words in the phrase. Default: 0.

**5. MATCH Parser**

A simplified parser focused on multi-term matching and match quality control.

* **Options Supported:**
  * minimumShouldMatch: (String) Defines how many terms must match for a document to be returned. Supports:
    * Integer: 2 (exactly two terms).
    * Percentage: 75% (at least 75% of terms).
    * Negative: -1 (all but one term).

## **Examples of Advanced Search Patterns**

### **1. Regex Pattern Matching (.\*text.\*)**

For highly specific patterns that go beyond simple wildcards, you can use regular expressions. This requires the STANDARD parser.

```javascript theme={null}
-- Finds any document containing the sequence "exception" anywhere in a term

SELECT * FROM MyTable 

WHERE TEXT_MATCH(comments, '/.*exception/', 'parser=STANDARD')
```

* **Matches**: "NullPointerException", "Runtime\_exception", "exception\_handler".
* **Note**: Regular expressions are wrapped in forward slashes (/pattern/).

### **2. Field and Anchor Search (^)**

The STANDARD parser allows for more granular control, such as boosting specific terms or using anchors if supported by the underlying analyzer. In Lucene-based searches, the ^ symbol is also frequently used for **Term Boosting**, allowing you to give more weight to certain words.

```javascript theme={null}
-- Finds "java" or "python", but gives "java" twice the relevance weight

SELECT * FROM MyTable 

WHERE TEXT_MATCH(comments, 'java^2 python', 'parser=STANDARD')
```

### **3. Prefix and Suffix Search**

Find documents containing words that start or end with a specific sequence.

**Prefix (Starts With):**

```javascript theme={null}
-- Matches "stream", "streaming", "streamed"

SELECT _FROM MyTable WHERE TEXT_MATCH(comments, 'stream_ ', 'parser=CLASSIC')
```

**Suffix (Ends With):**

```javascript theme={null}
-- Matches "streaming", "processing", "learning"

SELECT * FROM MyTable 

WHERE TEXT_MATCH(comments, '*ing', 'parser=CLASSIC,allowLeadingWildcard=true')
```

### **4. Contains Search**

Find a term regardless of its position or surrounding characters.

```javascript theme={null}
-- Matches "realtime" anywhere in the text

SELECT * FROM MyTable 

WHERE TEXT_MATCH(comments, '*realtime*', 'parser=CLASSIC,allowLeadingWildcard=true')
```

### **5. Exact Match**

Search for a specific term or a specific phrase in its exact order.

* **Single Term Exact Match:**

```javascript theme={null}
-- Matches documents containing the word "java"

SELECT * FROM MyTable WHERE TEXT_MATCH(comments, 'java')
```

* **Exact Phrase Match:** Use double quotes within the query string to ensure terms appear together and in order.

```javascript theme={null}
-- Matches "realtime data processing", but not "data processing realtime"
SELECT * FROM MyTable 
WHERE TEXT_MATCH(comments, '"realtime data processing"', 'parser=CLASSIC')
```

### Other Miscellaneous examples

```javascript theme={null}
-- Use CLASSIC parser with leading wildcard support
SELECT * FROM myTable WHERE TEXT_MATCH(myCol, '*search*', 'parser=CLASSIC, allowLeadingWildcard=true')

-- Use STANDARD parser with AND operator
SELECT * FROM myTable WHERE TEXT_MATCH(myCol, 'term1 term2', 'parser=STANDARD, defaultOperator=AND')

-- Use COMPLEX parser for advanced queries
SELECT * FROM myTable WHERE TEXT_MATCH(myCol, 'complex query', 'parser=COMPLEX')
```

## **Technical Syntax Reference**

For deep-dive syntax on how to write query\_string expressions (e.g., using \~ for fuzzy, ^ for boosting, or \[] for ranges), please refer to the official[Apache Lucene Query Parser Syntax](https://lucene.apache.org/core/2_9_4/queryparsersyntax.html) .

**Note:** Apache Pinot implements Lucene 8.x/9.x logic. While the core syntax remains the same, advanced field-specific features may behave differently based on your Pinot Table Index configuration.

## **Best Practices**

* **Leading Wildcards**: Using *term is slower than term* because it requires a more intensive index scan.
* **Index Requirement**: TEXT\_MATCH only works with Lucene-based text indexes, not native Pinot text indexes.

**Cluster Configuration for Text Search**

When text search queries contain too many terms or clauses, Lucene may throw `TooManyClauses` exceptions, causing query failures. This commonly occurs with:

* Complex boolean queries with many OR conditions
* Wildcard queries that expand to many terms
* Queries with large numbers of search terms To handle such cases, you can increase the maximum clause count at the cluster level. See the [cluster configuration reference](https://github.com/pinot-contrib/pinot-docs/blob/latest/configuration-reference/cluster.md) for the `pinot.lucene.max.clause.count` setting.


# Timestamp Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/timestamp-index

The Timestamp index accelerates time-based queries by pre-computing and indexing multiple time granularities to optimize filtering and grouping operations.

## Overview and Purpose

A timestamp index is a specialized indexing solution that accelerates time-based queries with different time granularities. It's designed to optimize analytics workflows that involve filtering and grouping by common time dimensions like day, month, or year.

The TIMESTAMP data type in StarTree Cloud stores values as millisecond-precision epoch timestamps (long values). While this high precision is valuable for exact timestamps, it's often excessive for analytical queries where users typically want to analyze data at coarser time granularities.

Timestamp indexes are particularly valuable for:

* Range queries on time columns (e.g., queries for specific date ranges)
* Group-by operations with different time granularities (day, week, month, year, etc.)
* Time-series analytics across large datasets
* Any query pattern that involves time transformations like `dateTrunc()` or `dateTimeConvert()`

> [!NOTE]
>
> Without a timestamp index, time-based queries require extracting values at query time, applying transformation functions, and then performing filters or group-by operations, which can be computationally expensive for large datasets.

## How the Index Works

### Core Concepts

Traditional querying of timestamp data requires time value conversions and transformations to be computed at query time. For example, to group data by month, the system would need to convert each timestamp to its corresponding month for every record in the dataset.

The timestamp index in StarTree Cloud optimizes these operations by:

1. **Pre-computing Time Granularities**: During segment generation, the system pre-computes additional derived columns for each configured time granularity.
2. **Derived Column Generation**: For each time granularity (e.g., DAY, MONTH, YEAR), a separate derived column is created following the naming pattern `$${ts_column_name}$${ts_granularity}`.
3. **Automatic Range Indexing**: Range indexes are automatically built for all granularity columns, enabling efficient filtering on time ranges.
4. **Query Rewriting**: At query time, functions like `dateTrunc('DAY', timestamp_col)` are automatically rewritten to use the pre-computed columns, dramatically reducing computation overhead.

### Example Illustration

For a timestamp column named `event_time` with configured granularities DAY, MONTH, and YEAR:

1. Three additional derived columns are generated:
   * `$event_time$DAY` - containing day-level timestamps
   * `$event_time$MONTH` - containing month-level timestamps
   * `$event_time$YEAR` - containing year-level timestamps
2. When a query includes `dateTrunc('MONTH', event_time)`, it's automatically rewritten to use the `$event_time$MONTH` column instead of computing this transformation during query execution.
3. Similarly, a filter condition like `dateTrunc('YEAR', event_time) = '2022'` would use the `$event_time$YEAR` column's range index for efficient filtering.

## Configuration

### Enabling Timestamp Index

To enable a timestamp index on a column in your StarTree Cloud table, add the following configuration to your table definition:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "event_time",
      "timestampConfig": {
        "granularities": [
          "DAY",
          "WEEK",
          "MONTH"
        ]
      }
    }
  ]
}
```

### Supported Granularities

The timestamp index supports the following granularities:

* `MILLISECOND`
* `SECOND`
* `MINUTE`
* `HOUR`
* `DAY`
* `WEEK`
* `MONTH`
* `QUARTER`
* `YEAR`

### Important Configuration Considerations

1. **Data Type Requirement**: Timestamp indexes can only be created on columns with the `TIMESTAMP` data type.
2. **Storage Implications**: Each configured granularity creates an additional derived column, which increases storage requirements. Choose granularities based on your common query patterns.
3. **Index Generation Timing**: Timestamp indexes are created during segment generation, not at query time.
4. **Performance Impact**: While timestamp indexes accelerate query performance, configuring too many granularities may impact segment generation time and storage requirements.

[YouTube video player](https://www.youtube.com/embed/DetGpHZuzDU)

*Mark Needham demos the Timestamp Index*


# Vector Index
Source: https://docs.startree.ai/corecapabilities/manage-data/indexes/vector-index

The Vector index enables efficient vector similarity searches, supporting high-dimensional data queries for applications like recommendation systems and anomaly detection.

## Overview and Purpose

A vector index is a specialized data structure that enables efficient similarity searches across high-dimensional vector embeddings. It transforms complex similarity computations from exhaustive comparisons to approximate nearest neighbor searches, dramatically improving query performance.

In StarTree Cloud (powered by Apache Pinot), vector indexing supports storing and querying embedding vectors generated from machine learning models for use cases like semantic search, recommendation systems, and AI-powered analytics.

Vector indexes are particularly valuable for:

* Semantic search applications using text embeddings
* Image similarity search using visual embeddings
* Recommendation engines based on user or item embeddings
* Anomaly detection using vector representations
* Any query pattern involving similarity comparisons between high-dimensional vectors

> [!NOTE]
>
> Without a vector index, finding similar vectors would require computing distances between the query vector and every vector in the database, which becomes prohibitively expensive as data volume grows.

## How the Index Works

### Core Concepts

Traditional database indexes are designed for exact matches or range queries. However, vector similarity search requires finding the "most similar" items in high-dimensional space, which is fundamentally different.

The vector index in StarTree Cloud:

1. **HNSW Implementation**: Uses Hierarchical Navigable Small World (HNSW) algorithm for efficient approximate nearest neighbor (ANN) search.
2. **Graph-Based Navigation**: Creates a navigable graph structure where similar vectors are connected, enabling efficient traversal to find nearest neighbors.
3. **Multi-Layer Structure**: Organizes vectors in a hierarchical structure that allows for quick pruning of the search space.
4. **Approximate Results**: Trades off perfect accuracy for dramatic speed improvements, finding the most similar vectors with high probability.

### Example Illustration

For a set of product embeddings representing semantic meaning:

1. A traditional approach would compute the distance between a query vector and every product vector in the database.
2. With a vector index, the system:
   * Starts at an entry point in the graph
   * Navigates through connections to increasingly similar vectors
   * Quickly narrows down to a small subset of candidate vectors
   * Returns the most similar vectors without exhaustive comparison

This approach can reduce search time from linear (O(n)) to logarithmic (O(log n)) complexity, enabling real-time similarity search even with millions of vectors.

## Configuration

### Enabling Vector Index

To enable a vector index on a column in your StarTree Cloud table, add the following configuration to your table definition:

```
{
  "fieldConfigList": [
    {
      "name": "embedding",
      "encodingType": "RAW",
      "indexes": {
        "vector": {
          "vectorIndexType": "HNSW",
          "vectorDimension": 1536,
          "vectorDistanceFunction": "COSINE",
          "version": 1
        }
      }
    }
  ]
}
```

### Alternative Configuration (Legacy Method)

You can also use the older configuration format:

```
{
  "fieldConfigList": [
    {
      "name": "embedding",
      "encodingType": "RAW",
      "indexType": "VECTOR",
      "properties": {
        "vectorIndexType": "HNSW",
        "vectorDimension": 1536,
        "vectorDistanceFunction": "COSINE",
        "version": 1
      }
    }
  ]
}
```

### Important Configuration Parameters

1. **vectorIndexType**: Specifies the algorithm used for vector indexing.
   * Currently, only "HNSW" (Hierarchical Navigable Small World) is supported.
2. **vectorDimension**: Defines the number of dimensions in the vector.
   * This must match the dimension of your embeddings (e.g., 1536 for OpenAI models, 768 for BERT base).
3. **vectorDistanceFunction**: Specifies the distance metric for similarity computation.
   * **COSINE**: Measures the angle between vectors (ideal for normalized vectors where orientation matters more than magnitude).
   * **INNER\_PRODUCT**: Computes the dot product (best when vectors are normalized and higher scores indicate greater similarity).
   * **L2**: Measures Euclidean distance (suitable for spatial closeness in high-dimensional space).
   * **L1**: Measures Manhattan distance (sum of absolute differences of coordinates).
4. **version**: Specifies the version of the vector index implementation.
5. **Data Type Requirements**, The column must be:
   * A multi-valued FLOAT array (set `singleValueField: false` in schema)
   * RAW encoded (set `encodingType: "RAW"` in the configuration)


# Merge JSON fields during partial upsert
Source: https://docs.startree.ai/corecapabilities/manage-data/merge-json

Retain and update JSON attributes during realtime ingestion with MERGE_JSON, including append and override modes, configuration examples, and operational guidance.

> [!NOTE]
>
> `MERGE_JSON` will be available starting with the **StarTree Cloud 0.17.0 release**.
> Contact StarTree support to confirm availability in your environment before enabling it. The configuration on
> this page requires StarTree's implementation and does not apply to an unmodified Apache Pinot deployment.

`MERGE_JSON` combines an incoming JSON object with the stored object for the same primary key during ingestion.
Use it when events update individual attributes and should retain attributes omitted from later events.
Queries and JSON indexes see the resulting merged value.

Configure `MERGE_JSON` per column in a realtime partial-upsert table and choose one object-merge mode for the table:

| Mode | Behavior |
| - | - |
| `append` (default) | Merge nested objects recursively and retain missing keys at every level. |
| `override` | Retain missing top-level keys, but replace each supplied value, including entire nested objects. |

Both modes **replace arrays**; `append` does not concatenate them. This is an ingestion configuration, not a SQL
function. Use the same `MERGE_JSON` function name for both modes.

## Example: update a customer profile

Suppose the stored `attributes` column contains:

```json theme={null}
{
  "account": "business",
  "settings": {"theme": "dark", "language": "en"},
  "tags": ["trial"],
  "contact": "old@example.com"
}
```

A later event for the same primary key supplies:

```json theme={null}
{
  "settings": {"language": "fr"},
  "tags": ["paid", "active"],
  "contact": null
}
```

With default `append`, the resulting column is:

```json theme={null}
{
  "account": "business",
  "settings": {"theme": "dark", "language": "fr"},
  "tags": ["paid", "active"],
  "contact": null
}
```

With `override`, the resulting column is:

```json theme={null}
{
  "account": "business",
  "settings": {"language": "fr"},
  "tags": ["paid", "active"],
  "contact": null
}
```

In both cases, `account` survives, `tags` is replaced, and `contact` remains a key with a null value.
Only `append` retains `settings.theme`.

## Configure a table

### Prerequisites

* Confirm with StarTree support that the feature is available on every server that can host the table.
* Start with a correctly configured [realtime partial-upsert table](/recipes/upserts-partial): primary keys, stream
  records partitioned by primary key, a comparison column, null handling, and appropriate routing and assignment.
  The custom-merger settings below replace the recipe's `partialUpsertStrategies` map.
* Include the primary key and comparison value in every update. Choose an ordering value that represents the
  intended update order for each key.
* Use a single-value `JSON` column for variable-shape JSON objects. Serialized JSON-object `STRING` columns and
  native `MAP` values are also supported. MAP keys and values must remain compatible with the column's complex schema.

This feature uses realtime partial-upsert semantics. It does not add partial-upsert support to
[OFFLINE tables](/corecapabilities/ingestdata/adv-concepts/batch/offline-upserts). Keep the existing
[off-heap upsert](/corecapabilities/manage-data/offheap-upsert) metadata and recovery settings for your table.

> [!WARNING]
>
> Existing partial-upsert restrictions still apply. Do not enable `SegmentRefreshTask` on this table. Coordinate
> maintenance, rebalancing, and recovery with StarTree support to preserve per-key update order.

### Schema example

This schema defines the columns used below. `attributes` contains the object to merge; `score` demonstrates a
built-in numeric merge function on the same row.

```json theme={null}
{
  "schemaName": "profile_events",
  "primaryKeyColumns": ["customerId"],
  "dimensionFieldSpecs": [
    {"name": "customerId", "dataType": "STRING"},
    {"name": "attributes", "dataType": "JSON"},
    {"name": "name", "dataType": "STRING"}
  ],
  "metricFieldSpecs": [
    {"name": "score", "dataType": "LONG"}
  ],
  "dateTimeFieldSpecs": [
    {"name": "eventTime", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS"}
  ]
}
```

### Table configuration

Merge these sections into your existing `profile_events` realtime table config. This is a **fragment**, not a
complete table definition: retain the stream, time-column, retention, tenant, routing, and partition-assignment
settings. Preserve existing `metadataManagerClass` and other `metadataManagerConfigs` entries.

```json theme={null}
{
  "tableIndexConfig": {
    "nullHandlingEnabled": true
  },
  "upsertConfig": {
    "mode": "PARTIAL",
    "comparisonColumns": ["eventTime"],
    "partialUpsertMergerClass": "ai.startree.pinot.upsert.merger.ConfigurablePartialUpsertMerger",
    "defaultPartialUpsertStrategy": "OVERWRITE",
    "metadataManagerConfigs": {
      "startree.merger.columns.attributes": "MERGE_JSON",
      "startree.merger.columns.score": "INCREMENT"
    }
  }
}
```

This configuration recursively merges `attributes`, adds each incoming `score` to the previous score, and uses
`OVERWRITE` for other non-key columns such as `name`. Primary-key and comparison columns are never merged.
With no previous value to merge, the incoming value becomes the initial value.

The example enables table-level null handling. Schema-level `enableColumnBasedNullHandling` is also supported;
when using it, keep the fields that can be omitted or null nullable rather than marking them `notNull`.

Do not also configure `partialUpsertStrategies`: it is mutually exclusive with `partialUpsertMergerClass`.
When migrating an existing table, move its column strategies into `startree.merger.columns.` entries.

To select `override`, add this entry to `upsertConfig.metadataManagerConfigs`:

```json theme={null}
{
  "startree.merger.json.mode": "override"
}
```

The mode applies to **every `MERGE_JSON` column in the table**. It cannot be selected separately per column.
It does not change built-in strategies. Built-in `OVERWRITE` replaces an entire non-null column value;
JSON `override` still preserves absent top-level keys inside the column.

### Configuration reference

| Setting | Default | Meaning |
| - | - | - |
| `partialUpsertMergerClass` | Not enabled | Set to the fully qualified class name above. |
| `metadataManagerConfigs.startree.merger.columns.` | No custom function | `MERGE_JSON` or a Pinot built-in strategy, such as `INCREMENT`, `OVERWRITE`, or `FORCE_OVERWRITE`. At least one entry is required. |
| `metadataManagerConfigs.startree.merger.json.mode` | `append` | `append` or `override`; applies to all JSON merge columns. |
| `defaultPartialUpsertStrategy` | `OVERWRITE` | Built-in strategy for non-key columns without a custom function. |

Function names and modes ignore case and surrounding whitespace. Column names are trimmed; duplicate names
after trimming are rejected. Null, blank, or unknown functions/modes, primary-key or
comparison-column targets, and an empty function map fail merger construction.

> [!WARNING]
>
> Merger construction does not validate column existence or type compatibility against the schema. A nonexistent
> column or an incompatible data type can therefore pass table creation. Check the configured column names and types
> before enabling the feature.

## Merge rules and invalid input

| Input at a supplied key | `append` | `override` |
| - | - | - |
| Object replacing an object | Recursively merge. | Replace with the incoming object. |
| Array, scalar, or a change of value type | Replace with the incoming value. | Replace with the incoming value. |
| Explicit JSON null | Set the key to null. | Set the key to null. |
| Key absent from the incoming top-level object | Keep the previous value. | Keep the previous value. |
| Empty incoming top-level object `{}` | Keep all previous keys. | Keep all previous keys. |

An incoming empty nested object such as `{"settings":{}}` retains the previous nested keys in `append` mode,
and clears that nested object's contents in `override` mode. Neither mode provides a key-deletion operator;
omitted top-level keys survive. Use whole-column replacement if that is the intended operation.

**SQL null and JSON null differ.** An absent or SQL-null incoming `attributes` column preserves its previous value
under `MERGE_JSON`. A JSON object such as `{"contact":null}` sets a key to null. The string `"null"` as the entire
serialized document is a non-object input. A built-in `FORCE_OVERWRITE` column can accept SQL null to clear the
whole column; that column then follows replacement semantics instead of JSON merging.

Object merging requires both serialized strings to parse as single JSON objects. Otherwise, the merger uses
the fallbacks below:

| Previous column value | Incoming column value | Result |
| - | - | - |
| Valid object | Malformed JSON, trailing garbage, concatenated documents, array, scalar, or JSON `null` | Keep the previous object. |
| Malformed JSON or any non-object | Any incoming string | Keep the incoming string verbatim, even if it is also invalid. |
| Valid object | Valid object | Merge using the configured mode. |

For example, `{"a":1}` plus `[2]` keeps `{"a":1}`, while `42` plus `{"a":2}` becomes `{"a":2}`.
These are merge-stage fallbacks, not record-validation guarantees: decoding, transforms, and schema conversion
can reject or normalize data before it reaches the merger. A first record has no previous object to preserve.

Two native MAP values merge as maps. A mixed representation such as MAP plus string, or unsupported runtime
values such as numbers, keeps the incoming value. Use a consistent representation for each column.
The merger preserves numeric precision when parsing JSON strings; it does not preserve original whitespace or
key order, or restore precision already lost during decoding or conversion.

## Ordering, recovery, and operations

`MERGE_JSON` follows Pinot's existing partial-upsert ordering and previous-row selection. An older event excluded
by that ordering cannot contribute missing keys later. Preserve per-key ordering upstream; merge operations are
not generally commutative. Equal comparison values are not a deduplication mechanism: an accepted replay can
reapply functions such as `INCREMENT`.

Deletion, metadata expiry, retention, and reinsertion keep their existing Pinot behavior. Updates after deletion
start from the incoming fields rather than inheriting the deleted object's attributes, subject to normal record
ordering. Once previous-key metadata expires, the next record also starts fresh. TTL applies to Pinot's record
and metadata lifecycle, not to individual keys inside a JSON object. The plugin does not implement key deletion,
scripted updates, or the complete Elasticsearch Update API.

For rollout and configuration changes:

1. Confirm availability with StarTree support and coordinate any required deployment update.
2. For a new table, apply the configuration and verify construction succeeds. For an existing table, pause table
   consumption before changing merger settings, then coordinate the configuration update and server restart with
   StarTree support. Resume consumption only after all replicas have recreated their partition mergers with the
   same settings. A rolling mode change
   during consumption can produce different accumulated documents across replicas; a segment reload alone does
   not recreate the partition merger.
3. Ingest two ordered updates for a test key and query the stored result. Check a retained key and a replaced key;
   if using a JSON index, also check that a predicate on the old value no longer matches.
4. Coordinate rollback with StarTree support. Before using a deployment without this feature, remove
   `partialUpsertMergerClass` and its merger-specific metadata settings, and configure the intended built-in upsert
   behavior. Already merged data is not undone.

For the schema above, query the test key after its updates have been consumed:

```sql theme={null}
SELECT customerId, attributes, score
FROM profile_events
WHERE customerId = 'customer-1';
```

The `attributes` result should match the chosen mode in the customer-profile example. If the table has a
[JSON index](/corecapabilities/manage-data/indexes/json-index), test its predicates against the merged document too.

Settings are captured at construction. A mode change is not retroactive, and hot configuration migration is not
provided. Each partition owns its merger and failure state.

Malformed JSON and unsupported types produce rate-limited fallback warnings. A built-in function that throws
keeps the incoming value; after 100 consecutive exceptions for a column, that column switches to `OVERWRITE`
until its merger is recreated. A successful merge resets that exception streak but not the per-column warning
timer; a later failure can still have its warning suppressed within the 10-second interval. The documented JSON
fallbacks return normally and do not trigger this exception latch. Warnings are not a per-record error counter,
and this plugin does not export a dedicated degraded-merge metric.

Keys accumulate over time. For serialized JSON values, a successful object merge parses both strings and serializes
the result. Native MAP values instead incur map allocation and copying, including recursive merging in `append`
mode. Check representative document sizes, nesting, update rates, and ingestion lag before production adoption.

## Troubleshooting

| Symptom | Check |
| - | - |
| Server cannot load `ConfigurablePartialUpsertMerger` | Verify the class name and confirm feature availability with StarTree support. |
| Table creation rejects the custom merger | Remove conflicting `partialUpsertStrategies` and check normal partial-upsert prerequisites. |
| Partition initialization rejects configuration | Check function/mode spelling, null values, empty/duplicate column names, and protected-column targets. |
| A configured column is not merged | Match its schema name and runtime type; inspect the missing-column warning emitted on the first merge. |
| A previous JSON document remains unchanged | Check whether the update was out of order, SQL null, malformed, or a top-level non-object. |
| Nested keys disappear | Check whether the table uses `override`, or the incoming value changes type. |
| A column starts replacing values | Inspect warnings for repeated built-in merge failures and the 100-exception latch; fix the cause before coordinating a restart. |

## Related documentation

* [Off-heap upserts](/corecapabilities/manage-data/offheap-upsert): metadata storage, snapshot, preload, and TTL settings.
* [Partial upserts](/recipes/upserts-partial): the base partial-upsert ingestion model and built-in strategies.
* [JSON index](/corecapabilities/manage-data/indexes/json-index): filter the merged JSON document efficiently.


# Off-Heap Deduplication
Source: https://docs.startree.ai/corecapabilities/manage-data/offheap-dedup

Learn about off-heap dedup and how to use them in StarTree

Apache Pinot natively supports deduplication [dedup](https://docs.pinot.apache.org/basics/data-import/dedup) during real-time ingestion to handle scenarios where duplicate primary keys must be invalidated. Common use cases include invalidation of duplicate user logins (e.g., IP/UserID), prevention of duplicate orders in shipment tracking systems etc. Internally, Pinot servers keep track of dedup metadata to tell which segment holds it right now, so that the server drops duplicate records during ingestion. In the open-source implementation, dedup metadata is stored on-heap, leading to scalability issues when the data grows.

### Challenges with Open Source Dedup Implementation

1. Decline in query performance due to increasing memory overhead.
2. Ingestion delays caused by heap exhaustion and garbage collection.
3. Prolonged server restart times for large tables as metadata is rebuilt.

## StarTree’s Off-Heap Dedup

StarTree introduces off-heap deduplication using RocksDB to manage dedup metadata on disk. This approach reduces heap usage and improves scalability. Key functionalities include:

* **In-memory caching:** Recent reads are served from memory, minimizing disk access.
* **Batched writes:** Latest writes are buffered in memory and flushed to disk in large batches, reducing write latency.
* **Improved server restart times:** Metadata snapshots allow faster bootstrap of dedup tables reducing the preload time.

### Benefits of StarTree Off-Heap Dedup

1. Reduced JVM heap pressure, enabling better scalability for large tables.
2. Improved query latency by reducing contention on heap memory.
3. Minimized ingestion delays through efficient memory and disk usage.
4. Streamlined server restarts with prebuilt metadata snapshots.

## Configuration for Off-Heap Deduplication

Enable off-heap deduplication by setting the `enablePreload` and `metadataManagerClass` parameters in the dedupConfig. These parameters are enabled by default starting from January, 2025.

```json theme={null}
{
    "dedupConfig": {
        "dedupEnabled": true,
        "hashFunction": "NONE",
        "enablePreload": true,
        "metadataManagerClass": "ai.startree.pinot.dedup.rocksdb.RocksDBTableDedupMetadataManager",
        "metadataManagerConfigs": {
             ...
        },
        "dedupTimeColumn": "timeCol"
    }
}
```

> [!NOTE]
>
> Off-heap dedup shares the same server-level RocksDB store as [off-heap upsert](./offheap-upsert), including its `max_open_files` setting — starting 0.16.0 this defaults to **`4096`** (previously `1024`). See [Off-Heap Upserts](./offheap-upsert#configuration) for the full RocksDB store configuration reference.

### Async Removal of Dedup Metadata

StarTree supports asynchronous removal of dedup metadata for deleted segments or expired keys based on Metadata TTL. This ensures consistent cleanup without impacting ingestion or query performance. The configs are enabled by default in StarTree, but one can change the cleanup frequency and key thresholds as shown:

```json theme={null}
{
    "dedupConfig": {
        "metadataManagerClass": "ai.startree.pinot.dedup.rocksdb.RocksDBTableDedupMetadataManager",
        "metadataManagerConfigs": {
            "rocksdb.asyncremoval.enable": "true",
            "rocksdb.asyncremoval.threads": "1",
            "rocksdb.asyncremoval.interval_in_seconds": 3600,
            "rocksdb.asyncremoval.time_threshold_in_seconds": 86400,
            "rocksdb.asyncremoval.key_threshold": 1000000
        }
    }
}
```

## Metadata TTL

With `metadataTTL` in [Open Source Pinot](https://docs.pinot.apache.org/basics/data-import/dedup#metadata-ttl), metadata is removed when next consuming segment starts which can be minutes or even hours, depending on the segment commit frequency. At StarTree, the dedup metadata is cleaned up by the async removal process as mentioned above. This is particularly useful for scenarios where duplicates are unlikely after a certain time.

```json theme={null}
{
    "dedupConfig": {
        "metadataTTL": 86400
    }
}
```

### RocksDB Customization

When using the off-heap dedup, the server creates one RocksDB store to be shared by all the dedup tables for better efficiency. RocksDB stores data in SST (Sorted String Table) files on disk. There are some configurations which can be useful like configuring buffer size at db level and at columnfamily level. DB-level buffer size: Maximum size of all memtables combined across column families (default: 5GB). ColumnFamily buffer size: Maximum size of each memtable before being flushed to disk (default: 100MB).

### Default cluster settings:

```json theme={null}
{
   "pinot.server.kvStoreFactory.class.rocksdb": "ai.startree.pinot.common.rocksdb.metastore.RocksDBStore",
   "pinot.server.kvStoreFactory.rocksdb.datadir": "/home/pinot/data/index/metadata/common",
   "pinot.server.kvStoreFactory.rocksdb.common.delete.on.exit": "true",
   "pinot.server.kvStoreFactory.rocksdb.db.db.write.buffer.size": "5368709120",
   "pinot.server.kvStoreFactory.rocksdb.columnfamily.write.buffer.size": "104857600"
   ...
}
```

Each table partition creates its own `ColumnFamily` in the shared RocksDB store. To customize the table's `ColumnFamily` add the following RocksDB configs in the `metadataManagerConfigs` section. The config names are kept consistent with [those available for RocksDB](https://betterprogramming.pub/navigating-the-minefield-of-rocksdb-configuration-options-246af1e1d3f9).

```json theme={null}
  "dedupConfig" : {
        "enablePreload": true,
        "metadataManagerClass": "ai.startree.pinot.dedup.rocksdb.RocksDBTableDedupMetadataManager",
        "metadataManagerConfigs": {
            "rocksdb.blockcache.size_bytes": "1073741824",
            "rocksdb.rowcache.size_bytes": "104857600"
            ...
        }
    }
```

## Dedup Snapshot Creation

For large tables with millions or billions of primary keys, server restart times can be significantly reduced using prebuilt dedup metadata snapshots during preload. These snapshots are created by minion using the `DedupSnapshotCreationTask`, and persisted in deep store, later imported by servers during restart. This mechanism minimizes restart delays by eliminating the need to rebuild metadata using read and write for each record.

![](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/manage-data/images/startree-offheap-dedup.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=50385ca8d7bbdd8c09c61314989cd9e0)

This minion task runs periodically to keep updating the prebuilt dedup metadata incrementally. To enable the minion task, add the following task configs. See [the Pinot docs](https://docs.pinot.apache.org/basics/concepts/components/cluster/minion) to understand how to schedule a minion task.

```json theme={null}
   "task": {
      "taskTypeConfigsMap": {
        "DedupSnapshotCreationTask": {
         "schedule": "0 0 12 * * ?"
        }
      }
    }
```

### Partition Configuration

The `DedupSnapshotCreationTask` requires partition information to schedule tasks effectively. If the table has`segmentPartitionConfig` with a single partition column, the task uses the `numPartitions` field:

```json theme={null}
     "tableIndexConfig": {
        "segmentPartitionConfig": {
        "columnPartitionMap": {
          "": {
            "numPartitions": ,
            ...
          }
        }
      }
    }
```

For tables with multiple partition columns or missing `segmentPartitionConfig`, explicitly specify the partitions:

```json theme={null}
    "dedupConfig": {
      ...
      "metadataManagerConfigs": {
        "rocksdb.preload.num_partition_overwrite": ""
      }
    }
```

As tasks complete, following Restful APIs can be used to inspect the prebuilt dedup metadata.

```
/dedupSnapshots/{tableNameWithType}/names
/dedupSnapshots/{tableNameWithType}/{snapshotName}/metadata
/dedupSnapshots/{tableNameWithType}/latest
```

Note that using the health check `PREBUILT_SNAPSHOT_CHECK`, we can identify those dedup tables whose snapshot generation task is disabled in a cluster.

## Segment Operation Throttling

When segments are added to a server, it undergoes various operations to come online and be ready for query processing. One of these expensive operations is related to updating the RocksDB state for dedup tables if snapshots are not available. Throttling has been added to limit the number of segments that can undergo this operation to limit the resource utilization usage during this operation. The following configs have been added to control the throttle threshold:

* Before serving queries config: `pinot.server.max.segment.rocksdb.parallelism.before.serving.queries`
* After serving queries config: `pinot.server.max.segment.rocksdb.parallelism`

More about segment operations throttling and how to make changes can be found in [the Pinot docs](https://docs.pinot.apache.org/operators/tutorials/segment-operations-throttling).

## Ingestion Pause on too many Primary Keys per Server

When the number of primary keys per server gets too large, this can result in various issues:

* RocksDB latency increase on query and ingestion path
* Server restarts and rebalance scenarios can become slow even when snapshots exist

To prevent the system from getting into the above state, we will now pause ingestion for dedup tables when a threshold is reached on any server for that table. The configs of interest:

* The primary key count threshold (defaults to 3 billion): `controller.primary.key.count.threshold`
* The primary key count check timeout (defaults to 30 seconds): `controller.primary.key.count.check.timeoutMs`

Other related configs common with the DiskUtilization check mechanism (which also pauses ingestion) are:

* Flag to enable / disable the resource utilization check: `controller.enable.resource.utilization.check`
* Resource utilization check frequency: `controller.resource.utilization.checker.frequency`
* Initial delay to start the resource utilization check: `controller.resource.utilization.checker.initial.delay`

## Frequently Asked Questions

> **Can we enable deduplication on existing real-time tables?**
>
> Enabling `dedupConfig` on existing real time tables is not advised and it will not eliminate any duplicate records from the existing segments, but newly ingested data will start to get deduplicated after enabling dedup. If the user still wishes to enable it on the existing table, add the `dedupConfig` and restart the server, beware they may see duplicates in existing segments until the segments get deleted from the table, for example due to data retention.

> **Is it possible to convert an upsert table to a dedup table?**
>
> It is possible to convert an upsert table to dedup table but it's recommended to run compaction task [upsert compaction](https://docs.pinot.apache.org/manage-data/data-import/upsert-and-dedup/segment-compaction-on-upserts) to remove invalid docs from existing segments first, so that those invalid docs won't become duplicates when enabling dedup. A server restart is needed to convert an upsert table to a dedup table.
>
> Currently, enabling both upsert and dedup on one table is not supported.

> **What if duplicates are seen in a Dedup table?**
>
> Note that the upstream data needs to be partitioned correctly, and a primary key should not exist in different partitions as it will be served by different servers. Make sure `metadataTTL` in `dedupConfig` is set to an approipriate value and duplicate primary keys are not ingested beyond this ttl window.


# Off-Heap Upserts
Source: https://docs.startree.ai/corecapabilities/manage-data/offheap-upsert

Learn about off-heap upserts and how to use them in StarTree

Pinot supports [upserts](https://docs.pinot.apache.org/build-with-pinot/ingestion/upsert-dedup/upsert) during real-time ingestion — updating or deleting records after they're ingested.

Servers track upsert metadata — a map from primary key → segment + docId of its latest version. When a new record arrives, the old one's bit is cleared from its segment's `validDocIds` bitmap; the old row stays on disk, but queries skip it. This metadata grows with the table.

Open-source Pinot keeps upsert metadata on the JVM heap. That's fine at small scale, but grows expensive as primary-key volume grows — it eats heap, slows ingestion and queries, and vanishes on restart (forcing a full segment scan to rebuild).

StarTree's off-heap upserts move that metadata onto a disk-backed [RocksDB](https://betterprogramming.pub/navigating-the-minefield-of-rocksdb-configuration-options-246af1e1d3f9) store shared across all upsert tables on the server. Scales with disk instead of heap, and prebuilt metadata lets servers hydrate on startup instead of rebuilding from segments.

![Off heap upsert](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/manage-data/images/startree-offheap-upsert.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=003c517745e5704d9cc7bed7a777898c)

Despite being disk-backed, recent reads are served from an in-memory cache and writes are batched before flushing to disk, so ingestion speed isn't affected. The query path itself is unchanged — it still reads the same bitmaps on-heap upserts use — so query latency stays comparable, and often improves thanks to the reduced heap pressure.

## Merge JSON fields during partial upsert

Starting with the **StarTree Cloud 0.17.0 release**, [MERGE\_JSON](/corecapabilities/manage-data/merge-json) will let
you update selected JSON attributes while retaining omitted attributes. It supports recursive `append` (default)
and nested-object `override` modes on realtime partial-upsert tables. Confirm availability in your environment before enabling it.
The guide covers the custom merger configuration and how it works alongside existing off-heap metadata settings.

## OFFLINE Upserts

StarTree also supports upserts on [OFFLINE tables](/corecapabilities/ingestdata/adv-concepts/batch/offline-upserts), currently in beta while testing continues. OFFLINE upsert shares the same off-heap RocksDB metadata store described below. The rest of this page describes real-time upserts unless noted otherwise.

Minion tasks are not supported in the latest release yet — planned for upcoming iterations. This includes OSS tasks (MergeRollup, UpsertPurgeTask), SegmentRefreshTask (SRT v1 and v2), StarTreeAlterTableTask, and UpsertCompactionTask / UpsertCompactMergeTask.

**Available today:** Only FULL upsert mode is supported — PARTIAL upsert does not work with OFFLINE upsert tables. Fast metadata recovery via UpsertSnapshotCreationTask is a work in progress.

## Configs

Nothing to configure to get started. Off-heap upsert, snapshot, and preload are on by default for real-time tables, and both [FULL and PARTIAL upsert modes](https://docs.pinot.apache.org/build-with-pinot/ingestion/upsert-dedup/upsert#upsert-modes) work out of the box — no configs beyond the standard [Apache Pinot upsert configs](https://docs.pinot.apache.org/build-with-pinot/ingestion/upsert-dedup/upsert) are required. The settings below are optional knobs for tuning or overriding defaults.

> [!NOTE]
>
> Adding, removing, or changing `upsertConfig` requires a server restart — the metadata manager is built once at server startup.

Two places to customize:

* Server / cluster configs — where to store upsert metadata on disk, RocksDB store initialization.
* RocksDB tuning configs — write buffer, block cache, row cache, etc. Names mirror [RocksDB's own](https://betterprogramming.pub/navigating-the-minefield-of-rocksdb-configuration-options-246af1e1d3f9); defaults work well for most workloads.

```json theme={null}
{
   "pinot.server.kvStoreFactory.class.rocksdb": "ai.startree.pinot.upsert.rocksdb.metastore.rocksdb.RocksDBStore",
   "pinot.server.kvStoreFactory.rocksdb.datadir": "/home/pinot/data/index/metadata/upsert",
   "pinot.server.kvStoreFactory.rocksdb.upsert.delete.on.exit": "true",
   "pinot.server.kvStoreFactory.rocksdb.db.db.write.buffer.size": "5368709120",
   "pinot.server.kvStoreFactory.rocksdb.columnfamily.write.buffer.size": "104857600"
   ...
}
```

Server-level configs — applied once at server startup, so a server restart is required to pick up changes.

Table-level upsert configs live in the table config. Defaults are already on — snapshot and preload let servers recover upsert metadata quickly on restart.

Each table partition gets its own RocksDB `ColumnFamily` in the shared store. To tune it, add RocksDB configs under `metadataManagerConfigs` — the config names match [RocksDB's own](https://betterprogramming.pub/navigating-the-minefield-of-rocksdb-configuration-options-246af1e1d3f9).

```json theme={null}
  "upsertConfig" : {
        "enableSnapshot": true,
        "enablePreload": true,
        "metadataManagerClass": "ai.startree.pinot.upsert.rocksdb.RocksDBTableUpsertMetadataManager",
        "metadataManagerConfigs": {
            "rocksdb.blockcache.size_bytes": "2147483648"
            ...
        }
    }
```

### Disable off-heap upsert for an entire environment

To turn off the RocksDB-backed upsert manager server-wide and fall back to the open-source on-heap implementation, set the following server config and restart (it's read once at server startup, and defaults to `true`):

```
"pinot.server.upsert.startree.default": "false"
```

This only stops the server from *injecting* the RocksDB manager as the default. A table with this already set in its `upsertConfig` keeps using it regardless — remove it too for that table to fall back to the on-heap manager:

```json theme={null}
"metadataManagerClass": "ai.startree.pinot.upsert.rocksdb.RocksDBTableUpsertMetadataManager"
```

This also requires a server restart to take effect.

### Enable async removal of upsert metadata

When a table has a lot of primary keys, its upsert metadata in RocksDB can be huge, and cleaning it up can take a long time. This async removal feature moves that cleanup onto a dedicated background thread pool, instead of blocking the thread doing the actual ingestion or segment-load work.

This only removes primary-key entries from the RocksDB metadata store, invalidating those keys for future upsert resolution — it does not delete or compact the underlying segment data itself. Reclaiming that storage still requires [Segment Refresh Task](#segment-refresh-task-srt).

This feature is **enabled by default**, and there's no need to specify the configs unless you want to modify the default configurations.

```json theme={null}
    "upsertConfig" : {
        "enableSnapshot": true,
        "enablePreload": true,
        "metadataManagerClass": "ai.startree.pinot.upsert.rocksdb.RocksDBTableUpsertMetadataManager",
        "metadataManagerConfigs": {
          "rocksdb.asyncremoval.enable": "true",
          "rocksdb.asyncremoval.threads": "1",
          "rocksdb.asyncremoval.interval_in_seconds": "3600"
        }
    }
```

Like every other `upsertConfig` field, this is read once when a table's upsert metadata manager is built — a server restart is required for changes to these configs to take effect.

## Accelerating Server Restarts

When a server restarts, it needs to recover the upsert metadata (primary key → record location) for every partition it hosts before it can resume real-time ingestion and serve correct query results. Off-heap upserts try to avoid the most expensive path — rebuilding metadata by scanning every segment — using a priority-ordered set of recovery paths:

1. **Reuse the local RocksDB state** left behind by the previous server process, if it is still on disk and still valid for the current table/segment configuration. This is the fastest path since no metadata is recomputed at all.
2. If local state can't be reused or is corrupted, **import a prebuilt snapshot** produced by the `UpsertSnapshotCreationTask` minion task from the deep store. This is fast and also works when the server has lost its local disk entirely (e.g. after a node replacement).
3. If neither is available, **fall back to a full rebuild** — reading local `validDocIds` snapshots (or, absent those, scanning raw segments) to reconstruct metadata from scratch. This is the slowest path and is the same mechanism the open-source on-heap implementation always uses.

The rest of this section explains [path 1](#reuse-mode) and [path 2](#use-upsertsnapshotcreationtask) in detail.

### Reuse Mode

On a **graceful** server shutdown, the RocksDB column family backing a partition's upsert metadata is flushed to disk (rather than deleted) and a completion marker is written. On the next restart, if the server determines that this on-disk state is still valid, it reuses it directly instead of rebuilding metadata — this is what we refer to as **reuse mode** (StarTree's engineering blog calls the same mechanism ["Smart Metadata Persistence"](https://startree.ai/resources/accelerating-pinot-server-restarts-for-upserts-at-scale/)).

Reuse mode is enabled by default and applies to REALTIME upsert tables in FULL mode only. It is not compatible with PARTIAL upsert, `dropOutOfOrderRecord` / `outOfOrderRecordColumn`, or pauseless consumption — these always fall back to snapshot or full rebuild.

See [Accelerating Pinot Server Restarts for Upserts at Scale](https://startree.ai/resources/accelerating-pinot-server-restarts-for-upserts-at-scale/) for a deeper explanation.

#### Disabling Reuse Mode

Reuse mode is on by default. To disable it cluster-wide, set the following server flag (dynamically updatable at runtime, no restart required, defaults to `true`):

```
pinot.server.kvStoreFactory.rocksdb.common.reuse.rocksdb: false
```

To disable it for a single table instead, set the per-table equivalent:

```json theme={null}
"upsertConfig": {
    "metadataManagerConfigs": {
        "partition.preload.reuse_rocksdb": "false"
    }
}
```

### Use UpsertSnapshotCreationTask

Reuse mode can't be used when the server's local disk is lost, when a crash leaves the on-disk snapshot stale, or when a rebalance moves partitions to a new server. In those cases, StarTree automatically falls back to `UpsertSnapshotCreationTask`.

`UpsertSnapshotCreationTask` addresses both problems. This minion task prebuilds a RocksDB-backed snapshot of the primary-key → record-location state for each table partition (incrementally updating the previous version where possible instead of always starting from scratch) and uploads it to the table's deep-store snapshot directory. Servers simply download and import this prebuilt snapshot when loading upsert tables.

This task runs automatically for every real-time upsert table in StarTree — no opt-in needed. If a table doesn't have an explicit `UpsertSnapshotCreationTask` entry under `task.taskTypeConfigsMap`, the controller schedules it on a default cron of `0 0 0/12 * * ?` (every 12 hours).

All fallbacks and the restart flow are handled automatically in the background — no explicit configuration is required. If reuse mode can't be applied, the server transparently falls back to importing a minion-built snapshot, or a full rebuild.

To change this default cadence cluster-wide instead of per table, set the following controller config:

```
"controller.startree.task.manager.UpsertSnapshotCreationTask.cron": "0 0 0/6 * * ?"
```

Defaults to `0 0 0/12 * * ?` (every 12 hours) if unset.

To customize the default schedule on a table:

```json theme={null}
   "task": {
      "taskTypeConfigsMap": {
        "UpsertSnapshotCreationTask": {
         "schedule": "0 0 0/6 * * ?"
        }
      }
    }
```

To disable the default schedule on a single table, add the task entry but omit `schedule`:

```json theme={null}
   "task": {
      "taskTypeConfigsMap": {
        "UpsertSnapshotCreationTask": {}
      }
    }
```

More about how to operate the minion tasks can be found in [the Pinot docs](https://docs.pinot.apache.org/architecture-and-concepts/components/cluster/minion).

As tasks complete, these Restful APIs can be used to inspect — or, if needed, clean up — the prebuilt upsert metadata:

```
GET    /upsertSnapshots/{tableNameWithType}/names                                           // List all snapshot names for the table
GET    /upsertSnapshots/{tableNameWithType}/latest                                          // Get the latest snapshot name for each partition
GET    /upsertSnapshots/{tableNameWithType}/{snapshotName}/metadata                          // Get metadata for one specific snapshot
DELETE /upsertSnapshots/{tableNameWithType}/snapshotAndMetadata?snapshotName={snapshotName}  // Delete a snapshot and its metadata
```

## Metadata TTL and deletedKeys TTL

The `metadataTTL` and `deletedKeysTTL` configs, as described in the [Pinot docs](https://docs.pinot.apache.org/build-with-pinot/ingestion/upsert-dedup/upsert#metadata-time-to-live-ttl), work with off-heap upserts too. `metadataTTL` ages out primary keys older than the TTL window. This cleanup only purges entries out of the RocksDB upsert-metadata store — it does not delete or compact the underlying segment data itself. Cleanup is performed by the async removal feature described above, so it doesn't block the start of new consuming segments, and the `UpsertSnapshotCreationTask` minion task is also TTL-aware, so stale metadata past the TTL is excluded from newly-built snapshots.

A few things to know specifically about `metadataTTL`:

* `metadataTTL` doesn't work with [SRT](#segment-refresh-task-srt).

A few things to know specifically about `deletedKeysTTL`:

* Used together with `deleteRecordColumn`, it purges tombstoned/deleted keys from the RocksDB upsert-metadata store after the TTL window — this doesn't delete or compact the underlying segment data itself, so use [Segment Refresh Task](#segment-refresh-task-srt) to actually reclaim segment storage for rows that TTL has aged out.
* `deletedKeysTTL` is in testing phase.
* For `deletedKeysTTL` to work reliably, set `preload.enforce_prebuilt_snapshot` in `metadataManagerConfigs` to enforce an incremental snapshot build:

  ```json theme={null}
  "upsertConfig" : {
      "metadataManagerConfigs": {
        "preload.enforce_prebuilt_snapshot": "true"
      }
  }
  ```

  > [!NOTE]
  >
  > `preload.enforce_prebuilt_snapshot` is **not** the same config as `rocksdb.preload.use_prebuilt_snapshot`. The latter is enabled by default and controls whether preload uses the minion-built snapshot to preload metadata during startup.
* Keep `UpsertSnapshotCreationTask` running regularly — cleanup is gated behind a fresh snapshot watermark, and simply stays disabled without one rather than risking key revival.

## Segment Refresh Task (SRT)

Off-heap upserts on their own don't reclaim space for records that have been overwritten or deleted — segments still contain the old, now-invalid rows until something physically removes them. StarTree's [Segment Refresh Task (SRT)](/corecapabilities/manage-data/upsert-compaction-srt) is the recommended minion task for this: it compacts out invalid rows and merges small segments together in a single pass, replacing segments atomically so queries never see a partial swap. See the full [Segment Refresh Task](/corecapabilities/manage-data/segment-refresh-task) documentation for configuration details, and the [SRT best practices](/corecapabilities/manage-data/upsert-compaction-srt#best-practices-for-running-segment-refresh-task-srt-on-upsert-tables) for scheduling and sizing guidance.

## Recommended Minion Tasks

StarTree recommends the following minion tasks for upsert tables — using them together is what keeps storage and query costs down and query performance up, without sacrificing correctness:

* **[Segment Refresh Task](#segment-refresh-task-srt)** for compaction and merging of overwritten/deleted rows, in a single atomic pass.
* **`UpsertSnapshotCreationTask`**, [enabled and scheduled by default](#use-upsertsnapshotcreationtask), for fast restart recovery. It's good practice to have it run between SRT passes, so restarts and rebalances always import a snapshot that already reflects SRT's latest compaction.
* **`SegmentPurgeTask`** for explicit, predicate-driven deletion of specific rows (for example a right-to-be-forgotten request) — a different problem from TTL-based cleanup. `metadataTTL`/`deletedKeysTTL` already invalidate aged-out keys automatically (see [Metadata TTL and deletedKeys TTL](#metadata-ttl-and-deletedkeys-ttl)); it's SRT, not a separate TTL-specific task, that physically compacts those invalidated rows out of segments.
* **File Ingestion Task** is supported for FULL upsert tables only — it does **not** work with PARTIAL upsert tables. Running it on a table that also runs SRT is not recommended — see [SRT with File Ingestion Task](#best-practices) in Best Practices.

The older OSS-inherited tasks are generally not recommended on StarTree, since SRT already covers the same compaction/merge functionality with cross-segment correctness guarantees and scale that they lack:

* **[`enableCommitTimeCompaction`](https://docs.pinot.apache.org/build-with-pinot/ingestion/upsert-dedup/upsert#enable-commit-time-compaction-for-storage-optimization)** compacts a segment only once at commit time and does not guarantee correctness across replicas, whereas SRT re-evaluates and compacts on every run.
* **`UpsertCompactionTask`** compacts one segment at a time and never merges, so it tends to leave behind many small segments — hurting query performance and, at scale, ingestion rate. Correctness may also not be guaranteed when it's enabled together with `deleteRecordColumn` or TTL-based configs (`metadataTTL`/`deletedKeysTTL`).
* **`UpsertCompactMergeTask`** does merge segments, but only schedules one eligible merge group per partition per cycle, which doesn't scale to tables with many partitions or frequent compaction needs.

## Guardrails

### Segment Operation Throttling

Concurrent segment loads and updates can trigger expensive RocksDB operations. Throttling caps how many segments can run these operations in parallel:

> [!NOTE]
>
> These configs are dynamically updatable at runtime — **no server restart is required** for changes to take effect.

* The before-serving-queries throttle threshold, defaults to `max(1, number of CPU cores / 4)`:

  ```
  "pinot.server.max.segment.rocksdb.parallelism.before.serving.queries": "4"
  ```

* The after-serving-queries throttle threshold, defaults to `max(1, number of CPU cores / 4)`:

  ```
  "pinot.server.max.segment.rocksdb.parallelism": "4"
  ```

More about segment operations throttling and how to make changes can be found in [the Pinot docs](https://docs.pinot.apache.org/operate-pinot/tuning/segment-operations-throttling).

### Restricting Primary Keys per Server

When the number of primary keys per server gets too large, this can result in various issues:

* RocksDB latency increase on query and ingestion path
* Server restarts and rebalance scenarios can become slow even when snapshots exist

To prevent the system from getting into the above state, we will now pause ingestion for upsert tables when a threshold is reached on any server for that table. The configs of interest:

```
"controller.primary.key.count.threshold": "3000000000"
```

> [!NOTE]
>
> A **controller restart** is required for this config change to take effect.

### Consuming Segment Consistency Mode

Upsert metadata can briefly diverge across replicas — a lagging replica can return a stale record. This is generally safe for FULL upsert, but unsafe for PARTIAL upsert and for FULL tables using `dropOutOfOrderRecord`/`outOfOrderRecordColumn`, since both depend on an accurate record-location pointer.

Control with `pinot.server.consuming.segment.consistency.mode` (no restart needed):

```
"pinot.server.consuming.segment.consistency.mode": "PROTECTED"
```

* `RESTRICTED` (default) — blocks force-commit/reload during the risky transition.
* `PROTECTED` — allows it, then reconciles afterward (resets each affected key's pointer on consuming-segment removal).
* `UNSAFE` — no reconciliation; not recommended for production.

On PARTIAL upsert tables (and FULL tables with `dropOutOfOrderRecord`/`outOfOrderRecordColumn`), force-commit, pause-consumption, and reload are blocked under `RESTRICTED` — without reconciliation, these operations can leave upsert metadata pointing at the wrong record. Switch to `PROTECTED` to allow them safely.

See the [Apache Pinot docs on handling inconsistencies](https://docs.pinot.apache.org/build-with-pinot/ingestion/upsert-dedup/upsert#handling-inconsistencies) for the metrics that surface inconsistent rows.

### Minion Task Segment Selection Consensus

Minion tasks that rewrite or merge segments — `UpsertCompactionTask`, `UpsertCompactMergeTask`, [Segment Refresh Task](#segment-refresh-task-srt), and `StarTreeAlterTableTask` — share the same safeguard:

* Every replica hosting the segment is queried for its `validDocIds` metadata.
* All responding replicas must report the same valid-doc count and a CRC matching ZooKeeper's segment metadata.
* If replicas disagree, or too few respond, the segment is skipped rather than rewritten on stale state.

### Table Config Immutability

We restrict changes to certain `upsertConfig` fields once a table is created, because changing them can silently break upsert correctness for data ingested before the change. Disallowed from changing — see the OSS docs on [Immutable upsert configuration fields](https://docs.pinot.apache.org/build-with-pinot/ingestion/upsert-dedup/upsert#immutable-upsert-configuration-fields):

* The schema's `primaryKeyColumns`
* `mode`
* `hashFunction`
* `comparisonColumns` (and `timeColumnName` when used as the implicit comparison column)
* `deleteRecordColumn`
* `dropOutOfOrderRecord`
* `outOfOrderRecordColumn`

This is only enforced by a rejected API call — `force=true` bypasses the check, but should be reserved as a last resort: pause ingestion and restart servers first, since using it without doing so can corrupt upsert correctness for data ingested before the change. The recommended way to change one of these is to create a new table with the desired configuration and reingest.

By contrast, `enableSnapshot`, `enablePreload`, `metadataTTL`, `deletedKeysTTL`, and the partial-upsert merge strategy/merger class are explicitly safe to change on an existing table at any time — though all of them are read once when a table's upsert metadata manager is built, so a server restart is needed before any of these new values actually take effect.

For the safest rollout when changing any of these, see [Safest rollout for mutable upsert config changes](#best-practices) in Best Practices.

## Record Comparison and Tie Breakers

> [!TIP]
>
> **The greater comparison value wins. Ties are broken by where the record sits, not by anything in the record.** If your application needs a particular update to win, put that order in the comparison value.

The comparison value comes from the comparison columns you configured for the table, or from `timeColumnName` when none is set. This holds for OFFLINE tables too, where segment push time only breaks ties (see [Which timestamp is used](#which-timestamp-is-used)). When two records share a primary key, the one with the greater value is current. With full upsert, the winning record replaces the current one whole. With partial upsert, StarTree merges a winning streaming record into the current one using the column strategies you configured.

Equal values happen even when every update carries its own value. The stream can deliver one event twice, and a commit, reload, refresh, or upload can leave copies of one record in two segments. The walk below decides those cases.

### Which record wins

StarTree asks these questions in order and stops at the first one that separates the two records.

1. **Is one comparison value greater?** That record wins.
2. **Did the record just arrive from the stream?** The new record wins.
3. **Are both records in the same segment?** The record with the higher document ID wins. Document IDs follow stored order, which can differ from arrival order when the segment is sorted.
4. **Is the segment that holds the current record being replaced, for example on commit or reload?** The record in the replacement segment wins.
5. **Is either segment a refreshed segment?** This step exists only with off-heap upsert. Three checks run in order:
   * The segment with the higher sequence number wins. A segment with a sequence number beats one without.
   * At the same sequence number, the refreshed segment beats the unrefreshed one. This is how a refreshed segment takes over from the segments it was built from.
   * If both are refreshed, the segment with the later timestamp wins.
6. **Are both segments LLC segments?** The segment with the higher sequence number wins.
7. **Otherwise**, the segment with the later timestamp wins.
8. **Still tied?** The current record stays. Two rare exceptions are listed under [edge cases](#edge-cases).

### How to recognize a segment from its name

Steps 5 and 6 depend on the segment type, and the name tells you which type you have. The examples use table `orders`, partition 3.

* **LLC segment**, built by the server from the stream: `orders__3__12__20260915T0930Z`. The third field, 12, is the sequence number.
* **Refreshed segment**, written by [Segment Refresh Task](#segment-refresh-task-srt): `refreshed__orders__3__20260915T0930Z__12`. It starts with `refreshed__`, and the last field is the highest sequence number among the segments it was built from.
* **Uploaded segment**, pushed by a job or a backfill: `uploaded__orders__3__20260915T0930Z__1`. The first field is a prefix the uploader chooses. It has no sequence number, so it can only win a tie on timestamp.

### Which timestamp is used

The timestamp is the one recorded in ZooKeeper, so every replica reads the same value and picks the same winner. REALTIME tables use the segment creation time. OFFLINE tables use the segment push time, and fall back to the creation time when the push time is missing.

### Three cases you will meet

**The same event arrived twice from Kafka.** Both copies have the same key and value. Step 2 makes the second copy current. With full upsert nothing visible changes, because the two records are identical. With partial upsert the merge runs again on the second copy, so a strategy that is not safe to repeat, such as `INCREMENT`, changes the stored value. Equal comparison values do not deduplicate replays.

**A backfill segment was uploaded for a key the stream also carries, with the same comparison value.** One segment is LLC and one is uploaded, so steps 5 and 6 do not apply. Step 7 picks the segment with the later ZooKeeper creation time. If you need the backfill to win, give it a greater comparison value instead of relying on upload timing.

**Segment Refresh Task rewrote a segment and both copies are loaded during the swap.** The source is `orders__3__12__…` and the output is `refreshed__orders__3__…__12`. Step 5 sees the same sequence number on both and picks the refreshed one.

### Edge cases

* If no ZooKeeper timestamp is set, StarTree uses the index creation time stored inside the segment.
* At step 8, when the timestamps are equal and neither segment is refreshed, StarTree prefers the segment named in the standard uploaded-realtime format. If both segments use that format, or neither does, the current record stays.

## Diagnosing Duplicates and Count Mismatches

If the surprise is which of two records with the same primary key won, start with [Record Comparison and Tie Breakers](#record-comparison-and-tie-breakers). The techniques below are for counts that are off.

If a query is returning more rows than expected for a primary key, or the table's total count is running ahead of the source system, these are the techniques StarTree engineers use to narrow down where the divergence is coming from:

* **Compare against the raw, un-deduplicated rows.** `SELECT ... FROM myTable OPTION(skipUpsert=true)` bypasses upsert masking entirely and returns every physically stored row, including ones the upsert layer considers superseded. A large gap between this count and the normal count is expected and healthy; if the two counts are close, masking itself has failed for some rows.

* **Trace a duplicate back to its segment, doc, and host.** Project the virtual columns `$segmentName`, `$docId`, and `$hostName` alongside the primary key to pinpoint exactly which segment, docId, and server a duplicate is coming from. If a duplicate for the same PK is served from a segment that should have been superseded, look at partitioning (see [Avoiding duplicates](#best-practices) below) or a stale local RocksDB state on that host.

* **Detect cross-replica upsert-metadata divergence.** Run the same query with `OPTION(useStrictReplicaGroup=true)` and without it, and compare counts. A difference means replicas disagree on which record is currently valid for some PKs — for example due to a bad [reuse-mode](#reuse-mode) checkpoint, or a consuming-segment-commit race (see [Consuming Segment Consistency Mode](#consuming-segment-consistency-mode)).

* **Recover from a snapshot that has baked in bad state.** A snapshot (from [reuse mode](#reuse-mode) or [`UpsertSnapshotCreationTask`](#use-upsertsnapshotcreationtask)) can capture bad state — a plain reload or restart just re-trusts it, so nothing changes. To actually fix it:
  * Pause ingestion for the table.
  * Reload the affected segments with `forceDownload=true` to rebuild metadata from a fresh segment scan instead of the snapshot.
  * If you need to restart instead, also disable reuse mode first — otherwise the restart just re-trusts the same bad state:

    ```
    "pinot.server.kvStoreFactory.rocksdb.common.reuse.rocksdb": "false"
    ```

* **Check whether `metadataTTL` evicted the key before a late update arrived.** `metadataTTL` assumes no more updates will come in for a primary key once its metadata is removed from RocksDB — if an update does arrive after that point, it's no longer recognized as an update at all, and lands as a brand-new record alongside the existing row instead of replacing it. If the duplicated PK's original ingestion time is older than your `metadataTTL` window, this is likely the cause.

## Best Practices

* **Always set a comparison column.** Neither `comparisonColumns` nor `timeColumnName` is actually required by validation — if you set neither, Pinot silently falls back to ordering by segment creation time, which has nothing to do with the actual event order in your data. Always configure a real comparison column so "latest wins" is deterministic and reflects your source data, not arrival order.
* **Ingestion transform functions**: Do not use a non-deterministic transform function (for example `now()`, `ago()`, or `agoMV()`) on the primary key, comparison column(s), or partition column — each server evaluates transforms independently per replica, so a non-deterministic function can produce a different value per replica for the same source record and diverge query results across replicas. Not validated or blocked today. This includes the case where `timeColumnName` is implicitly used as the comparison column: Pinot doesn't distinguish a genuine event-time column from one populated by an ingestion-time transform, so an ingestion-time `timeColumnName` degrades to the same non-determinism as using `now()` directly.
* **Partitioning**: Size the partition count so that each server-hosted partition's primary-key volume stays well within what a single RocksDB column family handles comfortably (typically well under a billion keys — see [Restricting Primary Keys per Server](#restricting-primary-keys-per-server)), and avoid so many partitions per server that RocksDB compaction/flush overhead starts to dominate. Treat the partition count as immutable once set: Pinot's backward-compatibility validation doesn't check partition-count changes today, and changing it re-maps primary keys to different partitions than their historical records live in, which can produce duplicate or incorrect upsert resolution.
* **Avoiding duplicates**: Make sure upstream partitioning is set up correctly so the same primary key always lands in the same partition. Upsert metadata is scoped per partition at the server level, so a key split across two partitions is never reconciled by either partition's metadata manager, and both copies can end up visible as duplicates. This isn't detected or validated at ingestion time today.
* **Comparison column type**: Use `INT`, `LONG`, `FLOAT`, `DOUBLE`, `BIG_DECIMAL`, or `STRING`. `BYTES` comparison columns are not supported by the RocksDB-backed upsert manager, even though the on-heap implementation supports any `Comparable` type.
* **Rebalance**: Before rebalancing an off-heap upsert table, make sure `UpsertSnapshotCreationTask` is enabled and every partition has a current upsert snapshot — the rebalance precheck flags an error if snapshots are missing, stale, or the task isn't enabled; no equivalent precheck exists for on-heap tables today.
* **Deleted-keys cleanup**: Configure `deletedKeysTTL` together with `deleteRecordColumn`, and keep snapshots enabled — TTL cleanup without a minion snapshot watermark is disabled by default, so it depends on `UpsertSnapshotCreationTask` running regularly.
* **Only enable `metadataTTL` if you're sure no key will get a legitimate update after the TTL window.** `metadataTTL` must be larger than the longest gap you realistically expect between two updates to the same primary key — not just "how long you want to keep the data." Once a key's metadata is evicted from RocksDB, a later update for that key is no longer recognized as an update at all: it lands as a brand-new record next to the existing, no-longer-invalidated row, instead of replacing it. This is the single most common cause of duplicate/count-mismatch issues in practice.
* **TTL and `validDocIdsType`**: Be careful when `metadataTTL`/`deletedKeysTTL` is set. Minion tasks that read `validDocIds` bitmaps (including the [minion task consensus check](#minion-task-segment-selection-consensus)) support a `validDocIdsType` of `SNAPSHOT` (the default) or `IN_MEMORY` — `IN_MEMORY` is documented upstream as inconsistency-prone during concurrent restarts, which is more likely while TTL cleanup is actively running. Prefer leaving `validDocIdsType` at its `SNAPSHOT` default for correctness and consistent results.
* **SRT with File Ingestion Task**: Running the two together is not recommended, since FIT segments may not follow LLC naming, which can lead to resolution conflicts with SRT's refreshed segments. FIT also does not support PARTIAL upsert tables — only FULL.
* **Rolling upgrades**: Upgrade servers before minions when adopting a new StarTree release with upsert snapshot changes.
* **Safest rollout for mutable upsert config changes**: When changing `enableSnapshot`, `enablePreload`, `metadataTTL`, `deletedKeysTTL`, or the partial-upsert merge strategy — pause ingestion, disable [reuse mode](#reuse-mode), trigger `UpsertSnapshotCreationTask` to build a fresh snapshot under the new config, then restart the servers, and resume ingestion once they're back up. Disabling reuse mode is required — none of these config changes invalidate the on-disk reuse checkpoint automatically, so a restart would otherwise reuse the old state. Re-enable reuse mode once servers are back up.


# Segment Purge Task
Source: https://docs.startree.ai/corecapabilities/manage-data/purge-task

Selectively remove records from StarTree Pinot analytics tables using powerful Segment Purge Task strategies and Dry Run mode.

## Overview

As organizations face increasing data governance requirements, compliance mandates, and the need for precise data management, the Segment Purge Task in StarTree Pinot provides a powerful solution for selectively removing records from your analytics tables. The task enables fine-grained data deletion based on time windows, dimension predicates, or external input files, helping organizations meet regulatory requirements and maintain data quality without resorting to expensive full table reprocessing.

## Key Capabilities

### 1. Flexible Delete Strategies

The Segment Purge Task supports two primary strategies for identifying records to remove:

* **Input File Strategy:** Delete records matching keys from external input files (e.g., CSV files containing user IDs or transaction IDs to delete)
* **Predicate Strategy:** Delete records matching time windows and/or dimension predicates (e.g., delete all records where status = 'inactive' and timestamp \< '2024-01-01')

### 2. Multiple Selection Modes

When using the Predicate strategy, you can filter segments and records in multiple ways:

* **TIME Mode:** Delete records within a specific time window
* **DIMENSION Mode:** Delete records matching dimension field predicates
* **TIME\_DIMENSION Mode:** Delete records matching both time window and dimension predicates

### 3. Advanced Filtering Capabilities

* **Comparison Operators:** Supports `==` (equal), `!=` (not equal), `<` (less than), `<=` (less than or equal), `>` (greater than), `>=` (greater than or equal)
* **Logical Operators:** Supports && (AND) and || (OR) for combining multiple field conditions

### 4. Support for Regular and Upsert Tables

* **Regular Tables:** Physically removes records from segments
* **Upsert Tables:** Marks records as deleted (soft delete) using the configured delete column. The compaction task subsequently cleans these deleted records.

### 5. Incremental Processing

The task maintains processing state in Zookeeper metadata, enabling incremental processing of large datasets and efficient handling of ongoing purge operations.

## Dry-Run Mode

Dry Run mode lets you preview exactly what a Segment Purge Task would do, without creating tasks or modifying data. It runs the same validation, segment selection, and planning logic with the current watermark metadata for the task, but stops before execution. This provides a safe way to view which segments would be deleted, allowing you to fine-tune the scope before consuming compute resources.

### How to run the Purge Task in Dry-Run mode?

Use the below mentioned API to run the Purge Task in dry-run mode.

`POST /tasks/SegmentPurgeTask//dryRun`

Auth: Authorization: Bearer `` Query Parameter: verbose (optional, default: false) - When set to true, includes detailed per-segment information Body: ``

#### Sample Request

```bash theme={null}
curl -sS -X POST 'https://abc.xyz.startree.cloud/tasks/SegmentPurgeTask/myTable_OFFLINE/dryRun?verbose=true' \
  -H "Authorization: Bearer $TOKEN" \
  -H 'content-type: application/json' \
  -d '{
    "purge.start.time.ms": "1609459200000",
    "purge.end.time.ms": "1640995200000",
    "purge.time.column": "timestamp"
  }'
```

#### Sample Response (`verbose=true`)

Once the Segment Purge Task is run in Dry Run mode, what you will get is a response similar to the following JSON:

```json theme={null}
{
  "mode": "dry-run",
  "table": "myTable_OFFLINE",
  "summary": {
    "purgeCount": 2,
    "deleteCount": 1,
    "purgePercent": 10.0,
    "deletePercent": 5.0,
    "purgeTaskCount": 2,
    "totalTaskCount": 2,
    "totalIterationRequired": 1,
    "maxSubtasksPerIteration": 100,
    "warnings": [
      "[WIDE_TIME_WINDOW] Purge window spans a large portion of the table: touched=15.00% (purge=10.00%, delete=5.00%). Consider smaller batches.",
      "[DIMENSION_TOUCHES_MOST_SEGMENTS] DIMENSION selector touches 60.00% of active segments (600/1000). Consider TIME_DIMENSION or narrower predicates to reduce minion work."
    ]
  },
  "segments": [
    {
      "action": "DELETE",
      "segment": "myTable_OFFLINE_2023-09-01_2023-09-02_0",
      "reason": "segment window [2023-09-01T00:00:00Z, 2023-09-02T00:00:00Z) fully inside selection window [2023-09-01T00:00:00Z, 2023-09-10T00:00:00Z)"
    },
    {
      "action": "PURGE",
      "segment": "myTable_OFFLINE_2023-09-02_2023-09-03_0",
      "reason": "time predicate match `timestamp in [2023-09-01T00:00:00Z, 2023-09-10T00:00:00Z)`; segment window [2023-09-02T00:00:00Z, 2023-09-03T00:00:00Z)"
    }
  ],
  "inputFilesProcessing": [
    "s3://bucket/purge-keys/file1.csv",
    "s3://bucket/purge-keys/file2.csv"
  ]
}
```

#### Descriptions of the above response:

* A summary with counts of segments to delete and purge, percentages, task counts, iterations, and any warnings.
* More details on each stage of the Purge Task (when verbose=true):
  * **DELETE** – lists segments that would be removed entirely, along with the reason a segment was selected (e.g., fully inside the selection window, or explicitly listed in purge.segment.list).
  * **PURGE** – lists segments that are partially deleted, which would also be rewritten to keep the rest of the records. (e.g., time windows, dimension filters like status='deleted' AND region='EU').
  * **inputFilesProcessing** – lists input files that would be processed (for the Input File strategy only, when verbose=true).

Please note that Dry Run does not generate task IDs, lineage IDs, progress logs, or build/upload URLs. It also does not provide per-segment row counts for deletion (since that requires reading segment data).

## Supported Warning Scenarios

Dry Run applies the same validations as a real run, returning errors for unsupported scenarios such as invalid predicates and time windows. When other tasks are active, it provides warnings with enough detail to help you decide whether to wait or adjust the scope. If selectors are too broad; for example, a DIMENSION filter that impacts more than half of the segments, Dry Run prompts you to refine them.

Importantly, the DELETE choices made in Dry Run always match those of a real run with the same configuration, ensuring consistency. Potential issues are clearly flagged with descriptive warning codes for easier troubleshooting. These warning codes are mentioned below:

| Category | Code | Description |
| :- | :- | :- |
| Task State | **ACTIVE\_TASKS\_PRESENT** | Another conflicting task is currently running (SegmentRefreshTask, StartreeAlterTableTask, UpsertCompactionTask, UpsertSnapshotCreationTask, or another SegmentPurgeTask). Only checked for upsert tables with delete enabled. |
| Task State | **ACTIVE\_TASKS\_UNKNOWN** | Failed to compute active tasks for the table. Only checked for upsert tables with delete enabled. |
| Task State | **INCOMPLETE\_TASKS** | Found incomplete purge tasks. Task generation would be skipped. |
| Selection Issues | **NO\_MATCH** | Selection matches no active segments. No tasks would be generated. |
| Selection Issues | **NO\_PURGE\_FILE\_FOUND** | No eligible input files to process (Input File strategy). Tasks would not be generated. |
| Selection Issues | **VALIDATION\_ERROR** | Task configuration validation failed. |
| Selection Issues | **DIMENSION\_TOUCHES\_MOST\_SEGMENTS** | DIMENSION selector impacts ≥50% of segments (purgePercent ≥ 50%). |
| Selection Issues | **WIDE\_TIME\_WINDOW** | Time selector covers ≥50% of segments (touchedPercent ≥ 50%). |
| Selection Issues | **MOSTLY\_PARTIAL\_OVERLAP** | ≥80% partial overlaps (purgeRatio ≥ 0.8). This is CPU/IO intensive on minions. |
| Selection Issues | **TIME\_WINDOW\_LARGE** | Time window ≥30 days on large tables (≥10k segments). |
| Selection Issues | **DIMENSION\_SCANS\_ALL** | DIMENSION selector scans all segments (applied to very large tables ≥10k segments). |

## Supported Error Scenarios

Dry Run validates all task configuration before any segment selection occurs. Errors are **fatal** — the dry-run stops immediately when one is encountered and no tasks would be generated. Unlike warnings, errors always require a configuration fix before re-triggering; retrying without changes will produce the same error.

All errors are reported under the single code VALIDATION\_ERROR. The description below identifies the specific misconfiguration.

| **Category** | **Code** | **Description** | **How to fix** |
| :- | :- | :- | :- |
| Strategy | VALIDATION\_ERROR | inputDirURI cannot be combined with predicate/time or query selector configs. | Remove either the inputDirURI key or all predicate/query keys — configure only one strategy per task. |
| Strategy | VALIDATION\_ERROR | Query selector configs (purge.query.predicate) are mutually exclusive with predicate/time configs. | Use either purge.query.predicate alone (QUERY strategy) or time/dimension keys alone (PREDICATE strategy), not both. |
| Strategy | VALIDATION\_ERROR | No strategy keys are present; cannot determine purge mode. | Configure at least one of: purge.query.predicate, purge.startTimeMs/purge.endTimeMs, purge.field.names, or purge.segment.list. |
| Query | VALIDATION\_ERROR | purge.query.predicate is blank or missing. | Provide a valid SQL WHERE predicate, e.g. status = 'deleted'. |
| Query | VALIDATION\_ERROR | purge.query.allowMultiStage must be true. Single-stage support is not yet available. | Do not set "purge.query.allowMultiStage" to false |
| Query | VALIDATION\_ERROR | purge.query.predicate contains invalid SQL syntax. | Correct the predicate. Run the equivalent query directly against the table in query console to identify the syntax issue. |
| Query | VALIDATION\_ERROR | Query references columns without required indexes and purge.query.force is not enabled. | Add the necessary index to the table, or set "purge.query.force": "true" to bypass the check (use with caution on large tables). |
| Time Window | VALIDATION\_ERROR | purge.startTimeMs is missing or ≤ 0. | Provide a valid epoch-millisecond timestamp greater than zero. |
| Time Window | VALIDATION\_ERROR | purge.endTimeMs is missing or ≤ 0. | Provide a valid epoch-millisecond timestamp greater than zero. |
| Time Window | VALIDATION\_ERROR | purge.endTimeMs is earlier than purge.startTimeMs. | Ensure purge.endTimeMs is strictly greater than purge.startTimeMs. |
| Time Window | VALIDATION\_ERROR | No time column configured — the table has no time column in its table config | Configure a time column in the table's validation config. |
| Dimension | VALIDATION\_ERROR | purge.field.names is empty. | Provide at least one field name, e.g. "purge.field.names": "userId". |
| Dimension | VALIDATION\_ERROR | A field listed in purge.field.names does not exist in the table schema. | Verify and correct the field name. |
| Dimension | VALIDATION\_ERROR | purge.field.names and purge.field.values have a different number of comma-separated entries. | Ensure both lists contain the same number of entries. |
| Dimension | VALIDATION\_ERROR | Unsupported value in purge.field.comparisonOperator. | Use one of the supported operators listed in the error message (==, !=, \<, \<=, >, >=). |
| Dimension | VALIDATION\_ERROR | Unsupported value in purge.field.logicalOperator. | Use one of the supported operators listed in the error message (&&, \` |
| Upsert Table | VALIDATION\_ERROR | deleteRecordColumn is not set in upsertConfig. | Add "deleteRecordColumn": "\" to the table's upsertConfig and table schema. |
| Upsert Table | VALIDATION\_ERROR | Neither SegmentRefreshTask nor UpsertCompactionTask is configured for this upsert table. | Enable either SegmentRefreshTask or UpsertCompactionTask in the table's task config. |

### How to run the Purge task in StarTree cloud?

You can run Purge Task either by using /execute API or /schedule API.

#### Using /schedule API

Add the Segment Purge Task config in the table configuration and use the /schedule API to run the Purge task. Sample for segment purge task config is mentioned in each section of the Purge Strategies.

##### Sample Request

```bash theme={null}
curl 'https://pinot.abc.xyz.startree.cloud/tasks/schedule?taskType=SegmentPurgeTask&tableName=anshultestcsv_OFFLINE' \
  -X 'POST' \
  -H "Authorization: Bearer $TOKEN" \
  -H 'content-type: application/json'
```

#### Using /execute API

Use the below mentioned API to run the Purge task in StarTree cloud.

##### Sample Request

```bash theme={null}
curl -X 'POST' \
  'https://pinot.abc.xyz.startree.cloud/tasks/execute' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Database: default' \
  -d '{
   "taskType": "SegmentPurgeTask",
   "tableName": "",
   "taskId": "",
   "taskConfigs": {
        "purge.query.predicate": "WHERE country = 'US' AND org_id = 'org_id_1' or product IN ('WEB', 'MOBILE')"
    }
  }'
```

## Purge Strategies

### 1. Input File Strategy

**Purpose:** Remove records based on keys provided in external input files.

#### How it works

* You provide input files (CSV, JSON, etc.) containing purge keys (e.g., user IDs, transaction IDs)
* The task reads these files and builds a set of keys to match against
* For each record in segments, it computes a purge key from the specified fields
* Records with matching purge keys are removed
* Records with null or empty purge keys are skipped for safety

#### Use cases

* GDPR right-to-be-forgotten requests (bulk user deletion)
* Removing specific transactions or records identified in external systems
* Compliance-driven bulk deletions from external audit files

#### Sample Task Configuration

Here is a detailed reference table for the Segment Purge Task Configuration Parameters in StarTree Pinot. This table explains each parameter, its purpose, accepted values, and examples to help guide correct usage.

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "inputDirURI": "s3://my-bucket/purge-keys/",
        "inputFormat": "CSV",
        "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
        "input.fs.prop.accessKey": "MY_ACCESS_KEY",
        "input.fs.prop.secretKey": "MY_SECRET_KEY",
        "input.fs.prop.region": "us-west-2",
        "max.num.purge.input.files": 10,
        "max.total.purge.input.file.size": 100000000
      }
    }
  }
}
```

| Parameter | Description | Accepted Values / Format | Example |
| :- | :- | :- | :- |
| inputDirURI | URI of directory containing purge key input files | File system URI (e.g., s3://, hdfs\://, file://) | `"inputDirURI": "s3://bucket/purge-keys/"` |
| inputFormat | Format of the input files | One of: CSV, JSON, AVRO, PARQUET, etc. | `"inputFormat": "CSV"` |
| max.num.purge.input.files | Maximum number of input files to process per cycle | Positive integer | `"max.num.purge.input.files": 10` |
| max.total.purge.input.file.size | Maximum total size of input files in bytes per cycle | Positive integer (bytes) | `"max.total.purge.input.file.size": 100000000` |

### 2. Predicate Strategy

**Purpose:** Delete records for both simple and complex predicates. Simple predicates like matching time windows and/or dimension field, and complex predicates where there are multiple different logical operators involved with different fields.

#### Modes

* **QUERY Mode:** Remove all records matching a query predicate like "WHERE country = 'US' AND org\_id = 'org\_id\_1' or product IN ('WEB', 'MOBILE')"
* **TIME Mode:** Remove all records within a specified time range
* **DIMENSION Mode:** Remove records matching dimension field conditions (e.g., status = 'deleted', region = 'EU')
* **TIME\_DIMENSION Mode:** Remove records matching both time and dimension conditions

#### Use cases

* Time-based data retention policies (delete data older than X days)
* Removing records matching specific business conditions
* Cleaning up test or invalid data based on multiple criteria

#### Considerations

When using DIMENSION mode without a time filter, all segments in the table will be scanned, which can be less efficient for large tables. It is recommended to use TIME or TIME\_DIMENSION modes when possible.

Use QUERY mode when you have more than a couple of dimension columns with many different operators and values, which can make the other modes complex.

#### Sample Task Configuration

Here is a detailed reference table for the Segment Purge Task Configuration Parameters in StarTree Pinot. This table explains each parameter, its purpose, accepted values, and examples to help guide correct usage.

### Predicate Strategy: QUERY Mode


  The support for query mode in Segment Purge Task will be available starting StarTree Cloud version 0.14.0


```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.query.predicate": "WHERE country = 'US' AND org_id = 'org_id_1' or product IN ('WEB', 'MOBILE')"
      }
    }
  }
}
```

#### Configuration Parameters

| Parameter | Description | Accepted Values / Format | Example |
| :- | :- | :- | :- |
| purge.query.predicate | Mandatory. A complex Pinot query predicate used to identify rows to purge. Must start with a WHERE clause. | Predicate starting with WHERE | `"purge.query.predicate": "WHERE country = 'US' AND org_id = 'org_id_1' or product IN ('WEB', 'MOBILE')"` |
| purge.query.timeout.ms | Optional. Bounds the query execution time to prevent long-running scans from impacting brokers and other queries. | Time in milliseconds. Default: 60000 | `"purge.query.timeout.ms": "120000"` |
| purge.query.max-rows | Optional. Maximum number of rows fetched per segment. | Numeric value. Default: 100000 | `"purge.query.max-rows": "100000"` |
| purge.query.max-segments | Optional. Maximum number of segments the query result can include. | Numeric value. Default: 30000 | `"purge.query.max-segments": "10000"` |

#### Limitations

* JOINs, subqueries, DISTINCT, GROUP BY, HAVING, and aggregation functions or operators are not supported. The task will fail if used.
* Every column referenced in the WHERE clause must exist in the table schema.
* Every column referenced in the WHERE clause must be indexed. Supported indexes include inverted, range, sorted, or a time column configured in the table.

#### Recommendations

**Writing safe and efficient queries:**

* Use selective predicates on indexed columns, such as equality or range filters.
* Avoid wide OR chains or unindexed columns that force full scans.
* If more than 100000 records per segment are expected, increase purge.query.max-rows. If this limit is exceeded, the task fails with the error: "Stopping execution due to reaching max-rows threshold. Increase the limit or narrow the query and retry."
* When testing new predicates, add an explicit LIMIT to validate behavior before running large scans.
* Prefer deterministic filters, such as immutable status values or bounded time ranges, to avoid inconsistent results across runs.

#### Operational Behavior and Guardrails

* If collected rows reach purge.query.max-rows, the task fails to prevent runaway deletions.
* If the segment list reaches purge.query.max-segments, task generation fails before workers are scheduled.

#### Troubleshooting

* **"requires indexed columns"**: Add an index, any one of (inverted, range, sorted, or star-tree) on all columns used in the WHERE clause.
* **"unsupported construct"**: Remove joins, subqueries, DISTINCT, aggregations, GROUP BY, HAVING, or OFFSET clauses.
* **Max rows or segments reached**: Tighten predicates, or increase purge.query.max-rows and purge.query.max-segments only after confirming the cluster can handle the load.
* **Timeouts**: Simplify the predicate, verify broker and server health, or cautiously increase purge.query.timeout.ms.

### Predicate Strategy: TIME Mode

Sample task configuration:

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.start.time.ms": 1609459200000,
        "purge.end.time.ms": 1640995200000,
        "purge.time.column": "timestamp"
      }
    }
  }
}
```

| Parameter | Description | Accepted Values / Format | Example |
| :- | :- | :- | :- |
| purge.start.time.ms | Start of time window for record deletion | Epoch timestamp in milliseconds | `"purge.start.time.ms": 1609459200000` |
| purge.end.time.ms | End of time window for record deletion | Epoch timestamp in milliseconds | `"purge.end.time.ms": 1640995200000` |
| purge.time.column | Name of the time column to evaluate | Column name as string | `"purge.time.column": "timestamp"` |

### Predicate Strategy: DIMENSION Mode

Sample task configuration:

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.field.names": "status,region",
        "purge.field.values": "deleted,EU",
        "purge.comparison.operator": "==",
        "purge.logical.operator": "&&"
      }
    }
  }
}
```

| Parameter | Description | Accepted Values / Format | Example | | |
| :- | :- | :- | :- | - | - |
| purge.field.names | Comma-separated field names to match | Comma-separated list of field names | `"purge.field.names": "userId,status"` | | |
| purge.field.values | Corresponding values for fields listed in field.names; matches positionally | Comma-separated list (same length as field.names) | `"purge.field.values": "12345,inactive"` | | |
| purge.comparison.operator | Comparison operator to use when evaluating field conditions | One of: `==`, `!=`, `<`, `<=`, `>`, `>=` | `"purge.comparison.operator": "=="` | | |
| purge.logical.operator | Logical operator to combine multiple field conditions | `&&` (AND) or \` | | \` (OR) | `"purge.logical.operator": "&&"` |

### Predicate Strategy: TIME\_DIMENSION Mode

Combine all configuration keys from the TIME mode and the DIMENSION mode described above. Sample task configuration:

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.start.time.ms": 1609459200000,
        "purge.end.time.ms": 1640995200000,
        "purge.time.column": "timestamp",
        "purge.field.names": "status",
        "purge.field.values": "inactive",
        "purge.comparison.operator": "=="
      }
    }
  }
}
```

### Common Configuration

| Parameter | Description | Accepted Values / Format | Example |
| :- | :- | :- | :- |
| table.max.num.tasks | Maximum number of tasks to generate per table | Positive integer | `"table.max.num.tasks": 100` |

## Key Use Case Scenarios

The Segment Purge Task supports a wide range of data deletion scenarios within Pinot tables. These use cases provide granular control over data removal, helping maintain compliance and data quality.

Below are the core supported scenarios:

### 1. Delete Based on Complex Predicates

Use query mode for filtering and deleting records based on complex predicates.

**Example Use Case:**

Implement a policy to delete all records older than 90 days for organisations "org\_id\_1" and country is US for WEB and MOBILE products.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.query.predicate": "WHERE \"timestamp\" <= 1761644708000 AND country = 'US' AND org_id = 'org_id_1' or product IN ('WEB', 'MOBILE')",
        "purge.query.timeout.ms": "120000",
        "purge.query.max-rows": "100000",
        "purge.query.max-segments": "10000"
      }
    }
  }
}
```

### 2. Delete Records Based on Dimension Filters

Remove records matching specific dimension field conditions.

**Example Use Case 1: Single Field Condition**

Delete all records where status equals 'deleted'.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.field.names": "status",
        "purge.field.values": "deleted",
        "purge.comparison.operator": "=="
      }
    }
  }
}
```

**Example Use Case 2: Multiple Field Conditions (AND)**

Delete records where status equals 'inactive' AND region equals 'EU'.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.field.names": "status,region",
        "purge.field.values": "inactive,EU",
        "purge.comparison.operator": "==",
        "purge.logical.operator": "&&"
      }
    }
  }
}
```

**Example Use Case 3: Multiple Field Conditions (OR)**

Delete records where currency equals 'EUR' OR currency equals 'GBP'.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.field.names": "currency,currency",
        "purge.field.values": "EUR,GBP",
        "purge.comparison.operator": "==",
        "purge.logical.operator": "||"
      }
    }
  }
}
```

**Example Use Case 4: Not Equal Condition**

Delete all records where status is not equal to 'active'.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.field.names": "status",
        "purge.field.values": "active",
        "purge.comparison.operator": "!="
      }
    }
  }
}
```

### 3. Delete Records Using Input File Keys

Remove records based on keys provided in external input files.

**Example Use Case:**

A user receives a GDPR right-to-be-forgotten request with a list of user IDs that need to be deleted from the system.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "inputDirURI": "s3://compliance-bucket/gdpr-requests/2024-01-15/",
        "inputFormat": "CSV",
        "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
        "input.fs.prop.accessKey": "MY_ACCESS_KEY",
        "input.fs.prop.secretKey": "MY_SECRET_KEY",
        "input.fs.prop.region": "us-west-2"
      }
    }
  }
}
```

### 4. Delete Entire Segments

Remove complete segments by explicitly listing segment names.

**Example Use Case:**

A user wants to delete specific segments that are known to contain corrupted or invalid data.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.segment.list": "segment_2024_01_15,segment_2024_01_16"
      }
    }
  }
}
```

### 5. GDPR/Compliance Data Removal

Combine time and dimension filters to remove records for compliance purposes.

**Example Use Case:**

Delete all records for EU region users that are older than 3 years to comply with GDPR data retention requirements.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.start.time.ms": "",
        "purge.end.time.ms": "",
        "purge.time.column": "timestamp",
        "purge.field.names": "region",
        "purge.field.values": "EU",
        "purge.comparison.operator": "=="
      }
    }
  }
}
```

### 6. Time-Based Data Retention Policies

Automatically remove old data based on age criteria.

**Example Use Case:**

Implement a policy to delete all transaction records older than 90 days to maintain optimal query performance and reduce storage costs.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.start.time.ms": "",
        "purge.end.time.ms": "",
        "purge.time.column": "transaction_timestamp"
      }
    }
  }
}
```

### 7. Delete Records Based on Time Range

Remove all records within a specific time window.

**Example Use Case:**

A user wants to implement a data retention policy that deletes all records older than 2 years to comply with storage optimization requirements.

**Configuration Sample:**

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentPurgeTask": {
        "purge.start.time.ms": "",
        "purge.end.time.ms": "",
        "purge.time.column": "timestamp"
      }
    }
  }
}
```

## How Does it Work?

The Segment Purge Task operates through a two-phase process that ensures data integrity and efficient processing:

### 1. Task Generation (Controller)

The Pinot Controller generates purge tasks based on your configuration:

* Reads processing state from Zookeeper metadata for incremental processing
* Selects eligible segments based on the configured purge strategy
* Generates individual PinotTaskConfig instances for each segment that needs processing
* Updates Zookeeper metadata to track processing progress

### 2. Task Execution (Minion)

Minion workers execute the purge operations:

* Download the segment that needs to be processed
* Build a RecordProcessor based on the configured strategy (Input File or Predicate)
* Process each record, applying the purge criteria to determine which records should be removed
* Create a new segment containing only the records that should be retained
* Upload the new segment using the same segment name, effectively replacing the original segment at the segment level

## FAQs & Recommendations

### 1. What is the most efficient way to run the Segment Purge Task?

The most efficient approach is to use TIME or TIME\_DIMENSION modes when possible, as they allow the task to efficiently identify which segments need processing based on time partitioning. If no time filter is provided (DIMENSION mode only), all segments in the table will be scanned, which can be less efficient for large tables.

### 2. Is custom dimension partitioning necessary to run the Purge Task efficiently?

No, the data does not need to be physically partitioned by dimensions. The task performs record-level scans within segments. However, having a time column configured and using time-based filters will significantly improve efficiency by limiting which segments need to be processed.

### 3. How does the Purge Task handle upsert tables differently from regular tables?

For regular tables, records are physically removed from segments. For upsert tables, records are marked as deleted (soft delete) using the configured delete column. The delete column must be configured in the table's upsertConfig. The matching records from the CONSUMING segments won't be marked for deletion; in case it is required, do run the Purge Task again to remove all the matching records.

### 4. Can I use the Purge Task to delete records based on multiple conditions?

Yes, you can combine multiple field conditions using a single logical operator (`&&` for AND, `||` for OR) and a single comparison operator (`==`, `!=`, `<`, `<=`, `>`, `>=`) that applies to all specified fields in DIMENSION and TIME\_DIMENSION modes. You can also combine time windows with dimension filters using TIME\_DIMENSION mode. You can use any number of conditions in QUERY mode.

### 5. What happens if I run the Purge Task while other tasks are running on the same table?

The Segment Purge Task cannot run concurrently with SegmentRefreshTask, StartreeAlterTableTask, UpsertCompactionTask, UpsertSnapshotCreationTask, or another Segment Purge Task instance on the same table.

### 6. How does the Input File strategy work with incremental processing?

The Input File strategy maintains state in Zookeeper metadata, tracking which input files have been processed. This allows you to add new files to the input directory over time, and the task will process only the new files in subsequent runs, enabling incremental bulk deletions.

### 7. Can I preview what will be deleted before running the task?

Yes, the Segment Purge Task supports a dry-run mode that provides a preview of what will be affected. Use the `POST /tasks/SegmentPurgeTask/{tableNameWithType}/dryRun` API endpoint with your task configuration. The dry-run returns a summary including the number of segments to purge and delete, percentages of affected segments, estimated task counts, and warnings about potential issues. When verbose mode is enabled, it also lists the specific segments that would be affected, along with the reasons for each action. See the "Dry-Run Mode" section above for detailed API documentation and response structure.

### 8. How do I schedule the Purge Task for regular data retention policies?

The task does not support native scheduling within Pinot. You will need to use external orchestration tools (e.g., cron, Airflow, Kubernetes CronJobs) to trigger the task via the controller's execution API on a regular schedule.

### 9. What is the difference between deleting segments and purging records?

Deleting segments removes entire segments from the table (used when all data in a segment should be removed). Purging records removes only specific records within segments while keeping the segment structure intact (used for fine-grained deletion).


# Managing Adhoc Minion Tasks Programmatically
Source: https://docs.startree.ai/corecapabilities/manage-data/recipes/adhoc-task-trigger

Learn how to programmatically trigger and manage Pinot Minion tasks using Schedule and Execute APIs.

Pinot provides two primary APIs to programmatically trigger ad hoc Minion tasks. Choosing the right API and strategy depends on your task type, data source characteristics, and runtime requirements.

> **Note:** `Schedule API` uses the task config defined in the table config, whereas `Execute API` relies on the API payload. Other than this, there is no difference between these two APIs.

***

## Schedule API

**API Reference:** [`/tasks/schedule`](/api-reference/task/schedule-tasks-and-return-a-map-from-task-type-to-task-name-scheduled-if-task-type-is-missing-schedules-all-tasks-if-table-name-is-missing-schedules-tasks-for-all-tables-in-the-database-if-database-is-missing-in-headers-uses-default)

**Behavior:**
This API schedules tasks using parameters defined under the table’s task configs. Pinot internally determines eligible segments, generates subtasks accordingly, and executes them **immediately**.

**When to Use:**

* When a scheduled task needs to be run immediately instead of waiting for its next schedule.
* If periodic execution is not required or the task should only be externally triggered (e.g., by Airflow), remove the schedule parameter in the task config.
* Recommended for tasks predefined in table config like:
  * Delta Ingestion Task
  * File Ingestion Task
  * SQL Ingestion Task
  * Segment Backfill Task

**Caveats:**

* Ensure `tableMaxNumTasks = -1` so all sub-tasks are created within a single task.
* No built-in retry/recovery mechanism. Tasks must be re-triggered manually if failures occur.

***

## Execute API

**API Reference:** [`/tasks/execute`](/api-reference/task/execute-a-task-on-minion)

**Behavior:**
Executes a task using runtime-supplied configuration (API payload), giving full control over parameters like input paths, filters, and processing windows.

**When to Use:**

* Tasks that require dynamic parameters (input sources, delta ranges, etc.).
* Ideal for external orchestration (e.g., Airflow) of the following:
  * Delta Ingestion Task
  * File Ingestion Task
  * SQL Ingestion Task
  * Segment Backfill Task

**Caveats:**

* No built-in retry/recovery mechanism. Tasks must be re-triggered manually if failures occur.
* Avoid using this API for scheduled maintenance tasks like Segment Import, Segment Refresh, or Alter Table Tasks (use cron-based schedules instead).

***

## File Ingestion Task

The **File Ingestion Task** is used to ingest data from external sources (e.g., S3, GCS, ADLS).
It operates in two main modes - **Append Mode** and **Sync Mode**. Each mode also supports **Consistent Push Mode** and **Consistent Push Full Swap Mode** for atomic ingestion.

### Mode Overview

| Mode | Consistent Push | Consistent Push Swap | Behavior | Segment to File Mapping |
| - | - | - | - | - |
| Sync Mode | false | false | No atomicity is guaranteed within the batch | 1:1 |
| Sync Mode | true | false | Atomic batch ingestion, it is used for incremental ingestion. | 1:1 |
| Sync Mode | false | true | Full table refresh, atomic per run | 1:1 |
| Append Mode | true | false | Atomic batch ingestion, it is used for incremental ingestion. | m:n |
| Append Mode | false | true | Full table refresh, atomic per run | m:n |

***

### Scaling and Stability Recommendations

| Parameter | Description | Recommended Setting |
| - | - | - |
| `tableMaxNumTasks` | Max concurrent subtasks per table | 1000 (adjust per controller capacity) |
| `taskMaxDataSize` | Max data processed per subtask | 1–2 GB (default: 1 GB) |
| `taskMaxNumFiles` | Max files per subtask | Increase for small-file datasets |
| `desiredSegmentSize` | Target Pinot segment size | 500 MB – 1 GB |

***

### Recommendations

#### 1. Large Ad-hoc Ingestion (\~2TB, `consistentPushSwapEnabled=false`)

* **Append Mode:** If `taskMaxDataSize = 1GB` and `tableMaxNumTasks = 1000`, a single batch can ingest 1TB. Run twice to complete 2TB ingestion.
* **Sync Mode:** If each file = 400 MB, with `tableMaxNumTasks = 1000`, total = 400 GB per batch → 5 batches required.
* **Tip:** Re-trigger `/tasks/schedule` or `/tasks/execute` in a loop until no subtasks are created. (No ingestion will take place unless there are new files).

#### 2. Full Table Refresh (`consistentPushSwapEnabled=true`)

* Refresh the whole table with new data.
* Ensures atomic ingestion for all or none.
* Atomicity applies only within a single task execution, not across batches.
* Set `tableMaxNumTasks = -1` to generate all subtasks in one batch.

#### 3. Atomic Table Ingestion (`consistentPushEnabled=true`)

* Similar atomic behavior as above; atomic only within one task execution.
* Set `tableMaxNumTasks = -1` to generate all subtasks in one batch.

#### 4. Rerun on Failure

* Use the [Debug API](/api-reference/task/fetch-information-for-the-given-task-name) to verify task status.
* If subtasks show `FAILED` or `TIMED_OUT`, re-trigger with the same config.
* Once all subtasks are `COMPLETED`, reruns have no effect (idempotent).

***

## Delta Ingestion Task

* Generates 1:1 mapping of Delta files to Pinot segments.
* Safe to rerun; each execution is independent.
* Supports both Schedule and Execute ad hoc APIs.

**Rerun Recommendation:**

* Check via [Debug API](/api-reference/task/fetch-information-for-the-given-task-name).
* If subtasks show `FAILED` or `TIMED_OUT`, re-trigger.
* Once all are `COMPLETED`, rerun has no effect.

***

## Segment Backfill Task

* Executable only on-demand (cannot be scheduled).
* Does **not** support the `schedule` parameter in task config.
* Avoid re-triggering successful runs — repeated runs will re-ingest the same data.

**Rerun Recommendation:**

* Check via [Debug API](/api-reference/task/fetch-information-for-the-given-task-name).
* If subtasks show `FAILED` or `TIMED_OUT`, re-trigger.
* Once all are `COMPLETED`, rerun has no effect.

***

## Segment Import Task

* Recommended frequency: **every 15 minutes**.
* Should be scheduled via the `schedule` parameter in task config.
* It is **not** recommended to trigger using ad hoc schedule or execute APIs.

***

## Segment Refresh Task / Alter Table Task

* Recommended frequency: **every 15 minutes**.
* Should be scheduled via the `schedule` parameter in task config.
* It is **not** recommended to trigger using ad hoc schedule or execute APIs.
* Use the **[Progress Tracker API](/api-reference/altertable/get-progress-and-estimated-completion-time-for-the-task)** to monitor execution status for Alter Table Task.


# Segment Count Limits
Source: https://docs.startree.ai/corecapabilities/manage-data/recipes/segment-count-limits

Per-table and cluster-wide segment count guardrails that gate ingestion and task generation before segment sprawl degrades the cluster.

## Overview

Ingestion tasks create segments continuously. Without a guard, a single misconfigured table or the cluster as a whole can accumulate hundreds of thousands of segments, which slows queries through per-segment fan-out overhead and puts pressure on ZooKeeper and Helix metadata.

StarTree Cloud includes a **segment limit checker**: a controller-side circuit breaker that tracks per-table and cluster-wide segment counts and automatically blocks new ingestion work when configured thresholds are reached. It runs inside the same resource-utilization framework as the disk utilization and primary-key count checkers, and it is **enabled by default**.

This makes it easier to:

* Stop a runaway table before it degrades query latency for the whole cluster
* Cap total segment metadata growth across all tables
* Recover automatically once segment counts drop back under the thresholds — there is nothing to reset

## Key concepts

### How segment counts are collected

On the controller's periodic resource-utilization cadence (`controller.resource.utilization.checker.frequency`), the checker counts every table's segments directly from Helix ideal state and caches per-table counts plus a cluster-wide total. No server calls are involved, so the collection is cheap at any cluster size.

### What a FAIL gates

A `FAIL` verdict gates the same operations as the other utilization checkers:

* Minion ingestion task generation
* Real-time ingestion
* Offline segment uploads
* External Table sync — the watcher consults the checker before each table and skips tables that are over the limit

Queries are unaffected. The breaker only gates the creation of new segments.

***

## Configuration

| Config key | Default | Scope |
| - | - | - |
| `controller.segmentLimitChecker.enable` | `true` | Controller config, read at startup. Disabling requires a controller restart — contact StarTree support. |
| `controller.segmentLimitChecker.maxSegmentsPerTable` | `350,000` | Controller config at startup, or cluster config at runtime. |
| `controller.segmentLimitChecker.maxSegmentsInCluster` | `1,000,000` | Controller config at startup, or cluster config at runtime. |

Non-positive values are rejected: at startup the checker falls back to the default, and at runtime it keeps the current value.

> [!NOTE]
>
> The per-table default of 350,000 matches the documented [capacity guardrail for External Tables](/corecapabilities/external-table/operational-guidance), where each source Parquet file becomes one segment.

***

## Tune limits at runtime

Both limit keys are watched as **cluster configs**, so they can be changed without a controller restart.

**API Reference:** [`POST /cluster/configs`](/api-reference/cluster/update-cluster-configuration) · [`GET /cluster/configs`](/api-reference/cluster/list-cluster-configurations) · [`DELETE /cluster/configs/{configName}`](/api-reference/cluster/delete-cluster-configuration)

Raise the per-table limit:

`POST /cluster/configs`

```json theme={null}
{
  "controller.segmentLimitChecker.maxSegmentsPerTable": "500000"
}
```

Verify what is active with `GET /cluster/configs`, and revert to the default by deleting the key:

`DELETE /cluster/configs/controller.segmentLimitChecker.maxSegmentsPerTable`

The controller logs every applied change and revert:

```text theme={null}
Updating 'controller.segmentLimitChecker.maxSegmentsPerTable' from 350000 to 500000
Cluster config 'controller.segmentLimitChecker.maxSegmentsPerTable' removed, reverting to default: 350000
```

***

## Monitoring and troubleshooting

**Count where a table stands** with `GET /tables/{tableName}/segments` (the segment list length), or watch the controller log for the cluster-wide picture after each periodic count:

```text theme={null}
Computed segment counts for  tables ( failed), cluster total: 
```

**When a limit trips**, the controller log states exactly which threshold was breached:

```text theme={null}
Table my_table_OFFLINE segment count 350000 >= limit 350000, returning FAIL
Cluster segment total 1000000 >= limit 1000000, returning FAIL for table my_table_OFFLINE
```

While the verdict stays `FAIL`, new ingestion work for the affected table (or every table, for a cluster-wide breach) is skipped on each cycle and re-evaluated on the next one.

***

## Recovering

Pick one, depending on whether the segment count is legitimate:

1. **Reduce the segment count.** Tighten retention so old segments age out, or merge small segments (see [Merging Small Segments](/recipes/merge-small-segments)). For External Tables, aim for larger source Parquet files — see the [operational guidance](/corecapabilities/external-table/operational-guidance) on file sizing.
2. **Raise the limit at runtime** with the cluster config API above, if the cluster is sized for it.
3. **Disable the checker** (`controller.segmentLimitChecker.enable=false`) — a controller-config change requiring a restart; contact StarTree support. Prefer options 1 and 2.

Once the next periodic count runs and the totals are back under the thresholds, gated work resumes automatically.


# Schema Evolution
Source: https://docs.startree.ai/corecapabilities/manage-data/schema-evolution



Apache Pinot provides great support for evolving a schema after the table has been created. This is frequently needed as business requirements evolve, and data formats or structures need to change.

At a high level schema evolution works as follows:

1. Create the new schema based on the existing schema.
2. Update the schema using the Controller API
3. Invoker the Table Reload operation to ensure the new schema is picked up by all Pinot segments and hence in the query results.

Details of each of these steps can be found in the Apache Pinot docs: [Schema Evolution](https://docs.pinot.apache.org/users/tutorials/schema-evolution)

**Segment reload** is the primary mechanism that enables schema evolution in Pinot by applying new schema changes to existing segments. When you update a schema with backwards compatible changes, the segments need to be reloaded to incorporate these changes into the queryable data.

### **Schema Update and Reload Integration**

When a schema is updated, Pinot provides two options for applying the changes to existing segments

1. **Immediate reload** (`reload=true`) - Triggers `reloadAllSegments()` for all tables using the schema
2. **Deferred reload - default** (`reload=false`) - Sends schema refresh messages to servers without immediate reload

By default, Pinot does not do a full reload operation but instead only instructs the servers to refresh the schema.

### **Backwards compatible changes**

Following changes are considered backwards compatible:

* **Adding new columns** - You can add new fields to the schema
* **Changing default values** - Modifying default values for existing columns is allowed
* **Time column granularity changes** - Modifications to time field granularity specifications are permitted
* **DateTime column format changes** - Changes to datetime field formats are backwards compatible

### **Incompatible Changes**

Generally speaking, following changes are deemed as incompatible:

* Removing existing columns
* Changing column data types

> [!NOTE]
>
> In StarTree Cloud, you can still make such incompatible changes by running the SegmentRefreshTask (SRT). Note that queries may not succeed until the SRT task has finished.

### **Query Semantics**

It is important to ensure that queries don't fail in the midst of a schema evolution operation. In order to make this easier, Pinot supports the notion of a virtual column which is used in lieu of the actual newly added column. Queries touching this virtual column will pick up the default values instead of failing (which was the old behavior). Here is the general guidance:

| **Scenario** | **Before** | **After** |
| :- | :- | :- |
| Rolling addition of a new column (newCol) | Queries fail with schema mismatch | Queries succeed; newCol is exposed as a virtual column |
| Add a new column without segment reload | Empty results due to server‑side pruning | Queries succeed and return all columns, with newCol filled with default values |
| Queries over existing columns only | No change | No change |
| Queries referencing a partially loaded column | Merge errors or empty responses | Queries succeed with consistent results, including the new virtual column |

Full details of this new functionality can be found [in this PR](https://github.com/apache/pinot/pull/15350).

### **Data Backfill**

Once a new column is added, often times we need to populate it with real values - typically from a batch data source like data lake. StarTree cloud provides a convenient way to accomplish this using the SegmentBackfillTask. For more details, please refer to: [https://docs.startree.ai/corecapabilities/manage-data/segment-backfill-task](https://docs.startree.ai/corecapabilities/manage-data/segment-backfill-task)


# Segment Backfill Task
Source: https://docs.startree.ai/corecapabilities/manage-data/segment-backfill-task



## Overview

As data governance, compliance, and quality expectations continue to evolve, organizations require more precise tools to manage and maintain data in real-time analytics platforms. The Minion-Based Segment Backfill Task in StarTree Cloud addresses this need by enabling selective deletion and backfill of data in a way that is both powerful and easy to control, without resorting to complex, manual workarounds.

## Key Capabilities

### Selective Data Deletion

The Backfill Task supports multiple modes to select data for deletion:

* **Time range** (e.g., a specific day or hour)
* **Dimension filters** (e.g., currency = EUR, region = APAC)
* **Combination** of time range and dimension filters

It is recommended to configure one of these criteria for data deletion. If deletion is not required, use the File Ingestion Task instead.

### Selective Data Backfill

Users can re-ingest targeted subsets of data, such as corrected records or missing slices, without reprocessing entire partitions or tables.

### Atomic and Consistent Updates

Backfill operations are atomic by design. When segment replacement is involved, either all related segments are updated or none are, maintaining table integrity and avoiding partial updates or broken transitions.

## How Does it Work?

The Backfill Task is initiated via an ad-hoc execution API and operates on time-partitioned data with all-or-none consistency. It consists of four phases:

### 1. Segment Selector

Filters segments based on deletion or purge criteria. It identifies full deletions and partial purges, generating tasks with types: `Purge`, `Delete`, or `Ingest`.

### 2. Segment Purger

Removes specific records from segments using defined filters, without deleting entire segments.

### 3. Segment Deletor

Completely deletes segments that the Segment Selector marked for removal.

### 4. Data Ingestor

Uses the FileIngestionTask to bring in new data by creating segments that replace or append to existing ones.

## Limitations

| Constraint | Description |
| - | - |
| No Native Scheduling Support | Triggered on demand via API; users must manage automation. |
| No Support for Upsert-Enabled Tables | Not supported currently. |
| Task Interference | Disable other tasks (FileIngestion, SegmentRefresh, AlterTableTask) during backfill execution. |
| No Support for Sync Mode enabled Tables | The Backfill Task should not be executed on the Pinot tables that were originally ingested using [sync-mode in File Ingestion Task](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/file-ingestion-task#sync-mode) |
| No Support for Delta Ingested Tables | The Backfill Task should not be executed on Pinot tables that were originally ingested from a Delta table. |

## Configuration Parameters

If Backfill also needs to ingest new data, it is important to configure the Segment Backfill Task using the [File Ingestion Task configuration parameters](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/file-ingestion-task). A reference table of key Backfill-related configuration parameters is provided below.

| Parameter | Description | Accepted Values / Format | Example |
| - | - | - | - |
| `backfill.start.time.ms` | Start timestamp for purge/replace | Epoch ms | `1727748000000` |
| `backfill.end.time.ms` | End timestamp | Epoch ms | `1727755200000` |
| `backfill.field.names` | Multi-dimensional filters | Comma-separated | `industry,country` |
| `backfill.field.values` | Values for field names | Comma-separated | `ENERGY,Germany` |
| `backfill.comparison.operator` | Field comparison logic | `==`(default), `!=`, `<`, `<=`, `>`, `>=` | `"!="` |
| `backfill.input.dir` | Data source path for ingestion. Make sure backfill input directory is different to the actual ingestion directory. | File path | `/data/clean/stocks/2024-10-01` |
| `backfill.logical.operator` | Combines multiple filters | `&&`(default), `\|\|` | `\|\|` |
| `backfill.input.format` | Input file format | `CSV`, `JSON`, etc. | `CSV` |

## Preview with Dry Run

Use dry run before scheduling a Backfill Task. Dry run validates task configs, selects matching segments, estimates generated purge and ingestion tasks, and returns warnings without mutating table data.

The dry-run endpoint supports `SegmentBackfillTask` and `SegmentPurgeTask`.

```bash theme={null}
curl -X POST \
  "https:///api/pinot/tasks/SegmentBackfillTask/orders_OFFLINE/dryRun?verbose=true" \
  -H "Authorization: Bearer $STARTREE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "segmentSelector": "TIME_DIMENSION",
    "backfill.start.time.ms": "1735689600000",
    "backfill.end.time.ms": "1735776000000",
    "backfill.field.names": "country",
    "backfill.field.values": "DE",
    "backfill.comparison.operator": "==",
    "backfill.input.dir": "s3://example-bucket/orders/corrections/dt=2025-01-01",
    "backfill.input.format": "parquet",
    "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
    "input.fs.prop.region": "us-west-2"
  }'
```

Use `verbose=false` or omit the query parameter when you only need summary counts.

### Dry-run response

```json theme={null}
{
  "mode": "dry-run",
  "table": "orders_OFFLINE",
  "summary": {
    "deleteCount": 2,
    "purgeCount": 4,
    "ingestFileCount": 12,
    "purgeTaskCount": 4,
    "ingestTaskCount": 3,
    "totalTaskCount": 7,
    "skippedRealtimeNotCompleted": 0,
    "deletePercent": 1.25,
    "purgePercent": 2.5,
    "warnings": [
      "[MOSTLY_PARTIAL_OVERLAP] 4/6 segments partially overlap and require record-level purge. This is CPU/IO intensive on minions."
    ]
  },
  "segments": [
    {
      "action": "DELETE",
      "segment": "orders_OFFLINE_2025-01-01_0",
      "reason": "segment window [2025-01-01T00:00:00Z, 2025-01-01T06:00:00Z) fully inside selection window [2025-01-01T00:00:00Z, 2025-01-02T00:00:00Z)"
    },
    {
      "action": "PURGE",
      "segment": "orders_OFFLINE_2025-01-01_1",
      "reason": "time predicate match `eventTime in [2025-01-01T00:00:00Z, 2025-01-02T00:00:00Z)`, segment window [2025-01-01T06:00:00Z, 2025-01-01T12:00:00Z). Dimension predicate will be matched during task execution;"
    },
    {
      "action": "INGEST",
      "files": [
        "s3://example-bucket/orders/corrections/dt=2025-01-01/part-000.parquet",
        "s3://example-bucket/orders/corrections/dt=2025-01-01/part-001.parquet"
      ],
      "totalBytes": "512.00MB"
    }
  ]
}
```

### How to read the output

| Field | Meaning |
| - | - |
| `deleteCount` | Segments fully covered by the selection and planned for deletion. |
| `purgeCount` | Segments partially covered by the selection and planned for record-level purge. |
| `ingestFileCount` | Input files discovered under `backfill.input.dir`. |
| `purgeTaskCount` | Estimated purge subtasks. |
| `ingestTaskCount` | Estimated ingestion subtasks based on input file grouping. |
| `totalTaskCount` | Estimated purge plus ingestion subtasks. |
| `deletePercent` / `purgePercent` | Percentage of active segments affected by each action. |
| `warnings` | Validation, scope, active-task, missing-input, and high-impact warnings to resolve before scheduling. |
| `segments` | Verbose per-segment and per-ingestion-group details. Returned when `verbose=true`. |

Treat warnings as blockers unless you intentionally want that outcome. For example, `[DELETE_ONLY]` means the plan deletes segments without replacement data, and `[DIMENSION_SCANS_ALL]` means a dimension-only selector must scan all active segments.

## Sample Task Configuration

```json theme={null}
{
  "task": {
    "taskTypeConfigsMap": {
      "SegmentBackfillTask": {
        "backfill.start.time.ms": 1735689601000,
        "backfill.end.time.ms": 1735775999000,
        "backfill.input.dir": "/path/to/data",
        "backfill.input.format": "json",
        "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
        "input.fs.prop.accessKey": "MY_ACCESS_KEY",
        "input.fs.prop.secretKey": "MY_SECRET_KEY",
        "input.fs.prop.region": "us-west-2"
      }
    }
  }
}
```

## Key Use Case Scenarios

### 1. Delete a Subset of Table

Remove data/segments for a specific time range.

```json theme={null}
{
  "backfill.start.time.ms": "",
  "backfill.end.time.ms": ""
}
```

### 2. Replace a Subset of the Table

Delete existing data and replace with clean data for a specific time range.

```json theme={null}
{
  "backfill.start.time.ms": "",
  "backfill.end.time.ms": "",
  "backfill.input.dir": "/path/to/clean/data",
  "backfill.input.format": "CSV"
}
```

### 3. Mixed Operation: Delete Some, Replace Others, and Add New Segments

A table has inconsistent data - some segments are invalid and should be removed, some are outdated and need replacing, and new records need to be appended.
Perform a combination of:

* Deleting some segments outright,
* Replacing a set of segments,
* Ingesting entirely new segments.

```json theme={null}
{
  "backfill.segment.list": "segment_old1, segment_old2",
  "backfill.start.time.ms": "",
  "backfill.end.time.ms": "",
  "backfill.input.dir": "/path/to/new/data",
  "backfill.input.format": "CSV"
}
```

### 4. Replace Data Matching Multiple Values

**Example: currency is EUR or USD**

```json theme={null}
{
  "backfill.field.names": "currency, currency",
  "backfill.field.values": "EUR,USD",
  "backfill.logical.operator": "||",
  "backfill.input.dir": "/path/to/data",
  "backfill.input.format": "CSV"
}
```

**Example: industry is ENERGY or country is Germany**

```json theme={null}
{
  "backfill.field.names": "industry,country",
  "backfill.field.values": "ENERGY,Germany",
  "backfill.logical.operator": "||",
  "backfill.input.dir": "/path/to/data",
  "backfill.input.format": "CSV"
}
```

**Example: currency is not EUR**

```json theme={null}
{
  "backfill.field.names": "currency",
  "backfill.field.values": "EUR",
  "backfill.comparison.operator": "!=",
  "backfill.input.dir": "/path/to/data",
  "backfill.input.format": "CSV"
}
```

### 5. Replace with Time and Multi-dimension Filter

```json theme={null}
{
  "backfill.start.time.ms": "",
  "backfill.end.time.ms": "",
  "backfill.field.names": "currency, currency",
  "backfill.field.values": "EUR,USD",
  "backfill.logical.operator": "||",
  "backfill.input.dir": "/path/to/data",
  "backfill.input.format": "CSV"
}
```

### 6. Add New Data or Segments Only

**Use the File Ingestion Task when no deletion is needed.**

## FAQs & Recommendations

> **Is custom dimension partitioning required?**
>
> No, segment purger performs record-level scans.

> **Can I use Backfill Task on Upsert tables?**
>
> No, it’s not supported at the moment.

> **Can Backfill Task be used for deletion only?**
>
> Yes, Backfill Task can be used for deletion only. A time based or dimension based filter can also be applied for selective deletion.

> **Should I use Backfill to only ingest new data?**
>
> No, use the File Ingestion Task for that.

> **Can it run with other tasks?**
>
> No, disable concurrent tasks.

> **What if new data is small and segments become uneven?**
>
> Cluster Health Check dashboard can give insights into Segment sizes and skewness if there is any. Use Alter Table Task after Backfill to optimize the Segment size.


# Segment Import Task
Source: https://docs.startree.ai/corecapabilities/manage-data/segment-import-task



The SegmentImportTask is an extended version of Apache Pinot's [RealtimeToOfflineSegmentsTask](https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows).
Both are built with the Pinot Minion framework to convert segments from real-time tables to offline tables.

The SegmentImportTask does this conversion with multiple tasks in parallel, which means it's less likely to lag behind. This task is a drop-in replacement for the `RealtimeToOfflineSegmentsTask`.

## Prerequisites

Like `RealtimeToOfflineSegmentsTask`, both real-time and offline tables should be created.
But, where the `RealtimeToOfflineSegmentsTask` is configured on the real-time table, the SegmentImportTask is configured on the OFFLINE table, following the convention that minion tasks to ingest data are configured on the destination tables.

## Pinot Table Configuration

The task configs used by [RealtimeToOfflineSegmentsTask](https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows#config) are supported by SegmentImportTask as well.
There are also some new parameters that control task parallelism as listed below.

| Property Name | Required | Description |
| - | - | - |
| tableMaxNumTasks | No | The max number of parallel tasks a table can run at any time. It's 10,000 by default. |
| maxNumRecordsPerTask | No | The max number of records one task can process, to spread workload among parallel tasks. It's 50M by default. |
| initialWatermarkMs | No | Where to start the task and by default starting from the smallest start time of real-time segments. It's -1 by default. |
| desiredSegmentSize | No | The segment size desired (Default is 500M. K for kilobyte, M megabyte, G for gigabyte). |
| schedule | No | CRON per [Quartz cron syntax](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) for when the job will be routinely triggered. If not set, the task is not cron scheduled but can still be triggered via endpoint /tasks/schedule. |

Below is a sample task configuration:

```json theme={null}
    "task": {
      "taskTypeConfigsMap": {
        "SegmentImportTask": {
          "bucketTimePeriod": "1d",
          "bufferTimePeriod": "1d",
          "schedule": "0 */10 * * * ?",
          "maxNumRecordsPerTask": "50000000",
          "maxNumRecordsPerSegment": "5000000",
          "tableMaxNumTasks": "4",
          "initialWatermarkMs": 1654680177745
        }
      }
    },
```

> [!NOTE]
>
> Usually, run the task every 10min+ is a good starting point.
> You can always adjust the schedule and task parallelism to make sure the task can catch up with the load from the real-time table.
>
> Don't run the task only few times a day when the bucketTimePeriod is set to 1d, as the task might just keep lagging until data retention kicks in and deletes the oldest segments in real-time table, causing data loss.

## Migrating from `RealtimeToOfflineSegmentsTask`

The key step is to find the current watermark of the `RealtimeToOfflineSegmentsTask` and set `initialWatermarkMs` to that value before starting the SegmentImportTask.

1. Remove the `RealtimeToOfflineSegmentsTask` task config from the REALTIME table and wait for existing tasks to finish.
2. Use tasks API `/tasks/RealtimeToOfflineSegmentsTask/{tableNameWithType}/metadata` to get current watermark.
3. Add `SegmentImportTask` task configs in offline table, and set `initialWatermarkMs` to what we get in the last step.

During this process, you can check Grafana dashboards to monitor that the `RealtimeToOfflineSegmentsTask` has stopped and `SegmentImportTask` started.


# Segment Operation Throttling
Source: https://docs.startree.ai/corecapabilities/manage-data/segment-operation-throttling

Control how much work a server spends on segment downloads, index builds, and upsert/dedup operations so that heavy operations on one table don't degrade query latency or starve other tables.

When a server loads segments — during a restart, a reset, a rebalance, a reload/refresh, or
while committing realtime segments — it competes for a few shared, resource-intensive
operations: downloading segments from the deep store, building indexes, and (for upsert/dedup
tables) updating RocksDB metadata. Left unbounded, a single heavy operation (for example,
resetting a table with tens of thousands of segments) can saturate these resources and slow
down query serving or block other tables.

**Segment operation throttling** puts a configurable concurrency limit on each of these
operations. Limits are enforced per server and can be tuned live — you do not need to restart
the server to change them.

> [!NOTE]
>
> Throttling limits **concurrency** (how many operations run at once), not throughput. A limit of
> `4` means at most 4 segments are processed by that operation simultaneously on a given server;
> the rest queue and run as slots free up.

## The throttlers

Each throttler guards one type of work. A segment operation may pass through several of them in
sequence (for example, an OFFLINE→ONLINE transition can download, then preprocess indexes).

| Throttler | What it protects | When it applies |
| - | - | - |
| **Download** | Concurrent segment downloads from the deep store | Any operation that needs to fetch a segment locally (restart, reset, rebalance, backfill) |
| **Index preprocess (all)** | Concurrent index building / preprocessing | Building or rebuilding indexes during segment load and reload |
| **StarTree preprocess** | StarTree index rebuilds (CPU- and memory-heavy) | Acquired *while holding* the index-preprocess slot, for segments with a StarTree index |
| **Multi-column text index preprocess** | Multi-column text index builds (heavy) | Acquired *while holding* the index-preprocess slot, for segments with a multi-column text index |
| **RocksDB** | Upsert / dedup metadata operations backed by RocksDB | Loading or preloading segments of an upsert or dedup table |

> [!NOTE]
>
> StarTree and multi-column text index preprocessing are acquired *on top of* the index-preprocess
> throttler, so they are intentionally given small limits (default `1`) — they are the most
> expensive per segment.

## Context-aware isolation (consuming vs. general)

By default, throttlers are split into **two isolated sets** so that operations on consuming
(realtime) segments are never blocked behind bulk background work:

* **`consuming`** — operations that involve consuming segments, such as committing a realtime
  segment. These keep ingestion healthy.
* **`general`** — everything else: server restart, table reset, rebalance, reload, refresh, and
  upsert/dedup preload.

Because the two sets hold separate permits, a large `general` operation (a 10k-segment reset)
cannot exhaust the permits that consuming-segment commits rely on.

This isolation is controlled at the server level:

| Cluster config | Default | Description |
| - | - | - |
| `pinot.server.throttler.context.aware.enabled` | `true` | Enable the `consuming` / `general` split. When `false`, all operations share one set. |
| `pinot.server.throttler.parallelism.ratio.consuming` | `0.75` | Fraction of the configured parallelism allotted to the `consuming` set. |
| `pinot.server.throttler.parallelism.ratio.general` | `0.75` | Fraction of the configured parallelism allotted to the `general` set. |

## Server-level limits

Server-level limits are set as **cluster configs** and apply to every server in the cluster.
Updating a cluster config takes effect immediately, without a restart.

| Throttler | Cluster config | Default\* |
| - | - | - |
| Download | `pinot.server.max.segment.download.parallelism` | `max(1, cores/4)` |
| Index preprocess (all) | `pinot.server.max.segment.preprocess.parallelism` | `max(1, cores/4)` |
| StarTree preprocess | `pinot.server.max.segment.startree.preprocess.parallelism` | `1` |
| Multi-col text index preprocess | `pinot.server.max.segment.multicol.text.index.preprocess.parallelism` | `1` |
| RocksDB (upsert/dedup) | `pinot.server.max.segment.rocksdb.parallelism` | `max(1, cores/4)` |

\*Defaults scale with the number of CPU cores on the server.

Each limit also has a `.before.serving.queries` variant (for example,
`pinot.server.max.segment.download.parallelism.before.serving.queries`). This variant is used
**only during server startup**, before the server begins serving queries, and is set higher
(up to all cores) so a starting server loads its segments quickly. Once the server starts
serving queries, the steady-state limit above takes over to protect query latency.

> [!WARNING]
>
> Setting a limit to **`0` acts as a kill switch**: it halts that operation entirely and parks any
> caller until permits are restored (by raising the limit again). This is useful to immediately
> stop a runaway operation, but remember to restore it afterward — leaving it at `0` will stall
> segment loading.

## Per-table tuning

To stop one table from monopolizing a server-level throttler, you can cap how many permits a
**single table** may hold within a throttler. This is set in the **table config**, under
`customConfigs`, using keys of the form:

```
segmentOperation.concurrencyLevel..
```

* `` — `general` or `consuming`
* `` — `download`, `allIndexPreprocess`, `starTreePreprocess`,
  `multiColTextIndexPreprocess`, or `rocksDB`

The value is the maximum number of permits this table may hold in that throttler at once. A
table with a low cap yields the remaining permits to other tables, so heavy work on it has a
smaller blast radius.

```json theme={null}
{
  "tableName": "myTable_OFFLINE",
  "tableType": "OFFLINE",
  "...": "...",
  "metadata": {
      "customConfigs": {
        "segmentOperation.concurrencyLevel.general.allIndexPreprocess": "5",
        "segmentOperation.concurrencyLevel.consuming.download": "1",
      }
  }
}
```

In the example above, while resetting or rebalancing `myTable`, the table can use at most 2 of
the server's index-preprocess permits and 2 download permits at a time, leaving headroom for
other tables. Table config changes are picked up live — no restart required.

> [!TIP]
>
> Use per-table caps for the few tables that are large or operationally heavy. Leave everything
> else unset so those tables draw from the full server-level limit. An unset (or invalid) key
> means "no per-table cap" for that throttler.

## Observability

Each throttler exposes server metrics so you can see saturation and tune limits with evidence:

* **Threshold** — the currently configured permit count for the throttler.
* **In-use count** — how many permits are held right now. Sustained `in-use ≈ threshold` means the
  throttler is saturated and is the bottleneck.
* **Queue length** — how many pending operations are waiting for this throttler from each table.
* **Wait time / hold time / acquisition** — how long operations wait to acquire a permit, how long
  they hold the throttler, and how often they acquire per table. High wait time confirms contention.

These metrics are available per throttler type (download, index preprocess, StarTree, multi-col
text, RocksDB) and are surfaced on the segment-operations panels of the StarTree Grafana
dashboards.

## How to tune

**1. Find the saturated throttler**

On the Grafana segment-operations panels, look for a throttler whose in-use count sits at its
threshold with rising wait times during the operation you are running (restart, reset,
rebalance, reload, ingestion spike).

**2. Decide where to apply the limit**

If one table is responsible (for example, a large reset), add a **per-table cap** so other
tables keep making progress. If the whole server is overloaded, lower the **server-level**
limit for that throttler instead.

**3. Adjust and observe**

Change the cluster config or table config and watch the metrics — limits apply live. Lower a
limit to reduce impact on query latency; raise it to drain a backlog faster.

**4. For emergencies**

Set the offending throttler's server-level limit to `0` to immediately halt that operation,
then restore it to a sensible value once the situation is stable.


# Segment Refresh Task
Source: https://docs.startree.ai/corecapabilities/manage-data/segment-refresh-task



The `SegmentRefreshTask` reads the latest table config and refreshes the segments if they are not consistent with the table config. When an inconsistency between segments and table config is detected, it will download the segments from the deep store, process and regenerate the segments, and then push them back to replace the old segments atomically.

## Supported Operations

The following operations can be applied to the segments to match the table config:

* Time partitioning: Re-partition the segments to be time partitioned (all the records within a segment are in the same time bucket).
* Value partitioning: Re-partition the segments according to the partitioning config.
* Merge/Split: Merge small segments or split large segments (with rollup support) to ensure segments are properly sized.
* Other table config changes that cannot be applied on the server side with segment reload:
  * Change time column
  * Change sorted column
  * Change column data type
  * Change column encoding

> [!WARNING]
>
> complexTypeConfig transforms are not supported with the Segment Refresh Task. The `complexTypeConfig` configuration is used as a part of the `ingestionConfig` to flatten and process complex-type data. **Do not** use the Segment Refresh Task when the `complexTypeConfig` configuration is enabled. This may result in records being duplicated and unintended changes to tables.

## How to configure the SegmentRefreshTask

Configure the SegmentRefreshTask under the `taskConfig` section in the table configuration.

| Property Name | Required | Description |
| - | - | - |
| bucketTimePeriod | Yes | Time bucket for segments (e.g. 1d). |
| maxNumRecordsPerSegment | No (default 5M) | Max (desired) number of records in each segment. The task will try to resize all segments to this size after applying the partitioning constraints. |
| skipSegmentIndexCheck | No (default false) | If set to true, the index check (see the next section) will be skipped. This check requires pulling all segments' metadata from the servers, which can be costly for large table. |
| tableMaxNumTasks | No (default 10) | Max number of parallel tasks a table can run at each schedule. This value can be tuned based on the Minion instances in the cluster. It has to be positive. |
| maxNumRecordsPerTask | No (default 50M) | Max number of records processed in a single task. Each task is executed by a single Minion instance, so the records processed should be limited to prevent the Minion from running out of resources. It has to be a positive value. |
| maxDataSizePerTask | No (default 5 GB) | Max size of data provided to a single task. |
| desiredSegmentSize | No (default 500 MB) | User specified size for a segment. |
| batchSegmentUpload | No | Boolean field for which the default value is `false`. When the value is set to `true` segments are uploaded in batch mode which is faster than uploading segments one after the other. |
| mergeType | No | Same definition as in the [MergeRollupTask](https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task#config). |
| roundBucketTimePeriod | No | Same definition as in the [MergeRollupTask](https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task#config). |
| \*.aggregationType | No | Same definition as in the [MergeRollupTask](https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task#config). |

## Segment Index Check

When segment index check is enabled, the task generator will pull the segment metadata for each segment from the servers, and compare it with the table config. If the segment metadata is not consistent with the table config, the segment will be refreshed.

The following properties are compared between the segment metadata and the table config:

* Whether the time column is the same.
* Whether the partitioning info matches:
  * Same partition column
  * Same partition function
  * Same partition count
  * Segment belong to a single partition
* Sorted column in the table config is sorted in the segment.
* Checks for all columns:
  * Column is added (while this can be handled through a server reload, the task has been extended, so that it can be performed in one go and through Minions.)
  * Column is deleted.
  * Column field type change.
  * Column data type change.
  * Column SV/MV change.
  * Column encoding change.

## Example Configuration

```json theme={null}
    "task": {
      "taskTypeConfigsMap": {
        "SegmentRefreshTask": {
          "bucketTimePeriod": "1d",
          "maxNumRecordsPerSegment": "2000000",
          "maxNumRecordsPerTask": "10000000",
          "schedule": "0 */30 * * * ?"
        }
      }
    }
```

## Real-time Tables

For real-time tables, the SegmentRefreshTask considers segments which are in COMPLETED state. The consuming segments are left untouched.

### Upsert Table

This task can work with real-time table enabled Upserts as well. The task can compact segments, i.e. removing invalid docs; and then merge segments into bigger one. There are some limitations when this task is enabled for upsert tables, mainly due to the complexity from managing the upsert metadata consistently for upsert to work.

1. Records can't be rolled up because the primary keys must be kept intact before and after refreshing.
2. The task performs M:1 merging, i.e. combining multiple input segments into one new segment, to simplify failure handling for tracking upsert metadata consistently. With M:1 mapping, data repartitioning and bucketizing are not supported.

Most of the task configs described above are ignored, except `"maxNumRecordsPerSegment": "2000000"` . This setting configures how large the output segment should be, and the input segments are automatically decided for each task, based on how many valid records they have.

For the in-depth documentation on running Segment Refresh Task on upsert tables, see [Compaction after Upserts](https://docs.startree.ai/corecapabilities/manage-data/upsert-compaction-srt).


# Tiered storage for Apache Pinot in StarTree Cloud
Source: https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/architecture

Learn how tiered storage is built for Apache Pinot, effectively decoupling the storage from the compute in your Pinot cluster.

With Tiered storage, your Pinot cluster is now, **not limited to use just the disk/SSD storage**. We are no longer strictly tightly coupled. You can have multiple tiers of storage, with **support for using a cloud object storage like S3 as a storage tier**. You can **configure exactly which portion of your data you want to keep locally, and which is offloaded to the cloud tier storage**.

![Introducing Tiered Storage for Apache Pinot in StarTree Cloud](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/manage-data/images/introducing-tiered-storage.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=4e7c73d340a25735234f786d2f9ab5b5)

One popular way to split data across local vs cloud is by **data age**, so you could configure in your table config, that you want **data less than 30 days old to be on disk, rest to be on S3**.

Users can then **query the entire table, across local and remote data**, like any other Pinot dataset.

With this decoupling, you can now store as much data as you want in Pinot, without worrying about the cost. And this is super flexible and configurable: This threshold is dynamic and can be changed at any point in time and Pinot will automatically reflect the changes.

You can still operate Pinot in fully -tightly coupled mode if you want, or completely in decoupled mode, or a hybrid approach (like in the picture above), where some nodes are dedicated for local data, and some for remote data.

Here's a blog post which goes in more details about the motivation, and some initial benchmarks: [Speed of Apache Pinot at the cost of Cloud Object Storage](https://www.startree.ai/blog/introducing-tiered-storage)

## Frequently asked questions

Questions and clarifications about this feature:

### What is different between this and the Tiered Storage in open-source Apache Pinot?

In Apache Pinot open-source, we have limited support for tiered storage (read more in the doc: [Moving data from one tenant to another based on segment age](https://docs.pinot.apache.org/operators/operating-pinot/moving-segments-across-tenants)). It simply lets you use multiple tenants for the same table, where each tenant can be configured to have servers of different specs (e.g. use SSD nodes for tenant 1 and lower performant cheaper HDDs as tenant 2). But the tiering stops there. We can separate the compute nodes using tenants, but are still using local storage, making us tightly coupled in terms of storage and compute.

![Tiered storage in oss vs in StarTree Cloud](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/manage-data/images/oss-vs-startree.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=7faea9ebbacf7e7eeec508d45256c85c)

### Doesn't Pinot already have deep store? I have set up Pinot with S3/GCS as my segment store, isn't that the same?

Many of you might be familiar with the architecture of Pinot or similar systems, and might know that these systems, along with their tightly coupled storage component, additionally also have a cloud object store like S3, GCS, ABS in their architecture. However, this segment store or deep store, **is only used for permanent backups, disaster recovery, and it is not involved in the query path**. With the tiered storage feature, for Apache Pinot we are now able to **use the cheap cloud storage directly in the query, and replace the disk/SSDs**.

![Not to be confused with deep-store](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/manage-data/images/not-deep-store.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=60e6bbfdf4bed58a9a8e9b5a97cd564e)

### Why can't we use the copy already in deep-store?

The segment in deep store is compressed, and typically will not have all indexes, especially if you've added indexes or derived columns to the segment later on. We use a separate location and copy for the tiered storage query execution, so we can keep it uncompressed and exactly in sync with what would have been on the server. Think of this S3 bucket as an exact replacement for the storage that would've been on the servers locally, with one major difference that we'll keep only 1 copy in S3 vs replication number of copies locally.

### Is this implemented like lazy loading?

One common way that other systems in data infra solve tiering, is by using lazy loading. In lazy loading, think of your shards/segments being stored in S3, and when your query needs a segment which is on S3, it will be downloaded entirely. This means that the first access will be very slow, and your second access would find the segment locally. However, in typical OLAP workloads,

1. more often than not, your next query will need completely different set of segments (think arbitrary slide n dice, point lookups on various dimensions)
2. or you will need more data for a single query than your local storage can keep
3. Predictable p99 query latency is crucial

So in an OLAP system with lazy loading, you'll end up with continuous churn and continuous segment downloads.
StarTree's approach is different, and strictly stays away from lazy loading. And while caches can be enabled as optimizations, the implementation doesn't rely on a cache or local storage to function.

### Are all indexes supported in Tiered Storage?

Yes, all indexes are supported. Text index support is very new (Beta) added in the StarTree 0.11.1 release.

## Architecture

### Lightboard session

[YouTube video player](https://www.youtube.com/embed/YUJXwzN2dok)

### Tech talk

[YouTube video player](https://www.youtube.com/embed/nEAvHAvh0tY)


# Motivation
Source: https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/motivation

Learn about the motivation behind the tiered storage project

### Tightly-coupled real-time analytics databases

Apache Pinot, like other typical OLAP real-time databses, was a strictly tightly coupled system for many years. Which meant, that we could only use disk/SSDs as storage, from local instance storage or remote attached storage (EBS volumes). This works great for such systems, as these types of storage have very fast access speeds (microseconds for local, milliseconds for remote attached storage), and help them achieve fast milliseconds/sub-seconds query latencies.

### Exploding cost with data volume increase

As Pinot became popular and adoption increased, users wanted to put more and more data into Pinot, and use it for use-cases beyond real-time user-facing analytics, such as internal dashboarding, metrics reporting, and so on. But using disks/SSDs for storing this vast amount of data, becomes very expensive.
There's 2 main reasons for this:

1. Since this **storage is tightly coupled with the compute**, you'll often have to add a lot of compute just to keep up with increasing storage, whether you need to utilize the compute or not.
2. Secondly, **compared to storage options like cloud object stores, disk/SSD cost almost 5x**

At the same time, the willingness to pay reduces for historical data, as usecases tend to be for internal audiences, with slightly relaxed latency and throughput SLAs.

### Decoupled analytics systems

One option for users, is to split the data up, keeping real-time data in the tightly-coupled OLAP system, and use a decoupled system like Presto for the historical data. The advantage is that these systems use cloud object storage giving you very low cost, but the disadvantage is that your queries are now in the several seconds range, plus you have 2 systems to operate, federate and manage.

### Need for best of both worlds

It was clear that we needed a true best of both worlds.

## Tiered storage with S3 for Apache Pinot

With this motivation in mind, we at StarTree set out to build Tiered Storage for Apache Pinot.


# Set up Tiered Storage with Cloud Object Store
Source: https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/setup

Learn how to set up tiered storage with cloud object store, effectively decoupling storage from compute in your Pinot cluster.

This page shows you how to enable tiered storage for a table.

## Set the segment age boundary

Set a segment age boundary to split data across your local and remote tiers.

For example, if you have 30 days of data, and you set the segment age boundary at 7 days, segments less than 7 days old will be on the local tier, and segments older than 7 days are moved to the remote tier.

## Set up a tenant

Do one of the following:

* (Recommended) Set up a server as a dedicated tenant
* (Default) Use the same server as your table as your tenant. We do not recommend this option for production environments.

### Set up a dedicated tenant server *(Recommended)*

Create a new tenant to serve as a compute-only node. This tenant will be dedicated to serving the segments which are on the remote tier backend. These do not require as much local storage as your table's default local tenant since the data is stored remotely.

To add one or more dedicated tenants, do the following:

1. Add servers. You'll need at least as many servers as in your table replication setting in the Pinot table configuration.
2. Tag these as `RemoteTenant_OFFLINE` (or whatever you prefer) using the `/instances/{instanceName}/updateTags` API in the Swagger UI on [the host](http://localhost:9000/help#/Schema/) where Pinot is running. This link is only accessible when Pinot is running.

### Use same tenant as your table *(Default)*

We don't recommend this option for production environments.

This option is selected by default. StarTree will use the same tenant that you set in `tableConfig->tenants->server` in your Pinot table configuration. This option is a no-op, requiring you to do nothing.

> [!NOTE]
>
> Processing segments from the cloud object store causes higher than usual memory and CPU utilization on nodes, and can affect local data query performance. If your local data queries are serving latency sensitive, user-facing traffic, we recommend using a separate server as your tenant.

## Add cluster configs and restart servers

To enable tiered storage, do one of the following:

* Add the configs as part of the server instance configs or
* From the Swagger UI on [the host](http://localhost:9000/help#/Schema/) where Pinot is running, use the `POST /cluster/configs` API to add the following cluster-level tier storage configurations.

After adding your configs, restart the servers for changes to take effect.

Where:

```json theme={null}
{
  "pinot.server.instance.tier.enabled": "true",
  "pinot.server.instance.segment.cache.directory": "/home/pinot/data/tieredStorage/segmentCache/",
  "pinot.server.instance.buffer.reference.manager.ondemand.total.size": "24G",
  "pinot.server.instance.buffer.reference.manager.ondemand.reserved.size": "4G"
}
```

Review the related configuration options here:

| Property | Description | Default value |
| - | - | - |
| tier.enabled | Configuration to enable tier storage for the cluster | false |
| segment.cache.directory | Where to cache segment headers to speed up server restarts | empty (disabled) |
| ondemand.total.size | Max memory for remote data access. Ensure this is smaller than direct memory available on server else you’ll get OutOfDirectMemory exceptions | 24G |
| ondemand.reserved.size | This is a portion of the above total size. Ensure this is larger than size of largest uncompressed segment | 4G |

> [!NOTE]
>
> The `pinot.server.instance.segment.cache.directory` enables caching to restart the server more quickly, but does not affect the query execution speed. The two `ondemand` configs are shown here for their importance in query performance. More information about these configs others can be found in the sections below. You may want to set some of them before restarting servers.

Restarting servers will take much longer than before (depending on the number of segments on the remote tier backend) because StarTree needs to read the segment metadata from the remote tier backend when loading the segment on the server.  To enable configurations that improve performance, see [recommended configurations for ease of operations](#configs-for-ease-of-operations).

## Update the Pinot table configuration

1. If not already set, set  in the Pinot table configuration.
2. Add `tierConfigs` to your Pinot table configuration (example below). The `segmentAge` will be calculated using the primary `timeColumnName` as set in Pinot table configuration.

After adding this configuration, a controller-side periodic task, `SegmentRelocator`, will check and migrate segments to the proper tier, such as when the segments cross the `segmentAge`. See the [SegmentRelocator configuration documentation](https://docs.pinot.apache.org/basics/components/controller#segmentrelocator) for more information.

```json theme={null}
"tierConfigs": [
      {
        "name": "remoteTier",
        "segmentSelectorType": "time",
        "segmentAge": "7d",
        "storageType": "pinot_server",
        "serverTag": "RemoteTenant_OFFLINE",
        "tierBackend": "s3",
        "tierBackendProperties": {
             "pathPrefix": "",
             "region": "",
             "bucket": "",
             ...
        }
      }
    ],
```

Review the related configuration options here:

| Property | Description |
| - | - |
| name | Set this to anything you want, this is more for referring to this tier uniquely. |
| segmentSelectorType | Can be `time` for moving segments by age, or `fixed` to move fixed segments by name |
| segmentAge | Applicable if `segmentSelectorType` is `time`. Set this to the segment age boundary you picked in Step 1. This takes any period string, examples - `0s`, `12h`, `30m`, `60d` |
| segmentList | Applicable if `segmentSelectorType` is `fixed`. Set this to a comma separated list of segments you wish to move to remote tier. This is a good way to do a quick test. |
| storageType | Has to be `pinot_server` |
| serverTag | If using setup 2.1, set this to table's server tag. If using setup 2.2, set this to the tag of the remote tier created in Step 2.2 |
| tierBackend | For AWS S3 tier backend, use `s3`. For Google Cloud Storage (GCS) tier backend using interoperable API, use `gcs_interoperable`. |
| tierBackendProperties (for S3) | Set these as per your S3 details. Supported properties are `region` (required), `bucket` (required, but can be same as deep store bucket. If using deep store bucket, be sure to use pathPrefix to avoid collision), `pathPrefix` (optional as a relative path prefix to use in the bucket). The `accessKey`, `secretKey` and `sessionToken` can be saved if servers use role based access control to access S3. |
| tierBackendProperties (for GCS) | For detail about tiered storage backend properties, see [Tiered storage with GCS](#tiered-storage-with-google-cloud-storage-gcs). |

> [!NOTE]
>
> This `segmentAge` setting is independent of the table's retention. Segments will continue to be forgotten from Pinot once the table's retention is reached, regardless of what tier the segments are on.

### Tiered storage with Google Cloud Storage (GCS)

Tiered storage with Google Cloud Storage (GCS) is supported using the [Cloud Storage XML API](https://cloud.google.com/storage/docs/interoperability). This API lets you access Pinot segments stored in a GCS bucket using a S3 client. To use a GCS bucket as the tier backend, complete the following steps:

1. To use GCS's interoperable APIs with a S3 client, you must [generate HMAC keys](https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create) for the GCP bucket. Each HMAC key consists of an access ID and a secret key and can be associated with any service account.

```
$ gsutil hmac create 
AccessId: 
SecretKey: 
```

2. For testing purposes, you can store the HMAC key pair in `tierBackendProperties`, as shown in the following example:

```json theme={null}
"tierBackendProperties": {
  "bucket": "",
  "endpoint": "https://storage.googleapis.com",
  "region": "auto",
  "accesskey": "",
  "secretkey": "",
  ...
}
```

**To avoid exposing the HMAC key pair** in the table configuration, we recommend storing the key pair as secrets in the [GCP secret manager](https://cloud.google.com/secret-manager/docs), and configuring the table to fetch them securely when creating the client. To do this, complete the following steps:

1. [Enable the GCP secret manager API](https://cloud.google.com/secret-manager/docs/configuring-secret-manager), and then [add the key pair as secrets](https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#create) (one for the access ID and the other for the secret key).
2. Ensure that the service account used to create the cluster has [access to read the added secrets](https://cloud.google.com/secret-manager/docs/manage-access-to-secrets).
3. Set the following properties in `tierBackendProperties`:

   ```json theme={null}
   "tierBackendProperties": {
     "bucket": "",
     "endpoint": "https://storage.googleapis.com",
     "accesskey": "",
     "secretkey": "",
     "region": "auto",
     "keyType": "SECRET",
     "secretmanagertype": "GCS",
     "gcpprojectid": "",
     "gcpkeypath": "",
     ...
     }
   ```

   The `gcpprojectid` and `gcpkeypath` are required to be able to access the GCP secret manager. The `gcpkeypath` should be set to the absolute path where the gcp key is mounted on the server (typically `/home/pinot/gcp/credentials.json`).

## Upload new segments and migrate existing segments

When uploading new segments to the table, the segments are put on the remote tier if their data is older than the specified `segmentAge`. For existing segments, the `SegmentRelocator` task moves them to the remote tier periodically as configured.

## Validate and query

For segments that move to the remote tenant, verify the following:

1. From the Swagger UI on [the host](http://localhost:9000/help#/Schema/) where Pinot is running use the `segments/{tableName}/tiers` API to check segments' current tier
2. Segments are not present on the pinot-server’s local disk (in dataDir and indexDir)
3. Segments are present in the remote tier backend in uncompressed form

When querying, you may need to increase [timeout of your query](https://docs.pinot.apache.org/basics/getting-started/frequent-questions/query-faq#how-do-i-increase-timeout-for-a-query).

We have adopted multiversion concurrency control (MVCC) to make remote segment management transparent to ongoing queries. MVCC can lead to multiple copies of segment data with different index or encoding types kept in the remote tier backend. Automatic periodic cleanup of unused segment directories can be enabled using the configs listed in the [Stale segment directory cleanup](#stale-segment-directory-cleanup) section.

With optimizations to be discussed below, the query latency can be reduced as needed.

## Fine tunings for query speed

You have the option to use the following tuning adjustments to speed up queries.

### Preloaded index

Pin some index types of some columns on the local disk to avoid frequent remote reads.

For example, pin bloom filters on local disk so that segments are pruned first to reduce the number of segments processed, minimizing remote reads. Bloom filters tend to be compact, not affecting local disk usage much. All index types can be pinned, including the tree nodes inside the star-tree index. Decide what to pin based on the need for cost savings and reduced query latencies.

Indexes to be preloaded can be specifed at a cluster level globally, and can be overriden at a table level by setting a table level override either via cluster configuration or table configuration.

1. To enable the buffer preloading, use [cluster config apis](https://docs.pinot.apache.org/configuration-reference/cluster#cluster-configs-apis) to add the following option to your cluster configuration:

```json theme={null}
{
   "pinot.server.instance.buffer.reference.manager.preload.enable": "true",
   "pinot.server.instance.buffer.reference.manager.preload.enable.index.consolidation": "true",
   "pinot.server.instance.buffer.reference.manager.preload.total.size": "100G",
   "pinot.server.instance.buffer.reference.manager.preload.dir": "/home/pinot/data/tieredStorage/preload/",
   "pinot.server.instance.buffer.reference.manager.preload.load.existing.buffers": "true",
   "pinot.server.instance.buffer.reference.manager.preload.index.keys.override": ",col1.bloom_filter,col2.sparse;,startree_index_0.tree"
}
```

The following buffer preload options are configured by default:

| Config | Description | Default value |
| - | - | - |
| preload.enable | Whether to use preload feature | false |
| preload.enable.index.consolidation | Whether to consolidate preload indexes. This is to make all column indexes the same file to prevent too many mmap | false |
| preload.total.size | How much disk space to hold preloaded index | 100G |
| preload.dir | Where to keep the preloaded index. Be sure this is different from `dataDir` which server processes segments while loading them | required |
| preload.load.existing.buffers | Whether to keep preloaded index across server restarts | true |
| preload.index.keys | What index types to pin, applied to all tables | empty |
| preload.index.keys.override | Same as above but for specific table. Be sure to use tableNameWithType like foo\_OFFLINE | empty |

2. Once enabled, set global buffer preloading using the following cluster configuration:

```json theme={null}
{
   "pinot.server.instance.buffer.reference.manager.preload.index.keys": "*.bloom_filter,col2.sparse,startree_index_0.tree"
}
```

Follow the `.` convention to specify the index type to preload. Use `*` as a wildcard for the column name to preload all index buffers of a given index type.

3. To override the global cluster level configuration, specify the configuration option on the table level:

* To specify the override in the cluster configuration, set the `pinot.server.instance.buffer.reference.manager.preload.index.keys.override` cluster configuration option as follows:

```json theme={null}
{
   "pinot.server.instance.buffer.reference.manager.preload.index.keys.override": ",*.bloom_filter,col2.sparse;,startree_index_0.tree"
}
```

Make sure to use `tableNameWithType` like `foo_OFFLINE`.

* To specify the override in the table configuration, set the `preload.index.keys.override` property inside `tierBackendProperties` as follows:

```json theme={null}
"tierConfigs": [
  {
    "name": "remoteTier",
      ...
    "tierBackendProperties": {
          "region": "",
          "bucket": "",
          "preload.index.keys.override": "*.bloom_filter,col2.sparse,col3.inverted_index"
          ...
    }
  }
]
```

4. Both clusterConfig and tableConfig changes for preloading index as said above can be applied by reloading segments. Btw, restarting servers also works if wanted.

### Automatic caching (Experimental)

Caching is different from preloading an index. You can decide what to cache and evict from local disk automatically, based on commonly accessed segment data by ongoing queries. If queries find cached index data, the remote reads can be saved.

Automatic caching is specified as part of the cluster configuration like this:

```json theme={null}
{
   "pinot.server.instance.buffer.reference.manager.mmap.enable": "true",
   "pinot.server.instance.buffer.reference.manager.mmap.total.size": "100G",
   "pinot.server.instance.buffer.reference.manager.mmap.dir": "/home/pinot/data/tieredStorage/mmap/",
   "pinot.server.instance.buffer.reference.manager.mmap.load.existing.buffers": "true"
}
```

Review the related configuration options here:

| Config | Description | Default value |
| - | - | - |
| mmap.enable | Whether to use this caching feature | false |
| mmap.total.size | How much disk space to hold cached index | 100G |
| mmap.dir | Where to keep the cached index. Be sure this is different from `dataDir` which server processes segments while loading them | required |
| mmap.load.existing.buffers | Whether to keep cached index across server restarts | true |
| mmap.num.buffers | How many index to cache | unlimited |
| mmap.cache.refresh.policy | Only LFU available for now | LFU |
| mmap.cache.refresh.initial.delay.seconds | When to start cache refresher | 30 |
| mmap.cache.refresh.frequency.seconds | How often to refresh the cache | 60 |

Because these are cluster configs, any changes will require you to restart the servers.

### On demand access

If index data isn't found in local cache, StarTree accesses the data from the remote tier backend on demand during query execution.

> [!NOTE]
>
> Using cache alone does not ensure stable and predicable query performance, particularly when the amount of data in the remote tier backend is more than the size of local disk by multiple orders. But the size of each Pinot segment is limited, and at any time only a limited number of segments are processed, which lets you configure the memory space for on demand access accordingly.

The size of `ondemand` space is an important configuration for query performance. If it's too small, it may become a bottleneck to prefetch segment data effectively or to fully parallelize the remote reads. The `reserved` size is to ensure query execution can always proceed, so it should be larger than the max segment size in the table. If query execution tends to wait for `ondemand` space, you should increase the `reserved` space. You've seen these two configs above in [Add cluster configs and restart servers](#add-cluster-configs-and-restart-servers).

```json theme={null}
{
  "pinot.server.instance.buffer.reference.manager.ondemand.total.size": "24G",
  "pinot.server.instance.buffer.reference.manager.ondemand.reserved.size": "4G"
}
```

Review the related configuration options here:

| Property | Description | Default value |
| - | - | - |
| ondemand.total.size | Max memory for remote data access. Ensure this is smaller than direct memory available on server else you’ll get OutOfDirectMemory exceptions | 24G |
| ondemand.reserved.size | This is a portion of the above total size. Ensure this is larger than size of largest uncompressed segment | 4G |

Because these are cluster configs, any changes will require you to restart the servers.

### Access methods

There are three methods for on demand data access: prefetch, read ahead, and block cache prefetch read.

Prefetching happens asynchronously before segment processing but it gets the index data as a whole.

Reading ahead happens on demand during segment processing synchronously, which adds I/O waiting time onto query latency directly, but only to get the index data requested by the query plus a small amount of bytes ahead, reading just a few small chunks from the index data.

By default, all remote data is obtained via prefetching. But based on the query, such as the indexes accessed, you can enable `readAhead` to speed up the query. The access methods can be turned on/off as a query option, making it easy to experiment with both access methods, as in this example:

```sql theme={null}
SET "readAhead.enable" = true;
SET "prefetch.column.index.list.processing.phase"='col1.dictionary,col2.dictionary';
SET "readAhead.column.index.list.processing.phase"='col3.dictionary.initialBytes=1K;maxBytes=1M;numTriesBeforeDouble=10,teamID.dictionary.downloadAll=true';
SELECT ....
```

Block cache prefetch reads data on demand in chunks of size blockCache.blockSize.bytes. Instead of fetching entire indexes, it uses the post-filter results to selectively retrieve only the needed blocks. Unlike read-ahead, which performs synchronous reads during query execution, block cache prefetch issues parallel asynchronous S3 reads for all selected blocks up front, improving efficiency and reducing latency.

To enable block cache prefetch, set the following query options:

```sql theme={null}
SET "readAhead.enable" = true;
SET "readAhead.use.blockCache" = true;
SET "blockCache.blockSize.bytes"=128000;
SELECT ....
```

We recommend using 128KB as the default block size. If the block size is too small, the overhead of each S3 request becomes significant. If it's too large, the latency of individual S3 requests starts increasing with the block size. However, if you know your query patterns tend to access documents in contiguous ranges, you may increase the block size to reduce the number of S3 requests.

Note that you can also specify these options in table configs under `tierBackendProperties`, it will apply to all queries towards this table.

```json theme={null}
"tierConfigs": [
  {
    "name": "remoteTier",
      ...
    "tierBackendProperties": {
        "region": "",
        "bucket": "",
        "readAhead.enable": "true",
        "readAhead.use.blockCache": "true",
        "blockCache.blockSize.bytes": "128000"
        ...
    }
  }
]
```

Even when readAhead or block cache prefetch is enabled, not all index data is accessed by reading ahead or block cache prefetch. Under the hood, a fetch planner decides which access method is used to read which column index.

For example, if a column is dictionary encoded and is present in the query projection clause, the column's dictionary is prefetched. Because when evaluating the projection clause, the column's dictionary is probed (via a binary search) for every single matching document.

Prefetching the whole dictionary allows us to handle those relatively random reads from binary search more efficiently.

By default, this happens like in this example, but remember the order can always be overridden by the query options shown above.

1. For columns in non-predicate clauses (such as `projection`, `groupBy`, and `orderBy`): if they are [dictionary encoded](https://docs.pinot.apache.org/basics/indexing/forward-index#dictionary-encoded-forward-index-with-bit-compression-default), their dictionaries are prefetched; but in any case, their [forward index](/corecapabilities/manage-data/indexes/forward-index) data is accessed via the `readAhead` method, because only part of the data is needed to evaluate the clauses.
2. For columns in predicate clauses (such as a where clause): if they have no index to be used to evaluate the predicates, their forward index data is prefetched as a whole, because the whole forward index data is scanned to evaluate the predicates anyway; otherwise, their index data (like an [inverted index](/corecapabilities/manage-data/indexes/inverted-index), [range index](/corecapabilities/manage-data/indexes/range-index) or [json index](/corecapabilities/manage-data/indexes/json-index)) is accessed via the readAhead method, because only part of them is needed to evaluate the predicates.

Review the related configuration options here:

| Config | Description | Default value |
| - | - | - |
| readAhead.enable | Whether to enable read ahead access method (need to set readAhead.use.blockCache=true first) | false, i.e. all by prefetching |
| prefetch.column.index.list.processing.phase | Force to use prefetching to access some index | empty |
| readAhead.column.index.list.processing.phase | Force to use readAhead to access some index | empty |
| initialBytes (readAhead specific) | Initial buffer size to access data by read ahead | 10KB |
| maxBytes (readAhead specific) | Max buffer size to access data by read ahead | 1MB |
| numTriesBeforeDouble (readAhead specific) | How many reads before double the buffer size | 1 |
| downloadAll (readAhead specific) | Whether to download all index data by a single read, avoiding the cost of growing buffer | false |
| readAhead.use.blockCache | Whether to use block cache prefetch read instead of read ahead (need to set readAhead.enable=true first) | false |
| blockCache.blockSize.bytes | The size of the block to prefetch in block cache prefetch read | 10000 |

Change of these configs do not require you to restart servers, because they are set as query options.

### Forward Block Index

Forward block index is a small utility index that provides lookup capability on forward indexes without scanning the entire column. It helps Pinot identify the exact block or blocks needed for a given row range, improving read efficiency, particularly under tiered storage. It works together with block cache prefetch to enhance data locality and reduce amount of remote data to read.

When using block cache prefetch, enable the forward block index on raw forward index columns. The block size should align with the chunk size used by the raw forward index (a block size of 1000 is a good default). Here's an example configuration:

```json theme={null}
"fieldConfigList": [
    {
      "name": "columnName",
      "encodingType": "RAW",
      "indexTypes": [],
      "indexes": {
        "forwardBlock": {
          "blockSize": 1000
        }
      },
      "tierOverwrites": null
    }
]
```

When using forward block index, you must pin it onto local disk. To do this, add `*.forward_block` to the `pinot.server.instance.buffer.reference.manager.preload.index.keys` cluster configuration. Additionally, set `pinot.server.instance.index.sparse.enabled` to true in the cluster config to enable prefetching support.

### Star-tree index

Enable the star-tree index with the `enable.startree.index` configuration, in `tierBackendProperties` (shown below). Changes here require you to reload the table or restart the servers. This is disabled by default.

```json theme={null}
"tierConfigs": [
      {
        "name": "remoteTier",
         ...
        "tierBackend": "s3",
        "tierBackendProperties": {
             "region": "",
             "bucket": "",
             "enable.startree.index": "true"
             ...
        }
      }
    ],
```

Each [star-tree Index](/corecapabilities/manage-data/indexes/startree-index) consists of two major parts: tree nodes and pre-aggregated values.

The tree nodes have a small number of metadata about where to find the pre-aggregated values. If a query fits a StarTree Index, the Pinot server would traverse the tree nodes to identify the pre-aggregated values and merge them.

Because the total size of tree nodes tends to be much smaller than that of all pre-aggregated values, and traversing tree nodes incurs relatively random reads, we suggest preloading the tree nodes on local disk and leaving the pre-aggregated values in remote store for best performance and cost balance when using the star-tree index. If not preloaded, the tree nodes are accessed via the `readAhead` method during query.

Here is an example to preload the tree nodes of StarTree Index. The count suffix is aligned with the order of star-ree index configurations set in the table configuration.

```json theme={null}
{
   "pinot.server.instance.buffer.reference.manager.preload.index.keys.override": ",startree_index_0.tree,startree_index_1.tree"
}
```

### Tier overwrites

The performance/cost requirements can be very different for data on the local and remote tiers, requiring different index across tiers.

For example, if you want to add a bloom filter, inverted index, and star-tree index for segments on the local tier to lower the query latency, and just add bloom filters on the remote tier to save cost.

To configure indexes according to tiers, see [Overwrite index configs at tier level](https://docs.pinot.apache.org/basics/getting-started/frequent-questions/operations-faq#overwrite-index-configs-at-tier-level).

### S3 client configs

The S3 client can also be tuned, for example, to tolerate longer request latency. However, the default of the S3 configurations should work in most cases.

In case there is a need, below is the list of current configs to customize when S3 clients fetch data from remote segments. All those configs are put inside `tierBackendProperties`. Most of them need to reload the table or restart servers to take effect.

Pinot supports both sync and async s3 client.

By default, sync s3 client is used and a thread pool is created to fetch data via the sync s3 clients in parallel, without blocking the query processing threads. The configs for sync s3 client are prefixed with `s3client.http`.

The async s3 client can reduce the size of the thread pool considerably. It uses async I/O to fetch data in parallel in a non-blocking manner; and uses the thread pool mentioned above to process the I/O completion callbacks only. The configs for the async s3 client are prefixed with `s3client.asynchttp`. Configs prefixed with `s3client.general` apply to both kinds of clients.

Review the related configuration options here:

| Config | Description | Default value |
| - | - | - |
| s3client.general.apiCallTimeout | Timeout for e2e API call, which may involve a few http request retries. | no limit |
| s3client.general.apiCallAttemptTimeout | Timeout for a single http request. | no limit |
| s3client.general.numRetries | How many times to retry the http request, with exponential backoff and jitter. | 3 |
| s3client.general.readBufferSize | Size of buffer to stream data into buffers for query processing. | 8KB |

| Config | Description | Default value |
| - | - | - |
| s3client.http.maxConnections | The max number of connections to pool for reuse | 50 |
| s3client.http.socketTimeout | Timeout to read/write data | 30s |
| s3client.http.connectionTimeout | Timeout to establish connection | 2s |
| s3client.http.connectionTimeToLive | Timeout to reclaim the idle connection | 0s |
| s3client.http.connectionAcquireTimeout | Timeout to get a connection from pool | 10s |
| s3client.http.connectionMaxIdleTimeout | Timeout to mark a connection as idle to be candidate to reclaim | 60s |
| s3client.http.reapIdleConnections | Reclaim the idle connection in pool | true |

| Config | Description | Default value |
| - | - | - |
| s3client.asynchttp.maxConcurrency | The max number of concurrent requests allowed. We use HTTP/1.1, so this controls max connections. | 50 |
| s3client.asynchttp.readTimeout | Timeout to read data | 30s |
| s3client.asynchttp.writeTimeout | Timeout to write data | 30s |
| s3client.asynchttp.connectionTimeout | Timeout to establish connection | 2s |
| s3client.asynchttp.connectionTimeToLive | Timeout to reclaim the idle connection | 0s |
| s3client.asynchttp.connectionAcquireTimeout | Timeout to get a connection from pool | 10s |
| s3client.asynchttp.connectionMaxIdleTimeout | Timeout to mark a connection as idle to be candidate to reclaim | 60s |
| s3client.asynchttp.reapIdleConnections | Reclaim the idle connection in pool | true |

## Configs for ease of operations

You have the option to use the following adjustments to simplify operations.

### Server restart optimizations

When tiered storage is enabled, server restarts can take a lot longer than otherwise. This is primarily due to servers having to make multiple object store calls when loading a segment. This high restart time can be problematic especially during upgrades and server outages.

To alleviate this, enable segment header caching by setting the `pinot.server.instance.segment.cache.directory` cluster configuration to a path. This path must be different from the dataDir. Setting the property leads to segment metadata files, such as `creation.meta`, `metadata.properties`, and `index_map`, along with certain byte slices (specifically, the header bytes of each of the index buffers) of the `columns.psf` file being cached onto the disk.

These cached files are used during server startup and eliminate the expensive object store calls, which helps in reducing the overall server startup time.

> [!NOTE]
>
> This feature only helps with server restarts and not when adding a new server since no cached files will be present when a server comes up for the first time.

### Minion Tasks

Segment reload or refresh operations can be resource-intensive, especially with tiered storage enabled tables, due to heavy remote fetches and uploads. It's recommended to offload this overhead from Pinot servers to minions.
This can be achieved by enabling `"directTierUpload": "true"` in the individual table's minion task config and adding the cluster config `"pinot.server.instance.table.data.manager.provider.class" : "ai.startree.pinot.data.tier.table.StarTreeDefaultTableDataManagerProvider"` but will be made default in later releases.
A server restart is required for the cluster config to take effect.

## Note

As of now, File ingestion or Segment import task is currently not supported.

```json theme={null}
    "task": {
      "taskTypeConfigsMap": {
        "SegmentRefreshTask": {
          "directUploadToTier": "true"
        }
      }
    }
```

```json theme={null}
    "task": {
      "taskTypeConfigsMap": {
        "StarTreeAlterTableTask": {
          "directUploadToTier": "true"
        }
      }
    }
```

Once the configs are added, initiate a segment refresh/ StarTreeAlterTableTask operation for the table using the `POST /tasks/schedule` API call.

```bash theme={null}
curl -X POST "http:///tasks/schedule?taskType=SegmentRefreshTask&tableName=" -H "accept: application/json"
```

### Stale segment directory cleanup

With the implementation of segment directory level MVCC, a new segment directory will be created in the object store whenever a segment is reloaded to apply changes in the schema or table configs. By default, these directories are not automatically removed, potentially resulting in a significant utilization of object storage space to store outdated segment directories that correspond to previous schema or table configs.

To address this issue, periodic cleanup of these stale segment directories can be enabled by adding the following cluster config:

```json theme={null}
{
  "controller.tieredStorage.segment.cleanup.enabled": "true"
}
```

> [!NOTE]
>
> All of the cleanup configs below are dynamic cluster configs — the controller picks up changes to them without a restart. The one exception is `controller.tieredStorage.segment.cleanup.frequencyPeriod`, which is read when the periodic task is scheduled at controller startup, so changing the schedule does require a controller restart.

In addition to stale versions of live segments, the cleanup task also reclaims *dangling* segment directories — directories still present in the tier backend that are no longer backed by any segment in the table's ideal state, for example directories left behind when a segment was deleted from the table. Dangling directories go through the same tombstone-then-delete flow, so they are subject to the same TTLs described below.

The following additional settings can also be used to further refine the behavior of deleting stale segment directories.

| Config | Description | Default value |
| - | - | - |
| controller.tieredStorage.segment.cleanup.enabled | Whether the stale segment directory cleanup task performs any work | false (disabled) |
| controller.tieredStorage.segment.cleanup.frequencyPeriod | The frequency at which the controller should check for stale segment directories | 1h |
| controller.tieredStorage.segment.cleanup.tombstoneTtlMillis | The amount of time in milliseconds before an unused segment directory can be tombstoned | 2h (7200000) |
| controller.tieredStorage.segment.cleanup.tombstoneToDeletionTtlMillis | The amount of time in milliseconds before a tombstoned segment directory is deleted | 6h (21600000) |

The periodic task is always scheduled at `frequencyPeriod`, but it is a no-op until `controller.tieredStorage.segment.cleanup.enabled` is set to `true`. When cleanup is disabled, each run logs `Tiered storage segment directory cleanup is disabled. Skipping cleanup task.` and exits without touching the tier backend.

Increasing `controller.tieredStorage.segment.cleanup.tombstoneTtlMillis` allows more time for reusing old segment directories if table config reverts. However, it prolongs the persistence of stale directories in the object store.

`controller.tieredStorage.segment.cleanup.tombstoneToDeletionTtlMillis` config specifies the wait time before deleting a tombstoned directory. This prevents the wrongful deletion of a segment directory if any server starts using the segment directory concurrently during the tombstoning process.

> [!TIP]
>
> Use `GET /tables/{tableName}/storageUsage` to see how much space stale versions are taking up in the tier backend before and after enabling cleanup. Stale versions are reported under `breakdown.segmentDirectory.unexpected.staleVersions`, along with a `totalStaleVersions` count.

## Monitoring metrics

Below are metrics we emit to help understand how tiered storage works. The metrics mainly cover how:

* Segments get uploaded to the tier backend, such as data volume and operation duration, rates or failures
* Queries fetch data from the remote segments, such as data volume and query duration breakdown over query operators, query rates or failures
* Segment data is kept on servers temporarily, such as data volume from segments temporarily held in memory or on disk

Review the available metrics here:

| Metrics | Description |
| - | - |
| pinot\_server\_s3Segments\_UploadedCount | How many segments uploaded to S3 tier backend when reloading segment. |
| pinot\_server\_s3Segments\_UploadedSizeBytes | How many bytes in total uploaded to S3 tier backend when reloading segment. |
| pinot\_server\_s3SegmentUploadTimeMs | How long segment uploading takes when reloading segment. |
| pinot\_server\_s3Segments\_DownloadedCount | How many segments downloaded to server when reloading segment. |
| pinot\_server\_s3Segments\_DownloadedSizeBytes | How many bytes in total downloaded to server when reloading segment. |
| pinot\_server\_s3SegmentDownloadTimeMs | How long segment downloading takes when reloading segment (this is not on query path). |
| pinot\_server\_totalPrefetch\_99thPercentile/OneMinuteRate | Rate and time spent to kick off data prefetching, and should be low. The prefetching is done in async. The metric is also broken down for cache layers: `mmapCacheLayer` and `ondemandCacheLayer`. |
| pinot\_server\_totalAcquire\_99thPercentile/OneMinuteRate | Rate and time spent to wait for segment data to be available and may be varying based on the effectiveness of prefetching. The metric is also broken down for cache layers: `mmapCacheLayer` and `ondemandCacheLayer`. |
| pinot\_server\_totalReleas\_99thPercentile/OneMinuteRate | Rate and time spent to release the segment data, and should be low. The metric is also broken down for cache layers: `mmapCacheLayer` and `ondemandCacheLayer`. |
| pinot\_server\_bufferPrefetchExceptions\_OneMinuteRate | Error rate when prefetching segment data. This metric is also broken down for cache layers: `mmapCacheLayer` and `ondemandCacheLayer`. |
| pinot\_server\_bufferAcquireExceptions\_OneMinuteRate | Error rate when waiting to acquire segment data. This metric is also broken down for cache layers: `mmapCacheLayer` and `ondemandCacheLayer`. |
| pinot\_server\_bufferReleaseExceptions\_OneMinuteRate | Error rate when releasing the prefetched segment data. This metris is also broken down for cache layers: `mmapCacheLayer` and `ondemandCacheLayer`. |
| pinot\_server\_ondemandCacheLayerReserve\_99thPercentile | Time for prefetching threads to wait for ondemand memory space to be available to fetch data. If this is high, try to configure more on-demand memory or adjust the query to access less segments like making query predicates more selective. |
| pinot\_server\_ondemandCacheLayerBufferDownloadBytes\_OneMinuteRate | Bytes rate of data prefetching. |
| pinot\_server\_ondemandCacheLayerSingleBufferFetchersCount\_OneMinuteRate | Number of column index pairs being prefetched. |
| pinot\_server\_ondemandCacheLayerBufferDownloadBytes\_OneMinuteRate | Bytes rate of data prefetching. |

> [!NOTE]
>
> In addition to the metrics emitted from tiered storage modules, the cpu util and network bandwidth util are important to help you fine tune the system.


# Use Sparse Index
Source: https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/use-sparse-index



## Introduction

Apache Pinot supports different types of indexes such as forward indexes, Bloom filters, hash and sorted inverse dense indexes, or range inverted indexes. Sparse indexes are a hybrid between inverted indexes and Bloom filters designed to speed up queries that filter results using equality (or IN) over a column with very high cardinality, such as columns storing hundreds of thousands of random values, UUIDs, or public IPs. They are a StarTree extension that shines particularly when tiered storage is used.

## Description

Sparse indexes group rows into contiguous chunks of a predefined size and classify column values into a fixed number of *partitions* using one or many *hash functions* (also called *mappers*).
Finally, it creates a pseudo-inverted index between each *partition* and each *chunk*.

When a filter such as `column = constant` is used, the sparse index looks for the partition where the constant belongs. Then it uses the pseudo-inverted index to obtain the *chunks* where this partition is found. These are the only row groups where the predicate may evaluate to true, so this information is used to optimize I/O access.

This method is useful when the segment is stored locally, but it is particularly notable when using tiered storage. In this case, instead of downloading the whole forward index, Pinot only downloads the interesting *chunks*. This significantly reduces the number of bytes downloaded from the cloud object storage.

Sparse indexes are probabilistic at chunk level. They can include extra candidate chunks, which are removed by final predicate evaluation.

> [!NOTE]
>
> Sparse index cannot be created on a column that also has an inverted index enabled, and it is not supported on MAP columns.

## Enable sparse indexes

To enable the sparse index StarTree extension, set `pinot.server.instance.index.sparse.enabled` to `true` in the server configuration. One way to do so is to use the `POST /cluster/configs` API from Swagger.

After adding new configurations, restart existing servers.

> [!NOTE]
>
> If the `pinot.server.instance.index.sparse.enabled` property is **not** set to `true`, you can create sparse indexes, but they won't be used during query execution.

## Create sparse indexes

Sparse indexes have to be declared in the table configuration, specifically in the `fieldConfigList` section. For example, the following JSON defines a sparse index in the `deviceId` column:

```json theme={null}
{
  "tableName": "example",
  //...
  "fieldConfigList": [
    //...
    {
      "name": "deviceId",
      "indexes": {
        "sparse": {
          "chunkSize": 1024,
          "partitions": 10000,
          "hashFunctionCount": 3
        }
      }
    }
    //...
  ]
}
```

Sparse indexes support several configuration options:

| Property | Type | Default | Recommended | Affects size | Description |
| - | - | - | - | - | - |
| chunkSize | int | 1024 | 128 to 8192 | inverse trend, not strictly linear | Chunk size. Must be a power of 2. |
| partitions | int | 10000 | depends on cardinality and target FP | generally increases, often sub-linear | Number of partitions used. |
| hashFunctionCount | int | 3 | 1 to 8 | often close to linear | Number of partition mappers (hashes). |

### `chunkSize`

Specifies the number of documents that form a chunk.\
This option must be a power of 2.\
Smaller chunks typically reduce false positives and `numEntriesScannedInFilter`, but may increase index size.

### `partitions`

Specifies how many partitions values are classified into.\
The recommended value depends on expected cardinality and false-positive targets.\
Increasing partitions generally increases index size and can improve selectivity.

### `hashFunctionCount`

Specifies the number of function mappers used.\
Larger values improve selectivity but increase index lookups and index size.\
In practice, index size often grows close to linearly with this parameter.

### Extra options

The following configurations should only be used in extremely unique use cases. Typically, we do not recommend configuring these options:

#### `seedGenerator`

An integer used as a seed to later generate seeds used to generate each hash function.
Any integer value is valid, and they produce similar values and do not affect the index size.
This option should only be defined in the strange case that collisions are very frequent in one or many hash functions.

#### `mapperId`

A string that identifies the type of hash function used.
At this moment, the only valid value is `murmur`.

## Quick tuning workflow

1. Pick high-cardinality columns queried with `=` or `IN`.
2. Benchmark representative queries and track:
   * `timeUsedMs`
   * `numEntriesScannedInFilter`
   * `numDocsScanned`
3. Tune in this order:
   * `chunkSize` first
   * `partitions` second
   * `hashFunctionCount` third
4. Compare p50/p95 latency, scan metrics, and index size.
5. Reload/rebuild segments after configuration changes.

### Quick formula intuition

Let $R$ be rows, $c$ chunk size, $m$ partitions, $k$ hash count, and $v$ average values per row.\
Define $N_c = \lceil R / c \rceil$ and $n = c \cdot v$.

$$
p_{\mathrm{fp},\mathrm{chunk}} \approx \left(1 - e^{-n/m}\right)^k
$$

Here, $p_{\mathrm{fp},\mathrm{chunk}}$ is the approximate probability that a chunk is selected by sparse index even when it does not contain the target value (chunk-level false positive).

$$
E_{\mathrm{entries}} \approx N_c \cdot k \cdot m \cdot \left(1 - e^{-n/m}\right)
$$

Here, $E_{\mathrm{entries}}$ is an index-size proxy: the expected number of sparse index entries in the segment. Larger $E_{\mathrm{entries}}$ usually means a larger on-disk sparse index.

For formulas and deeper sizing guidance, see [Sparse Index](/corecapabilities/manage-data/indexes/sparse-index).


# Text Index support for Tiered Storage
Source: https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/use-text-index



## Context

Having text index support for data stored in remote tiered storage is crucial for accelerating performance for text search based queries. Previously, text indexes were not supported in StarTree Tiered storage due to various reasons (text index stored separate from Pinot segment and not integrated in columns.psf)

Starting **Startree release 0.11.1**, text indexes can now be used in conjunction with Startree Tiered storage configuration.

Key changes made to enable this support

* **File Consolidation**: Text index directories are now consolidated into the columns.psf file.
* **Tiered Storage Enabled**: The inclusion of text indexes within `columns.psf` automatically enables tiered storage support.
* **Minimal Configuration**: Activation of the new format merely requires setting the storeInSegmentFile flag.

**Operational Mechanism**

* **Default Mode**: Text indexes continue to employ separate directories, thereby lacking tiered storage support.
* **Consolidated Mode**: When `storeInSegmentFile: "true"` is configured, text indexes are stored within columns.psf, enabling tiered storage support.

**Backward Compatibility**: Existing segments containing text indexes in separate directories remain fully functional without alteration.

## Sample configuration

The `storeInSegmentFile` flag controls whether text indexes are stored within the segment file or in separate directories:

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "your_text_column",
      "indexes": {
        "text": {
		  ...
          "storeInSegmentFile": "true"
        }
      },
	  ...
```

With this, users can leverage text index with tiered storage enabled tables.

# **How to Enable Text Index for S3 Tiered Storage**

## New Table with Text Index and S3 Tiered Storage Support

For new tables, you have two options for configuring text indexes with S3 tiered storage support:

### Case 1: Both Local and S3 in Consolidated Format

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "text_column",
      "indexes": {
        "text": {
          "storeInSegmentFile": "true"  // Both local and S3 in consolidated format
        }
      },
      "properties": {
        "luceneUseCompoundFile": "false",
        "luceneMaxBufferSizeMB": "128",
        "useANDForMultiTermQueries": "true",
        "luceneAnalyzerClass": "org.apache.lucene.analysis.standard.StandardAnalyzer",
        "luceneQueryParserClass": "org.apache.lucene.queryparser.classic.QueryParser"
      }
    }
  ]
}
```

### Case 2: Local in Default Format, S3 in Consolidated Format

In this case, we use the `tierOverwrites` to decouple local vs S3 format.

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "text_column",
      "indexes": {
        "text": {
          "storeInSegmentFile": "false"  // Local storage in default format
        }
      },
      "properties": {
        "luceneUseCompoundFile": "false",
        "luceneMaxBufferSizeMB": "128",
        "useANDForMultiTermQueries": "true",
        "luceneAnalyzerClass": "org.apache.lucene.analysis.standard.StandardAnalyzer",
        "luceneQueryParserClass": "org.apache.lucene.queryparser.classic.QueryParser"
      },
      "tierOverwrites": {
        "s3Tier": {
          "indexes": {
            "text": {
              "storeInSegmentFile": "true"  // S3 storage in consolidated format
            }
          }
        }
      }
    }
  ]
}
```

## Enable Text Index on Existing Table with S3 Tiered Storage

For existing tables, local storage can remain as it is (default format), and tierOverwrites with storeInSegmentFile: "true" can be used to enable text index on S3 tiered storage:

**Note**: For existing tables, the new flag will take effect on existing segments after a table reload but new segments(data) will pick the changes. Table reload can take some time based on the data size in tables.

**Important**: Local storage can remain in default format (separate directories), but when moving to S3, storeInSegmentFile must be set to "true" for tiered storage to work.

```json theme={null}
{
  "fieldConfigList": [
    {
      "name": "your_text_column",
      "indexes": {
        "text": {
         ...
        }
      },
      "properties": {
       ...
      },
      "tierOverwrites": {
        "myS3Tier": {
          "indexes": {
            "text": {
              "deriveNumDocsPerChunk": false,
              "rawIndexWriterVersion": 4,
              "storeInSegmentFile": "true"  // Enable consolidation for new segments
            }
          }
        }
      }
    }
  ]
}
```

## How to Pin the Text Index

\*\*Text index pinning works the same way as inverted index pinning and is specified using \*\*preload.keys in the tier configuration. For detailed information on preloaded index configuration, refer to the[StarTree documentation on preloaded index](https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/setup#preloaded-index) .

To pin text indexes to specific storage tiers, configure the preload.keys in your tierConfigs:

```json theme={null}
{
  "tierConfigs": [
    {
      "name": "remoteTier",
      "segmentSelectorType": "time",
      "segmentAge": "7d",
      "storageType": "pinot_server",
      "serverTag": "RemoteTenant_OFFLINE",
      "tierBackend": "s3",
      "tierBackendProperties": {
        "pathPrefix": "",
        "region": "",
        "bucket": ""
      },
      "preload.index.keys.override": "*.text_index"  // Pin text index to this tier
    }
  ]
}
```

**Note**\
Tiered storage features for text indexes work similar to other indexes (for example, inverted index). Once text indexes are consolidated into the pinot segment file with storeInSegmentFile: "true", they can be moved between storage tiers just like any other index type.


# Storage efficiency
Source: https://docs.startree.ai/corecapabilities/manage-data/storage-efficiency

Reduce a table's footprint without hurting query performance — choose encodings and compression codecs, size segments, prune indexes, and measure the effect of each change on your own data.

[Table Storage Usage](/corecapabilities/manage-data/table-storage-usage) tells you how much a table is consuming and where. This page is about the next question: what to change, and how to know whether the change helped.

There is no universally correct configuration. Compression ratio depends on your data's cardinality, sortedness, and value distribution, so the honest answer to "which codec is smallest" is always "measure it on this column." What this page gives you is the set of levers, in the order that usually pays best, and a method for measuring each one.

## Where the bytes actually go

Before changing anything, find out what is large. A table's on-disk footprint is the sum of:

| Component | Typically dominated by |
| - | - |
| **Forward indexes** | The raw column values. Usually the largest single contributor. |
| **Dictionaries** | High-cardinality string columns, where the dictionary approaches the size of the data |
| **Auxiliary indexes** | Inverted, range, text, JSON, and star-tree indexes — each one is additional bytes on top of the forward index |
| **Metadata** | Per-segment and per-column metadata. Small individually, significant across tens of thousands of segments |

Open the table's **Storage** tab and note three figures before you start: **Used**, **Orphaned**, and **Avg Segment Size** alongside the segment-size percentiles.

> [!WARNING]
>
> **Check `Orphaned` first.** If a meaningful share of the footprint is orphaned — unknown files, stale versions, or deleted segments past retention — no amount of codec tuning is the right fix. That storage is not holding your data. See [What "Orphaned" includes](/corecapabilities/manage-data/table-storage-usage#what-orphaned-includes).

## Lever 1: retention and tiering

The cheapest byte is the one you are not storing.

* **Retention.** Confirm the table has a retention period and that it matches what the data is actually used for. A table with no retention configured fails the [`SEGMENT_RETENTION_CHECK`](/corecapabilities/observability/health-checks#segment_retention_check) health check and grows until it hits its quota.
* **Tiered storage.** Move cold data to object storage rather than keeping it on server disks. This changes where the bytes live and what they cost, without deleting anything. See [Tiered storage](/corecapabilities/manage-data/set-up-tiered-storage/motivation).

Both of these usually move more bytes than any encoding change, and neither requires touching the schema. Do them first.

## Lever 2: segment sizing

Segment size affects both storage and query cost. Many small segments compress worse than the same rows in fewer segments — compression works on blocks, and small blocks give it less to work with — while also multiplying per-segment metadata and per-query overhead.

Signals that this is your problem:

* [`SEGMENT_SIZE_CHECK`](/corecapabilities/observability/health-checks#segment_size_check) failing (more than a quarter of segments under 5 MB)
* [`SEGMENT_COUNT_CHECK`](/corecapabilities/observability/health-checks#segment_count_check) failing (more than 50,000 segments)
* A large gap between **Avg Segment Size** and the p90/p99 figures on the Storage tab

What to do:

* For real-time tables, raise the flush threshold so segments seal larger. See [Configuring the segment threshold](/recipes/configuring-segment-threshold).
* For existing small segments, merge them — [real-time](/recipes/merge-segments-realtime) or [offline](/recipes/merge-small-segments).
* For offline tables, review push granularity: a daily push of a small table produces small segments by construction.

## Lever 3: encoding — dictionary or raw

Every column is stored either dictionary-encoded or as raw values, and this choice matters more than the codec applied on top of it.

| Encoding | Stores | Best for |
| - | - | - |
| **Dictionary** | A dictionary of distinct values, plus per-row dictionary IDs | Low to medium cardinality. Repeated values collapse to a small ID, so the saving grows as cardinality falls relative to row count. |
| **Raw** | The values themselves, in compressed chunks | High cardinality. When almost every value is distinct, the dictionary is nearly as large as the data and buys nothing while adding an indirection on every read. |

The failure mode to look for is a **high-cardinality string column left dictionary-encoded** — request IDs, URLs, free-text fields, UUIDs. The dictionary is large, poorly compressible, and held in memory. Switching such a column to raw encoding with a codec is frequently the single largest saving available on a wide table.

Dictionary encoding is also a prerequisite for some index types, so check what else a column is used for before switching it. See [Dictionary index](/corecapabilities/manage-data/indexes/dictionary-index) and [Forward index](/corecapabilities/manage-data/indexes/forward-index).

## Lever 4: compression codec

Codecs are set per column through `fieldConfigList`. See [Forward index](/corecapabilities/manage-data/indexes/forward-index) for the configuration syntax; this section is about which one to pick.

### Codecs for raw-encoded columns

| Codec | Character |
| - | - |
| `PASS_THROUGH` | No compression. Smallest CPU cost, largest footprint. Reasonable only when the column is already incompressible and latency-critical. |
| `LZ4` | Fast to compress and decompress, moderate ratio. The usual default for raw columns on a query-latency-sensitive table. |
| `SNAPPY` | Similar profile to LZ4 — balanced ratio and speed. |
| `ZSTANDARD` | Noticeably better ratio, more CPU on decompress. The usual choice when the column is large and not on the hot path of your tightest queries. |
| `GZIP` | High ratio, slowest. Rarely the right answer for query-serving data; occasionally right for very cold columns. |

The trade-off is always the same shape: better ratio costs decompress CPU on every query that reads the column. That is why "which codec" cannot be answered without knowing whether the column is read by your latency-critical queries or only by occasional ad-hoc ones — the same table can justify `ZSTANDARD` on one column and `LZ4` on another.

### Specialized codecs

| Codec | Applies to |
| - | - |
| `MV_ENTRY_DICT` | Multi-value **dictionary-encoded** columns. Adds a second level of dictionary encoding over the multi-value entries themselves, which pays when the same combinations of values recur across rows. The only codec applicable to dictionary-encoded columns. |
| `DELTA`, `DELTADELTA` | Numeric `INT` and `LONG` columns whose consecutive values differ by small amounts — monotonic timestamps, sequence numbers, counters. Stores differences rather than values, then compresses those with LZ4. `DELTADELTA` takes the difference twice, which suits values increasing at a near-constant rate. Ineffective, and possibly worse than `LZ4`, on values that jump around. |
| `CLP`, `CLPV2`, `CLPV2_ZSTD`, `CLPV2_LZ4` | Log-line text. These exploit the repeated structure of log messages — a static template with variable fields — and are not general-purpose string codecs. Use them for columns holding log messages and nowhere else. |

> [!NOTE]
>
> The [forward block index](/corecapabilities/manage-data/indexes/forward-block-index) has its own compression configuration with its own codec names. Do not assume a name from one applies to the other.

## Lever 5: prune indexes you are not using

Each auxiliary index is additional storage. It is worth periodically asking, per index, which query needs it — an inverted index added during a past investigation and never removed is pure cost.

Two things help you answer that:

* [`system_query_log`](/corecapabilities/query_data/advanced_operations/query-logger) shows the predicates your queries actually use. Columns nobody filters on do not need filter indexes.
* `numEntriesScannedInFilter` on those queries tells you whether an index is being used effectively. A filter index that is not reducing scanned entries is not earning its bytes.

The same logic applies in reverse: a column that is scanned heavily might be *worth* more index bytes. Storage efficiency is not the only objective — the goal is knowing what each byte buys.

## Measuring a change

The only credible answer about compression is one measured on your data. The method:

**1. Record the baseline**

From the table's **Storage** tab, note **Used**, **Avg Segment Size**, and the segment-size percentiles. For a per-column view, pull segment metadata for a representative segment through the [segment APIs](/api-reference/segment/reload-a-segment) — per-column index sizes are what tell you whether the column you changed is the one that moved.

**2. Change one thing**

One column, or one codec, at a time. Changing an encoding and a codec together leaves you unable to attribute the result — and the two interact, so the combined effect is not the sum of the separate ones.

**3. Reload, and wait for it to finish**

Configuration changes apply to **new** segments immediately and to existing segments only on reload. Until you reload, the measurement is of the old layout. Use [Reload all segments](/api-reference/segment/reload-all-segments) — or the **Reload segments** remediation on the [`TABLE_SEGMENTS_RELOAD_CHECK`](/corecapabilities/observability/health-checks#table_segments_reload_check) health check — and confirm completion before measuring. For larger structural changes, see the [Alter Table task](/corecapabilities/manage-data/alter-table-task).

**4. Measure both sides**

Compare storage *and* query latency. A codec change that saves 30% of a column's bytes and adds 15 ms to your p99 may or may not be a win — that depends on your workload, and you cannot tell from the storage number alone. Check p95 and p99 for the queries that read the column before and after.

**5. Test on a copy where the stakes are high**

For a large production table, validate on a table built from a representative subset of the same data rather than iterating on the production table. Reloads on a large table are not free, and reverting is another reload.

> [!NOTE]
>
> Compression ratios reported elsewhere — in benchmarks, in other companies' blog posts, in Pinot's own documentation — are properties of the data they were measured on, not of the codec. A codec that halves one team's log column may gain a few percent on your numeric one. Treat published ratios as an ordering hint, never as a projection for your table.

## Related

* [Table Storage Usage](/corecapabilities/manage-data/table-storage-usage) — measuring the footprint and understanding orphaned storage
* [Forward index](/corecapabilities/manage-data/indexes/forward-index) — encodings and the `compressionCodec` configuration syntax
* [Dictionary index](/corecapabilities/manage-data/indexes/dictionary-index) — when dictionary encoding pays
* [Tiered storage](/corecapabilities/manage-data/set-up-tiered-storage/motivation) — moving cold data to object storage
* [Health check reference](/corecapabilities/observability/health-checks) — the segment-size, segment-count, and retention checks
* [Configuring the segment threshold](/recipes/configuring-segment-threshold) — sizing real-time segments


# Table Storage Usage
Source: https://docs.startree.ai/corecapabilities/manage-data/table-storage-usage

Understand a table's storage in the StarTree Data Portal — Local, Deep, and Tiered Storage, what Used and Orphaned mean, and the underlying API.

## Overview

Unexpected growth in table storage increases both infrastructure and object-storage costs. Pinot keeps a table's data in several places — the Pinot servers' local disks, the deep store, and the remote object stores used by tiered storage — and the **Storage** tab in the StarTree Data Portal reports the size at each location and highlights the gap between what *should* be there and what is *actually* stored.

Use it to:

* Track the total storage footprint across local disk, deep store, and tiered storage
* Spot untracked segments and stale versions that no longer belong to the table
* Find deleted segments that are past retention but still consuming deep store space
* Debug discrepancies during rebalances, segment replacement, and tiered-storage cleanup

Most of what you need is on the Storage tab. If you want to script checks or drill into the specific items behind the numbers, the same data is available from the [storage usage API](#using-the-storage-usage-api).

This page is about *measuring* the footprint. Once you know where the space is going, [Storage efficiency](/corecapabilities/manage-data/storage-efficiency) covers what to change to reduce it — compression codecs, index choices, and segment sizing — and how to confirm a change actually helped.

> [!NOTE]
>
> The Storage tab and its API are available starting StarTree Cloud version 0.12.0.

## Reading the Storage tab

Open a table in the Data Portal and select the **Storage** tab — it is the default landing tab for every table, including External (catalog) tables. The tab header has a **Documentation** button linking to this page and a **Refresh** button; the data also auto-refreshes about every 30 seconds. A fourth sub-tab, **Segments**, sits alongside Local/Deep/Tiered Storage — see [Browsing segments](#browsing-segments) below.

> [!NOTE]
>
> The Storage tab and its API cover **External (catalog) tables too**. External Tables don't have true local or deep storage — their segments are auto-enrolled in an S3-backed [tier](/corecapabilities/manage-data/set-up-tiered-storage/motivation) instead, so their **Tiered Storage** card is the one with meaningful numbers; **Local Storage** shows only each server's preload-index/metadata-cache bytes (segments aren't kept on local disk), and **Deep Storage** stays near-empty. Sizing an External Table requires crawling its S3 prefix, parallelized by key range so it fits the metadata fan-out budget — on older releases this call was skipped for External Tables and their size was left unset.

Storage is grouped into up to three cards, one per physical location:

| Card | What it holds |
| :- | :- |
| **Local Storage** | Segment data on the Pinot servers' local disks — the copy that serves queries with the lowest latency. |
| **Deep Storage** | The durable backup copy of every segment in external object storage (S3, GCS, HDFS), used for recovery and replication. |
| **Tiered Storage** | Cost-optimized tiers holding colder data in remote object storage. Shown only when the table uses [tiered storage](/corecapabilities/manage-data/set-up-tiered-storage/motivation) — every External Table is auto-enrolled in an S3-backed tier, so this card is always present for them. |

**Local Storage**

![Local Storage tab in the Data Portal](https://mintcdn.com/startree/98YChggxv7Uq_-AO/images/localstorage.png?fit=max\&auto=format\&n=98YChggxv7Uq_-AO\&q=85\&s=cd7ed300e45b3272c9a58f0175070314)

**Deep Storage**

![Deep Storage tab in the Data Portal](https://mintcdn.com/startree/98YChggxv7Uq_-AO/images/deepstorage.png?fit=max\&auto=format\&n=98YChggxv7Uq_-AO\&q=85\&s=248d8e3737977770e63fe8041b242924)

**Tiered Storage**

![Tiered Storage tab in the Data Portal](https://mintcdn.com/startree/98YChggxv7Uq_-AO/images/tieredstorage.png?fit=max\&auto=format\&n=98YChggxv7Uq_-AO\&q=85\&s=7b25720149f118f88c723ee65d95f9ac)

### What each number means

Every card shows the same set of figures:

| Number | What it tells you | Derived from API fields |
| :- | :- | :- |
| **Used** | The **live** storage at this location — physical bytes minus the orphaned bytes, across all replicas. *Used + Orphaned = total physical footprint.* On the **Local Storage** card, when the table's replication factor is greater than 1, **Used** also shows a **per replica** figure (Used ÷ replication factor) alongside the total. | `actualSizeInBytes − unexpectedSizeInBytes` |
| **Orphaned** | Storage not backed by the table's current data, and possibly reclaimable. It bundles three things — unknown files, stale versions, and beyond-retention deleted segments (see [What "Orphaned" includes](#what-orphaned-includes)). Hover the stat for a per-category breakdown. | `unexpectedSizeInBytes` |
| **Live segments** | Number of live segments/items at this location (physical items minus orphaned items). On the Deleted Segments card this is labeled **Retained segments**, and on the Upsert/Dedupe cards **Snapshots**. | `actualItemsCount − unexpectedItemsCount` |
| **Orphaned segments** | How many individual segments or files are orphaned — unknown items plus beyond-retention. Stale versions add to *Orphaned* size but are **not** counted here. | `unexpectedItemsCount` |
| **Avg Segment Size** | Average size of the table's segments, shown per server. A large gap between this and your p90/p99 sizes points to segment-size skew. | `avgSegmentSizeInBytes` |

> [!NOTE]
>
> Because **Used** counts only live storage, cleaning up orphaned data shrinks the **Orphaned** figure while **Used** stays the same. Exact counts are shown in the stat tooltips (displayed values are compacted).

### Deep Storage breakdown

The **Deep Storage** card expands into per-area cards so you can see which part of the deep store is using space:

| Card | What it is |
| :- | :- |
| **Online Segments** | Live segments that belong to the table. |
| **Deleted Segments** | Segments in the deep store's retention queue after deletion. Within retention they are expected; past retention they count as orphaned. |
| **Upsert Snapshot** | Snapshot files for upsert-enabled tables. Never counted as orphaned. |
| **Dedupe Snapshot** | Snapshot files for dedup-enabled tables. Never counted as orphaned. |

The Deleted, Upsert, and Dedupe cards appear only when they hold data.

The **Local Storage** sub-tab also lists every server under a **Servers (N)** header, with a search box, a **Sort by** selector (Name, Used storage, Orphaned storage, Live segments, Orphaned segments, Avg segment size), and an ascending/descending toggle — useful for finding the server that holds the orphaned bytes on a large cluster.

### Segment-size health (Local Storage)

The **Local Storage** tab also shows two segment-size percentiles — **p50** (the median) and **p90** — plus a **Health Score** from 0 to 100. The Health Score stat links back to this section.

Think of the Health Score as a **report card for how well your segments are sized**. It has nothing to do with orphaned storage — it looks only at segment sizes. It starts at 100 and subtracts points for two problems:

* **Skew** — segments of very uneven size, i.e. a big gap between p50 and p90.
* **Oversized segments** — a p90 that is large in absolute terms, measured against a \~500 MB reference.

```
skew penalty  =  min( (p90 / p50 − 1) × 20 , 40 )     # 0 when segments are uniform
size penalty  =  min( (p90 / 500 MB)  × 20 , 40 )      # grows as segments get larger

Health Score  =  100 − skew penalty − size penalty     # rounded, clamped to 0–100
```

Roughly how your numbers map to a score:

| p50 | p90 | p90 / p50 | Health Score | Reading |
| :- | :- | :- | :- | :- |
| 100 MB | 120 MB | 1.2× | \~91 | Healthy — uniform and right-sized |
| 100 MB | 200 MB | 2× | \~72 | Some skew |
| 100 MB | 500 MB | 5× | \~40 | Noticeably skewed |
| 200 MB | 1.2 GB | 6× | 20 | Skewed and oversized |

A low score points to segment-sizing work: if segments are small and uneven, [merge small segments](/recipes/merge-small-segments); if they are too large, revisit the [segment threshold](/recipes/configuring-segment-threshold). A high score means your segments are consistently sized in a healthy range.

> [!NOTE]
>
> The Health Score is a **heuristic computed by the Data Portal**, not a value the API returns — the 500 MB reference and the point weights are guidance, not a hard SLA. To reproduce or alert on it, recompute it from the `p50` / `p90` figures the API returns under `serverAggregateSegmentSizeStats`. It shows **N/A** until both percentiles are available.

### Browsing segments

The **Segments** sub-tab lists every segment in the table individually — a grid with **Segment Name**, **Status** (Good / Bad / Updating / Unknown, with a summary chip per status), **Servers**, **Size per replica**, **Size (all replicas)**, **Total docs**, and **Created At**, plus a size-distribution chart, a storage-concentration chart, and a per-server filter. Clicking a server on the **Local Storage** tab jumps here pre-filtered to that server.

Opening a segment (or its **View Metadata** action) shows a **data vs. index size** breakdown and a per-column **index matrix** — which index types (inverted, forward, min/max, etc.) each column carries and how many bytes each one uses — plus the top columns by size. Each segment also exposes **Reload**, **Reset**, and **Delete** actions from a row menu.

> [!NOTE]
>
> The Segments tab and its data-vs-index / index-matrix breakdown reflect what the underlying Pinot segment metadata reports; they are a Data Portal convenience view, not a separate API surface documented on this page.

### Reloading segments

The Segments sub-tab header carries a **Reload all** button alongside the per-segment **Reload** row action, and a warning banner appears on the Storage tab whenever existing segments are out of date with the table's current config or schema. All three routes — **Reload all**, the banner's **Estimate & Analyze**, and the per-segment action — open the same **Reload segments** dialog, which runs a dry-run estimate before anything changes:

* **Segments / copies** affected, sampled across up to 3 segments per server.
* **Rebuild time**, estimated from the sampled segments.
* **Disk**, the net size change across all copies.
* **Net index changes**, added/removed indexes per column.

Nothing runs until you confirm with **Reload now**. If sampling finds segments that would fail to reload, the dialog lists each failure (segment, server, message) and blocks the reload by default, though a **Reload anyway** option is still available. On a table with tiered storage, segments held remotely are counted but not sampled, so the figures are called out as lower bounds.

> [!NOTE]
>
> The dry-run estimate requires a StarTree Pinot build with the fix for the `/tmp`-filling issue on large tables, and Data Portal detects support for it automatically — no action is needed once the connected cluster is upgraded. Until then, **Reload all**, **Estimate & Analyze** (labeled **Reload segments** in this state), and the post-save prompt skip straight to a confirm-only dialog: it still waits for **Reload** before doing anything, but it no longer pre-samples segments for failures or reports segments/copies affected, rebuild time, disk impact, or index changes up front — a failure surfaces only if the reload itself is attempted and fails.

Once a reload is started, a progress strip tracks copies reloaded, elapsed and estimated-remaining time, and any failed server calls, and a **Reload history** drawer lists past reload jobs for the table with a status per job (**In Progress**, **Completed**, **Completed with Errors**, **Aborted**, **Record Expired**) — a running job can be aborted from there.

> [!NOTE]
>
> Saving a table config or schema change now surfaces this same dialog automatically: a **Configs updated successfully** prompt checks whether the existing segments are now stale and offers to estimate the impact or reload immediately. This replaces the previous flow, which left finishing the reload to Pinot Admin.

## Understanding the numbers

### Used, orphaned, and expected

Two ideas explain every figure on the tab:

* **Used** is the live, expected storage — what *should* be there.
* **Orphaned** is storage present at the location but not backed by the table's current metadata.

Together they make up the physical footprint:

```
total physical footprint = Used + Orphaned
```

(In the API, `Used = actualSizeInBytes − unexpectedSizeInBytes` and `Orphaned = unexpectedSizeInBytes`.)

Storage is *expected* when it is a live segment recorded in the table's metadata, or a deleted segment still inside its retention window. Everything else is orphaned.

### What "Orphaned" includes

The **Orphaned** figure is a single number, but it bundles three different situations, each with a different cause and a different fix:

| Category | What it is |
| :- | :- |
| **Unknown items** | Files on storage with **no** record in the table at all — also called *untracked* segments. Usually left by interrupted operations, failed uploads, or manual copies. |
| **Stale versions** | Older copies of a segment that is still current, kept alongside the latest version when version cleanup lags. |
| **Beyond-retention segments** | Deleted segments still in the deep store past their retention deadline. |

> [!NOTE]
>
> **"Orphaned" means the combined total, not one category.** The Storage tab shows a single number labeled **Orphaned** for all three of the above added together. This trips people up because "orphan segment" is also used loosely to mean just the first category — files with no metadata. Throughout this page, **Orphaned** always refers to the combined Data Portal total; the no-metadata files are called **unknown items** (the API's `unknownItems`). A large **Orphaned** figure can therefore be entirely stale versions or beyond-retention segments, with zero unknown items. **Hovering the Orphaned stat shows the per-category split** — stale versions + unknown items on the Local and Tiered cards, unknown items + beyond-retention segments on the Deep Storage card. The API calls this same combined total **unexpected**; to see the split with the individual item names, call it at `verbose=2` (see [Using the storage usage API](#using-the-storage-usage-api)).

> [!NOTE]
>
> Stale versions count toward the **Orphaned** size but **not** toward the **Orphaned segments** count. This is why a table can show a large **Orphaned** size with zero **Orphaned segments** — the excess is entirely stale versions.

### Why these numbers differ from the old Pinot table size

If you are used to the OSS Pinot **table size** API (`GET /tables/{tableName}/size`, shown as *Reported Size* and *Estimated Size* in the Pinot console), the numbers here will look different, and usually larger. That is expected: the two answer different questions.

| | Pinot `getTableSize` | Table Storage Usage (this page) |
| :- | :- | :- |
| **Question** | How big is the table *as served on Pinot servers*? | How many bytes is the table *physically storing, everywhere*? |
| **Where it looks** | Server local disk only — it polls each server for the on-disk size of the segments it hosts | Server local disk **and** deep store **and** every tiered/remote object store, each reported separately |
| **What counts** | Only segments in the table's ideal state (currently served) | Every object found at each location, including orphaned, stale, deleted-but-within-retention, and snapshot data |
| **Replicas** | `reportedSizeInBytes` sums all replicas; `reportedSizePerReplicaInBytes` is a single copy | Local Storage **Used** sums all replicas' on-disk bytes |
| **Unreachable servers** | Estimates the missing size from other replicas (`estimatedSizeInBytes` ≥ `reportedSizeInBytes`) | Lists the actual objects present at each location; no estimation |

**How `getTableSize` computes Reported Size.** For each served segment it polls every server that hosts a replica and reads that replica's on-disk size:

```
size(s, r) = on-disk bytes of replica r of segment s, as reported by its server

Reported Size              =  Σ segments s  Σ reporting replicas r   size(s, r)
Reported Size Per Replica  =  Σ segments s   max over replicas r   size(s, r)      (≈ one full copy)
Estimated Size             =  Reported Size  +  Σ segments s ( unreachable replicas of s  ×  largest replica of s )
```

In words: **Reported Size adds up every replica that actually responded** — roughly one copy's size × replication factor. Replicas on an unreachable server are left out of Reported Size and only approximated in *Estimated Size* (so `Estimated ≥ Reported`), a segment whose replicas all fail to report is flagged *missing* and contributes nothing, and nothing outside the Pinot servers is ever counted — no deep store, no tiered storage.

Mapping the old numbers:

* The old **Reported Size** maps most closely to **Local Storage → Used**: both sum the on-disk size of served segments across all replicas. Local Storage can differ slightly because it also includes the tiered-storage preload and metadata caches (orphaned local files are excluded from Used and reported under Orphaned instead).
* The old **Reported Size Per Replica** ≈ **Local Storage → Used ÷ replication factor**.
* **Deep Storage** and **Tiered Storage** have **no counterpart** in `getTableSize`. This is usually why the totals look much larger than before: the deep store keeps a durable copy of every segment, and tiered storage holds cold data that servers no longer keep fully on local disk. The old API never counted either.

> [!NOTE]
>
> The two views are complementary. Use `getTableSize` for "how much data is my query tier serving," and the Storage tab for "what is my total footprint, and how much of it is reclaimable." A higher total does not mean the table grew — it means you can now see storage that was always there but previously invisible to the table size API (deep store, tiered storage, and orphaned data).

## What to do when Orphaned is high

The Storage tab and the API **report** the Orphaned total but never delete it. What to do depends on which of the three [categories](#what-orphaned-includes) it is made of, so start by identifying the category rather than acting on the headline number.

> [!NOTE]
>
> The **Orphaned** number bundles all three categories — unknown items, stale versions, and beyond-retention segments. The stat tooltip on the Storage tab shows which category dominates; for the individual item names, call the API with `verbose=2` and inspect the `unexpected` object before taking any action.

### Recommended workflow

1. **Rule out transient growth first.** During and shortly after a rebalance, segment replacement, or heavy ingestion, storage legitimately exceeds the expected size because of overlapping placement and intermediate artifacts. Re-check after the operation settles, and across a couple of the controller's periodic cycles, before treating anything as a real orphan.
2. **Identify the exact items and their category** with `verbose=2`. The remedy is driven entirely by category.
3. **Apply the matching remedy:**

| Category (`unexpected.*`) | Typical cause | Remedy |
| :- | :- | :- |
| `beyondRetentionSegments` | Deleted-segment cleanup is not keeping up | The controller's retention job purges these automatically. A persistent non-zero count means retention is not running or is lagging — verify the deleted-segment retention configuration and that the controller retention task is active. |
| `staleVersions` | Re-indexed or reconfigured segments leaving older versions behind | Run [Deep Store Sync](/corecapabilities/manage-data/deep-store-sync) (the Alter Table Task in `reloadOnly` mode) and enable tiered-storage version cleanup so superseded versions age out. |
| `unknownItems` | Interrupted ingestion or rebalance, failed uploads, `.tmp` / `.tar.compressed` leftovers, or manually copied files | Transient artifacts on server local disk usually clear on the next segment reload or server restart. For confirmed, persistent items in the deep store or remote object store, contact StarTree support to reconcile them. |

> [!WARNING]
>
> Do not manually delete objects from the deep store or remote object store based on this list alone — a wrong deletion removes live data. Confirm an item is genuinely unreferenced, and for deep store or object-store orphans prefer StarTree support over manual deletion.

### Preventing recurrence

Most orphaned growth traces back to interrupted work. To keep it from returning:

* Avoid stopping ingestion, rebalance, or segment-refresh operations mid-run.
* Right-size batch ingestion so failed uploads are rare; for External Tables, aim for larger source files.
* Monitor the **Orphaned** figure so growth is caught while it is small, rather than after it accumulates.

If what is growing is **Used** rather than **Orphaned**, this is not a cleanup problem — the storage is holding real data. See [Storage efficiency](/corecapabilities/manage-data/storage-efficiency) for the levers that reduce a legitimate footprint, in order of payoff.

## Glossary

> **Unknown items**
>
> Segments or files physically present on storage with **no** matching entry in Pinot (ZooKeeper) metadata — also called *untracked* or *orphan* segments. In the API these are `unexpected.unknownItems`. In the Data Portal they contribute to both **Orphaned** (size) and **Orphaned segments** (count). Common causes: interrupted ingestion or rebalance, failed deep store uploads, and manually copied files.

> **Stale versions**
>
> Older copies of a segment that is still part of the table, kept alongside the version currently in use in tiered/remote storage or in the server's preload/metadata caches. A version string encodes the segment CRC and a hash of the table config and schema, so changing the schema or config, or re-indexing a segment, can produce a new expected version — but a segment holding only a **single** version on storage is never reported as stale, even when that version doesn't match the expected one: the version string alone can't tell a legitimate reused version (for example, a new index added without rebuilding the segment) apart from a real leftover, so the lone version is assumed to be the one in use. Only once **more than one** version is present for a segment are versions reported: any version matching the expected one is in use, and if none matches, the newest version (by CRC/config-hash recency) is assumed in use and the older ones are reported stale. Reported in `totalStaleVersions` and `unexpected.staleVersions`; their bytes count toward **Orphaned** but not toward **Orphaned segments**.

> **Beyond-retention segments**
>
> Deleted segments that remain in the deep store `Deleted_Segments` area past their retention deadline. The deadline comes from the `__RETENTION_UNTIL__` suffix on the file name, or, if absent, from the last-modified time plus the configured deleted-segment retention. Reported in `unexpected.beyondRetentionSegments`. A persistent non-zero count usually means the retention/cleanup job is not running.

> **Online segments**
>
> Live segments that belong to the table, stored in the deep store under `/`. These make up the bulk of *expected* deep store size.

> **Deleted segments**
>
> Segments moved to the deep store retention queue at `/Deleted_Segments/`. Those within retention are expected; those past retention are counted as beyond-retention (orphaned).

> **Upsert / dedup snapshots**
>
> Snapshot files written by upsert- or dedup-enabled tables, stored in the deep store. Always reported as expected (never counted as orphaned).

> **Preload index & metadata cache**
>
> Server-local caches used by tiered storage and External Tables, all part of the same persistent disk cache (see [Data and Index Caching](/corecapabilities/external-table/data-and-index-caching)). The **preload index** stages remote-tier segment indexes for low-latency serving; the **metadata cache** holds cached metadata for remote-tier segments; the **LFU mmap cache** holds admitted index buffers. These three are keyed by segment version, so all are checked for stale versions. The **deletion-vector Puffin cache** is also included in this location's size total, but it has no segment/version mapping, so it never contributes to stale-version or unknown-item counts.

> **Segment size stats (p50 / p90 / p99 / avg)**
>
> Percentile and average segment sizes computed over the table's expected segments. Returned per server (`segmentSizeStats`) and aggregated across all servers (`serverAggregateSegmentSizeStats`). A large gap between p50 and p90/p99 indicates size skew.

***

## Using the storage usage API

The Storage tab is powered by a controller endpoint. Most users never need to call it directly. Reach for the API when you want to script storage checks, feed a dashboard, or drill into the **specific** orphaned or stale items behind the aggregate numbers — the Data Portal shows the totals but not the individual item names.

The endpoint provides a consolidated view for a table across server instances, deep store, and the remote object store, aggregating results collected from each server.

```
GET /tables/{tableName}/storageUsage?verbose={0|1|2}
```

* `tableName` — table name **with type**, for example `myTable_OFFLINE` or `myTable_REALTIME`.
* `verbose` — verbosity level (integer, default `0`):

| Level | Contents | Use for |
| :- | :- | :- |
| `0` | Summary only: per-location totals and segment size stats. No `breakdown`. | Dashboards, alerting, routine monitoring. |
| `1` | Adds the per-sub-directory `breakdown`, with counts and sizes, but **without** the lists of individual unexpected items. | Locating which area is growing. **This is what the Data Portal Storage tab uses.** |
| `2` | Full breakdown **including** the `unexpected` item lists (specific segment/version names). | Investigating and cleaning up specific orphaned or stale items. |

> [!WARNING]
>
> Listing large deep store or remote object-store prefixes can be slow and may incur object-store request charges. Use `verbose=0` for routine monitoring and reserve `verbose=1`/`2` for investigation.

The size and count invariants below always hold (and are enforced by integration tests):

```
unexpectedSizeInBytes  =  unknownItems.sizeInBytes
                       +  staleVersions.sizeInBytes
                       +  beyondRetentionSegments.sizeInBytes

unexpectedItemsCount   =  unknownItems.count
                       +  beyondRetentionSegments.count        (staleVersions are NOT counted)

unexpectedSizeInBytes  ≤  actualSizeInBytes
```

Note that `staleVersions` appears in the **size** total but not in the item **count** — this is exactly why a location can report non-zero Orphaned bytes with zero Orphaned segments. Stale versions are counted separately in `totalStaleVersions`.

### Response: summary (`verbose=0`)

Per-location scalar totals plus segment size statistics. `breakdown` is omitted.

> **Example response**
>
> ```json theme={null}
> {
>   "tableName": "myTable_OFFLINE",
>   "serverInstances": {
>     "Server_pinot-server-0": {
>       "actualItemsCount": 1305,
>       "unexpectedItemsCount": 38,
>       "totalStaleVersions": 0,
>       "actualSizeInBytes": 1119418454,
>       "unexpectedSizeInBytes": 253932,
>       "segmentSizeStats": {
>         "segmentCount": 999,
>         "p50SegmentSizeInBytes": 1100000,
>         "p90SegmentSizeInBytes": 1200000,
>         "p99SegmentSizeInBytes": 1300000,
>         "avgSegmentSizeInBytes": 1150000
>       }
>     }
>   },
>   "serverAggregateSegmentSizeStats": {
>     "segmentCount": 999,
>     "p50SegmentSizeInBytes": 1100000,
>     "p90SegmentSizeInBytes": 1200000,
>     "p99SegmentSizeInBytes": 1300000,
>     "avgSegmentSizeInBytes": 1150000
>   },
>   "deepStore": {
>     "actualItemsCount": 2890,
>     "unexpectedItemsCount": 3,
>     "actualSizeInBytes": 21805648,
>     "unexpectedSizeInBytes": 14193
>   },
>   "remoteS3": {
>     "s3Tier": {
>       "actualItemsCount": 5,
>       "unexpectedItemsCount": 0,
>       "totalStaleVersions": 3,
>       "actualSizeInBytes": 1331507185,
>       "unexpectedSizeInBytes": 1006228697
>     }
>   }
> }
> ```

### Response: full breakdown (`verbose=2`)

`verbose=1` adds a `breakdown` for each area but omits the `unexpected` item lists. `verbose=2` additionally lists the individual unexpected contributors — unknown items, stale versions, and beyond-retention segments.

> **Example response**
>
> ```json theme={null}
> {
>   "tableName": "fake_table_SRT_Compaction_REALTIME",
>   "serverInstances": {
>     "Server_pinot-server-server-0-1": {
>       "actualItemsCount": 1305,
>       "unexpectedItemsCount": 38,
>       "totalStaleVersions": 0,
>       "actualSizeInBytes": 1119418454,
>       "unexpectedSizeInBytes": 253932,
>       "breakdown": {
>         "localSegmentsDirectory": {
>           "actualItemsCount": 1305,
>           "unexpectedItemsCount": 38,
>           "actualSizeInBytes": 1119418454,
>           "unexpectedSizeInBytes": 253932,
>           "unexpected": {
>             "unknownItems": {
>               "count": 38,
>               "sizeInBytes": 253932,
>               "items": [
>                 "fake_table_SRT_Compaction__0__14199__20251222T2339Z.tar.compressed",
>                 "..."
>               ]
>             }
>           }
>         }
>       }
>     }
>   },
>   "deepStore": {
>     "actualItemsCount": 2890,
>     "unexpectedItemsCount": 3,
>     "actualSizeInBytes": 21805648,
>     "unexpectedSizeInBytes": 14193,
>     "breakdown": {
>       "onlineSegments": {
>         "actualSizeInBytes": 19301473,
>         "unexpectedItemsCount": 2,
>         "unexpectedSizeInBytes": 14193,
>         "unexpected": {
>           "unknownItems": {
>             "count": 2,
>             "sizeInBytes": 14193,
>             "items": [
>               "fake_table_SRT_Compaction__1__162105__20260107T1819Z.tmp",
>               "fake_table_SRT_Compaction__0__144799__20260107T1819Z.tmp"
>             ]
>           }
>         }
>       },
>       "deletedSegments": {
>         "actualItemsCount": 1,
>         "unexpectedItemsCount": 1,
>         "actualSizeInBytes": 1201,
>         "unexpectedSizeInBytes": 1201,
>         "unexpected": {
>           "beyondRetentionSegments": {
>             "count": 1,
>             "sizeInBytes": 1201,
>             "items": [
>               "fake_table_SRT_Compaction__1__10702__20251222T1506Z"
>             ]
>           }
>         }
>       }
>     }
>   },
>   "remoteS3": {
>     "s3Tier": {
>       "actualItemsCount": 5,
>       "unexpectedItemsCount": 0,
>       "totalStaleVersions": 3,
>       "actualSizeInBytes": 1331507185,
>       "unexpectedSizeInBytes": 1006228697,
>       "breakdown": {
>         "segmentDirectory": {
>           "actualItemsCount": 5,
>           "unexpectedItemsCount": 0,
>           "totalStaleVersions": 3,
>           "actualSizeInBytes": 1331507185,
>           "unexpectedSizeInBytes": 1006228697,
>           "unexpected": {
>             "staleVersions": {
>               "count": 3,
>               "sizeInBytes": 1006228697,
>               "items": [
>                 "3769746207_-1890732847",
>                 "3811220169_-1890732847",
>                 "3118351592_-1890732847"
>               ]
>             }
>           }
>         }
>       }
>     }
>   }
> }
> ```

### Response schema

The response is hierarchical. It is defined below in three parts:

1. [Root response fields](#1-root-response-fields)
2. [The StorageStats object](#2-the-storagestats-object) (reused recursively)
3. [Breakdown definitions](#3-breakdown-definitions) (per-location keys)

#### 1. Root response fields

| Field | Type | Description |
| :- | :- | :- |
| `tableName` | `string` | Table name with type, for example `myTable_OFFLINE`. |
| `serverInstances` | `map` | Map of server instance ID (for example `Server_pinot-server-0`) to a [StorageStats object](#2-the-storagestats-object). Storage on the actual server nodes; powers the **Local Storage** card. |
| `serverAggregateSegmentSizeStats` | `object` | [Segment size stats](#segment-size-stats) computed over all expected segments across all servers. Present in every verbosity level, including `verbose=0`. |
| `deepStore` | `object` | A [StorageStats object](#2-the-storagestats-object) for the deep store (PinotFS/S3); powers the **Deep Storage** card. |
| `remoteS3` | `map` | Map of tier name to a [StorageStats object](#2-the-storagestats-object) for each tiered-storage object store; powers the **Tiered Storage** card. Absent when the table has no S3-backed tier. |

#### 2. The `StorageStats` object

This structure is used for `serverInstances`, `deepStore`, `remoteS3`, and every nested `breakdown` object.

| Field | Type | Description |
| :- | :- | :- |
| `actualItemsCount` | `integer` | Total count of unique items (segments/files) physically found at the location. |
| `actualSizeInBytes` | `long` | Total physical size, in bytes, of all items found. Shown as **Used**. |
| `unexpectedItemsCount` | `integer` | Count of items found that are **not** expected — unknown items plus beyond-retention segments. Excludes stale versions. The UI's **Orphaned segments** figure. |
| `unexpectedSizeInBytes` | `long` | Total size, in bytes, of all unexpected storage — unknown items **plus** stale versions **plus** beyond-retention segments. Shown as **Orphaned**. |
| `totalStaleVersions` | `integer` | *(Tiered/remote and server caches only)* Number of unique segment versions present that differ from the version expected from table config and segment CRC. |
| `segmentSizeStats` | `object` | *(Server instances only)* [Segment size stats](#segment-size-stats) for the expected segments on that server. |
| `breakdown` | `map` | Per-sub-directory breakdown. See [Breakdown definitions](#3-breakdown-definitions). Present only when `verbose ≥ 1`. |
| `unexpected` | `object` | Inside a breakdown entry: itemized unexpected contributors. Contains `unknownItems`, `staleVersions`, and/or `beyondRetentionSegments`. Item lists are present only when `verbose=2`. |
| `error` | `string` | *(`remoteS3` tiers only)* Present only when that tier's crawl failed — a failed tier still returns all-zero counts, so check this field before reading them as an empty tier. Absent (not `null`) on a successful crawl. |

> [!NOTE]
>
> A remote-tier crawl can fail on a very large External Table (millions of objects under the tier prefix) — the controller now reports the tier as failed via `error` instead of silently returning zeros for it. The Data Portal's Tiered Storage card does not yet distinguish this from a genuinely empty tier; use the API directly (or retry) if a tier unexpectedly shows all zeros.

##### Segment size stats

Returned as `serverAggregateSegmentSizeStats` at the root and as `segmentSizeStats` per server instance. Computed over the table's expected segments.

| Field | Type | Description |
| :- | :- | :- |
| `segmentCount` | `integer` | Number of segments included in the statistics. |
| `p50SegmentSizeInBytes` | `long` | 50th percentile (median) segment size. |
| `p90SegmentSizeInBytes` | `long` | 90th percentile segment size. |
| `p99SegmentSizeInBytes` | `long` | 99th percentile segment size. |
| `avgSegmentSizeInBytes` | `long` | Average segment size. Shown as **Avg Segment Size**. |

##### The `unexpected` object

| Key | Description |
| :- | :- |
| `unknownItems` | Items on storage with no entry in Pinot metadata (untracked segments, temp files). |
| `staleVersions` | Older versions of known segments in tiered/remote storage or local caches. |
| `beyondRetentionSegments` | Deleted segments past their retention deadline in the deep store. |

Each entry is an object with `count` (integer), `sizeInBytes` (long), and `items` (array of segment/version names). The `items` array is populated only at `verbose=2`.

#### 3. Breakdown definitions

The keys inside a `breakdown` object depend on the location.

**Server instance** (`serverInstances`):

| Key | Description |
| :- | :- |
| `localSegmentsDirectory` | Segment index directories for locally stored segments, on the server's local data directory. |
| `remoteSegmentsPreloadIndex` | Local preload-index cache for a tiered table's remote segments, when preload is configured. |
| `remoteSegmentsMetadataCache` | Local metadata cache for a tiered table's remote segments, when configured. |

**Deep store** (`deepStore`):

| Key | Description |
| :- | :- |
| `onlineSegments` | Live segments under `/`. |
| `deletedSegments` | Segments in the retention queue at `/Deleted_Segments/`. |
| `upsertSnapshots` | Snapshot files for upsert-enabled tables. |
| `dedupSnapshots` | Snapshot files for dedup-enabled tables. |

**Remote / tiered store** (`remoteS3`):

| Key | Description |
| :- | :- |
| `segmentDirectory` | Objects under the configured prefix for the tier, `//`. |

## Replication considerations

Replication increases Local Storage usage, because segment directories exist per replica. Deep store and remote object-store sizes are reported independently of replica placement.

## Limitations

During cluster transitions such as rebalances or segment replacement, **Used** can temporarily exceed the expected size because of overlapping placement and intermediate artifacts. Re-check after the operation completes before treating the excess as reclaimable.


# Compaction after Upserts
Source: https://docs.startree.ai/corecapabilities/manage-data/upsert-compaction-srt



> [!NOTE]
>
> **Known limitations when running Segment Refresh Task on upsert tables:**
>
> * **File Ingestion Task (FIT) with Segment Refresh Task is currently unsupported** — `metadataTTL` is not supported with SRT.
> * **`deletedKeysTTL` needs an extra config** in `upsertConfig` to work correctly with SRT — see [A few things to know specifically about `deletedKeysTTL`](/corecapabilities/manage-data/offheap-upsert#metadata-ttl-and-deletedkeys-ttl).

After enabling upserts, steps must be taken to optimize storage and maintain performance. Old and invalid documents are to be physically removed from the system rather than just being logically hidden. Merging small segments into larger ones reduces the number of segments scanned by queries, improving response times. Where applicable, rolling up records by time windows or other dimensions further reduces data volume and increases efficiency. Segments that no longer contain valid documents should also be cleaned up, since retaining them adds overhead without benefit. Together, these actions keep upsert-enabled tables performant and storage-efficient.

## How does StarTree Cloud solve it?

To simplify this process and reduce the burden on operators, **StarTree Cloud** provides the **Segment Refresh Task**. Instead of requiring manual compaction and cleanup, this task automates the entire process of segment replacement. It takes a set of input segments, processes them to generate new segments by merging, transforming, or rolling up the data, and then atomically replaces the old segments with the new ones. With this mechanism, queries always see a consistent dataset, even when replacement is in progress. The Segment Refresh Task, therefore, offers a better, more reliable way to manage upserts in Pinot, ensuring correctness while optimizing storage and performance.

## What is Segment Refresh Task and How It Works?

The Segment Refresh Task runs as a background minion task within **StarTree Cloud** and is designed to improve both storage and query efficiency.
For example, imagine a table with three segments: `S1`, `S2`, and `S3`. Over time, updates have invalidated many rows across these segments. When the Segment Refresh Task is executed, it collects `S1`, `S2`, and `S3` as input and creates a new merged segment, say `S3_refreshed`.

This new segment is immediately made visible to queries, while the old segments with no valid documents are removed eventually. The result is that queries only need to scan the refreshed segment, which reduces latency and saves storage space. Importantly, this process guarantees **atomicity**: queries never encounter a partial replacement where some data is missing or duplicated.
Read more about [Segment Refresh Task](https://docs.startree.ai/corecapabilities/manage-data/segment-refresh-task)

## How StarTree Segment Refresh Task competes with Apache Pinot Upsert Compaction Task?

### Segment Refresh Task automatically compacts and merges segments while rebuilding indexes, ensuring new data stays optimized.

Unlike the Upsert Compaction Task, which only handled basic compaction, the Segment Refresh Task goes further by merging small, fragmented segments into larger ones, dropping invalidated records, and rebuilding indexes in the same process. This not only reduces storage overhead but also ensures that queries benefit from fully optimized index structures on refreshed data, leading to consistently faster response times even as datasets grow.

### Segments are replaced atomically across servers, so queries continue to run smoothly while Minion rolls out the refreshed segments in the background.

For upsert tables, this atomicity is achieved by leveraging bitmaps instead of just lineage-based routing flips. New segments are made visible to queries immediately, but their bitmaps start empty and fill as updates are applied, while old segments gradually drain to empty. This guarantees that queries never see partial data or miss valid documents. Because replacement happens atomically across all servers, query consistency is preserved end-to-end, even while Minion is refreshing and swapping segments in the background.

## Things to Keep an Eye On While Configuring the Segment Refresh Task

While the Segment Refresh Task is powerful, careful configuration is key to getting the best results. Running the task too frequently can put unnecessary strain on the system, as it may end up processing nearly all segments repeatedly. On the other hand, running it too infrequently can cause storage to grow and query performance to degrade.

It is important to configure thresholds that determine when segments should be refreshed, ensuring that only segments with a significant number of invalid documents are selected. Another consideration is the **buffer period**, which allows the system to skip the most recent segments that are still ingesting new records. This avoids conflicts between ingestion and refresh.

Segment naming conventions also matter, as consistently naming refreshed segments makes tie-breaking deterministic when conflicts arise.

## Example: Configuring Segment Refresh Task for Upsert Table

You can configure the Segment Refresh Task as shown below. Include this in your table config:

```json theme={null}
"tasks": {
  "SegmentRefreshTask": {
    "schedule": "0 0 2 * * ?",       // Cron schedule (here: run daily at 2 AM)
    "bufferTimePeriod": "1h",      // Skip segments committed in the last 1 hour
    "invalidRecordsThresholdPercent": 20,   // Refresh only if more than 20% records are invalid
    "desiredSegmentSize": "500MB", // Target size of  output segment
    "tableMaxNumTasks": "100"
   }
}
```

### Explanation of Key Parameters

* **schedule**: Defines when the task should run. In the above example, it runs every day at 2 AM.
* **bufferTimePeriod**: Ensures the most recent segments are skipped, so ingestion is not disrupted. (This is required because the EndOffset of the latest committed segment is used to resume ingestion and hence this segment shouldn’t be merged)
* **invalidRecordsThresholdPercent**: Only triggers a refresh if a large enough portion of the segment’s records have been invalidated by upserts.
* **maxNumRecordsPerSegment**: Prevents the creation of overly large output segments by setting a record count limit.
* **desiredSegmentSize**: Defines the desired output segment size. Larger segments improve query performance by reducing segment lookups, but may increase scan time. The default size is 500 MB.
* **tableMaxNumTasks**: Max number of sub tasks a table can run at each schedule. Refer best practices for details on how to configure this

## Best Practices for Running Segment Refresh Task (SRT) on Upsert Tables

Use SRT scheduling and configuration as part of your table lifecycle management strategy.
Balancing frequency, task limits, and cleanup intervals ensures stable performance and predictable cluster behavior over time.

### 1. Schedule SRT and Upsert Snapshot Tasks Carefully

Ensure that the **Upsert Snapshot Task** is scheduled **after** the **SRT task** completes.
This ensures that snapshots are created using the latest segments produced by SRT, avoiding inconsistencies between snapshot data and segment versions.

***

### 2. Tune SRT Frequency and Parallelism

Running SRT efficiently requires balancing **Minion task cost**, **query performance**, and **cluster stability**.
Use the following two-step approach to optimize configuration.

#### a. Optimize SRT Frequency

Understand the trade-off between **minion cost** (to run SRT) and **query performance** (affected by segment growth).

* **Recommended setup:** Schedule SRT based on the average rate of data updates.
  * Example: If data updates every 12 hours, run SRT every 12 hours.
* Adjust SRT frequency up or down to balance Minion compute cost with acceptable query latency.

#### b. Set an Appropriate `tableMaxNumTasks`

Avoid spawning too many subtasks, which can degrade **controller performance**, cause **Helix state transition delays**, and extend **segment replacement times** — potentially impacting availability.

Use the following guideline to determine a safe `tableMaxNumTasks` value:

| Symbol | Description | Example |
| - | - | - |
| **X** | Time to replace a segment in the server (approx. 30 seconds) | 30 |
| **Y** | Parallelism (25% of total CPU cores) | — |
| **Z** | `pinot.broker.new.segment.expiration.seconds` (default: 3600 seconds) | 3600 |
| **A** | Number of servers where the table is hosted | — |
| **B** | Table replication factor | — |

**Recommended formula:**

```text theme={null}
tableMaxNumTasks = (Z * Y * A / X / B) * 0.75

Note: The 0.75 multiplier adds a buffer for random variance in timing.
```

If tableMaxNumTasks is insufficient to complete SRT in a single run, **increase SRT frequency** to trigger multiple runs within the desired schedule determined by step (a).

#### c. First-Time SRT Execution

If running SRT for the first time on an existing table (with many old segments), temporarily increase
pinot.broker.new\.segment.expiration.seconds to **\~12 hours**.

This prevents throttling and server-side congestion, since the initial SRT run may pick up a large number of old segments from the same partition.

***

### 3. Manage Segment Cleanup Between SRT Runs

To maintain data freshness and prevent segment buildup, ensure the table commits **one segment per partition** between consecutive SRT runs.
This helps cleanup the older segments and ensures subsequent SRT runs execute efficiently.


# Accessing logs
Source: https://docs.startree.ai/corecapabilities/observability/access-logs

Search your cluster's component logs in Grafana with LogQL, pull raw log files from a specific Pinot instance over the API, and temporarily raise a log level to reproduce a problem.

Metrics tell you that something happened; logs tell you what it said. StarTree Cloud gives you three ways in, and they answer different questions:

| Surface | Best for |
| - | - |
| **Grafana Logging dashboards** | Searching across components and time — "which pod logged this error, and when did it start?" |
| **Pinot `/loggers` API** | Pulling the raw log file from one named instance, unparsed and complete |
| **Data Portal task logs** | Logs for one minion task or subtask, next to its progress |

Start with Grafana. It is indexed, searchable across every component at once, and does not require you to know which pod to look at.

## Searching logs in Grafana

Open [Grafana](/corecapabilities/observability/grafana) and go to the **Logging** dashboards. They search the log lines your cluster's components have written, indexed so you can filter and count rather than scroll.

### The dashboards

| Dashboard | What it gives you |
| - | - |
| **Logging / Common (historical log search)** | The general-purpose one. Log volume over time, level distribution, per-pod breakdown, an errors-only panel, and a free-text search box. Filter by log group, application, pod, and level. |
| **Logging / Application Logs** | Scoped to application logs, with a timeline and an error/warning summary. |
| **Logging / Query Stats** | Query log lines with latency, rows returned, and the query text. Includes slowest-query and most-frequent-query panels. |
| **Logging / Slow Queries Drilldown** | Queries above a latency threshold you set, with per-table maximum latency and drill-down by request ID. |
| **Logging / Table Queries** | The same query view, scoped to one table. |
| **Logging / Kubernetes Events** | Container OOM kills, failed liveness checks, evictions, and warnings. This is where you confirm a component was killed rather than crashed. |
| **Logging / Tool & K8s Cluster Logs** | Platform and tooling component logs. |

### Filtering

Each Logging dashboard has selectors at the top, and they map onto the labels every log line carries:

| Selector / label | What it narrows to |
| - | - |
| `log_group` | The broad category of log — `applications` for your Pinot components, `query_stats` for query log lines, `kubernetes_events` for pod-level events such as out-of-memory kills |
| `app_name` | The component that wrote the line |
| `pod_name` | A single instance of that component |
| `log_level` | `INFO`, `WARN`, `ERROR`, and so on |

Start broad and narrow down. `app_name` plus `log_level="ERROR"` over a wide time range is usually enough to find the window worth reading in detail, and `pod_name` then tells you whether the problem is one replica or all of them.

### Writing your own queries

The dashboards are built from LogQL you can adapt. In Grafana, choose **Explore** with the Loki data source.

Find every error from one application in the current time range:

```logql theme={null}
{log_group="applications", app_name="pinot-server", log_level="ERROR"}
```

Search for a string across a component's logs, regardless of level:

```logql theme={null}
{log_group="applications", app_name="pinot-server"} |= "OutOfMemoryError"
```

Count errors per pod, to see whether a problem is one replica or all of them:

```logql theme={null}
sum by (pod_name) (
  count_over_time({log_group="applications", app_name="pinot-server", log_level="ERROR"}[$__range])
)
```

Query-stats lines expose their fields individually, so you can filter on latency numerically:

```logql theme={null}
{log_group="query_stats"} | json | qs_timeMs > 5000
```

Query-stats lines expose `qs_requestId`, `qs_table`, `qs_timeMs`, `qs_docs`, `qs_entries`, and `qs_query` — the same values shown as columns on the query dashboards. `qs_requestId` is the one to carry between dashboards when drilling into a single slow query.

> [!NOTE]
>
> For anything beyond ad-hoc searching of query behaviour, prefer [`system_query_log`](/corecapabilities/query_data/advanced_operations/query-logger) over the `query_stats` log group. It is a queryable Pinot table with a much richer set of columns, and you can join and aggregate it with SQL. The log view is the fallback when the query logger is not enabled on your cluster.

### Retention

Logs are retained for a bounded window, set for your environment — days rather than months. Confirm the exact retention with your StarTree team if you need to rely on it. Beyond that window the lines are gone, so for a problem you may need to explain later, export what you need while it is still in range: use **Inspect** → **Data** on a log panel, or copy the lines out of the log view.

This matters most for intermittent problems. If something fires monthly, do not plan to go back and look at the previous occurrence — capture each one as it happens.

## Pulling raw log files from an instance

When you need the unparsed file from one specific component — a full stack trace, a startup sequence, everything around a single second — use the Pinot controller's logger API. This is also the route when you need to hand a complete log file to support.

All calls go to your controller host and use the same authentication as the rest of the Pinot API; see [Manage API tokens](/corecapabilities/security/manage-api-tokens).

**1. List the instances that have logs**

```
GET /loggers/instances
```

```bash theme={null}
curl -X GET "https:///loggers/instances" \
  -H "Authorization: Bearer "
```

See [Collect log files from all the instances](/api-reference/logger/collect-log-files-from-all-the-instances).

**2. List the log files on one instance**

```
GET /loggers/instances/{instanceName}
```

Use the instance name exactly as it appears in the previous response — `Server_pinot-server-0.…_8098`, for example. See [Collect log files from a given instance](/api-reference/logger/collect-log-files-from-a-given-instance).

For the controller you are already talking to, `GET /loggers/files` lists its local files directly ([reference](/api-reference/logger/get-all-local-log-files)).

**3. Download the file**

```
GET /loggers/instances/{instanceName}/download?filePath=
```

```bash theme={null}
curl -X GET "https:///loggers/instances//download?filePath=" \
  -H "Authorization: Bearer " -o component.log
```

See [Download a log file from a given instance](/api-reference/logger/download-a-log-file-from-a-given-instance), or [Download a log file](/api-reference/logger/download-a-log-file) for the local-controller equivalent.

> [!WARNING]
>
> Log files are rotated. A file that existed when you listed it may be gone minutes later, and the file you want may already have rotated out — which is the main reason to search in Grafana first and reach for raw files only when you know what you are after.

## Raising a log level to reproduce something

If a problem is reproducible but the logs are not saying enough, you can raise a logger's level at runtime without a restart.

**1. Find the logger**

```
GET /loggers
```

Returns every logger name currently configured. See [Get all the loggers](/api-reference/logger/get-all-the-loggers). `GET /loggers/{loggerName}` returns the level of one ([reference](/api-reference/logger/get-logger-configs)).

**2. Raise the level**

```
PUT /loggers/{loggerName}?level=DEBUG
```

See [Set logger level](/api-reference/logger/set-logger-level).

**3. Reproduce, collect, and put it back**

Reproduce the problem, collect what you need from Grafana or the file API, then set the level back to `INFO`.

> [!WARNING]
>
> **Always restore the level.** `DEBUG` on a busy component produces a very large volume of log lines, which consumes log retention for every other component sharing the environment and can shorten how far back you are able to search. Change the narrowest logger that covers the code you care about rather than the root logger, and change it back as soon as you have what you need.

## Task and subtask logs

For minion work, the Data Portal shows logs in context rather than making you search for them. Open a task plan and use its **Logs** tab, or open an individual subtask for its own log view alongside its progress. See [Task observability](/corecapabilities/observability/minion-task-runs).

## What is not available to you

Some logs exist but are not exposed:

* **Logs from the managed platform itself.** The services StarTree runs to operate and manage your environment are not exposed here.
* **Cloud provider infrastructure logs.** Network and cloud-API audit logs for StarTree-managed infrastructure are not exposed through the Data Portal. In a BYOC or BYOK deployment the data plane runs in your own cloud account, so those logs are in your account and under your control.
* **Anything older than retention.** Once the window has passed, the data is deleted rather than archived.

If you have worked through the Logging dashboards and the file API and the answer is not there, that is itself a useful result to report — say so in the ticket, with the queries you ran and the window you searched, so support does not repeat the same search.

## Related

* [Grafana dashboards](/corecapabilities/observability/grafana) — where the Logging dashboards live
* [Query Logger](/corecapabilities/query_data/advanced_operations/query-logger) — `system_query_log`, the structured alternative for query analysis
* [Pinot audit logs](/corecapabilities/security/audit/pinot-audit-logs) — who did what to the cluster, as opposed to what it logged
* [Task observability](/corecapabilities/observability/minion-task-runs) — logs for minion tasks
* [Troubleshooting: start here](/corecapabilities/observability/troubleshooting)


# Batch ingestion and minion task alerts
Source: https://docs.startree.ai/corecapabilities/observability/alerts/batch-and-minion-tasks

Alerts that watch file, SQL, segment import, and external table sync tasks and the minions that run them: when each fires, what it means, and what to do.

File, SQL, segment import and external table sync tasks, and the minion workers that run them.

> [!NOTE]
>
> **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the [overview](/corecapabilities/observability/alerts/overview) for how to read an entry.

## Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

```mermaid placement="top-right" actions={true} theme={null}
flowchart TD
  A["Batch data didn't land"] --> B{"Was a task generated?"}
  B -- "No, generation failed" --> B1["FailedToGenerate*Task
MinionTaskGenerationFailures
credentials, input path, file names"]
  B -- "No, skipped" --> B2["MinionTaskGenerationSkipped
previous run still active: space out the schedule"]
  B -- "Yes" --> C{"GET /tasks/{type}/{table_TYPE}/debug
Subtask states?"}
  C -- "ERROR" --> C1["*SubtasksInError
read the subtask error"]
  C -- "Dropped" --> C2["MinionSubtasksDroppedForTable
re-run the task"]
  C -- "Waiting for hours" --> C3["MinionSubTaskHighWaitTime
minion capacity, overlapping schedules"]
  C -- "Running for hours" --> C4["MinionSubTaskHighRunningTime
very large inputs"]
  C -- "Push rejected" --> C5["MinionConsistentPushFailures
raise quota.storage"]
```

## Critical alerts

### `FailedToGenerateFileIngestionTask`

> [!TIP]
>
> Usually your fix: the cause is most often in your stream, source system, or table config.

| Field | Detail |
| - | - |
| **Fires when** | File ingestion task generation failed in the last 30 min. |
| **What it means** | No new files are loaded for that table. |
| **Check first** | Task status message in the Data Portal |
| **What you can do** | Fix credentials, input path or non-ASCII file names |

### `FailedToGenerateSqlIngestionTask`

> [!TIP]
>
> Usually your fix: the cause is most often in your stream, source system, or table config.

| Field | Detail |
| - | - |
| **Fires when** | SQL ingestion task generation failed in the last 30 min. |
| **What it means** | No new rows are pulled from the source database. |
| **Check first** | Task status message in the Data Portal |
| **What you can do** | Fix source credentials or the query |

### `FailedToGenerateSegmentImportTask`

| Field | Detail |
| - | - |
| **Fires when** | Segment import task generation failed in the last 30 min. |
| **What it means** | Real-time data isn't moved to the offline table. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Fixes the task configuration |

### `FailedToGenerateExternalTableSyncTask`

> [!TIP]
>
> Usually your fix: the cause is most often in your stream, source system, or table config.

| Field | Detail |
| - | - |
| **Fires when** | External table sync generation failed in the last 30 min. |
| **What it means** | The table stops picking up new snapshots. |
| **Check first** | External table status in the Data Portal |
| **What you can do** | Fix catalog credentials or snapshot retention |
| **See also** | [External table observability: metrics and alerting](/corecapabilities/external-table/observability#metrics-and-alerting) [External table troubleshooting](/corecapabilities/external-table/troubleshooting) |

### `MinionTaskGenerationFailures`

| Field | Detail |
| - | - |
| **Fires when** | Generation failed in the last hour and hasn't succeeded for 6 hours. |
| **What it means** | Scheduled work is stuck. |
| **Check first** | Pinot Minion Tasks → **Time Since Last Successful Task Generation** |
| **What you can do** | Fix the config or input named in the error |
| **What StarTree does** | Clears locks or resource limits |

### `MinionTaskGenerationSkipped`

| Field | Detail |
| - | - |
| **Fires when** | Generation was skipped because a previous run is still active. |
| **What it means** | Runs are overlapping. |
| **Check first** | Pinot Minion Tasks → **Cron Scheduler Job Skipped Count** |
| **What you can do** | Space out the task schedule |
| **See also** | [Minion task orchestration: FAQs](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration#faqs) |

### `MinionSubtasksInErrorForTableAndTaskType`

| Field | Detail |
| - | - |
| **Fires when** | Over 50% of a table's subtasks are in ERROR, for 20 min. |
| **What it means** | That table's task is failing. |
| **Check first** | `GET /tasks/{taskType}/{tableNameWithType}/debug`, with the typed table name such as `orders_OFFLINE` |
| **What you can do** | Fix source or config errors in the subtask message |
| **What StarTree does** | Fixes minion-side failures |

### `FileIngestionSubtasksInErrorForTable`

| Field | Detail |
| - | - |
| **Fires when** | Over 90% of a table's file ingestion subtasks fail, for 20 min. |
| **What it means** | Files aren't loading. |
| **What you can do** | Check the files and credentials |

### `FileIngestionSubtasksInError`

| Field | Detail |
| - | - |
| **Fires when** | Over 90% of file ingestion subtasks fail cluster-wide, for 60 min. |
| **What it means** | File ingestion is failing broadly. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Investigates a shared cause |

### `SqlIngestionSubtasksInError`

| Field | Detail |
| - | - |
| **Fires when** | Over 90% of SQL ingestion subtasks fail, for 60 min. |
| **What it means** | SQL ingestion is failing broadly. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Investigates a shared cause |

### `SegmentImportSubtasksInError`

| Field | Detail |
| - | - |
| **Fires when** | Over 90% of segment import subtasks fail, for 60 min. |
| **What it means** | Segment import is failing broadly. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Investigates a shared cause |

### `MinionSubtasksDroppedForTable`

| Field | Detail |
| - | - |
| **Fires when** | More than 10 subtasks dropped, for 60 min. |
| **What it means** | A minion went away mid-task; dropped work isn't retried. |
| **What you can do** | Re-run the task |
| **What StarTree does** | Checks minion stability |

### `MinionLongWaitingSubtasksForTaskType`

| Field | Detail |
| - | - |
| **Fires when** | Waiting subtasks haven't decreased in 20 min and minions aren't picking any up, for 60 min. |
| **What it means** | Task execution has stalled. |
| **Check first** | Pinot Minion Tasks → **Total Number of SubTasks Waiting** |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Restarts or scales minions |
| **See also** | [Minion task orchestration: FAQs](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration#faqs) |

### `MinionSubTaskHighWaitTime`

> [!NOTE]
>
> Often transient: this alert usually clears on its own.

| Field | Detail |
| - | - |
| **Fires when** | A subtask has waited over 4 hours. |
| **What it means** | Not enough minion capacity, or schedules collide. |
| **Check first** | Pinot Minion Tasks → **Sub Task Queueing Time P95** |
| **What you can do** | Stagger task schedules |
| **What StarTree does** | Scales minions |

### `MinionSubTaskHighRunningTime`

> [!NOTE]
>
> Often transient: this alert usually clears on its own.

| Field | Detail |
| - | - |
| **Fires when** | A subtask has run over 3 hours. |
| **What it means** | Usually very large inputs. |
| **Check first** | Pinot Minion Tasks → **Sub Task Execution Time P95** |
| **What you can do** | Split large inputs |

### `MinionConsistentPushFailures`

| Field | Detail |
| - | - |
| **Fires when** | A consistent push failed in the last hour. |
| **What it means** | Usually the table's storage quota rejected the upload. |
| **What you can do** | Raise `quota.storage` |

### `MinionsUnderUtilized`

| Field | Detail |
| - | - |
| **Fires when** | Fewer than 20% of minions run tasks, for 12 hours. |
| **What it means** | Autoscaling may be stuck high. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Fixes autoscaling |

### `OfflineSegmentDelayHoursCrossedThreshold`

| Field | Detail |
| - | - |
| **Fires when** | The offline table is 7+ days behind while segment import is active. |
| **What it means** | Offline data is stale. |
| **What you can do** | Run segment import more often |
| **See also** | [Managed offline flow: troubleshooting](/recipes/managed-offline-flow#troubleshooting) |

## Warning alerts

| Alert | Fires when | What it means | What you can do |
| - | - | - | - |
| `MinionSubtaskQueueSizeHigh` | More than 20 waiting subtasks per running minion, for 6 hours. | A backlog is building. | Nothing needed |
| `PossibleDataLossDuringSegmentImport` | Real-time segments hold data older than the import buffer. | Late events may be skipped by import. | Raise `bufferTimePeriod` |

## Metrics

The series below are available in [Grafana](/corecapabilities/observability/grafana) through the Prometheus data source. For how names, suffixes, and labels are built, see [Reading Pinot metrics](/corecapabilities/observability/alerts/overview#reading-pinot-metrics).

### Metrics worth watching

These are the ones to reach for first; they are not the complete set.

| Metric | What it tells you |
| - | - |
| `pinot_controller_percentMinionSubtasksInError_Value` | Share of a task type's subtasks in error, per table. |
| `pinot_controller_numMinionSubtasksWaiting_Value` | Queue depth, per task type. |
| `pinot_controller_numMinionSubtasksDropped_Value` | Subtasks dropped rather than run. |
| `pinot_controller_maxSubtaskWaitTimeMs_Value`, `pinot_controller_maxSubtaskRunningTimeMs_Value` | Worst-case wait and run time, per table and task type. |
| `pinot_controller_taskGenerationFailureCount_Count` | Generation failed — no subtasks were even created. |
| `pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value` | How long since anything generated successfully. Pair it with the failure count: failures matter far more when this is also large. |
| `pinot_controller_taskGenerationSkippedDueToConflict_Count` | Generation skipped because a conflicting task was in flight. |
| `pinot_controller_consistentPushFailure_Count` | Consistent-push failures. |
| `pinot_controller_sqlIngestionFailure_Count`, `pinot_controller_fileIngestionFailure_Count`, `pinot_controller_segmentImportFailure_Count` | Per-ingestion-type generation failures. |
| `pinot_minion_taskQueueing_Count` | Minions actually dequeuing work. Zero while the queue is deep means no minion is picking anything up. |
| `pinot_controller_offlineSegmentDelayHours_Value` | Age of the newest offline segment — how far behind the real-time-to-offline job is. |

### Metrics behind each alert

The Prometheus series each alert rule evaluates.

| Alert | Metrics in the rule |
| - | - |
| `FailedToGenerateFileIngestionTask` | `pinot_controller_fileIngestionFailure_Count` |
| `FailedToGenerateSqlIngestionTask` | `pinot_controller_sqlIngestionFailure_Count` |
| `FailedToGenerateSegmentImportTask` | `pinot_controller_segmentImportFailure_Count` |
| `FailedToGenerateExternalTableSyncTask` | `pinot_controller_externalTableSyncTaskGenerationFailure_Count` |
| `MinionTaskGenerationFailures` | `pinot_controller_taskGenerationFailureCount_Count`, `pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value` |
| `MinionTaskGenerationSkipped` | `pinot_controller_taskGenerationSkippedDueToConflict_Count`, `pinot_controller_timeMsSinceLastSuccessfulMinionTaskGeneration_Value` |
| `MinionSubtasksInErrorForTableAndTaskType` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `FileIngestionSubtasksInErrorForTable` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `FileIngestionSubtasksInError` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `SqlIngestionSubtasksInError` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `SegmentImportSubtasksInError` | `pinot_controller_percentMinionSubtasksInError_Value` |
| `MinionSubtasksDroppedForTable` | `pinot_controller_numMinionSubtasksDropped_Value` |
| `MinionLongWaitingSubtasksForTaskType` | `pinot_controller_numMinionSubtasksWaiting_Value`, `pinot_minion_taskQueueing_Count` |
| `MinionSubTaskHighWaitTime` | `pinot_controller_maxSubtaskWaitTimeMs_Value` |
| `MinionSubTaskHighRunningTime` | `pinot_controller_maxSubtaskRunningTimeMs_Value` |
| `MinionConsistentPushFailures` | `pinot_controller_consistentPushFailure_Count` |
| `MinionsUnderUtilized` | `pinot_minion_numberOfTasks_Value` |
| `OfflineSegmentDelayHoursCrossedThreshold` | `pinot_controller_offlineSegmentDelayHours_Value`, `pinot_controller_scheduledSegments_Count` |
| `MinionSubtaskQueueSizeHigh` | `pinot_controller_numMinionSubtasksWaiting_Value`, `kube_pod_status_phase` |
| `PossibleDataLossDuringSegmentImport` | `pinot_controller_segmentsWithLateData_Value` |

## Related

* [Alerts and metrics overview](/corecapabilities/observability/alerts/overview): severity, the symptom router, the alert index, and how metric names are built.
* [Troubleshooting by feature: Batch ingestion and minion tasks](/corecapabilities/observability/troubleshooting-by-feature#batch-ingestion-and-minion-tasks)
* [Minion task runs](/corecapabilities/observability/minion-task-runs)
* [Ingestion troubleshooting](/corecapabilities/ingestdata/troubleshooting)


# Data availability and storage alerts
Source: https://docs.startree.ai/corecapabilities/observability/alerts/data-availability-and-storage

Alerts that watch serving replicas, segments in error, and disk, key, and storage limits: when each fires, what it means, and what you or StarTree can do.

Whether every segment has enough serving replicas, and whether a table is close to a disk, key or segment limit.

> [!NOTE]
>
> **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the [overview](/corecapabilities/observability/alerts/overview) for how to read an entry.

## Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

```mermaid placement="top-right" actions={true} theme={null}
flowchart TD
  A["Results incomplete or
change between runs"] --> B{"Segments in ERROR?
GET /tables/{t}/externalview"}
  B -- "Yes" --> B1["SegmentError
POST /segments/{t}/reset?errorSegmentsOnly=true"]
  B -- "No" --> C{"Restart, rebalance or
scale-out in progress?"}
  C -- "Yes" --> C1["Replicas are reloading
SegmentReplicasCriticallyLowForHATable clears on its own"]
  C -- "No" --> D{"Servers restarting or OOM?"}
  D -- "Yes" --> D1["See Platform health"]
  D -- "No" --> E{"Ingestion or uploads blocked?"}
  E -- "Yes" --> E1["ResourceUtilizationLimitExceeded
StorageQuotaUtilizationAlert"]
  E -- "No" --> F["Upsert table? See Upsert tables"]
```

## Critical alerts

### `SegmentReplicasCriticallyLowForHATable`

> [!NOTE]
>
> Often transient: this alert usually clears on its own.

| Field | Detail |
| - | - |
| **Fires when** | A table with replication 2+ has 34% or fewer replicas serving, for 10 min. |
| **What it means** | Queries may return partial results. Usually clears once replicas finish loading. |
| **Check first** | • Pinot Controllers → **Percent Of Replicas**, **Segments In Error State** • Compare `GET /tables/{table}/externalview` with `GET /tables/{table}/idealstate` |
| **What you can do** | • Use replication 3 for critical tables • Reduce minion task parallelism if it overlaps |
| **What StarTree does** | • Resets segments in ERROR • Investigates servers that keep restarting |

### `ResourceUtilizationLimitExceeded`

| Field | Detail |
| - | - |
| **Fires when** | A table hits a server disk, primary-key or segment-count limit, for 10 min. |
| **What it means** | Real-time consumption pauses and new tasks aren't generated for that table. |
| **Check first** | • `GET /tables/{table}/pauseStatus` shows the reason • Total Provisioned Capacity dashboard |
| **What you can do** | • Shorten retention, roll up or merge segments • Set an upsert `metadataTTL` where supported |
| **What StarTree does** | • Adds disk or servers • Consumption resumes on its own once below the limit |
| **See also** | [Segment count limits: monitoring and troubleshooting](/corecapabilities/manage-data/recipes/segment-count-limits#monitoring-and-troubleshooting) |

### `PinotZnodeSizeTooCloseToZkBufferLimit`

| Field | Detail |
| - | - |
| **Fires when** | A table's ideal state or external view is within 10 KB of ZooKeeper's size limit. |
| **What it means** | The table can stop accepting new segments. |
| **Check first** | Pinot Controllers → **Idealstate Znode Size** |
| **What you can do** | Reduce segment count with merge-rollup or retention |
| **What StarTree does** | Raises the ZooKeeper buffer if needed |
| **See also** | [Segment count limits: monitoring and troubleshooting](/corecapabilities/manage-data/recipes/segment-count-limits#monitoring-and-troubleshooting) |

### `TierStorageServerPreloadIndexReachingLimit`

| Field | Detail |
| - | - |
| **Fires when** | A server's tiered-storage preload cache is over 90% reserved, for 5 min. |
| **What it means** | When full, reads fall back to object storage: slower, still correct. |
| **What you can do** | Narrow the preloaded index keys |
| **What StarTree does** | Raises the preload budget |
| **See also** | [Tiered storage setup: monitoring metrics](/corecapabilities/manage-data/set-up-tiered-storage/setup) |

### `PinotPodVolumeCrossed90Percent`

| Field | Detail |
| - | - |
| **Fires when** | A Pinot volume is over 90% full, for 5 min. |
| **What it means** | Ingestion can stop and queries slow down. |
| **Check first** | Kubernetes / Storage / Volumes dashboard |
| **What you can do** | Shorten retention |
| **What StarTree does** | Grows the volume or adds servers |

### `SystemQueryLogStorageQuotaUtilization`

| Field | Detail |
| - | - |
| **Fires when** | The `system_query_log` table reaches 85% of its storage quota, for 10 min. |
| **What it means** | At 100%, new query-log segments are rejected until the quota rises. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Expands the quota |

## Warning alerts

| Alert | Fires when | What it means | What you can do |
| - | - | - | - |
| `SegmentReplicasUnavailable` | 67% or fewer replicas serving, for 10 min. | Early warning before the critical alert. | Nothing needed |
| `SegmentError` | A table has segments in ERROR for 30 min. | Results can differ between runs. | Nothing needed |
| `ConfiguredRFLessThanThreshold` | A table has replication below 2 on a multi-server cluster, for 1 hour. | Any server restart takes part of the table offline. | Raise replication, or accept the risk |
| `StorageQuotaUtilizationAlert` | A table uses over 80% of its storage quota, for 10 min. | Uploads are rejected at 100%. | Raise `quota.storage` or shorten retention |
| `PinotPodVolumeCrossed80Percent` | A Pinot volume is over 80% full, for 5 min. | Early warning before 90%. | Nothing needed |

## Metrics

The series below are available in [Grafana](/corecapabilities/observability/grafana) through the Prometheus data source. For how names, suffixes, and labels are built, see [Reading Pinot metrics](/corecapabilities/observability/alerts/overview#reading-pinot-metrics).

### Metrics worth watching

These are the ones to reach for first; they are not the complete set.

| Metric | What it tells you |
| - | - |
| `pinot_controller_percentOfReplicas_Value` | Share of segment replicas available, per table. Non-lead controllers emit a negative placeholder, so filter `>= 0`. |
| `pinot_controller_replicationFromConfig_Value` | Configured replication factor. |
| `pinot_controller_segmentsInErrorState_Value` | Segments stuck in error. |
| `pinot_controller_tableStorageQuotaUtilization_Value` | Percentage of the table's storage quota in use. |
| `pinot_controller_idealstateZnodeByteSize_Value`, `pinot_controller_externalviewZnodeByteSize_Value`, `pinot_controller_propertystoreSegmentChildrenByteSize_Value` | ZooKeeper znode sizes — the constraint that eventually bounds segment count. |
| `pinot_server_preloadCacheLayerReservedSizeBytes_Value`, `pinot_server_preloadCacheLayerMaxSizeBytes_Value` | Tiered-storage preload index headroom. Chart the ratio. |

### Metrics behind each alert

The Prometheus series each alert rule evaluates.

| Alert | Metrics in the rule |
| - | - |
| `SegmentReplicasCriticallyLowForHATable` | `pinot_broker_percentOfReplicas_Value`, `pinot_controller_percentOfReplicas_Value`, `pinot_controller_replicationFromConfig_Value`, `pinot_controller_numberOfReplicas_Value` |
| `ResourceUtilizationLimitExceeded` | `pinot_controller_resourceUtilizationLimitExceeded_Value` |
| `PinotZnodeSizeTooCloseToZkBufferLimit` | `pinot_controller_idealstateZnodeByteSize_Value`, `pinot_controller_externalviewZnodeByteSize_Value`, `pinot_controller_propertystoreSegmentChildrenByteSize_Value` |
| `TierStorageServerPreloadIndexReachingLimit` | `pinot_server_preloadCacheLayerReservedSizeBytes_Value`, `pinot_server_preloadCacheLayerMaxSizeBytes_Value` |
| `PinotPodVolumeCrossed90Percent` | `kubelet_volume_stats_used_bytes`, `kubelet_volume_stats_capacity_bytes` |
| `SystemQueryLogStorageQuotaUtilization` | `pinot_controller_tableStorageQuotaUtilization_Value` |
| `SegmentReplicasUnavailable` | `pinot_controller_percentOfReplicas_Value` |
| `SegmentError` | `pinot_controller_segmentsInErrorState_Value` |
| `ConfiguredRFLessThanThreshold` | `pinot_controller_replicationFromConfig_Value`, `kube_statefulset_status_replicas` |
| `StorageQuotaUtilizationAlert` | `pinot_controller_tableStorageQuotaUtilization_Value` |
| `PinotPodVolumeCrossed80Percent` | `kubelet_volume_stats_used_bytes`, `kubelet_volume_stats_capacity_bytes` |

## Related

* [Alerts and metrics overview](/corecapabilities/observability/alerts/overview): severity, the symptom router, the alert index, and how metric names are built.
* [Troubleshooting by feature: Segments and storage](/corecapabilities/observability/troubleshooting-by-feature#segments-and-storage)
* [Health checks](/corecapabilities/observability/health-checks)
* [Table health and email alerts](/corecapabilities/observability/table-health-and-alerting)


# Alerts and metrics
Source: https://docs.startree.ai/corecapabilities/observability/alerts/overview

How StarTree alerts on your StarTree Cloud environment and how Pinot metrics are named: severity levels, where to start from a symptom, an index of every alert, and how to find a metric.

StarTree runs 159 Prometheus alerts on every StarTree Cloud environment. They're split across five pages by what you'd notice, plus one page for the platform underneath. Use this page to find the right one, look an alert up by name in the index below, or learn how the Pinot metrics behind them are named.

> [!NOTE]
>
> **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds on these pages are defaults; StarTree may tune them for your environment.

## Start from the symptom

> **[New data is late or missing](/corecapabilities/observability/alerts/real-time-ingestion)**
>
> Real-time ingestion stopped, lagging, or dropping rows. 9 alerts.

> **[Queries fail, time out, or get rejected](/corecapabilities/observability/alerts/queries)**
>
> Error rates, timeouts, query quota, broker availability. 12 alerts.

> **[Batch data or a scheduled task didn't land](/corecapabilities/observability/alerts/batch-and-minion-tasks)**
>
> File, SQL, segment import, and external table sync tasks, and minion capacity. 20 alerts.

> **[Results are incomplete, inconsistent, or near a limit](/corecapabilities/observability/alerts/data-availability-and-storage)**
>
> Serving replicas, segments in error, disk, key, and storage limits. 11 alerts.

> **[Upsert results or storage look wrong](/corecapabilities/observability/alerts/upsert-tables)**
>
> Primary-key growth, compaction, snapshots, replica consistency. 7 alerts.

> **[Components restart, run hot, or infrastructure is unhealthy](/corecapabilities/observability/alerts/platform-health)**
>
> Out-of-memory kills, CPU, GC, ZooKeeper, Kubernetes, platform services. 100 alerts.

## How to read an alert entry

Each critical alert on the category pages has its own entry:

| Field | Meaning |
| - | - |
| **Fires when** | The condition and how long it must hold before the alert fires |
| **What it means** | The effect on your data or queries |
| **Check first** | The Grafana panel or API call that confirms the cause |
| **What you can do** | Changes on your side: stream, source, table config, queries |
| **What StarTree does** | Actions StarTree on-call takes |

Some entries carry a note. **Often transient** means the alert usually clears on its own. **Usually your fix** means the cause is most often in your stream, source system, or table config. Warning alerts are listed in a table on each page, and every page ends with the metrics each alert evaluates.

Grafana dashboards named on these pages are described in [Grafana dashboards](/corecapabilities/observability/grafana). API paths are Pinot controller endpoints, available from **Cluster Manager → Swagger API** in the Data Portal.

## Alert index

All 159 alerts, 119 critical and 40 warning, in alphabetical order.

| Alert | Severity | Page |
| - | - | - |
| `AuthServiceHighHttp5xxErrorRate` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `BrokerDataTableDeserializationExceptions` | Critical | [Queries](/corecapabilities/observability/alerts/queries#brokerdatatabledeserializationexceptions) |
| `BrokerNoServerFoundAlert` | Warning | [Queries](/corecapabilities/observability/alerts/queries#brokernoserverfoundalert) |
| `BrokerQueryExecutionLatency` | Warning | [Queries](/corecapabilities/observability/alerts/queries#brokerqueryexecutionlatency) |
| `BrokerResponseAlert` | Warning | [Queries](/corecapabilities/observability/alerts/queries#brokerresponsealert) |
| `CertificateExpiration` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `CertificateRenewal` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ConfiguredRFLessThanThreshold` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#configuredrflessthanthreshold) |
| `ContainerOOMKills` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health#containeroomkills) |
| `DetectionTaskHealth` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `FailedToGenerateExternalTableSyncTask` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#failedtogenerateexternaltablesynctask) |
| `FailedToGenerateFileIngestionTask` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#failedtogeneratefileingestiontask) |
| `FailedToGenerateSegmentImportTask` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#failedtogeneratesegmentimporttask) |
| `FailedToGenerateSqlIngestionTask` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#failedtogeneratesqlingestiontask) |
| `FileIngestionSubtasksInError` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#fileingestionsubtasksinerror) |
| `FileIngestionSubtasksInErrorForTable` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#fileingestionsubtasksinerrorfortable) |
| `HighAvailablePinotStatefulsetReplicasMismatch` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#highavailablepinotstatefulsetreplicasmismatch) |
| `HighNonServerPodsRestarts` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#highnonserverpodsrestarts) |
| `HighQueryCriticalErrorRate` | Critical | [Queries](/corecapabilities/observability/alerts/queries#highquerycriticalerrorrate) |
| `HighQueryTimeoutPercentage` | Critical | [Queries](/corecapabilities/observability/alerts/queries#highquerytimeoutpercentage) |
| `HighRealtimeIngestionOffsetLag` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#highrealtimeingestionoffsetlag) |
| `HighServerPodsRestarts` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#highserverpodsrestarts) |
| `HighSystemQueryErrorRate` | Critical | [Queries](/corecapabilities/observability/alerts/queries#highsystemqueryerrorrate) |
| `HighTotalDocsToPKCountRatio` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#hightotaldocstopkcountratio) |
| `HighTotalNumberOfPrimaryKeysPerServer` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#hightotalnumberofprimarykeysperserver) |
| `KubernetesContainerOOMKilled` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesContainerTerminated` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesMinionPodNotHealthy` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeDiskPressure` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeMemoryPressure` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeNetworkUnavailable` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeNotReady` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesNodeOutOfPodCapacity` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesPinotContainerOOMKilled` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#kubernetespinotcontaineroomkilled) |
| `KubernetesPinotPodCrashLooping` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#kubernetespinotpodcrashlooping) |
| `KubernetesPodCrashLooping` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesPodFailed` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesPodNotHealthy` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesPodPending` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesReplicasetReplicasMismatch` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesStatefulsetUpdateNotRolledOut` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesThirdeyeContainerOOMKilled` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `KubernetesThirdeyePodCrashLooping` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `LowBrokerAvailability` | Critical | [Queries](/corecapabilities/observability/alerts/queries#lowbrokeravailability) |
| `MariadbConnectionsExhausted` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbDown` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHANotAvailable` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationIOThreadStopped` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationLagHigh` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationRecvQueueSize` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationSendQueueSize` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MariadbHAReplicationSQLThreadStopped` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MinionConsistentPushFailures` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionconsistentpushfailures) |
| `MinionLongWaitingSubtasksForTaskType` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionlongwaitingsubtasksfortasktype) |
| `MinionSubTaskHighRunningTime` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtaskhighrunningtime) |
| `MinionSubTaskHighWaitTime` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtaskhighwaittime) |
| `MinionSubtaskQueueSizeHigh` | Warning | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtaskqueuesizehigh) |
| `MinionSubtasksDroppedForTable` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtasksdroppedfortable) |
| `MinionSubtasksInErrorForTableAndTaskType` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsubtasksinerrorfortableandtasktype) |
| `MinionsUnderUtilized` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#minionsunderutilized) |
| `MinionTaskGenerationFailures` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#miniontaskgenerationfailures) |
| `MinionTaskGenerationSkipped` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#miniontaskgenerationskipped) |
| `MySQLStorageExhaustion` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `MySQLStorageUsage` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NettyResponseFetchExceptions` | Critical | [Queries](/corecapabilities/observability/alerts/queries#nettyresponsefetchexceptions) |
| `NoBrokerAvailable` | Critical | [Queries](/corecapabilities/observability/alerts/queries#nobrokeravailable) |
| `NoControllerLeaderAlert` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#nocontrollerleaderalert) |
| `NodeConntrackExhausted` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeConntrackLow` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeDataDiskUsageCritical` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeDataDiskUsageHigh` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeNetworkBandwidthInThrottled` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeNetworkBandwidthOutThrottled` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NodeNetworkPPSThrottled` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `NoRealtimeIngestionPerServer` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#norealtimeingestionperserver) |
| `OfflineSegmentDelayHoursCrossedThreshold` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#offlinesegmentdelayhourscrossedthreshold) |
| `OperatorDisabled` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `OperatorOffline` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PartitionConsumerCreateExceptions` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#partitionconsumercreateexceptions) |
| `PinotCertificateExpiration` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PinotCertificateRenewal` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PinotComponentHighCPUUsage` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotcomponenthighcpuusage) |
| `PinotControllerHighThreadsBlocked` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotcontrollerhighthreadsblocked) |
| `PinotHighGcPercent` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinothighgcpercent) |
| `PinotPodImagePullErrors` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotpodimagepullerrors) |
| `PinotPodStuckInContainerCreating` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotpodstuckincontainercreating) |
| `PinotPodVolumeCrossed80Percent` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#pinotpodvolumecrossed80percent) |
| `PinotPodVolumeCrossed90Percent` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#pinotpodvolumecrossed90percent) |
| `PinotPodWaiting` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotpodwaiting) |
| `PinotStatefulsetReplicasMismatch` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health#pinotstatefulsetreplicasmismatch) |
| `PinotZnodeSizeTooCloseToZkBufferLimit` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#pinotznodesizetooclosetozkbufferlimit) |
| `PlatformAuthAPIP95Latency` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformAuthErrorLogs` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformAuthInfoP95Latency` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformAuthP95Latency` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformAuthZP95Latency` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformDeploymentReplicasMismatch` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformFrontendErrorLogs` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformPodWaiting` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PlatformStatefulsetReplicasMismatch` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PodVolumeCrossed80Percent` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PodVolumeCrossed90Percent` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PossibleDataLossDuringSegmentImport` | Warning | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#possibledatalossduringsegmentimport) |
| `PrometheusContainerHighMemoryUsage` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusContainerOOMKilled` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusPodVolumeCrossed65Percent` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusPodVolumeCrossed75Percent` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbCheckpointCreationFailures` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbCheckpointDeletionFailures` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbCompactionsFailed` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbHeadTruncationsFailed` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbReloadFailures` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbWalCorruptions` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `PrometheusTsdbWalTruncationsFailed` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `QueriesFailingWithUnavailableSegment` | Warning | [Queries](/corecapabilities/observability/alerts/queries#queriesfailingwithunavailablesegment) |
| `QueryQuotaAlert` | Critical | [Queries](/corecapabilities/observability/alerts/queries#queryquotaalert) |
| `RBACServiceHighHttp5xxErrorRate` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `RealtimeConsumptionExceptions` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimeconsumptionexceptions) |
| `RealtimeIngestionHighRowsWithErrorsRate` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimeingestionhighrowswitherrorsrate) |
| `RealtimeIngestionHighSkippedRowsRate` | Warning | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimeingestionhighskippedrowsrate) |
| `RealtimeIngestionStopped` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimeingestionstopped) |
| `RealtimePartitionMismatch` | Warning | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#realtimepartitionmismatch) |
| `ResourceUtilizationLimitExceeded` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#resourceutilizationlimitexceeded) |
| `RocksDBHighCompactionRate` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#rocksdbhighcompactionrate) |
| `RocksDBHighFlushRate` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#rocksdbhighflushrate) |
| `RocksDBHighMemoryUsage` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#rocksdbhighmemoryusage) |
| `SegmentError` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#segmenterror) |
| `SegmentImportSubtasksInError` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#segmentimportsubtasksinerror) |
| `SegmentReplicasCriticallyLowForHATable` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#segmentreplicascriticallylowforhatable) |
| `SegmentReplicasUnavailable` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#segmentreplicasunavailable) |
| `SnapshotCreationFailed` | Critical | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#snapshotcreationfailed) |
| `SqlIngestionSubtasksInError` | Critical | [Batch and minion tasks](/corecapabilities/observability/alerts/batch-and-minion-tasks#sqlingestionsubtasksinerror) |
| `StorageQuotaUtilizationAlert` | Warning | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#storagequotautilizationalert) |
| `StreamDataLoss` | Critical | [Real-time ingestion](/corecapabilities/observability/alerts/real-time-ingestion#streamdataloss) |
| `SystemQueryLogStorageQuotaUtilization` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#systemquerylogstoragequotautilization) |
| `ThirdeyeCertificateExpiration` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyeCertificateRenewal` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyeMultipleNamespaceConfigurations` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyeNotificationFailure` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyePodVolumeCrossed80Percent` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyePodVolumeCrossed90Percent` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ThirdeyePodWaiting` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TierStorageServerPreloadIndexReachingLimit` | Critical | [Data availability and storage](/corecapabilities/observability/alerts/data-availability-and-storage#tierstorageserverpreloadindexreachinglimit) |
| `TraefikDeploymentReplicasMismatch` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikHighHttp499PinotErrorRateService` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikHighHttp4xxErrorRateService` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikHighHttp5xxErrorRateService` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikPinotBrokerHttp403BurnRateHigh` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikPinotBrokerHttp5xxBurnRateHigh` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TraefikServiceConnectionsAbnormalGrowth` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `TSDBDataInsufficient` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `UpsertInconsistentReplicas` | Warning | [Upsert tables](/corecapabilities/observability/alerts/upsert-tables#upsertinconsistentreplicas) |
| `X509ExporterReadErrors` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health) |
| `ZookeeperErrorCount` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeepererrorcount) |
| `ZookeeperIncorrectNumberOfLeaders` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeeperincorrectnumberofleaders) |
| `ZookeeperIncorrectQuorumSize` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeeperincorrectquorumsize) |
| `ZookeeperLeaderElectionTooLong` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeeperleaderelectiontoolong) |
| `ZookeeperLeaderUnavailable` | Warning | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeeperleaderunavailable) |
| `ZookeeperMetricsMissing` | Critical | [Platform health](/corecapabilities/observability/alerts/platform-health#zookeepermetricsmissing) |

## Reading Pinot metrics

Your cluster's Pinot metrics are available in [Grafana](/corecapabilities/observability/grafana), through the Prometheus data source. Each alert page lists the metrics worth watching for its area and the series each alert evaluates. This section explains how those names are built, so you can find the one you need rather than guess.

> [!WARNING]
>
> **A metric that has never been recorded does not exist.** If no value has been published — a table that has never been queried, a task type that has never run — the metric is absent rather than present with a zero. An empty chart can mean "nothing happened," not "the metric is wrong."

### Naming convention

Metric names follow a consistent shape:

```
pinot___
```

| Part | Values | Example |
| - | - | - |
| `component` | `controller`, `broker`, `server`, `minion` | `pinot_server_…` |
| `metricName` | The Pinot metric name, in camelCase | `…_realtimeIngestionDelayMs_…` |
| `Suffix` | The statistic — see below | `…_Value` |

So a real-time ingestion delay gauge on a server becomes `pinot_server_realtimeIngestionDelayMs_Value`, and a broker query counter becomes `pinot_broker_queries_Count`.

A few metrics do not follow the pattern. RocksDB metrics, used by off-heap upsert, appear as `pinot_server_rocksdb_ticker_` and `pinot_server_rocksdb_histogram__`. JVM metrics such as `jvm_gc_collection_seconds_sum` and `jvm_threads_state` keep their standard JVM names and carry a `component` label instead of a `pinot_` prefix.

#### Suffixes

| Suffix | Meaning | Applies to |
| - | - | - |
| `_Value` | Current value | Gauges |
| `_Count` | Monotonically increasing total | Meters and timers |
| `_OneMinuteRate`, `_FiveMinuteRate` | Exponentially weighted rate | Meters |
| `_50thPercentile`, `_75thPercentile`, `_95thPercentile`, `_99thPercentile` | Latency distribution | Timers |

Not every statistic is available for every metric. If a suffix such as `_Mean` or `_FifteenMinuteRate` returns nothing, use one from the table above instead — `_50thPercentile` for a typical value, or a rate over `_Count` for throughput. Confirm what a metric actually offers by running the bare name in **Explore** and reading the series that come back.

#### Labels

Metrics are labelled so you can slice them without name explosion:

| Label | Meaning |
| - | - |
| `table` | Table name with type suffix, for example `orders_REALTIME` |
| `database` | Database the table belongs to, when the table is not in the default database |
| `tableType` | `OFFLINE` or `REALTIME` |
| `partition` | Stream partition, on ingestion metrics |
| `topic`, `topic_name` | Source stream topic, on ingestion metrics |
| `taskType` | Minion task type, for example `SegmentGenerationAndPushTask` |
| `scope` | Metric scope where a metric is emitted at more than one level |
| `column` | Column name, on the few per-column metrics |
| `throttlerSet` | Throttler pool, on server throttling metrics |

Some metrics also carry pod-level labels such as `pod` or `kubernetes_pod_name`. These identify the component instance rather than a table, so resource questions ("which server is hot?") are answered by grouping on those, and workload questions ("which table is slow?") by grouping on `table`.

> [!NOTE]
>
> The `table` label always carries the type suffix. `table="orders_REALTIME"` and `table="orders_OFFLINE"` are separate series for a hybrid table, so a per-table chart of a hybrid table needs both, or a `table=~"orders_.*"` matcher.

### Finding a metric that is not listed

The lists above are curated, not exhaustive — Pinot exports far more than this. To find something specific:

**1. Browse in Grafana Explore**

Open Grafana, choose **Explore**, and select the Prometheus data source. Type `pinot_server_` in the metric field and the metric browser will complete against everything currently present in your cluster. This is the fastest way to check whether a metric exists at all.

**2. Search by fragment**

Metric names are camelCase inside a snake\_case wrapper, so search on a distinctive fragment — `Upsert`, `Ingestion`, `Segment` — rather than a whole name. Prometheus's metric browser matches on substring.

**3. Check the labels before charting**

Run the bare metric name first and look at the label set on the returned series. Whether a metric is per-table, per-partition, or per-pod determines whether you need `sum by (table)` or `avg by (pod)` — and getting this wrong is the most common cause of a chart that looks alarming and means nothing.

**4. Copy from an existing panel**

Every panel on the [built-in dashboards](/corecapabilities/observability/grafana) exposes its query. Open the panel menu, choose **Explore**, and you get a working query with the right labels and aggregation to modify.

## Related

* [Health checks](/corecapabilities/observability/health-checks) and [Table health and email alerts](/corecapabilities/observability/table-health-and-alerting): Data Portal checks and email alerts, separate from these Prometheus alerts.
* [Grafana dashboards](/corecapabilities/observability/grafana): the built-in dashboards over these metrics.
* [Accessing logs](/corecapabilities/observability/access-logs): when a metric tells you something happened and you need the log line.
* [Query Logger](/corecapabilities/query_data/advanced_operations/query-logger): per-query records rather than aggregates.
* [Troubleshooting: start here](/corecapabilities/observability/troubleshooting): step-by-step triage when something is wrong.


# Platform health alerts
Source: https://docs.startree.ai/corecapabilities/observability/alerts/platform-health

Alerts that watch Pinot components, ZooKeeper, Kubernetes, platform services, certificates, and monitoring, most of which StarTree handles end to end.

This page covers the Pinot components themselves, then the infrastructure underneath them. Most of these alerts are handled by StarTree end to end.

> [!NOTE]
>
> **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the [overview](/corecapabilities/observability/alerts/overview) for how to read an entry.

## Cluster health

Pinot controllers, brokers, servers, minions and ZooKeeper: restarts, memory, CPU and coordination.

### Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

```mermaid placement="top-right" actions={true} theme={null}
flowchart TD
  A["Components restart or run hot"] --> B{"Last termination reason?"}
  B -- "OOMKilled" --> B1["KubernetesPinotContainerOOMKilled
heavy queries or task waves"]
  B -- "Crash on start" --> B2["PinotPodWaiting
config, keystore or deep store access"]
  B -- "Liveness kill" --> B3["PinotHighGcPercent
long GC pauses"]
  A --> C{"CPU above 85% of request
for 30 min?"}
  C -- "With query errors" --> C1["Capacity problem
scale or limit queries"]
  C -- "Alone" --> C2["Often an ingestion or
reload peak"]
  A --> D{"Coordination?"}
  D -- "No lead controller" --> D1["NoControllerLeaderAlert"]
  D -- "ZooKeeper quorum or leader" --> D2["Zookeeper* alerts"]
```

### Critical alerts

#### `NoControllerLeaderAlert`

| Field | Detail |
| - | - |
| **Fires when** | No controller is leader, for 10 min. |
| **What it means** | Table changes, segment commits and tasks stall. |
| **Check first** | Pinot Controllers → **Pinot Controller Leader** |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Restores leadership |

#### `PinotControllerHighThreadsBlocked`

| Field | Detail |
| - | - |
| **Fires when** | Blocked controller threads exceed half the runnable ones, for 10 min. |
| **What it means** | The controller is contended; operations slow down. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Takes thread dumps and restarts |

#### `PinotHighGcPercent`

| Field | Detail |
| - | - |
| **Fires when** | A Pinot or ZooKeeper JVM spends over 50% of time in GC, for 15 min. |
| **What it means** | Memory pressure; queries and ingestion slow down. |
| **Check first** | JVM dashboard |
| **What you can do** | Fix queries that build large results |
| **What StarTree does** | Adds memory or finds the leak |

#### `PinotComponentHighCPUUsage`

> [!NOTE]
>
> Often transient: this alert usually clears on its own.

| Field | Detail |
| - | - |
| **Fires when** | A server, broker or controller uses over 85% of its CPU request for 30 min. Servers still starting are skipped. |
| **What it means** | Sustained load. Paired with query errors, it's a capacity problem. |
| **Check first** | Pinot Servers → **CPU % Usage** |
| **What you can do** | Move heavy reloads or backfills off peak hours |
| **What StarTree does** | Scales or rebalances |

#### `HighServerPodsRestarts`

| Field | Detail |
| - | - |
| **Fires when** | A server restarts more than twice in 30 min. |
| **What it means** | Replicas drop and ingestion can stop. |
| **Check first** | Logging / Kubernetes Events |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Finds the OOM or crash cause |

#### `HighNonServerPodsRestarts`

| Field | Detail |
| - | - |
| **Fires when** | A broker, controller or ZooKeeper pod restarts more than twice in 15 min. |
| **What it means** | Queries or coordination are disrupted. |
| **Check first** | Logging / Kubernetes Events |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Finds the OOM or crash cause |

#### `KubernetesPinotContainerOOMKilled`

| Field | Detail |
| - | - |
| **Fires when** | A Pinot container was killed for running out of memory. |
| **What it means** | Often heavy queries, task waves or on-heap upsert metadata. |
| **Check first** | Logging / Kubernetes Events |
| **What you can do** | Avoid exact distinct counts and very large LIMITs |
| **What StarTree does** | Raises memory or moves upsert metadata to RocksDB |

#### `KubernetesPinotPodCrashLooping`

| Field | Detail |
| - | - |
| **Fires when** | A Pinot pod restarts more than 5 times in 10 min. |
| **What it means** | The pod can't stay up. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Fixes the crash cause |

#### `PinotPodWaiting`

| Field | Detail |
| - | - |
| **Fires when** | A Pinot pod is in CrashLoopBackOff for 15 min. |
| **What it means** | Usually bad config, keystore or deep store access. |
| **What you can do** | Check any credentials you supplied for deep store |
| **What StarTree does** | Fixes startup |

#### `PinotPodStuckInContainerCreating`

| Field | Detail |
| - | - |
| **Fires when** | A Pinot pod is stuck creating its container for 15 min. |
| **What it means** | Usually a volume or node problem. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Fixes scheduling |

#### `PinotPodImagePullErrors`

| Field | Detail |
| - | - |
| **Fires when** | A Pinot pod can't pull its image for 15 min. |
| **What it means** | The pod can't start. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Fixes the image reference or registry |

#### `HighAvailablePinotStatefulsetReplicasMismatch`

| Field | Detail |
| - | - |
| **Fires when** | A Pinot component with 2+ replicas has fewer than 2 ready, for 1 hour. |
| **What it means** | High availability is lost. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Restores replicas |

#### `ZookeeperErrorCount`

| Field | Detail |
| - | - |
| **Fires when** | ZooKeeper reports an unrecoverable error. |
| **What it means** | A ZooKeeper thread died. |
| **Check first** | Pinot ZooKeeper dashboard |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Checks quorum and restarts the member |

#### `ZookeeperLeaderElectionTooLong`

| Field | Detail |
| - | - |
| **Fires when** | ZooKeeper keeps electing a leader, for 15 min. |
| **What it means** | Coordination is unstable. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Restores quorum |

#### `ZookeeperIncorrectNumberOfLeaders`

| Field | Detail |
| - | - |
| **Fires when** | ZooKeeper doesn't have exactly one leader, for 10 min. |
| **What it means** | Coordination is unstable. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Restores quorum |

#### `ZookeeperIncorrectQuorumSize`

| Field | Detail |
| - | - |
| **Fires when** | ZooKeeper quorum drops below 3, for 10 min. |
| **What it means** | Fault tolerance is reduced. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Restores members |

#### `ZookeeperMetricsMissing`

| Field | Detail |
| - | - |
| **Fires when** | ZooKeeper leader metrics are missing for 20 min. |
| **What it means** | A monitoring gap, not necessarily an outage. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Restores metrics |

### Warning alerts

| Alert | Fires when | What it means | What you can do |
| - | - | - | - |
| `ZookeeperLeaderUnavailable` | ZooKeeper reports time without a leader, for 10 min. | Brief coordination gaps. | Nothing needed |
| `PinotStatefulsetReplicasMismatch` | A Pinot component has fewer ready replicas than desired, for 1 hour. | Capacity is reduced. | Nothing needed |
| `ContainerOOMKills` | A container was OOM-killed in the last 24 hours. | Repeated memory pressure. | Nothing needed |

## Infrastructure StarTree manages

StarTree handles these alerts end to end. They're listed so you know what is watched; none of them needs action from you.

> **Kubernetes nodes and network (7 critical, 5 warning)**
>
> | Alert | Severity | Fires when |
> | - | - | - |
> | `NodeDataDiskUsageCritical` | Critical | Node data disk over 90%, 5 min |
> | `NodeConntrackExhausted` | Critical | Node connection tracking exhausted; new connections drop |
> | `KubernetesNodeNotReady` | Critical | A node not Ready, 5 min |
> | `KubernetesNodeMemoryPressure` | Critical | Node memory pressure, 15 min |
> | `KubernetesNodeDiskPressure` | Critical | Node disk pressure, 15 min |
> | `KubernetesNodeNetworkUnavailable` | Critical | Node network unavailable, 5 min |
> | `NodeConntrackLow` | Critical | Connection tracking allowance running low, 5 min |
> | `NodeDataDiskUsageHigh` | Warning | Node data disk over 80%, 15 min |
> | `KubernetesNodeOutOfPodCapacity` | Warning | Node over 90% of pod capacity, 15 min |
> | `NodeNetworkBandwidthInThrottled` | Warning | Inbound bandwidth throttled, 5 min |
> | `NodeNetworkBandwidthOutThrottled` | Warning | Outbound bandwidth throttled, 5 min |
> | `NodeNetworkPPSThrottled` | Warning | Packets-per-second throttled, 5 min |

> **Kubernetes pods and volumes (6 critical, 5 warning)**
>
> | Alert | Severity | Fires when |
> | - | - | - |
> | `KubernetesPodPending` | Critical | A pod pending, 10 min |
> | `KubernetesPodFailed` | Critical | Over 50 failed instances of a pod, 10 min |
> | `KubernetesMinionPodNotHealthy` | Critical | A minion pod not running, 25 min |
> | `KubernetesPodCrashLooping` | Critical | A platform pod restarts 5+ times in 10 min |
> | `KubernetesContainerOOMKilled` | Critical | A platform container OOM-killed |
> | `PodVolumeCrossed90Percent` | Critical | A platform volume over 90%, 5 min |
> | `PodVolumeCrossed80Percent` | Warning | A platform volume over 80%, 5 min |
> | `KubernetesPodNotHealthy` | Warning | A pod Pending, Unknown or Failed, 10 min |
> | `KubernetesContainerTerminated` | Warning | A container terminated with an error |
> | `KubernetesReplicasetReplicasMismatch` | Warning | ReplicaSet not fully ready, 10 min |
> | `KubernetesStatefulsetUpdateNotRolledOut` | Warning | StatefulSet update not rolled out, 10 min |

> **Platform services (16 critical, 5 warning)**
>
> | Alert | Severity | Fires when |
> | - | - | - |
> | `PlatformAuthP95Latency` | Critical | Sign-in p95 over 100 ms, 15 min |
> | `PlatformAuthZP95Latency` | Critical | Authorization p95 over 100 ms, 15 min |
> | `PlatformAuthInfoP95Latency` | Critical | Auth info API p95 over 100 ms, 15 min |
> | `PlatformAuthAPIP95Latency` | Critical | Other auth APIs p95 over 200 ms, 15 min |
> | `AuthServiceHighHttp5xxErrorRate` | Critical | Auth service 5xx over 10%, 15 min |
> | `RBACServiceHighHttp5xxErrorRate` | Critical | RBAC service 5xx over 10%, 15 min |
> | `PlatformStatefulsetReplicasMismatch` | Critical | Platform StatefulSet not fully ready, 15 min |
> | `PlatformPodWaiting` | Critical | Platform pod stuck waiting, 15 min |
> | `MariadbDown` | Critical | Metadata database down, 5 min |
> | `MariadbHAReplicationIOThreadStopped` | Critical | Database replication I/O thread stopped |
> | `MariadbHAReplicationSQLThreadStopped` | Critical | Database replication SQL thread stopped |
> | `MariadbHANotAvailable` | Critical | Database cluster below 2 nodes, 5 min |
> | `MariadbHAReplicationLagHigh` | Critical | Database replication lag high, 10 min |
> | `MariadbHAReplicationRecvQueueSize` | Critical | Database replication write latency high, 10 min |
> | `MariadbHAReplicationSendQueueSize` | Critical | Database replication send latency high, 10 min |
> | `MariadbConnectionsExhausted` | Critical | Database connections nearly exhausted, 5 min |
> | `PlatformDeploymentReplicasMismatch` | Warning | Platform deployment not fully ready, 15 min |
> | `PlatformFrontendErrorLogs` | Warning | Rising error logs in the frontend |
> | `PlatformAuthErrorLogs` | Warning | Rising error logs in auth |
> | `OperatorDisabled` | Warning | Operator disabled, 48 h |
> | `OperatorOffline` | Warning | Operator offline, 48 h |

> **API gateway (Traefik) (5 critical, 2 warning)**
>
> | Alert | Severity | Fires when |
> | - | - | - |
> | `TraefikDeploymentReplicasMismatch` | Critical | Gateway replicas not all available, 5 min |
> | `TraefikServiceConnectionsAbnormalGrowth` | Critical | Open connections jump 4× above the 5-min average and over 500 |
> | `TraefikHighHttp499PinotErrorRateService` | Critical | Over 20% of Pinot requests closed by the client (499), 5 min |
> | `TraefikPinotBrokerHttp5xxBurnRateHigh` | Critical | Broker 5xx burning the 30-day error budget 6× too fast |
> | `TraefikPinotBrokerHttp403BurnRateHigh` | Critical | Broker 403s burning the 30-day error budget 6× too fast |
> | `TraefikHighHttp4xxErrorRateService` | Warning | Over 5% 4xx on a service, 1 min |
> | `TraefikHighHttp5xxErrorRateService` | Warning | Over 5% 5xx on a service, 1 min |

> **Certificates (3 critical, 2 warning)**
>
> | Alert | Severity | Fires when |
> | - | - | - |
> | `CertificateExpiration` | Critical | A platform certificate expires in under 4 days |
> | `PinotCertificateExpiration` | Critical | A Pinot endpoint certificate expires in under 4 days |
> | `X509ExporterReadErrors` | Critical | Certificate exporter can't read certificates, 15 min |
> | `CertificateRenewal` | Warning | A platform certificate expires in under 7 days |
> | `PinotCertificateRenewal` | Warning | A Pinot certificate expires in under 7 days |

> **Monitoring stack (11 critical, 1 warning)**
>
> | Alert | Severity | Fires when |
> | - | - | - |
> | `PrometheusPodVolumeCrossed75Percent` | Critical | Prometheus volume over 75%, 5 min |
> | `PrometheusContainerHighMemoryUsage` | Critical | Prometheus over 80% of its memory limit, 10 min |
> | `PrometheusContainerOOMKilled` | Critical | Prometheus OOM-killed |
> | `TSDBDataInsufficient` | Critical | Less than 15 days of metrics retained |
> | `PrometheusTsdbCompactionsFailed` | Critical | Metrics storage compaction failed in 2 h |
> | `PrometheusTsdbWalCorruptions` | Critical | Write-ahead log corrupted in 2 h |
> | `PrometheusTsdbHeadTruncationsFailed` | Critical | Head truncation failed in 2 h |
> | `PrometheusTsdbWalTruncationsFailed` | Critical | WAL truncation failed in 2 h |
> | `PrometheusTsdbReloadFailures` | Critical | Storage reload failed in 2 h |
> | `PrometheusTsdbCheckpointCreationFailures` | Critical | Checkpoint creation failed in 2 h |
> | `PrometheusTsdbCheckpointDeletionFailures` | Critical | Checkpoint deletion failed in 2 h |
> | `PrometheusPodVolumeCrossed65Percent` | Warning | Prometheus volume over 65%, 5 min |

> **ThirdEye (9 critical, 3 warning)**
>
> | Alert | Severity | Fires when |
> | - | - | - |
> | `DetectionTaskHealth` | Critical | Over 20% of detection tasks fail over 2 h |
> | `ThirdeyeNotificationFailure` | Critical | A notification failed to send in 15 min |
> | `ThirdeyePodWaiting` | Critical | A ThirdEye pod stuck waiting, 15 min |
> | `MySQLStorageExhaustion` | Critical | ThirdEye database volume over 90%, 5 min |
> | `ThirdeyeCertificateExpiration` | Critical | A ThirdEye certificate expires in under 4 days |
> | `ThirdeyeMultipleNamespaceConfigurations` | Critical | Conflicting namespace configuration in the database |
> | `ThirdeyePodVolumeCrossed90Percent` | Critical | A ThirdEye volume over 90%, 5 min |
> | `KubernetesThirdeyeContainerOOMKilled` | Critical | A ThirdEye container OOM-killed |
> | `KubernetesThirdeyePodCrashLooping` | Critical | A ThirdEye pod restarts 5+ times in 10 min |
> | `MySQLStorageUsage` | Warning | ThirdEye database volume over 80%, 5 min |
> | `ThirdeyeCertificateRenewal` | Warning | A ThirdEye certificate expires in under 7 days |
> | `ThirdeyePodVolumeCrossed80Percent` | Warning | A ThirdEye volume over 80%, 5 min |

## Metrics

The series below are available in [Grafana](/corecapabilities/observability/grafana) through the Prometheus data source. For how names, suffixes, and labels are built, see [Reading Pinot metrics](/corecapabilities/observability/alerts/overview#reading-pinot-metrics).

### Metrics worth watching

These are the ones to reach for first; they are not the complete set.

| Metric | What it tells you |
| - | - |
| `pinot_controller_pinotControllerLeader_Value` | `1` on the leader. Summing to zero across the cluster means no leader. |
| `jvm_gc_collection_seconds_sum` | GC time, by `component` and `gc`. Rate of this over wall-clock time is the GC percentage. |
| `jvm_threads_state` | Thread counts by `state` and `component`. Blocked-versus-runnable is how a wedged controller shows up. |

### Metrics behind each alert

The Prometheus series each alert rule evaluates, for the Pinot component alerts.

| Alert | Metrics in the rule |
| - | - |
| `NoControllerLeaderAlert` | `pinot_controller_pinotControllerLeader_Value` |
| `PinotControllerHighThreadsBlocked` | `jvm_threads_state` |
| `PinotHighGcPercent` | `jvm_gc_collection_seconds_sum` |
| `PinotComponentHighCPUUsage` | `container_cpu_user_seconds_total`, `container_cpu_system_seconds_total`, `kube_pod_container_resource_requests`, `pinot_server_startupStatusCheckInProgress_Value`, `kube_pod_status_phase` |
| `HighServerPodsRestarts` | `kube_pod_container_status_restarts_total` |
| `HighNonServerPodsRestarts` | `kube_pod_container_status_restarts_total` |
| `KubernetesPinotContainerOOMKilled` | `kube_pod_container_status_restarts_total`, `kube_pod_container_status_last_terminated_reason` |
| `KubernetesPinotPodCrashLooping` | `kube_pod_container_status_restarts_total` |
| `PinotPodWaiting` | `kube_pod_container_status_waiting_reason` |
| `PinotPodStuckInContainerCreating` | `kube_pod_container_status_waiting_reason` |
| `PinotPodImagePullErrors` | `kube_pod_container_status_waiting_reason` |
| `HighAvailablePinotStatefulsetReplicasMismatch` | `kube_statefulset_status_replicas_ready`, `kube_statefulset_status_replicas` |
| `ZookeeperErrorCount` | `unrecoverable_error_count` |
| `ZookeeperLeaderElectionTooLong` | `election_time_count` |
| `ZookeeperIncorrectNumberOfLeaders` | `leader_uptime` |
| `ZookeeperIncorrectQuorumSize` | `quorum_size` |
| `ZookeeperMetricsMissing` | `leader_unavailable_time_sum` |
| `ZookeeperLeaderUnavailable` | `leader_unavailable_time_sum` |
| `PinotStatefulsetReplicasMismatch` | `kube_statefulset_status_replicas_ready`, `kube_statefulset_status_replicas` |
| `ContainerOOMKills` | `kube_pod_container_status_restarts_total`, `kube_pod_container_status_last_terminated_reason` |

## Related

* [Alerts and metrics overview](/corecapabilities/observability/alerts/overview): severity, the symptom router, the alert index, and how metric names are built.
* [Cluster operations troubleshooting](/corecapabilities/cluster-operations/troubleshooting)
* [Grafana dashboards](/corecapabilities/observability/grafana)


# Query alerts
Source: https://docs.startree.ai/corecapabilities/observability/alerts/queries

Alerts that watch query errors, timeouts, quota rejections, and broker availability: when each fires, what it means, and what you or StarTree can do.

Whether queries succeed and return on time. Error-rate alerts count only Pinot-side errors, not bad SQL.

> [!NOTE]
>
> **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the [overview](/corecapabilities/observability/alerts/overview) for how to read an entry.

## Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

```mermaid placement="top-right" actions={true} theme={null}
flowchart TD
  A["Queries fail or are slow"] --> B{"What does the response say?"}
  B -- "429 / quota exceeded" --> B1["QueryQuotaAlert
raise quota.maxQueriesPerSecond"]
  B -- "Timeout (250, 240, 400)" --> C{"Server CPU or GC high?"}
  C -- "Yes" --> C1["See Platform health
scale or spread load"]
  C -- "No" --> C2["HighQueryTimeoutPercentage
expensive query: rewrite or index"]
  B -- "Segments missing or unavailable (235, 305)" --> D["See Data availability"]
  B -- "Server not responding (427)" --> C1
  B -- "Connection refused / 5xx" --> E["LowBrokerAvailability
NoBrokerAvailable"]
  B -- "Other internal errors" --> F["HighSystemQueryErrorRate
Broker dashboard: Query Error Code Rate"]
```

## Critical alerts

### `HighQueryCriticalErrorRate`

| Field | Detail |
| - | - |
| **Fires when** | Over 2% of queries hit a critical error for 5 min, with more than 2 queries/s. |
| **What it means** | Queries are failing right now. |
| **Check first** | • Pinot - Broker → **Critical Error One Minute Rate** • **Query Error Code Rate** to see which code |
| **What you can do** | If one query shape dominates, rewrite it |
| **What StarTree does** | • Restarts a faulty server • Scales the cluster |

### `HighSystemQueryErrorRate`

| Field | Detail |
| - | - |
| **Fires when** | Pinot-side errors exceed 5% of queries for 15 min, with more than 10 queries/s. |
| **What it means** | Errors come from Pinot, not from your SQL. |
| **Check first** | • Pinot - Broker → **Query Error Code Rate** • 235 / 305 segment missing · 427 server not responding · 250 execution timeout · 240 scheduling timeout · 400 broker timeout · 720 planning |
| **What you can do** | 400 or 720: simplify the query (for example, large IN lists) |
| **What StarTree does** | • Compares ideal state with external view • Fixes the server behind the error code |

### `HighQueryTimeoutPercentage`

| Field | Detail |
| - | - |
| **Fires when** | Over 50% of a table's queries time out, for 5 min. |
| **What it means** | Timeouts are the main failure for that table. |
| **Check first** | • Execution timeouts with no server-not-responding errors: an expensive query • Both rising together: servers are saturated • Logging → **Slow Queries Drilldown** |
| **What you can do** | • Rewrite heavy queries or add indexes • Set `timeoutMs` on the table or per query (broker default 10 s) |
| **What StarTree does** | Adds capacity when servers are saturated |
| **See also** | [Memory-throttled scheduler: troubleshooting](/corecapabilities/query_data/advanced_operations/workload-isolation/memory-throttled-scheduler#troubleshooting) |

### `QueryQuotaAlert`

> [!TIP]
>
> Usually your fix: the cause is most often in your stream, source system, or table config.

| Field | Detail |
| - | - |
| **Fires when** | A table rejects queries over its QPS quota for 1 hour. |
| **What it means** | Pinot returns 429 for some queries. It's a quota, not an outage. |
| **Check first** | • Pinot - Broker → **Query Quota Exceeding** • The quota is split across brokers and enforced per second, so short bursts can trip it |
| **What you can do** | • Raise `quota.maxQueriesPerSecond` • Smooth out client bursts |

### `LowBrokerAvailability`

| Field | Detail |
| - | - |
| **Fires when** | Fewer than half the brokers are ready, for 5 min. |
| **What it means** | Query capacity is reduced. Brokers often fail from heavy queries. |
| **Check first** | Pinot - Broker → **Pod Restart Count**, **JVM Used Memory** |
| **What you can do** | • Use `DISTINCTCOUNTHLL` instead of exact distinct counts on large data • Cap very large result sizes |
| **What StarTree does** | • Restores brokers • Raises broker memory |

### `NoBrokerAvailable`

| Field | Detail |
| - | - |
| **Fires when** | No broker pods exist, for 5 min. |
| **What it means** | Every query fails. Ingestion continues. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Restores brokers immediately |

### `BrokerDataTableDeserializationExceptions`

| Field | Detail |
| - | - |
| **Fires when** | Brokers fail to read server responses, for 5 min. |
| **What it means** | Some queries fail while merging results. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Finds the server and version mismatch behind it |

### `NettyResponseFetchExceptions`

| Field | Detail |
| - | - |
| **Fires when** | Brokers fail to fetch responses from servers, for 15 min. |
| **What it means** | Some queries fail or return partial results. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Checks server health and network |

## Warning alerts

| Alert | Fires when | What it means | What you can do |
| - | - | - | - |
| `QueriesFailingWithUnavailableSegment` | Over 1% of queries hit unavailable segments, for 15 min. | Results are partial. See Data availability. | Nothing needed |
| `BrokerQueryExecutionLatency` | Broker p95 query time is over 5 s, for 10 min. | Queries are slow across the broker. | Look at Logging → Slow Queries Drilldown |
| `BrokerResponseAlert` | Over 50 queries answered with only some servers in 5 min. | Those results are partial. | Nothing needed |
| `BrokerNoServerFoundAlert` | Over 50 queries found no server for a segment in 5 min. | Those queries miss data. | Nothing needed |

## Metrics

The series below are available in [Grafana](/corecapabilities/observability/grafana) through the Prometheus data source. For how names, suffixes, and labels are built, see [Reading Pinot metrics](/corecapabilities/observability/alerts/overview#reading-pinot-metrics).

### Metrics worth watching

These are the ones to reach for first; they are not the complete set.

| Metric | What it tells you |
| - | - |
| `pinot_broker_queries_Count` | Single-stage query volume. The denominator for most error-rate calculations. |
| `pinot_broker_multiStageQueries_Count`, `pinot_broker_multiStageQueriesGlobal_Count` | Multi-stage engine query volume, per table and cluster-wide. |
| `pinot_broker_queryTotalTimeMs_95thPercentile` | Broker-side P95 latency, per table. |
| `pinot_broker_queryErrorBrokerTimeout_OneMinuteRate` and siblings | Error rate split by cause — `ExecutionTimeout`, `QuerySchedulingTimeout`, `ServerNotResponding`, `ServerSegmentMissing`, `BrokerSegmentUnavailable`, `QueryPlanning`, `SqlRuntime`, `Internal`, `MergeResponse`, `QueryCancellation`, `ServerShuttingDown`, `BrokerRequestSend`. Splitting by cause is what turns "queries are failing" into a diagnosis. |
| `pinot_broker_queryCriticalError_OneMinuteRate` | Errors classed as critical. |
| `pinot_broker_brokerResponsesWithTimeouts_Count` | Responses that timed out. |
| `pinot_broker_brokerResponsesWithUnavailableSegments_Count` | Responses returned with segments missing — results may be incomplete. |
| `pinot_broker_brokerResponsesWithPartialServersResponded_Count` | Responses built without every server answering. |
| `pinot_broker_noServerFoundExceptions_Count` | No server could serve the table. Routing or availability, not query shape. |
| `pinot_broker_queryQuotaExceeded_Count` | Queries rejected by quota. |
| `pinot_server_schedulingTimeoutExceptions_Count` | Queries that timed out waiting to be scheduled rather than while executing — the distinction between "too busy" and "too slow". |
| `pinot_server_numMissingSegments_Count` | Segments a server was asked for and did not have. |

For per-query rather than aggregate analysis, use [`system_query_log`](/corecapabilities/query_data/advanced_operations/query-logger) — metrics tell you the shape of the problem, the query log tells you which query.

### Metrics behind each alert

The Prometheus series each alert rule evaluates.

| Alert | Metrics in the rule |
| - | - |
| `HighQueryCriticalErrorRate` | `pinot_broker_queryCriticalError_OneMinuteRate`, `pinot_broker_queries_Count`, `pinot_broker_multiStageQueriesGlobal_Count` |
| `HighSystemQueryErrorRate` | `pinot_broker_queryErrorSqlRuntime_OneMinuteRate`, `pinot_broker_queryErrorInternal_OneMinuteRate`, `pinot_broker_queryErrorQuerySchedulingTimeout_OneMinuteRate`, `pinot_broker_queryErrorExecutionTimeout_OneMinuteRate`, `pinot_broker_queryErrorBrokerTimeout_OneMinuteRate`, `pinot_broker_queryErrorServerSegmentMissing_OneMinuteRate`, `pinot_broker_queryErrorBrokerSegmentUnavailable_OneMinuteRate`, `pinot_broker_queryErrorServerNotResponding_OneMinuteRate`, `pinot_broker_queryErrorBrokerRequestSend_OneMinuteRate`, `pinot_broker_queryErrorMergeResponse_OneMinuteRate`, `pinot_broker_queryErrorQueryCancellation_OneMinuteRate`, `pinot_broker_queryErrorServerShuttingDown_OneMinuteRate`, `pinot_broker_queryErrorQueryPlanning_OneMinuteRate`, `pinot_broker_queries_Count`, `pinot_broker_multiStageQueriesGlobal_Count` |
| `HighQueryTimeoutPercentage` | `pinot_broker_brokerResponsesWithTimeouts_Count`, `pinot_broker_queries_Count` |
| `QueryQuotaAlert` | `pinot_broker_queryQuotaExceeded_Count` |
| `LowBrokerAvailability` | `kube_statefulset_status_replicas_ready`, `kube_statefulset_status_replicas` |
| `NoBrokerAvailable` | `kube_pod_info` |
| `BrokerDataTableDeserializationExceptions` | `pinot_broker_exceptions_dataTableDeserialization_Count` |
| `NettyResponseFetchExceptions` | `pinot_broker_exceptions_responseFetch_Count` |
| `QueriesFailingWithUnavailableSegment` | `pinot_broker_brokerResponsesWithUnavailableSegments_Count`, `pinot_broker_queries_Count`, `pinot_broker_multiStageQueries_Count` |
| `BrokerQueryExecutionLatency` | `pinot_broker_queryTotalTimeMs_95thPercentile` |
| `BrokerResponseAlert` | `pinot_broker_brokerResponsesWithPartialServersResponded_Count` |
| `BrokerNoServerFoundAlert` | `pinot_broker_noServerFoundExceptions_Count` |

## Related

* [Alerts and metrics overview](/corecapabilities/observability/alerts/overview): severity, the symptom router, the alert index, and how metric names are built.
* [Troubleshooting by feature: Queries, indexes and performance](/corecapabilities/observability/troubleshooting-by-feature#queries-indexes-and-performance)
* [Query troubleshooting](/corecapabilities/query_data/troubleshooting)
* [Error codes](/corecapabilities/observability/error-codes)


# Real-time ingestion alerts
Source: https://docs.startree.ai/corecapabilities/observability/alerts/real-time-ingestion

Alerts that watch real-time ingestion from Kafka, Kinesis, and other streams: when each fires, what it means, and what you or StarTree can do.

Freshness and completeness of data coming from Kafka, Kinesis and other streams. These page first because stale data is usually what users notice.

> [!NOTE]
>
> **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the [overview](/corecapabilities/observability/alerts/overview) for how to read an entry.

## Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

```mermaid placement="top-right" actions={true} theme={null}
flowchart TD
  A["New data is late or missing"] --> B{"GET /tables/{t}/pauseStatus
Is consumption paused?"}
  B -- "Paused by a resource limit" --> B1["ResourceUtilizationLimitExceeded
free disk or reduce keys/segments"]
  B -- "Paused by you" --> B2["Resume consumption"]
  B -- "Not paused" --> C{"Is any replica consuming
the partition?"}
  C -- "No" --> D{"Consumer creation errors?"}
  D -- "Yes" --> D1["PartitionConsumerCreateExceptions
credentials, TLS, ACLs, topic exists"]
  D -- "No" --> D2{"Server restarting
or out of memory?"}
  D2 -- "Yes" --> D3["See Platform health"]
  D2 -- "No" --> D4["RealtimeIngestionStopped
consuming segment in ERROR: reset it"]
  C -- "Yes" --> E{"Rows arriving
but dropped?"}
  E -- "Decode or transform errors" --> E1["RealtimeIngestionHighRowsWithErrorsRate"]
  E -- "Filtered out" --> E2["RealtimeIngestionHighSkippedRowsRate"]
  E -- "No" --> F{"Is the backlog growing?"}
  F -- "Yes, throttled" --> F1["Raise topic.consumption.rate.limit"]
  F -- "Yes, not throttled" --> F2["HighRealtimeIngestionOffsetLag
producer burst, skew, CPU"]
  F -- "One server only" --> F3["NoRealtimeIngestionPerServer
often a segment commit"]
  F -- "No backlog" --> F4["Producer is sending old timestamps
not a Pinot delay"]
  A --> G{"A gap, not a delay?"}
  G -- "Yes" --> G1["StreamDataLoss
check stream retention"]
```

## Critical alerts

### `RealtimeIngestionStopped`

> [!NOTE]
>
> Often transient: this alert usually clears on its own.

| Field | Detail |
| - | - |
| **Fires when** | No replica has consumed a partition for 30 min, and the table isn't disabled or paused. |
| **What it means** | Data in that partition stops arriving. |
| **Check first** | • Pinot Server Ingestion → **Realtime Consumption Status by Table Partition** • `GET /tables/{table}/consumingSegmentsInfo` • `GET /tables/{table}/pauseStatus` |
| **What you can do** | • Confirm the topic still exists and credentials are valid • If paused by a resource limit, see ResourceUtilizationLimitExceeded |
| **What StarTree does** | • Resets the consuming segment • Restarts a stuck server |
| **See also** | [Pauseless consumption: debugging APIs](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#debugging-apis) |

### `HighRealtimeIngestionOffsetLag`

| Field | Detail |
| - | - |
| **Fires when** | Average lag is over 500,000 messages and still growing for 60 min. |
| **What it means** | Pinot is consuming but falling behind, so queries see older data. |
| **Check first** | • Pinot Server Ingestion → **Is Ingestion Throttled?** • **Ingestion Quota Utilization** (100% means throttled) • **Realtime Rows Consumed PerSecondRate** vs your producer rate |
| **What you can do** | • Raise `topic.consumption.rate.limit` if throttled • Fix a skewed partition key • Add partitions only on non-upsert tables; changing the partition count of a live upsert table isn't supported |
| **What StarTree does** | • Scales servers • Tunes the server consumption limit |
| **See also** | [Pauseless consumption: operational troubleshooting](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#operational-troubleshooting) [Upsert operations guide: supported configurations](/corecapabilities/manage-data/upsert-operations-guide#supported-configurations-at-a-glance) |

### `NoRealtimeIngestionPerServer`

> [!NOTE]
>
> Often transient: this alert usually clears on its own.

| Field | Detail |
| - | - |
| **Fires when** | On one server, a partition's delay grew 500 s+ in 10 min with no new offsets consumed. |
| **What it means** | One replica has stalled, so results can differ between replicas. |
| **Check first** | • Pinot Server Ingestion → **Last Realtime Segment Creation Duration Seconds** • **Realtime Consumption Status by Table Partition** |
| **What you can do** | Consider pauseless consumption for tables with long segment commits. It isn't supported with partial or full upsert |
| **What StarTree does** | • Restarts the consumer if it's stuck • Clears stale metrics |
| **See also** | [Pauseless consumption: operational troubleshooting](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#operational-troubleshooting) [Upsert operations guide: supported configurations](/corecapabilities/manage-data/upsert-operations-guide#supported-configurations-at-a-glance) |

### `PartitionConsumerCreateExceptions`

> [!TIP]
>
> Usually your fix: the cause is most often in your stream, source system, or table config.

| Field | Detail |
| - | - |
| **Fires when** | Any failure creating a stream consumer in the last 5 min. |
| **What it means** | A server can't connect to your stream. |
| **Check first** | • One server: a host problem. All servers on one topic: the topic. All topics: credentials or network • Logging / Application Logs, filtered to the server |
| **What you can do** | Check credentials, TLS certificates, ACLs and that the topic exists |
| **What StarTree does** | Force-commits or resets the partition once the cause is fixed |

### `RealtimeConsumptionExceptions`

| Field | Detail |
| - | - |
| **Fires when** | A server keeps hitting consumption exceptions for 30 min. |
| **What it means** | Fetches from the stream fail. After repeated errors the consumer stops. |
| **Check first** | • Pinot Server Ingestion → **Realtime Consumption Exceptions PerSecondRate** • Server logs for the exception class |
| **What you can do** | • Disconnects or timeouts: check stream broker health • Offset out of range: retention is too short • Schema registry errors: fix the schema |
| **What StarTree does** | Restarts consumption after the cause clears |

### `RealtimeIngestionHighRowsWithErrorsRate`

> [!TIP]
>
> Usually your fix: the cause is most often in your stream, source system, or table config.

| Field | Detail |
| - | - |
| **Fires when** | Over 50% of fetched rows fail to decode or transform, at more than 0.1 rows/s, for 30 min. |
| **What it means** | Rows are being dropped. |
| **Check first** | Pinot Server Ingestion → **Rows filtered due to decode exception** and **transform exception** |
| **What you can do** | Fix the producer's message format or the table's transform config |
| **What StarTree does** | Confirms which column or decoder fails |
| **See also** | [Decoder troubleshooting](/corecapabilities/observability/troubleshooting-by-feature#streaming-ingestion) |

### `StreamDataLoss`

> [!TIP]
>
> Usually your fix: the cause is most often in your stream, source system, or table config.

| Field | Detail |
| - | - |
| **Fires when** | A server finds a gap between the offset it expected and the first one the stream still has. Fires at once. |
| **What it means** | Some messages were never ingested. The alert clearing doesn't mean the data came back. |
| **Check first** | • Gap of 1 with no lag: a transactional topic, not real loss • Large gap after a restart: stream retention ran out • Large gap with high lag: ingestion was too slow |
| **What you can do** | • Set `stream.kafka.isolation.level` to `read_committed` for transactional topics • Keep stream retention at least 2× the segment flush time |
| **What StarTree does** | Confirms the gap and its cause |

## Warning alerts

| Alert | Fires when | What it means | What you can do |
| - | - | - | - |
| `RealtimeIngestionHighSkippedRowsRate` | Over 90% of rows are filtered out for 15 min. | Usually a `filterConfig` that drops more than intended. | Check the table's filterConfig |
| `RealtimePartitionMismatch` | Rows land in a partition that doesn't match the table's partition config, for 30 min. | Partition pruning can return wrong results. | Match `functionName` and `numPartitions` to the producer, or stop pruning on that column |

## Metrics

The series below are available in [Grafana](/corecapabilities/observability/grafana) through the Prometheus data source. For how names, suffixes, and labels are built, see [Reading Pinot metrics](/corecapabilities/observability/alerts/overview#reading-pinot-metrics).

### Metrics worth watching

These are the ones to reach for first; they are not the complete set.

| Metric | What it tells you |
| - | - |
| `pinot_server_llcPartitionConsuming_Value` | `1` if the partition is consuming, `0` if it is not. The single most direct answer to "is ingestion running?" |
| `pinot_server_realtimeIngestionDelayMs_Value` | Age of the newest ingested record, per partition. Rising steadily means falling behind. |
| `pinot_server_realtimeIngestionOffsetLag_Value` | Offset distance behind the stream head. Compare its *direction* with the delay: a large but shrinking lag is a recovery. |
| `pinot_server_highestStreamOffsetConsumed_Value` | The last offset consumed. Flat while delay grows is the signature of a stuck consumer. |
| `pinot_server_realtime_consumptionExceptions_Count` | Consumption threw. The stream is reachable but records are failing. |
| `pinot_server_rowsWithErrors_Count` | Rows rejected during decode or transform. Almost always a schema mismatch. |
| `pinot_server_streamConsumerCreateExceptions_Count` | Could not create the consumer at all — credentials, topic, or network path. |
| `pinot_server_realtimePartitionMismatch_Count` | Pinot's partition count no longer matches the stream's. |
| `pinot_controller_tableConsumptionPaused_Value` | `1` when consumption is paused. Check this before diagnosing "stopped" ingestion. |
| `pinot_controller_tableDisabled_Value` | `1` when the table is disabled. |

### Metrics behind each alert

The Prometheus series each alert rule evaluates.

| Alert | Metrics in the rule |
| - | - |
| `RealtimeIngestionStopped` | `pinot_server_llcPartitionConsuming_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |
| `HighRealtimeIngestionOffsetLag` | `pinot_server_realtimeIngestionOffsetLag_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |
| `NoRealtimeIngestionPerServer` | `pinot_server_realtimeIngestionDelayMs_Value`, `pinot_server_highestStreamOffsetConsumed_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |
| `PartitionConsumerCreateExceptions` | `pinot_server_streamConsumerCreateExceptions_Count` |
| `RealtimeConsumptionExceptions` | `pinot_server_realtime_consumptionExceptions_Count` |
| `RealtimeIngestionHighRowsWithErrorsRate` | `pinot_server_rowsWithErrors_Count`, `pinot_server_realtimeRowsFetched_Count` |
| `StreamDataLoss` | `pinot_server_streamDataLoss_Value`, `pinot_server_documentCount_Value`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |
| `RealtimeIngestionHighSkippedRowsRate` | `pinot_server_realtimeRowsFiltered_Count`, `pinot_server_realtimeRowsConsumed_Count` |
| `RealtimePartitionMismatch` | `pinot_server_realtimePartitionMismatch_Count`, `pinot_controller_tableDisabled_Value`, `pinot_controller_tableConsumptionPaused_Value` |

## Related

* [Alerts and metrics overview](/corecapabilities/observability/alerts/overview): severity, the symptom router, the alert index, and how metric names are built.
* [Troubleshooting by feature: Streaming ingestion](/corecapabilities/observability/troubleshooting-by-feature#streaming-ingestion)
* [Ingestion troubleshooting](/corecapabilities/ingestdata/troubleshooting)
* [Pauseless consumption](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption)


# Upsert table alerts
Source: https://docs.startree.ai/corecapabilities/observability/alerts/upsert-tables

Alerts that watch upsert and dedup tables: primary-key growth, compaction, snapshots, RocksDB, and replica consistency.

Primary-key growth, compaction and snapshots for upsert and dedup tables.

> [!NOTE]
>
> **Critical** alerts page StarTree on-call 24/7. **Warning** alerts reach StarTree's alert channels without paging anyone. Thresholds are defaults; StarTree may tune them for your environment. See the [overview](/corecapabilities/observability/alerts/overview) for how to read an entry.

## Decision tree

Follow the tree from the symptom to the alert most likely to explain it. Use the zoom controls at the top right of the diagram to enlarge it.

```mermaid placement="top-right" actions={true} theme={null}
flowchart TD
  A["Upsert table looks wrong"] --> B{"Same query, different
results per replica?"}
  B -- "Yes" --> B1["UpsertInconsistentReplicas
enforceConsumptionInOrder"]
  B -- "No" --> C{"Storage growing
faster than keys?"}
  C -- "Yes" --> C1["HighTotalDocsToPKCountRatio
compaction is behind"]
  C -- "No" --> D{"Key count near the
per-server limit?"}
  D -- "Yes" --> D1["HighTotalNumberOfPrimaryKeysPerServer
more servers, or metadataTTL where supported"]
  D -- "No" --> E["SnapshotCreationFailed
slow restarts, no query impact"]
```

## Critical alerts

### `HighTotalNumberOfPrimaryKeysPerServer`

| Field | Detail |
| - | - |
| **Fires when** | A server holds over 2 billion primary keys, for 30 min. |
| **What it means** | Early warning before the key limit pauses ingestion. |
| **Check first** | Pinot Upsert → **Upsert Table Primary Key Count (per server)** |
| **What you can do** | Set `metadataTTL` on the upsert config, except on tables that run `SegmentRefreshTask`. Ask StarTree support before changing it on a live table |
| **What StarTree does** | Spreads partitions across more servers |
| **See also** | [Upsert operations guide: supported configurations](/corecapabilities/manage-data/upsert-operations-guide#supported-configurations-at-a-glance) |

### `HighTotalDocsToPKCountRatio`

| Field | Detail |
| - | - |
| **Fires when** | An upsert table holds over 10 documents per primary key, for 1 hour. |
| **What it means** | Compaction is behind, so storage and cost grow. |
| **Check first** | A daily sawtooth is normal. A floor that keeps rising means compaction is failing |
| **What you can do** | Run compaction more often |
| **What StarTree does** | Fixes failing compaction tasks |
| **See also** | [Upsert operations guide: troubleshooting](/corecapabilities/manage-data/upsert-operations-guide#troubleshooting) |

### `SnapshotCreationFailed`

| Field | Detail |
| - | - |
| **Fires when** | A snapshot task failed in the last hour. |
| **What it means** | No query impact. Restarts take longer. |
| **Check first** | Pinot Upsert → **Snapshot Creation Failure Rate** |
| **What you can do** | Check deep store permissions and the partition config |
| **What StarTree does** | Fixes the task |
| **See also** | [Upsert operations guide: troubleshooting](/corecapabilities/manage-data/upsert-operations-guide#troubleshooting) |

### `RocksDBHighMemoryUsage`

| Field | Detail |
| - | - |
| **Fires when** | RocksDB memtables on a server exceed 10 GB. |
| **What it means** | Upsert metadata is using too much memory. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Tunes RocksDB or adds memory |

### `RocksDBHighFlushRate`

| Field | Detail |
| - | - |
| **Fires when** | RocksDB flushes more than 500 times per 2 min on a server. |
| **What it means** | Heavy write churn on upsert metadata. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Tunes RocksDB |

### `RocksDBHighCompactionRate`

| Field | Detail |
| - | - |
| **Fires when** | RocksDB compacts more than 10 times per 2 min on a server. |
| **What it means** | Heavy write churn on upsert metadata. |
| **What you can do** | Nothing. StarTree handles this one. |
| **What StarTree does** | Tunes RocksDB |

## Warning alerts

| Alert | Fires when | What it means | What you can do |
| - | - | - | - |
| `UpsertInconsistentReplicas` | Keys weren't replaced while replacing segments, for 2 min. | Replicas can disagree; partial upserts can show defaults. | Set `enforceConsumptionInOrder` and `parallelSegmentConsumptionPolicy: DISALLOW_ALWAYS` |

## Metrics

The series below are available in [Grafana](/corecapabilities/observability/grafana) through the Prometheus data source. For how names, suffixes, and labels are built, see [Reading Pinot metrics](/corecapabilities/observability/alerts/overview#reading-pinot-metrics).

### Metrics worth watching

These are the ones to reach for first; they are not the complete set.

| Metric | What it tells you |
| - | - |
| `pinot_server_upsertPrimaryKeysCount_Value` | Primary keys held in memory. Grows with distinct keys, not rows — the number that outgrows a cluster silently. |
| `pinot_server_documentCount_Value` | Documents per table. Its ratio to primary-key count tells you whether compaction is keeping up. |
| `pinot_server_realtimeUpsertInconsistentRows_Count` | Rows that could not be reconciled between replicas. |
| `pinot_server_partialUpsertKeysNotReplaced_Count` | Partial-upsert keys not replaced during segment replacement. |
| `pinot_server_rocksdb_ticker_total_memtable_bytes` | Off-heap upsert metadata memory. |
| `pinot_server_rocksdb_histogram_flush_time_count`, `pinot_server_rocksdb_histogram_compaction_time_count` | Off-heap upsert write amplification. |

### Metrics behind each alert

The Prometheus series each alert rule evaluates.

| Alert | Metrics in the rule |
| - | - |
| `HighTotalNumberOfPrimaryKeysPerServer` | `pinot_server_upsertPrimaryKeysCount_Value` |
| `HighTotalDocsToPKCountRatio` | `pinot_server_documentCount_Value`, `pinot_server_upsertPrimaryKeysCount_Value`, `pinot_controller_cronSchedulerJobScheduled_Value` |
| `SnapshotCreationFailed` | `pinot_minion_snapshotCreationFailure_Count` |
| `RocksDBHighMemoryUsage` | `pinot_server_rocksdb_ticker_total_memtable_bytes` |
| `RocksDBHighFlushRate` | `pinot_server_rocksdb_histogram_flush_time_count` |
| `RocksDBHighCompactionRate` | `pinot_server_rocksdb_histogram_compaction_time_count` |
| `UpsertInconsistentReplicas` | `pinot_server_realtimeUpsertInconsistentRows_Count`, `pinot_server_partialUpsertKeysNotReplaced_Count` |

## Related

* [Alerts and metrics overview](/corecapabilities/observability/alerts/overview): severity, the symptom router, the alert index, and how metric names are built.
* [Upsert operations guide](/corecapabilities/manage-data/upsert-operations-guide)
* [Diagnosing duplicates and count mismatches](/corecapabilities/manage-data/offheap-upsert#diagnosing-duplicates-and-count-mismatches)
* [Troubleshooting by feature: Upserts and dedup](/corecapabilities/observability/troubleshooting-by-feature#upserts-and-dedup)


# Data Portal Error Codes
Source: https://docs.startree.ai/corecapabilities/observability/error-codes

Every failure the Data Portal reports carries a stable DPE-#### code. Look one up here for what it means and what to do next.

The Data Portal is the console where you connect data sources, create and configure tables, run
ingestion, and query your data. When one of those actions cannot complete, the message it shows
carries a code of the form `DPE-####` — short for **D**ata **P**ortal **E**rror.

The code names the condition, not the wording. Copy gets improved over time; the code does not
change with it, so it is safe to search for on this page, to bookmark, or to quote to support. The
same codes appear in Data Portal API responses, so a failure reads the same whether you hit it in
the console or from a script.

This page lists every code the product can return, what each one means, and what to do about it.
If you do not have a code in hand, start from [Troubleshooting](/corecapabilities/observability/troubleshooting)
instead, which routes by symptom.

> [!NOTE]
>
> Alongside the code, each error shows a short **reference**. That identifies your specific
> occurrence rather than the class of problem, so include it when you contact support — it is
> what lets us find the exact event in our logs.

## How to read an entry

Every entry says what happened and, where you can act on it, what to do next. The category tells
you whether acting is worth your time at all:

| Category | What it tells you |
| - | - |
| **User correctable** | Something in the request or configuration needs changing. The steps say what. |
| **Transient** | A temporary condition. Retrying the same action usually succeeds. |
| **Upstream** | A service StarTree Cloud depends on did not respond as expected. Not something you can fix from the console. |
| **Internal** | A defect on our side. Quote the reference shown with the error when you contact support. |
| **Informational** | Not a failure — nothing is broken. Most describe a state you can change if you want a different outcome, and say how. |

Some entries go on to give steps for particular kinds of connection, under **On** and the connection
type — for example **On AWS Glue**. When your connection is one of those listed, the Data Portal
shows you those steps in place of the general ones.

The first digit groups the code by the area it comes from:

| Prefix | Area |
| - | - |
| `DPE-0000` | The general code, for a failure no more specific code describes. Listed under [Platform](#platform) |
| `DPE-1###` | Connections and data sources |
| `DPE-2###` | Schema and table configuration |
| `DPE-3###` | Ingestion and tasks |
| `DPE-4###` | Queries |
| `DPE-5###` | Platform and request validation |

[Connections and data sources](#connections-and-data-sources) | [Platform](#platform) | [Request validation](#request-validation) | [Ingestion and tasks](#ingestion-and-tasks) | [Queries](#queries) | [Schema and table configuration](#schema-and-table-configuration)

## Connections and data sources

### DPE-1041 — Not enough records to sample

The source was reached, but no records came back to sample, so the schema could not be inferred.

**User correctable** — you can correct this.

**What to do**

1. For a stream, confirm the topic still holds messages: the sample starts at the earliest retained message, so an empty or fully expired topic returns nothing.
2. Check Data Format on the Data Setup step matches what producers write. Messages that do not decode are skipped, so a wrong format samples nothing.
3. For files, confirm the selected files contain rows, not only a header or empty marker files.
4. Then select Show Sample Data again, or Fetch Preview on the dataset's Schema tab.

**On Upload File**

The uploaded file was read, but it has no rows to sample, so the schema could not be inferred.

1. Check the file holds rows below its header; a file with only a header, or an empty one, has nothing to sample.
2. For CSV, check Custom Delimiter and Skip File Header on Connection Setup match the file.
3. Then upload the file again and select Show Sample Data.

**On Kafka, Confluent Cloud, Redpanda, Aiven Kafka, WarpStream, StreamNative Kafka, Amazon MSK, Kinesis, Pulsar**

The stream was reached, but no messages came back to sample, so the schema could not be inferred.

1. Confirm the topic still holds messages: the sample starts at the earliest retained message, so an empty or fully expired topic returns nothing.
2. Check Data Format on the Data Setup step matches what producers write. Messages that do not decode are skipped, so a wrong format samples nothing.
3. Then select Show Sample Data again, or Fetch Preview on the dataset's Schema tab.

**On S3, GCS, ADLS**

The files were reached, but no rows came back to sample, so the schema could not be inferred.

1. Confirm the selected files contain rows, not only a header or empty marker files.
2. Check Data Format and the Include Glob Pattern on the Data Setup step select the data files.
3. Then select Show Sample Data again, or Fetch Preview on the dataset's Schema tab.

[Related documentation](/corecapabilities/ingestdata/dataportal/streaming/kafka)

### DPE-1042 — Catalog rejected our credentials

The catalog refused to list its databases with the credentials on this connection.

**User correctable** — you can correct this.

**What to do**

1. For AWS Glue or Amazon S3 Tables, check the Metastore Region is the catalog's region, and that the Access Key and Secret Key are active or IAM Role ARN names a role that trusts StarTree.
2. Grant that identity the catalog actions in the IAM Policy under IAM Policy Creation Steps, such as glue:GetDatabases or s3tables:ListNamespaces.
3. For Unity Catalog, replace an expired or revoked Personal Access Token.
4. Then select Test Connection, or Edit, Test and Update on the dataset's Connection tab.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1043 — Catalog configuration incomplete

This connection was saved without the catalog settings needed to browse it, such as which catalog it points at.

**User correctable** — you can correct this.

**What to do**

1. Create the connection again with Connect external table: choose the catalog under Select Source Type, fill in Connection Setup and select Test Connection. The form sets the catalog type for you.
2. If the connection was created through the API, send catalogType together with that catalog's catalog.\* properties.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1044 — Kafka credentials rejected

The Kafka brokers rejected the Username and Password on this connection for its SASL mechanism.

**User correctable** — you can correct this.

**What to do**

1. Check Username and Password for a typo or a rotated password.
2. Confirm the brokers accept the selected SASL Mechanism and the user exists for it. SCRAM SHA 256 and SCRAM SHA 512 credentials are created separately, so a user made for one is rejected by the other.
3. Correct them on Connection Setup and select Test Connection. For an existing dataset, select Edit on its Connection tab, change them, then Test and Update.

**On Confluent Cloud**

Confluent Cloud refused the API Key and API Secret on this connection. A key that was deleted, one created for another cluster, or a Cloud API key used in place of a cluster API key is refused this way.

1. In Confluent Cloud, open the cluster, then API Keys, and create a key for this cluster.
2. Enter it in API Key and API Secret on Connection Setup, or with Edit on the Connections page or the dataset's Connection tab.
3. Click Test, then Update.

[Related documentation](/recipes/kafka-sasl)

### DPE-1045 — Could not create the stream consumer

The Kafka client could not be created from this connection's settings, so the brokers were never contacted.

**User correctable** — you can correct this.

**What to do**

1. Check Broker URL is a comma-separated list of host:port entries, and that each host name resolves from StarTree.
2. If Security Protocol is SSL, paste Server Certificate, Client Certificate and Client Key as complete PEM blocks, including their BEGIN and END lines.
3. Correct it on Connection Setup and select Test Connection. For an existing dataset, select Edit on its Connection tab, change it, then Test and Update.

[Related documentation](/corecapabilities/ingestdata/dataportal/streaming/kafka)

### DPE-1046 — The decoder for this data format could not be set up

The settings for the selected Data Format could not be turned into a working message decoder, so no messages were read.

**User correctable** — you can correct this.

**What to do**

1. For Kafka with AVRO, fill in Schema Registry URL, plus Schema Registry Key and Schema Registry Secret if the registry requires them.
2. For PROTOBUF with a Descriptor File, give a Descriptor File Path StarTree can read and the Proto Class Name.
3. For Kinesis with AVRO, Schema must be a valid Avro schema in JSON.
4. Fix it on the Data Setup step, then select Show Sample Data.

**On Kafka, Confluent Cloud, Redpanda, Aiven Kafka, WarpStream, StreamNative Kafka, Amazon MSK**

The settings for the selected Data Format could not be turned into a working message decoder, so no messages were read.

1. For AVRO or JSON, check Decoder Class Name, and fill in Schema Registry URL, plus Schema Registry Key and Schema Registry Secret if the registry requires them.
2. For PROTOBUF, check Schema Source and the settings it asks for.
3. Fix it on the Data Setup step, then select Show Sample Data.

**On Kinesis**

The settings for the selected Data Format could not be turned into a working message decoder, so no records were read.

1. For AVRO, Schema must be a valid Avro schema in JSON.
2. Fix it on the Data Setup step, then select Show Sample Data.

[Related documentation](/corecapabilities/ingestdata/dataportal/streaming/kafka)

### DPE-1047 — Delta table could not be read

The Delta table could not be opened at the configured location.

**User correctable** — you can correct this.

**What to do**

1. Point the table URI at the Delta table's root folder, the one that contains \_delta\_log, using the s3a:// scheme.
2. Confirm the access key and secret key, or the role, can list and read that folder, including \_delta\_log.
3. Send the same test or preview request again. Delta Lake is not offered in the Data Portal wizard, so this is checked through the API only.

[Related documentation](/corecapabilities/ingestdata/adv-concepts/batch/delta-lake-connector)

### DPE-1048 — Preview could not read the source data

We reached the source but could not read the data needed to build a preview.

**User correctable** — you can correct this.

**What to do**

1. Set the data format to the files' real format: Data Format on the Data Setup step, or inputFormat in the request when calling the API.
2. Show details lists the files that could not be decoded and the format that was tried.
3. Then run the preview again.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/s3)

### DPE-1049 — Not authorized to preview this table

Your account is not allowed to read the connection or table this preview uses.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. Ask a workspace administrator for a role that can read this connection and its tables, then reload the page.

[Related documentation](/corecapabilities/security/manage-security)

### DPE-1050 — No files matched the path

There were no data files to sample at the location you chose: the catalog table has no Parquet files yet, or nothing under the path matched.

**User correctable** — you can correct this.

**What to do**

1. If you picked a catalog table, confirm it holds data. A new or emptied table has no files to preview.
2. If you picked a path, choose a Prefix, or an S3 Input Directory URI and Include Glob Pattern, that reaches the Parquet files rather than a parent or sibling folder.
3. Then select Show Sample Data again.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1051 — Preview is busy right now

Too many previews are running in this environment at once, so this one was turned away before it read any data. Nothing is wrong with the connection.

**Transient** — temporary; retrying usually works.

**What to do**

1. Wait a few seconds, then run the preview again.

### DPE-1052 — A setting on this connection was not accepted

A setting on this connection or its data setup was checked and rejected before any data was read: a sampling limit, a file format, or a catalog setting outside what the source accepts.

**User correctable** — you can correct this.

**What to do**

1. Select Show details: it names the setting and the value it accepts.
2. Correct it where it is set: Data Format, Include Glob Pattern or Additional Config on the Data Setup step, or the connection's own settings for a catalog. If it names a region, set Region to the bucket's region.
3. Then select Show Sample Data, or Test Connection, again.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1053 — The file could not be read

The file `{file}` could not be read. It may be incomplete, or it may not be in the selected Data Format (`{format}`).

**User correctable** — you can correct this.

**What to do**

1. Set Data Format to the file's real format, on the Data Setup step (on Connection Setup for Upload File).
2. If the folder also holds other files, such as \_SUCCESS markers or manifests, add an Include Glob Pattern that matches only the data files, such as one ending in \*.parquet. The smallest file is read first, so one stray file fails the preview.
3. If Show details mentions Access Denied, grant this connection read access to that path.
4. Then select Show Sample Data again.

**On Upload File**

The file `{file}` could not be read. It may be incomplete, or it may not be in the selected Data Format (`{format}`).

1. Set Data Format on Connection Setup to the file's real format.
2. If the format is right, the file may be cut short or damaged; export it again and upload it.
3. Then select Show Sample Data again.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/s3)

### DPE-1054 — No files matched that path

Nothing could be read at `{path}`: no files there matched, or the storage service refused to list it.

**User correctable** — you can correct this.

**What to do**

1. Check S3 Input Directory URI (or the GCS or ADLS equivalent) for a typo, and that files exist under it.
2. On the Data Setup step, check the Include Glob Pattern does not exclude every file.
3. If Show details says Access Denied, grant this connection's identity s3:ListBucket on the bucket. If it says to use the specified endpoint, set Region to the bucket's region.
4. Then select Test Connection or Show Sample Data again.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/s3)

### DPE-1055 — Access was denied while reading this table

The credentials on this connection were refused the `{action}` permission needed to read the table, so it could not be previewed.

**User correctable** — you can correct this.

**What to do**

1. Grant `{action}` to the identity on this connection: the Metastore credentials for catalog actions, the Storage credentials for s3: and kms: actions. For Unity Catalog, grant it in Databricks to the principal behind this connection.
2. A new grant can take a minute to apply.
3. Then select Show Sample Data, or Fetch Preview on the dataset's Schema tab. Test Connection only lists the catalog, so it cannot confirm this grant.

[Related documentation](/corecapabilities/security/manage-security)

### DPE-1056 — The storage credentials cannot read that data

The storage credentials on this connection were refused when reading the data file `{path}`, so the table could not be previewed or created. The catalog listed the table; only reading its files was denied.

**User correctable** — you can correct this.

**What to do**

1. In AWS IAM, allow s3:GetObject on that bucket and prefix for the Storage identity: the user behind Access Key, the role in IAM Role ARN, or the role in IAM Role Name.
2. If the files are encrypted with a KMS key, allow kms:Decrypt on that key for the same identity.
3. Then click Show Sample Data again. Test Connection does not read data files, so it can pass while this read is refused.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1057 — The catalog would not let us sign in

The catalog refused to sign this connection in, so its tables could not be listed. Credentials that expired or were revoked, AWS keys that are no longer accepted, or credentials issued for a different workspace or account than the catalog all end this way.

**User correctable** — you can correct this.

**What to do**

1. Open the connection from the Connections page or the dataset's Connection tab and click Edit.
2. Under Metastore, re-enter what the authentication type asks for: Personal Access Token, OAuth Client ID and OAuth Client Secret, or Access Key and Secret Key, from the workspace or account that owns the catalog.
3. Click Test, then Update.

**On Databricks Unity Catalog**

1. Open the connection from the Connections page or the dataset's Connection tab and click Edit.
2. Under Metastore, re-enter the Personal Access Token, or the OAuth Client ID and OAuth Client Secret, from the Databricks workspace that owns the catalog.
3. Click Test, then Update.

**On Apache Polaris, Snowflake Open Catalog**

1. Open the connection from the Connections page or the dataset's Connection tab and click Edit.
2. Under Metastore, re-enter the OAuth Client ID and OAuth Client Secret of a principal in the account that owns the catalog.
3. Click Test, then Update.

**On Snowflake Horizon Catalog**

1. Open the connection from the Connections page or the dataset's Connection tab and click Edit.
2. Under Metastore, re-enter the Personal Access Token (PAT) of a Snowflake user in the account that owns the catalog.
3. Click Test, then Update.

**On AWS Glue**

1. Open the connection from the Connections page or the dataset's Connection tab and click Edit.
2. Under Metastore, re-enter the Access Key and Secret Key, or the IAM Role ARN, of an identity in the AWS account that owns the catalog.
3. Click Test, then Update.

**On Amazon S3 Tables**

1. Open the connection from the Connections page or the dataset's Connection tab and click Edit.
2. Check Table Bucket ARN names a table bucket in the AWS account the connection signs in to, then re-enter the Access Key and Secret Key, or the IAM Role ARN, of an identity in that account.
3. Click Test, then Update.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1058 — The bucket could not be listed

Listing bucket `{bucket}` was refused for the identity this connection uses, so its folders and files could not be browsed.

**User correctable** — you can correct this.

**What to do**

1. Allow s3:ListBucket on `{bucket}` and s3:GetObject on its objects for the user behind Access Key or the role in IAM Role ARN; for IAM Role, the IAM Policy under IAM Policy Creation Steps lists exactly these.
2. With IAM Node Role, the role in IAM Role Name is in StarTree's account, so grant it in the bucket policy. For a GCS bucket, give the HMAC key's service account read access.
3. Then click Test Connection, or Test and Update on a saved connection.

**On GCS Data Lake**

1. Check HMAC Access ID and HMAC Secret on the connection are an active key pair; Cloud Storage answers Access denied for a key it does not recognise as well.
2. Grant the service account that owns the key storage.objects.list on the bucket and storage.objects.get on its objects, for example through the Storage Object Viewer role.
3. Click Test, then Update.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/s3)

### DPE-1059 — The streaming source rejected our credentials

The streaming source refused this connection's credentials while the topic's partitions were being read, so no sample could be fetched.

**User correctable** — you can correct this.

**What to do**

1. Open the connection from the Connections page or the dataset's Connection tab, click Edit, and re-enter API Key and API Secret (Confluent Cloud) or Username, Password and SASL Mechanism.
2. Use a key created for this Kafka cluster rather than a global or Cloud API key, and allow its principal to describe and read the topic.
3. Click Test, then Update, then click Show Sample Data again.

[Related documentation](/corecapabilities/ingestdata/dataportal/streaming/kafka)

### DPE-1060 — The storage credentials were rejected

AWS rejected the storage credentials on this connection (`{reason}`). Keys that are wrong, inactive or temporary, or a role StarTree is not trusted to assume, are refused this way.

**User correctable** — you can correct this.

**What to do**

1. Edit the connection from the Connections page or the dataset's Connection tab.
2. With Access Key and Secret Key, re-enter both from one active, long-term key pair in AWS IAM.
3. With IAM Role, apply the Trust Policy under IAM Policy Creation Steps, with its StarTree Account ID and External ID, to the role in IAM Role ARN.
4. Click Test, then Update. If Reuse metastore credentials is Yes, fix them under Metastore.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1061 — That role could not be assumed

The role named in IAM Role ARN on this connection refused to let StarTree assume it, so the storage behind it could not be read.

**User correctable** — you can correct this.

**What to do**

1. Check IAM Role ARN is the exact ARN of the role in your AWS account.
2. In the AWS console, set that role's trust policy to the Trust Policy shown under IAM Policy Creation Steps on this connection. It names the StarTree principal and the External ID to require.
3. A trust policy change can take a minute to apply.
4. Then select Test on the dataset's Connection tab, or Test Connection in the wizard, rather than waiting for the next ingestion run.

**On Kinesis**

The role named in IAM Role ARN on this Kinesis connection refused to let StarTree assume it, so its streams could not be listed.

1. Check IAM Role ARN is the exact ARN of the role in your AWS account.
2. In AWS IAM, set that role's trust policy to trust the Startree IAM Role ARN shown on the connection form, and to require the External ID shown with it.
3. A trust policy change can take a minute to apply; then click Test Connection, or Test then Update on a saved connection.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/s3)

### DPE-1062 — The starting snapshot is not in this table

This dataset is set to start syncing `{table}` from snapshot `{snapshotId}`, which that table does not have, so its first sync cannot start. An id copied from another table, or one expired before the first sync, causes this.

**User correctable** — you can correct this.

**What to do**

1. On the dataset's Configuration tab, click Edit, remove iceberg.snapshotProcessing.initialSnapshotId and iceberg.snapshotProcessing.initialVersionSelector, then click Apply and Save; the next sync starts from the latest snapshot.
2. To start earlier, use an id from this table's own snapshot history.
3. If the dataset was created with SQL, recreate it without snapshot\_id.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1063 — This catalog type is not available in this environment yet

The connection type you picked is newer than the version of StarTree running in this environment, so its storage could not be opened for browsing, previewing or syncing. This is on our side; nothing you entered caused it.

**Internal** — a defect on our side.

**What to do**

1. Contact StarTree support and quote the reference shown with this error. The environment needs an upgrade before this connection type can be used.
2. There is nothing to change on the connection in the meantime. Once the upgrade is in place, click Test on it again.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1064 — That sign-in needs a one-time passcode

Snowflake refused the sign-in because the Database User on this connection must also enter a one-time passcode, which a stored password cannot supply.

**User correctable** — you can correct this.

**What to do**

1. In Snowflake, give StarTree a user that signs in with a key pair, ideally a service user with an RSA public key assigned. Key-pair sign-in does not ask for a passcode.
2. Open the connection from the Connections page or the dataset's Connection tab, click Edit, set Database User to that user, choose Key Pair Authentication and paste its Private Key, plus Private Key Password if the key is encrypted.
3. Click Test, then Update.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/snowflake)

### DPE-1065 — No readable Parquet files at that path

No readable Parquet file was found under `{path}`, so the table's columns could not be worked out. Only objects whose names end in .parquet are read.

**User correctable** — you can correct this.

**What to do**

1. On the Select Path step, set Prefix to the folder that holds the table's .parquet files, as deep as possible, then click Show Sample Data again.
2. Confirm the files' names end in .parquet; files without that extension are skipped.
3. If the files are still being written, try again once the writer has finished.

[Related documentation](/corecapabilities/external-table/getting-started/query-without-a-catalog)

### DPE-1066 — That table is no longer in the catalog

The catalog has no table `{table}`, so it could not be read. A table dropped or renamed at the source, or a connection whose Catalog Name now points at another catalog, causes this.

**User correctable** — you can correct this.

**What to do**

1. Check in the source catalog that the table still exists under that exact name, and that this connection's credentials can see it.
2. If the connection's Catalog Name was changed, set it back: Edit the connection, then click Test and Update.
3. If the table was renamed or moved, connect it as a new dataset with Connect external table on the Tables page.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1067 — That access key is not recognised

AWS does not recognise the Access Key on this connection, so the data could not be read. A key that was deleted or deactivated fails this way, and so does a temporary key (one starting ASIA), which also needs a session token, a field this form lacks.

**User correctable** — you can correct this.

**What to do**

1. In AWS IAM, check the key is listed as Active for its user, or create a new long-term access key for an IAM user.
2. Open the connection from the Connections page or the dataset's Connection tab, click Edit, and enter the new Access Key and Secret Key.
3. Click Test, then Update; the new key reaches every table on this connection that is still in sync.

**On Kinesis**

AWS does not recognise the Access Key on this Kinesis connection, so its streams could not be listed. A key that was deleted or deactivated fails this way. So does a temporary key (one starting ASIA), because this form has no field for its session token.

1. In AWS IAM, check the key is listed as Active for its user, or create a new long-term access key for an IAM user.
2. Set Authentication Type to Basic, then enter the key in Access Key and Secret Key, on Connection Setup or with Edit on a saved connection.
3. Click Test Connection, or Test then Update on a saved connection.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/s3)

### DPE-1068 — Object storage refused the request

Object storage answered Forbidden when we read an object. That answer carries no reason: a missing permission, a bucket policy denial and an inactive key look the same.

**User correctable** — you can correct this.

**What to do**

1. Grant this connection's identity s3:GetObject on the files and s3:ListBucket on the bucket. The IAM Policy under IAM Policy Creation Steps (shown for IAM Role) lists both.
2. Check no bucket policy denies that identity, and that any Access Key and Secret Key are still active.
3. Select Show Sample Data again to confirm. Test Connection only lists the bucket, so it can pass while reads are still refused.

### DPE-1069 — That table is not an Iceberg table

The table you picked is in the catalog but cannot be read as Iceberg, so it cannot be used as an external table. Unity Catalog says this of a Delta table without Iceberg reads turned on; AWS Glue of a table not registered as Iceberg.

**User correctable** — you can correct this.

**What to do**

1. For a Unity Catalog Delta table, enable Iceberg reads (UniForm) on it in Databricks, then pick it again on the Select Table step.
2. For AWS Glue, pick a table whose table type is ICEBERG. Plain Parquet files in S3 can be connected through an S3 Data Lake connection instead.
3. Views and tables in other formats cannot be used, even though they are listed.

**On Databricks Unity Catalog**

Unity Catalog lists this table, but it is a Delta table without Iceberg reads turned on, so it cannot be read as an external table.

1. In Databricks, enable Iceberg reads (UniForm) on this Delta table, then pick it again on the Select Table step.
2. Views and tables in other formats cannot be used, even though they are listed.

**On AWS Glue**

AWS Glue lists this table, but it is not registered as an Iceberg table, so it cannot be read as an external table.

1. Pick a table whose table type is ICEBERG. Plain Parquet files in S3 can be connected through an S3 Data Lake connection instead.
2. Views and tables in other formats cannot be used, even though they are listed.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1070 — The connection test could not run

The test could not start because no file system is set for this connection, so whether its other settings are correct is still unknown.

**Internal** — a defect on our side.

**What to do**

1. For a Custom connection, Datasource Properties must name the file system in input.fs.className, as the batch ingestion docs show; add it, then click Test Connection.
2. For any other source type, contact StarTree support and quote the reference shown with this error. Trying the test again gives the same result.

**On S3, GCS, ADLS**

The Connection Properties on this connection no longer name its file system, so the test could not start and whether the other settings are right is still unknown.

1. Restore the input.fs.className line in Connection Properties as the form filled it in; the quickest way is to start again from Create Connection, which fills it for you.
2. Then click Test Connection. For a saved connection, click Edit on the Connections page, restore it, then Test and Update.

[Related documentation](/corecapabilities/ingestdata/ingest_data_overview)

### DPE-1071 — The source took too long to answer

The source did not answer within 30 seconds while its topics were being listed, so the request stopped before it finished.

**Transient** — temporary; retrying usually works.

**What to do**

1. Try again in a moment.
2. If it times out again, check Broker URL, that Security Protocol matches the listener at that address, and that the cluster accepts traffic from StarTree through its allow-list or private link.

### DPE-1072 — The catalog's reply was cut off

The Iceberg catalog began sending the details of `{table}`, but the reply stopped partway through, so the table could not be read. When a reply stops at the same point on every attempt, a size limit is the usual cause, and trying again on its own will not help.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Expire old snapshots on `{table}`. The catalog sends the table's full snapshot history in this reply, and a long history makes it very large.
2. If a proxy or gateway sits in front of the catalog, check whether it limits the size of responses.
3. If the reply is still cut off, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1073 — StarTree's connections can't be changed

This connection was created by StarTree, so it can't be edited or deleted. StarTree sets it up again from its own configuration every time the service starts.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. To use different settings, create a new connection and build your tables from it.

### DPE-1074 — Databricks refused the OAuth scope this connection requests

Databricks would not sign this connection in because the scope StarTree requests for Unity Catalog, `{scope}`, is not assigned to its OAuth client. That happens when the client's OAuth secret was created for a narrower set of scopes.

**User correctable** — you can correct this.

**What to do**

1. In Databricks, open the service principal whose application ID is this connection's OAuth Client ID and generate a new OAuth secret that includes the `{scope}` scope.
2. Edit the connection, paste the new secret into OAuth Client Secret, then click Test and Update.
3. Alternatively, switch REST Authentication Type to Personal Access Token.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1075 — This catalog type is not available in this environment yet

The connection type you picked is newer than the version of StarTree running in this environment, so the catalog could not be opened. This is on our side; nothing you entered caused it.

**Internal** — a defect on our side.

**What to do**

1. Contact StarTree support and quote the reference shown with this error. The environment needs an upgrade before this connection type can be used.
2. There is nothing to change on the connection in the meantime. Once the upgrade is in place, click Test on it again.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1076 — AWS access keys are needed to sign in to this catalog

This connection signs in to AWS with an access key, but its Access Key, Secret Key or Region was empty. It also happens when IAM Role or IAM Node Role is chosen in an environment that cannot yet use a role for this catalog.

**User correctable** — you can correct this.

**What to do**

1. Edit the connection and, under Metastore, choose the Authentication Type that asks for an Access Key and Secret Key.
2. Fill in Access Key, Secret Key and Region, then click Test and Update. If Reuse metastore credentials is set to No, do the same under Storage.
3. To sign in with an IAM role instead, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1077 — The AWS account ID for this Glue catalog is missing

An AWS Glue catalog is identified by the AWS account that owns it, and this connection did not include that account ID, so the catalog could not be opened.

**User correctable** — you can correct this.

**What to do**

1. Edit the connection and fill in AWS Account ID with the 12-digit ID of the account that owns the Glue Data Catalog. Connections created with the older Iceberg REST option call this field Warehouse.
2. Click Test, then Update.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1078 — That catalog could not be found

The catalog service found no catalog matching the name saved on this connection, so its databases could not be listed. A misspelled name is the usual cause; a catalog the connection's identity is not allowed to see can look the same.

**User correctable** — you can correct this.

**What to do**

1. Check Catalog Name on the connection (Database on Snowflake Horizon Catalog) against the catalog's exact name in the source system.
2. Confirm the Catalog Endpoint URI points at the workspace or account that holds the catalog, and that the connection's identity has access to it.
3. Edit the connection to correct it, then click Test and Update.

**On Databricks Unity Catalog, Apache Polaris, Snowflake Open Catalog**

1. Check Catalog Name on the connection against the catalog's exact name in the source system.
2. Confirm Catalog Endpoint URI points at the workspace or account that holds the catalog, and that the connection's identity can use it.

**On Snowflake Horizon Catalog**

1. Check Database on the connection against the Snowflake database's exact name.
2. Confirm Catalog Endpoint URI points at the account that holds it.

**On AWS Glue**

1. Check AWS Account ID on the connection is the numeric ID of the account that owns the Glue Data Catalog.
2. Check Region under Metastore is the region of that catalog.

**On Amazon S3 Tables**

1. Check Table Bucket ARN on the connection is the full ARN of the table bucket, copied from the S3 console.
2. Check Region under Metastore is the region in that ARN.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1079 — The source table for this preview was not chosen

The preview did not know which table in the catalog to read: it needs a database and a table name, and at least one was missing, so no rows could be sampled.

**User correctable** — you can correct this.

**What to do**

1. Go back to Select Table (Select Path for an S3 or GCS data lake), choose the table or folder again, and preview once more. Choosing it fills in both values.
2. If you call the API directly, set both the namespace and the table name in the catalog's table settings. The setup flow uses default for both on an S3 or GCS data lake.
3. If the setup flow still shows this, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1080 — A decimal column could not be read

Column `{column}` stores its decimal values in an integer-based encoding that the version of StarTree in this environment cannot read, so the preview stopped. This is on our side; the data itself is fine.

**Internal** — a defect on our side.

**What to do**

1. Contact StarTree support and quote the reference shown with this error. A newer version reads this encoding, so the fix is an upgrade, not a change to your data.

### DPE-1081 — A data file for this table is missing from storage

The preview found `{file}` in the table's file list, but storage said the file does not exist when it was opened. A file deleted or moved after it was listed, by a cleanup job, a lifecycle rule or a rewrite in progress, causes this.

**User correctable** — you can correct this.

**What to do**

1. Check whether that file is still in the bucket.
2. If a cleanup job or lifecycle rule removed it, restore the file, or rewrite the table so it no longer lists the file.
3. If files under this path are being rewritten right now, wait for the write to finish and preview again.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1082 — No credentials were available to read the data files

The table's data files are in a bucket that needs AWS credentials, but no access key or role reached the reader that opens them, so the preview could not read them.

**User correctable** — you can correct this.

**What to do**

1. Edit the connection and, under Storage, choose the Authentication Type that asks for an Access Key and Secret Key (on AWS Glue and Amazon S3 Tables, first set Reuse metastore credentials to No).
2. Enter a key that can read the bucket holding the table's data files, set Region, then click Test and Update.
3. If keys were already set, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1083 — The JSON file stops partway through a record

Preview reads about the first 1 MB of a JSON file, and `{file}` ended partway through a record inside that part. Its records are too large, or are wrapped in one top-level object, to fit there, or the file itself is cut short.

**User correctable** — you can correct this.

**What to do**

1. Save the data as newline-delimited JSON, one compact record per line without indentation, so whole records fit in the part preview reads.
2. Do not wrap the records in a single top-level object. A list of records, or one record per line, both work.
3. If the file is under 1 MB, it is incomplete: export it again, replace or re-upload it, and preview the data again.

### DPE-1084 — The Kafka sign-in setting is not valid

The connection's SASL sign-in setting (sasl.jaas.config) does not start with a login module and the word required, so the stream consumer could not be created.

**User correctable** — you can correct this.

**What to do**

1. If you set sasl.jaas.config under Datasource Properties, give only its value: the login module name, the word required, username and password in double quotes, then a semicolon.
2. Remove anything before the module name, such as a pasted property name and equals sign, or a \*\*\*\*\*\* placeholder.
3. Or create a new connection with just Username and Password (API Key and API Secret for Confluent Cloud), then run Test Connection.

[Related documentation](/recipes/kafka-sasl)

### DPE-1085 — No AWS credentials are set on this connection

The Kinesis connection has no access key or role to sign in to AWS with, so the list of streams could not be read.

**User correctable** — you can correct this.

**What to do**

1. On the Connection Setup step, or after Edit on the dataset's Connection tab, set Authentication Type and fill in Access Key and Secret Key, or the IAM Role ARN for StarTree to assume.
2. Run Test Connection, or Test then Update on a saved connection; the streams load once AWS accepts the credentials.
3. If Authentication Type is already IAM Role, the environment could not get the identity it uses to assume your role: contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/ingestdata/dataportal/streaming/kinesis)

### DPE-1086 — The AWS identity lacks a Kinesis permission

AWS refused `{action}` for the user or role this connection signs in with: no policy attached to it allows that action, or one denies it.

**User correctable** — you can correct this.

**What to do**

1. In AWS IAM, allow `{action}` in a policy attached to that user or role.
2. Allow kinesis:ListShards, kinesis:GetShardIterator and kinesis:GetRecords on the stream as well; ingestion needs them to read it.
3. Run Test Connection again once the policy is saved. IAM changes can take a minute to apply.

[Related documentation](/corecapabilities/ingestdata/dataportal/streaming/kinesis)

### DPE-1087 — The Google Cloud key is incomplete

The Authentication Json on this connection is not a complete service account key, so the connection could not sign in to Google Cloud.

**User correctable** — you can correct this.

**What to do**

1. In the Google Cloud console, open IAM & Admin, then Service Accounts, choose the account, and add a new JSON key under Keys.
2. Paste the whole downloaded file into Authentication Json without editing it. It must include type, client\_id, client\_email, private\_key and private\_key\_id; a base64-encoded copy of the file also works.
3. Run Test Connection, or Test then Update on a saved connection.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/gcs)

### DPE-1088 — That AWS region is not valid

The Region on this connection is not an AWS region code, so the address used to sign in to AWS could not be built.

**User correctable** — you can correct this.

**What to do**

1. Set Region to the code of the region your bucket or stream is in, in lowercase with hyphens, for example us-east-1 or eu-west-2.
2. Run Test Connection before saving.

**On Kinesis**

The Region on this Kinesis connection is not an AWS region StarTree can reach, so its streams could not be listed. A region written with spaces or capitals, or a code that does not exist, fails this way.

1. Set Region to the code of the region the stream is in, in lowercase with hyphens, for example us-east-1 or eu-west-2.
2. Click Test Connection, or Test then Update on a saved connection.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/s3)

### DPE-1089 — The sample data connection could not sign in

The access key behind StarTree's built-in sample data connection is no longer accepted, so its files could not be listed. This is on our side, not a problem with your settings.

**Internal** — a defect on our side.

**What to do**

1. If your connection list includes the newer sample connection, s3DemoV4, use it. Datasets built from an older sample connection keep its expired key and need to be created again from the new one.
2. Otherwise contact StarTree support and quote the reference shown with this error; your environment needs an update to restore the sample data.

### DPE-1090 — The starting snapshot is not in this table

This dataset is set to start syncing `{table}` from a snapshot id the catalog does not have for this table, so the first sync stops before reading anything. An id copied from another table, or one since expired, causes this.

**User correctable** — you can correct this.

**What to do**

1. On the Configuration tab, click Edit, remove iceberg.snapshotProcessing.initialSnapshotId and iceberg.snapshotProcessing.initialVersionSelector, then click Apply and Save. The first sync then starts from the latest snapshot.
2. To start earlier, use an id from this table's own snapshot history.
3. If the dataset was created with SQL, recreate it without snapshot\_id.

### DPE-1091 — The catalog refused this table's sync sign-in

The lakehouse catalog refused the sign-in this table's sync used. A table keeps its own copy of its connection's sign-in details and syncs with that copy, so it can be refused even when Test Connection passes for the connection.

**User correctable** — you can correct this.

**What to do**

1. On the table's Connection tab, click Edit and re-enter the sign-in details the catalog refused: the Personal Access Token, or the OAuth Client ID and OAuth Client Secret.
2. If the tab shows the connection, click Test, then Update; every table in sync with the connection gets the new details. If it shows this table's own settings, click Save; it tests them first and changes this table only.
3. If the sync is still refused, this table's copy was not reached or still holds an older sign-in: open the Configuration tab, click Edit, and in the ExternalTableSyncTask settings correct the sign-in entries and delete those for a method the connection no longer uses; then Apply and Save.
4. Click Sync Now to run the sync again.

**On Databricks Unity Catalog**

1. On the table's Connection tab, click Edit and re-enter the sign-in the catalog refused: the Personal Access Token, or the OAuth Client ID and OAuth Client Secret, whichever the connection uses.
2. If the tab shows the connection, click Test, then Update; every table in sync with the connection gets the new details. If it shows this table's own settings, click Save; it tests them first and changes this table only.
3. If the sync is still refused, this table's copy was not reached or still holds an older sign-in: open the Configuration tab, click Edit, and in the ExternalTableSyncTask settings correct the sign-in entries and delete those for a method the connection no longer uses; then Apply and Save.
4. Click Sync Now to run the sync again.

**On Apache Polaris, Snowflake Open Catalog**

1. On the table's Connection tab, click Edit and re-enter the OAuth Client ID and OAuth Client Secret the catalog refused.
2. If the tab shows the connection, click Test, then Update; every table in sync with the connection gets the new details. If it shows this table's own settings, click Save; it tests them first and changes this table only.
3. If the sync is still refused, this table's copy was not reached or still holds an older sign-in: open the Configuration tab, click Edit, and in the ExternalTableSyncTask settings correct the sign-in entries and delete those for a method the connection no longer uses; then Apply and Save.
4. Click Sync Now to run the sync again.

**On Snowflake Horizon Catalog**

1. On the table's Connection tab, click Edit and re-enter the Personal Access Token (PAT) the catalog refused, with a Snowflake role restriction that matches OAuth Scopes.
2. If the tab shows the connection, click Test, then Update; every table in sync with the connection gets the new details. If it shows this table's own settings, click Save; it tests them first and changes this table only.
3. If the sync is still refused, this table's copy was not reached or still holds an older sign-in: open the Configuration tab, click Edit, and in the ExternalTableSyncTask settings correct the sign-in entries and delete those for a method the connection no longer uses; then Apply and Save.
4. Click Sync Now to run the sync again.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1092 — The catalog refused the requested OAuth scope

The catalog's sign-in service would not grant the OAuth scope this connection asks for, so the connection could not sign in to the catalog.

**User correctable** — you can correct this.

**What to do**

1. Set OAuth Scopes on the connection to a scope the catalog grants this client.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Apache Polaris, Snowflake Open Catalog**

1. Set OAuth Scopes to PRINCIPAL\_ROLE:ALL, or to PRINCIPAL\_ROLE: followed by a principal role granted to the client.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Snowflake Horizon Catalog**

1. Set OAuth Scopes to session:role: followed by the Snowflake role the PAT is restricted to; the two must name the same role.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Databricks Unity Catalog**

1. Unity Catalog accepts the all-apis scope. In the connection's Connection Properties, set catalog.iceberg-rest.auth.rest.oauthScopes back to all-apis, then click Test.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1093 — The OAuth sign-in address did not issue a token

The URI this connection uses to request OAuth tokens did not answer like an OAuth sign-in endpoint, so the connection could not sign in to the catalog.

**User correctable** — you can correct this.

**What to do**

1. In the connection's OAuth settings, enter the full https URI for requesting tokens, not a path.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Databricks Unity Catalog**

1. Set the URI for requesting OAuth tokens to the workspace URL followed by /oidc/v1/token.
2. Do not leave it empty: Unity Catalog has no path of its own for issuing tokens.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Apache Polaris, Snowflake Open Catalog, Snowflake Horizon Catalog**

1. Set the URI for requesting OAuth tokens to the Catalog Endpoint URI followed by /v1/oauth/tokens.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1094 — AWS rejected the access key and secret key

AWS refused the request because the access key and secret key on this connection do not work together: the secret does not belong to the key, or the key is not recognised.

**User correctable** — you can correct this.

**What to do**

1. Re-enter the access key and its matching secret wherever this connection uses a pair: under Metastore, under Storage, or both. A key that was rotated or deactivated at the source has to be replaced here as well.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On AWS Glue, Amazon S3 Tables**

1. Re-enter the Access Key and its matching Secret Key under Metastore, and under Storage as well when Reuse metastore credentials is No. A key rotated or deactivated in AWS has to be replaced here too.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Databricks Unity Catalog, Apache Polaris, Snowflake Open Catalog, Snowflake Horizon Catalog**

1. Re-enter the Access Key and its matching Secret Key under Storage. A key rotated or deactivated in AWS has to be replaced here too.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On GCS Data Lake**

1. Re-enter the HMAC Access ID and the HMAC Secret created with it, from the Interoperability settings of Cloud Storage.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1095 — The Region does not match the catalog's region

AWS refused the catalog request because it was signed for a different region from the catalog endpoint it was sent to.

**User correctable** — you can correct this.

**What to do**

1. Set Region to the region of the AWS catalog this connection reads.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On AWS Glue**

1. Set Region under Metastore to the region of your AWS Glue Data Catalog.
2. AWS Glue connections send catalog requests to the Catalog Endpoint URI in the form's Connection Properties, which defaults to us-west-1. For a catalog in another region, change the region in that address to match, for example glue.us-east-1.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Amazon S3 Tables**

1. Set Region under Metastore to the region in the Table Bucket ARN.
2. Amazon S3 Tables connections send catalog requests to the Catalog Endpoint URI in the form's Connection Properties, which defaults to us-west-1. For a table bucket in another region, change the region in that address to match, for example s3tables.us-east-1.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/glue/onboarding-data-portal)

### DPE-1096 — The connection's AWS identity lacks a permission

AWS refused `{action}` for the identity this connection uses, so the request could not be completed.

**User correctable** — you can correct this.

**What to do**

1. Add `{action}` to the IAM policy of the role or user this connection uses. Where the connection form offers IAM Policy Creation Steps, Show lists every permission it needs.
2. With IAM Node Role, grant it to the role shown under IAM Role Name.
3. AWS can take a minute to apply a policy change; then try again.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1097 — The catalog account lacks access to this data

The catalog accepted the sign-in but refused the request, because the account behind it has not been granted the privileges needed to read this catalog, schema or table.

**User correctable** — you can correct this.

**What to do**

1. Grant the account behind this connection's sign-in read access to the catalog, schema and table at the source.
2. Then test the connection again, or click Sync Now on an existing table.

**On Databricks Unity Catalog**

1. In Databricks, grant the user or service principal USE CATALOG on the catalog, USE SCHEMA and EXTERNAL USE SCHEMA on the schema, and SELECT on the table.
2. Enable external data access on the metastore if it is off.
3. Then test the connection again, or click Sync Now on an existing table.

**On Apache Polaris, Snowflake Open Catalog**

1. Grant the client's principal role a catalog role that can list namespaces and tables and read table data.
2. Then test the connection again, or click Sync Now on an existing table.

**On Snowflake Horizon Catalog**

1. In Snowflake, grant the role the PAT is restricted to USAGE on the database and schema, and SELECT on the table.
2. Then test the connection again, or click Sync Now on an existing table.

[Related documentation](/corecapabilities/external-table/unity/onboarding-data-portal)

### DPE-1098 — The catalog's host name could not be found

The host name in Catalog Endpoint URI could not be resolved, so StarTree could not reach the catalog.

**User correctable** — you can correct this.

**What to do**

1. Check Catalog Endpoint URI for a typo in the host name.
2. If the catalog is reachable only on a private network, contact StarTree support about private connectivity.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Databricks Unity Catalog**

1. Check Catalog Endpoint URI: https\://, your workspace host, then /api/2.1/unity-catalog/iceberg-rest.
2. If the catalog is reachable only on a private network, contact StarTree support about private connectivity.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Apache Polaris**

1. Check Catalog Endpoint URI: https\://, your Polaris host, then /api/catalog.
2. If the catalog is reachable only on a private network, contact StarTree support about private connectivity.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Snowflake Open Catalog, Snowflake Horizon Catalog**

1. Check Catalog Endpoint URI: https\://, your Snowflake account host, then /polaris/api/catalog.
2. If the catalog is reachable only on a private network, contact StarTree support about private connectivity.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1099 — The catalog could not be reached

StarTree could not open a connection to the catalog, or the catalog stopped answering before it replied, so the request did not complete.

**Transient** — temporary; retrying usually works.

**What to do**

1. Try again in a moment. A catalog that is restarting or busy can refuse connections briefly.
2. If it keeps failing, check the host and port in Catalog Endpoint URI, and that the catalog accepts traffic from StarTree, for example through an allow-list or private connectivity.
3. If the settings are right and it still fails, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1100 — The catalog's TLS certificate was not accepted

The catalog presented a certificate StarTree could not verify: it has expired, it is not issued by a public certificate authority, or it does not name the host in Catalog Endpoint URI.

**User correctable** — you can correct this.

**What to do**

1. Use the host name the certificate was issued for in Catalog Endpoint URI, rather than an IP address or an alias.
2. If the catalog uses a private or self-signed certificate, replace it with one from a public certificate authority, or contact StarTree support.
3. If the certificate has expired, renew it on the catalog's side.
4. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1101 — The Catalog Endpoint URI is not a catalog address

The address in Catalog Endpoint URI is missing, malformed, or points at a web page rather than the catalog's Iceberg REST endpoint.

**User correctable** — you can correct this.

**What to do**

1. Enter the catalog's full Iceberg REST address in Catalog Endpoint URI, with no spaces.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Databricks Unity Catalog**

1. Set Catalog Endpoint URI to https\://, your workspace host, then /api/2.1/unity-catalog/iceberg-rest, with no spaces.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Apache Polaris**

1. Set Catalog Endpoint URI to https\://, your Polaris host, then /api/catalog, with no spaces.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Snowflake Open Catalog, Snowflake Horizon Catalog**

1. Set Catalog Endpoint URI to https\://, your Snowflake account host, then /polaris/api/catalog, with no spaces.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1102 — The connection needs a Region

No region is set for the storage or catalog this connection uses, so its requests could not be signed or routed.

**User correctable** — you can correct this.

**What to do**

1. Set Region to the AWS region of the bucket that holds the data. It is required even when the catalog vends storage credentials, because vended credentials carry no region.
2. With IAM Role as the Authentication Type, the role cannot be assumed until the region is set.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Databricks Unity Catalog, Apache Polaris, Snowflake Open Catalog, Snowflake Horizon Catalog**

1. Set Region under Storage to the AWS region of the bucket that holds the table's data. It is required even when the catalog vends storage credentials, because those carry no region.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On AWS Glue, Amazon S3 Tables**

1. Set Region under Metastore to the catalog's region, and under Storage as well when Reuse metastore credentials is No.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On S3 Data Lake**

1. Set Region to the region of the bucket.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1103 — The Bucket Name is missing or malformed

The connection's Bucket Name is empty or is not a plain bucket name, so the files in it could not be listed.

**User correctable** — you can correct this.

**What to do**

1. Enter only the bucket's name in Bucket Name, for example sales-data, without s3:// or gs\://, slashes or a path. Put the folder in the prefix, without a trailing slash.
2. When browsing, stay within the connection's bucket; to read another bucket, create a connection for it.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On S3 Data Lake**

1. Enter only the bucket's name in Bucket Name, for example sales-data, without s3://, slashes or a path. Put the folder in the prefix, without a trailing slash.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On GCS Data Lake**

1. Enter only the bucket's name in Bucket Name, for example sales-data, without gs\://, slashes or a path. Put the folder in the prefix, without a trailing slash.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1104 — The storage Endpoint is not a valid address

The Endpoint on this connection is not a complete https address, so the object store could not be contacted.

**User correctable** — you can correct this.

**What to do**

1. Enter the full address of the object store in Endpoint, starting with https\:// and with no spaces.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On GCS Data Lake**

1. Set Endpoint to [https://storage.googleapis.com](https://storage.googleapis.com), the default.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/gcs/onboarding-data-portal)

### DPE-1105 — The HMAC keys could not be read from Secret Manager

This connection reads its HMAC keys from GCP Secret Manager, and they could not be read: a secret name may be wrong, have no enabled version, or be unreadable with the configured service account.

**User correctable** — you can correct this.

**What to do**

1. Check that Access ID Secret Name and Secret Key Secret Name match secrets in the project set in GCP Project ID, and that each has an enabled version.
2. Grant the service account in Service Account Key Path the Secret Manager Secret Accessor role on both secrets.
3. Or switch Authentication Type to Inline HMAC Keys and enter the HMAC Access ID and the HMAC Secret directly.
4. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/gcs/onboarding-data-portal)

### DPE-1106 — The bucket does not exist

The object store reports that the bucket named on this connection does not exist.

**User correctable** — you can correct this.

**What to do**

1. Check Bucket Name for a typo, and enter only the name, without s3:// or a path.
2. If the bucket was deleted or renamed, point the connection at the bucket that now holds the data.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1107 — The bucket is in a different region

The bucket that holds this data is in a different AWS region from the one set on the connection, so the object store redirected or refused the request instead of answering it.

**User correctable** — you can correct this.

**What to do**

1. Set the Region in the connection's storage settings to the bucket's region, shown in the S3 console's list of buckets.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Databricks Unity Catalog, Apache Polaris, Snowflake Open Catalog, Snowflake Horizon Catalog**

1. Set Region under Storage to the bucket's region, shown in the S3 console's list of buckets.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On AWS Glue, Amazon S3 Tables**

1. Set Region under Storage to the bucket's region, shown in the S3 console's list of buckets. If Reuse metastore credentials is Yes, set it to No first, so the storage Region can differ from the catalog's.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On S3 Data Lake**

1. Set Region to the bucket's region, shown in the S3 console's list of buckets.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1108 — The storage Region does not match the bucket

Bucket `{bucket}` is in `{actualRegion}`, but the connection's storage Region is set to `{configuredRegion}`.

**User correctable** — you can correct this.

**What to do**

1. Set the Region in the connection's storage settings to `{actualRegion}`.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Databricks Unity Catalog, Apache Polaris, Snowflake Open Catalog, Snowflake Horizon Catalog**

1. Set Region under Storage to `{actualRegion}`.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On AWS Glue, Amazon S3 Tables**

1. Set Region under Storage to `{actualRegion}`. If Reuse metastore credentials is Yes, set it to No first, so the storage Region can differ from the catalog's.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On S3 Data Lake**

1. Set Region to `{actualRegion}`.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1109 — Tables stored outside Amazon S3 cannot be read yet

This table's data files are in Google Cloud Storage or Azure storage, and an external table backed by a catalog can read data files only from Amazon S3 today.

**User correctable** — you can correct this.

**What to do**

1. Pick a table whose data files are stored in Amazon S3. The support matrix lists the storage each catalog can use.
2. For Parquet files in Google Cloud Storage that no catalog manages, use a GCS Data Lake connection instead.

[Related documentation](/corecapabilities/external-table/support-matrix)

### DPE-1110 — The source table has no data yet

The table exists in the catalog but has no snapshot with data files yet, so there is nothing to preview or sync.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. Write data to the table at the source, then preview it again.
2. An existing table picks the data up on its next sync; nothing needs to change in StarTree.

[Related documentation](/corecapabilities/external-table/observability)

### DPE-1111 — The table changed while it was being read

The table snapshot being read was expired at the source before the read finished, which happens when snapshot retention is short or expiry runs often.

**Transient** — temporary; retrying usually works.

**What to do**

1. Try again: the next attempt reads the latest snapshot, and an existing table's sync recovers on its next run.
2. If it keeps happening, keep snapshots longer at the source, or expire them less often.

[Related documentation](/corecapabilities/external-table/snapshot-retention)

### DPE-1112 — A data file the table lists is missing

A Parquet file that the table points to no longer exists in storage, usually because compaction, vacuum or file-retention cleanup removed it while it was still referenced.

**User correctable** — you can correct this.

**What to do**

1. Make sure cleanup at the source, such as VACUUM, orphan-file removal or compaction, keeps every file the table still reads. Then try again once a fresh snapshot is written.
2. If the file was removed by mistake, restore it at the same path, then click Sync Now.

**On S3 Data Lake, GCS Data Lake**

1. Files removed or rewritten by compaction cannot be read again from a data lake source. Read sources that compact through a catalog such as AWS Glue or Unity Catalog, or recreate the table.
2. If the file was removed by mistake, restore it at the same path, then click Sync Now.

**On Databricks Unity Catalog, Apache Polaris, Snowflake Open Catalog, Snowflake Horizon Catalog, AWS Glue, Amazon S3 Tables**

1. Make sure cleanup at the source, such as VACUUM or orphan-file removal, keeps every file the current snapshots still use. Then try again once a fresh snapshot is written.
2. If the file was removed by mistake, restore it at the same path, then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1113 — A source file is not readable Parquet

A file under the source path is not a Parquet file the reader can open: it is in another format, incomplete, corrupt, or encrypted.

**User correctable** — you can correct this.

**What to do**

1. Keep only Parquet data files under the table's path, or narrow the prefix so other files, such as \_SUCCESS markers, CSV files or a \_delta\_log folder, are left out.
2. Parquet files with encrypted footers cannot be read; rely on bucket encryption instead.
3. To let a sync skip unreadable files instead of failing, set continueOnFileError to true in the ExternalTableSyncTask settings on the table's Configuration tab.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1114 — Reading the source files took too long

Reading data from the source files did not finish in time, so the sample could not be built.

**Transient** — temporary; retrying usually works.

**What to do**

1. Try again in a moment; object storage can be slow to answer under load.
2. If it fails the same way every time, check that the Region in the connection's storage settings matches the bucket's region; a mismatch can show up like this.
3. If it still fails, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1115 — A column in the source could not be decoded

Column `{column}` holds values in a form the reader could not decode, so the sample could not be built.

**Internal** — a defect on our side.

**What to do**

1. Contact StarTree support and quote the reference shown with this error and the column name; the type mapping may not cover how this column is encoded yet.

[Related documentation](/corecapabilities/external-table/data-type-mapping)

### DPE-1116 — The catalog returned an error

The catalog answered with a server error, said it was unavailable, or asked StarTree to slow down, so the request did not complete. This is on the catalog's side.

**Transient** — temporary; retrying usually works.

**What to do**

1. Try again in a moment.
2. If it keeps failing, check the catalog's status page or service health, then contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1117 — The catalog rejected the connection's settings

The catalog rejected a request built from this connection's settings as malformed, usually because the catalog name, prefix or table bucket it names is not valid there.

**User correctable** — you can correct this.

**What to do**

1. Check the settings that name where the catalog is against the catalog: Catalog Name, AWS Account ID, Prefix or Table Bucket ARN, whichever the form shows.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Snowflake Open Catalog**

1. Check Catalog Name and Prefix against the catalog in Snowflake Open Catalog.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Databricks Unity Catalog, Apache Polaris**

1. Check Catalog Name against the catalog's exact name at the source.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Snowflake Horizon Catalog**

1. Check Database against the Snowflake database's exact name.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On Amazon S3 Tables**

1. Check Table Bucket ARN is the full ARN of the table bucket, copied from the S3 console.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On AWS Glue**

1. Check AWS Account ID is the numeric ID of the account that owns the Glue Data Catalog.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1118 — That namespace is no longer in the catalog

The catalog no longer has the namespace being browsed; it may have been renamed or dropped since the list was loaded.

**User correctable** — you can correct this.

**What to do**

1. Go back to the list of namespaces, reload it, and pick the namespace again.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1119 — No AWS credentials were available

The connection asked to use AWS credentials from the environment, but none were available where the request ran.

**User correctable** — you can correct this.

**What to do**

1. Set Authentication Type to IAM Role, with the IAM Role ARN of a role that trusts this environment, or to access keys with a Secret Key.
2. IAM Node Role works only where the cluster's servers have an AWS identity; if you expected it to, contact StarTree support and quote the reference shown with this error.
3. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1120 — The IAM Role ARN is not valid

AWS rejected the IAM Role ARN on this connection because it is not a well-formed role ARN.

**User correctable** — you can correct this.

**What to do**

1. Copy the role's ARN from the role's summary page in the IAM console into IAM Role ARN, whole and without spaces; a role name or a policy ARN is not accepted.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1121 — The storage address could not be found

The object store's address for this bucket could not be resolved, usually because the Region is misspelled or the Endpoint is wrong.

**User correctable** — you can correct this.

**What to do**

1. Check the Region in the connection's storage settings is a real AWS region name, written with hyphens, for example us-east-1.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

**On GCS Data Lake**

1. Keep Endpoint as [https://storage.googleapis.com](https://storage.googleapis.com), and check Region; the default auto works for most buckets.
2. On an existing table, make the change from its Connection tab: click Edit, then, if the tab shows the connection, Test and Update, which reaches every table in sync with it; if it shows this table's own settings, Save. Then click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1122 — Object storage did not answer the request

The object store was busy, asked StarTree to slow down, or could not be reached, so the files could not be listed or read this time.

**Transient** — temporary; retrying usually works.

**What to do**

1. Try again in a moment; an existing table's next sync retries on its own.
2. If it keeps failing, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1123 — The temporary AWS credentials expired

The temporary AWS credentials used for this request expired before it finished.

**Transient** — temporary; retrying usually works.

**What to do**

1. Try again; fresh credentials are requested on the next attempt, and an existing table's next sync does so on its own.
2. If it keeps happening with an IAM Role, raise the role's maximum session duration in AWS.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1124 — A file name under the source cannot be read

A file under the source path has a name with a space or another character that is not allowed in a storage address, so the sync cannot read past it.

**User correctable** — you can correct this.

**What to do**

1. Rename or move files whose names contain spaces, percent signs, or # and ? characters at the source, or narrow the prefix so they are left out.
2. Click Sync Now.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1125 — The table's namespace is not set

The table's sync settings name a table but no namespace, so the catalog could not look it up.

**User correctable** — you can correct this.

**What to do**

1. Open the table's Configuration tab, click Edit, and in the ExternalTableSyncTask settings set the namespace (the database or schema) the table belongs to; then Apply and Save.
2. Click Sync Now.

[Related documentation](/corecapabilities/external-table/getting-started/connect-a-catalog)

### DPE-1126 — The preview needed more memory than was available

Reading the sample rows needed more memory than the cluster had free, so the preview stopped. This is on our side, not your configuration.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Try again in a moment.
2. If the table is very wide and it keeps failing, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-1131 — Some tables are no longer linked to this connection

The update named tables that aren't linked to this connection: `{tables}`. Nothing was changed. A table may have been deleted, unlinked or moved to another connection since this page loaded.

**User correctable** — you can correct this.

**What to do**

1. Reload the page so the Associated Tables list is current.
2. Update the connection again.

### DPE-1132 — A table can't be both updated and unlinked

The update asked to both update and unlink `{tables}`. Nothing was changed.

**User correctable** — you can correct this.

**What to do**

1. Reload the page and update the connection again.

### DPE-1133 — This table's ingestion settings can't be edited here

Table `{table}` doesn't ingest from a source type that has connection settings, so there is no settings form to apply.

**User correctable** — you can correct this.

**What to do**

1. Edit the table's ingestion config on its Configuration tab instead.

### DPE-1134 — The brokers do not offer that SASL Mechanism

The Kafka brokers do not accept the `{mechanism}` mechanism, so they never checked the Username and Password. They offer `{enabled}`.

**User correctable** — you can correct this.

**What to do**

1. Set SASL Mechanism on Connection Setup to one the brokers offer: `{enabled}`. Confluent Cloud brokers offer Plain.
2. For a saved connection, click Edit on the Connections page or the dataset's Connection tab, change SASL Mechanism, then Test and Update.

[Related documentation](/recipes/kafka-sasl)

### DPE-1135 — An AWS policy explicitly denies this action

AWS refused `{action}` because a `{policy}` explicitly denies it for this identity. An explicit deny overrides any permission granted elsewhere, so adding `{action}` to the connection's IAM policy does not help.

**User correctable** — you can correct this.

**What to do**

1. Ask whoever manages that `{policy}` in your AWS account or organization to exempt this connection's identity, or the bucket, from the deny.
2. Then click Test Connection, or Test then Update on a saved connection.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/s3)

### DPE-1136 — The uploaded file is no longer available

The preview reads an uploaded file from a temporary copy on the server, and that copy is gone, so there was nothing to read. The copy does not survive a restart of the service that holds it.

**User correctable** — you can correct this.

**What to do**

1. Go back to Connection Setup and upload the file again with Upload File.
2. Then select Show Sample Data, and continue to the step you were on.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/upload)

## Platform

### DPE-0000 — Something went wrong on our side

This action did not complete because of an unexpected problem on our side.

**Internal** — a defect on our side.

**What to do**

1. Retry the action.
2. If it happens again, contact StarTree support and quote the reference shown with this error.

### DPE-5101 — The system is busy

We could not complete your request because the service is at capacity right now.

**Transient** — temporary; retrying usually works.

**What to do**

1. Wait a moment and try again. If you were creating or changing something, first check whether the change was saved, so you do not apply it twice.
2. If it keeps failing for more than a few minutes, contact StarTree support and quote the reference shown with this error.

### DPE-5102 — The system is busy

A database operation could not complete right now.

**Transient** — temporary; retrying usually works.

**What to do**

1. Wait a moment and try again. If you were creating or changing something, first check whether the change was saved, so you do not apply it twice.
2. If it keeps failing for more than a few minutes, contact StarTree support and quote the reference shown with this error.

### DPE-5103 — The system is busy

A database operation could not complete right now.

**Transient** — temporary; retrying usually works.

**What to do**

1. Wait a moment and try again. If you were creating or changing something, first check whether the change was saved, so you do not apply it twice.
2. If it keeps failing for more than a few minutes, contact StarTree support and quote the reference shown with this error.

### DPE-5104 — A service we depend on is unavailable

A service this action depends on could not be reached. This is on our side, not your configuration.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Retry in a few minutes.
2. If it persists, contact StarTree support and quote the reference shown with this error.

### DPE-5105 — A service we depend on is unavailable

A service this action depends on did not respond. This is on our side, not your configuration.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Retry in a few minutes.
2. If it persists, contact StarTree support and quote the reference shown with this error.

### DPE-5106 — The cluster is not reachable

We could not reach the cluster that stores this dataset. This is on our side, not your configuration.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Retry in a few minutes.
2. If it persists, contact StarTree support and quote the reference shown with this error.

### DPE-5107 — The cluster is at capacity

Too many requests to the cluster were already in progress, so this one was held back to protect the cluster and was not sent.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Retry in a few minutes.

### DPE-5108 — Something went wrong on our side

The cluster returned an error we could not interpret, so the action did not complete.

**Internal** — a defect on our side.

**What to do**

1. Retry the action.
2. If it happens again, contact StarTree support and quote the reference shown with this error.

### DPE-5109 — The request could not be read

The request body could not be read: it was missing, was not well-formed JSON, or a field held a value of the wrong type or outside its allowed options. Nothing was changed.

**User correctable** — you can correct this.

**What to do**

1. Reload the page and repeat the action.
2. If you are calling the API directly, send a well-formed JSON body whose fields use the types and values the API reference lists; Show details has the parser's own message.
3. If it happens again in Data Portal, contact StarTree support and quote the reference shown with this error.

### DPE-5110 — A service we depend on is unavailable

A service this action depends on could not be reached or did not answer in time. This is on our side, not your configuration.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Retry in a few minutes.
2. If it persists, contact StarTree support and quote the reference shown with this error.

### DPE-5111 — That action is not available here

This request used a method this address does not accept, so nothing was changed.

**Internal** — a defect on our side.

**What to do**

1. Reload the page and try again.
2. If you are calling the API directly, use the method the API reference lists for this path.
3. If it happens again in Data Portal, contact StarTree support and quote the reference shown with this error.

### DPE-5112 — That address does not exist

Nothing on this service answers that address, so the request was not processed.

**Internal** — a defect on our side.

**What to do**

1. Reload the page and try again.
2. If you are calling the API directly, check the path, including its version prefix, against the API reference.
3. If it happens again in Data Portal, contact StarTree support and quote the reference shown with this error.

### DPE-5117 — A stored setting could not be read

One of the connection settings this request carried is incomplete, such as a dropdown value sent without its list of options, so it could not be read.

**Internal** — a defect on our side.

**What to do**

1. If you are calling the API directly, send each setting with its complete options list, as the properties endpoint returns it.
2. In Data Portal, reload the page and repeat the action. If it keeps happening on a saved connection, open it from Connections, select Edit, then Update, which saves its settings again in full.
3. If that does not help, contact StarTree support and quote the reference shown with this error.

### DPE-5118 — The request was cancelled

The request ended before a reply could be sent, usually because the page was closed or navigated away from.

**Informational** — not a failure; act only if you want a different outcome.

### DPE-5119 — Cluster health is not available right now

The cluster answered the health request without any detail, so there is nothing to show yet.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Try again in a few moments.
2. If it stays empty, contact StarTree support and quote the reference shown with this message.

### DPE-5120 — Something went wrong on our side

This action did not complete because a database request could not be carried out.

**Internal** — a defect on our side.

**What to do**

1. Retry the action.
2. If it happens again, contact StarTree support and quote the reference shown with this message.

### DPE-5121 — The cluster is not reachable

We could not get an answer from the cluster, so this action could not finish. This is on our side, not your configuration.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Retry in a few minutes; this usually means the cluster was restarting or briefly unavailable.
2. If it still fails after about 15 minutes, contact StarTree support and quote the reference shown with this error.

### DPE-5122 — A service we depend on returned an error

An internal service this action needs returned an error, so the action could not complete.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Try again in a moment.
2. If it keeps happening, contact StarTree support and quote the reference shown with this error.

### DPE-5123 — An answer from the cluster could not be read

The cluster replied in a shape this action did not expect, so the result could not be read.

**Internal** — a defect on our side.

**What to do**

1. Retry the action.
2. If it happens again, contact StarTree support and quote the reference shown with this error.

### DPE-5124 — That data source type is not recognised

This environment has no data source type with that id. This usually happens when the page was opened before the environment was updated, or when the id came from another environment.

**User correctable** — you can correct this.

**What to do**

1. Reload the page, then try again. If you were adding a table or connection, choose the data source type again from the refreshed list.
2. If you are calling the API directly, get the id from GET /api/datasources/types in this environment. Don't reuse an id from another environment or release.
3. If the type you need is missing from the list after reloading, contact StarTree support and quote the reference shown with this message.

### DPE-5125 — The change is still being applied

This sync registered new data for `{table}`, but the servers had not finished loading it when the sync's waiting time ran out, so the run stopped. The servers keep loading it, so the new data may take longer to appear.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Check the dataset's Monitoring tab after the next sync, which waits for this data again; to start one now, select Retry Sync there.
2. If it stops this way again, open the Storage tab, then Segments, select Bad, tick the segments listed, select Reset and confirm.
3. If no segments are Bad, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-5126 — That file is not served here

A browser asked this service for a static file it does not serve. Nothing the reader did caused it and nothing is affected.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. No action is needed.

### DPE-5127 — Some copies of this data are in an error state

Some of this dataset's segments failed to load on the servers holding them and stay failed until they are reset, so this sync stopped rather than leave the dataset half-updated. Queries may return partial results until then.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. On the dataset's Storage tab, open Segments, select Bad, tick the segments listed, select Reset and confirm. Then select Retry Sync on the Monitoring tab.
2. If they turn Bad again after the reset, or many segments are Bad at once, contact StarTree support and quote the reference shown with this error; the servers holding them need attention.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-5128 — The change stopped making progress

This sync registered new data, but the servers made no progress loading it for longer than the sync waits, so the run was stopped rather than left waiting indefinitely.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Select Retry Sync on the dataset's Monitoring tab, or wait for the next scheduled sync; it waits for the same data again and completes once the servers catch up.
2. If it stops this way again, contact StarTree support and quote the reference shown with this error; the servers loading this data need attention.

[Related documentation](/corecapabilities/external-table/troubleshooting)

### DPE-5129 — The API reference could not be generated

The page that describes this service's API could not be built because of a fault on our side. The API itself is unaffected.

**Internal** — a defect on our side.

**What to do**

1. Keep using the API as normal — only the reference page is affected.
2. If you need the reference page, contact StarTree support and quote the reference shown with this error.

### DPE-5130 — A service component failed to load

Part of the service that reads this source could not be loaded on our side, so the request stopped before any data was read. Your connection and data are not the cause.

**Internal** — a defect on our side.

**What to do**

1. Contact StarTree support and quote the reference shown with this error; this needs a fix on our side.
2. Your connection settings do not need to change.

### DPE-5131 — New data could not be registered in time

This sync prepared its new data, but the cluster took longer than allowed to accept it, so the run stopped. Large runs on a busy cluster can exceed that limit.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. No action is needed if the next sync succeeds; it picks up any files this run did not finish.
2. Check the dataset's Monitoring tab after the next scheduled run.
3. If several runs in a row stop this way, contact StarTree support and quote the reference shown with this error: the cluster needs more time or capacity for runs of this size.

[Related documentation](/corecapabilities/external-table/overview)

### DPE-5132 — Table health results could not be saved

The background check that tracks table health could not record its latest results, so health alert emails are held back until it succeeds. This is on our side, not your configuration.

**Internal** — a defect on our side.

**What to do**

1. No action is needed on your side.
2. If health alert emails stop arriving, contact StarTree support and quote the reference shown with this error.

### DPE-5133 — Table health tracking skipped one round

A table was removed while the background table health check was saving its results, so that round was set aside. The next round starts from the current list of tables, so no health alert is lost.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. No action is needed.

### DPE-5134 — The preview request was missing its settings

Some of the connection or data setup settings this preview needs did not arrive with the request, so the sample data could not be built. This is a fault on our side, not in your data.

**Internal** — a defect on our side.

**What to do**

1. Go back to Select Data Source, choose the source again, fill in Connection Setup and select Show Sample Data.
2. If you are calling the preview API directly, send each settings object with its complete properties list, as the properties endpoints return it.
3. If it happens again, contact StarTree support and quote the reference shown with this error.

### DPE-5135 — This source type has no Data Setup step

The Data Setup step was opened for a source type that does not use one, so there was nothing to load. This is a fault on our side, not in your connection.

**Internal** — a defect on our side.

**What to do**

1. Upload File, Custom and Databricks skip Data Setup: after Connection Setup, select Show Sample Data, then continue to Data Modeling.
2. To load a file, select Create table on the Tables page and choose Upload File under Select Data Source.
3. If you call the API directly, these types have no data setup properties to request.
4. If it happens again, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/ingestdata/ingest_data_overview)

### DPE-5136 — Your account cannot access this table in the cluster

The cluster's access control did not allow this request for your account.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. Ask a workspace administrator to grant your account access to this table.
2. If access was granted just now, sign out and in again so a fresh session is used.

[Related documentation](/corecapabilities/security/manage-security)

### DPE-5137 — The cluster is not reachable

We could not reach the cluster that stores this dataset. This is on our side, not your configuration.

**Upstream** — a service we depend on did not answer as expected.

**What to do**

1. Retry in a few minutes.
2. If it persists, contact StarTree support and quote the reference shown with this error.

## Request validation

### DPE-5201 — Ingestion type not recognised

This table was created outside Data Portal with a stream type that Data Portal does not recognise, so its ingestion details could not be read.

**User correctable** — you can correct this.

**What to do**

1. Keep querying and managing the table as usual; only the ingestion details shown in Data Portal are affected.
2. If you need those details, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/ingestdata/ingest_data_overview)

### DPE-5202 — A required value was missing

The request did not name a `{param}`. This environment hosts several workspaces, so every request has to say which one it is for.

**User correctable** — you can correct this.

**What to do**

1. Choose a workspace in the Workspace selector at the top of the page, then repeat the action.
2. If the selector offers no workspace, ask an administrator to give you access to one.
3. If you are calling the API directly, send a Workspace header with the workspace id; Copy Workspace Id in the Workspace selector gives it.

### DPE-5203 — You do not have access to this

Your account is not authorized for this action in this workspace.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. Ask an administrator to assign you a role that allows this action, under Settings > Security Manager > Role Assignments, then reload the page.

[Related documentation](/corecapabilities/security/manage-security)

### DPE-5204 — You do not have access to this

Your account is missing the `{action}` permission, which this needs.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. Ask an administrator to grant `{action}` through a role assigned to you, under Settings > Security Manager, then reload the page.

[Related documentation](/corecapabilities/security/manage-security)

### DPE-5205 — You do not have access to this

Your account is not authorized for this action in this workspace.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. Ask an administrator to assign you a role that allows this action, under Settings > Security Manager > Role Assignments, then reload the page.

[Related documentation](/corecapabilities/security/manage-security)

### DPE-5206 — Connection not found

The connection this refers to no longer exists: it was deleted, or the page or link that named it is out of date.

**User correctable** — you can correct this.

**What to do**

1. Open Connections and pick the connection from the list; a deleted connection is no longer listed.
2. If this appeared while creating a table, go back to Connection Setup and choose one under Existing Connections, or set one up under New Connection and run Test Connection.
3. If it appears on a table's page although its connection still exists, contact StarTree support and quote the reference shown with this error.

[Related documentation](/corecapabilities/ingestdata/ingest_data_overview)

### DPE-5207 — Uploaded file could not be read

The upload could not be read: the request ended before the whole file arrived, or it was not sent as a multipart form.

**User correctable** — you can correct this.

**What to do**

1. Upload the file again; an interrupted upload cannot be resumed.
2. Keep the file under 30 MB. For a larger file, put it in object storage such as Amazon S3 and create the table from there instead of uploading it.
3. If you are calling the API directly, send multipart/form-data with a file part and a request part, and let your HTTP client set the boundary.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/upload)

### DPE-5208 — Uploaded file was empty

The preview request carried no file contents, so there was nothing to read.

**User correctable** — you can correct this.

**What to do**

1. If you are calling POST /api/v1/data-manager/datasources/preview-ingestion with ingestionType UPLOAD, put the file's contents, base64-encoded, in payloadBase64.
2. For preview-ingestion-v2, put them in the locator under upload.payloadBase64.
3. In Data Portal, choose Upload File as the source when creating a table; the page sends the file for you.

[Related documentation](/corecapabilities/ingestdata/dataportal/batch/upload)

### DPE-5209 — Setting not recognised

This environment has no setting by that name. That happens when Data Portal is newer than the service behind it, or when a request names a setting that does not exist.

**User correctable** — you can correct this.

**What to do**

1. Nothing needs changing in Data Portal; the setting becomes available once this environment is updated.
2. If you are calling the API directly, use a key listed by GET /api/v1/app-settings.

### DPE-5210 — That dataset could not be found

The dataset `{srn}` no longer exists, or it was renamed while this page was open.

**User correctable** — you can correct this.

**What to do**

1. Reload the page and open the dataset again from the Tables list.
2. If it is not in the list, it was deleted — nothing further is needed.
3. If you reached this from a bookmark or a shared link, the name in the link is out of date.

### DPE-5211 — You do not have access to that

Your account does not have permission to `{action}`.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. Ask an administrator to assign you a role that allows this, under Settings > Security Manager > Role Assignments.
2. Reload the page once it is granted; permissions are read when the page loads.

### DPE-5212 — That value is not valid here

The value supplied for `{param}` is not one this request accepts.

**User correctable** — you can correct this.

**What to do**

1. If you are calling the API directly, send `{param}` in the form the endpoint documents: a number where it takes a numeric id, or one of its listed values where it takes a fixed set.
2. For example, GET /api/datasources/\ takes a numeric data source type id, not a connection reference; list connections with GET /api/datasources instead.
3. If this appeared in Data Portal, reload the page and repeat the action from its list.

### DPE-5213 — A required value was not supplied

`{field}` is required, and the request did not include it.

**User correctable** — you can correct this.

**What to do**

1. Fill in `{field}` on the step or form that asks for it, then try again; required fields are marked with an asterisk.
2. If Data Portal shows no field by that name, reload the page and repeat the action; the page sent an incomplete request.
3. If you are calling the API directly, include `{field}` in the request body; it has no default.

### DPE-5214 — A required value was not supplied

The request was missing a value it needs, so it was not processed.

**User correctable** — you can correct this.

**What to do**

1. Reload the page and repeat the action.
2. If a field on screen is empty, fill it in; required fields are marked with an asterisk.
3. If you are calling the API directly, the error's technical detail names the missing value; add it to the request.
4. If it keeps happening in Data Portal, contact StarTree support and quote the reference shown with this error.

### DPE-5215 — Choose a source type

The request did not set ingestionType, so it is not known which kind of source it is for.

**User correctable** — you can correct this.

**What to do**

1. Set ingestionType in the request body to one of: `{types}`.
2. In Data Portal, start from Create table on the Tables page and pick the source under Select Data Source; the wizard sets the type for you.

### DPE-5216 — That name is already taken

Another connection in this workspace already uses this name, and connection names must be unique within a workspace.

**User correctable** — you can correct this.

**What to do**

1. Change Source Name (Connection Name on a catalog connection) to a name no other connection in this workspace uses, then try again.
2. To reuse the existing connection instead, choose it under Existing Connections when creating a table.

### DPE-5217 — That dataset belongs to a different workspace

The dataset named in the request is not in the workspace the request was made against.

**User correctable** — you can correct this.

**What to do**

1. Choose the dataset's workspace in the Workspace selector at the top of the page, then open the dataset again from Tables.
2. A link saved in another workspace keeps failing here, because the workspace is part of the dataset's address. If that workspace is not offered, ask an administrator for access to it.
3. If you are calling the API directly, send the Workspace header for the workspace in the dataset reference.

### DPE-5218 — That value is not one of the accepted options

The setting with id `{parameter}` accepts only values from a fixed list, and the request sent a value that is not on it, or none at all.

**User correctable** — you can correct this.

**What to do**

1. In Data Portal, open the connection or data setup form, choose a value from each dropdown again, and retry; a value saved by an earlier version may no longer be offered.
2. If you are calling the API directly, set the property whose id is `{parameter}` to one of the option values listed for it, matching case exactly.
3. If it persists in Data Portal, contact StarTree support and quote the reference shown with this error.

### DPE-5219 — That data source type is not available here

This environment does not offer that data source type. Which types are offered can differ between environments and releases.

**User correctable** — you can correct this.

**What to do**

1. Reload the page, then choose a data source type from the list. The list shows only the types this environment offers.
2. If you are calling the API directly, use a type listed by GET /api/datasources/types in this environment.
3. If you expect this type to be available, contact StarTree support and quote the reference shown with this message.

### DPE-5220 — A required setting is missing

The connection has no value for `{property}`, which this source needs before it can connect.

**User correctable** — you can correct this.

**What to do**

1. On a Kinesis connection, fill in Region with the AWS region the stream is in, such as us-east-1, then test the connection again.
2. If you are calling the API directly, include `{property}` in the connection properties.

**On Kinesis**

This Kinesis connection has no Region, so it cannot connect to a stream.

1. Open the connection from the Connections page, click Edit, and fill in Region with the AWS region the stream is in, such as us-east-1.
2. Click Test, then Update.

[Related documentation](/corecapabilities/ingestdata/dataportal/streaming/kinesis)

### DPE-5221 — Ingestion settings are needed to preview rows

Sample rows are read through the table's ingestion task, and this table has none: its data is loaded another way, or its source has not been set up yet.

**User correctable** — you can correct this.

**What to do**

1. While creating a table, complete Data Setup under Configure Connection, then preview again.
2. On an existing table's Schema tab, carry on without the preview: a table with no ingestion task has no source rows to sample, and its schema can still be edited.

[Related documentation](/corecapabilities/ingestdata/ingest_data_overview)

### DPE-5222 — A source in the list is no longer there

The tables that use this connection could not be listed because the connection no longer exists; it was deleted, or the page that asked for it is out of date.

**Informational** — not a failure; act only if you want a different outcome.

**What to do**

1. Reload Connections and pick the connection from the list; if it is not listed, it was deleted.

[Related documentation](/corecapabilities/ingestdata/ingest_data_overview)

### DPE-5223 — Notification preferences need a recipient or a dataset

Listing notification preferences needs a recipientEmail or a tableKey query parameter, and the request had neither.

**User correctable** — you can correct this.

**What to do**

1. On GET /api/v1/table-health-check-notifications/preferences, add recipientEmail=\
 to list one recipient's preferences, or tableKey=\
to list one table's. 2. To find who is subscribed, call GET /api/v1/table-health-check-notifications/preferences/recipients first. 3. In Data Portal, open Monitoring > Health Dashboard > Settings and pick a recipient; the page sends the address for you. [Related documentation](/corecapabilities/observability/table-health-and-alerting) ### DPE-5224 — That data source belongs to a different workspace This connection belongs to another workspace, so it cannot be opened or changed from the workspace selected now. **User correctable** — you can correct this. **What to do** 1. Choose the connection's workspace in the Workspace selector at the top of the page, then open the connection again from Connections. 2. To use the same system from this workspace, create a connection here with Create Connection on the Connections page. 3. If you are calling the API directly, send the Workspace header for the workspace the connection belongs to. ### DPE-5225 — That recipient has no preferences set No notification preferences that you can see are saved for this recipient. You see your own, plus those in workspaces where you can manage notifications. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. To subscribe them, open Monitoring > Health Dashboard > Settings and click Add recipient, or call PUT /api/v1/table-health-check-notifications/preferences/scopes. [Related documentation](/corecapabilities/observability/table-health-and-alerting) ### DPE-5226 — That name is reserved for a StarTree connection A connection created by StarTree already uses this name. StarTree's connections appear in every workspace, so the name can't be used for another connection. **User correctable** — you can correct this. **What to do** 1. Enter a different name for the connection. ### DPE-5227 — A task plan chart setting was out of range The task plan chart request's `{param}` `{rule}`. **User correctable** — you can correct this. **What to do** 1. In Data Portal, reload the Tasks page and pick the time range again; the page sends valid values for you. 2. If you are calling GET /api/tasks/plans/histogram directly, send from and to as epoch milliseconds with from before to, bucketMs between one minute and one year giving at most 500 buckets, and tzOffsetMs within 18 hours of UTC. ### DPE-5229 — That file is larger than the upload limit The file is larger than the 30 MB an upload accepts, so it was not read. **User correctable** — you can correct this. **What to do** 1. Upload a file under 30 MB. 2. For a larger file, put it in object storage such as Amazon S3 and create the table from there, with S3, GCS or ADLS under Select Data Source. [Related documentation](/corecapabilities/ingestdata/dataportal/batch/upload) ### DPE-5230 — The upload did not include its file The request had no `{part}` part, so there was nothing to read. **User correctable** — you can correct this. **What to do** 1. Reload the page and upload the file again. 2. If you are calling the API directly, send multipart/form-data with a file part and a request part. [Related documentation](/corecapabilities/ingestdata/dataportal/batch/upload) ## Ingestion and tasks ### DPE-3104 — No ingestion runs yet This dataset has no ingestion task scheduled, so there is no run history to show. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Schedule an ingestion task to get started. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3105 — No ingestion runs yet No ingestion run has been recorded for this dataset yet, so there is no progress to report. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Schedule an ingestion task, then check back once the first run starts. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3106 — No files matched this run The ingestion task ran but found no new input files matching the configured path, so nothing was scheduled. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Confirm the input path and file pattern point at the files you expect. 2. If files were already ingested, this is expected — each file is processed once. [Related documentation](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration) ### DPE-3107 — Ingestion service unavailable We could not reach the service that reports ingestion progress. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Retry in a few minutes. 2. If it persists, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3108 — Preview is not available for this dataset Sample rows can be fetched only for a dataset that loads its data through file ingestion, a SQL connector, Delta Lake or a catalog sync. This dataset's table config names `{taskType}` instead, so there is no source to sample. The dataset itself is unaffected; only the preview is. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. To look at its rows, query the dataset from the Query Console. [Related documentation](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration) ### DPE-3109 — Task configuration missing Setting, pausing or resuming a schedule changes the `{taskType}` entry in this dataset's table config, and the table config has no such entry. It was removed after this page loaded, or it never existed. **User correctable** — you can correct this. **What to do** 1. Reload the dataset's page; without that entry there is no schedule to change, so Pause Sync and Resume Sync are no longer offered. 2. To run on a schedule again, restore the entry under task on the Configuration tab (Edit, Apply, then Save). If you do not have the settings it held, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration) ### DPE-3110 — Ingestion task scheduling failed The request to start this ingestion run did not get a usable answer from the cluster's task scheduler, so it is not known whether the run started. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Wait a minute, then look for a new run on the dataset's Tasks tab, or on the Monitoring tab for a catalog table; it may have started anyway. 2. If none appears, select Run Now (Sync Now for a catalog table) again. 3. If it keeps failing, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3111 — Cluster state for this dataset is missing The cluster has no record of how this dataset's data is placed across its servers. That is expected for a moment while a dataset is being created or deleted; if it lasts, the cluster's state for this dataset is incomplete. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Refresh the page in a minute; if the dataset was just created or deleted, this clears on its own. 2. If it persists, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3112 — Could not read ingestion progress We could not read this dataset's stream ingestion progress from the cluster, so it cannot be shown right now. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Wait a moment, then refresh the page. 2. If it persists, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3113 — Could not reach the ingestion service We could not reach the service that reports this dataset's ingestion runs, so their status cannot be shown right now. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Wait a moment and refresh. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. ### DPE-3115 — The source path could not be listed Listing the files under `{uri}` failed, so the run had nothing to ingest. The connection could not list that location: a missing list permission, a wrong bucket, path or region, or credentials that no longer work. **User correctable** — you can correct this. **What to do** 1. On the dataset's Connection tab, select Test. If it fails, select Edit, correct the input directory URI, region or credentials, then Test and Update. 2. If the settings are right, let the connection's credentials list that location, not only read files: on AWS, s3:ListBucket on the bucket. 3. Then select Run Now at the top of the dataset's page, or wait for the next scheduled run. [Related documentation](/corecapabilities/ingestdata/troubleshooting) ### DPE-3116 — Ingestion could not be scheduled The cluster could not prepare an ingestion run for `{table}`, so no run was started and no data was read. The reason it gave is not one we recognise; it is recorded with the reference shown with this error. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Try again in a moment. 2. If it persists, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/ingestdata/troubleshooting) ### DPE-3117 — Ingestion status is unavailable The service that reports ingestion progress did not return this dataset's task details, so its status cannot be shown right now. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Wait a moment and refresh. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. ### DPE-3118 — Ingestion run failed One or more input files in the latest ingestion run failed to load, so their rows are not in the dataset yet. The run recorded which files failed and why. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. At the top of the dataset's page, select the icon beside Sync Status to see the failed files and the error; the run's Task plan on the Tasks tab holds its full log. 2. If the error is about a file's content or format, fix or replace the file at the source, then select Run Now to load it again. 3. If it reports running out of disk space or memory, or every file fails, contact StarTree support. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3119 — You do not have access to task details Reading task details for this dataset requires a permission this account does not have. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Ask a workspace administrator for access to view this dataset's ingestion tasks. [Related documentation](/corecapabilities/security/manage-security) ### DPE-3120 — Pinot answered with nothing The cluster answered that it has nothing to return for this yet, and gave no reason. The usual case is cluster health results asked for before the first scheduled health check has finished. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Try again in a few minutes; the result appears once the cluster has produced it. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. ### DPE-3121 — No new ingestion run was needed The scheduled check ran and found nothing new to ingest since the previous run, so no run was started. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Nothing to do — the next scheduled check starts a run once there is new data. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3122 — Streaming ingestion has stalled One stream partition of this dataset stopped being read after the server reading it hit an error, so its new records are not arriving. The cluster usually restarts it within about an hour; if the stream refuses the read, it stops again. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Watch Ingestion delay on the dataset's Monitoring tab; once it falls again, reading has resumed and nothing else is needed. 2. If it stays stopped, check the topic still exists and the credentials still work: on the dataset's Connection tab, click Test. 3. If the connection passes, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/ingestdata/troubleshooting) ### DPE-3123 — A source file could not be ingested The sync stopped at `{file}`, which could not be turned into data. The time column is the usual culprit: missing, empty, or a legacy INT96 timestamp. Otherwise the file is damaged or a column type differs from the schema. **User correctable** — you can correct this. **What to do** 1. On the dataset's Monitoring tab, click View full error to see what failed, and fix the file at the source. 2. If its time column is missing or empty, set segmentTimeValueCheck to false under ingestionConfig on the Configuration tab (Edit, Apply, Save). 3. To skip unreadable files instead, set continueOnFileError to true in the ExternalTableSyncTask settings there. Then click Retry Sync. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-3124 — Ingestion paused while capacity frees up No run was started for this dataset: the cluster holds back new ingestion while a capacity limit is reached, either disk usage on the servers that hold it or, for an external table, the number of segments it or the environment may have. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Runs start again on their own once usage is back under the limit, and the next run picks up what arrived meanwhile. 2. If it lasts, contact StarTree support to review this environment's capacity, and quote the reference shown with this error. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3125 — Some rows have timestamps out of range Data was refused because its time range, `{range}`, falls outside what StarTree accepts, `{allowed}`. A placeholder date or a timestamp read in the wrong unit causes this. **User correctable** — you can correct this. **What to do** 1. Correct or remove those rows at the source. A unit mismatch is the usual cause: seconds read as milliseconds land in 1970, the reverse far in the future. 2. If the dates are right, set segmentTimeValueCheck to false under ingestionConfig on the Configuration tab (Edit, Apply, Save). 3. Then run the ingestion again: Retry Sync on the Monitoring tab for an external table, otherwise Run Now or the next scheduled run. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-3126 — This dataset has no data to show This dataset holds no data. It was loaded from an uploaded file, and data whose timestamps are all older than the retention period is removed automatically. **User correctable** — you can correct this. **What to do** 1. Check the newest timestamp in the uploaded file; if it is older than the retention period, the data was removed. 2. Raising retentionTimeValue under segmentsConfig on the Configuration tab does not bring back removed data. 3. To load the file again, use Upload File, which creates a new dataset; raise retention at its Validate step first. 4. If the timestamps are recent, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/ingestdata/dataportal/batch/upload) ### DPE-3127 — The source warehouse is suspended The source Snowflake account's free trial has ended, so Snowflake has suspended all of its warehouses and refuses the reads this ingestion needs. **User correctable** — you can correct this. **What to do** 1. Add billing details to that Snowflake account in the Snowflake web interface; its warehouses stay suspended until then. 2. Nothing in StarTree needs changing: the next scheduled run ingests as usual, and Next Run at the top of the dataset's page shows when. [Related documentation](/corecapabilities/ingestdata/dataportal/batch/snowflake) ### DPE-3128 — Preview needs exactly one task in the table config Sample rows are read through the table config's ingestion task, and this table config lists more than one task, or none, so the preview cannot tell which one reads the source. The dataset's ingestion is not affected. **User correctable** — you can correct this. **What to do** 1. While creating a dataset: on the Validate step's Table Config tab, select Edit configuration, keep only the ingestion task under task, then Apply Changes. Add other tasks on the Configuration tab once the dataset exists. 2. On an existing dataset, keep its tasks; only Fetch Preview on the Schema tab is affected. To look at its rows, query the dataset from the Query Console. ### DPE-3129 — Ingestion tasks could not be generated The cluster could not prepare this dataset's next ingestion run, so nothing was started. The full error names the reason; most often the source could not be listed or read with the connection's settings. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. At the top of the dataset's page, click the icon next to Sync Status to read the full error. 2. If it points at the source, open the dataset's Connection tab and click Test; correct the settings with Edit, then Update. 3. The next scheduled run retries on its own. If runs keep failing, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3130 — Another run is already being prepared Task generation for this dataset is already in progress elsewhere in the cluster, so this attempt stopped rather than duplicating it. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. No action is needed: the run already being prepared continues normally. 2. It appears on the dataset's Tasks tab, or on the Monitoring tab for an external table; use Run Now or Sync Now again only if no new run shows up there. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3131 — The ingestion schedule is not firing correctly This dataset's next scheduled run time has passed without the run starting, so its schedule has stopped firing. This is a fault in the cluster's scheduler, not in your settings. **User correctable** — you can correct this. **What to do** 1. On the Configuration tab click Edit, change the schedule value under the dataset's ingestion task (a minute's shift is enough), then Apply and Save. Saving it unchanged does not restart the schedule. 2. Next Run at the top of the dataset's page then shows the coming run. If it falls behind again, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration) ### DPE-3132 — That file could not be read The dataset was not created because a record in the uploaded file could not be read or converted to its column's type. The cluster does not say which record. **User correctable** — you can correct this. **What to do** 1. At the Data Setup step, check Data Format matches the file; for CSV also check Custom Delimiter and Skip File Header. 2. At the Data Modeling step, check each column's type fits every value: text in a number column fails the whole file. 3. Correct or remove the records that differ and create the dataset again, or set continueOnError to true under ingestionConfig at the Validate step to skip them. [Related documentation](/corecapabilities/ingestdata/dataportal/batch/upload) ### DPE-3133 — The time column is missing from the uploaded data The data could not be stored because it carries no time range for `{timeColumn}`, the dataset's time column. That happens when the column is missing from the source files or holds no usable values. **User correctable** — you can correct this. **What to do** 1. Make sure every source file has `{timeColumn}`, with values, in the type the schema expects. 2. To store such data anyway, set segmentTimeValueCheck to false under ingestionConfig on the Configuration tab (Edit, Apply, Save); without a time range, retention will not remove it. 3. Then run the ingestion again: Retry Sync on the Monitoring tab for an external table, otherwise the next scheduled run. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-3134 — Preview is not available for this dataset Sample rows cannot be fetched for a dataset that ingests both batch and streaming data. The dataset itself is unaffected; only the preview is. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. To look at its rows, query the dataset from the Query Console. ### DPE-3135 — Two of this dataset's tasks cannot be used together The table config turns on both `{task}` and `{conflictingTask}`. They rewrite the same data, so only one of them may be configured and the configuration was not saved. **User correctable** — you can correct this. **What to do** 1. Open the table config JSON — the Configuration tab of an existing dataset (Edit) or, while creating one, the Validate step's Table Config tab (Edit configuration) — delete one of the two entries under task, and apply the change. 2. If one is UpsertCompactionTask or UpsertCompactMergeTask and the other refreshes segments, keep the refresh task: it already does the compaction work. ### DPE-3136 — The last scheduled run did not start The schedule's last run time passed with no record of a run being prepared for it, so nothing was ingested then. Usually an earlier run was still in progress, or the run was held back by a capacity limit or another run being prepared. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Open the dataset's Tasks tab: if a run is still in progress there, nothing is wrong and the schedule resumes once it finishes. 2. To ingest now, use Run Now at the top of the dataset's page where offered; its message says why no run started. 3. If runs keep being skipped with nothing in progress, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3137 — Your access could not be checked The cluster tried to check whether your account may do this, but the check itself failed, so the request was refused without a decision. This is not a missing permission; it is on our side. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Reload the page and try again; a check that fails once usually succeeds on the next attempt. 2. If it keeps failing, contact StarTree support and quote the reference shown with this error. No permission change is needed on your side. ### DPE-3140 — No stream ingestion for this dataset The cluster holds no stream ingestion for `{table}`, so there is no live progress to show. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Refresh the page, then open the dataset from the list. 2. If it should be ingesting from a stream, contact StarTree support and quote the reference shown with this message. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-3141 — A sync setting needs a feature the source lacks The table's sync settings turn on `{flag}`, but its catalog or source does not support it, so the sync stopped before reading any data. **User correctable** — you can correct this. **What to do** 1. Open the table's Configuration tab, click Edit, and in its ExternalTableSyncTask settings set `{flag}` to false or remove it; then Apply and Save. 2. Click Sync Now to run the sync again. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-3142 — The starting snapshot setting is not valid The table's sync settings ask to start from a snapshot this source cannot select. S3 and GCS Data Lake sources read only the latest files; Iceberg sources accept LATEST, EARLIEST or a numeric snapshot ID. **User correctable** — you can correct this. **What to do** 1. Open the Configuration tab, click Edit, and in the ExternalTableSyncTask settings set the initial version selector to LATEST, or for an Iceberg source to EARLIEST, or to SNAPSHOT\_ID together with a numeric initial snapshot ID; then Apply and Save. 2. Click Sync Now. **On S3 Data Lake, GCS Data Lake** The table's sync settings ask to start from a snapshot, but a data lake source has no snapshots to choose from: it always reads the latest files. 1. Open the Configuration tab, click Edit, and in the ExternalTableSyncTask settings set the initial version selector to LATEST; then Apply and Save. 2. Click Sync Now. **On Databricks Unity Catalog, Apache Polaris, Snowflake Open Catalog, Snowflake Horizon Catalog, AWS Glue, Amazon S3 Tables** The table's sync settings name a starting snapshot the catalog cannot select: the selector is not LATEST, EARLIEST or SNAPSHOT\_ID, or the snapshot ID is not a number. 1. Open the Configuration tab, click Edit, and in the ExternalTableSyncTask settings set the initial version selector to LATEST or EARLIEST, or to SNAPSHOT\_ID together with a numeric initial snapshot ID; then Apply and Save. 2. Click Sync Now. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-3143 — A sync setting has a value out of range One of the table's numeric sync settings, such as the files scanned per run, the batch size or a wait time, has a value the sync cannot use. **User correctable** — you can correct this. **What to do** 1. Open the Configuration tab, click Edit, and in the ExternalTableSyncTask settings check the value the error names: the maximum files scanned per run must be at least the batch size, and wait and poll times must be positive numbers of milliseconds. Then Apply and Save. 2. Click Sync Now. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-3144 — The table's source type changed after it was created The table's saved sync progress belongs to a different kind of source from the one its settings now name, so the sync cannot continue from where it stopped. **User correctable** — you can correct this. **What to do** 1. Change the catalog settings back in the table's Configuration tab, or, to read from the new source, delete the table and create it again from Data & AI, Tables, Connect external table. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-3145 — The sync could not save or read its progress The cluster could not read or record how far this table's sync has got, so the run stopped without losing any data. **Transient** — temporary; retrying usually works. **What to do** 1. Click Sync Now to run it again; the next run continues from the last saved point. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/observability) ### DPE-3146 — Servers did not finish loading the new data The sync built and uploaded the new data, but some servers did not report it ready in the time allowed, so the new snapshot was not made visible to queries. **Transient** — temporary; retrying usually works. **What to do** 1. Click Sync Now to try again. Servers that were busy usually finish on a later run, and queries keep returning the previous snapshot meanwhile. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error; the servers may need more capacity. [Related documentation](/corecapabilities/external-table/observability) ### DPE-3147 — The sync could not upload the segments it built The sync built segments from the source files but could not upload them to the cluster's storage, so the run stopped. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Click Sync Now to try again. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-3148 — The cluster ran out of disk space during the sync The sync ran out of local disk space while building segments, so the run stopped. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Contact StarTree support and quote the reference shown with this error; the cluster needs more disk. 2. Click Sync Now once support confirms the capacity. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-3149 — One file stopped a grouped sync This table groups files into shared segments, which needs every file in the run to convert, and only `{succeeded}` of `{total}` did. **User correctable** — you can correct this. **What to do** 1. Find the failing file in the earlier errors for this run, and fix or remove it at the source. 2. Or turn off segment groups in the table's Configuration tab, so the sync can proceed file by file. 3. Click Sync Now. [Related documentation](/corecapabilities/external-table/segment-groups) ### DPE-3150 — The table's definition was missing during the sync The sync could not read the table's schema or configuration while it ran, so it stopped. This is on our side, unless the table was being deleted at the time. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. If the table still exists, click Sync Now to run the sync again. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-3151 — External table sync is not available on this cluster The cluster could not start a sync for this table because external table sync is not running there. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Contact StarTree support and quote the reference shown with this error. 2. Queries on data that has already synced keep working meanwhile. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-3152 — The sync status is unavailable right now The cluster could not report this table's sync status just now. **Transient** — temporary; retrying usually works. **What to do** 1. Refresh the page in a moment. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/observability) ### DPE-3153 — This table has no sync settings The table's configuration has no ExternalTableSyncTask settings, so its sync cannot be paused, resumed or started. Tables created with an older sync task lack them. **User correctable** — you can correct this. **What to do** 1. Open the Configuration tab and look for an ExternalTableSyncTask entry in the task settings. If the table uses an older task instead, create it again from Data & AI, Tables, Connect external table, then delete the old one. 2. If the entry was removed by an edit, restore it, or contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-3154 — The last sync failed without a recorded reason The table's last sync run failed in its `{phase}` step, and the cluster recorded no reason for it. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. LISTING\_FILES usually means the storage credentials or path; GENERATING\_SEGMENTS a data type that does not match the schema; COMPRESSING\_SEGMENTS server resources; UPLOADING\_SEGMENTS the cluster's storage. 2. Click Sync Now to run it again; the run records its reason if it fails again. 3. If it keeps happening, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-3155 — The sync is still running Sync Now runs the sync while the page waits, and this one took longer than the page waits for. It keeps running in the background. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Follow its progress on the table's Monitoring tab. 2. Do not click Sync Now again until it finishes; a second request waits for the first. [Related documentation](/corecapabilities/external-table/observability) ### DPE-3156 — The table's storage quota is full The table's storage quota has no room for the segments this sync built, so the cluster refused them. **User correctable** — you can correct this. **What to do** 1. Raise the storage value under quota in the table's Configuration tab, or remove it; then Apply and Save. 2. Click Sync Now. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-3157 — The sync could not read the table's data files Object storage refused to let the sync read the table's files, so no data was loaded. Storage gives no reason with that refusal: a missing read permission, a bucket policy, files owned by another account and an encryption key the identity cannot use all look the same. **User correctable** — you can correct this. **What to do** 1. Grant the identity in the connection's storage settings read access to the table's data files, and use of the encryption key if the bucket encrypts them with one. 2. Check that no bucket policy denies that identity. 3. Click Sync Now to run the sync again. To use other credentials instead, change them from the table's Connection tab: click Edit, then, if the tab shows the connection, Test and Update; if it shows this table's own settings, Save. **On Databricks Unity Catalog, Apache Polaris, Snowflake Open Catalog, Snowflake Horizon Catalog** 1. Grant the identity under Storage on the connection s3:GetObject on the table's data files, and kms:Decrypt on the key if the bucket uses AWS KMS. If the catalog hands out its own storage credentials, grant the same to the storage credential it uses for this table. 2. Check that no bucket policy denies that identity. 3. Click Sync Now to run the sync again. To use other credentials instead, change them from the table's Connection tab: click Edit, then, if the tab shows the connection, Test and Update; if it shows this table's own settings, Save. **On AWS Glue** 1. Grant the identity under Storage s3:GetObject on the table's data files, and kms:Decrypt on the key if the bucket uses AWS KMS. When Reuse metastore credentials is Yes, that is the identity under Metastore. 2. Check that no bucket policy denies that identity. 3. Click Sync Now to run the sync again. To use other credentials instead, change them from the table's Connection tab: click Edit, then, if the tab shows the connection, Test and Update; if it shows this table's own settings, Save. **On Amazon S3 Tables** 1. Grant the identity under Storage s3tables:GetTableData on the table bucket, and kms:Decrypt on the key if the bucket uses AWS KMS. When Reuse metastore credentials is Yes, that is the identity under Metastore. 2. Click Sync Now to run the sync again. To use other credentials instead, change them from the table's Connection tab: click Edit, then, if the tab shows the connection, Test and Update; if it shows this table's own settings, Save. **On S3 Data Lake** 1. Grant the connection's identity s3:GetObject on the files under its bucket and prefix, and kms:Decrypt on the key if the bucket uses AWS KMS. 2. Check that no bucket policy denies that identity, and that the files belong to the bucket owner's account. 3. Click Sync Now to run the sync again. To use other credentials instead, change them from the table's Connection tab: click Edit, then, if the tab shows the connection, Test and Update; if it shows this table's own settings, Save. **On GCS Data Lake** 1. Grant the service account that owns the connection's HMAC key storage.objects.get on the bucket, for example through the Storage Object Viewer role. 2. Click Sync Now to run the sync again. To use other credentials instead, change them from the table's Connection tab: click Edit, then, if the tab shows the connection, Test and Update; if it shows this table's own settings, Save. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-3160 — Preview cannot read through that task Sample rows are read through the table config's ingestion task, and its task, `{taskType}`, is not one preview can read, so no rows could be sampled. **User correctable** — you can correct this. **What to do** 1. While creating the dataset, choose Edit configuration on the Validate step's Table Config tab, put the ingestion task back under task, such as FileIngestionTask for files, then Apply Changes. 2. Add other tasks on the Configuration tab once the dataset exists. [Related documentation](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration) ### DPE-3161 — The uploaded file is not kept for another run This dataset was created from an uploaded file, and the file is not kept once the dataset exists, so Run Now has nothing to read. **User correctable** — you can correct this. **What to do** 1. To add data, select Upload File next to the dataset's name and upload the file again. 2. Run Now and scheduled runs do not apply to an uploaded file; there is nothing to fix on the connection. [Related documentation](/corecapabilities/ingestdata/dataportal/batch/upload) ### DPE-3162 — The previous run is still finishing Run Now did not start a new run because tasks from the previous run, or from before the last configuration change, are still finishing. Nothing is wrong with the dataset. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Select Run Now again in a few minutes, or wait for the next scheduled run. [Related documentation](/corecapabilities/observability/minion-task-runs) ## Queries ### DPE-4301 — Query could not be parsed The SQL could not be read, so nothing was run. **User correctable** — you can correct this. **What to do** 1. Select Run in the Query Console to see the line and column where Pinot stopped reading the statement; a query that cannot be parsed does not run. 2. Fix what is at that spot: often an unclosed quote, bracket or comment, or a missing comma between columns. 3. Wrap any table or column name that is also an SQL keyword, such as timestamp or date, in double quotes. 4. If part of the editor text is selected, only that part is analyzed; clear the selection or select one whole statement, then choose Analyze Query again. [Related documentation](/corecapabilities/query_data/query_interfaces/query_console) ### DPE-4302 — Query took too long The query did not finish within the time allowed and was stopped. **Transient** — temporary; retrying usually works. **What to do** 1. Narrow the time range or add a filter, then run it again. [Related documentation](/corecapabilities/query_data/query_interfaces/query_console) ### DPE-4303 — Table not found for this query The query names a table this workspace does not have, so it was not analyzed. The analyzer matches table names exactly, including upper and lower case, even where the query itself runs. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Type the table name exactly as it appears under Tables on the left of the Query Console, matching upper and lower case, then choose Analyze Query again. ### DPE-4304 — Query analysis is not set up Query analysis is switched on for this environment, but the service that performs it has no endpoint or credential configured, so no analysis can run. Nothing about the query caused this. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Contact StarTree support to finish enabling query analysis for this environment. 2. On a self-managed install, ask its administrator to set QUERY\_ANALYZER\_LLM\_API\_URL and QUERY\_ANALYZER\_LLM\_API\_KEY for Data Manager; Show details names the one that is missing. ### DPE-4305 — That query could not be parsed The SQL could not be read, so nothing was run. **User correctable** — you can correct this. **What to do** 1. Select Run in the Query Console to see the line and column where Pinot stopped reading the statement; a query that cannot be parsed does not run. 2. Fix what is at that spot: often an unclosed quote, bracket or comment, or a missing comma between columns. 3. Wrap any table or column name that is also an SQL keyword, such as timestamp or date, in double quotes. 4. If part of the editor text is selected, only that part is analyzed; clear the selection or select one whole statement, then choose Analyze Query again. ### DPE-4306 — Query analysis usage limit reached Query analysis has used up its usage limit for this environment, so it cannot analyze queries until the limit resets. Nothing about the query caused this. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Run the analysis again once the limit resets; the details give the reset time when the limit is weekly. 2. Contact StarTree support if you need query analysis before then. ### DPE-4307 — Query analysis is temporarily unavailable The service that performs query analysis could not complete this request, so no recommendations were produced. This is on our side, not the query. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Run the analysis again in a few minutes. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. ### DPE-4308 — Analysis of this query was too large to finish The analysis of this query ran past the most the analysis service can return, so it was cut off and discarded. Analyzing the same query again is likely to be cut off the same way. **User correctable** — you can correct this. **What to do** 1. Select one subquery or join in the Query Console editor, then choose Analyze Query; only the selected text is analyzed, so the analysis is shorter. 2. If you chose High Effort Analysis, choose Low Effort Analysis, which reports less detail for each finding. 3. Contact StarTree support to raise the analysis output limit if you need the whole query analyzed. ### DPE-4309 — Query analysis service refused access The service that performs query analysis did not accept the credentials StarTree set up for this environment, so no analysis can run. This is on our side; workspace permissions do not affect it. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Contact StarTree support and quote the reference shown with this error so the credentials can be restored. ### DPE-4310 — Query analysis request was rejected The service that performs query analysis rejected the request we sent for this query, so nothing was analyzed. This is a problem on our side, not in the query. **Internal** — a defect on our side. **What to do** 1. Contact StarTree support and quote the reference shown with this error. ### DPE-4311 — Query analysis is turned off for this organization Query analysis runs only after your organization turns it on in Settings, and it is currently off, so nothing was analyzed. This is a setting, not a permission on your account. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Open Settings and turn on Query Analyzer. It shows what query analysis sends to the model before you confirm. 2. If the switch is unavailable to you, ask someone who can change settings to turn it on. ### DPE-4312 — Query too large to analyze This query, the metadata of every table it reads and, for High Effort Analysis, its execution statistics are more than the analysis service can take in at once, so nothing was analyzed. Analyzing the same query again stops at the same point. **User correctable** — you can correct this. **What to do** 1. Select one subquery or join that reads fewer tables in the Query Console editor, then choose Analyze Query; only the selected text is analyzed. 2. If you chose High Effort Analysis, choose Low Effort Analysis, which does not send execution statistics. 3. Contact StarTree support if you need the whole query analyzed. ## Schema and table configuration ### DPE-2201 — Dataset name not allowed A dataset name may use only letters, numbers, hyphens and underscores, and this one contains another character, such as a space or a symbol. **User correctable** — you can correct this. **What to do** 1. Change Table Name on the Additional Configuration step to letters, numbers, hyphens and underscores only, then submit again. 2. If you are calling the API directly, apply the same rule to the name in the request. 3. Choose with care: a dataset cannot be renamed once it exists. [Related documentation](/corecapabilities/ingestdata/dataportal/additional-config-create-table) ### DPE-2202 — Dataset name is required The request to create a dataset did not include a name, so the dataset could not be created. **User correctable** — you can correct this. **What to do** 1. Fill in Table Name on the Additional Configuration step, then submit again. 2. If you are calling the API directly, set the dataset name in the request; letters, numbers, hyphens and underscores are accepted. [Related documentation](/corecapabilities/ingestdata/dataportal/additional-config-create-table) ### DPE-2203 — Double underscore not allowed in name Dataset names cannot contain a double underscore, which Pinot reserves for internal naming. **User correctable** — you can correct this. **What to do** 1. Change Table Name on the Additional Configuration step so it has no double underscore; a single underscore is fine. 2. The schema takes the same name, so one change fixes both. [Related documentation](/corecapabilities/ingestdata/dataportal/additional-config-create-table) ### DPE-2204 — Column name not allowed A column name contains a forward slash or a backslash. Those two characters are reserved because column names become path segments internally. **User correctable** — you can correct this. **What to do** 1. On the Schema tab choose Edit, open the column with its Edit (pencil) action, and change Column Name so it has no forward slash or backslash. 2. Choose Update Column, then Save. 3. Spaces are refused too; letters, numbers and underscores are always safe. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2205 — Schema is not backward-compatible The updated schema removes or changes a column the dataset already has, so it cannot replace the current schema. **User correctable** — you can correct this. **What to do** 1. On the Schema tab choose Reset All to restore the saved schema, then make only additions and Save. 2. Adding columns is always allowed. Removing a column, or changing an existing column's Data Type or Multi Value setting, or the primary key, is refused. 3. To change one of those, create a new dataset with the schema you want and load the data into it; stored data cannot be reinterpreted in place. [Related documentation](/corecapabilities/manage-data/schema-evolution) ### DPE-2206 — Table configuration rejected The table configuration for this dataset could not be read: a value has the wrong type, a required section is missing, or the JSON is malformed. **User correctable** — you can correct this. **What to do** 1. Choose Show details on this message to see the field that could not be read. 2. On the Configuration tab choose Edit, correct that field, then Apply and Save. While creating, use Edit configuration under Table Config on the Validate step. 3. Look for text where a number or true/false is expected, or a missing offline or realtime section; in a pasted config, also a trailing comma. [Related documentation](/api-reference/table/validate-table-config-for-a-table) ### DPE-2207 — Datetime format not recognised The Format on a datetime column is missing, or does not match the values in the source data. **User correctable** — you can correct this. **What to do** 1. Open the column with its Edit (pencil) action on the Data Modeling step or the Schema tab, and set Format to match the source values exactly. 2. Epoch numbers use EPOCH|MILLISECONDS (or SECONDS); text dates use SIMPLE\_DATE\_FORMAT| followed by their pattern, such as yyyy-MM-dd. 3. Choose Fetch Preview to check the values now parse. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2208 — This dataset already exists A dataset with this name already exists in the workspace. **User correctable** — you can correct this. **What to do** 1. Choose a different dataset name. 2. To change the existing dataset instead, open it from the Tables list — creating it again is not needed. 3. A deleted dataset's name is free again only once its removal finishes. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2209 — Dataset not found This dataset is no longer available. It may have been deleted, or the link may be out of date. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Refresh the page, then open the dataset from the list. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2210 — Dataset not found This dataset is no longer available. It may have been deleted, or the link may be out of date. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Refresh the page, then open the dataset from the list. ### DPE-2211 — Transform refers to a field that does not exist The transform function for `{columnName}` reads `{property}`, but no value of that name reaches the script. **User correctable** — you can correct this. **What to do** 1. Open `{columnName}` with its Edit (pencil) action on the Data Modeling step or the Schema tab, and check its Transform Function. 2. In a Groovy transform, list every field the script reads after the script, as in Groovy(\{a + b}, a, b); names are case-sensitive. 3. If `{property}` sits inside a field that arrives as JSON text, read it with jsonPathString instead of a dot. 4. Choose Fetch Preview to check the column now resolves. [Related documentation](/recipes/groovy-transformation-functions) ### DPE-2212 — Transform expression could not be evaluated A column's transform function failed on the source data, or two transforms depend on each other. **User correctable** — you can correct this. **What to do** 1. Open the column named in the details with its Edit (pencil) action on the Data Modeling step or the Schema tab, and check its Transform Function. 2. Confirm every field it reads exists in the source with the type it expects, and that no two transforms read each other's output. 3. If only some records fail, set continueOnError to true under Ingestion Configuration in the Json view and Apply; those values are then left empty. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/concepts) ### DPE-2213 — Column value did not match its type A value in column `{columnName}` could not be converted to the column's Data Type, so the rows could not be loaded. **User correctable** — you can correct this. **What to do** 1. While creating, open `{columnName}` with its Edit (pencil) action on the Data Modeling step and choose a Data Type its values fit, such as STRING for a dimension with mixed values. 2. Or correct the values at the source; an empty string or text in a numeric column is the usual cause. 3. To keep loading and leave such values empty, set continueOnError to true under Ingestion Configuration in the Json view, then Apply. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2214 — Dataset not found in Pinot This dataset does not exist in the cluster. It may have been deleted, or never finished creating. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Refresh the page, then open the dataset from the list. ### DPE-2215 — That identifier is not valid One of the identifiers in this request is not in a form this service recognises. **User correctable** — you can correct this. **What to do** 1. Reload the page and repeat the action from the list rather than from a saved link. 2. If you are calling the API directly, the identifier must be the full reference the list endpoint returns, not just a name. ### DPE-2216 — Dataset removed The dataset `{table}` was removed. Its entry here will clear on its own shortly. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. Refresh the list in a moment. ### DPE-2217 — Dimension table needs a primary key A dimension table has to declare at least one primary key column, and this one declares none. **User correctable** — you can correct this. **What to do** 1. While creating, pick at least one column in Primary Key Columns under Table Type Dimension on the Additional Configuration step. 2. For an existing dataset, open the Schema tab, choose Edit, switch to Json, add the column names to primaryKeyColumns under Column Configuration, then Apply and Save. 3. If it should not be a dimension table, choose Table Type Fact instead, or set isDimTable to false on the Configuration tab. [Related documentation](/corecapabilities/ingestdata/dataportal/additional-config-create-table) ### DPE-2218 — That server tenant does not exist The server tenant this dimension table names has no servers at all, so the table has nowhere to be placed and the configuration was rejected. **User correctable** — you can correct this. **What to do** 1. On the Tenants step of Additional Configuration, choose another Server Tenant; its list shows the server tenants this cluster has. 2. For an existing dataset, set tenants.server in the Configuration tab (Edit, then Apply) to a tenant that has servers. The name is case-sensitive. 3. If this is the tenant you expect to use, ask a workspace administrator to assign servers to it. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2219 — A dataset of that name already exists `{table}` already exists, so it cannot be created again. **User correctable** — you can correct this. **What to do** 1. Choose a different name for the new dataset. 2. To change the existing one, open it from the Tables list and edit it instead of creating it again. ### DPE-2220 — That setting cannot be changed after creation This edit changes `{fields}`. Those settings cannot change on an existing dataset, because the data already stored depends on the current value. Each item shows the current value, then the one you entered. **User correctable** — you can correct this. **What to do** 1. In the Configuration tab, choose Edit, set each setting listed back to the value before the arrow, then Apply; your other changes can then save. 2. To use the new value, create a new dataset with it and load the data into that; an existing dataset cannot be converted in place. ### DPE-2221 — A value in the configuration has the wrong type The default null value `{value}` is not a valid `{targetType}`, the data type of its column, so the configuration could not be read. **User correctable** — you can correct this. **What to do** 1. Edit the column whose Default Null Value is `{value}` (Data Modeling step, or the Schema tab after Edit), enter a `{targetType}` value or clear the field, then choose Update Column. 2. A number column takes a number such as 0 or -1, not text like N/A; to keep a text placeholder, set Data Type to STRING. 3. In the Json view, the same setting is defaultNullValue on that column. ### DPE-2222 — The configuration could not be read Part of the dataset configuration is not in the expected structure, so it could not be read and nothing was changed. Usually a column sits in the list for another field type, a fieldType value is misspelled, or a required section such as tenants was removed. **User correctable** — you can correct this. **What to do** 1. In the Json view (Data Modeling step, or the Schema tab after Edit), keep each column in the list matching its fieldType, such as a DIMENSION in dimensionFieldSpecs. 2. To change a column's field type, use Field Type in the Table view; it moves the column for you. 3. In the Configuration tab, restore any section you removed, then Apply. ### DPE-2223 — A timestamp did not match its declared format The time value `{value}` could not be read with this SQL source's Time Column Format. That setting takes millisecondsSinceEpoch, secondsSinceEpoch or hoursSinceEpoch for epoch numbers, or a date pattern such as yyyy-MM-dd HH:mm:ss for text. **User correctable** — you can correct this. **What to do** 1. On the Data Setup step, enter the Time Column Format that matches the column, and give Start Time in that same format. 2. The size:unit:EPOCH form, such as 1:MILLISECONDS:EPOCH, is not accepted here, and there is no microsecond option; expose the column in milliseconds or seconds through a view at the source. ### DPE-2224 — No servers are assigned to that tenant No servers are assigned to `{tag}` for this kind of data, so the dataset could not be placed on it and the configuration was rejected. A tenant can have servers for batch data but not streaming, or the reverse. **User correctable** — you can correct this. **What to do** 1. Choose another Server Tenant on the Tenants step of Additional Configuration; for an existing dataset, set tenants.server in the Configuration tab (Edit, then Apply). 2. If a storage tier uses `{tag}`, change Tier Server Tag on the Storage step, or serverTag under tierConfigs in the Configuration tab. 3. If `{tag}` is the tenant you expect, ask a workspace administrator to assign servers to it. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2225 — Granularity value not recognised The granularity in this configuration is not a value the cluster accepts, so the configuration was rejected and nothing was changed. **User correctable** — you can correct this. **What to do** 1. Set Granularity on the Datetime column to a size and a unit, such as 1:DAYS or DAYS|1. It is required, so it cannot be left empty. 2. Units are NANOSECONDS, MICROSECONDS, MILLISECONDS, SECONDS, MINUTES, HOURS or DAYS, in capitals. A value such as 1:MILLISECONDS:EPOCH belongs in Format, not Granularity. 3. Enter only the value: a field that also holds the property name is the most common cause. Edit it on the column's row, on the Data Modeling step or the Schema tab. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/concepts) ### DPE-2226 — The two halves name different time columns A dataset that ingests both batch and streaming data must use one time column on both halves. This one names `{offline}` on the batch half and `{realtime}` on the streaming half, so the configuration was rejected. **User correctable** — you can correct this. **What to do** 1. In the Configuration tab, choose Edit, set timeColumnName under segmentsConfig to the same column in both the offline and realtime sections, then Apply. 2. Choose a Datetime column that both batch and streaming data fill; a query spans the two halves on that column. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2227 — The dataset still has tasks running It could not be deleted because `{tasks}` task(s) are still running on it. Its schedules have been cleared, so no further tasks will start. **Transient** — temporary; retrying usually works. **What to do** 1. Wait for the running tasks to finish, then send the delete again. 2. To delete it now, add ignoreActiveTasks=true to the delete request; Delete on the dataset's page in Data Portal already does this and does not wait for running tasks. [Related documentation](/corecapabilities/observability/minion-task-runs) ### DPE-2228 — The primary key names a column that is not there The schema sent to infer a configuration lists `{column}` as a primary key column but defines no column of that name, so no configuration was produced. **User correctable** — you can correct this. **What to do** 1. In the infer-config request, make every name in the schema's primaryKeyColumns match one of its columns exactly, including case. A dimension table needs at least one. 2. Or create the dataset in Data Portal: Primary Key Columns, on the Details step of Additional Configuration, offers only columns that exist. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2229 — The dataset has neither batch nor streaming settings A dataset configuration has to contain an offline section for batch data, a realtime section for streaming data, or both. This one contains neither, so it could not be checked. **User correctable** — you can correct this. **What to do** 1. Add the offline or realtime section, holding the table configuration, next to tableName and schema, and send it again. 2. In Data Portal, open the Configuration tab, choose Edit, restore the offline or realtime section, then Apply. 3. For a new dataset, use Create table on the Tables page; the source you pick fills in the right section. [Related documentation](/corecapabilities/ingestdata/ingest_data_overview) ### DPE-2230 — The time column is not in the schema The table config uses `{column}` as its time column, but the schema has no Datetime column of that name, so the change could not be saved. Either timeColumnName names a column the schema does not have, or the column was removed, renamed or given another Field Type. **User correctable** — you can correct this. **What to do** 1. If you changed timeColumnName, set it on the Configuration tab (Edit, then Apply) to a Datetime column the schema has. 2. If the schema changed, set `{column}` back to Field Type Datetime on the Schema tab or the Data Modeling step; on the Schema tab, Cancel drops the whole edit. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2231 — The star-tree index names a column that is not there The star-tree index lists `{column}` as a dimension, but the schema has no column with that name, so the configuration could not be saved. **User correctable** — you can correct this. **What to do** 1. On the Indexes tab choose Edit, open the star-tree index with its pencil icon, remove `{column}` from Dimension Split Order or pick an existing column, then Update and Save. 2. Or add `{column}` first: on the Schema tab choose Edit, then Add Column, then Save. 3. For a time bucket such as $ts$DAY, add no column: give the base column a timestamp index with that granularity on the Configuration tab instead. [Related documentation](/corecapabilities/manage-data/indexes/startree-index) ### DPE-2232 — The schema is missing a column the table config uses The table configuration for `{table}` still uses `{column}`, in an index or a field setting, but the schema has no column of that name. **User correctable** — you can correct this. **What to do** 1. To keep `{column}`, add it back: on the Schema tab, Cancel discards an unsaved deletion, or choose Edit, then Add Column, then Save. While creating, use Add Column on the Data Modeling step. 2. To drop it, first remove every mention of `{column}` on the Configuration tab: choose Edit, delete it from the index and field settings, then Apply and Save. Then delete the column. 3. Names must match exactly, including case. ### DPE-2233 — Dimension table storage quota is above the limit A dimension table is copied in full to every server, so its storage quota may not exceed `{allowed}` bytes. This configuration sets it to `{requested}` bytes. **User correctable** — you can correct this. **What to do** 1. On the Configuration tab choose Edit, lower storage under quota to the limit or remove it so the limit applies, then Apply and Save. While creating, do this under Table Config on the Validate step. 2. If the data needs more room, make it a fact table instead: Table Type Fact on the Additional Configuration step. 3. Ask StarTree support if this workspace needs a higher dimension-table limit. [Related documentation](/corecapabilities/ingestdata/dataportal/additional-config-create-table) ### DPE-2234 — Every column needs raw encoding An external table reads its data in place from the source files, so every column must use raw encoding, without a dictionary. These columns have no field config and would default to a dictionary: `{columns}`. **User correctable** — you can correct this. **What to do** 1. In the Configuration tab, choose Edit, add a fieldConfigList entry with encodingType RAW for each column listed, then Apply. While creating, use Edit configuration on the Table Config step. 2. If a listed column is not needed, delete it on the Schema tab instead. [Related documentation](/corecapabilities/external-table/indexes) ### DPE-2235 — A transform expression could not be read The Transform Function on `{column}` could not be parsed, so the configuration was rejected and nothing was changed. **User correctable** — you can correct this. **What to do** 1. Edit `{column}` (Data Modeling step, or the Schema tab after Edit) and check Transform Function for an unclosed quote or bracket, the usual cause. 2. A single quote inside a quoted argument ends it early; write it as two single quotes. Function names need parentheses, even with no arguments. 3. Use Fetch Preview to confirm the expression now works. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/concepts) ### DPE-2236 — A transform writes to a column that is not there A transform writes to `{column}`, which is not one of this dataset's columns and is not read by another transform, so its value would have nowhere to go. **User correctable** — you can correct this. **What to do** 1. Open the Json view (Data Modeling step, or the Schema tab after Edit). Under Ingestion Configuration, change that transform's columnName to an existing column or remove the entry, then Apply. 2. To keep `{column}`, add it first with Add Column; the name must match exactly, including case. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2237 — Dataset creation interrupted by a concurrent update Another part of the service recorded a dataset with this name at the same moment, so this creation could not be saved and was undone. **Transient** — temporary; retrying usually works. **What to do** 1. Create the dataset again in a moment; the name does not need to change. [Related documentation](/corecapabilities/ingestdata/ingest_data_overview) ### DPE-2238 — The schema was missing from the request The table configuration could not be worked out because the request carried no schema for the new dataset. **User correctable** — you can correct this. **What to do** 1. In the infer-config request, set outputSchema's schema to an object holding the Pinot schema under schema and its column mappings under mappings, not to the Pinot schema on its own. 2. Or create the dataset in Data Portal with Create table on the Tables page; it does not use this request. ### DPE-2239 — That function is not available when data is ingested The Transform Function for `{column}` calls `{function}`, which ingestion cannot run. Ingestion has fewer functions than queries, and some query functions go by another name there. **User correctable** — you can correct this. **What to do** 1. To read a value out of JSON, use jsonPathString(col, '$.path') in place of jsonExtractScalar(col, '$.path', 'STRING'), or jsonPathLong or jsonPathDouble for numbers. 2. For any other function, check its name and argument count against the ingestion transformation functions; both mistakes are reported this way. 3. Fix it in Transform Function (Data Modeling step, Edit on the column), then use Fetch Preview. [Related documentation](/recipes/json-transformation-functions) ### DPE-2240 — A transform reads the column it writes to The Transform Function for `{column}` takes `{column}` as one of its inputs. A transform has to write to a column other than the ones it reads, so the configuration was rejected and nothing was changed. **User correctable** — you can correct this. **What to do** 1. Keep `{column}` as the source: on the Data Modeling step choose Add Column, give the new column its own name, and enter the expression that reads `{column}` as its Transform Function. 2. Then clear the Transform Function on `{column}` itself. If you only need the new column, delete `{column}`: the new column's transform still reads it from the source. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2241 — This request did not name a dataset The request refers to a dataset, but the dataset name in it is blank, so there is nothing to look up. **User correctable** — you can correct this. **What to do** 1. Refresh the page, then open the dataset from the list. 2. If you are calling the API directly, end the dataset reference with the dataset name exactly as the list endpoint returns it: lowercase letters, numbers, hyphens and underscores. ### DPE-2242 — The ingestion filter calls a function it cannot use The ingestion filter calls `{function}`, which ingestion cannot run, or calls it with the wrong number of arguments. IN and NOT IN lists are among the forms it cannot use, so the configuration was rejected. **User correctable** — you can correct this. **What to do** 1. Write a list as strcmp comparisons: for col NOT IN ('A', 'B') use strcmp(col, 'A') != 0 AND strcmp(col, 'B') != 0; for IN, join strcmp(col, 'A') = 0 terms with OR. 2. Compare other text with strcmp too; for any other function, check its name and argument count. 3. Edit filterFunction in Ingestion Configuration (Data Modeling step, Json view) and Apply; rows for which the filter is true are left out. [Related documentation](/recipes/filtering-ingestion) ### DPE-2243 — The ingestion filter could not be read The ingestion filter is not a valid expression, so the configuration was rejected and nothing was changed. **User correctable** — you can correct this. **What to do** 1. Test equality with a single =, not ==, and inequality with != or \<>; join conditions with AND, OR and NOT. 2. Put text in single quotes and close every quote and bracket; a quote inside text is written twice. 3. Compare text through strcmp, as in strcmp(col, 'value') = 0; a plain = on text fails once rows are read. 4. Edit filterFunction in Ingestion Configuration (Data Modeling step, Json view) and Apply. Rows for which the filter is true are left out. [Related documentation](/recipes/filtering-ingestion) ### DPE-2244 — The datetime format is not written correctly The datetime format `{format}` is not in a form the cluster can read, so the configuration was rejected and nothing was changed. **User correctable** — you can correct this. **What to do** 1. For numbers counted from 1970, write EPOCH|unit, optionally |size: EPOCH|SECONDS or EPOCH|MILLISECONDS|1. The unit is NANOSECONDS, MICROSECONDS, MILLISECONDS, SECONDS, MINUTES, HOURS or DAYS. 2. For dates written as text, write SIMPLE\_DATE\_FORMAT|pattern, such as SIMPLE\_DATE\_FORMAT|yyyy-MM-dd HH:mm:ss. MM is the month and mm the minute. 3. For a TIMESTAMP column, write TIMESTAMP. 4. Set it in the column's Format: Data Modeling step, Edit on the column, or the Schema tab after Edit. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/timezones) ### DPE-2245 — Text was compared as a number during ingestion An ingestion filter or transform compares `{column}` with a text value using =, != or a similar operator. During ingestion those operators compare numbers, so the text could not be compared and the rows could not be read. **User correctable** — you can correct this. **What to do** 1. Compare text through strcmp, which returns 0 when the values match: strcmp(`{column}`, 'value') != 0 in place of `{column}` != 'value', and strcmp(`{column}`, 'value') = 0 in place of `{column}` = 'value'. 2. Edit the filter in Ingestion Configuration (Data Modeling step, Json view) or the column's Transform Function, then preview the rows again. [Related documentation](/recipes/filtering-ingestion) ### DPE-2246 — That retention unit is not accepted Retention Time Unit is set to `{unit}`, but retention has to be given in days or a smaller unit such as hours, so the configuration was not saved. **User correctable** — you can correct this. **What to do** 1. On the Storage step of Additional Configuration, set Retention Time Unit to DAYS and enter the period in days in Retention Time Value: 365 for one year, 730 for two. 2. For an existing dataset, open the Configuration tab, choose Edit, set retentionTimeUnit to DAYS with retentionTimeValue in days, and choose Apply. 3. YEARS is listed as a unit but is not accepted yet, so convert years to days. ### DPE-2247 — A column is set to both raw and dictionary encoding Column `{column}` is listed in noDictionaryColumns, so it is stored raw, but its fieldConfigList entry sets encodingType to DICTIONARY. The two disagree, so the configuration was not saved. **User correctable** — you can correct this. **What to do** 1. Open the table config JSON: the Configuration tab of an existing dataset (Edit) or, while creating one, the Validate step's Table Config tab (Edit configuration). 2. To keep a dictionary, remove `{column}` from noDictionaryColumns (and noDictionaryConfig, if present) under tableIndexConfig. To store it raw, set encodingType to RAW in its fieldConfigList entry instead. Then apply the change. ### DPE-2248 — A range index on a text column needs a dictionary Column `{column}` is not numeric and has no dictionary, and a range index can only be built on such a column through its dictionary, so the configuration was not saved. **User correctable** — you can correct this. **What to do** 1. On the Indexes tab, choose Edit, then either set the Encoding of `{column}` to Dictionary or remove its range index, and choose Save. 2. In the table config JSON (the Validate step's Table Config tab while creating, or the Configuration tab), remove `{column}` from noDictionaryColumns and set its encodingType to DICTIONARY, or drop range from its indexes. ### DPE-2249 — A primary key column is not in the schema The schema lists a primary key column that is not one of its columns, or that is a built-in name starting with \$, so the configuration was not saved. **User correctable** — you can correct this. **What to do** 1. Choose the key again in Primary Key Columns, on the Ingestion Type step for an upsert or deduplication dataset or the Details step for a dimension table. That list offers only columns that exist. 2. If you edited the schema JSON, make every name in primaryKeyColumns match a column exactly, including case. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2250 — An upsert dataset cannot have a star-tree index This dataset uses upsert and also declares a star-tree index, or turns on the default one. The two cannot be combined, so the configuration was not saved. **User correctable** — you can correct this. **What to do** 1. On the Indexes tab, choose Edit, delete each row under StarTree Index, and choose Save. 2. If enableDefaultStarTree is true, or while creating the dataset, edit the JSON instead (the Validate step's Table Config tab, or the Configuration tab): empty starTreeIndexConfigs and set enableDefaultStarTree to false under tableIndexConfig. 3. If the star-tree index matters more than upsert, create the dataset again with Ingestion Type set to Append. ### DPE-2251 — A column's index settings could not be read One column's index settings could not be read, usually because the same index is declared twice: in a list under tableIndexConfig and in that column's fieldConfigList entry. **User correctable** — you can correct this. **What to do** 1. Open the table config JSON: the Configuration tab of an existing dataset (Edit) or, while creating one, the Validate step's Table Config tab (Edit configuration). 2. For each column with an indexes block in fieldConfigList, remove that index from the lists under tableIndexConfig (such as invertedIndexColumns or rangeIndexColumns) and keep every value under indexes an object. Then apply the change. ### DPE-2252 — The schema name was changed before the dataset was named Until the dataset is named, its schema keeps the temporary name preview\_table. The schema JSON was edited to name it `{schemaName}`, so it no longer matches and the change was not applied. **User correctable** — you can correct this. **What to do** 1. On the Data Modeling step, open the Json view and either choose Cancel to drop the edit, or set schemaName under Column Configuration back to preview\_table and choose Apply. 2. Give the dataset its name in Table Name on the Details step of Additional Configuration. The schema takes that name automatically. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2253 — Stream settings are declared in two places This streaming configuration has stream settings both in tableIndexConfig.streamConfigs and in ingestionConfig.streamIngestionConfig. Only one may hold them, so the configuration was not saved. **User correctable** — you can correct this. **What to do** 1. Delete streamConfigs from tableIndexConfig and keep the settings in ingestionConfig.streamIngestionConfig.streamConfigMaps, moving across any key you still need, such as a flush threshold. 2. In Data Portal, edit it on the Validate step's Table Config tab (Edit configuration, then Apply Changes), or on an existing dataset's Configuration tab (Edit, then Apply). ### DPE-2254 — That column name is reserved `{column}` is the name of a built-in column every dataset already has, so a column of your own cannot use it and the configuration was not saved. **User correctable** — you can correct this. **What to do** 1. On the Data Modeling step, open the Table view and delete the `{column}` row, then continue. 2. If you need its values, rename the column in the source data, for example without the leading \$, and preview again. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2255 — A dataset with this name already stores data elsewhere A dataset named `{table}` already exists and keeps its data at a fixed storage location. This configuration points that storage somewhere else, which would cut off the data already stored, so it was not saved. **User correctable** — you can correct this. **What to do** 1. If you are creating a new dataset, go back to Additional Configuration and choose a different Table Name. 2. If you are editing `{table}`, put bucket and pathPrefix under tierConfigs back to their saved values. 3. To keep the name but store the data somewhere new, delete `{table}` and create it again. [Related documentation](/corecapabilities/external-table/overview) ### DPE-2256 — The dataset was deleted, but we could not confirm it The cluster removed `{table}`, but its confirmation came back in a form we could not read, so the request was reported as failed. This is on our side. The dataset's entry in the list clears on its own within a few minutes. **Internal** — a defect on our side. **What to do** 1. Refresh the Tables list in a few minutes; the dataset should no longer be listed. Deleting it again is not needed and would show this same message. 2. If it is still listed after 10 minutes, contact StarTree support and quote the reference shown with this error. ### DPE-2257 — No brokers are assigned to that tenant The broker tenant `{tenant}` has no brokers assigned to it, so this dataset could not be placed on it and the configuration was rejected. **User correctable** — you can correct this. **What to do** 1. Choose a tenant that has brokers: Broker Tenant on the Tenants step of Additional Configuration lists only those; in the JSON, set tenants.broker on the Validate step's Table Config tab or the Configuration tab. 2. Until a tenant is chosen, the Data Modeling step checks against the default tenant. If this appeared there, or `{tenant}` is the one you want, ask a workspace administrator to assign brokers to `{tenant}`. ### DPE-2258 — Batch and streaming settings could not be checked This dataset has both a batch and a streaming half. When columns are edited, Data Portal checks the change with a temporary time column standing in for the streaming half's, and the batch half's time column does not match it. The mismatch comes from that placeholder, not from your settings. **Internal** — a defect on our side. **What to do** 1. To change this dataset's columns now, open Cluster Manager > Pinot Components, select the dataset on the Tables tab, choose Edit Schema, make the change and choose Save. 2. Contact StarTree support and quote the reference shown with this error so the check can be corrected. ### DPE-2259 — That function cannot be used in a streaming transform The Transform Function on `{column}` calls `{function}`, which can return a different value each time it runs. A streaming dataset must get the same result every time a record is read, so its transforms may only use functions whose result depends on their inputs. **User correctable** — you can correct this. **What to do** 1. Choose Edit on `{column}`, on the Data Modeling step or on the Schema tab after Edit, and remove `{function}` from its Transform Function. 2. To record arrival time, map the column from a timestamp carried in the source records instead. 3. Clock functions such as now() and ago() are refused; string, math and date-conversion functions are fine. ### DPE-2260 — Time Column Format not accepted for this source The Start Time or End Time `{value}` could not be read with the Time Column Format `{format}`. For a SQL source that setting takes millisecondsSinceEpoch, secondsSinceEpoch or hoursSinceEpoch for epoch numbers, or a date pattern such as yyyy-MM-dd HH:mm:ss for text. **User correctable** — you can correct this. **What to do** 1. On the Data Setup step, write Start Time, and End Time if set, in the Time Column Format: 2020-04-01 00:00:00 for yyyy-MM-dd HH:mm:ss, or a number such as 1585699200000 for millisecondsSinceEpoch. 2. If the format itself is wrong, set Time Column Format to the column's own: millisecondsSinceEpoch, secondsSinceEpoch, hoursSinceEpoch, or a date pattern. 3. The 1:MILLISECONDS:EPOCH form is not accepted here, and there is no microsecond option; expose the column in milliseconds or seconds through a view at the source. ### DPE-2261 — The Transform Function holds JSON, not an expression The Transform Function for `{column}` contains configuration text, such as a transformFunction key or a whole column entry in braces, instead of the expression on its own, so it could not be read and nothing was changed. **User correctable** — you can correct this. **What to do** 1. Edit `{column}` on the Data Modeling step and leave only the expression in Transform Function, with no key, surrounding double quotes or braces: for example jsonPathString(source\_column, '\$.field'). 2. The result is written to the column you are editing. To write it to a different column, choose Add Column and give the new column that name. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2262 — External tables run only the sync task The table configuration includes the `{taskType}` task, which an external table cannot run; only ExternalTableSyncTask is allowed. **User correctable** — you can correct this. **What to do** 1. In the Configuration tab, click Edit and remove the `{taskType}` entry from the task settings, keeping ExternalTableSyncTask; then Apply and Save. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-2263 — That setting is not supported on external tables External tables do not support `{detail}`, which this configuration sets. **User correctable** — you can correct this. **What to do** 1. Remove it in the Indexes tab or the Configuration tab; the external table indexes page lists what is supported. 2. Save the configuration again. [Related documentation](/corecapabilities/external-table/indexes) ### DPE-2264 — A star-tree index cannot be used with deletion vectors The configuration has a star-tree index and enables deletion vectors, and external tables cannot combine the two. **User correctable** — you can correct this. **What to do** 1. In the Configuration tab, remove the star-tree index settings, including enableDefaultStarTree, or set enableDeletionVectors to false; then Apply and Save. [Related documentation](/corecapabilities/external-table/deletion-vectors) ### DPE-2265 — External tables cannot transform columns The configuration adds a derived column or an ingestion transform, which an external table cannot apply because it reads the source files as they are. **User correctable** — you can correct this. **What to do** 1. Remove transform functions from the schema and transformConfigs from ingestionConfig in the Configuration tab, and compute derived values in queries instead. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-2266 — The table's sync settings are incomplete The table's ExternalTableSyncTask settings are missing a value the sync needs, such as the catalog type or one of the catalog's settings. **User correctable** — you can correct this. **What to do** 1. Open the Configuration tab and compare the ExternalTableSyncTask settings with the connection's: catalogType and the settings for that catalog must be present. Restore any value an edit removed, then Apply and Save. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-2267 — A sync setting must be true or false Setting `{key}` is set to `{value}`, which is neither true nor false. **User correctable** — you can correct this. **What to do** 1. In the Configuration tab set `{key}` to true or false; then Apply and Save. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-2268 — The table's storage tier settings are incomplete An external table keeps its segments in a cloud storage tier, and this configuration's tier settings are missing or incomplete. **User correctable** — you can correct this. **What to do** 1. Restore the tier settings that Connect external table generated for this table, rather than writing them by hand; each tier needs its storage backend properties. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-2269 — A storage tier's location cannot be changed The change would move the table's storage tier to another backend or location, which would leave the segments already stored there unreachable. **User correctable** — you can correct this. **What to do** 1. Keep the existing tier settings. To keep the table's data somewhere else, create a new table and delete this one. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-2270 — The segment group settings are not valid The table's segmentGroupConfig is not valid JSON or has a value out of range. **User correctable** — you can correct this. **What to do** 1. In the Configuration tab, make segmentGroupConfig valid JSON with minSegmentsPerGroup and maxSegmentsPerGroup of at least 1; then Apply and Save. [Related documentation](/corecapabilities/external-table/segment-groups) ### DPE-2271 — That index needs a dictionary on the column An inverted, FST, IFST or star-tree index on column `{column}` needs a dictionary, and the column has none. **User correctable** — you can correct this. **What to do** 1. Add a dictionary to column `{column}` alongside the index, in the Indexes tab or the Configuration tab, keeping its forward index raw if the table requires raw encoding; then save. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-2272 — This table uses a retired sync task The table was created with an older sync task that this cluster no longer runs, so changes to it are rejected. **User correctable** — you can correct this. **What to do** 1. Create the table again from Data & AI, Tables, Connect external table, so it uses the current sync, then delete this one. 2. If you need to keep this table, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-2273 — A table with this name is still being removed A table with the same name was deleted moments ago and the cluster is still cleaning it up, so a new one cannot be created yet. **Transient** — temporary; retrying usually works. **What to do** 1. Wait a few minutes, then create the table again. 2. If it still fails after ten minutes, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-2274 — Data from an earlier table with this name remains Parts of an earlier table with the same name were not fully cleaned up, so a new table with that name cannot be created. **User correctable** — you can correct this. **What to do** 1. If a table with this name is still listed under Data & AI, Tables, delete it again, then create the new one. 2. If none is listed, create the table under a different name, or contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-2275 — Materialized views depend on this table The table cannot be deleted while materialized views read from it: `{views}`. **User correctable** — you can correct this. **What to do** 1. Drop the dependent materialized views, then delete the table again. 2. The table's sync schedule was cleared by this attempt; if you keep the table, click Resume Sync to restore it. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-2276 — The cluster could not read the new table back The table was submitted, but the cluster could not read its configuration back, so it was not created. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Try creating the table again in a few minutes. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-2277 — The cluster's restart settings rejected the change The cluster is set up to restart servers in batches, and that setting rejected this table change. This is on our side, not your configuration. **Upstream** — a service we depend on did not answer as expected. **What to do** 1. Try again in a few minutes; the setting can be unavailable just after a restart. 2. If it keeps happening, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/external-table/troubleshooting) ### DPE-2278 — The configuration was saved; nothing was reloaded The new configuration was saved, but no loaded segments were found to reload, so it applies to segments as they are next loaded. **Informational** — not a failure; act only if you want a different outcome. **What to do** 1. No action is needed for a new table, which has no segments yet. 2. If the table has data and servers were restarting, reload it later from the table's Storage tab, or save the configuration again. [Related documentation](/corecapabilities/external-table/observability) ### DPE-2279 — The task schedule is not a valid cron expression The schedule set for the `{task}` task is not a cron expression the scheduler accepts. **User correctable** — you can correct this. **What to do** 1. In the Configuration tab, set the task's schedule to a Quartz cron expression, which starts with a seconds field and uses ? for the day of the month or week: for example 0 \*/5 \* ? \* \* \* runs every five minutes. Then Apply and Save. [Related documentation](/corecapabilities/external-table/operational-guidance) ### DPE-2284 — A time value is not a number A number in the Data Setup settings could not be read: `{value}`. Usually it is Start Time or End Time, which have to be numbers when Time Column Format counts time since the epoch. **User correctable** — you can correct this. **What to do** 1. When Time Column Format is millisecondsSinceEpoch, secondsSinceEpoch or hoursSinceEpoch, write Start Time and End Time as numbers in that unit, such as 1585699200000 for milliseconds. 2. Otherwise check the other number fields on the Data Setup step for text. 3. Then select Show Sample Data again. ### DPE-2285 — That date pattern must start with the year A Datetime column's text has to sort in the same order as the dates it holds, so its pattern must run year, month, day, then time. `{format}` puts the parts in another order, so it cannot be used, even though it matches the values. **User correctable** — you can correct this. **What to do** 1. Keep the column as text: on the Data Modeling step or the Schema tab, set its Field Type back to Dimension with Data Type STRING. 2. To use it as time, add a LONG column with Add Column, Field Type Datetime and Format EPOCH|MILLISECONDS, whose Transform Function is fromDateTime applied to the text column with its pattern. 3. Or have the source write dates year first, such as yyyy-MM-dd. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2286 — The schema name does not match the dataset name The schema is named `{schemaName}`, but the dataset is `{table}`. The two must match, so the change was not applied. **User correctable** — you can correct this. **What to do** 1. Set schemaName back to `{table}`: on the Validate step's Schema Config tab (Edit configuration, then Apply Changes), or in the Schema tab's Json view on an existing dataset. 2. To rename the dataset, change Table Name on the Details step of Additional Configuration instead; the schema takes that name. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2287 — Groovy transforms are turned off here The Transform Function on `{column}` uses Groovy, which this environment does not allow when data is ingested, so the configuration was rejected. **User correctable** — you can correct this. **What to do** 1. Rewrite the Transform Function on `{column}` with built-in functions instead, such as upper, concat, arithmetic, jsonPathString or fromDateTime. 2. Edit it on the Data Modeling step, or on the Schema tab after Edit, then Apply. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) ### DPE-2288 — A dimension table needs batch ingestion settings A dimension table is replaced whole on each load, so its table config needs batch ingestion settings with segmentIngestionType REFRESH, and this one has none. **User correctable** — you can correct this. **What to do** 1. On the Configuration tab choose Edit, add batchIngestionConfig under ingestionConfig with segmentIngestionType REFRESH and segmentIngestionFrequency DAILY, then Apply and Save. 2. If it should not be a dimension table, set isDimTable back to false instead. [Related documentation](/corecapabilities/ingestdata/dataportal/additional-config-create-table) ### DPE-2289 — The preview failed on the file's rows The preview failed while reading the uploaded file's rows, and the server did not report why. When the schema and table config are accepted and only the preview fails, the usual cause is a column's Data Type, a filter or a transform that does not fit the values in the file. **Internal** — a defect on our side. **What to do** 1. On the Data Modeling step, check each column's Data Type against the file: a text value in an INT, LONG, FLOAT or DOUBLE column fails here. 2. In the Json view's Ingestion Configuration, check filterConfig and transformConfigs: a filter that compares text with a number, or two transforms that read each other, fail only on the rows. 3. Undo your last change and select Fetch Preview to find the one that fails. If the preview fails with the settings the wizard filled in, contact StarTree support and quote the reference shown with this error. [Related documentation](/corecapabilities/ingestdata/dataportal/data-modeling/overview) # Grafana dashboards Source: https://docs.startree.ai/corecapabilities/observability/grafana Reach Grafana from the Data Portal, find the dashboard for what you are investigating, read the Pinot panels, and build your own charts on your cluster's metrics and logs. Grafana is the metrics and log surface for your StarTree Cloud cluster. It is where you see a signal over time — the last hour, the last week, before and after a change — as opposed to the current-state views in the Data Portal. It reads only your own cluster's data. ## Getting there **1. Open it from the Data Portal** In the Data Portal left navigation, go to **Monitoring** → **Grafana Dashboard**. It opens in a new tab. **2. Sign in with the same identity provider** Grafana uses the same single sign-on as the Data Portal, so in most cases you land already authenticated. If you are prompted, sign in with the same credentials you use for StarTree Cloud. > [!NOTE] > > Each cluster environment has its own Grafana. If you have more than one environment, there is no combined view across them, and a dashboard link from one environment will not resolve against another — always follow the link from the Data Portal you are working in. ## Panels embedded in the Data Portal You do not always have to leave the Data Portal. The task views embed Grafana panels directly, scoped to the work you are looking at: | Where | What it shows | | - | - | | A **task plan's Observability** tab | Minion task progress for that plan, alongside the affected table's segment count and total document count | | A **task plan's Logs** tab | Minion task progress panels for the plan's generation | | A **submitted task** or **subtask** detail view | The minion progress panel for that specific task or subtask | Embedded panels carry a **Refresh** and a **Reconnect to Grafana** action, and a time-range selector on the task-plan views. For anything beyond that — a wider window, a different query, a table-level view — go to the dashboards directly. > [!NOTE] > > Embedded panels need a live Grafana session in the same browser. When one is not detected you will see a **Sign in to Grafana** prompt — click it, complete sign-in in the tab that opens, and return; the panels mount shortly afterwards. Panel embedding is enabled per environment, so on some environments the panels link out to Grafana rather than rendering inline. ## Dashboard catalogue Dashboards are organized into folders. Use the search icon or **Dashboards** in the Grafana navigation to list them. ### Pinot The core set. Most investigations start here. | Dashboard | Use it for | | - | - | | **Pinot Servers** | The busiest dashboard: query rate and latency per table and per server, docs and entries scanned, segment counts, real-time consumption, CPU and heap. Start here for both query and ingestion questions. | | **Pinot Server Ingestion** | Real-time ingestion in detail — consumption rate, ingestion delay and offset lag per partition, consumption exceptions, rows with errors. | | **Pinot - Broker** (listed as *Pinot - Broker - Parameterized*) | Broker-side query volume, latency, error breakdown by cause, and routing exceptions. | | **Pinot Controllers** | Controller health, leadership, segment state convergence, ZooKeeper znode sizes. | | **Pinot Minions** | Minion pod resource use and availability. | | **Pinot Minion Tasks** | Task and subtask counts by type and state, error rates, wait and run times. | | **Minion Task Progress** | Progress of a specific task generation — the dashboard behind the task-plan panels in the Data Portal. | | **Pinot Upsert** | Primary-key counts, document-to-key ratio, upsert metadata memory, off-heap upsert internals. | | **Pinot MultiStage Engine** | Multi-stage query volume, latency, serialization and deserialization cost, data exchanged between stages. | | **Pinot System Table** | The query-log pipeline, built over `system_query_log`. | | **Table Rebalance** | Segment movement during a rebalance — use it to confirm a rebalance is progressing rather than stalled. | | **Pinot ZooKeeper** | Quorum, leadership, request latency, znode counts. | | **Pinot - Proxy** | Request volume and latency through the query proxy, including gRPC. | ### Logging Log search for your cluster's components. Covered in detail in [Accessing logs](/corecapabilities/observability/access-logs). | Dashboard | Use it for | | - | - | | **Logging / Common** | Historical log search across any component, filtered by log group, application, pod, and level | | **Logging / Application Logs** | Pinot and StarTree application logs | | **Logging / Query Stats** | Query log lines with latency, rows, and the query text | | **Logging / Slow Queries Drilldown** | Slowest queries over a window, drillable by request ID | | **Logging / Table Queries** | Query activity for one table | | **Logging / Kubernetes Events** | OOM kills, failed liveness checks, evictions | | **Logging / Tool & K8s Cluster Logs** | Supporting component logs | ### Capacity and resources | Dashboard | Use it for | | - | - | | **Total Provisioned Capacity** | What your environment is provisioned for, against what it is using. The one to open before asking for a resize. | | **JVM** | Heap, garbage collection, and thread state per component. | Other folders cover the infrastructure your environment runs on. They are useful for one specific question — confirming that a component was restarted or killed for memory rather than crashing on its own — and [Troubleshooting resource pressure](/corecapabilities/cluster-operations/troubleshooting) covers when that matters. Otherwise, start from the Pinot dashboards, which are the ones expressed in terms of tables, queries, and segments. ## Reading the Pinot dashboards Three things to do before drawing conclusions from a panel: **1. Set the template variables** Most Pinot dashboards have selectors at the top — table, server, partition, task type. A panel with no table selected is showing you either everything summed together or nothing. Table names carry their type suffix, so `orders_REALTIME` and `orders_OFFLINE` are separate entries for a hybrid table. **2. Widen the time range past the incident** The default range is short. Widen to at least several times the duration of what you are investigating — a latency step change is only visible if the window contains both sides of it. When comparing against an upgrade or config change, put the change roughly in the middle. **3. Check whether a flat line is zero or absent** A metric with no recorded value is absent rather than zero, so an empty panel can mean "this never happened" — a task type that has not run, a table never queried. Confirm against a related panel that you know should have data before treating emptiness as a finding. ## Building your own charts You have full access to query your cluster's metrics, and creating dashboards is the normal way to track something specific to your workload. **1. Start from a panel that nearly works** Open the panel menu on any built-in panel and choose **Explore**. You get its query with the correct metric name, labels, and aggregation already in place — far quicker and less error-prone than composing one from scratch. **2. Adjust the query** See [Reading Pinot metrics](/corecapabilities/observability/alerts/overview#reading-pinot-metrics) for the naming convention, the available suffixes, and the label set. The most common mistake is aggregating across the wrong label — check what labels the series actually carries before adding a `sum by`. **3. Save into your own folder** Create a new dashboard rather than editing a built-in one, and save it in a folder of your own. > [!WARNING] > > **Do not edit the built-in dashboards.** StarTree maintains them, so an edit you make in place can be replaced without warning when they are updated. Duplicate the dashboard instead (**Dashboard settings** → **Save as**), put the copy in your own folder, and edit that. ## Retention and limits Metrics and logs are retained for a bounded window — long enough for post-incident analysis, not long enough for capacity planning across quarters. If you need a trend over a longer period than Grafana holds, export the panel data (**Panel** → **Inspect** → **Data**) while it is still in range, or ask your StarTree team what the retention is for your environment. For log retention specifically, see [Accessing logs](/corecapabilities/observability/access-logs#retention). ## If you cannot get in | Symptom | Likely cause | | - | - | | Sign-in loops or is rejected | Your account may not yet have access to this environment's Grafana. Ask your StarTree Cloud admin. | | Grafana loads but dashboards are missing | You are on a different environment's Grafana. Go back to the Data Portal you were working in and follow its **Grafana Dashboard** link. | | Panels in the Data Portal show a sign-in prompt that never clears | The Grafana session was not established. Open Grafana directly in another tab, confirm you are signed in, then reload the Data Portal page. | | A dashboard exists but every panel is empty | Either the time range predates the environment, or nothing has published those metrics yet. Try the **Pinot Servers** dashboard over the last hour as a control. | ## Related * [Alerts and metrics](/corecapabilities/observability/alerts/overview) — metric names, suffixes, and labels, and the alerts built on them * [Accessing logs](/corecapabilities/observability/access-logs) — the Logging dashboards and the log APIs * [Troubleshooting: start here](/corecapabilities/observability/troubleshooting) — symptom-first diagnostic guides * [Table Health Dashboard](/corecapabilities/observability/table-health-and-alerting) — current-state checks in the Data Portal # Health checks Source: https://docs.startree.ai/corecapabilities/observability/health-checks The Cluster Health dashboard and every health check StarTree Cloud runs: what each verifies, the threshold that fails it, the diagnostic fields it returns, and what to do about a failure. StarTree Cloud continuously checks your Apache Pinot cluster and reports each check as pass or fail. Checks run against one of three entity types — a **table**, an **instance** (a single server, broker, or controller), or the **cluster** as a whole. This page is both the dashboard guide and the full reference: how to reach the results, how to run checks on demand, and what every individual check verifies. ## Where results appear Two views surface these checks, and they differ in scope rather than in what they measure: * **Cluster Health** — log in to the Data Portal and click **Cluster Health** in the left navigation. This lists every check across all three entity types with a **PASS**/**FAIL** indicator and the additional details a failure returned: error messages, affected instances, or the relevant metric. You can filter by instance or by table. * **[Table Health Dashboard](/corecapabilities/observability/table-health-and-alerting)** — the per-table view, with email alerting and one-click remediation. It surfaces a subset of the table-level checks; that page covers which ones and how to change it. ## How to read a check Checks are evaluated periodically by the `ClusterHealthCheckTask`, which runs every **20 minutes** by default — the interval is set by Pinot's `controller.cluster.healthCheck.frequencyPeriod`, and setting it to `-1` disables periodic evaluation entirely. Results are cached in memory and overwritten on each run, so both views show the last evaluation — with a **Last Checked** timestamp — not a live probe. Three things follow from that: * **A failure can already be fixed.** If you have just changed something, refresh or trigger a new run before acting. * **Not every failure is urgent.** Several checks are advisory: they flag a configuration that will bite you during an upgrade or a node loss, not something broken now. Each entry below says which kind it is. * **Only a subset appears on the Table Health Dashboard.** The others still run and are still visible on Cluster Health and through the APIs below. Every check returns a `pass` boolean plus whatever diagnostic fields are listed below. In the Data Portal, simple fields are shown inline and nested ones open in a JSON viewer. ## Running checks on demand For immediate results, or to wire checks into a script, call the Controller APIs directly. **Run the checks now**, bypassing the 20-minute interval. Returns confirmation that the task was queued or executed: ```bash theme={null} curl -X GET "https:///periodictask/run?taskName=ClusterHealthCheckTask" ``` **Fetch the latest report** — the cached pass/fail status and details of every check, the same data the dashboard renders: ```bash theme={null} curl -X GET "https:///clusterHealth" | json_pp ``` **List the check types** the cluster monitors, with brief descriptions — useful for confirming what your version actually runs: ```bash theme={null} curl -X GET "https:///clusterHealth/list" | json_pp ``` ## Table-level checks Eighteen checks run per table, grouped below by what they are about. These are the checks the Table Health Dashboard draws from. **Remediable.** These two can be fixed from the Table Health Dashboard without leaving the page; see [remediating failing checks](/corecapabilities/observability/table-health-and-alerting#remediating-failing-checks) for the buttons and the permissions they need. ### `TABLE_SEGMENT_ASSIGNMENT_CHECK` **Verifies:** whether the table's segments are optimally assigned across servers, by running a rebalance dry run and inspecting the result. **Fails when:** the dry run reports that segments would move — the current assignment is no longer what a rebalance would produce. | Field | Meaning | | - | - | | `balanced` | Whether the assignment is already balanced | | `numSegments` | Total segments considered | | `numSegmentsToMove` | How many segments a rebalance would relocate | | `numSegmentsToDelete` | How many replicas a rebalance would remove | | `message` | The dry run's own summary | **What to do:** rebalance the table — the Table Health Dashboard can [do it in one click](/corecapabilities/observability/table-health-and-alerting#remediating-failing-checks). `numSegmentsToMove` is the size of the job: a large number means a long-running rebalance with real network and disk cost, so schedule it rather than firing it during a peak. This check commonly fails after servers are added or removed, after a tenant change, or after tier configuration changes. **Severity:** advisory unless query latency is skewed. Unbalanced assignment costs you performance and headroom, not correctness. ### `TABLE_SEGMENTS_RELOAD_CHECK` **Verifies:** whether any segment needs reloading, by asking every server hosting the table whether its segments are consistent with the current table and schema config. **Fails when:** at least one server reports a segment needing reload. | Field | Meaning | | - | - | | `tableSegmentsConsistent` | `false` when at least one segment needs a reload | **What to do:** reload the table's segments — again [one click](/corecapabilities/observability/table-health-and-alerting#remediating-failing-checks) from the Table Health Dashboard. This check fails whenever you change a schema or index configuration — existing segments keep the old layout until reloaded, which is exactly why a newly added index appears to have no effect on query performance. Reloading is the step that applies it. **Severity:** act on it. Until you reload, the table's data does not match its configuration. **Segment health and availability.** ### `IDEAL_STATE_EV_MISMATCH_CHECK` **Verifies:** that every segment's actual state (ExternalView) matches its intended state (IdealState). **Fails when:** at least one segment's states disagree. | Field | Meaning | | - | - | | `numberOfSegmentsMismatch` | How many segments disagree | | `mismatchSegmentsName` | The specific segment names | **What to do:** a mismatch means Helix wants a segment somewhere it has not arrived. Transient mismatches during a rebalance, a restart, or a fresh segment push are normal and clear on their own — re-run the check before treating it as a problem. Persistent mismatches usually mean a server cannot load the segment: check the server's logs for that segment name (see [Accessing logs](/corecapabilities/observability/access-logs)) and whether the server is under [resource pressure](/corecapabilities/cluster-operations/troubleshooting). Segments stuck in error state will also make queries return incomplete results. **Severity:** act on it if it persists. This is the check that correlates with unavailable-segment query errors. ### `SEGMENT_COUNT_CHECK` **Verifies:** the table's total segment count. **Fails when:** the table has more than 50,000 segments. | Field | Meaning | | - | - | | `numberOfSegments` | Current count | **What to do:** high segment counts inflate the metadata every query must consider, grow ZooKeeper state, and slow every cluster operation. Usually the cause is segments that are too small — check [`SEGMENT_SIZE_CHECK`](#segment_size_check) alongside this one, review your [segment threshold configuration](/recipes/configuring-segment-threshold), and consider merging small segments ([real-time](/recipes/merge-segments-realtime), [offline](/recipes/merge-small-segments)) for older data. See [storage efficiency](/corecapabilities/manage-data/storage-efficiency) for the full set of levers. **Severity:** advisory, but the trend matters more than the value. A table crossing this threshold while still growing needs attention before it doubles. ### `SEGMENT_SIZE_CHECK` **Verifies:** the distribution of segment sizes. **Fails when:** more than a quarter of the table's segments are smaller than 5 MB. | Field | Meaning | | - | - | | `smallSegmentsCount` | Segments below the size floor | | `totalSegmentsCount` | Total segments | | `smallSegmentsName` | Which segments are small | **What to do:** many small segments means more metadata, more per-segment query overhead, and worse compression than the same data in fewer segments. For real-time tables, raise the flush threshold so segments are sealed at a larger size; for offline tables, adjust the push granularity or roll up older data. See [Configuring the segment threshold](/recipes/configuring-segment-threshold). **Severity:** advisory. It is a performance and cost signal, not a failure. ### `SEGMENT_RETENTION_CHECK` **Verifies:** that the table has segment retention configured. **Fails when:** no retention is configured. Returns only `pass`. **What to do:** without retention, the table grows without bound and will eventually hit its storage quota. Set a retention period appropriate to the data. If unbounded retention is genuinely intended, this check will keep failing; treat it as an acknowledged exception rather than something to fix. **Severity:** advisory, but it is the leading indicator of a storage-quota incident months later. ### `SEGMENTS_TIME_PARTITION_CHECK` **Verifies:** that segments are cleanly partitioned by time — sorted by start and end time, no segment ends before its predecessor. **Fails when:** segment time ranges overlap. | Field | Meaning | | - | - | | `overlappedSegments` | The overlapping segments | **What to do:** overlapping time ranges defeat time-based segment pruning, so time-filtered queries scan more segments than they need to. It commonly follows a backfill that wrote data into a range already covered by existing segments. Review how the overlapping segments were produced; see [Backfill](/recipes/backfill). **Severity:** advisory. It shows up as queries slower than the time filter suggests they should be. **Configuration and schema.** ### `REPLICATION_CHECK` **Verifies:** the table's configured replication factor against the cluster's threshold. **Fails when:** replication is below the threshold — 3 unless your cluster configures otherwise. | Field | Meaning | | - | - | | `tableReplication` | The table's configured replication | | `tableReplicationThreshold` | The threshold being compared against | **What to do:** replication below 2 means a single server loss or a rolling upgrade can make part of the table unavailable or return inconsistent results. Raising replication increases storage and memory proportionally, so it is a deliberate trade-off: development tables at 1 are reasonable, production tables generally are not. **Severity:** advisory now, decisive during an incident. ### `TABLE_COLUMN_COUNT_CHECK` **Verifies:** the number of columns in the schema. **Fails when:** the schema has more than 500 columns. | Field | Meaning | | - | - | | `numColumns` | Current column count | **What to do:** every column carries per-segment metadata and index structures, so wide schemas cost memory and slow segment load even for columns nobody queries. Look for columns that are never referenced, and for semi-structured data that would be better held in a JSON or map column than flattened into hundreds of physical ones. See [JSON index](/corecapabilities/manage-data/indexes/json-index). **Severity:** advisory. ### `TIME_COLUMN_GRANULARITY_CHECK` **Verifies:** the granularity of the table's time columns. **Fails when:** a time column is defined at `MILLISECONDS`, `MICROSECONDS`, or `NANOSECONDS` granularity. | Field | Meaning | | - | - | | `timeColumnsWithHighGranularity` | The columns concerned | **What to do:** granularity finer than your queries need makes every distinct timestamp a distinct value, which hurts segment pruning and dictionary efficiency. If queries filter and group by minute or hour, a millisecond primary time column earns nothing. Consider a coarser time column for partitioning and retention, with the precise timestamp kept as an ordinary column, and add a [timestamp index](/corecapabilities/manage-data/indexes/timestamp-index) for the granularities you actually query. **Severity:** advisory. ### `BROKER_RESOURCE_CHECK` **Verifies:** that the table's broker resource includes brokers from its tenant. **Fails when:** none of the tenant's brokers are present in the table's broker resource. | Field | Meaning | | - | - | | `missingBrokerList` | Brokers expected but absent | **What to do:** this is a routing problem — queries against the table have no broker able to serve them, which surfaces as `BrokerNoServerFoundAlert` or table-not-found style errors. It usually follows a tenant rename or a broker tenant change. Confirm the table's tenant configuration matches an existing broker tenant, and open a ticket if it does and the check still fails. **Severity:** act on it. The table may be unqueryable. ### `INSTANCE_POOLS_N_REPLICA_GROUPS_CHECK` **Verifies:** that the table uses instance pools and replica groups. **Fails when:** it does not. Returns only `pass`. **What to do:** pools and replica groups isolate failures and make rolling operations safe, and they are a prerequisite for [scheduled server scaling](/corecapabilities/cluster-operations/scheduled-server-scaling). Adopting them on an existing table changes segment assignment, so plan for a rebalance. **Severity:** advisory. ### `KAFKA_CHECK` **Verifies:** for real-time tables ingesting from Kafka, that the topic's `retention.ms` is more than twice the table's real-time segment flush threshold — enough headroom to re-consume after downtime without the data having aged out. **Fails when:** topic retention is at or below twice the flush threshold. Unlimited retention (`-1`) always passes. | Field | Meaning | | - | - | | `topic` | The topic checked | | `retentionMillis` | The topic's configured retention | | `flushThresholdMillis` | The table's flush threshold | | `details` | Per-topic breakdown | | `errorMessage` | Why the check could not be evaluated, when applicable | **What to do:** either raise topic retention or lower the flush threshold. Without headroom, a consumer outage longer than retention means the missing records are unrecoverable — the stream has already discarded them. This is one of the few checks whose fix is on the Kafka side rather than in Pinot. **Severity:** advisory until you have an outage, at which point it determines whether you lose data. **Skew and distribution.** Skew checks compare one server against the average for the table. They matter because a single overloaded server sets the latency of every query that touches it, however healthy the rest of the cluster is. ### `SEGMENT_SKEW_HEALTH_CHECK` **Verifies:** segment distribution across the servers hosting the table. Only evaluated for tables with more than 50 segments. **Fails when:** any server holds more than 50% above the average segment count for that table. | Field | Meaning | | - | - | | `numberOfSegments` | Segments in the table | | `numberOfSegmentsWithReplicas` | Segments counted with replicas | | `numberOfServers` | Servers hosting the table | | `expectedNumberOfSegmentsPerServer` | The even-distribution figure | | `serversWithHighNumberOfSegments` | The servers over the line | **What to do:** a rebalance is the usual answer — check [`TABLE_SEGMENT_ASSIGNMENT_CHECK`](#table_segment_assignment_check), which will typically be failing too. If it passes while this fails, the assignment strategy itself is producing the skew rather than drift from it, so review the table's instance assignment configuration. **Severity:** advisory; a strong hint when investigating latency that varies by query. ### `CONSUMING_PARTITION_SKEW_HEALTH_CHECK` **Verifies:** distribution of *consuming* segments — the real-time ingestion load — across the table's servers. Only evaluated for tables with more than 10 consuming segments. **Fails when:** any server holds more than 50% above the average consuming-segment count. | Field | Meaning | | - | - | | `numberOfConsumingSegments` | Consuming segments in the table | | `numberOfConsumingSegmentsWithReplicas` | Counted with replicas | | `numberOfServers` | Servers hosting the table | | `expectedNumberOfConsumingSegmentsPerServer` | The even-distribution figure | | `serversWithHighNumberOfConsumingSegments` | The servers over the line | **What to do:** consuming segments cost far more than completed ones — memory for the in-flight segment, CPU for decode and index building, and stream connections. A server carrying a disproportionate share becomes the table's ingestion bottleneck and will be the first to lag. Rebalance including consuming segments, and check whether stream partition count and server count are compatible. **Severity:** act on it if the table is also lagging. This is a common root cause behind ingestion delay on one replica. **Upsert and dedup tables.** ### `UPSERT_TABLE_SEGMENT_ASSIGNMENT_CHECK` **Verifies:** for an upsert table, that all segments of a partition within a replica group live on one server — the invariant upsert correctness depends on. **Fails when:** a partition's segments for a single replica group are spread across more than one server. | Field | Meaning | | - | - | | `partitionId` | The offending partition | | `hostedSegmentCount` | Segments of that partition on the server | | `totalSegmentCount` | Segments in the partition | | `instanceSegmentCountMetadata` | Per-instance breakdown | **What to do:** this is a correctness concern, not a performance one — upsert resolution needs every record for a primary key co-located. Do not rebalance an upsert table ad hoc to fix it; open a support ticket with the partition ID and the instance breakdown. **Severity:** act on it. Query results may be wrong. ### `HIGH_PK_PER_PARTITION_CHECK` **Verifies:** primary-key count per partition on an instance. **Fails when:** a partition holds more than 500 million primary keys. | Field | Meaning | | - | - | | `permissiblePKCountPerPartition` | The threshold | | `partitionWithHighPKCount` | The partition over it | **What to do:** upsert metadata is held per primary key and grows with distinct keys rather than with rows, so a table that was comfortable at launch can exhaust memory much later with no configuration change. Options, in rough order of preference: increase the stream's partition count so keys spread further, move the table to [off-heap upsert](/corecapabilities/manage-data/offheap-upsert), or shorten the metadata TTL if your key space has a natural lifetime. **Severity:** act on it. This is the check that precedes an out-of-memory server. ### `PK_SKEW_PER_PARTITION_CHECK` **Verifies:** how evenly primary keys are distributed across a table's partitions. **Fails when:** the skew falls outside the range −2 to 2. | Field | Meaning | | - | - | | `skew` | Measured skew | | `lowerBoundForSkew`, `upperBoundForSkew` | The acceptable range | **What to do:** skewed keys mean one partition — and so one server — carries a disproportionate share of upsert metadata, and it will hit memory limits while the others look fine. The cause is upstream: the partition key does not distribute evenly. Check for a dominant value (a default, a null placeholder, one high-volume tenant) in the key. **Severity:** advisory, and the explanation for why one server out of many keeps running out of memory. ## Instance-level checks These evaluate a single server, broker, or controller rather than a table, so they do not appear on the Table Health Dashboard. They are reported on Cluster Health and through the APIs above. | Check | Description | | :- | :- | | `INSTANCE_HEALTH_API_CHECK` | Checks if any of the instance `/health` API endpoints are not live, indicating that a Pinot instance (Server or Broker) is unresponsive. | | `HIGH_NUMBER_OF_SEGMENTS_CHECK` | Checks whether the number of segments hosted on a server exceeds 10,000, which can lead to performance degradation. | | `HIGH_NUMBER_OF_CONSUMING_PARTITIONS_CHECK` | Checks whether the number of consuming segments hosted on a server exceeds 50, indicating a potential bottleneck for real-time ingestion on that server. | | `HIGH_NUMBER_OF_TABLES_CHECK` | Checks whether the number of tables hosted by a server is greater than 10, indicating a potential concentration of tables on a single server which might affect resource utilization. | | `HIGH_PRIMARY_KEYS_CHECK` | Checks if the primary key count for all tables hosted by the instance exceeds 800 Million, indicating a very high primary key load on that instance. | | `HELIX_HOST_NAME_INSTANCE_NAME_MISMATCH_CHECK` | Checks if the Instance ID/Name does not match the expected value derived from the Instance Config. This can indicate a misconfiguration. | ## Cluster-level checks These evaluate the cluster as a whole. | Check | Description | | :- | :- | | `QUERY_HEALTH_CHECK` | Performs a simple `SELECT $docId FROM TABLE_NAME LIMIT 1` query for each table to check if it executes unsuccessfully, indicating fundamental queryability issues. | | `CLUSTER_LEVEL_SEGMENT_SKEW_CHECK` | Checks whether the number of segments on any server exceeds 50% of the average segment count across all servers within the entire cluster, indicating overall segment distribution imbalance. | | `CLUSTER_LEVEL_CONSUMING_PARTITION_SKEW_CHECK` | Checks whether the number of consuming segments on any server exceeds 50% of the average consuming segment count across all servers within the entire cluster, highlighting overall real-time ingestion load imbalance. | | `TABLE_SKEW_CHECK` | Checks whether the number of tables hosted by a server exceeds 50% of the average number of tables hosted across all servers, indicating an imbalance in table distribution. | | `DATA_SIZE_SKEW_CHECK` | Checks whether the skew in the size of data hosted by instances is asymmetric (specifically if the skew is not in the range `[-2, 2]`), indicating an uneven distribution of data storage across your cluster. | | `CONTROLLER_TABLES_SKEW_CHECK` | Checks whether the skew of table count hosted by lead controllers is asymmetric (specifically if the skew is not in the range `[-2, 2]`), indicating an uneven distribution of table management load among controllers. | | `PRIMARY_KEY_SKEW_CHECK` | Checks whether the skew in the distribution of primary keys among servers is asymmetric (specifically if skew is not in the range `[-2, 2]`), indicating an uneven load of primary keys across servers. | | `PREBUILT_SNAPSHOT_CHECK` | Checks for any upsert or dedup table in the cluster that is missing prebuilt snapshot configurations, and identifies partitions without snapshots. Prebuilt snapshots can significantly improve preload performance for these table types. | | `INSTANCE_POOLS_CHECK` | Checks if any server is not part of instance pools, or if any pool is not disjoint when the cluster has more than 8 servers. Proper instance pooling is crucial for fault isolation and resource management. | ## Related * [Table Health Dashboard & Alerting](/corecapabilities/observability/table-health-and-alerting) — the per-table view of these checks, with email alerting and one-click remediation * [Troubleshooting: start here](/corecapabilities/observability/troubleshooting) — when a check fails and you need a diagnostic path * [Scheduled Server Scaling](/corecapabilities/cluster-operations/scheduled-server-scaling) — `INSTANCE_POOLS_CHECK` and `INSTANCE_POOLS_N_REPLICA_GROUPS_CHECK` cover the same pool and replica-group setup that scheduled scale-down requires * [Storage efficiency](/corecapabilities/manage-data/storage-efficiency) — for the segment-count and segment-size checks # Task Observability Source: https://docs.startree.ai/corecapabilities/observability/minion-task-runs Watch background minion tasks — ingestion, purge, refresh, snapshotting, and more — run to completion for a table or across your whole workspace, directly from the Data Portal. ## Overview Pinot runs a lot of work in the background on your tables — ingesting new files, purging rows, refreshing segments, snapshotting upserts, syncing external catalogs, and more — using its minion **task** framework. Historically, checking on one of these jobs meant calling the controller's task REST API directly. **Task Observability** brings that same information into the Data Portal as a readable, real-time view: a list of every **task plan** (a tracked run of one task type against one table, potentially spanning several batches) with drill-down into each batch and its subtasks, without leaving the UI. > [!NOTE] > > A task plan is the same concept described in [Minion Task Orchestration](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration) — this page covers how that data shows up in the Data Portal. Not every task type you'll see here supports full orchestration (automatic batch chaining); task types that don't are still tracked for progress and history, they just don't automatically chain their next batch for you. ## Requirements Task Observability is built on the same task-plan machinery as [Minion Task Orchestration](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration), so the two share a cluster-level dependency: | Setting | What it actually controls | | - | - | | `controller.startree.task.manager.enableTaskOrchestration` (cluster config) | The master switch for all task-plan tracking, orchestrated or not. On by default since StarTree 0.16.0 — if a cluster predates that or has it explicitly turned off, no task plans exist anywhere and Task Observability has nothing to show. | | `enableTaskOrchestration` (per task type, in the table/task config) | Only controls whether a plan **automatically chains its next batch**. It is *not* required for a scheduled task to show up here — a cron-triggered run of a StarTree task is still tracked as a plan (status, progress, batch history) without this flag, it just won't self-submit its next batch. Ad hoc (manual) runs are the exception: they still need this flag set to produce a trackable plan at all. | > [!NOTE] > > In short: cluster-level orchestration has to be on for anything to appear in Task Observability, but you don't need to opt every scheduled task into full orchestration just to watch it — cron-driven runs are tracked automatically. Automatic batch chaining stays opt-in per task type, and today only [supported task types](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration#supported-task-types) benefit from it. ## Where to find it Task Observability has two entry points: * **Tasks** — a top-level nav item in the **Monitoring** group, showing task plans across every table you have access to. * **Monitoring tab** — on a table's details page, showing the task history for that one table. If you don't see **Tasks** in the navigation, the feature isn't turned on for your environment yet — contact your StarTree account team. > [!NOTE] > > The global **Tasks** list is a live/recent view, not a full historical log: it shows every plan that's currently active, plus completed plans this browser has previously seen (remembered per workspace in browser local storage, and shown for as long as the controller still has the plan — Pinot keeps roughly the newest 10 completed plans per table and task type). If you need a dependable history for a specific table, use that table's **Monitoring** tab instead — it's backed by a per-table query rather than what your browser happened to see. ## Reading the task list Both the global list and the per-table tab show one row per task plan, refreshing automatically (roughly every 20 seconds) with a manual refresh option: | Column | What it shows | | - | - | | **Task plan** | The plan's ID; opens the plan's detail page. | | **Table** | The table the plan runs against (hidden on the per-table tab, since it's implied). | | **Task Type** | Which minion task this plan is running, e.g. File Ingestion, Segment Purge, Segment Refresh. | | **Status** | Active, Completed, Cancelled, Aborting, Failed, or Unknown. | | **Progress** | A completion bar with a tooltip showing percent complete and units processed. | | **Created At / Completed At** | When the plan started and (if finished) ended. | | **Trigger** | **Manual** (an ad hoc run) or **Automatic** (a scheduled/cron run). | | **Note** | A human-readable status message, when one is available. | Additional columns (input unit, input units to process, in progress, and processed) are hidden by default but available from the column picker. Every column except the progress bar can be filtered independently. The global list also has a **Create task plan** action — a short wizard for starting a new orchestrated run (currently File Ingestion and Segment Purge) by picking the task type, the table, and the task configuration. ## Task plan detail Opening a plan shows its full detail page, which refreshes on its own more aggressively than the lists (roughly every 10 seconds while the plan is running): * **Refresh** and **Copy link** (a shareable URL to this exact plan). * **Schedule Now** — appears only for a non-orchestrated plan that still has work left and nothing currently in flight. It nudges the task type to pick up the remaining work immediately. * **Abort** — enabled only while the plan is **Active**. You're asked to type "abort" to confirm. Aborting stops the plan from generating any further batches; a batch that's already running is left to finish rather than being killed outright. An overview strip at the top shows the plan's progress donut alongside its key execution charts. Below it, the page is organized into tabs: * **Logs** (the default tab) — the plan's execution charts, with a time-range selector. For a finished plan, the window freezes to the run's start and end. * **Tasks** — every batch submitted so far for this plan, each with its own status, progress, sequence, and submission time (newest first). * **Properties** — the plan's configuration, as a flat, copyable key/value list. * **Custom** — task-type-specific statistics reported by the running task, also copyable. A collapsible sidebar shows the plan's metadata (ID, table, task type, status, subtask counts, timestamps) with a one-click **Copy** of the full plan as JSON. ## Drilling into an individual run From a plan's **Tasks** tab, opening one submitted batch shows Pinot's actual per-task subtask table: each subtask's ID, state (waiting, running, completed, error, dropped, timed out, aborted), start/finish time, and duration. Opening a subtask goes one level deeper, into its progress log: * A switch between **Live Progress** (what the minion is reporting right now) and **Log History** (the recorded log), with an explanation when live progress is no longer available. * A search box, multi-select filters for log level and label, and a show-timestamps toggle. * A running count of how many lines are shown out of the total, newest first. * A metadata panel with the subtask's configuration in plain terms — task type, table, sequence ID, size settings, and the input files it's processing. ## Retention of task execution details Once a task finishes, Pinot keeps its detailed execution metadata — subtask states, progress, and configuration — for a limited window before cleaning it up (on the order of a day by default; your cluster's operator can tune this). Once that window passes, drilling into an old batch shows a clear **Task Execution Data No Longer Available** message instead of an empty or broken page. The execution *charts* have their own, longer retention: after about two weeks a plan's charts show a **Task Metrics No Longer Available** notice. These are separate from the global Tasks list's own "recently seen" behavior described [above](#where-to-find-it) — one is how long Pinot keeps a task's execution details, the other is how long your browser remembers a completed plan is worth showing in the all-tables view. If you need to look back at a plan that finished a while ago, the per-table **Monitoring** tab is the more reliable place to start, but the underlying execution detail can still have aged out on the Pinot side either way. ## Permissions Viewing and acting on tasks from Data Portal uses the same fine-grained authorization as the rest of Data Portal. Viewing task plans requires the `GetTaskPlan` action on the plan's table — the global list silently omits plans for tables you can't see — and aborting a plan requires `AbortTaskPlan`. Triggering a run (Schedule Now, Create task plan) goes through Pinot's task APIs and their standard task actions, and the per-subtask debug and log views use Pinot's task status/debug actions. See [RBAC Actions](/corecapabilities/security/actions) for the full list. If a control is missing or disabled for you, ask your StarTree Cloud admin to confirm your permissions. ## Related * [Minion Task Orchestration](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration) — the task plan concept and REST API this feature is built on. * [File Ingestion Task](/corecapabilities/ingestdata/adv-concepts/batch/file-ingestion-task), [Purge Task](/corecapabilities/manage-data/purge-task), [Segment Backfill Task](/corecapabilities/manage-data/segment-backfill-task), [Segment Import Task](/corecapabilities/manage-data/segment-import-task), [Alter Table Task](/corecapabilities/manage-data/alter-table-task) — task types you can monitor here. * [External Tables: Observability](/corecapabilities/external-table/observability) — status, checkpoint, and schedule-now APIs for External Table sync runs specifically, also available from a table's details page in Data Portal. # Table Health Dashboard & Alerting Source: https://docs.startree.ai/corecapabilities/observability/table-health-and-alerting Monitor per-table health checks in real time from the Data Portal, drill into failures, remediate them, and configure email alerts for check failures and recoveries. The **Table Health Dashboard** gives you a real-time, table-centric view of your tables' operational status directly in the StarTree Cloud Data Portal. Instead of digging through the cluster-wide Pinot admin console, you get a single grid of every table you manage, a pass/fail rollup per table, and one click to drill into exactly which checks are failing and why. Layered on top of the dashboard is **Table Health Alerting**: a per-recipient, per-check email subscription system. Instead of having to keep the dashboard open, you can subscribe yourself (or a distribution list) to email alerts whenever a check starts failing, keeps failing, or recovers — scoped as broadly as "every table, every check" or as narrowly as a single check on a single table. > [!NOTE] > > Both the dashboard and alerting live under the **Monitoring** section in the Data Portal left navigation, and require the Pinot Admin app permission (`PinotAdminSystemAdmin`). The underlying APIs additionally check the Pinot RBAC [`GetHealth`](/corecapabilities/security/actions) action. If your account doesn't have this section enabled yet, ask your StarTree Cloud admin — the dashboard and the alerting poller are each enabled per environment. ## Key features * **Per-table rollup:** every table you have access to, with a Passing / N Failing summary and a `Last Checked` timestamp, sortable and filterable by health status. * **Drill-down detail:** click into a table to see every check that runs against it, with a pass/fail chip, a human-readable description, and a read-only viewer for the raw diagnostic payload. * **One-click remediation:** for checks that support it (and when you have permission), fix the underlying issue — trigger a segment rebalance or a segment reload — without leaving the page. * **Email alerting:** subscribe any email address to failure/recovery notifications for all tables and checks, or fine-tune down to individual (table, check) pairs. * **Mute, don't just disable:** snooze a specific failing check's alerts until it next passes, instead of permanently turning notifications off. * **Bulk controls:** enable, disable, mute, or unmute alerts across an entire check type or an entire recipient's subscriptions in one action. ## Accessing the Table Health Dashboard **1. Open Data Portal** Log in to the StarTree Cloud Data Portal for your cluster. **2. Go to Monitoring** In the left navigation, click **Monitoring**, then **Health Dashboard**. **3. Review the table grid** You'll land on the Table Health Dashboard, showing every table in scope with its current health rollup. ## Reading the dashboard The dashboard header shows two summary chips — **Tables** (total count) and **Failing Tables** (highlighted in red when non-zero) — plus a **Last run** indicator with a manual **Refresh** button so you don't have to wait for the next automatic evaluation cycle. The grid itself has four columns: | Column | What it shows | | - | - | | **Table Name** | The table's display name (suffix-stripped), with an **OFFLINE** / **REALTIME** badge. Click the name to open the table's detail page. | | **Workspace** | The workspace that owns the table (useful once you have more than the default workspace). | | **Health** | Either **Passing**, or **N failing / M checks** with a row of colored dots — one per check, green for pass and red for fail (hover a dot to see its check name). Tables with no checks configured show **No checks configured**. | | **Last Checked** | Relative time since the last health-check run (for example, "12 min ago"). | Use the **Health** column's filter icon to narrow the grid to **Passing**, **Failing**, or **No checks** tables. If you haven't created any tables yet, the dashboard shows an empty state with a **Create table** shortcut instead of a grid. > [!NOTE] > > If your cluster has disabled periodic health checks entirely (Pinot's `controller.cluster.healthCheck.frequencyPeriod` set to `-1`), the dashboard shows a warning banner and the grid will not update. See [Health checks](/corecapabilities/observability/health-checks#how-to-read-a-check) for how that setting is controlled. ## Table details page Clicking a table name opens its detail page: an accordion list with one entry per health check that applies to that table, sorted failing-first (then alphabetically by check key). At the top of the page is a **Recipient** picker listing every subscribed recipient across all workspaces. The **Email** and **Mute** controls on each check act on whichever recipient is selected here, so the page shows an informational message and leaves those controls unavailable until at least one recipient has been subscribed. Opening the page from a digest email preselects the recipient the email was addressed to. Each accordion row shows: * The check's short label and its raw check key (for example, **Segment Assignment** / `TABLE_SEGMENT_ASSIGNMENT_CHECK`). * A one-line description of what the check verifies. * A **passed** / **failed** status chip. * An **Email: On/Off** control — toggles this check's email alerts *for the currently selected recipient* on this specific table (see [Alerting](#alerting-email-notifications) below). * For failing checks with email alerts on, a **Mute/Unmute** control to snooze that specific alert. Expanding a row (failing checks auto-expand) reveals additional diagnostic detail returned by the check — for example, how many segments need to move, or which segments are affected. Simple values are shown inline; any nested field gets an eye icon that opens the raw JSON in a read-only viewer. Checks that return no extra detail show **No additional details**. Where a check supports it, one or more remediation buttons appear alongside this detail. ## Remediating failing checks Two of the table-level checks currently ship with one-click remediation, gated by RBAC permissions: | Check | What it means | Remediation actions | | - | - | - | | `TABLE_SEGMENT_ASSIGNMENT_CHECK` | The table needs rebalancing — segment assignment across servers is no longer optimal. | **Rebalance segments** (triggers a rebalance) and **Rebalance status** (shows progress of the last rebalance). | | `TABLE_SEGMENTS_RELOAD_CHECK` | One or more segments need to be reloaded (for example, after a config or schema change). | **Reload segments** (triggers a reload) and **Reload status** (shows progress of the last reload). | > [!WARNING] > > Remediation is gated purely by RBAC: the buttons appear only on a **failing** check when your account holds cluster-level write access ([`UpdateClusterConfig`](/corecapabilities/security/actions)). This is evaluated once at the cluster level, not per workspace — so remediation is either available to you across the whole cluster or nowhere. If you don't see a **Rebalance segments** or **Reload segments** button on a failing check that supports it, ask your StarTree Cloud admin to confirm your permissions. Only a subset of the table-level checks Pinot runs is surfaced on the Table Health Dashboard, and that subset is set per environment. **By default it is exactly the two remediable checks above** — the other table-level checks (segment size, segment retention, segment availability, ingestion, column count, and more) are filtered out of the dashboard entirely rather than shown as pass/fail. Reach out to your StarTree Cloud team if you'd like additional checks surfaced; see [Health checks](/corecapabilities/observability/health-checks) for the full check catalog. For any check you see here — surfaced by default or enabled for your environment — the [health check reference](/corecapabilities/observability/health-checks) explains what it verifies, what the diagnostic fields in the expanded row mean, and what a failure asks of you. ## Alerting: email notifications Alerting is managed from the **Notification Settings** page — click the gear icon next to **Last run** on the dashboard, or next to the table title on any table's detail page. ### How subscriptions work A recipient's alert coverage is resolved from preference rows, each pairing them with a `(workspace, table, check)` tuple where the table and check may be a `*` wildcard: * A **wildcard subscription** — "every table, every check" — created in each workspace the first time you subscribe a recipient. * **Concrete overrides** — a specific (table, check) pair for that recipient. For a given failing check, the **most specific matching row wins** and only that row's settings apply — specificity runs exact `(table, check)` → `(table, *)` → `(*, check)` → `(*, *)`. In practice the Data Portal writes only the two ends of that range: a wildcard row when you subscribe someone, and a concrete `(table, check)` row whenever you change an individual cell (including when a bulk action fans out across many cells). The partial-wildcard forms in between are accepted by the API for scripted setups. This lets you subscribe someone to everything and then carve out exceptions (for example, turn off a single noisy check on one table) without touching the rest of their subscriptions. ### Subscribing a new recipient **1. Open Notification Settings** From the dashboard or a table detail page, click the gear icon. **2. Click Subscribe new user** Enter their email address. This creates one wildcard subscription per workspace that currently reports tables, so they'll start receiving alerts for every table and check cluster-wide. Any workspace you don't have write access to is skipped. **3. Fine-tune from the grid** Use the recipient picker to select them, then adjust individual checks or tables as needed (see below). ### Managing a recipient's preferences Once a recipient is selected in the picker, the page shows one collapsible section per health-check type (for example, **Table Segment Assignment Check**). Each section lists every (workspace, table) pair for that check, with its current **Passing** / **Failing** status and that recipient's effective **Email** and **Muted** state. From here you can: * Toggle **Email** for a single table/check cell, or flip the section header's switch to enable/disable the check for every table at once. * **Mute** or **Unmute** a single failing cell, or use the section's **Mute All / Unmute All** to snooze every currently-failing, email-enabled row in that section. * Use the page-level bulk actions — **Enable Email for All**, **Disable Email for All**, **Mute All Failing**, **Unmute All** — to act across the recipient's entire subscription in one click. * Click the trash icon next to the recipient picker to **remove** them entirely. After you confirm, this deletes their wildcard subscriptions and every override, unsubscribing them from all table-health alerts. This can't be undone (they'd need to be re-subscribed from scratch). > [!TIP] > > Muting is a temporary snooze, not a permanent opt-out: a muted check automatically un-mutes itself the next time it passes — the recovery email is delivered and the snooze ends — so you won't need to remember to turn it back on. This applies to mutes set on a specific (table, check) cell, which is what the Data Portal creates. A mute applied to a whole wildcard subscription is not self-clearing and suppresses the recovery notification too, so unmute it explicitly with **Unmute All**. ### What triggers an email Alerting runs its own evaluation cycle, polling the cluster's health state every 20 minutes by default. (That is independent of how often Pinot itself re-runs the checks, which the dashboard reflects directly.) For each subscribed (recipient, table, check), an alert is generated when: * A check goes from **passing to failing** — an immediate "new failure" notification. * A previously-failing check **recovers** — an immediate "recovery" notification. * A check is **still failing** — it is always listed alongside any new failure or recovery in the same email. On its own, with nothing else to report, it becomes a reminder sent at most once every 24 hours by default, so you aren't paged repeatedly for a known, ongoing issue. Any new-failure or recovery email resets that 24-hour clock, so the standalone reminder is skipped on days you've already heard from us. * Nothing is sent on the **first cycle after alerting starts** — that run just records the current state, so subscribing doesn't immediately blast you with the state of the world. Note this baseline is also reset if every recipient is unsubscribed; after that gap, the next cycle with a subscriber starts clean again. Once a baseline exists, a check that appears for the first time already failing does alert straight away. Each recipient gets at most one digest email per evaluation cycle, covering **all** of their workspaces in a single message — every table is labelled with the workspace that owns it. Within the email, **new failures**, **still failing**, and **recovered** checks are grouped into separate sections, each listing up to 5 affected tables (ordered by failing-check count) with a "+N more" note for the rest. The email links back to the Table Health Dashboard and to the specific failing table's detail page, so you can jump straight from your inbox to the fix. > [!NOTE] > > Alerting is currently **email only**. There is no Slack, PagerDuty, or webhook channel for Table Health alerts today. ## Permissions required The entire Monitoring section — the dashboard, table detail pages, and Notification Settings — requires Pinot Admin access in the Data Portal (the `PinotAdminSystemAdmin` app permission), and the health APIs behind it check the Pinot RBAC [`GetHealth`](/corecapabilities/security/actions) action. Triggering remediation additionally requires cluster-level [`UpdateClusterConfig`](/corecapabilities/security/actions) write access, as described above. ## Related * [Health check reference](/corecapabilities/observability/health-checks) — per-check detail: what each check verifies, its threshold, the diagnostic fields it returns, and what to do about a failure. * [Health checks](/corecapabilities/observability/health-checks) — the Cluster Health view and the full catalog: table-, instance-, and cluster-level checks, with per-check detail. Table Health surfaces a table-scoped subset of the same checks with a friendlier, per-table workflow plus alerting. * [RBAC Actions](/corecapabilities/security/actions) — reference for `GetHealth`, `UpdateClusterConfig`, and the other Pinot permission actions referenced above. * [Troubleshooting](/corecapabilities/observability/troubleshooting) — what to do once a check is failing: diagnostic paths for query failures, ingestion problems, and resource pressure. # Troubleshooting: start here Source: https://docs.startree.ai/corecapabilities/observability/troubleshooting Start here when something is wrong. Identify the symptom, follow the matching guide, and know what to collect before you open a support ticket. Most production problems in StarTree Cloud fall into a small number of shapes. There is a diagnostic path for each, using only what you can see and do yourself from the Data Portal, the Query Console, and the APIs your cluster exposes. Each one lives with the part of the product it concerns — queries under Query Data, ingestion under Ingestion, capacity under Cluster Operations. This page routes you to the right one. > [!NOTE] > > If you already know the shape of the problem, go straight to the relevant guide. If you do not, work through **Triage** below — it takes about two minutes and will tell you which guide to open. ## Choose a guide > **[Queries failing or slow](/corecapabilities/query_data/troubleshooting)** > > Timeouts, execution errors, multi-stage engine failures, sudden latency increases, error-rate spikes. > **[Data not arriving](/corecapabilities/ingestdata/troubleshooting)** > > Real-time ingestion stopped or lagging, batch and minion tasks failing, tasks that never got scheduled. > **[Resource pressure](/corecapabilities/cluster-operations/troubleshooting)** > > Out-of-memory conditions, high CPU, servers filling up, degradation after an upgrade or a traffic peak. > **[An error code on screen](/corecapabilities/observability/error-codes)** > > You have a `DPE-####` code from the Data Portal or an API response, and want to know what it means and what to do about it. > **[A failing health check](/corecapabilities/observability/health-checks)** > > What a specific check verifies, what its diagnostic fields mean, and what to do about the failure. > **[An alert from StarTree](/corecapabilities/observability/alerts/overview)** > > StarTree support mentioned an alert by name, and you want to know when it fires, what it means, and what you can do. ## Triage Answer these in order. The first one that matches tells you where to go. **1. Is the table healthy?** Open **Observability → Health Dashboard** in the Data Portal and find the affected table. Failing checks name the subsystem in trouble, which is usually enough to pick a guide. A failing `TABLE_INGESTION_CHECK` points at [Data not arriving](/corecapabilities/ingestdata/troubleshooting). A failing `TABLE_SEGMENT_AVAILABILITY_CHECK` or `BROKER_RESOURCE_CHECK` points at [Queries failing](/corecapabilities/query_data/troubleshooting). See [Table Health Dashboard & Alerting](/corecapabilities/observability/table-health-and-alerting) for how to read the dashboard, and [Health checks](/corecapabilities/observability/health-checks) for cluster-wide checks. > [!NOTE] > > Not every failing check is a fault. Some checks are best-practice recommendations that do not apply to every deployment — a check that flags a configuration you deliberately chose is telling you about a convention, not an outage. Read the check description before acting on it. **2. Is it one table, or everything?** Run a trivial query (`SELECT 1`) and then a `SELECT COUNT(*)` against an unrelated table. If both succeed, the problem is scoped to one table or one query pattern — start with the table's health checks. If the trivial query also fails or hangs, the problem is cluster-wide, and [Resource pressure](/corecapabilities/cluster-operations/troubleshooting) is usually the better starting point. **3. What changed?** The overwhelming majority of sudden production changes follow a change somewhere. Before investigating deeply, check whether any of these happened recently: * A cluster or engine version upgrade * A table config, schema, or index change * A change in your upstream source — new schema version, different partition count, a producer deploy * A change in query volume or shape from your application * A rebalance, reload, or scaling operation A symptom that started at the same time as one of these is very likely caused by it, and saying so in a support ticket saves a round trip. **4. Is it constant or intermittent?** A constant failure is usually configuration, schema, or capacity. An intermittent failure is usually contention, a specific data shape, or one unhealthy replica. For intermittent problems, note whether failures correlate with a time of day, a particular query, or a particular tenant — that correlation is often the whole diagnosis. ## Tools you will use | Tool | What it answers | Where | | - | - | - | | **Health Dashboard** | Is this table structurally healthy right now? | Data Portal → Observability | | **Query Console** | Does this query still fail, and what does it return? | Data Portal → Query Console | | **Query Analyzer** | Why is this query slow, and how should it be rewritten or indexed? | Data Portal → Query Console → Analyze. See [Query Analyzer](/corecapabilities/query_data/query-analyzer) | | **`system_query_log`** | What did the cluster actually serve, how long did it take, and what errored? | Query Console, as an ordinary table. See [Query Logger](/corecapabilities/query_data/advanced_operations/query-logger) | | **Task views** | Did my ingestion or maintenance tasks run, and what did they do? | Data Portal → Tasks. See [Task Observability](/corecapabilities/observability/minion-task-runs) | | **Grafana** | How does this compare with an hour, a day, or a week ago? | Data Portal → Monitoring → Grafana Dashboard. See [Grafana dashboards](/corecapabilities/observability/grafana) and [Alerts and metrics](/corecapabilities/observability/alerts/overview) | | **Logging dashboards** | What did the component actually log, and on which pod? | Grafana → Logging. See [Accessing logs](/corecapabilities/observability/access-logs) | | **Health check reference** | What does this failing check mean, and what do its diagnostic fields say? | See [Health check reference](/corecapabilities/observability/health-checks) | > [!WARNING] > > Query Logger is on by default in current releases, and by default it records slow (1 second or longer) and failed queries. If `system_query_log` does not exist on your cluster, contact StarTree Support to have it enabled — it is the single most useful diagnostic surface for query problems, and it only captures traffic from the point it is switched on. Enabling it before you need it is worth doing. ## Before you open a support ticket A ticket that arrives with this information attached is typically resolved in one exchange rather than three. Collect what applies: > **Always** > > * Cluster and environment, and the table name including its `_OFFLINE` / `_REALTIME` suffix > * When the problem started, in a stated timezone, and whether it is ongoing > * Whether it is constant or intermittent, and what fraction of requests are affected > * What changed recently — see step 3 of Triage > * The business impact, and whether a workaround is in place > * If the failure showed an error code, the `DPE-####` code and the **reference** beside it — see [Data Portal error codes](/corecapabilities/observability/error-codes). The reference identifies your exact occurrence in our logs > **For query problems** > > * The full query text, unedited > * The complete error message, including any nested cause > * The `requestId` from the response, or from `system_query_log` > * Whether the query uses the single-stage or multi-stage engine > * Whether the same query previously worked, and roughly when it last did > * If you have Query Logger enabled, the row from `system_query_log` for a failing execution > **For ingestion problems** > > * How far behind the data is, and whether the gap is growing, flat, or shrinking > * Whether every partition is affected or only some > * Any recent change to the upstream topic, stream, or source bucket — including partition count and schema version > * For batch and minion tasks: the task type, the task or plan ID, and the failure message > * Whether the table paused consumption, and whether it was paused deliberately > **For resource problems** > > * Which component is under pressure — server, broker, controller, or minion > * Whether the pressure is memory, CPU, or disk > * When it began, and whether it coincided with a traffic change or an upgrade > * Your current tenant sizing and whether it has changed recently > [!TIP] > > Where a support engineer would ask "can you reproduce it," answering that in the first message is the biggest single accelerator. A query that fails every time is far faster to diagnose than one that failed once yesterday. ## What these guides do not cover These pages cover problems you can diagnose from your own cluster. Some classes of problem are visible only to StarTree Operations — control-plane faults, telemetry pipeline outages, and infrastructure events in the managed layer. If you have worked through the relevant guide and the symptom does not match anything in it, that is a meaningful diagnostic result in itself, and worth stating in your ticket. # Troubleshooting by feature Source: https://docs.startree.ai/corecapabilities/observability/troubleshooting-by-feature A directory of the feature-specific troubleshooting, FAQ and limitations sections across the documentation, grouped by the problem they solve. The [symptom guides](/corecapabilities/observability/troubleshooting) narrow a problem down to a subsystem. This page takes you the rest of the way: it indexes the troubleshooting, FAQ and limitations material that lives on individual feature pages, so you do not have to know which page it is on. > [!NOTE] > > Feature-specific troubleshooting is deliberately kept next to the feature it belongs to, alongside the configuration and concepts it refers to. This page is a directory, not a copy. ## Streaming ingestion | If you are dealing with | Go to | | - | - | | A StarTree alert about ingestion, such as `RealtimeIngestionStopped` or `HighRealtimeIngestionOffsetLag` | [Real-time ingestion alerts](/corecapabilities/observability/alerts/real-time-ingestion) | | Consumption stalls, segment commit and ordering behaviour, and the debug APIs for inspecting consuming segments | [Pauseless consumption → Operational Troubleshooting](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#operational-troubleshooting), and [Debugging APIs](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#debugging-apis) | | Which log lines indicate which consumption problem | [Key logging patterns for debugging](/corecapabilities/ingestdata/adv-concepts/realtime/pauseless-consumption#key-logging-patterns-for-debugging) | | Records failing to decode, or a schema-registry problem | The decoder's own page: [Glue](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/glue#troubleshooting) · [OpenTelemetry](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/opentelemetry#troubleshooting) · [Prometheus remote write](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/prometheus-remote-write#troubleshooting) · [Pulsar key-value Avro](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/pulsar-key-value-avro#troubleshooting) | | Change-data-capture pipelines behaving unexpectedly | [MongoDB CDC](/corecapabilities/ingestdata/adv-concepts/realtime/mongodb-cdc#troubleshooting) · [Debezium CDC recipe](/recipes/debezium-cdc#troubleshooting) | | Kinesis-specific questions | [Kinesis FAQs](/corecapabilities/ingestdata/dataportal/streaming/kinesis#faqs) | | Pulsar connection problems | [Pulsar recipe](/recipes/pulsar#troubleshooting) | ## Batch ingestion and minion tasks | If you are dealing with | Go to | | - | - | | A StarTree alert about a task, such as `FailedToGenerateFileIngestionTask` or `MinionSubTaskHighWaitTime` | [Batch ingestion and minion task alerts](/corecapabilities/observability/alerts/batch-and-minion-tasks) | | Tasks not being scheduled, queue behaviour, concurrency and retries | [Minion task orchestration → FAQs](/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration#faqs) | | Purge tasks failing or not removing what you expect | [Purge task](/corecapabilities/manage-data/purge-task), *FAQs & Recommendations* section | | Backfill not producing the segments you expect | [Segment backfill task](/corecapabilities/manage-data/segment-backfill-task), *FAQs & Recommendations* section | | The real-time to offline flow not running | [Managed offline flow recipe](/recipes/managed-offline-flow#troubleshooting) | | JFR record ingestion | [JFR record reader](/corecapabilities/ingestdata/adv-concepts/batch/jfr-record-reader#troubleshooting) | | A file preview failing during table creation | [Preview from file → Troubleshooting](/api-reference/table/preview-from-file#troubleshooting) | ## Upserts and dedup | If you are dealing with | Go to | | - | - | | A StarTree alert about an upsert table, such as `HighTotalDocsToPKCountRatio` | [Upsert table alerts](/corecapabilities/observability/alerts/upsert-tables) | | Duplicate rows, or counts running ahead of the source | [Diagnosing duplicates and count mismatches](/corecapabilities/manage-data/offheap-upsert#diagnosing-duplicates-and-count-mismatches) — includes `skipUpsert`, the `$segmentName` / `$docId` / `$hostName` virtual columns, and detecting cross-replica divergence | | Compaction or segment refresh tasks failing on an upsert table | [Upsert operations guide → Troubleshooting](/corecapabilities/manage-data/upsert-operations-guide#troubleshooting) | | Offline upsert constraints | [Offline upserts → Limitations](/corecapabilities/ingestdata/adv-concepts/batch/offline-upserts#limitations) | ## Queries, indexes and performance | If you are dealing with | Go to | | - | - | | A StarTree alert about query errors, timeouts, or quota | [Query alerts](/corecapabilities/observability/alerts/queries) | | Interpreting what Query Analyzer told you, including its internal diagnostic values | [Reading Query Analyzer results → Diagnostic Values](/corecapabilities/ai/query-analyzer/reading-results#diagnostic-values) | | A slow geospatial query | [Geospatial performance → Troubleshooting performance issues](/corecapabilities/manage-data/indexes/geospatial-performance#troubleshooting-performance-issues) | | Checking what a reload would do before running it, or a dry run that failed | [Reload dry run → Troubleshooting](/corecapabilities/manage-data/reload-dry-run#troubleshooting) | | A star-tree index that is not being used | [StarTree index recipe → Troubleshooting](/recipes/startree-index#troubleshooting) | | An index-inference apply that failed | [Index inference apply → Troubleshooting](/api-reference/table/index-inference-apply#troubleshooting) | | Queries being throttled or starved by other workloads | [Memory-throttled scheduler → Troubleshooting](/corecapabilities/query_data/advanced_operations/workload-isolation/memory-throttled-scheduler#troubleshooting) | | Materialized view behaviour | [Multi-stage materialized views → FAQs](/corecapabilities/query_data/materialized-views/multi-stage-materialized-views#faqs) | | `JSON_EXTRACT_INDEX` returning unexpected results | [json\_extract\_index → Error handling](/corecapabilities/query_data/functions/json_extract_index#error-handling) | | Gaps in what `system_query_log` captures | [Query Logger](/corecapabilities/query_data/advanced_operations/query-logger), *Limitations & caveats* section | ## Segments and storage | If you are dealing with | Go to | | - | - | | A StarTree alert about replicas, segments in error, or a storage limit | [Data availability and storage alerts](/corecapabilities/observability/alerts/data-availability-and-storage) | | Hitting a segment-count limit, or wanting to see where a table stands | [Segment count limits → Monitoring and troubleshooting](/corecapabilities/manage-data/recipes/segment-count-limits#monitoring-and-troubleshooting) | | Columnar segment processing failures | [Columnar segment processing → Troubleshooting](/corecapabilities/manage-data/columnar-segment-processing#troubleshooting) | | Deep-store sync behaviour and constraints | [Deep store sync → Limitations](/corecapabilities/manage-data/deep-store-sync#limitations) | | A self-managed MinIO deep store | [MinIO deep store recipe](/recipes/minio-deep-store#troubleshooting) | | Understanding what storage a table is consuming | [Table storage usage](/corecapabilities/manage-data/table-storage-usage) | ## External tables External tables have the most developed troubleshooting material in the docs — [External table troubleshooting](/corecapabilities/external-table/troubleshooting) is organised by symptom and indexed by the exact error string to search for, covering onboarding, schema and table creation, queries, and sync operations. Start there for anything external-table shaped. | If you are dealing with | Go to | | - | - | | A specific external-table error string — region errors, `Access Denied`, `NumberFormatException`, `servers not responded`, distributed-lock failures | [External table troubleshooting](/corecapabilities/external-table/troubleshooting) | | Operational problems, including ZooKeeper packet-size overflow as segment count grows | [Operational guidance → Additional troubleshooting](/corecapabilities/external-table/operational-guidance#additional-troubleshooting) | | General questions and behaviour | [External tables FAQ](/corecapabilities/external-table/faq) | | Monitoring sync state, lag and freshness | [External table observability](/corecapabilities/external-table/observability) | | A type not mapping as you expect | [Data type mapping → Known limitations](/corecapabilities/external-table/data-type-mapping#known-limitations) | | Deletion vector or segment group behaviour | [Deletion vectors → FAQs](/corecapabilities/external-table/deletion-vectors#faqs) · [Large External Tables → FAQs](/corecapabilities/external-table/segment-groups#faqs) | | Catalog-specific constraints | The `Limitations` section on your catalog's onboarding page — [Glue](/corecapabilities/external-table/glue/onboarding-api#limitations) · [Unity](/corecapabilities/external-table/unity/onboarding-api#limitations) · [Polaris](/corecapabilities/external-table/polaris/onboarding-api#limitations) · [Nessie](/corecapabilities/external-table/nessie/onboarding-api#limitations) · [S3](/corecapabilities/external-table/s3/onboarding-api#limitations) · [S3 Tables](/corecapabilities/external-table/s3tables/onboarding-api#limitations) · [GCS](/corecapabilities/external-table/gcs/onboarding-api#limitations) | ## Security and access | If you are dealing with | Go to | | - | - | | Audit logs not appearing or not containing what you expect | [Pinot audit logs → Troubleshooting](/corecapabilities/security/audit/pinot-audit-logs#troubleshooting) | | Row-level policies not filtering as intended | [Row-level access control → Limitations](/corecapabilities/security/row-level-access-control#limitations) | | Working out which action governs an operation | [RBAC actions](/corecapabilities/security/actions) | ## Interfaces and integrations | If you are dealing with | Go to | | - | - | | Elasticsearch-compatible queries | [Elasticsearch gateway → Troubleshooting](/corecapabilities/query_data/query_interfaces/elasticsearch-gateway#troubleshooting) and its [Limitations](/corecapabilities/query_data/query_interfaces/elasticsearch-gateway#limitations) | | Semantic search behaviour | [Semantic search → Troubleshooting](/corecapabilities/ai/semantic-search#troubleshooting) | | The MCP server | [StarTree MCP → Limitations](/corecapabilities/query_data/mcp/startree-mcp#limitations) | | Generative AI recipes | [GenAI recipe → Troubleshooting](/recipes/genai#troubleshooting) | ## Scaling and cluster operations | If you are dealing with | Go to | | - | - | | A scheduled scaling operation that did not take effect | [Scheduled server scaling → Troubleshooting](/corecapabilities/cluster-operations/scheduled-server-scaling#troubleshooting) and its [Notes and limitations](/corecapabilities/cluster-operations/scheduled-server-scaling#notes-and-limitations) | | Cluster-, broker-, controller- or minion-level health checks | [Health checks: instance-level](/corecapabilities/observability/health-checks#instance-level-checks) and [cluster-level](/corecapabilities/observability/health-checks#cluster-level-checks) | | A specific table health check failing, and what its diagnostic fields mean | [Health check reference](/corecapabilities/observability/health-checks) | | Finding the log line behind a symptom | [Accessing logs](/corecapabilities/observability/access-logs) | | A table whose storage footprint is growing | [Storage efficiency](/corecapabilities/manage-data/storage-efficiency) and [Table Storage Usage](/corecapabilities/manage-data/table-storage-usage) | ## ThirdEye ThirdEye has its own troubleshooting section under its own tab: * [ThirdEye FAQ](/thirdeye/troubleshooting/faq) * [ThirdEye observability and monitoring](/thirdeye/troubleshooting/thirdeye-observability-and-monitoring) * [Getting help with ThirdEye](/thirdeye/troubleshooting/get-help) > [!TIP] > > Not finding what you need here? Work through the [symptom guides](/corecapabilities/observability/troubleshooting) instead — they start from what you are seeing rather than from which feature is involved, which is the better route when you do not yet know what is at fault. # Index-Based DISTINCT in Apache Pinot Source: https://docs.startree.ai/corecapabilities/query_data/advanced_operations/fast-distinct-with-inverted-index From StarTree release 0.15 onwards, we have added support for two index-based execution paths for `SELECT DISTINCT` that skip the normal scan + projection pipeline and read distinct values directly from existing indexes. Both are opt-in behind a single query option. In many cases, this will lead to a significant increase in performance for distinct operators. ## Enabling ```sql theme={null} SET useIndexBasedDistinctOperator = true; ``` This is off by default. Set it per query (or session) to turn on the optimization. When enabled, Pinot automatically picks the cheapest path for each query — index-based or the regular scan — based on filter selectivity and column cardinality. Ineligible queries silently fall back to the regular DISTINCT path, so it is safe to set globally. The following two features share this flag: 1. **Inverted-index DISTINCT** — for single-column `SELECT DISTINCT` on a column with a dictionary plus an inverted (or sorted) index. 2. **JSON-index DISTINCT** — for `SELECT DISTINCT jsonExtractIndex(...)` on a column with a JSON index. *** ## 1. Inverted-Index DISTINCT ### What it does Computes `SELECT DISTINCT col` by walking the column's dictionary and checking which values still have matching docs after the filter, instead of scanning every row. ### When it helps * Single-column `SELECT DISTINCT` on a column with **dictionary + inverted index**, or a **sorted column**. * Selective filters (the filter eliminates most rows). * Low-to-moderate cardinality columns (cities, country codes, status codes, tenant IDs, etc.). * `ORDER BY LIMIT N` on the DISTINCT column — the operator can short-circuit on sorted columns. ### When it does not help (and Pinot falls back automatically) * Multiple columns in the `SELECT DISTINCT` list. * Very high cardinality columns with a weak or no filter — scanning is cheaper. * Column without a dictionary or without an inverted/sorted index. * Multi-value columns. ### Sample queries ```sql theme={null} SET useIndexBasedDistinctOperator = true; SELECT DISTINCT city FROM myTable; SELECT DISTINCT city FROM myTable WHERE country = 'US'; SELECT DISTINCT city FROM myTable WHERE country = 'US' ORDER BY city LIMIT 100; ``` ### Tuning (optional) If the default cost-based pick is too conservative or too aggressive for your workload, you can nudge it: ```sql theme={null} SET invertedIndexDistinctCostRatio = 10; ``` * Higher value → bias toward the index path. * Lower value → bias toward scanning. * `0` → always use the inverted-index bitmap path whenever the filter has any match. Defaults already scale with cardinality (`30` for ≤1K, `10` for ≤10K, `6` above), so most workloads do not need to set this. *** ## 2. JSON-Index DISTINCT ### What it does Computes `SELECT DISTINCT jsonExtractIndex(col, '$.path', ...)` by reading the JSON index's value-to-docId map directly, instead of parsing the JSON for every matching row. ### When it helps * High-cardinality JSON fields where parsing dominates cost (e.g., distinct user IDs, distinct event types out of a JSON blob). * Faceting / dropdown style queries that just need the set of distinct values for a path. * Queries where the JSON path is indexed. ### When it does not help (and Pinot falls back automatically) * The JSON path is not covered by the index. * Multi-value arrays inside the JSON where each matching row contributes many values. * Multiple expressions in the `SELECT DISTINCT` list. * Multi-value return types like `STRING_ARRAY`. ### Prerequisite — JSON index on the column ```json theme={null} { "fieldConfigList": [ { "name": "myJsonCol", "encodingType": "RAW", "indexTypes": ["JSON"] } ] } ``` ### Sample queries ```sql theme={null} SET useIndexBasedDistinctOperator = true; SELECT DISTINCT jsonExtractIndex(myJsonCol, '$.user.country', 'STRING') FROM myTable; SELECT DISTINCT jsonExtractIndex(myJsonCol, '$.event.type', 'STRING') FROM myTable WHERE timestamp > 1700000000000 ORDER BY jsonExtractIndex(myJsonCol, '$.event.type', 'STRING') LIMIT 1000; ``` ### Supported patterns | Pattern | Supported | | - | - | | `DISTINCT jsonExtractIndex(col, '$.path', 'STRING')` and all single-value primitive types | Yes | | `defaultValue` argument for missing paths | Yes | | Inline JSON filter expression (5th arg of `jsonExtractIndex`) | Yes | | `WHERE` clause filters | Yes | | `ORDER BY` + `LIMIT` | Yes | | Multi-value return types (`STRING_ARRAY`, etc.) | No — falls back | | Multiple columns in `SELECT DISTINCT` | No — falls back | *** ## Verifying the optimization kicked in * `EXPLAIN PLAN FOR ` shows `InvertedIndexDistinct` or `JsonIndexDistinct` in place of the default distinct node. * Query response metadata: `numEntriesScannedPostFilter` drops to `0` for the JSON path, and near-zero for the inverted-index bitmap path. * If the plan still shows the regular DISTINCT, either the query is ineligible (see "When it does not help" above) or the cost-based picker chose scan because that was cheaper. *** ## Background — how the path is picked Pinot does cost-based selection for the inverted-index path at query time. For each segment it estimates the cost of scanning vs. probing the index using the rule: ```text theme={null} dictionaryCardinality × costRatio ≤ filteredDocCount → use index path ``` Otherwise it falls back to scanning. This means the same query can use different paths on different segments depending on data distribution, and you do not have to know in advance which one is cheaper. The JSON-index path is unconditional once eligible — reading from the index is always cheaper than parsing JSON per row. # Decoupled Intermediate Stages for MSE Source: https://docs.startree.ai/corecapabilities/query_data/advanced_operations/intermediate-stage-on-broker Offload multi-stage intermediate work — joins, aggregations, sorts, window functions — from servers to a dedicated broker pool, when the bottleneck is genuinely above the leaf stage. # Overview By default, Pinot's multi-stage query engine (MSE) executes all intermediate stages (joins, window functions, aggregations, sorts, unions, subqueries etc.) on server instances. Decoupled mode (also called broker-as-worker) is a StarTree Cloud feature that runs these intermediate stages on brokers instead, which moves compute-heavy operations off the machines that store and scan your data. # Why Use This ### Reduce server load   Intermediate stages like joins, sorts, and aggregations can be CPU- and memory-intensive. These operations compete for resources with leaf-stage work (segment scanning, filtering, indexing) on servers. Offloading intermediate stages to brokers frees server resources to focus on what they do best - serving data from disk. ### Scale intermediate stages independently of data   Servers are stateful - they hold segment data on disk and must be carefully scaled, rebalanced, and provisioned around storage requirements. Adding server capacity means rebalancing segments across the new nodes, which is a heavyweight operation. Brokers, on the other hand, are stateless. They hold no segment data and can be added or removed freely with no rebalancing or data movement. This makes brokers a natural fit for horizontally scaling intermediate-stage compute: if your complex queries (joins, large aggregations, multi-table unions) are bottlenecked on intermediate-stage resources, you can spin up more brokers without touching your server fleet. ### Right-size instance types for the workload   In cloud deployments, servers are typically provisioned with large disks and balanced CPU/memory for segment storage and scanning. Intermediate stages have a very different resource profile - they are often memory- and CPU-bound (hash joins, sort-based aggregations, large intermediate result sets) with no disk requirements. With this feature, you can run your intermediate-stage broker pool on memory-optimized instances (e.g., AWS r-family, GCP m-family) that are purpose-built for this workload, rather than paying for disk-heavy server instances that carry unused storage capacity. This can lead to both better query performance and lower infrastructure cost. ### Workload isolation Using broker tags, you can dedicate specific broker pools to MSE intermediate work. This prevents heavy analytical queries from starving latency-sensitive queries of resources, and lets you scale the broker pool that handles complex workloads on its own. # When to Use It Decoupled mode pays off when the expensive part of a query is genuinely the intermediate stages — joins, `GROUP BY`/aggregation, sorts, window functions — rather than the leaf-stage scan and filter. Typical fits: * **Heavy analytical and download queries over long time ranges.** Reports that scan multi-day, weekly, or monthly ranges and then aggregate or sort over a large number of groups. Run these on the multi-stage engine and send the intermediate work to a dedicated broker tenant, so servers only scan, filter, and pre-aggregate. * **Isolating heavy queries from latency-sensitive traffic.** Tag a separate broker pool for intermediate stages so the intermediate stages of an expensive analytical query cannot starve the brokers serving application queries, and so that pool can be scaled on its own. * **Join-heavy workloads sharing a cluster with an application workload** — funnel queries are the common case. These spend their time in the intermediate stages, so moving that work off the servers isolates them at the point where the cost actually is, with per-query granularity. > [!NOTE] > > Tagging a worker pool isolates **intermediate-stage** work only. The broker that receives the query still plans it, dispatches the stages, and runs the root-stage final merge before returning the result — so a large download, or a query with expensive final result handling, still consumes resources on whichever broker it was routed to. To keep that off your application brokers as well, route these queries to a separate broker pool in the first place, in addition to tagging a worker pool. # When It Does Not Help This feature moves **intermediate-stage** work and nothing else. If the cost sits somewhere else, offloading it changes nothing. > [!WARNING] > > Confirm the intermediate stages are the bottleneck before enabling this for a workload — see [Confirming the bottleneck is in intermediate stages](#confirming-the-bottleneck-is-in-intermediate-stages). * **Leaf-stage-bound queries.** When most of the time goes into scanning, filtering, and aggregating inside the leaf stage, the intermediate stages are already cheap and there is nothing meaningful to offload. A common example is a high-cardinality `DISTINCTCOUNTBITMAP` with few output groups: the query returns a handful of rows, but each group carries a large bitmap built at the leaf stage. Reduce leaf-stage cost instead: where an approximate count is acceptable, switch to a bounded sketch such as `DISTINCTCOUNT_HLL` or `DISTINCTCOUNT_THETA_SKETCH`, then pre-aggregate it with a [star-tree index](/corecapabilities/manage-data/indexes/startree-index). Adding a star-tree index while the query still calls `DISTINCTCOUNTBITMAP` does not help — the index is only used when every aggregation in the query is one it is configured for, and the [supported functions](/corecapabilities/manage-data/indexes/startree-index#supported-aggregation-functions) are the bounded ones. * **Single-stage engine queries.** A single-stage query has no intermediate stages to move; the query must run on the multi-stage engine (`SET useMultistageEngine=true;`) for this to apply at all. Rewriting a single-stage-shaped query as multi-stage purely to enable offloading only pays off if the rewritten plan genuinely puts substantial work into an intermediate stage. * **Short, selective queries.** Point lookups and small filtered scans do little or no work above the leaf stage, so there may be nothing to offload. If the plan does contain a small intermediate stage, moving it to a broker can add scheduling and network overhead without enough savings to compensate. * **Server pressure that is not intermediate-stage CPU or memory.** Saturated disk I/O, segment loading, or GC driven by leaf-stage work is not relieved by this feature. # Confirming the Bottleneck Is in Intermediate Stages **1. Look at the stage breakdown** Per-stage and per-operator statistics tell you where the time actually goes. If a leaf stage dominates the total, offloading intermediate work will not move the needle. * **With [Query Logger](/corecapabilities/query_data/advanced_operations/query-logger)** (0.15 and later), `system_query_log` records `stageStats` as JSON for each captured query, so you can look at history rather than re-running anything. See [Query troubleshooting](/corecapabilities/query_data/troubleshooting#multi-stage-engine-issues). * **Without it**, turn the streaming stats channel on **before** running the query, then read `stageStats` from the response: ```sql theme={null} SET streamStats = true; SELECT ... ``` It has to be set on the query you want to measure — it cannot be recovered afterwards. On the [StarTree Execution Engine](/corecapabilities/query_data/query_languages/startree-engine) the streaming channel is required for per-stage and per-operator stats: without it the response carries broker-side stats only. The [Query Analyzer](/corecapabilities/query_data/query-analyzer) presents the same breakdown for a single query. **2. Read the plan** `EXPLAIN PLAN FOR ` shows how the query splits into stages, where the exchanges sit, and which operators — joins, aggregations, sorts, window functions — are placed above the leaf. It answers **whether** there is intermediate work worth moving, not what that work costs: the plan is static and carries no timings, row counts, or bytes shuffled. Use it to rule the feature out quickly when nothing substantial sits above the leaf, and use the stage statistics from the previous step to establish the bottleneck. See [MSQE](/corecapabilities/query_data/query_languages/msqe#checking-execution-plans). **3. A/B the query (0.16 and later)** Once brokers run workers (`startree.broker.mse.worker.enabled=true`, the default on recent builds), run a representative query both ways — `SET useBrokerForIntermediateStage = false;` and `= true;` — and compare latency and server load. The per-query option makes this cheap to test without changing cluster behavior. If your workers are in their own broker tenant, also set `mseWorkerBrokerTag`. Before you compare, run `EXPLAIN IMPLEMENTATION PLAN FOR ` with the option set to `true`. The intermediate stages must show broker hosts. If no live broker carries the worker tag, the query runs on the servers without an error, and both runs measure the same plan. Before 0.16, `startree.broker.mse.use.broker.for.intermediate.stage=true` turns decoupled mode on for **every** query and requires a broker restart, so there is no per-query opt-in. After you turn it on, you can still compare per query: run the baseline with `SET useBrokerForIntermediateStage = false;`. # Trade-offs to Weigh * **Brokers sit on the query path.** A heavy intermediate stage on the brokers that route your application queries competes with them for CPU and heap. That heap also holds the routing tables, which grow with segment count. Sharing the pool is fine when those brokers are mostly idle (low QPS, small tables). Otherwise, dedicate a broker pool by tag — see [Workload isolation with broker tags](#workload-isolation-with-broker-tags). * **Broker capacity becomes a sizing exercise.** Brokers that act as workers need CPU and memory budgeted for the intermediate stages they will run, not just for routing. Size the worker pool for the heaviest concurrent analytical load you expect. * **Shuffle traffic moves to the brokers.** Data that would have shuffled between servers now goes from the servers to the worker brokers, and from there to the broker that received the query. The number of hops does not change, but no part of the shuffle stays local to a server, so the worker pool needs enough network bandwidth. * **Some changes need a restart.** Brokers read `startree.broker.mse.worker.enabled` and the worker tag at startup. The per-query option needs no restart. On recent builds, `startree.broker.mse.use.broker.for.intermediate.stage` set as a cluster config applies live, and it also takes a percentage (for example, `10%`) for a gradual rollout. * **One query can occupy the whole pool.** Cap it with `numBrokersForIntermediateStage` — see [Limiting broker parallelism](#limiting-broker-parallelism). Brokers also do not kill runaway queries by default. For a worker pool, ask StarTree Support to turn on broker query killing (`pinot.broker.query.accounting.oom.enable.killing.query`). # Decoupled Mode vs. Replica-Group Workload Isolation Both isolate workloads, but at different points in the query. [Replica-group based workload isolation](/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation) partitions the **server** fleet; decoupled mode moves intermediate-stage compute off the servers entirely. | | Decoupled mode | Replica-group isolation | | :- | :- | :- | | What it isolates | Intermediate-stage compute (joins, aggregations, sorts, windows) | Server-side execution, by replica group | | Where the work runs | A tagged broker pool | The servers, restricted to the chosen replica groups | | Applies to | Multi-stage engine queries only | Any query the broker routes | | Per-query control | `useBrokerForIntermediateStage`, `mseWorkerBrokerTag`, `numBrokersForIntermediateStage` | `preferredReplicas`, `fallbackReplicas` | | Setup cost | Broker tags, and a broker restart for broker-config changes; brokers are stateless, so the pool scales without rebalancing | Table must use replica groups and the StarTree replica-group instance selector; changing it requires a rebalance | | Best fit | Joins and large aggregations competing with leaf-stage work for server resources | Separating whole workloads (for example adhoc from application traffic) that are server-bound | The two are complementary: you can isolate server-side execution by replica group and still offload intermediate stages to a dedicated broker pool. # Enabling the Feature **For StarTree Release before 0.16** Add the following to your broker / cluster configuration: ```text theme={null} startree.broker.mse.use.broker.for.intermediate.stage=true ``` This will turn on decoupled mode for all queries. No server-side changes are required. Each broker with this config enabled will start an embedded query server that can accept dispatched intermediate stage plans from other brokers. **For StarTree Release starting 0.16 (and beyond)** Add the following to your broker / cluster configuration: ```text theme={null} startree.broker.mse.worker.enabled=true ``` This will simply enable the feature cluster wide. At this point, users can choose to turn on decoupled mode on a per query basis by adding this option: ```text theme={null} SET useBrokerForIntermediateStage = true; SELECT ... ``` Or by adding the following to your broker / cluster configuration: ```text theme={null} startree.broker.mse.use.broker.for.intermediate.stage=true ``` **Note**: this property is mandatory and must be set to true for this feature to work. This property cannot be dynamically configured in a running cluster and needs a restart for the feature to work. # Gradual Rollout by Percentage > [!NOTE] > > Available in StarTree release 0.16 and later. You do not have to turn decoupled mode on for all multi-stage queries at once. `startree.broker.mse.use.broker.for.intermediate.stage` also takes a percentage. Brokers then send that share of multi-stage queries to the worker brokers. The other queries run their intermediate stages on the servers, as before. When you set the value as a cluster config, brokers apply each change live, **with no restart**. As a result, you can start small, increase the percentage in steps, and roll back at once. A small first step also shows early if the worker pool is too small: query timeouts and errors appear on a small share of queries first. | Value | Result | | :- | :- | | `false` (default) | Intermediate stages run on the servers. A query can still opt in with `SET useBrokerForIntermediateStage = true;`. | | `0` to `100`, for example `10` or `10%` | That percentage of multi-stage queries runs its intermediate stages on the worker brokers. Up to two decimals are allowed, for example `0.5%`. | | `true` | All multi-stage queries run their intermediate stages on the worker brokers. | ### Roll out step by step 1. Make sure that the worker brokers are running and that they carry the worker tag. See [Workload isolation with broker tags](#workload-isolation-with-broker-tags). 2. Set a small percentage as a cluster config, for example with the controller API: ```text theme={null} POST /cluster/configs {"startree.broker.mse.use.broker.for.intermediate.stage": "10%"} ``` 3. Monitor query errors, timeouts and latency, and the CPU and heap of the worker brokers. 4. Increase the percentage in steps, for example 10%, then 30%, then 100%. 5. To roll back, set the value to `0` or `false`. New queries use the new value. Running queries are not affected. ### How brokers pick queries * Each broker decides at random for each query. The decision covers all intermediate stages of that query. The same SQL can take either path on different runs. * A query that sets `useBrokerForIntermediateStage` itself is not part of the rollout. For example, `SET useBrokerForIntermediateStage = false;` keeps one query on the servers at any percentage. * With a percentage, each query response gets `"rolloutBrokerAsWorker": true` or `false` in its `responseMetadata`. You can use it to compare the two groups. Failed queries, `EXPLAIN`, and queries that set the option themselves do not get the entry. * The entry shows the routing decision only. If no worker broker is available, a picked query still runs on the servers. * The value comes from, in order: the query option, the cluster config, the broker's own config, then `false`. If you delete the cluster config, brokers go back to their own config. * If the cluster config value is not valid, brokers log a warning and use their own config until you correct the value. Brokers on older releases read a percentage as `false`. > [!WARNING] > > If your cluster has more than one broker tenant, set the worker tag on each tenant's brokers before you exceed 0%. Otherwise, those tenants send their intermediate stages to the `DefaultTenant_BROKER` workers. Each broker uses its own `startree.broker.mse.default.worker.broker.tag`, and a change to it needs a broker restart. # Configuration Reference ### Broker Configs | Config Key | Default | Description | | :- | :- | :- | | [startree.broker.mse.use.broker](http://startree.broker.mse.use.broker).for.intermediate.stage | false | Mandatory. Enable using tagged brokers as workers for MSE intermediate stages. | | [startree.broker.mse.default.worker.broker](http://startree.broker.mse.default.worker.broker).tag | DefaultTenant\_BROKER | Helix tag identifying which brokers serve as intermediate-stage workers. | | [startree.broker](http://startree.broker).mse.num.brokers.for.intermediate.stage | -1 (all) | Max number of tagged brokers to use for intermediate stages; -1 means use all available | ### Query Options (per-query overrides) These can be set via SET statements to override cluster defaults on a per-query basis. | Query Option | Description | | :- | :- | | useBrokerForIntermediateStage | Turn decoupled mode on or off for this query (true/false). It overrides the cluster default in both directions. From 0.16, `true` sends this query's intermediate stages to worker brokers even when the default is off, as long as brokers run workers. `false` keeps them on the servers (old behavior). | | mseWorkerBrokerTag | Target a specific broker pool by tag for this query's intermediate stages. | | numBrokersForIntermediateStage | Override the max broker count for this query's intermediate stages. Default = all brokers in the given tenant. | ### Example ```sql theme={null}   SET useBrokerForIntermediateStage = true;   SET mseWorkerBrokerTag = 'highMem_BROKER';   SET numBrokersForIntermediateStage = 4;   SELECT a.col1, b.col2   FROM tableA a JOIN tableB b ON a.id = b.id ``` # Workload Isolation with Broker Tags   You can partition brokers into pools using Helix tags and route intermediate stage work to a specific pool:   1. Tag a subset of brokers with a custom tag (e.g., heavyQuery\_BROKER).   2. Set the default tag in broker config: ```text theme={null}   startree.broker.mse.default.worker.broker.tag=heavyQuery_BROKER ```   3. Or override per query: ```sql theme={null} SET mseWorkerBrokerTag = 'heavyQuery_BROKER';   SELECT ... ``` Only brokers carrying the specified tag (and that are live, enabled, and have the feature turned on) will be selected as intermediate-stage workers. ## How Query Routing works with this feature Let's assume we're using two broker pools (pool0 and pool1) - in other words brokers in each of these pools have these tags respectively: 'pool0' and 'pool1'. Let's also assume that brokers in pool0 are exposed for query routing. 1. A query is routed to a broker (brokerA) in pool 0. 2. brokerA will plan the query and 1. Schedule leaf node stages on the respective servers. 2. Schedule intermediate stages on pool 1. 3. Eventually intermediate stages on pool 1 once completed will stream results back to the original brokerA. 4. brokerA will return the final response. # Limiting Broker Parallelism   By default, all brokers matching the tag are used. To cap the number of brokers used for a query's intermediate stages: ```text theme={null}   startree.broker.mse.num.brokers.for.intermediate.stage=4 ```   Or per query: ```text theme={null}   SET numBrokersForIntermediateStage = 2;   SELECT ... ``` This is useful when you want to limit resource consumption for individual queries or prevent a single query from monopolizing the entire broker pool. # Pagination Source: https://docs.startree.ai/corecapabilities/query_data/advanced_operations/pagination > **StarTree Pagination is deprecated. Use cursors instead** > > The StarTree version of Pagination has been deprecated. Instead please use the Cursor functionality available in Apache Pinot. You can find more details  in this [doc](https://docs.pinot.apache.org/build-with-pinot/querying-and-sql/query-execution-controls/query-using-cursors) > **Note:** APIs and configuration knobs may change in future releases. This is useful when: * The result set is large and you want to **limit memory usage** on the client. * An application UI needs **paged navigation** through query results (tables, graphs, drilldowns). * You want **simpler application logic** for scrolling or pagination instead of manually managing `LIMIT`/`OFFSET` in SQL. Pagination should be used **through the StarTree Cloud Proxy service only**. *** ## How Pagination Work A pagination-enabled query follows this lifecycle: 1. A query is executed **once** on the cluster. 2. The **full result set** is stored in a **temporary result store**. 3. The **first page** of results is returned as part of the query response. 4. The client then iterates through the results by calling pagination API with: * `offset` (start row) * `numRows` (page size) 5. The client can: * Move **forward or backward** in the result set. * Change page size (`numRows`) between calls. *** ## Result Store The **Result Store** is responsible for storing query results for pagination-based access. Supported storage options: * `memory` * `file` (local filesystem or deep store such as S3/GCS, depending on configuration) * `s3` * `gcs` > **Important** > > * `memory` is **not suitable for production**. > * `memory` and local filesystem storage do **not** work well with more than one broker, because pagination requests may be routed to brokers that do not hold the result set. *** ## Configuration ### Result Store Configuration These configurations control where and how query results are stored. | Configuration | Default | Description | | - | - | - | | `pinot.broker.pagination.protocol` | `memory` | Protocol to use for storage (`memory`, `file`, `s3`, `gcs`). | | `pinot.broker.pagination.temp.dir` | `/tmp/pinot/query/broker` | Local filesystem directory for temporary files. | | `pinot.broker.pagination.` | — | Configuration prefix for options required by the chosen protocol. | #### Example: Local File Storage ```properties theme={null} pinot.broker.pagination.protocol=file pinot.broker.pagination.temp.dir=/tmp/pinot/broker/query pinot.broker.pagination.file.data.dir=file:///home/pinot/data/query ``` #### Example: File Storage Using S3 ```properties theme={null} pinot.broker.storage.factory.s3.region=us-west-2 pinot.broker.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.broker.pagination.protocol=file pinot.broker.pagination.temp.dir=/tmp/pinot/broker/query_results pinot.broker.pagination.file.data.dir=s3://bucket/dir/query-results/ ``` *** ## Storage Cleaner StarTree supports two cleanup mechanisms: ### PaginationCleaner (Time-Based Cleanup) PaginationCleaner is a periodic job that runs on the controller and deletes query results older than a configured TTL. It must be able to call all brokers and therefore needs appropriate authentication configuration. #### Example: ```properties theme={null} controller.cluster.pagination.cleaner.frequencyPeriod=1h controller.admin.auth.provider.class=org.apache.pinot.common.auth.UrlAuthProvider controller.admin.auth.url=file:///var/run/secrets/kubernetes.io/serviceaccount/token ``` ### Miscellaneous Settings These settings control the default page size and expiry behavior for queries. | Configuration | Default | Description | | :- | :- | :- | | `pinot.broker.query.page.size` | 10000 | Page size if `numRows` is not specified in API calls. | | `pinot.broker.pagination.expiration` | 1h | Time after which a query result will be deleted. | | `controller.cluster.pagination.cleaner.frequencyPeriod` | 1h | Frequency of the periodic task that deletes expired results. | | `controller.cluster.pagination.cleaner.initialDelay` | random | Initial delay before the first run of the cleaner task. | ### Quota-Based Storage Cleaner The Quota-Based Storage Cleaner automatically cleans up temporary query result storage when disk usage exceeds a configured limit. This prevents broker nodes from running out of disk due to: * Long-running queries * Many concurrent cursor queries * Large result sets #### Cleanup Behavior 1. Removes **expired results** first. 2. If storage is still above the target threshold, it cleans additional data based on size. > **Note:** Configuration for this feature can be updated without restarting the broker. #### Example Configuration ```properties theme={null} { "pinot.broker.pagination.file.enableStorageCleaner": "true", "pinot.broker.pagination.file.maxStoragePerBrokerMB": "1024", "pinot.broker.pagination.file.criticalStorageThresholdPercent": "90", "pinot.broker.pagination.file.targetStorageThresholdPercent": "60" } ``` ### Configuration Parameters | Configuration | Default | Description | | - | - | - | | `pinot.broker.pagination.file.enableStorageCleaner` | `false` | Enables or disables the storage cleaner. | | `pinot.broker.pagination.file.maxStoragePerBrokerMB` | `1024` | Maximum storage quota for query results per broker (in MB). | | `pinot.broker.pagination.file.criticalStorageThresholdPercent` | `90` | Cleanup is triggered when usage exceeds this percentage of the max quota. | | `pinot.broker.pagination.file.targetStorageThresholdPercent` | `60` | Cleaner attempts to reduce storage to this percentage of the max quota. | *** ## User APIs ### Submit a Cursor-Enabled Query **Endpoint** ```text theme={null} POST /query/sql ``` **Query Parameters** * `doPaginate` (boolean): Enables cursor-based pagination when `true`. * `numRows` (int, optional): Number of rows to return in the first page. **Example Request** ```bash theme={null} curl -H "Content-Type: application/json" -X POST \ -d '{"sql":"select * from upsertMeetupRsvpLLC limit 10"}' \ "http://localhost:8099/query/sql?doPaginate=true&numRows=1" ``` **Example Response (truncated)** ```json theme={null} { "resultTable": { "dataSchema": { "columnNames": ["event_id", "mtime"], "columnDataTypes": ["STRING", "TIMESTAMP"] }, "rows": [ [ "event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_3461491536", "2023-03-10 22:18:46.394" ] ] }, "requestId": "1599500010000000000", "brokerId": "Broker_dogfood-newer-pinot-broker-0.dogfood-newer-pinot-broker-headless.managed.svc.cluster.local_8099", "numRowsResultSet": 10, "offset": 0, "nextOffsetParams": "offset=1&numRows=1", "numRows": 1, "exceptions": [] } ``` *** ### Iterate Over Results **Endpoint** ```text theme={null} GET /query/{requestId}/results ``` **Query Parameters** * `offset` (int, required): Start offset of the page. * `numRows` (int, optional): Number of rows in this page. Defaults to `pinot.broker.query.page.size`. **Example Request** ```bash theme={null} curl -X GET \ "http://localhost:8099/query/1599500010000000000/results?offset=1&numRows=1" -H "Content-Type: application/json" \\ -H "Authorization: Bearer " \\ -H "FORWARD_HOST: " \\ -H "FORWARD_PORT: 8099" ``` **Example Response (truncated)** ```json theme={null} { "resultTable": { "dataSchema": { "columnNames": ["event_id", "mtime"], "columnDataTypes": ["STRING", "TIMESTAMP"] }, "rows": [ [ "event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_event_id_3458370505", "2023-03-11 01:57:26.394" ] ] }, "requestId": "1599500010000000000", "brokerId": "Broker_dogfood-newer-pinot-broker-0.dogfood-newer-pinot-broker-headless.managed.svc.cluster.local_8099", "numRowsResultSet": 10, "offset": 1, "nextOffsetParams": "offset=2&numRows=1", "numRows": 1, "exceptions": [] } ``` **Cursor Fields** * `requestId`: Unique ID for the query * `offset`: First record in current response * `numRows`: Number of rows in this page * `nextOffsetParams`: Pre-filled query parameters for the next page *** ### Iterate Over Results for Local Filesystem When using the broker's local filesystem as the result store, the StarTree Cloud Proxy must route calls to the correct broker. Each response includes: * `brokerHost` * `brokerPort` These must be included as request headers in subsequent pagination calls: **Initial Query Request** ```bash theme={null} curl -H "Content-Type: application/json" -X POST \ -d '{"sql":"select * from upsertMeetupRsvpLLC limit 10"}' \ "http:///query/sql?doPaginate=true&numRows=1" ``` **Example Response Fragment** ```json theme={null} { "brokerHost": "broker.host.name", "brokerPort": 8443 } ``` **Subsequent Pagination Request** ```bash theme={null} curl -X GET "http:///query/1599500010000000000/results?offset=1&numRows=1" -H "Content-Type: application/json" \\ -H "Authorization: Bearer " \\ -H "FORWARD_HOST: " \\ -H "FORWARD_PORT: 8443" ``` *** ### Retrieve Query Metadata **Endpoint** ```text theme={null} GET /query/{requestId}/metadata ``` **Example Request** ```bash theme={null} curl -H "Content-Type: application/json" -X GET \ "http://localhost:8099/query/1599500010000000000/metadata" ``` **Example Response (truncated)** This returns the broker response metadata for the query, such as: ```json theme={null} [ { "requestId": "1599500010000000000", "brokerId": "Broker_dogfood-newer-pinot-broker-0.dogfood-newer-pinot-broker-headless.managed.svc.cluster.local_8099", "exceptions": [], "numServersQueried": 1, "numServersResponded": 1, "numSegmentsQueried": 24, "numSegmentsProcessed": 22, "numSegmentsMatched": 2, "numConsumingSegmentsQueried": 2, "numConsumingSegmentsProcessed": 2, "numConsumingSegmentsMatched": 0, "numDocsScanned": 20, "numEntriesScannedInFilter": 0, "numEntriesScannedPostFilter": 40, "numGroupsLimitReached": false, "maxRowsInJoinReached": false, "totalDocs": 10030562, "timeUsedMs": 416, "offlineThreadCpuTimeNs": 0, "realtimeThreadCpuTimeNs": 0, "offlineSystemActivitiesCpuTimeNs": 0, "realtimeSystemActivitiesCpuTimeNs": 0, "offlineResponseSerializationCpuTimeNs": 0, "realtimeResponseSerializationCpuTimeNs": 0, "offlineTotalCpuTimeNs": 0, "realtimeTotalCpuTimeNs": 0, "brokerReduceTimeMs": 12, "segmentStatistics": [], "traceInfo": {}, "partialResult": false, "numRowsResultSet": 0, "minConsumingFreshnessTimeMs": 1715864004434, "explainPlanNumEmptyFilterSegments": 0, "numSegmentsPrunedByServer": 2, "numSegmentsPrunedInvalid": 0, "numSegmentsPrunedByLimit": 0, "numSegmentsPrunedByValue": 0, "explainPlanNumMatchAllFilterSegments": 0, "numSegmentsPrunedByBroker": 0 } ] ``` *** ## Admin APIs ### List Stored Query Results **Endpoint** ```text theme={null} GET /query/resultStore ``` **Example Request** ```bash theme={null} curl -H "Content-Type: application/json" -X GET \ "http://localhost:8099/query/resultStore/" ``` **Example Response (truncated)** ```json theme={null} { "resultMetadataList": [ { "requestId": "1599500010000000000", "brokerId": "Broker.local_8099", "submissionTimeMs": 1715864073212, "expirationTimeMs": 1715864076812, "responseMetadata": [ { "type": "fs", "numRows": 10, "dataFile": "s3://.../query-results//1599500010000000000/resultTable_0.json", "metadataFile": "s3://.../query-results//1599500010000000000/response_0.json" } ] } ] } ``` *** ### Delete Query Results **Endpoint** ```text theme={null} DELETE /query/resultStore/{requestId} ``` **Example Request** ```bash theme={null} curl -H "Content-Type: application/json" -X DELETE \ "http://localhost:8099/query/resultStore/1599500010000000000" ``` **Example Response** ```text theme={null} Query Results for 1599500010000000000 deleted. ``` # Query Logger Source: https://docs.startree.ai/corecapabilities/query_data/advanced_operations/query-logger > [!NOTE] > > Minimum Startree version needed for this feature is 0.15 ## **Overview** Query Logger captures a **record of the queries your Pinot cluster serves** and stores it in a queryable Pinot table called `system_query_log`. By default it keeps the queries worth investigating, slow ones and failed ones, and you can switch it to capture every query (see [What gets captured](#what-gets-captured)). Instead of hunting through broker logs, you run ordinary SQL against your own query history to answer questions like: * Which queries are the **slowest** or the **most expensive** right now? * Which query **patterns** (templates) dominate load? * Which queries are **scanning too much data**, hitting **limits**, or **erroring**? * How has a specific query's latency or resource usage **changed over time**? Each row = one query executed by a broker, enriched with timing, data-scan counts, CPU/memory usage, and the query text itself. ## **How it's enabled** On current StarTree releases, **Query Logger is on by default**. StarTree provisions the `system_query_log` table and configures the capture pipeline and storage location for you, so you can query the table like any other (via the Query Console or the query API) with no client-side setup. On older releases it is off by default; **contact StarTree Support to enable it for your cluster.** To turn capture off, set `querylog.enabled` to `false` (see [Configuration](#configuration)). The change applies without restarting brokers, and data already captured stays until retention removes it. ## **Browse query history in the Data Portal** You don't have to write SQL to look through `system_query_log`. In the Data Portal, open **Monitoring → Query Insights**. The **History** tab reads the same table and shows totals for success, partial-response and failed queries, p99 latency, query-rate and latency charts, and a sortable list of the captured queries. Filter by time range, engine, status or table. ![Query Insights History tab in the Data Portal](https://mintcdn.com/startree/UNjAl9AuYBniRVw2/corecapabilities/query_data/images/query-insights-history.png?fit=max\&auto=format\&n=UNjAl9AuYBniRVw2\&q=85\&s=721e4daf649032e5b4e77eedebbc7c38) The counts cover captured queries only. With the default capture settings that means slow and failed queries, not your total query volume. History refreshes about every 10 minutes, the same delay as the table itself. ## **What gets captured** By default, a query is captured when **either** of these is true: * It took **1 second or longer** (`timeUsedMs >= 1000`). * It **failed** with an error. Fast, successful queries are not stored. This keeps the table focused on the queries you are likely to investigate, and keeps its storage footprint small on busy clusters. > [!NOTE] > > Clusters that had Query Logger enabled before this default was introduced keep capturing **every** query after upgrading. Set `querylog.capture.threshold.enabled` explicitly to choose either behavior. A few kinds of query are left out by default: * **Queries against `system_query_log` itself**, such as the ones you run while browsing query history. This stops the log from filling up with its own lookups. * **StarTree health checks and uptime probes.** These are identified by the client that sends them, not by the SQL, so your own queries are never excluded by accident. * **Queries you opt out individually.** Prepend `SET skipQueryLog=true;` to a statement you don't want recorded, such as a high-frequency health check of your own: ```sql theme={null} SET skipQueryLog=true; SELECT * FROM myTable LIMIT 1; ``` This only affects logging. The query still runs and returns results normally. ## **Configuration** Query Logger settings are cluster configs. Change them with the controller's cluster-config API: ```shellscript theme={null} curl -X POST \ 'http://localhost:9000/cluster/configs' \ -H 'Content-Type: application/json' \ -d '{ "querylog.capture.threshold.latency.ms": "500" }' ``` To return a setting to its default, delete it: ```shellscript theme={null} curl -X DELETE 'http://localhost:9000/cluster/configs/querylog.capture.threshold.latency.ms' ``` ### Capture settings These apply to new queries without restarting brokers. | Setting | Default | What it does | | - | - | - | | `querylog.enabled` | `true` | Turns capture on or off. | | `querylog.capture.threshold.enabled` | `true` | `true` captures only slow and failed queries. `false` captures every query. | | `querylog.capture.threshold.latency.ms` | `1000` | A query is "slow" when it takes at least this many milliseconds. Set `-1` to capture failed queries only. | | `querylog.capture.threshold.errors` | `true` | Whether failed queries are always captured, regardless of latency. | | `querylog.capture.table.allowlist` | empty (all tables) | Comma-separated list of tables to capture, for example `orders, events`. A query is captured if **any** table it touches is on the list. Names are case-insensitive, and the `_OFFLINE` / `_REALTIME` suffix is optional. | | `querylog.include.self.queries` | `false` | Set `true` to also capture queries against `system_query_log`. | | `querylog.max.query.length` | `4096` | Maximum characters of SQL text stored in the `query` column. Longer queries are truncated. Raising it increases storage per row and only affects newly captured queries. | | `querylog.capture.query.options` | `true` | Whether each query's options (for example `timeoutMs`) are stored in the `queryOptions` column. | > [!WARNING] > > Setting `querylog.capture.threshold.errors` to `false` together with `querylog.capture.threshold.latency.ms` set to `-1` leaves nothing to capture. Use `querylog.enabled` to turn capture off instead. > [!NOTE] > > The table setting is an allowlist only. There is currently no setting to capture every table **except** a few. ### Table settings These control the `system_query_log` table itself. | Setting | Default | What it does | | - | - | - | | `querylog.systemtable.retention.ttl.days` | `7` | How many days of query history are kept. | | `querylog.systemtable.retention.max.size.bytes` | `322122547200` (300 GiB) | Storage cap for the table. `-1` removes the cap. | | `querylog.systemtable.broker.tenant` | `DefaultTenant` | Broker tenant that serves the table. | | `querylog.systemtable.server.tenant` | `DefaultTenant` | Server tenant that stores the table. Its servers must be tagged for offline tables. | > [!WARNING] > > Table settings take effect **when the table is first created**. Changing them on a cluster where `system_query_log` already exists does not update the table. To change retention, the storage cap, or the tenant on an existing cluster, contact StarTree Support. Set the tenants before the table is created if your cluster has no `DefaultTenant`, or if you want query history kept off the servers that handle production traffic. ## **The data: key fields** The table has \~70 columns. The ones you'll use most, grouped by purpose: **Identity & context** | Field | Type | Meaning | | - | - | - | | `logTimestampMs` | LONG (epoch ms) | When the query was logged — your **time filter** column | | `requestId` | LONG | Unique broker request ID | | `query` | STRING | The SQL text (truncated if very long) | | `queryHash` | STRING | Stable hash of the query *template* (literals normalized) — group by this to bucket parameterized queries\* | | `tableName` / `tablesQueried` | STRING | Table(s) the query targeted | | `brokerId` | STRING | Broker that served it | | `queryEngine` | STRING | `singleStage` or `multiStage` | **Timing & result** | Field | Meaning | | - | - | | `timeUsedMs` | Total end-to-end latency (ms) — the headline "how slow" number | | `brokerReduceTimeMs` | Time spent merging results on the broker | | `numRowsResultSet` | Rows returned | | `numExceptions` / `exceptions` | Error count and error text | | `partialResult` | Query returned incomplete results | **Data scanned (why a query is expensive)** | Field | Meaning | | - | - | | `numDocsScanned` | Rows actually read | | `numEntriesScannedInFilter` | Entries scanned evaluating filters — high = missing/weak index | | `numEntriesScannedPostFilter` | Entries scanned after filtering (projection cost) | | `numSegmentsQueried` / `numSegmentsProcessed` / `numSegmentsMatched` | Segment fan-out | | `totalDocs` | Total rows in the table at query time | **Resource usage** | Field | Meaning | | - | - | | `offlineTotalCpuTimeNs` / `realtimeTotalCpuTimeNs` | CPU time (ns) on offline / realtime servers | | `offlineTotalMemAllocatedBytes` / `realtimeTotalMemAllocatedBytes` | Memory allocated | **Limit / safety flags** (booleans): `numGroupsLimitReached`, `groupsTrimmed`, `maxRowsInJoinReached`, `maxRowsInWindowReached`, `maxRowsInDistinctReached` — these tell you a query hit a guardrail and may have returned trimmed/partial data. **Deep diagnostics**: `stageStats` (JSON, multi-stage per-operator stats), `serverStats`, `traceInfo`, `responseMetadataJson` — for detailed drill-down. > \* `queryHash` is populated only when query fingerprinting is enabled for your cluster; otherwise it may be empty. Ask StarTree Ops if you plan to rely on it. ## **Sample queries** > All examples filter to the **last 24 hours** using `logTimestampMs`. Adjust the window as needed. Always keep a time filter on these queries (see Limitations & caveats). **Top 10 slowest queries (last 24h)** ```sql theme={null} SELECT logTimestampMs, tableName, timeUsedMs, numDocsScanned, query FROM system_query_log WHERE logTimestampMs > now() - 86400000 ORDER BY timeUsedMs DESC LIMIT 10 ``` **Top 10 most CPU-expensive queries** ```sql theme={null} SELECT logTimestampMs, tableName, timeUsedMs, offlineTotalCpuTimeNs + realtimeTotalCpuTimeNs AS totalCpuNs, query FROM system_query_log WHERE logTimestampMs > now() - 86400000 ORDER BY totalCpuNs DESC LIMIT 10 ``` **Worst query *patterns* — aggregate by template** (find the query shapes driving load) ```sql theme={null} SELECT queryHash, COUNT(*) AS executions, AVG(timeUsedMs) AS avgLatencyMs, MAX(timeUsedMs) AS maxLatencyMs, AVG(numEntriesScannedInFilter) AS avgFilterScan, ANY_VALUE(query) AS sampleQuery FROM system_query_log WHERE logTimestampMs > now() - 86400000 GROUP BY queryHash ORDER BY avgLatencyMs DESC LIMIT 20 ``` **Heavy scanners (candidates for a better index)** ```sql theme={null} SELECT logTimestampMs, tableName, numEntriesScannedInFilter, numDocsScanned, timeUsedMs, query FROM system_query_log WHERE logTimestampMs > now() - 86400000 AND numEntriesScannedInFilter > 1000000 ORDER BY numEntriesScannedInFilter DESC LIMIT 25 ``` **Failed / errored queries** ```sql theme={null} SELECT logTimestampMs, tableName, numExceptions, exceptions, query FROM system_query_log WHERE logTimestampMs > now() - 86400000 AND numExceptions > 0 ORDER BY logTimestampMs DESC LIMIT 50 ``` **Queries that hit a limit guardrail** (possibly returning trimmed results) ```sql theme={null} SELECT logTimestampMs, tableName, numGroupsLimitReached, groupsTrimmed, maxRowsInJoinReached, timeUsedMs, query FROM system_query_log WHERE logTimestampMs > now() - 86400000 AND (numGroupsLimitReached = true OR groupsTrimmed = true OR maxRowsInJoinReached = true) ORDER BY logTimestampMs DESC LIMIT 50 ``` **Query volume & latency by table** ```sql theme={null} SELECT tableName, COUNT(*) AS queries, AVG(timeUsedMs) AS avgMs, PERCENTILETDIGEST(timeUsedMs, 99) AS p99Ms FROM system_query_log WHERE logTimestampMs > now() - 86400000 GROUP BY tableName ORDER BY queries DESC ``` ## **How the data is kept tidy** You don't manage any of this — it's handled automatically — but it's useful to know: * **Automatic segment compaction.** The pipeline lands data as many small segments (one every few minutes per broker). A **background merge/rollup job runs on a schedule and compacts those small segments into fewer, larger ones**, keeping the **total segment count bounded** as the table grows. No action is required on your part. * **Automatic retention.** Data older than the configured retention window is dropped automatically, so the table stays within its storage budget. ## **Limitations & caveats** > ⚠️ **It consumes storage on your cluster.** `system_query_log` is a real Pinot table — its segments occupy disk on your servers. StarTree keeps it bounded with an automatic **retention window** (older data is dropped, **7 days** by default) and a **storage quota** (**300 GiB** by default), and the background merge job (above) keeps the segment count in check, so the footprint stays finite. When the table reaches its quota, newly captured queries stop landing until retention frees space; existing history is not deleted early. High-QPS clusters, or clusters that capture every query, generate more log data; see [Table settings](#table-settings) and talk to StarTree Support if you need a longer window or expect large volume. > ⚠️ **Your analysis queries run on the same cluster.** Queries against `system_query_log` execute on the **same brokers and servers serving your production traffic** and compete for the same resources. **Always filter by** `logTimestampMs` to a bounded window and avoid unbounded `SELECT *` scans, so debugging doesn't add load to an already-busy cluster. * **Near-real-time, not instant.** Logged queries appear after background segment generation runs — expect a **short delay (roughly 10–15 minutes)** before a just-executed query is queryable. It's for analysis and debugging, not live monitoring. * **Long fields are truncated.** `query` (4096 characters by default, see `querylog.max.query.length`), `exceptions`, `serverStats`, and `traceInfo` have maximum lengths; extremely large values are cut off. * **One row per captured query.** By default only slow and failed queries are stored (see [What gets captured](#what-gets-captured)), so counts from this table are not total query volume unless capture is set to every query. * **Best-effort capture.** Under extreme broker load, or during an error storm where many queries qualify at once, some records may be dropped rather than slow down query serving. # Query Quotas in Apache Pinot Source: https://docs.startree.ai/corecapabilities/query_data/advanced_operations/query-quotas Apache Pinot supports three types of query-level quotas that work together in a hierarchical system to control query rates and prevent resource exhaustion. ## **Overview of Quota Types** The three quota types are enforced in hierarchical order, with the **most restrictive (lowest QPS) quota taking precedence** ## **1. Table-Level Quotas** Table quotas limit QPS for specific tables and have the highest precedence in the hierarchy. This can be specified in the table config as shown in this example: ```json theme={null} { "tableName": "pinotTable", "tableType": "OFFLINE", "quota": { "maxQueriesPerSecond": 300 }, ... } ``` ## **2. Database-Level Quotas** Database quotas apply to all tables within a specific database and are checked before table quotas. This is typically set in the cluster config as shown below: ```shellscript theme={null} curl -X POST \ 'http://localhost:9000/cluster/configs' \ -d '{ "databaseMaxQueriesPerSecond" : "1000" }' ``` This can also be set using the database API ```shellscript theme={null} # to set database specific quota curl -X POST 'http://localhost:9000/databases/{databaseName}/quotas?maxQueriesPerSecond=1200' ``` ## **3. Application-Level Quotas** Application quotas limit QPS based on the application identifier passed in query options. This can be set in the cluster config as shown below: ```shellscript theme={null} curl -X POST \ 'http://localhost:9000/cluster/configs' \ -d '{ "applicationMaxQueriesPerSecond" : "1000" }' ``` You could also use the API directly: ```shellscript theme={null} # to set application's quota curl -X POST 'http://localhost:9000/applicationQuotas/{applicationName}?maxQueriesPerSecond=1200' ``` In case of application specific quota, you do need to specify the application name in the query as shown: ```javascript theme={null} set applicationName='test'; select * from tables ``` For more details on the syntax, how they're imposed - please refer to the Apache Pinot documentation page that describes this in detail: [Query Quotas](https://docs.pinot.apache.org/users/user-guide-query/query-quotas) ## **Example Scenarios** | **Table Quota** | **Database Quota** | **Application Quota** | **Effective Limit** | **Reason** | | :- | :- | :- | :- | :- | | 10 QPS | 25 QPS | 50 QPS | **10 QPS** | Table quota is lowest | | 30 QPS | 15 QPS | 50 QPS | **15 QPS** | Database quota is lowest | | 30 QPS | 25 QPS | 5 QPS | **5 QPS** | Application quota is lowest | | Not set | 25 QPS | 50 QPS | **25 QPS** | Database quota applies | # Replica Group based Workload Isolation Source: https://docs.startree.ai/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation ## Context Workload isolation is a new feature in StarTree Cloud (Starting 0.14 release) that allows users to physically isolate queries from different workloads on the same Pinot table. For instance, let’s assume we have following 2 workloads 1. User facing (application) queries: characterized by strict SLA, high impact to business 2. Adhoc queries: Required for triaging/debugging, low impact to business With this feature, we can isolate A and B using replica group based routing isolation. > [!NOTE] > > Replica groups isolate workloads across the **server** fleet. If the workload you want to isolate is dominated by multi-stage intermediate work (joins, large aggregations, sorts), [decoupled mode](/corecapabilities/query_data/advanced_operations/intermediate-stage-on-broker#decoupled-mode-vs-replica-group-workload-isolation) gives you more granular control and needs no rebalance. The two can be used together. ## Pre-requisite For this feature to work, the table must be configured to use replica groups and use the ‘*StarTreeReplicaGroupInstanceSelector*’ instance selector type. For instance let’s consider this sample table config ```json theme={null} { "instanceAssignmentConfigMap": { "OFFLINE": { "tagPoolConfig": { "tag": "Tag1_OFFLINE" }, "replicaGroupPartitionConfig": { "replicaGroupBased": true, "numReplicaGroups": 3, "numInstancesPerReplicaGroup": 3 } } }, "routing": { "instanceSelectorType": "ai.startree.pinot.broker.routing.instanceselector.StarTreeReplicaGroupInstanceSelector" } } ``` In this case, we’ve configured to use replica group based instance assignment & routing with a total of 3 replica groups (each with 3 instances). The layout will look like this **Note**: If the current setup does not include replica groups, it must be enabled before using this feature. This involves changing the table config for all relevant tables and doing a rebalance operation. ## How to use this feature ### Preferred Replicas Once enabled, you can configure isolation at the query level via the preferredReplicas query option. It accepts a comma-separated list of replica group IDs (derived from ideal state, not from instance partitions) that the broker should use to route the query. Suppose we want the following configuration: * Workload A: allowed to use both replica groups (0 and 1) * Workload B: restricted to replica group 2 Queries from Workload A can be written like this ```sql theme={null} SET preferredReplicas='0,1'; select ... from table where ... ``` Whereas, queries from Workload B can we written like this ```sql theme={null} SET preferredReplicas='2'; select ... from table where ... ``` #### **Preferred Replica list semantics** When multiple replica groups are listed, the broker rotates across them by request ID for load balancing — it does not treat the first entry as primary and the rest as failover. So preferredReplicas='0,1' routes roughly half of the requests entirely through RG 0 and the other half entirely through RG 1. To express "prefer RG 0, and fall over to RG 1 only when RG 0 cannot serve a segment", use preferredReplicas='0' together with fallbackReplicas='1' (see the next section). Single-replica case. preferredReplicas='2' means the query is routed only to instances in replica group 2. If any segment is unavailable on RG 2, it is reported as unavailable (partial result) unless fallbackReplicas is set. Therefore, in the happy path, Workload A will be served from RG 0 and RG 1 (load balanced), and Workload B will be served exclusively from RG 2. ### Fallback replicas The fallbackReplicas query option is an optional companion to preferredReplicas. It accepts a comma-separated list of replica group IDs that the broker should consider only when none of the preferredReplicas can serve a given segment. Example: ```sql theme={null} SET preferredReplicas = '0'; SET fallbackReplicas = '1,2'; SELECT ... FROM table WHERE ... ``` In the example above, the broker tries to serve every segment from RG 0; for any segment that RG 0 cannot serve, it tries RG 1, then RG 2, before declaring the segment unavailable. #### **Selection order** For each segment, the broker walks the preferred replicas first (rotated by request ID for load balancing within the preferred set), and only falls through to the fallbackReplicas (in the listed order, no rotation) when no preferred replica has the segment online. Cross-preferred-replica failover takes priority over fallback: with preferredReplicas='0,1' and fallbackReplicas='2', a segment that is offline on RG 0 but online on RG 1 is served from RG 1, and the fallback metric is not incremented. #### **Upsert / dedup tables** Falling back to a different replica group is safe even for upsert / dedup tables: if any segment in a partition is unavailable on an instance, the underlying selector removes that instance as a candidate for all segments in that partition, so the fallback replica is chosen consistently across the whole partition. ## Observability Whenever any segment in a query is served by a fallback replica, the broker increments the per-table meter ***pinot.broker.\
.queriesUsingFallbackReplica*** (exposed to Prometheus as pinot\_broker\_queriesUsingFallbackReplica\_\* with a "table" label). Use this metric to alert on workload isolation being silently violated. Recommended pattern. For strict isolation, set preferredReplicas only. For best-effort isolation with availability under partial outages, set preferredReplicas to your isolation target and fallbackReplicas to the rest — and alert on the queriesUsingFallbackReplica meter so you know when isolation is being broken. Here are the key failover scenarios: | **Failure Scenario** | **Implication (no fallbackReplicas)** | **Implication (with fallbackReplicas set)** | | :- | :- | :- | | Segment replica failure within a preferred RG, but another preferred RG has the segment | Served from the other preferred RG (no metric increment) | Same as left column — fallback is not used | | Segment replica failure across all preferred RGs | Segment reported unavailable → partial result | Served from the first fallback RG that has the segment; queriesUsingFallbackReplica meter is incremented | | Segment replica failure on a preferred RG with upserts enabled | Entire affected partition is failed over to the next preferred RG | Entire partition fails over to the next available RG (preferred or fallback) that has it (consistent across the partition) | | Segment unavailable on every preferred and fallback RG | Partial result | Partial result | ## Limitations * During any kind of segment movement (eg: rebalance or segment relocator -> realtime to offline or across tiers) - workload isolation guarantees are lost (this is something to do with the internals of Apache Pinot as it works today). * Only homogeneous replica groups are supported today. In other words, each replica group must contain the same amount of instances. * When preferredReplicas lists more than one replica group, request-ID-based rotation is the only load-balancing strategy — adaptive server selection is not consulted within the preferred set. # Binary scheduler for workload isolation Source: https://docs.startree.ai/corecapabilities/query_data/advanced_operations/workload-isolation/binary-scheduler > [!WARNING] > > Binary scheduler feature has not been certified for production use in Startree Cloud. This is an experimental feature. ## **How Does Binary Scheduler Work?** The Binary Workload Scheduler is designed to handle two distinct types of workloads: primary workloads (regular application queries) and secondary workloads (adhoc queries, testing, etc.) **Primary Workload Processing:** * Primary workload queries are executed with priority and submitted to runner threads immediately upon arrival * Resources used by primary workload queries are not capped **Secondary Workload Processing:** * Secondary workload queries are identified using the query option `SET isSecondaryWorkload=true` * These queries are processed through a constrained system with restrictions on runner threads, worker threads per query, and total worker threads for all in-progress secondary queries The key benefit is to provide workload isolation: this scheduler can help to prevent expensive adhoc queries from impacting production workloads. Secondary workload can be used for adhoc triaging, testing and other non critical use cases on the same datasets. ## **How to Enable** 1. **Configure the Scheduler**: Set the query scheduler algorithm to `binary_workload` in your Pinot cluster configuration: ``` pinot.query.scheduler.name=binary_workload ``` 2. **Mark Secondary Workload Queries**: Use the query option to identify secondary workload queries ``` SET isSecondaryWorkload=true ``` 3. **Optional Configuration**: Configure the maximum number of pending secondary queries ``` pinot.query.scheduler.binarywlm.maxPendingSecondaryQueries = <> ``` ## **Limitations** * Only works for Single Stage Engine queries * If primary workload queries end up occupying all threads - this will block any secondary workload queries * Secondary workload queries can be rejected with "OutOfCapacityException" when the system cannot accommodate them ## Relevant Metrics * numSecondaryQueries\_Count # JSON Extract Index Source: https://docs.startree.ai/corecapabilities/query_data/functions/json_extract_index `json_extract_index` is a high-performance JSON extraction function in Apache Pinot and StarTree Cloud. Unlike `json_extract_scalar`, which parses raw JSON strings at query time, this function reads values directly from the **JSON index**. By leveraging the prebuilt index, it reduces parsing overhead and improves query latency for large or deeply nested JSON documents. For smaller JSON payloads or highly selective queries, the standard `json_extract_scalar` function may still perform better. ### Syntax ```sql theme={null} json_extract_index(jsonFieldName, 'jsonPath', 'resultType', ['defaultValue']) ``` #### Parameters | Parameter | Description | | - | - | | **`jsonFieldName`** | The name of the JSON column. The column must have a JSON index configured. | | **`jsonPath`** | The JSONPath expression that identifies which field to extract. | | **`resultType`** | The expected output type: `INT`, `LONG`, `FLOAT`, `DOUBLE`, `BIG_DECIMAL`, `STRING`, or their `_ARRAY` variants. | | **`defaultValue`** *(optional)* | The value to return when the path does not exist. If omitted, Pinot throws an error. | #### Examples ```sql theme={null} select ... group by json_extract_index(col, '$.keyA, 'STRING', 'null') ``` ```sql theme={null} select ... regexp_like(json_extract_index(json_data, '$.keyA, 'STRING', 'null'), 'val') ``` ### How It Works When a query uses json\_extract\_index, Pinot accesses the JSON index to fetch the requested value directly, instead of deserializing the entire JSON document. This lookup uses internal index mappings between JSON paths, values, and document IDs, allowing for faster query execution on large datasets. This function supports both single-value and multi-value results, depending on the structure of the field being extracted. ### When to Use Use json\_extract\_index when: • Your dataset contains large JSON blobs or deeply nested objects. • You need to repeatedly extract the same fields across many records. • The column has a JSON index(on String or MAP Column) configured in the table schema. • Ideal for leaf-stage query execution where direct access to the index is available. ### Error Handling If the specified JSON path doesn’t exist and no default value is defined, the query will throw an error. To make your ingestion resilient, always include a defaultValue when paths may be missing. # Lookup UDF Join Source: https://docs.startree.ai/corecapabilities/query_data/functions/lookup-udf-join > [!NOTE] > > The Lookup UDF Join is only supported with the single-stage query engine (v1). For more information about using JOINs with the multi-stage query engine, see JOINs. Lookup UDF is used to get dimension data via primary key from a dimension table allowing a decoration join functionality. Lookup UDF can only be used with a dimension table in Pinot. ## Syntax The UDF function syntax is listed as below: ```json theme={null} lookupUDFSpec: LOOKUP '(' '''dimTable''' '''dimColToLookup''' [ '''dimJoinKey''', factJoinKey ]* ')' ``` * `dimTable` Name of the dim table to perform the lookup on. * `dimColToLookUp` The column name of the dim table to be retrieved to decorate our result. * `dimJoinKey` The column name on which we want to perform the lookup i.e. the join column name for dim table. * `factJoinKey` The column name on which we want to perform the lookup against e.g. the join column name for fact table > [!NOTE] > > * all the dim-table-related expressions are expressed as literal strings, this is the LOOKUP UDF syntax limitation: we cannot express column identifier which doesn't exist in the query's main table, which is the f`actTable` table. > * the syntax definition of `[ '''dimJoinKey''', factJoinKey ]*` indicates that if there are multiple dim partition columns, there should be multiple join key pair expressed. ## Examples ### Single-partition-key-column Example Consider the table `baseballStats` | Column | Type | | - | - | | playerID | STRING | | yearID | INT | | teamID | STRING | | league | STRING | | playerName | STRING | | playerStint | INT | | numberOfGames | INT | | numberOfGamesAsBatter | INT | | AtBatting | INT | | runs | INT | And the dim table `dimBaseballTeams` | Column | Type | | :- | :- | | teamID | STRING | | teamName | STRING | | teamAddress | STRING | Valid queries include the following: ### Dim-Fact LOOKUP example ```json theme={null} SELECT playerName, teamID, LOOKUP('dimBaseballTeams', 'teamName', 'teamID', teamID) AS teamName, LOOKUP('dimBaseballTeams', 'teamAddress', 'teamID', teamID) AS teamAddress FROM baseballStats ``` | playerName | teamID | teamName | teamAddress | | :- | :- | - | - | | David Allan | BOS | Boston Red Caps/Beaneaters (from 1876–1900) or Boston Red Sox (since 1953) | 4 Jersey Street, Boston, MA | | David Allan | CHA | null | null | | David Allan | SEA | Seattle Mariners (since 1977) or Seattle Pilots (1969) | 1250 First Avenue South, Seattle, WA | | David Allan | SEA | Seattle Mariners (since 1977) or Seattle Pilots (1969) | 1250 First Avenue South, Seattle, WA | ### Self LOOKUP example ```json theme={null} SELECT teamID, teamName AS nameFromLocal, LOOKUP('dimBaseballTeams', 'teamName', 'teamID', teamID) AS nameFromLookup FROM dimBaseballTeams ``` | teamID | nameFromLocal | nameFromLookup | | :- | :- | :- | | ANA | Anaheim Angels | Anaheim Angels | | ARI | Arizona Diamondbacks | Arizona Diamondbacks | | ATL | Atlanta Braves | Atlanta Braves | | BAL | Baltimore Orioles (original- 1901–1902 current- since 1954) | Baltimore Orioles (original- 1901–1902 current- since 1954) | ### Complex-partition-key-columns Example Consider a single dimension table with the following schema: BILLING SCHEMA | Column | Type | | :- | :- | | customerId | INT | | creditHistory | STRING | | firstName | STRING | | lastName | STRING | | isCarOwner | BOOLEAN | | city | STRING | | maritalStatus | STRING | | buildingType | STRING | | missedPayment | STRING | | billingMonth | STRING | ### Self LOOKUP example ```json theme={null} select customerId, missedPayment, LOOKUP('billing', 'city', 'customerId', customerId, 'creditHistory', creditHistory) AS lookedupCity from billing ``` | customerId | missedPayment | lookedupCity | | :- | :- | :- | | 341 | Paid | Palo Alto | | 374 | Paid | Mountain View | | 398 | Paid | Palo Alto | | 427 | Paid | Cupertino | | 435 | Paid | Cupertino | ## Usage * The data return type of the UDF will be that of the `dimColToLookUp` column type. * When multiple primary key columns are used for the dimension table (e.g. composite primary key), ensure that the order of keys appearing in the lookup() UDF is the same as the order defined in the `primaryKeyColumns` from the dimension table schema. # StarTree MCP server Source: https://docs.startree.ai/corecapabilities/query_data/mcp/startree-mcp StarTree provides an ability to deploy a MCP server within the environment. This can be leveraged for various agentic workflows and conversational analytics use cases. ![StarTree MCP](https://mintcdn.com/startree/db5_Hpj5dSqLuq1y/corecapabilities/query_data/images/startree-mcp.png?fit=max\&auto=format\&n=db5_Hpj5dSqLuq1y\&q=85\&s=568000ea4758892ed63ae3f6ac85c1c7) This MCP server is based off of OSS GitHub - [startreedata/mcp-pinot](https://github.com/startreedata/mcp-pinot): MCP Server for Apache Pinot and exposes several tools such as: list\_tables, table\_details, create\_schema, read\_query etc. It provides OAuth based authentication flow for the clients. > [!NOTE] > > Once, deployed, MCP server will be available at [https://mcp.\/mcp](https://mcp.\{domain}/mcp) **For installation** - please reach out to your StarTree representative. ## **Overview** Here are the set of tools exposed by the MCP server: | Tool | What it does | | :- | :- | | `list_tables` | List the tables visible in the cluster | | `get_schema` | Get one table's column schema | | `get_table_config` | Get one table's indexing and ingestion configuration | | `get_table_size` | Reported and estimated storage size for a table | | `list_segments` | List segment names for a table | | `list_segment_metadata` | Page through segment metadata | | `get_segment_index_metadata` | Per-column index details for one segment | | `read_query` | Run a single read-only SQL query | | `test_connection` | Diagnose broker and controller connectivity | | `create_schema` / `update_schema` | Preview or apply a schema change | | `create_table_config` / `update_table_config` | Preview or apply a table-config change | | `reload_table_filters` | Preview or apply the table-filter configuration | ## Conversational Queries on Startree One common pattern we see is use of this MCP server in conversational analytics on top of Pinot and StarTree. You can hook up this MCP server to your favorite AI tool: Claude, Librechat and be able to simply ask natural language questions against the Pinot table. ### Example: Claude Code * Install claude code: [Quickstart - Claude Docs](https://docs.claude.com/en/docs/claude-code/quickstart) * Add mcp server and use it following: [Connect Claude Code to tools via MCP - Claude Docs](https://docs.claude.com/en/docs/claude-code/mcp#authenticate-with-remote-mcp-servers) * Ex: `claude mcp add --transport http pinot-mcp https://mcp./mcp` * In Claude code, use the command `/mcp` to authenticate with StarTree MCP server * Start asking questions. For example, in our demo clickstream Pinot table, we asked this question: "For clickstream table in Pinot, what's the conversion rate ?" Here's the output from Claude Code ![Claude Code Output Pn](https://mintcdn.com/startree/OEv_LmKt0guvQI7k/images/claude-code-output.png?fit=max\&auto=format\&n=OEv_LmKt0guvQI7k\&q=85\&s=a5ef6284002209c05d20b7ba3f94461e) ### **Using it with other MCP clients** The server implements the standard MCP authorization specification — OAuth 2.1 discovery, dynamic client registration and PKCE — so any compliant client can connect with only the URL. **Codex CLI** — add to `~/.codex/config.toml`: `[mcp_servers.pinot]`\ `url = "https://mcp./mcp"` Codex runs the same browser sign-in the first time it connects. **MCP Inspector, LibreChat and other clients** connect the same way, using the same URL. ## **What the server can and cannot do** * **Queries are read-only.**`read_query` accepts a single `SELECT` (or `WITH … SELECT`) statement. Multiple statements, DDL and administrative commands are rejected before anything reaches the cluster. * **Changes are preview-first.** The schema and table-config tools return a preview of exactly what would be sent, together with a single-use confirmation token. Applying a change requires that token, so a change cannot be made in one unreviewed step. * **Results are bounded.** Query results are returned in pages with a flag indicating whether more rows exist, rather than returning unbounded result sets. * **Requests are rate limited** per authenticated user. * **A read-only deployment is available.** Ask your StarTree representative to enable it and the tools that modify schemas and table configurations are not offered to clients at all. ## **Troubleshooting** `needs authentication` — run `/mcp` and choose **Authenticate**. If the server was working previously, choose **Reconnect** instead; the existing authorization is usually still valid. `Unable to connect` — the client cannot reach the URL. Confirm you have the same network access you would need to reach the StarTree console for that environment; if the environment's endpoints are private, connect to your VPN first. The sign-in callback goes to your own machine, so nothing needs to reach you from outside. **Queries return no rows or an unexpected error** — ask the client to run `list_tables` first. Table names are case-sensitive, and the client will use the exact name once it has seen it. ## **Limitations** * **Claude Code is the recommended client.** Claude Desktop's remote MCP support has had compatibility issues with OAuth-protected servers; use Claude Code, or an MCP client with full remote-server support. * **Queries run with the permissions granted to the MCP server** in your environment, not per individual user. Consider a read-only deployment for exploratory or wide access. * **One environment per server.** Each deployment is scoped to the environment it is installed in; connect to several environments by adding one MCP server entry for each. # Query Analyzer Source: https://docs.startree.ai/corecapabilities/query_data/query-analyzer Query Analyzer is an AI-powered tool built into StarTree Cloud that helps you understand and optimize Apache Pinot queries. It analyzes your SQL query alongside table metadata, explain plans, and execution statistics to produce prioritized, evidence-backed recommendations — all without requiring deep Pinot expertise. > [!NOTE] > > Query Analyzer now lives under **AI**. Head to **[AI → Query Analyzer](/corecapabilities/ai/query-analyzer/overview)** for the full overview, usage guide, API reference, and more. # Connecting to StarTree Cloud via Java Source: https://docs.startree.ai/corecapabilities/query_data/query_interfaces/connect-via-java Pinot provides a native Java client to execute queries on the cluster. The client is tenant-aware, and able to direct queries to the correct broker. ### Prerequisite You can use the client by installing and including `Maven` or `Gradle`. For information on how to install Maven, visit [Installing Apache Maven](https://maven.apache.org/install.html). #### Example of how to including the Maven dependency to use the client: ``` org.apache.pinot pinot-java-client 1.0.0 ``` For information on how to install Gradle, visit [Gradle Build Tool Installation](https://gradle.org/install/). #### Example of how to including the Gradle dependency to use the client: ``` include 'org.apache.pinot:pinot-java-client:0.5.0' ``` ### Usage Here's an example of how to use the `pinot-java-client` to query Pinot. ``` import org.apache.pinot.client.Connection; import org.apache.pinot.client.ConnectionFactory; import org.apache.pinot.client.Request; import org.apache.pinot.client.ResultSetGroup; import org.apache.pinot.client.ResultSet; /** * Demonstrates the use of the pinot-client to query Pinot from Java */ public class PinotClientExample { public static void main(String[] args) { // pinot connection String brokerURL = "https://pinot..cp.s7e.startree.cloud"; // this URL should be available with you String pinotClusterName = "PinotCluster"; JsonAsyncHttpPinotClientTransportFactory factory = new JsonAsyncHttpPinotClientTransportFactory(); Connection pinotConnection = ConnectionFactory.fromHostList(Collections.singletonList(brokerURL), factory.buildTransport()); String query = "SELECT COUNT(*) FROM myTable GROUP BY foo"; // set queryType=sql for querying the sql endpoint Request pinotClientRequest = new Request("sql", query); ResultSetGroup pinotResultSetGroup = pinotConnection.execute(pinotClientRequest); ResultSet resultTableResultSet = pinotResultSetGroup.getResultSet(0); int numRows = resultTableResultSet.getRowCount(); int numColumns = resultTableResultSet.getColumnCount(); String columnValue = resultTableResultSet.getString(0, 1); String columnName = resultTableResultSet.getColumnName(1); System.out.println("ColumnName: " + columnName + ", ColumnValue: " + columnValue); } } ``` ### Connection Factory The client provides a `ConnectionFactory` class to create connections to a Pinot cluster. To create a connection, include a comma-separated list of the brokers in the cluster. For example, `("broker-1:1234", "broker-2:1234", ...);` ```java theme={null} Connection connection = ConnectionFactory.fromZookeeper ("some-zookeeper-server:2191/zookeeperPath"); Connection connection = ConnectionFactory.fromProperties("demo.properties"); Connection connection = ConnectionFactory.fromHostList(Collections.singletonList("https://pinot..cp.s7e.startree.cloud"), factory.buildTransport()); ``` ### Query methods Query using one of the following methods: * For blocking queries, use the following request format: `Connection.execute(org.apache.pinot.client.Request)` * For asynchronous queries that return a future object, use the following request format: `Connection.executeAsync(org.apache.pinot.client.Request)` ```java theme={null} ResultSetGroup resultSetGroup = connection.execute(new Request("sql", "select * from foo...")); // OR Future futureResultSetGroup = connection.executeAsync(new Request("sql", "select * from foo...")); ``` To escape query parameters, you can use the `PreparedStatement` (which isn't stored, so won't decrease the subsequent query performance). ```java theme={null} PreparedStatement statement = connection.prepareStatement(new Request("sql", "select * from foo where a = ?")); statement.setString(1, "bar"); ResultSetGroup resultSetGroup = statement.execute(); // OR Future futureResultSetGroup = statement.executeAsync(); ``` ### Result set Access results using various `get` methods on the first `ResultSet` with the `getResultSet(int)` method. ```java theme={null} Request request = new Request("sql", "select foo, bar from baz where quux = 'quuux'"); ResultSetGroup resultSetGroup = connection.execute(request); ResultSet resultSet = resultSetGroup.getResultSet(0); for (int i = 0; i < resultSet.getRowCount(); ++i) { System.out.println("foo: " + resultSet.getString(i, 0)); System.out.println("bar: " + resultSet.getInt(i, 1)); } ``` ### Authentication Pinot supports [basic HTTP authorization](https://docs.pinot.apache.org/operators/tutorials/authentication), which can be enabled for your cluster using configuration. To support basic HTTP authorization in your client-side Java applications, make sure you are using Pinot Java Client 0.10.0+ or building from the latest Pinot snapshot. The following code snippet shows you how to connect to and query a Pinot cluster that has basic HTTP authorization enabled when using the Java client. ```java theme={null} final String username = "admin"; final String password = "verysecret"; // Concatenate username and password and use base64 to encode the concatenated string String plainCredentials = username + ":" + password; String base64Credentials = new String( Base64.getEncoder().encode(plainCredentials.getBytes())); String authorizationHeader = "Basic " + base64Credentials; Map headers = new HashMap(); headers.put("Authorization", authorizationHeader); JsonAsyncHttpPinotClientTransportFactory factory = new JsonAsyncHttpPinotClientTransportFactory(); factory.setHeaders(headers); PinotClientTransport clientTransport = factory .buildTransport(); Connection connection = ConnectionFactory.fromProperties( Collections.singletonList("localhost:8000"), clientTransport); String query = "select count(*) FROM baseballStats limit 1"; ResultSetGroup rs = connection.execute(query); System.out.println(rs); connection.close(); ``` ### Configuring client time-out The following timeouts can be set: * brokerConnectTimeoutMs (default 2000) * brokerReadTimeoutMs (default 60000) * brokerHandshakeTimeoutMs (default 2000) * controllerConnectTimeoutMs (default 2000) * controllerReadTimeoutMs (default 60000) * controllerHandshakeTimeoutMs (default 2000) Timeouts for the Java connector can be added as a connection properties. The following example configures a very low timeout of 10ms: ```java theme={null} Properties connectionProperties = new Properties(); connectionProperties.setProperty("controllerReadTimeoutMs", "10"); connectionProperties.setProperty("controllerHandshakeTimeoutMs", "10"); connectionProperties.setProperty("controllerConnectTimeoutMs", "10"); connectionProperties.setProperty("brokerReadTimeoutMs", "10"); connectionProperties.setProperty("brokerHandshakeTimeoutMs", "10"); connectionProperties.setProperty("brokerConnectTimeoutMs", "10"); // Register new Pinot JDBC driver DriverManager.registerDriver(new PinotDriver()); // Get a client connection and set the connection timeouts Connection connection = DriverManager.getConnection(DB_URL, connectionProperties); // Test that your query successfully times out Statement statement = connection.createStatement(); ResultSet rs = statement.executeQuery("SELECT count(*) FROM baseballStats LIMIT 1;"); while (rs.next()) { String result = rs.getString("count(*)"); System.out.println(result); } ``` ## Obtaining Username and Password You can cenerate an API token in the [Data Portal](/corecapabilities/security/manage-api-tokens#generating-an-api-token) or using the [REST API](/api-reference/introduction#authentication-%26-prerequisites). Then, [obtain the username and password from the service token](/corecapabilities/security/manage-api-tokens#extracting-username-and-password-from-a-startree-bearer-token) Th username and password are used to authorize your API requests. ## Finding Your Broker URL To find the correct Broker URL for your table in StarTree Cloud: 1. Access the Data Portal. 2. Click on **Tables**. 3. Select the specific table you want to query. 4. From the browser address bar, copy the URL to your Pinot cluster. For example, if the URL shown is `https://dp.1abcde6.cp.s7e.startree.cloud/tables`, then the Broker URL for the table will be `broker.pinot.1abcde6.cp.s7e.startree.cloud`. # Connecting to StarTree Cloud via Node.js Source: https://docs.startree.ai/corecapabilities/query_data/query_interfaces/connect-via-nodejs Applications can use this sample client Node.js driver to query Pinot. > [!WARNING] > > This solution may not scale to meet performance and throughput requirements for your use case. ### Integrate Pinot in Node.js applications 1. Install axios. ``` npm install axios ``` 2. Contact StarTree Support to obtain the following information: * **Controller URL:** The general format is `https://pinot..cp.s7e.startree.cloud` * **Broker URL:** The general format is `https://pinot..cp.s7e.startree.cloud` * **AUTH\_TOKEN** 3. Run the Node.js driver script: ``` node queryPinot.js ``` ## queryPinot.js: Node.js driver script ``` const axios = require('axios'); // Pinot broker and controller URLs const pinotBrokerUrl = ''; const pinotControllerUrl = ''; // Function to query Pinot broker async function queryPinotBroker() { try { const response = await axios.post(`${pinotBrokerUrl}/query/sql`, { // Add your SQL query here sql: 'SELECT count(*) FROM , }, { headers: { Authorization: 'Basic ', } }); console.log('Query Result:', response.data); } catch (error) { console.error('Error querying Pinot broker:', error.message); } } // Function to fetch Pinot table schema from controller async function getPinotTableSchema(tableName) { try { const response = await axios.get(`${pinotControllerUrl}/tables/${tableName}/schema`, { headers: { Authorization: 'Basic ', } }); console.log('Table Schema:', response.data); } catch (error) { console.error('Error fetching table schema:', error.message); } } // Example usage queryPinotBroker(); getPinotTableSchema('companies_v1'); ``` ## Obtaining Username and Password You can cenerate an API token in the [Data Portal](/corecapabilities/security/manage-api-tokens#generating-an-api-token) or using the [REST API](/api-reference/introduction#authentication-%26-prerequisites). Then, [obtain the username and password from the service token](/corecapabilities/security/manage-api-tokens#extracting-username-and-password-from-a-startree-bearer-token) Th username and password are used to authorize your API requests. ## Finding Your Broker URL To find the correct Broker URL for your table in StarTree Cloud: 1. Access the Data Portal. 2. Click on **Tables**. 3. Select the specific table you want to query. 4. From the browser address bar, copy the URL to your Pinot cluster. For example, if the URL shown is `https://dp.1abcde6.cp.s7e.startree.cloud/tables`, then the Broker URL for the table will be `broker.pinot.1abcde6.cp.s7e.startree.cloud`. # Connecting to StarTree Cloud using Python Source: https://docs.startree.ai/corecapabilities/query_data/query_interfaces/connect-via-python Applications can use the Python client library to query StarTree Cloud. ## Prerequisites 1. Ensure you have the latest `pinotdb` library installed. To install the library, run the following command in your terminal: ``` pip install pinotdb ``` 2. [Obtain username and password for authentication](#obtaining-username-and-password) 3. [Find your Pinot Broker URL](#finding-your-broker-url) ## Usage ### Query the Pinot Broker Directly ``` from pinotdb import connect USERNAME = '' PASSWORD = '' BROKER_HOST_URL = 'broker.pinot..startree.cloud' conn = connect( host=f'{USERNAME}:{PASSWORD}@{BROKER_HOST_URL}', port=443, path='/query/sql', scheme='https' ) curs = conn.cursor() curs.execute("select * from myTable limit 10") for row in curs: print(row) ``` ### Use SQLAlchemy to Query Pinot The db engine connection string is formatted like this: `pinot://:?controller=://:/` ``` from sqlalchemy import * from sqlalchemy.engine import create_engine from sqlalchemy.schema import * engine = create_engine('pinot://localhost:8099/query/sql?controller=http://localhost:9000/') # uses HTTP by default # engine = create_engine('pinot+http://localhost:8099/query/sql?controller=http://localhost:9000/') # engine = create_engine('pinot+https://localhost:8099/query/sql?controller=http://localhost:9000/') places = Table('places', MetaData(bind=engine), autoload=True) print(select([func.count('*')], from_obj=places).scalar()) ``` ## Clone the Python pinot-dbapi Repository ``` git clone git@github.com:python-pinot-dbapi/pinot-dbapi.git cd pinot-dbapi ``` ## Obtaining Username and Password You can cenerate an API token in the [Data Portal](/corecapabilities/security/manage-api-tokens#generating-an-api-token) or using the [REST API](/api-reference/introduction#authentication-%26-prerequisites). Then, [obtain the username and password from the service token](/corecapabilities/security/manage-api-tokens#extracting-username-and-password-from-a-startree-bearer-token) Th username and password are used to authorize your API requests. ## Finding Your Broker URL To find the correct Broker URL for your table in StarTree Cloud: 1. Access the Data Portal. 2. Click on **Tables**. 3. Select the specific table you want to query. 4. From the browser address bar, copy the URL to your Pinot cluster. For example, if the URL shown is `https://dp.1abcde6.cp.s7e.startree.cloud/tables`, then the Broker URL for the table will be `broker.pinot.1abcde6.cp.s7e.startree.cloud`. # Using the Pinot Proxy Source: https://docs.startree.ai/corecapabilities/query_data/query_interfaces/pinot-proxy Use the Pinot Proxy to route queries to the right broker tenant and access advanced query, cursor, routing, and debug APIs. ## Proxy URL Use the StarTree Pinot Proxy URL in place of the broker URL: ```text theme={null} https://proxy.broker.XXXX.startree.cloud/ ``` where `broker.XXXX` is your regular broker host. The Pinot Proxy routes query requests to brokers while preserving broker-tenant isolation. A table can be mapped to one broker tenant, multiple tables can share a broker tenant, and one broker can serve multiple tenants. ![Pinot Proxy](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/query_data/images/pinot_proxy_tenant.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=a801f6797c7658ebcfb42e2c3f943d39) ## Request Routing The proxy routes a query to a broker only when that broker can serve every table used by the query. If a multi-table query references tables that are not served by one common broker, the query fails. In the diagram above: * A `Table 1 JOIN Table 2` query can route to **Broker 1** or **Broker 2**. * A `Table 1 JOIN Table 3` query can route only to **Broker 2**. * A `Table 1 JOIN Table 4` query cannot be routed. By default, the proxy parses the SQL query to determine the referenced tables. To avoid that parsing overhead, pass table names with the `FORWARD_TABLE` header. > [!NOTE] > > * For multi-table queries, pass one `FORWARD_TABLE` header per table. > * If you use databases, use `.
`. > * Do not include the `_OFFLINE` or `_REALTIME` suffix unless the endpoint explicitly asks for a typed table name. > * Keep the headers consistent with the SQL. The proxy trusts the headers for routing, so an incorrect header can route to a broker that cannot serve the query. ### Header-Routed Query ```bash theme={null} curl "$PROXY_URL/query/sql" \ -H "Authorization: Bearer $TOKEN" \ -H "FORWARD_TABLE: orders" \ -H "FORWARD_TABLE: customers" \ -H "Content-Type: application/json" \ --data @- <<'JSON' { "sql": "SET useMultistageEngine=true; SELECT count(*) FROM orders JOIN customers ON orders.customerId = customers.customerId" } JSON ``` To confirm a query is using the headers, intentionally pass a mismatched table in a non-production environment. The request should fail when the selected broker cannot serve the SQL table. ## Query Endpoints Use these endpoints for normal query execution through the proxy: | Method | Path | Use | | - | - | - | | `POST` | `/query/sql` | Standard Pinot SQL query endpoint. | | `GET` | `/query/sql?sql=...` | GET form of the standard SQL endpoint. | | `POST` | `/sql` | Controller-compatible alias for `/query/sql`. | | `GET` | `/sql?sql=...` | GET form of the controller-compatible SQL alias. | | `POST` | `/query` | Broker query endpoint that uses the multi-stage query engine by default. | | `POST` | `/dal/sql` | Data access layer endpoint with optional engine selection. | ### Standard SQL ```bash theme={null} curl "$PROXY_URL/query/sql" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ --data '{"sql":"SELECT orderStatus, count(*) FROM orders GROUP BY orderStatus LIMIT 10"}' ``` ### Multi-Stage Query Endpoint ```bash theme={null} curl "$PROXY_URL/query" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ --data '{"sql":"SELECT customerId, count(*) FROM orders GROUP BY customerId LIMIT 10"}' ``` ### Data Access Layer Query ```bash theme={null} curl "$PROXY_URL/dal/sql" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ --data '{"sql":"SELECT count(*) FROM orders","engine":"PINOT"}' ``` ## Paginated Query Results For large result sets, enable proxy pagination on `POST /query/sql` by adding `doPaginate=true`. The proxy forwards the request to the broker's StarTree pagination namespace and returns a `requestId` that can be used to fetch result pages. ```bash theme={null} curl "$PROXY_URL/query/sql?doPaginate=true&numRows=1000" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ --data '{"sql":"SELECT * FROM orders ORDER BY orderDate DESC"}' ``` Fetch result pages with the returned request ID: ```bash theme={null} curl "$PROXY_URL/query/$REQUEST_ID/results?offset=0&numRows=1000" \ -H "Authorization: Bearer $TOKEN" ``` Read metadata for the paginated query: ```bash theme={null} curl "$PROXY_URL/query/$REQUEST_ID/metadata" \ -H "Authorization: Bearer $TOKEN" ``` List and clean stored paginated results: ```bash theme={null} curl "$PROXY_URL/query/resultStore" \ -H "Authorization: Bearer $TOKEN" curl -X DELETE "$PROXY_URL/query/resultStore/$REQUEST_ID" \ -H "Authorization: Bearer $TOKEN" ``` > [!WARNING] > > The `/query/{requestId}/results`, `/query/{requestId}/metadata`, and `/query/resultStore` proxy endpoints forward to the older StarTree pagination namespace and are marked deprecated in the proxy implementation. Prefer cursor response-store APIs for new integrations when available in your environment. ## Cursor Response Store The proxy also forwards Pinot cursor response-store APIs. | Method | Path | Use | | - | - | - | | `GET` | `/responseStore` | List response-store metadata. | | `GET` | `/responseStore/{requestId}` | Get response metadata without the result table. | | `GET` | `/responseStore/{requestId}/results?offset=0&numRows=1000` | Fetch a result page. | | `DELETE` | `/responseStore/{requestId}` | Delete one response store. | | `DELETE` | `/responseStore?expiredBefore=` | Delete expired response stores. | Example: ```bash theme={null} curl "$PROXY_URL/responseStore/$REQUEST_ID/results?offset=0&numRows=1000" \ -H "Authorization: Bearer $TOKEN" ``` Deleting response stores requires cluster-level authorization for response-store deletion. ## Routing And Debug APIs Use routing and debug endpoints only for validation, incident response, or with StarTree support guidance. They expose broker routing state and can change routing state when using `PUT` or `DELETE`. | Method | Path | Use | | - | - | - | | `GET` | `/debug/timeBoundary/{tableName}` | Inspect hybrid table time-boundary information. | | `GET` | `/debug/routingTable/{tableName}` | Inspect routing for a typed or untyped table. | | `GET` | `/debug/routingTable?pql=...` | Inspect routing for a PQL query. | | `GET` | `/debug/routingTable/sql?query=...` | Inspect routing for a SQL query. | | `PUT` | `/routing/{tableNameWithType}` | Build or rebuild routing for a table. | | `PUT` | `/routing/refresh/{tableNameWithType}/{segmentName}` | Refresh routing for one segment. | | `DELETE` | `/routing/{tableNameWithType}` | Remove routing for a table. | Examples: ```bash theme={null} curl "$PROXY_URL/debug/timeBoundary/orders" \ -H "Authorization: Bearer $TOKEN" curl --get "$PROXY_URL/debug/routingTable/sql" \ -H "Authorization: Bearer $TOKEN" \ --data-urlencode "query=SELECT * FROM orders_OFFLINE WHERE orderDate >= '2026-01-01' LIMIT 10" ``` Typed table names and segment names in path parameters must be URL encoded when they contain characters that are not path-safe. ## Health And Controller Pass-Through Check proxy health with: ```bash theme={null} curl "$PROXY_URL/health" ``` Requests that do not match a proxy-specific broker/query route are forwarded to the controller. This lets controller APIs continue to work through the proxy, but query clients should prefer the explicit query endpoints listed above. ## Usage Considerations * Use `FORWARD_TABLE` headers for high-throughput applications to avoid repeated SQL parsing at the proxy. * Pass every table used by a join as a separate `FORWARD_TABLE` header. * Keep customer applications on `/query/sql`, `/query`, or cursor endpoints. Reserve debug and routing mutation APIs for operational workflows. * When using databases, include the database-qualified table name in `FORWARD_TABLE`. * Use typed table names such as `orders_OFFLINE` only for APIs that explicitly ask for a table name with type. # Querying Pinot using the REST API Source: https://docs.startree.ai/corecapabilities/query_data/query_interfaces/query-using-rest-API You can query your Pinot data by sending a POST request to the Pinot broker's `/query/sql` endpoint. This endpoint accepts a JSON body containing your SQL query. The basic structure of the request involves: * **Method:** POST * **Endpoint:** `/query/sql` * **Request Body:** A JSON object with a single key, `sql`, whose value is your SQL query string. * **Headers:** You will typically need to set the `Content-Type` header to `application/json`. If you are using StarTree Cloud, you will also need an `Authorization` header with a Bearer token. StarTree Trial accounts require an additional `database` header. ## Prerequisites 1. [Obtain a service token](#obtaining-a-service-token) for authentication. 2. [Find your Pinot Broker URL](#finding-your-broker-url). ## Examples Here are some examples using the curl command-line tool: ### Example 1: Querying StarTree Cloud When querying a Pinot cluster hosted on StarTree Cloud, you need to include an API token for authentication. ```bash theme={null} curl --location --request POST 'https://broker.pinot..startree.cloud/query/sql' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{"sql":"select * from myTable limit 10"}' ``` Remember to replace `` with your specific StarTree Cloud URL and `` with your generated token. ### Example 2: Querying a local Pinot broker If you have a Pinot broker running locally (for example, on localhost:8099), you can query it directly: ```bash theme={null} curl -H "Content-Type: application/json" -X POST \ -d '{"sql":"select * from myTable limit 10"}' \ http://localhost:8099/query/sql ``` Replace `yourTableName` with the actual name of your table. ## Obtaining a Service Token You can cenerate an API token in the [Data Portal](/corecapabilities/security/manage-api-tokens#generating-an-api-token) or using the [REST API](/api-reference/introduction#authentication-%26-prerequisites). This token is used in the Authorization header of your API requests. ## Finding Your Broker URL To find the correct Broker URL for your table in StarTree Cloud: 1. Access the Data Portal. 2. Click on **Tables**. 3. Select the specific table you want to query. 4. The Broker URL for that table will be displayed, and you can copy it to your clipboard. # Query Console Source: https://docs.startree.ai/corecapabilities/query_data/query_interfaces/query_console The StarTree Query Console is a web-based interface for running SQL queries on real-time data stored in StarTree Cloud. It’s designed for fast, interactive analysis and is ideal for developers, data analysts, and engineers working with high-volume streaming datasets. Query Console features include: * **An Interactive SQL Editor**: Run ad-hoc queries with syntax highlighting and auto-complete. * **Live Data Preview**: Instantly view results with sub-second latency on fresh data. * **Table Browser**: Explore available tables, schemas, and metadata. * **Query Templates**: Save, reuse, and share frequently used queries. * **Performance Insights**: View query execution stats like latency and scanned records. * **Multi-Tab Support**: Work on multiple queries simultaneously. * **Save Query Support**: One-click way to save frequently used queries. * **Syntax Highlighting**: Improve readability and reduce errors. * **Semantic Validation**: Ensure query accuracy before execution. * **Modern Design**: A sleek and user-friendly interface. ![Query Console Pn](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/query_data/images/query_console.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=bc76c82b6d5ff72633735ecf57bea16b) # Multi-Stage Query Engine Source: https://docs.startree.ai/corecapabilities/query_data/query_languages/msqe ## Overview and Purpose The Multi-Stage Query Engine (MSQE) is StarTree Cloud's advanced query processing system that enables complex analytical operations such as joins, window functions, and set operations across multiple tables. Unlike the traditional single-stage engine that uses a simple scatter-gather approach, MSQE breaks queries into multiple processing stages that can be efficiently executed across distributed servers. ![Multi Stage Query Engine 1 Sv](https://mintcdn.com/startree/IOs9sznG9CLVp633/images/Multi-stage-query-engine-1.svg?fit=max\&auto=format\&n=IOs9sznG9CLVp633\&q=85\&s=f82535f68f40ba993adf875346778e39) MSQE is particularly valuable for: * Joining data across multiple tables * Complex analytical queries requiring multi-step processing * Queries involving window functions and advanced aggregations * Set operations (UNION, INTERSECT, MINUS) * Applications requiring insights from related data in separate tables **Note:** MSQE is the primary query mode in StarTree Cloud and is recommended for most analytical workloads, especially those involving joins or complex processing. ## Enabling MSQE To query using distributed joins, window functions, and other multi-stage operators, you need to enable the multi-stage query engine. There are several ways to do this: ### Using the Query Console In the StarTree Cloud Query Console, simply select the **Use Multi-Stage Engine** checkbox before running your query. ### Using REST APIs When using the Controller Admin API: ```bash theme={null} curl -X POST http://localhost:9000/sql -d ' { "sql": "select * from baseballStats limit 10", "trace": false, "queryOptions": "useMultistageEngine=true" }' ``` When using the Broker Query API: ```bash theme={null} curl -X POST http://localhost:8000/query/sql -d ' { "sql": "select * from baseballStats limit 10", "trace": false, "queryOptions": "useMultistageEngine=true" }' ``` ### Using the SET Command You can enable MSQE directly in your SQL query by adding a SET command at the beginning: ``` SET useMultistageEngine=true; SELECT * from baseballStats limit 10 ``` ## Key Capabilities ### Table Joins MSQE supports joining tables in a distributed manner using various strategies: ``` SELECT customer.name, orders.amount FROM customer JOIN orders ON customer.id = orders.customer_id WHERE orders.amount > 100 ``` Join strategies include: * Lookup joins (for small dimension tables) * Colocated joins (when data is partitioned on join keys) * Query-time partition joins * Broadcast joins ### Window Functions MSQE enables analytical functions over partitions of result sets: ``` SELECT product_id, date, sales, SUM(sales) OVER (PARTITION BY product_id ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS rolling_7day_sales FROM sales ``` ### Set Operations MSQE supports operations across multiple query results: ``` SELECT customer_id FROM active_customers UNION SELECT customer_id FROM new_signups ``` ## How MSQE Works MSQE processes queries through a multi-stage execution pipeline: 1. **Query Planning**: The query is parsed and transformed into a logical plan 2. **Stage Assignment**: The plan is divided into multiple execution stages 3. **Distributed Execution**: Stages are distributed across servers for parallel processing 4. **Data Shuffling**: When needed, data is moved between stages using optimized shuffling 5. **Result Collection**: Final results are gathered at the root stage and returned Each stage serves a specific purpose: * **Leaf Stages**: Read data directly from tables * **Intermediate Stages**: Process operations like joins and aggregations * **Root Stage**: Collects and finalizes results for the client ## Using MSQE ### Checking Execution Plans To understand how a query will be executed, use the EXPLAIN PLAN command: ```sql theme={null} EXPLAIN PLAN FOR SELECT customer.name, SUM(orders.amount) FROM customer JOIN orders ON customer.id = orders.customer_id GROUP BY customer.name ``` The output shows the logical plan with stages indicated by `PinotLogicalExchange` operators. > [!NOTE] > > **Starting 0.16.0**, `EXPLAIN PLAN FOR` on the multi-stage engine includes **per-segment physical plans** by default, in addition to the logical plan (config: `pinot.query.multistage.explain.include.segment.plan`, now defaulting to `true`). If you need the old logical-plan-only output — for example, to keep existing tooling that parses `EXPLAIN` output working unchanged — use: > > ```sql theme={null} > EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR > SELECT customer.name, SUM(orders.amount) > FROM customer > JOIN orders ON customer.id = orders.customer_id > GROUP BY customer.name > ``` > > or set `pinot.query.multistage.explain.include.segment.plan=false` explicitly. ### Broker-Side Segment Pruning **Starting 0.16.0**, the multi-stage engine prunes segments at the broker during query routing **by default**, matching the behavior of the single-stage engine. The broker applies the table's configured segment pruners — such as time-based and partition-based pruning — before dispatching leaf stages, so queries with selective time or partition filters fan out to fewer segments and servers. No action is needed to benefit from this: any table whose routing config already defines segment pruners gets it automatically. To control the behavior: * Per query, set the `useBrokerPruning` query option (`SET useBrokerPruning=false;`). * Cluster-wide, set the broker config `pinot.broker.multistage.logical.planner.use.broker.pruning=false` (default `true` as of 0.16.0) to restore the previous behavior. ### StarTree Execution Engine StarTree Cloud also offers the StarTree Execution Engine, an opt-in native execution runtime for the multi-stage engine — see [StarTree Execution Engine](/corecapabilities/query_data/query_languages/startree-engine) for how to enable it, supported join hints, and its spill-to-disk behavior for large queries. ### Optimization Tips 1. **Join Key Selection**: Choose join keys that match table partitioning when possible 2. **Filtering Early**: Apply filters before joins to reduce data movement 3. **Dimension Table Design**: Keep dimension tables small for efficient lookup joins 4. **Column Selection**: Select only necessary columns to minimize data transfer ### Offloading Intermediate Stages to Brokers By default every intermediate stage runs on servers, competing with leaf-stage scan and filter work. StarTree Cloud can instead dispatch those stages to a tagged pool of brokers — see [Decoupled intermediate stages for MSE](/corecapabilities/query_data/advanced_operations/intermediate-stage-on-broker) for when this helps, when it does not, and how to enable it. ## Performance Considerations 1. **Join Complexity**: Performance scales with join complexity and table sizes 2. **Data Distribution**: Evenly distributed data performs better than skewed distributions 3. **Partitioning Strategy**: Align table partitioning with common join keys 4. **Memory Requirements**: Complex joins may require additional memory # PromQL Source: https://docs.startree.ai/corecapabilities/query_data/query_languages/promql Query metrics data using the Prometheus Query Language (PromQL). PromQL support is built as a plugin on top of Apache Pinot's Time Series Engine, which is specifically designed to handle time series workloads efficiently. This integration allows you to: * Use familiar Prometheus-style queries while leveraging Pinot's scalable analytics capabilities. * Query high-cardinality time series data efficiently. * Take advantage of StarTree Cloud optimized storage and indexing. The [Time Series Engine](https://docs.pinot.apache.org/basics/releases/1.3.0#timeseries-engine-support-in-pinot-design-doc) in Pinot provides a pluggable framework that enables native support for various time-series query languages like PromQL. This engine is purpose-built for observability use cases and optimized for processing time series data in a more efficient way than traditional SQL queries. ## Prerequisites Before querying metrics with PromQL, ensure you have: * A running Pinot cluster. * Metrics data ingested into tables using the [Prometheus message decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/prometheus). ## Configuration No additional configuration is required to enable TSE or PromQL. The Time Series Engine and PromQL support are enabled by default in StarTree Cloud. The PromQL endpoint is available at `:8000/timeseries`. Supported PromQL queries can be executed on this endpoint - see [Supported PromQL Expressions](#supported-promql-expressions) for details. ## Data Ingestion Using Real-time Tables > [!WARNING] > > The Prometheus Message Decoder is only supported with REALTIME tables in StarTree Cloud. It cannot be used with OFFLINE tables. To ingest Prometheus metrics data in real-time, you'll need to configure both the schema and table configuration properly. The Prometheus Message Decoder expects data in the following format: ``` {=,=,...} ``` ### Required Schema Configuration Your schema must include these specific fields with the exact names and data types: | Field | Type | Description | | - | - | - | | `metric` | STRING | Name of the Prometheus metric | | `labels` | JSON | Key-value pairs of Prometheus labels | | `value` | DOUBLE | The numeric value of the metric | | `ts` | TIMESTAMP | Timestamp in milliseconds since the Unix epoch | ![Table Schema](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/query_data/images/prometheus-schema.jpg?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=6443da136efaaab4c12ea6509f1d3235) ### Table Configuration For real-time ingestion, configure your table with the Prometheus Message Decoder: 1. Set the table type to `REALTIME` 2. Configure the stream settings to use `PrometheusMessageDecoder` 3. Set appropriate flush thresholds for your use case ![Table Configuration](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/query_data/images/prometheus-table-config.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=915d78b2e9588bd525fc1d1001d408db) ### Sample Data Format ![Sample Data](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/query_data/images/prometheus-sample-data.jpg?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=f3898842c471bb08fad687c96973ab0a) ### Important Considerations 1. The schema field names (`metric`, `labels`, `value`, `ts`) must match exactly as specified. 2. If you need different field names, use transform functions to alias them. 3. The timestamp must be in milliseconds since Unix epoch. 4. The labels field must be properly formatted JSON. For more information, see [Prometheus Message Decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/prometheus). ## How to Query PromQL Data There are two main ways to query PromQL data in Pinot: ### 1. Grafana Pinot Plugin You can query PromQL data using the Grafana Pinot plugin: 1. Install the Grafana Pinot plugin ([link](https://grafana.com/grafana/plugins/startree-pinot-datasource/)) 2. Configure your Pinot data source in Grafana. ![Configure Pinot in Grafana](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/query_data/images/configure-pinot-promql.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=bf1379bdb68d27b3333a22992c87a7b7) 3. Use the PromQL query editor in your Grafana dashboards. ![Grafana PromQL Query Editor](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/query_data/images/promql-query-grafana.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=6b436532e5fcc43de701dac57dda7f2e) ### 2. REST API Endpoints You can query PromQL data using timeseries endpoint: #### PromQL API endpoint ```bash theme={null} curl "http://localhost:8000/timeseries/api/v1/query_range" \ -G \ --data-urlencode "query=http_in_flight_requests" \ --data-urlencode "start=1741011820" \ --data-urlencode "end=1741021820" \ --data-urlencode "step=30" \ --data-urlencode "language=promql" \ --data-urlencode "table=prometheusMsg_REALTIME" ``` Parameters explained: * `query`: The PromQL expression to evaluate. * `start`: Start timestamp in Unix epoch seconds. * `end`: End timestamp in Unix epoch seconds. * `step`: Query resolution in seconds (interval between data points). * `language`: Query language type (promql). * `table`: Pinot table name containing the metrics data. ## Supported PromQL Expressions ### Series Selectors * Query Example: `http_requests_total` * Description: Selects time series that match the given metric name. * Usage: To retrieve multiple time series. ### Label Matching and Filtering * Query Examples: ```promql theme={null} http_requests_total{method="GET", status="200"} http_requests_total{method="GET", status="200"}[5m] ``` * Operators: `=`, `!=`, `=~` (regex match), `!~` (regex not match). * Usage: To narrow down queries to specific metrics or labels. ### Offset * Query Example: `http_requests_total offset 10m` * Description: Shifts the time series back or forward in time. * Usage: To compare current data with past data. ### Functions * Supported Functions: * Rate: `rate(http_requests_total[5m])` * Increase: `increase(http_requests_total[5m])` * Delta: `delta(http_requests_total[5m])` * Instant Rate: `irate(http_requests_total[5m])` * Usage: To compute rates, changes, and other complex calculations. ### Aggregation Operators * Supported operators: * Sum: `sum(http_requests_total)` * AVG: `avg(http_requests_total)` * MAX: `max(http_requests_total)` * MIN: `min(http_requests_total)` * Count: `count(http_requests_total)` * Topk: `topk(1, http_requests_total)` * Bottom: `bottomk(2, http_requests_total)` * Usage: To calculate sums, averages, and other metrics across multiple time series. ### By and Without Operators * Examples: ```promql theme={null} sum(http_requests_total) by (status) sum(http_requests_total) without(job) ``` * Description: Apply aggregation operator by or without labels. ### Binary Operators * Supported operators: `+`, `-`, `*`, `/` * Examples: ```promql theme={null} http_requests_total / http_requests_failed http_requests_total - http_requests_failed ``` * Usage: To combine multiple series mathematically. For detailed information about these operators and functions, please refer to the [Prometheus Query Language Documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/). > [!NOTE] > > Only the operators and functions listed above are supported in Pinot, which is a subset of all Prometheus functions and operators. # SQL Source: https://docs.startree.ai/corecapabilities/query_data/query_languages/sql Apache Pinot provides a SQL interface for querying, which uses the **Calcite SQL** parser to parse queries and the **MYSQL\_ANSI** dialect. The multi-stage query engine in Pinot supports inner join, left-outer, semi-join, and nested queries out of the box. Several different types of functions are supported, such as Aggeregation Functions, Transformation Functions, and query operators for querying JSON data. For the complete list of supported functions and description of query syntax, see [Querying Pinot](https://docs.pinot.apache.org/users/user-guide-query/querying-pinot). ## Sample Queries ### Selection ```sql theme={null} //default to limit 10 SELECT * FROM myTable SELECT * FROM myTable LIMIT 100 ``` ```sql theme={null} SELECT "date", "timestamp" FROM myTable ``` ### Aggregation ```sql theme={null} SELECT COUNT(*), MAX(foo), SUM(bar) FROM myTable ``` ### Filtering ```sql theme={null} SELECT COUNT(*) FROM myTable WHERE foo = 'foo' AND bar BETWEEN 1 AND 20 OR (baz < 42 AND quux IN ('hello', 'goodbye') AND quuux NOT IN (42, 69)) ``` ### Ordering on Selection ```sql theme={null} SELECT foo, bar FROM myTable WHERE baz > 20 ORDER BY bar DESC LIMIT 100 ``` StarTree Cloud supports a comprehensive set of functions for data manipulation and analysis. For a complete reference, see the [Apache Pinot Function Documentation](https://docs.pinot.apache.org/functions/supported-aggregations). # StarTree Execution Engine Source: https://docs.startree.ai/corecapabilities/query_data/query_languages/startree-engine ## Overview > [!WARNING] > > The StarTree Execution Engine is a beta capability. Both its opt-in mechanism and the controls described on this page change between releases — a configuration key, query option, or statistic documented here may not exist, or may behave differently, on an older deployment. Check behavior against your cluster's version, and contact StarTree support if something described here is not available in your environment. The StarTree Execution Engine is an alternative, high-performance execution runtime for the [Multi-Stage Query Engine (MSQE)](/corecapabilities/query_data/query_languages/msqe). It keeps everything you already know about MSQE — the SQL dialect, the query planner, and Pinot's segment/index-powered table scans — but executes most of the query's operators (joins, aggregations, sorts, window functions, set operations) in native code instead of Java. Under the hood it is built on [Apache DataFusion](https://datafusion.apache.org/), a vectorized query engine written in Rust, extended with a native Arrow-based data exchange layer for moving data between query stages. ### What changes — and what doesn't A multi-stage query runs in three kinds of stages: | Stage | Java MSQE | StarTree Execution Engine | | - | - | - | | **Leaf stages** (segment scans, filter/index pushdown) | Pinot Java execution | Pinot Java execution (unchanged) | | **Intermediate stages** (joins, aggregations, sorts, windows) | Java operators | Native, vectorized operators | | **Root stage** (final merge on the broker) | Java operators | Native operators | | **Data exchange between stages** | Java mailbox | Native Arrow-based exchange | Query planning is unchanged: the broker still parses, plans, and optimizes the query the same way, so `EXPLAIN` plans, query options, and the general query lifecycle look the same. Because leaf stages stay in Java, all of Pinot's indexing (inverted, sorted, star-tree, JSON, etc.) keeps working exactly as before. ## Enabling the StarTree Execution Engine The native runtime is available on every broker and server. Queries still run on the Java MSQE runtime until something routes them to it — either the [per-query option](#per-query-switch), or the [broker-wide default](#broker-wide-default), which routes every multi-stage query without a per-query choice. > [!NOTE] > > **Use `v2.196.47` or later on the 0.16 line.** Earlier 0.16 patches lack several controls on this page, and on those patches a setting from this page can be rejected or have no effect. Contact StarTree support to confirm the version that your cluster runs. ### The native runtime The runtime starts by default — `pinot.startree.rust.mailbox.port` defaults to `0`, which picks an ephemeral free port. No configuration and no restart are needed to make it available. To stop a broker or server from starting it, set the port to a negative value: ```text theme={null} # Negative: do not start the native runtime on this process. pinot.startree.rust.mailbox.port=-1 ``` A query that sets `workerRuntime = 'startree'` against a process that has opted out, or whose runtime failed to start, fails with a `runtimeKey is not registered` error. > [!NOTE] > > In StarTree Cloud these settings are managed as part of the deployment. Contact StarTree support if the StarTree Execution Engine is not available in your environment. Optional tuning (defaults shown): | Key | Default | Purpose | | - | - | - | | `pinot.startree.rust.metrics.port` | `8079` | Prometheus `/metrics` endpoint for the native runtime (`<= 0` disables) | | `pinot.startree.datafusion.batch.size` | `8192` | Rows per vectorized batch | Memory and spill-to-disk behavior has its own set of settings — see [Spill to disk for large queries](#spill-to-disk-for-large-queries). ### Per-query switch Route an individual multi-stage query to the StarTree Execution Engine with the `workerRuntime` query option. Using the `SET` command: ```sql theme={null} SET useMultistageEngine = true; SET workerRuntime = 'startree'; SELECT ... ``` Or through the query options field of the REST APIs: ```bash theme={null} curl -X POST http://localhost:8000/query/sql -d ' { "sql": "select * from baseballStats limit 10", "queryOptions": "useMultistageEngine=true;workerRuntime=startree" }' ``` > [!NOTE] > > The StarTree Execution Engine only applies to multi-stage queries. Single-stage queries are unaffected by the `workerRuntime` option — `workerRuntime` on its own does nothing unless `useMultistageEngine = true` is also set. ### Broker-wide default To route every multi-stage query to the StarTree Execution Engine without setting the option on each one, set the following in the cluster configuration: ```text theme={null} pinot.startree.datafusion.default.enabled=true ``` This takes effect live and is reversible the same way, with no restart. Routing per query keeps the change scoped to the queries you picked, so prefer the query option while you are still evaluating the engine. ### Returning a query to Java MSQE `DEFAULT` is the explicit identifier for the Java runtime: ```sql theme={null} SET workerRuntime = 'DEFAULT'; ``` An explicit `workerRuntime` always beats the broker-wide default, so this works even on a cluster where `pinot.startree.datafusion.default.enabled` is `true`. Use it to A/B a single query, and to unblock one that misbehaves without changing anything for other queries. ### Confirming which runtime ran Set the option and the query still looks unchanged? Check before concluding anything about performance: * `EXPLAIN` showing a `BROKER_REDUCE` node means the query never reached the multi-stage engine at all — `useMultistageEngine` is missing. * A `queryEngineDowngraded` field in the response metadata means the query reached MSQE but was routed back to Java — see [Monitoring](#monitoring) for how to read it. ## Leaf-stage work and the columnar pull-up The `LeafOperator` keeps running in Java, and it normally carries more than a scan: the filters, the projections, and the partial aggregate are all folded into it. Pushing work down into the leaf is the right choice on Java MSQE, where every row is a Java object either way. On the StarTree Execution Engine it can backfire — work done in the leaf is done on Java objects, and the result then has to be converted into Arrow to cross into the native runtime, which is exactly the allocation the engine exists to avoid. So the engine does the opposite where it pays: it takes the projection, and sometimes the partial aggregate, **out** of the `LeafOperator` and runs it natively above. The scan then emits Arrow columns directly and nothing is ever built as a Java row. ### What moves, and what stays | | Pulled up, runs above the leaf | Stays in the `LeafOperator` | | :- | :- | :- | | **Projections** | On a non-aggregate leaf, when every expression is a cast, arithmetic, a comparison, `AND`/`OR`, or `CASE` | Projections that use any other function (string, date, JSON) | | **Aggregates** | `COUNT`; `SUM`, `MIN`, `MAX`, `AVG` and variance over `FLOAT`/`DOUBLE`; `MIN`, `MAX` over `INT`; `DISTINCTCOUNT` over a column without a dictionary, except `BIG_DECIMAL` | `SUM`, `AVG` and variance over `INT`/`LONG`; `SUM`, `AVG` over `BIG_DECIMAL`; `MIN`, `MAX` over `LONG`/`STRING`; percentiles, sketches, funnel functions; `DISTINCTCOUNT` over a dictionary-encoded or `BIG_DECIMAL` column | | **Query shapes** | — | Aggregates with no `GROUP BY`; group-trim aggregates (their own `ORDER BY` + `LIMIT`); multi-value group keys; `is_leaf_return_final_result` combined with `DISTINCTCOUNT`; anything a star-tree index can already serve | | **Group keys** | Numeric group keys. With `enableNullHandling = true`, dictionary-encoded string keys too | Dictionary-encoded string group keys — grouping on dictionary ids in Java beats interning UTF-8 in the native runtime | The type-specific entries above are not arbitrary. With the engine on, the planner renames `SUM` over `INT`/`LONG` to `SUMINT`/`SUMLONG`, and `MIN`/`MAX` over `LONG`/`STRING` to type-specific names; those names are not on the pull-up list, so that work stays in the leaf. `AVG` and variance are rewritten into `SUM` and `COUNT` first, which is why they pull up over `FLOAT`/`DOUBLE`. > [!NOTE] > > The pull-up decides **where** an aggregate runs. Every aggregate it can take has a native implementation. An aggregate that needs the Java bridge (see [Function support](#function-support)) is never pulled up. Apart from the cost check on group keys, these are structural gates: the decision reads the column's **type and encoding — never its cardinality**. ### Overriding it ```sql theme={null} SET columnarLeaf = 'on'; -- pull up whenever eligible (no cost check) SET columnarLeaf = 'off'; -- never; leave the work in the LeafOperator ``` `'on'` only skips the cost check — the structural gates in the table above still apply, so an aggregate with no `GROUP BY`, a group-trim aggregate, a multi-value key, a star-tree-servable query, or an aggregate that is not on the list stays in the leaf regardless. Only `'on'` forces the pull-up. `'true'` keeps the default, and `'false'`, `'disabled'` or `'none'` act like `'off'`. > [!WARNING] > > **The case that needs `'off'`.** The cost check never keeps a numeric group key in Java, whatever its cardinality. `SELECT status_code, COUNT(*) FROM t GROUP BY status_code` over a billion rows therefore ships every row into the native runtime in order to produce a handful of groups. A low-cardinality numeric `GROUP BY` over a large table is the one shape where the pull-up reliably costs more than it saves — set `columnarLeaf = 'off'` for it. ## When to expect benefits The StarTree Execution Engine accelerates the parts of the query that run *above* the table scan. It shines when intermediate stages do significant work: * **Join-heavy queries**, especially large hash joins between fact tables. * **Large aggregations and group-bys** over data shuffled between servers. * **Sorts and window functions** over sizeable intermediate results. * **Queries that move a lot of data between stages** — the native exchange layer transfers Arrow batches with far less serialization and garbage-collection overhead than the Java mailbox. * **Memory-intensive queries**: the native runtime uses a bounded off-heap memory pool with spill-to-disk for sorts and aggregations, so large queries degrade gracefully instead of pressuring the JVM heap. ### Single-query latency vs. behavior under load These are two different results, and conflating them is the most common way to misread a benchmark. **Single-query latency is mixed and shape-dependent.** In internal benchmarks, join- and aggregation-heavy queries — where the work above the leaf dominates — run meaningfully faster than Java MSQE, while queries whose cost sits in the leaf, or which lean on shapes listed under [When it doesn't help](#when-it-doesnt-help), can be slower. Benchmark the queries you actually care about rather than assuming a uniform speedup. **Behavior under concurrency is the dependable win.** Because intermediate rows stop being Java objects and the exchange moves Arrow buffers instead of serialized Java rows, the JVM stops allocating and collecting that working set. In internal concurrency runs — a synthetic workload on a single machine, so directional rather than a guarantee — this showed up as a large reduction in both allocation and total GC pause time, and in queries completing that had been killed under memory pressure on Java MSQE. Both effects scale with *load*, not with query size, which is precisely why a single-query benchmark hides them. Note that the leaf stage is not covered by this: see [The leaf stage is not covered by any of this](#the-leaf-stage-is-not-covered-by-any-of-this). > [!WARNING] > > **Do not evaluate on a small dataset.** Crossing into the native runtime has a fixed per-query cost. On a small enough dataset that fixed cost *is* the measurement, and queries can look several times slower than Java; the same queries at realistic scale converge or come out ahead. Size the evaluation dataset to the workload you actually run. ### When it doesn't help * **Leaf-dominated queries.** If most of the time is spent scanning segments (highly selective filters, simple aggregations pushed down to the leaf), the leaf stage is identical in both runtimes and the StarTree Execution Engine has little to accelerate. * **Very fast, small queries.** Crossing between the JVM and the native runtime has a fixed cost per query and per batch. Below roughly 100 ms on Java MSQE, that crossing cost dominates and the query can come out slower. * **Bulk casts out of `BIG_DECIMAL`.** There is no native decimal type, so `BIG_DECIMAL` is carried as order-preserving binary. That is why comparison, sort, join, and `GROUP BY` on it are native and lossless — and why converting *out* of it pays per value. A query that casts a large number of `BIG_DECIMAL` values, for example `CAST(price AS FLOAT)` over millions of distinct decimals, pays that cost on every one. * **A few specific query shapes** currently regress and are being actively closed (for example, some ASOF joins and certain UNION ALL patterns). > [!TIP] > > Switching runtimes is a one-line query option, so A/B comparison on your own workload is cheap. If a query matters to you, benchmark it on both runtimes. ### Join strategies Beyond the default hash join, two join strategies are available natively through the standard Calcite join hint: ```sql theme={null} SELECT /*+ joinOptions(join_strategy='sort_merge') */ a.*, b.* FROM a JOIN b ON a.k = b.k ``` | Strategy | Hint value | Notes | | :- | :- | :- | | Sort-merge join | `sort_merge` | INNER/LEFT joins with at least one equality key (including `BIG_DECIMAL` keys). Can spill to disk, so prefer it for very large joins. Falls back to hash join (same results) for unsupported shapes. | | Native lookup join | `lookup` | Dimension-table joins (INNER/LEFT/SEMI/ANTI; non-equality conditions are allowed for INNER/LEFT only) run natively; the dimension-table lookup itself still happens in Java. Unsupported shapes are query errors, not silent fallbacks. | ### Spill to disk for large queries Aggregations (`GROUP BY`/`DISTINCT`), sorts, and sort-merge joins can spill to local disk under memory pressure instead of failing with an out-of-memory error — the query gets slower, not killed. Spill is backed by a shared native memory pool on each broker and server, sized from the available off-heap memory (container memory minus the JVM max heap). Native operators **reserve memory from the pool before consuming it**. If the pool cannot grant a reservation the engine knows before the allocation happens and can act on it — spilling, or cancelling that one query. This is the structural difference from the Java heap, where a query finds out it is out of memory only after the fact. > [!NOTE] > > Two things about the pool are easy to get wrong: > > * **It is per worker process, not per query.** One pool is created at startup and shared by every query and every stage that worker runs. A query that succeeds when run alone can therefore be cancelled under concurrency, because its neighbours take a share of the pool. > * **The pool is divided evenly between op-chains, not between queries.** The engine splits a multi-stage query into stages, and each stage into op-chains. Each op-chain registers with the pool as a separate consumer. If a server runs one op-chain of query Q1 and three op-chains of query Q2, each op-chain gets one quarter of the pool, so Q2 holds three quarters. The share of a query thus depends on how many op-chains its neighbours run on that server, not on how much memory they use. The same query can also get a different share on each server. > * **It does not bound the process's memory.** It bounds native intermediate execution only. The JVM heap — including everything in the leaf stage — sits outside it. | Config | Default | Description | | :- | :- | :- | | `pinot.startree.datafusion.spill.enabled` | `true` | Master on/off switch. When off, the pool is unbounded. | | `pinot.startree.datafusion.spill.memory.size` | `50%` | Percent of off-heap headroom, or an absolute size (`4G`, `512M`). | | `pinot.startree.datafusion.spill.compression` | `uncompressed` | Spill file compression: `uncompressed`, `lz4_frame`, or `zstd`. | | `pinot.startree.datafusion.spill.max.disk.size` | `100G` | Max spill data size before the query fails, or `unlimited`. | | `pinot.startree.datafusion.spill.dir` | OS temp dir | Absolute path override for spill files. | | `pinot.startree.datafusion.udaf.size.accounting.enabled` | `false` | Charge Java-heap-backed UDAF state (e.g. `DISTINCTCOUNT`, percentiles, sketches) toward the spill trigger. Off by default since the estimate is heap-based, not off-heap. | | `pinot.startree.datafusion.udaf.groupby.max.capacity` | `1,000,000` | Max simultaneously-live groups for a `GROUP BY` over a Java-bridged aggregate. The limit always applies, whether UDAF size accounting is enabled or not. | Malformed or out-of-range values for these settings fail broker/server startup rather than silently falling back to a default. Under memory pressure a query therefore meets one of three outcomes: | Outcome | Which work | What you see | | :- | :- | :- | | **Spills — slower, still finishes** | Final `GROUP BY` and `DISTINCT`, sorts, sort-merge joins | Added latency; spill counters on the operator | | **Cancelled** | Hash joins, nested-loop joins, cross joins, window aggregates | A resources-exhausted error. The query dies; the process and its neighbours keep serving | | **Never charged to the pool at all** | Everything inside the `LeafOperator`, and Java-bridged aggregate state — sketches, percentiles, `DISTINCTCOUNT` variants | No pool pressure. This state lives on the JVM heap, outside the pool, unless UDAF size accounting is enabled. A `GROUP BY` over it still stops at 1,000,000 live groups (`udaf.groupby.max.capacity`) | > [!WARNING] > > **Hash joins cannot spill.** The build side (the right input, as in Java MSQE) has to fit in the pool; above that the query is cancelled. If a large join hits this, use the `sort_merge` join hint instead of the default hash join, or raise the memory pool size. Note that sort-merge buys *completion*, not speed — reach for it deliberately, on the queries that would otherwise die. #### The leaf stage is not covered by any of this Everything above describes the native pool. The `LeafOperator` does not use it: it runs the original model, on the JVM heap, unbounded and without spilling. A large partial aggregate in the leaf can still cause GC pressure and still kill the query — in exactly the situations where Java MSQE kills it today. **Moving a query to this engine does not, by itself, fix a leaf-side memory problem.** Three things help: * **Streaming leaf aggregates**, which flush partial groups instead of holding them all, are on by default — this is why the problem is rarer than it used to be. * **Let the pull-up take the aggregate.** If it qualifies, the work moves into the native runtime and onto the bounded pool — see [Leaf-stage work and the columnar pull-up](#leaf-stage-work-and-the-columnar-pull-up). * **Skip the leaf aggregate entirely** with the standard Pinot hint, so grouping happens above the leaf: ```sql theme={null} SELECT /*+ aggOptions(is_skip_leaf_stage_group_by='true') */ ... ``` `is_skip_leaf_stage_group_by` is an existing Pinot aggregate hint and is not specific to this engine. ## Unsupported cases Not every query pattern runs natively yet. There are two very different behaviors, depending on *when* the engine detects the unsupported pattern. ### Transparent fallback to Java MSQE (query still succeeds) At planning time the broker checks the query plan for patterns the StarTree Execution Engine does not support. If it finds one, it silently ignores the `workerRuntime` option and runs the **whole query on the Java MSQE runtime**. The query succeeds and returns the same results as if you had never set the option. The response metadata reports the fallback with `queryEngineDowngraded` and its reason, and the broker writes an INFO log line (`Disabling DataFusion for this query: ...`). See [Monitoring](#monitoring). Patterns that currently fall back at planning time: * `INTERSECT ALL` and `EXCEPT ALL` / `MINUS ALL` set operations. * Aggregations that return arrays or collections (e.g. `ARRAY_AGG`). * Certain optimizer-hint-driven aggregation paths that have no native equivalent, such as `is_leaf_return_final_result` combined with `SUMPRECISION`, funnel functions, or raw sketch aggregations. * `UNNEST` when the column-pruning optimization is enabled for it. * `GROUPING SETS`, `ROLLUP`, and `CUBE`. * Window frames using `EXCLUDE`, and `LEAD`/`LAG ... IGNORE NULLS` alongside another call. * Unsupported column types, when they are visible in the logical plan (see below). If the planner finds any of these patterns, the whole query falls back to the classic Java engine — not just the affected stage. ### Query fails with an error (no fallback) Some patterns can only be detected after planning, while the plan is being prepared for native execution on the servers. At that point there is **no fallback**: the query fails and the client receives an error. To run such a query, set `workerRuntime = 'DEFAULT'` on it (see [Returning a query to Java MSQE](#returning-a-query-to-java-msqe)), or rewrite the query. Removing the option is not enough when the [broker-wide default](#broker-wide-default) is on: the broker sets it again, and the query runs on the StarTree Execution Engine. Cases that fail at runtime: * **Unsupported column types** reaching an intermediate stage when not detectable at planning time: `MAP`, `OBJECT`, and exotic array shapes. Supported types are INT, LONG, FLOAT, DOUBLE, BOOLEAN, TIMESTAMP, STRING, JSON, BIG\_DECIMAL, BYTES, and their common array variants. * **Unsupported join sub-cases**: LOOKUP joins outside INNER/LEFT/SEMI/ANTI, SEMI/ANTI lookup joins with non-equality conditions, and certain ASOF join match conditions. * **Functions outside the supported envelope** — see the function support section below for the specific lists of native, bridged, and unsupported functions. ### Function support **Aggregation functions.** The common aggregations run fully natively: `COUNT`, `SUM`, `MIN`, `MAX`, `AVG`, `DISTINCTCOUNT` (except over `BIG_DECIMAL`), `BOOL_AND`, `BOOL_OR`, and the statistics `VAR_POP`, `STDDEV_POP`, `COVAR_POP`, `COVAR_SAMP`, `VAR_SAMP`, and `STDDEV_SAMP`. Every other Pinot aggregation function — percentiles, the `DISTINCTCOUNT` sketch variants, `SUMPRECISION`, funnel functions, and so on — executes through a built-in Java bridge: results are identical to Java MSQE, at some extra cost when aggregation state crosses the native/Java boundary, and with a cap of 1,000,000 simultaneously-live groups (`udaf.groupby.max.capacity`). Aggregations that return arrays or collections (e.g. `ARRAY_AGG`) fall back to Java MSQE at planning time, as noted above. **Scalar functions.** Comparison, arithmetic, and logical operators, `CASE`, `IN`/`NOT IN`, and array construction run natively. All other registered Pinot scalar functions are automatically bridged to their Java implementation as long as their argument and return types are among INT, LONG, FLOAT, DOUBLE, BOOLEAN, STRING, BYTES, and TIMESTAMP (BIG\_DECIMAL and JSON results are carried as strings). A scalar function outside that envelope — for example one that takes or returns arrays or Java objects (theta-sketch set operations being a supported exception) — fails at runtime with an `Unsupported function` error. **Window functions.** The standard window functions run natively: `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `PERCENT_RANK`, `CUME_DIST`, `NTILE`, `LAG`, `LEAD`, `FIRST_VALUE`, `LAST_VALUE`, and `NTH_VALUE`, plus the native aggregations listed above used with an `OVER` clause. There is no Java bridge for window functions: a Pinot-specific window function outside this set fails at runtime. > [!NOTE] > > **`BIG_DECIMAL` precision.** `BIG_DECIMAL` columns are fully supported in comparisons, sorting, joins, `GROUP BY`, `DISTINCT`, and `MIN`/`MAX`/`SUM`/`AVG`/`COUNT`, using an order-preserving binary encoding. However, arithmetic (`+ - * / %`) and aggregation over `BIG_DECIMAL` are computed in double-precision floating point — matching Java MSQE behavior, not arbitrary-precision decimal arithmetic. If your workload depends on exact decimal arithmetic beyond double precision, verify results before relying on it. Custom Java UDFs/UDAFs over `BIG_DECIMAL` are not yet supported on the StarTree Execution Engine. ### Behavioral differences to be aware of Results are identical to Java MSQE apart from a short list of cases. Two of them are worth flagging to users **before** you enable the engine for their queries, because they change what comes back rather than only how it is computed. | Difference | What happens | What to do | | :- | :- | :- | | **Integer division** | `1 / int_col` truncates to an integer, following standard SQL. Java MSQE historically evaluated it as floating-point division. | Add an explicit `CAST(... AS DOUBLE)` where the legacy behavior is wanted. | | **`BIG_DECIMAL` scale** | A `BIG_DECIMAL` value that crosses a native stage comes back scale-stripped — `1.00` renders as `1`. The numeric value is preserved; only the trailing scale is lost. | Flag before enabling, if output formatting matters. | | **`numGroupsLimit`** | Java MSQE silently drops groups to stay inside a memory bound. Above the leaf, this engine applies no group limit: it returns groups that Java MSQE dropped, and pays for them in memory. An aggregate that stays in the Java leaf still applies `numGroupsLimit` per segment. | Nothing — but expect a query that was quietly truncating to start returning more groups, and more memory pressure with it. | | **Group-trim hints** | The hint runs but its limit is ignored. That limit was also acting as a memory bound, so a query that completes on Java MSQE can need much more memory here, and spill or be cancelled. | Flag before enabling. | | **`ASOF` with extra predicates** | Rejected outright rather than executed. Java MSQE silently drops the extra predicate and returns a wrong result. | Rewrite the query. The rejection is the correct behavior; do not report it as a defect. | * **Detailed stats require the streaming stats channel.** Per-stage and per-operator response stats for natively executed stages — including leaf counters such as `numSegmentsQueried` — are only delivered when MSQE's streaming stats channel is active. Enable it per query with `SET streamStats = true;` or cluster-wide with the broker configuration `pinot.broker.mse.stream.stats=true` (all servers must run a version that supports it). Without it, StarTree Execution Engine responses carry only broker-side stats. In addition, a few plan shapes (e.g. semi-join and dynamic-broadcast sub-plans) do not yet surface their leaf-stage stats. All of this affects observability, not results. ## Reading per-operator statistics Every native operator reports into the query's stage-stats tree, so a slow query can be read operator by operator instead of guessed at. These stats travel over MSQE's streaming stats channel — `SET streamStats = true;` if your build does not have it on by default. The tree uses thirteen native node types: `NATIVE_JOIN`, `NATIVE_AGGREGATE`, `NATIVE_SORT`, `NATIVE_WINDOW`, `NATIVE_FILTER`, `NATIVE_TRANSFORM`, `NATIVE_SET`, `NATIVE_LIMIT`, `NATIVE_LITERAL`, `NATIVE_MAILBOX_SEND`, `NATIVE_MAILBOX_RECEIVE`, `NATIVE_LEAF_BRIDGE`, and `NATIVE_ROOT`. ### Finding the operator that is actually slow Compare each node's wall time against its **self** time. Do not use `EXECUTION_TIME_MS` for this: it is cumulative, covering every operator below the node as well as the Java leaf. The broker emits self time as `selfExecutionTimeMs`. | Reading | Meaning | | :- | :- | | `WALL_TIME_MS` ≈ self time (`selfExecutionTimeMs`) | The operator is doing the work. If it is slow, it is the bottleneck. | | `WALL_TIME_MS` >> self time (`selfExecutionTimeMs`) | The operator mostly waited rather than worked. The wait counters below say in which direction. | Java MSQE has no per-operator wall time, so a starved operator and a busy one are harder to tell apart there. Its mailbox receive does report wait time. > [!NOTE] > > Node types are written in upper snake case, but the keys in the response JSON are camelCase — `WALL_TIME_MS` appears as `wallTimeMs`. Supporting counters: * `UPSTREAM_WAIT_MS` / `DOWNSTREAM_WAIT_MS` — these turn a wall-vs-self gap into a direction. A high `UPSTREAM_WAIT_MS` means the operator was starved by its input, so the bottleneck is below it in the tree; a high `DOWNSTREAM_WAIT_MS` means it was back-pressured by its consumer, so the bottleneck is above it. Reading the gap on its own and always walking downwards can lead away from the real cause. * `TIME_BUILDING_HASH_TABLE_MS` / `TIME_PROBING_HASH_TABLE_MS` — a join split into its two halves. Build-heavy means a large right input, which is the build side (as in Java MSQE). `BUILD_MEM_BYTES` shows its size. > [!NOTE] > > Two gaps to know about: `NUM_GROUPS` is always `0`, because the native runtime emits no such metric; and pipeline-breaker sub-plans (semi-join, dynamic broadcast) do not surface their leaf-stage stats. ### Did this query spill to disk? Per query, the clearest signal is on the operator itself. With `SET streamStats = true;`, `NATIVE_SORT` and `NATIVE_AGGREGATE` nodes carry `spillCount`, `spilledBytes`, and `spilledRows` in `stageStats`. Zero values are omitted from the JSON, so **the key being present means that operator spilled**, and its absence means it did not. Across the process, the native metrics endpoint carries the same story: `rust_operator_spills_total` and `rust_operator_spilled_bytes_total` for spill activity, and `rust_spill_pool_reserved_bytes` against `rust_spill_pool_size_bytes` for how full the pool is. Both failures surface as `ResourcesExhausted`, so the error type does not tell them apart — the message text does: * `...exceeded the allowable limit... max_temp_directory_size` — the query spilled and then hit the disk cap (`pinot.startree.datafusion.spill.max.disk.size`, default `100G`). All queries on a worker share that cap. * `Failed to allocate additional ...` — the pool had no memory left, which usually means an operator that cannot spill. > [!WARNING] > > **Two blind spots.** A sort-merge join spills through its inner sorts, but the join node itself carries no spill keys — that spill does not show up. And with `streamStats` off there are no per-operator statistics at all. ## Monitoring ### Did this query run natively? When the planner routes a query back to Java MSQE, the response metadata says so: ```json theme={null} "queryEngineDowngraded": true, "queryEngineDowngradeReason": "INTERSECT ALL" ``` Two broker meters count the two outcomes of that same planning decision: | Metric | What it counts | | :- | :- | | `pinot_broker_datafusionQueriesExecuted_Count` | Queries sent to the StarTree Execution Engine, including queries that then fail | | `pinot_broker_datafusionQueriesDegradedToMse_Count` | Queries that were routed back to Java MSQE at planning time | `Executed` counts the planner's choice, not the outcome: a query the engine accepted and then failed on still increments it. Read it next to your error rate rather than as a success count. Because both are read at the same decision point: ```text theme={null} coverage = executed / (executed + degraded) ``` > [!NOTE] > > **What this ratio does and does not measure.** It is the share of queries *that asked for the engine* and got it — not the share of your workload running natively. When you route per query, every multi-stage query you did not select runs on Java MSQE and appears in neither meter, so this can read 100% while most traffic is still on Java. Compare against `pinot_broker_multiStageQueriesGlobal_Count` for a workload-wide denominator. Queries that fail before planning — on parse, compile, or authorization — increment neither meter and sit outside the ratio entirely. ### Other signals * The native runtime exposes its own Prometheus metrics endpoint (`pinot.startree.rust.metrics.port`, default `8079`). All metrics are prefixed `rust_`: | Metric | Type | What it tells you | | - | - | - | | `rust_spill_pool_reserved_bytes` / `rust_spill_pool_size_bytes` | gauge | Native memory pool usage vs. configured size (labelled by runtime) | | `rust_operator_spills_total` / `rust_operator_spilled_bytes_total` | counter | Spill events and bytes written by sorts/aggregations under memory pressure | | `rust_operator_hash_build_seconds_total` | counter | Cumulative time building hash-join tables | | `rust_mailbox_compress_*` / `rust_mailbox_decompress_*` | counter | Inter-stage exchange traffic: batches per codec, bytes before/after compression, and time spent | | `rust_mailbox_fanout_payload_reuse_*` | counter | Broadcast exchange payload reuse | | `rust_mailbox_grpc_recv_tasks_active` / `_total` | gauge / counter | Active and total cross-server exchange receive tasks | | `rust_tokio_workers` | gauge | Network runtime worker threads | | In StarTree Cloud these metrics are also stored in Prometheus and can be read using Grafana as usual. | | | * In addition, brokers and servers export three engine-related meters through the standard Pinot metrics pipeline. `datafusionOpchainAllocatorLeakCloseCount` and `datafusionOpchainAllocatorLeakedBytes` count native memory reclaimed when a query's allocator is closed while it still holds buffers. `datafusionOpchainResourceLeakReclaimedCount` counts op-chains whose native resources a backstop reclaimed because the normal close never ran. All three are safety-net counters and must stay at zero; a non-zero rate means a query path leaks. * **To watch for a leak while it develops**, the live-count gauges are the earlier signal. They follow the standard Pinot naming scheme (`pinot___Value`) and are exported by both brokers and servers: | Gauge | What it counts | | - | - | | `pinot_broker_datafusionArrowLiveQueries_Value` / `pinot_server_datafusionArrowLiveQueries_Value` | Queries currently holding Arrow allocations | | `pinot_broker_datafusionOpchainLiveResources_Value` / `pinot_server_datafusionOpchainLiveResources_Value` | Live operator-chain resources | | `pinot_broker_datafusionNativeStreamHandlesLive_Value` / `pinot_server_datafusionNativeStreamHandlesLive_Value` | Open native stream handles | Any of these failing to return to zero when the cluster goes idle is the thing to investigate — the allocator-leak meters above only confirm a leak afterwards. > [!NOTE] > > There is no dedicated Grafana panel for these yet; add them to a dashboard by hand. Watching three questions covers most of an evaluation: is the engine actually being used (the executed/degraded ratio), is memory getting tight (`rust_spill_pool_reserved_bytes` climbing toward `rust_spill_pool_size_bytes`, and `rust_operator_spills_total` rising), and is anything leaking (the live gauges above). * Standard Pinot query response metadata is returned for all queries, with the stats-coverage caveat noted above. # Troubleshooting queries Source: https://docs.startree.ai/corecapabilities/query_data/troubleshooting Diagnose query timeouts, execution errors, multi-stage engine failures, and sudden latency or error-rate increases. This guide covers queries that fail, queries that time out, and queries that used to be fast and no longer are. > [!TIP] > > If StarTree support named an alert, such as `HighQueryTimeoutPercentage` or `QueryQuotaAlert`, look it up in [Query alerts](/corecapabilities/observability/alerts/queries) first. Each entry says what fired and what to check. > [!NOTE] > > If you have [Query Logger](/corecapabilities/query_data/advanced_operations/query-logger) enabled, start there — `system_query_log` records the slow and failed queries the cluster served (or every query, if configured), with timings, scan counts and error text. Most of the diagnosis below is a single SQL query against it. ## Start by classifying the failure The error text tells you which section to read. Match on the most specific string in the message, not the outermost one. | What you see | Most likely class | Go to | | - | - | - | | `BrokerTimeoutError`, `Timed out while planning query` | Planning or fan-out cost | [Timeouts](#timeouts) | | Timeout with no planning mention | Execution cost or contention | [Timeouts](#timeouts) | | `QueryExecutionError` with a nested cause | Data, function, or type problem | [Execution errors](#execution-errors) | | `Unsupported function`, or an error naming an operator | Engine capability | [Multi-stage engine](#multi-stage-engine-issues) | | HTTP 5xx, or a sudden rise in failures across many queries | Cluster-level, not query-level | [Error-rate spikes](#error-rate-spikes) | | Results returned, but wrong or inconsistent | Availability or data problem | [Inconsistent results](#inconsistent-or-partial-results) | | Quota or rate-limit rejection | Quota configuration | [Query Quotas](/corecapabilities/query_data/advanced_operations/query-quotas) | ## Timeouts A timeout means the query did not finish inside its budget. The useful question is *where* the time went. **1. Separate planning from execution** An error mentioning **planning** means the query did not get as far as reading data. That points at query shape — very large `IN` lists, many joins, a wide fan-out across segments, or a complex multi-stage plan — rather than at data volume. A timeout without a planning mention means execution was too slow, and the scan counts below will tell you why. **2. Look at what the query scanned** In `system_query_log`, the scan fields distinguish an expensive query from a slow cluster: ```sql theme={null} SELECT logTimestampMs, tableName, timeUsedMs, numDocsScanned, numEntriesScannedInFilter, numEntriesScannedPostFilter, numSegmentsQueried, numSegmentsProcessed, numSegmentsMatched, query FROM system_query_log WHERE logTimestampMs > now() - 3600000 AND timeUsedMs > 1000 ORDER BY timeUsedMs DESC LIMIT 20 ``` Read the result like this: * **High `numEntriesScannedInFilter`** — the filter is not being served by an index. This is the most common cause of a slow query, and the most fixable. * **High `numEntriesScannedPostFilter`** — the filter is fine, but the projection is reading a lot of columns or rows. Select fewer columns, or add a `LIMIT`. * **`numSegmentsProcessed` close to `numSegmentsQueried`** — segment pruning is not working. Check that the query filters on the time column, and on the partition column if the table is partitioned. * **All counts low but `timeUsedMs` high** — the query is not the problem. The cluster is. Go to [Resource pressure](/corecapabilities/cluster-operations/troubleshooting). **3. Check whether an index is actually being used** A high `numEntriesScannedInFilter` on a column you have indexed means the index is not being applied. Common reasons: * The index was added but segments have not been reloaded since. Check the `TABLE_SEGMENTS_RELOAD_CHECK` on the Health Dashboard, which will tell you how many segments are pending reload. * The filter predicate does not match a form the index can serve — for example a function applied to the indexed column, or a type mismatch that forces a cast. * The index exists on some segments and not others, because it was added partway through the table's life. [Query Analyzer](/corecapabilities/query_data/query-analyzer) will tell you which of these applies and what to change — see [Reading Query Analyzer results](/corecapabilities/ai/query-analyzer/reading-results) for how to interpret its output. Before triggering a reload, [Reload dry run](/corecapabilities/manage-data/reload-dry-run) shows you what a reload would actually change, and its [Troubleshooting](/corecapabilities/manage-data/reload-dry-run#troubleshooting) section covers dry runs that fail or return nothing. **4. Confirm it is this query and not the load** Run the same query when the cluster is quiet. If it is fast then and slow under load, you have a contention problem rather than a query problem — see [Resource pressure](/corecapabilities/cluster-operations/troubleshooting). Scheduler wait time rising while execution time stays flat is the signature. ## Execution errors An execution error means the query started and then failed. The cause is almost always in the innermost exception, not the outermost message. > [!WARNING] > > Read the full error, including nested causes. A generic outer message with a specific inner cause is common, and the inner cause usually names the column, value, or type at fault. Work through these in order: 1. **Does it fail for all inputs, or only some?** Narrow the time filter until the query succeeds. The boundary tells you which data is involved, and that is usually the whole answer — a null in an unexpected column, a value that overflows its type, a malformed JSON document. 2. **Did the schema or a type change?** A column whose type changed will fail on segments written under the old type until those segments are rebuilt. Check your schema history against the age of the segments the query touches. 3. **Is a specific function involved?** Remove functions from the projection one at a time. Errors during result serialization — particularly with sketch and aggregate types — usually point at a function that cannot handle the cardinality or the value range it is being given. 4. **Is it one replica?** If the same query fails intermittently with the same error, one replica may hold a bad segment. The `TABLE_SEGMENT_AVAILABILITY_CHECK` on the Health Dashboard will show unavailable segments. A segment reset or reload is the usual remedy, and both are available from the Health Dashboard where the check supports them. ## Multi-stage engine issues The multi-stage engine (MSE) supports joins and more complex plans than the single-stage engine, and has a different capability surface. See [Multi-stage query engine](/corecapabilities/query_data/query_languages/msqe) for what it supports. When an MSE query fails: **1. Try the same query on the single-stage engine** If it works there, you have an MSE-specific issue and you have your workaround while it is investigated. If it fails on both, the problem is not MSE-specific — treat it as an [execution error](#execution-errors). **2. Check for unsupported constructs** An `Unsupported function` error naming a specific function or operator means the plan reached a construct the engine cannot handle in that position. This is often reachable by rewriting — extracting a subquery, materializing an intermediate result, or replacing the construct with an equivalent. **3. Look at per-stage timings for slow MSE queries** `system_query_log` records `stageStats` as JSON, giving per-operator statistics. One stage dominating total time tells you which operator to attack — usually a join whose build side is too large, or a shuffle moving more data than it needs to. **4. Check partition-aware pruning on partitioned tables** If your table is partitioned and the query filters on the partition column, the broker should be able to prune. High `numSegmentsProcessed` despite a partition filter means pruning is not being applied — worth raising, with the query and the table config attached. ## Error-rate spikes When many queries start failing at once, the cause is rarely any individual query. **1. Establish the shape of the spike** With Query Logger enabled: ```sql theme={null} SELECT tableName, COUNT(*) AS total, SUM(CASE WHEN numExceptions > 0 THEN 1 ELSE 0 END) AS failures, SUM(CASE WHEN partialResult THEN 1 ELSE 0 END) AS partial FROM system_query_log WHERE logTimestampMs > now() - 3600000 GROUP BY tableName ORDER BY failures DESC ``` Failures concentrated on one table point at that table. Failures spread evenly across tables point at the cluster. **2. Check segment availability** Server restarts, failed reloads and stuck rebalances all remove segments from service temporarily. `TABLE_SEGMENT_AVAILABILITY_CHECK` and `IDEAL_STATE_EV_MISMATCH_CHECK` on the Health Dashboard both surface this. A mismatch between ideal state and external view means the cluster has not converged — that is usually transient, and worth escalating if it persists. **3. Check whether you are hitting a quota** Rejections due to query quotas look like failures to the application. See [Query Quotas](/corecapabilities/query_data/advanced_operations/query-quotas) for how quotas are configured and what a rejection looks like. **4. Check resource pressure** Servers under memory or CPU pressure fail queries rather than serving them slowly, and garbage-collection pauses show up as timeouts clustered in time. See [Resource pressure](/corecapabilities/cluster-operations/troubleshooting). ## Inconsistent or partial results A query that returns results but the wrong ones is a different problem from a query that fails. * **Check `partialResult` and the limit flags.** `system_query_log` records `partialResult`, plus `numGroupsLimitReached`, `groupsTrimmed`, `maxRowsInJoinReached`, `maxRowsInWindowReached` and `maxRowsInDistinctReached`. Any of these being true means the query hit a guardrail and the result was trimmed — the numbers are not wrong so much as incomplete. * **Check segment availability.** A query that silently reads fewer segments than it should returns a smaller answer. Compare `numSegmentsQueried` between a good and a bad execution. * **On hybrid tables, check the time boundary.** Real-time and offline halves are separated by a time boundary, and rows can appear missing or doubled if the boundary and the data do not line up. See [Hybrid tables](/corecapabilities/manage-data/hybrid-tables). * **On upsert tables, check replica consistency.** Upsert tables can serve different answers from different replicas if their state diverges. See the [Upsert operations guide](/corecapabilities/manage-data/upsert-operations-guide). * **Different results from different tools?** Confirm both are querying the same table type — a query against the base table name behaves differently from one against an explicit `_OFFLINE` or `_REALTIME` suffix on a hybrid table. ## Feature-specific guidance If the problem is specific to a feature — geospatial queries, star-tree indexes, materialized views, the Elasticsearch gateway, workload isolation — see [Troubleshooting by feature](/corecapabilities/observability/troubleshooting-by-feature#queries-indexes-and-performance), which indexes the troubleshooting sections on those pages. **Querying an external table?** [External table troubleshooting](/corecapabilities/external-table/troubleshooting#queries) covers timeouts and `servers not responded`, why a first query on a column is slow while later ones are fast, and how to tune a large scan. ## Escalating If you have worked through this guide, include in your ticket: the full query, the complete nested error, the `requestId`, the engine used, whether it reproduces, and the relevant `system_query_log` row. See [what to collect](/corecapabilities/observability/troubleshooting#before-you-open-a-support-ticket). # Actions Source: https://docs.startree.ai/corecapabilities/security/actions There are several commonly used actions that can be specified in a policy. Understanding actions will help you effectively define and manage access control rules within your environment. For the full list of available actions, see [Actions.java](https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/auth/Actions.java). StarTree Cloud adds a few actions of its own on top of the upstream Apache Pinot list — for example `GetTaskPlan`, `AbortTaskPlan`, and `GetTaskEstimatedTime`, which govern [Task Observability](/corecapabilities/observability/minion-task-runs); these are included in the tables below. ## Cluster | Action name | Description | | - | - | | CancelQuery | Cancel a query that's currently running | | CleanupTask | Clean up finished tasks from the task queue for the given task type (COMPLETED, FAILED) | | CommitSegment | Commit a segment | | CreateInstance | Add a CONTROLLER, BROKER, SERVER, or MINION in the Pinot cluster | | CreateTask | Create a new task | | CreateTenant | Create a new tenant in your cluster | | CreateUser | Create a new user in Pinot | | DebugResourceUsage | View the resource usage of a cluster | | DebugTask | Fetch debug information for tasks | | DeleteClusterConfig | Delete cluster configuration | | DeleteDatabase | Delete a Database in your Pinot cluster | | DeleteInstance | Delete a CONTROLLER, BROKER, SERVER, or MINION in the Pinot cluster | | DeleteResponseStore | Delete the response store used to support cursors | | DeleteTask | Delete a task | | DeleteTenant | Delete a Pinot tenant | | DeleteUser | Delete a user in Pinot | | DeleteZnode | Delete a ZooKeeper znode | | EstimateUpsertMemory | Get the estimated memory usage for an upsert table | | ExecuteTask | Run a task | | GetAppConfig | Retrieve the configuration of a broker or server | | GetApplicationQueryQuota | Get the query per second (QPS) quota defined for broker or server | | GetAuth | Retrieve information about the authentication workflow | | GetBroker | Retrieve the configuration details of a broker | | GetClusterConfig | Retrieve the configuration details of a cluster | | GetDatabase | Retrieve the details of a database in your Pinot cluster | | GetDatabaseQueryQuota | Get the current database query quota | | GetDatabaseQuota | Get a database's quota configuration | | GetForceCommitStatus | Get the status of a force commit operation | | GetHealth | Get the health of a cluster and all related Pinot entities | | GetInstance | Retreive the tags for current tenant instance | | GetInstancePartitions | Get the instance partitions of a tenant | | GetLogFile | Fetch a log file | | GetLogger | Retrieve the configuration of a logger | | GetRebalanceStatus | Get a details status of a rebalance operation | | GetResponseStore | Get the response store used to support cursors | | GetRunningQuery | Get information on running queries from all brokers | | GetSchedulerInfo | Retrieve cron scheduler information | | GetSchema | Retrieve the schema of a table | | GetSchemasInfo | Retrieve information of schemas | | GetSegment | Retrieve the list of segments for a table | | GetSegmentReloadStatus | Retreive the reload status of a segment | | GetServerRoutingStats | Retireve the server routing stats | | GetTable | Retrieve the list of all tables in a cluster | | GetTableConfig | Retrieve the configuration of a table | | GetTask | Retrieve configurations and various stats of tasks | | GetTaskCount | Get the number of tasks of a specific task type | | GetTenant | Retrieve the configuration of a tenant | | GetUser | Retrieve details of Pinot users | | GetVersion | Fetch the version of a specific component | | GetZnode | Get the content of a ZooKeeper znode | | IngestFile | Ingest a file into an offline table | | RebalanceTenantTables | Rebalance the tables in a Pinot tenant | | RecommendConfig | Get the recommended configuration for tables \[ref ] | | ResetSegment | Reset a segment by disabling it, waiting for the external view to stabilize, and then enabling it again | | ResumeTask | Resume a stopped task | | StopTask | Stop a running task | | UpdateApplicationQuota | Set or update the queries per second quota for an application \[ref ] | | UpdateBrokerResource | Update the tables served by a specific broker instance | | UpdateClusterConfig | Update the configuration of a cluster | | UpdateDatabaseQuota | Update the max queries per second (QPS) quota of a database | | UpdateInstance | Update the properties of a CONTROLLER, BROKER, SERVER, or MINION in a Pinot cluster | | UpdateInstancePartitions | Update an instance partition of a server | | UpdateLogger | Set the log level of a logger | | UpdateQPS | Enable or disable the query rate limiting for a broker instance | | UpdateTaskQueue | Update a task queue | | UpdateTenant | Update the configuration of a tenant | | UpdateTenantMetadata | Update the metadata of a tenant | | UpdateTimeInterval | Update the start and end time of segments | | UpdateUser | Update the details of a Pinot user | | UpdateZnode | Update the configuration of a ZooKeeper znode | | UploadSegment | Upload a segment | ## Table | Action name | Description | | - | - | | AbortTaskPlan | Abort an active task plan (StarTree-specific) | | BuildRouting | Build the routing for a table | | CancelRebalance | Cancel a rebalance task | | CreateDedupSnapshot | Create a dedupe snapshot | | CreateSchema | Create a schema for a table | | CreateTable | Create a new table | | CreateUpsertSnapshot | Create an upsert snapshot | | DeleteDedupSnapshot | Delete a dedupe snapshot | | DeleteMinionTaskMetadata | Delete the metadata of a minion task | | DeleteRouting | Remove the routing for a table | | DeleteSchema | Delete the schema of a table | | DeleteSegment | Delete a segment | | DeleteTable | Delete a table | | DeleteTimeBoundary | Delete the query time boundary of a hybrid table | | DeleteUpsertSnapshot | Delete an upsert snapshot | | DisableTable | Disable a table | | DownloadSegment | Download a segment | | EnableTable | Enable a table | | EndDataIngest | End data ingestion for a table | | ForceCommit | Force commit the current segments in consuming state and restart consumption | | GetBroker | List brokers for a given table | | GetConsumingSegments | Returns the state of all consuming segments | | GetControllerJobs | Get the list of controller jobs for a table | | GetDebugInfo | Get debug information for a table | | GetDedupSnapshot | Retrieve dedup snapshot information | | GetExternalView | Get the external view of a table | | GetIdealState | Get the ideal state of a table | | GetInstance | List the instances of a table | | GetMetadata | Retrieve the metadata of a table | | GetPauseStatus | Get the pause status of a realtime table, along with a list of consuming segments | | GetRoutingTable | Get the routing for a table | | GetSchedulerJobDetails | Get the job details for table tasks | | GetSchema | Retrieve the schema of a table | | GetSegment | Retrieve segments | | GetSegmentLineage | List the lineage of a segment | | GetSegmentMap | Get the CRC of a segment | | GetSegmentStatus | Get the status of a segment | | GetServerMap | Get a list of the segments hosted by the server | | GetSize | Get table size information | | GetState | Get the state of a table (enabled or disabled) | | GetStorageTier | Get storage tier for segments or a table | | GetTableConfig | Gets a configuration for specific table | | GetTableConfigs | Lists the configurations of a table | | GetTableQueryQuota | Get the query per second quota for a table | | GetTaskEstimatedTime | Get the estimated run time for a task | | GetTaskPlan | View task plans and their execution details for a table (StarTree-specific) | | GetTimeBoundary | Get the time boundary information for a table | | GetUpsertSnapshot | Get upsert snapshot information | | IngestRecord | Insert records into a table | | PauseConsumption | Pause the consumption for a realtime table | | Query | Query a table | | RebalanceTable | Rebalance a table | | RebuildBrokerResource | Rebuild a broker | | RefreshRouting | Refresh the routing for a table | | ReloadSegment | Reload a segment | | ReplaceSegment | Replace a segment | | RestreamTable | Restream a table | | ResumeConsumption | Resume the consumption for a realtime table | | StartDataIngest | Start data ingestion | | UpdateConsistentPush | Update consistentPush state | | UpdateInstancePartitions | Update the partitions of a table | | UpdateSchema | Update the configuration of a schema | | UpdateTableConfig | Update the configuration of a table | | UpdateTableConfigs | Update the configuration of multiple tables | | UploadSegment | Upload a segment | | ValidateSchema | Validate the configuration of a schema | | ValidateTableConfigs | Validate the configuration of a table | ### Query | Action name | Description | | - | - | | DeleteQueryStore | Delete a query response store | | GetQueryStore | Retrieve information about a query response store | # Pinot Audit Configuration Guide Source: https://docs.startree.ai/corecapabilities/security/audit/pinot-audit-logs > [!NOTE] > > Audit logging support is a new capability available in Apache Pinot. This is available in StarTree Cloud starting 0.11.1 release ([Release notes](https://docs.startree.ai/reference/startree-release-notes/0.11.1#pinot-audit-logs)) for auditing Controller Logs. Broker audit logs are supported starting StarTree Cloud version 0.12.0. > [!WARNING] > > **Starting StarTree Cloud 0.16.0, audit logging is seeded on by default** — including request-payload capture (up to 64 KB per request) — on both new and existing clusters, on the controller's next restart after upgrade. See [0.16.0 defaults and how to opt out](#0160-defaults-and-how-to-opt-out) before upgrading if you rely on audit logging being off, or want to control the specific defaults yourself. ## Overview Apache Pinot provides a comprehensive audit logging framework that captures API requests across different components (Controller, Broker). The audit system logs structured JSON events containing request details, user information, and payload data for security monitoring and compliance purposes. ## Log Management By default, audit logs are stored in the same bucket as the Pinot deep store under this path: `audit-logs/`. Default retention is 30 days. ## Configuration Properties All audit configurations follow the pattern: `pinot.audit.{component}.{property}` Where component can be: * controller - For Pinot Controller instances * broker - For Pinot Broker instances The different properties available are as follows: ### 1. enabled **Type:** Boolean\ **Default:** false (OSS) — **seeded `true` starting StarTree Cloud 0.16.0**, see [0.16.0 defaults](#0160-defaults-and-how-to-opt-out)\ **Description:** Master switch to enable/disable audit logging for the component.\ **Example:** ```shellscript theme={null} pinot.audit.controller.enabled=true ``` ### 2. capture.request.payload.enabled **Type:** Boolean\ **Default:** false (OSS) — **seeded `true` starting StarTree Cloud 0.16.0**\ **Description:** Enables capturing of request body payloads in audit logs. When enabled, POST/PUT request bodies will be logged (subject to size limits).\ **Example** ```shellscript theme={null} pinot.audit.controller.capture.request.payload.enabled=true ``` ### 3. capture.request.headers **Type:** String (comma-separated list)\ **Default:** "" (empty) (OSS) — **seeded `Content-Length,x-forwarded-for,x-real-ip` starting StarTree Cloud 0.16.0**\ **Description:** Comma-separated list of HTTP headers to capture in audit logs. Headers are matched case-insensitively.\ **Example** ```shellscript theme={null} pinot.audit.controller.capture.request.headers=Content-Type,X-Request-Id,Authorization ``` ### 4. request.payload.size.max.bytes **Type:** Integer\ \*\*Default:\*\*8192 (8KB) (OSS) — **seeded 65536 (64KB, the hard maximum) starting StarTree Cloud 0.16.0**\ \*\*Maximum:\*\*65536 (64KB - hard limit)\ **Description:** Maximum size of request payload to capture in bytes. Payloads exceeding this limit will be truncated with a marker.\ **Example** ```shellscript theme={null} pinot.audit.controller.request.payload.size.max.bytes=20480 ``` ### 5. url.filter.exclude.patterns **Type:** String (comma-separated list) **Default:** "" (empty) (OSS) — **seeded `health` starting StarTree Cloud 0.16.0** **Description:** Comma-separated list of URL patterns to exclude from audit logging. Useful for excluding health checks, metrics endpoints, etc. Exclusion patterns have priority over inclusion patterns. **Note:** * Since this is a comma-separated list, individual patterns cannot contain commas. Use multiple separate patterns instead of glob alternatives like api,v1. * Patterns should NOT start with '/'. The patterns are matched against the URL path without the leading slash. **Example** ```shellscript theme={null} pinot.audit.controller.url.filter.exclude.patterns=health,metrics,swagger.* ``` ### 6. url.filter.include.patterns **Type:** String (comma-separated list) **Default:** "" (empty) **Description:** Comma-separated list of URL patterns to include in audit logging. When specified, only URLs matching these patterns will be audited (unless excluded). If not specified, all URLs are audited by default (except excluded ones). **Note:** * Since this is a comma-separated list, individual patterns cannot contain commas. Use multiple separate patterns instead of glob alternatives like api,v1. * Patterns should NOT start with '/'. The patterns are matched against the URL path without the leading slash. **Pattern Priority:** 1. **Exclusion always wins**: URLs matching exclude patterns are never audited, even if they match include patterns 2. **Include acts as allowlist**: When include patterns are defined, only matching URLs are audited 3. **Default behavior**: Without include patterns, all non-excluded URLs are audited **Example:** ```shellscript theme={null} # Only audit table and schema operations, but never audit health checks pinot.audit.controller.url.filter.include.patterns=tables/**,schemas/**,segments/** pinot.audit.controller.url.filter.exclude.patterns=health,metrics ``` ### 7. userid.header **Type:** String\ **Default:** "" (empty)\ **Description:** HTTP header name containing the user identifier. Takes precedence over JWT claim if both are configured.\ **Example** ```shellscript theme={null} pinot.audit.controller.userid.header=X-User-Id ``` ### 8. userid.jwt.claim **Type:** String\ **Default:** "" (empty)\ **Description:** JWT claim name to extract user identity from Authorization Bearer tokens. Used when userid.header is not found.\ **Example** ```shellscript theme={null} pinot.audit.controller.userid.jwt.claim=sub ``` ### 9. capture.response.enabled **Type:** Boolean\ **Default:** false\ **Description:** Enables capturing of HTTP response information in audit logs. When enabled, response status codes, request IDs, and response duration will be logged along with request details.\ **Example** ```shellscript theme={null} pinot.audit.controller.capture.response.enabled=true ``` ## Audit Event Structure Audit events are logged as structured JSON with the following fields: ```json theme={null} { "timestamp": "2024-01-15T10:30:45.123Z", "service_id": "Controller_192.168.1.10_9000", "endpoint": "/schemas", "method": "POST", "origin_ip_address": "10.0.0.5", "user_id": { "principal": "john.doe@example.com" }, "request_id": "req-abc-123-456", "response_code": 201, "duration_ms": 1245, "request": { "query_params": { "override": "true" }, "headers": { "Content-Type": "application/json", "X-Request-Id": "abc-123" }, "body": "{\"schemaName\":\"myTable\",\"schema\":{...}}", "error": null } } ``` ## URL Pattern Syntax Both include and exclude patterns support powerful pattern matching using glob and regex patterns: ### Glob Patterns (default) * \* - matches any characters within a path segment * \*\* - matches any characters across multiple path segments * ? - matches a single character * \[abc] - matches any character in the set * \[a-z] - matches any character in the range * \[!abc] - matches any character NOT in the set **Important Note:** Patterns containing commas (like api,v1,v2) are NOT supported because the configuration uses comma-separated lists. Each pattern must be specified separately. **Examples:** * health - exact match for /health * tables/\* - matches /tables/myTable but not /tables/myTable/segments * tables/\*\* - matches /tables/myTable and /tables/myTable/segments * schemas/\* - matches /schemas/mySchema * segments/\*/metadata - matches /segments/mySegment/metadata **Note:** Patterns should not start with '/' as they are matched against paths without the leading slash. ### Regex Patterns Prefix with regex: to use regular expressions: **Important Note:** Avoid using commas in regex patterns as they will be interpreted as list separators. Use character classes or alternation without commas instead. **Examples:** * regex:tables/\[a-zA-Z0-9\_]+\$ - matches table names with alphanumeric characters * regex:^health(check)?\$ - matches only "health" or "healthcheck" * regex:segments/.\*\\.tar\\.gz\$ - matches segment tar.gz files * regex:tables/(realtime|offline)/.\* - matches realtime or offline table operations ## Complete Configuration Examples ### Controller with Full Audit Logging ```shellscript theme={null} # Enable audit logging pinot.audit.controller.enabled=true # Capture request payloads pinot.audit.controller.capture.request.payload.enabled=true # Capture response information pinot.audit.controller.capture.response.enabled=true # Capture specific headers pinot.audit.controller.capture.request.headers=Content-Type,X-Request-Id,Authorization,X-Correlation-Id # Set payload size limit to 20KB pinot.audit.controller.request.payload.size.max.bytes=20480 # Exclude health and metrics endpoints pinot.audit.controller.url.filter.exclude.patterns=health,metrics,swagger.*,api-docs.* # User identification from header pinot.audit.controller.userid.header=X-User-Email # Fallback to JWT claim if header not found pinot.audit.controller.userid.jwt.claim=email ``` ### Broker with Security-Focused Configuration ```shellscript theme={null} # Enable audit logging pinot.audit.broker.enabled=true # Capture payloads for query auditing pinot.audit.broker.capture.request.payload.enabled=true # Capture response information for query performance tracking pinot.audit.broker.capture.response.enabled=true # Capture auth headers pinot.audit.broker.capture.request.headers=Authorization,X-User-Id # Limit payload to 16KB pinot.audit.broker.request.payload.size.max.bytes=16384 # User identification from JWT pinot.audit.broker.userid.jwt.claim=sub ``` ### Controller with Include Pattern Allowlist ```shellscript theme={null} # Enable audit logging pinot.audit.controller.enabled=true # Only audit specific API endpoints (allowlist approach) pinot.audit.controller.url.filter.include.patterns=schemas/**,tables/**,segments/**,tenants/** # But always exclude health, status and size checks even if they match include patterns pinot.audit.controller.url.filter.exclude.patterns=health,tables/*/size,tables/*/stats,segments/*/metadata # Capture request details pinot.audit.controller.capture.request.payload.enabled=true pinot.audit.controller.capture.response.enabled=true pinot.audit.controller.capture.request.headers=Content-Type,Authorization ``` ## Audit Metrics The audit system includes built-in metrics collection to monitor performance and health of the audit logging infrastructure. These metrics are automatically exposed through Pinot's metrics system and can be monitored via JMX or other metrics collection systems. ### Available Metrics The audit system tracks the following metrics: * **Audit processing time**: Duration of audit event processing * **Audit event count**: Number of audit events processed * **Audit errors**: Count of errors during audit processing * **Request processing**: Time spent processing requests for audit logging ### Metrics Integration Audit metrics are automatically integrated with: * **Controller Metrics**: When audit is enabled on Controller instances * **Broker Metrics**: When audit is enabled on Broker instances No additional configuration is required - metrics are automatically collected when audit logging is enabled for any component. ## Dynamic Configuration Updates The audit configuration supports dynamic updates through Pinot's cluster configuration mechanism. Changes to audit settings are applied without requiring service restarts. ## Log Output Configuration Audit logs are written using SLF4J to the logger named org.apache.pinot.audit. Configure your logging framework (e.g., Log4j2) to direct these logs appropriately: ```java theme={null} ``` ## Best Practices ### Security Considerations 1. **Sensitive Data**: Be cautious when enabling payload capture as it may log sensitive information 2. **Header Selection**: Only capture necessary headers to minimize exposure of authentication tokens 3. **Payload Size**: Keep payload limits reasonable to prevent logging excessive data 4. **Log Storage**: Ensure audit logs are stored securely with appropriate access controls 5. **Log Rotation**: Implement proper log rotation to manage disk space ### Performance Considerations 1. **Selective Enabling**: Only enable audit logging for components that require it 2. **URL Filtering**: Exclude high-frequency endpoints like health checks and metrics 3. **Payload Capture**: Disable payload capture for high-throughput services if not required 4. **Header Filtering**: Limit captured headers to only what's necessary ### Monitoring and Alerting 1. Set up monitoring for audit log volume and errors 2. Alert on authentication failures or unauthorized access patterns 3. Regularly review audit logs for suspicious activity 4. Integrate with SIEM systems for centralized security monitoring ## Troubleshooting ### Audit Logs Not Appearing 1. Verify enabled is set to true for the component 2. Check the logging configuration for org.apache.pinot.audit logger 3. Ensure the service has reloaded configuration after changes ### Payload Truncation If payloads are being truncated: 1. Check the request.payload.size.max.bytes setting 2. Remember the hard limit is 64KB (65536 bytes) 3. Look for ...\[truncated] marker in logs ### User Identity Not Captured 1. Verify userid.header or userid.jwt.claim is correctly configured 2. Ensure the specified header is present in requests 3. For JWT claims, verify the Authorization header contains a valid Bearer token ## Migration Notes When migrating from older Pinot versions: * The configuration prefix has changed from a global pinot.audit to per-component prefixes * Update your configuration to use pinot.audit.controller or [pinot.audit.broker](http://pinot.audit.broker) * Each component now maintains its own audit configuration independently # Custom Policy Configuration Source: https://docs.startree.ai/corecapabilities/security/custom-policies Define and implement custom RBAC policies in StarTree Cloud by configuring granular permissions using StarTree Resource Names (SRNs), supporting over 150 distinct actions for precise access control across environments, clusters, and tables. A policy configuration is a collection of one or more statement objects, each containing: * A description of the policy statement. * One or more [actions](/corecapabilities/security/actions) that the policy statement applies to. * The resources that the policy statement is applied to, identified using the [SRN format](#identifying-resources-using-srns). * The effect of the policy statement (`allow` or `deny`). Here’s an example of a policy that allows querying a table called “myTable”: ```json theme={null} { "version": "v1", "statements": [ { "description": "A sample policy - allow query on a table called 'myTable', deny everything else", "resources": [ "srn2:cluster#*:table#myTable" ], "effect": "allow", "actions": [ "query" ] } ] } ``` > [!NOTE] > > * Within a policy, **deny statements take precedence** regardless of their location in the policy. > * If actions aren’t explicitly specified, then all actions are implicitly included in the statement (the equivalent of writing `“actions”: “*”`). > * If the effect isn’t explicitly specified, then `deny` is implicitly applied (the equivalent of writing `“effect”: “deny”`). ## Identifying resources using SRNs A StarTree Resource Name (SRN) is a unique identifier for any resource within your StarTree environment. It follows a standardized format that makes it easy to understand and use. ### SRNv2 format The basic format of an SRNv2 is: `srn2:#`. For example: srn2:table#myTable identifies the table called “myTable”. Some resources may have a more complex format to represent hierarchical relationships. ### Constructing an SRNv2 string Each SRNv2 string starts with the prefix `srn2:`, followed by the resource type and then the resource identifier. A simple SRNv2 would follow this format: `srn2:resource-type#resource-id` Since resources in StarTree are hierarchical, the SRNv2 string can include any number of levels of hierarchy. An SRNv2 with two levels of hierarchy would look like this: `srn2:resource-type#resource-id:sub-resource-type#sub-resource-id` Commonly used resource types include: environment, cluster, workspace, and table. Hierarchy levels can be omitted. For example, you can write a policy that applies to all tables called “myTable” across the entire environment by omitting the cluster resource type. Alternatively, you can explicitly include all clusters by adding `cluster#*` in the SRN. ## Examples of Policy Configurations for Common Functional Roles With over 150 distinct [actions](/corecapabilities/security/actions), Apache Pinot and StarTree allow for very granular custom policy creation. This section provides guidance on mapping common functional roles and application usage to the specific permissions required in Apache Pinot. This information serves as a guide for RBAC administrators to create and maintain policies and roles efficiently. ### System Administrator A System Administrator can perform all actions on any resource in the environment. > [!NOTE] > > The `system-admin` policy is predefined in StarTree, so you won't need to define it yourself. We're just showing it here as an example. ```json theme={null} { "version": "v1", "statements": [ { "description": "Can do everything in an environment", "resources": "*", "effect": "allow", "actions": "*" } ] } ``` ### Cluster Administrator A cluster administrator can perform all actions in a Pinot cluster ```json theme={null} { "version": "v1", "statements": [ { "description": "Allow all actions on the 'Pinot' cluster, and all tables in the cluster", "resources": [ "srn2:cluster#pinot", "srn2:cluster#pinot:table#*" ], "effect": "allow", "actions": "*" } ] } ``` The same policy can also be written like this: ```json theme={null} { "version": "v1", "statements": [ { "description": "Allow all actions on the 'Pinot' cluster, including all resources in the cluster.", "resources": "srn2:cluster#pinot:*#*", "effect": "allow", "actions": "*" } ] } ``` ### Table Admin A table admin can perform any action on a table. ```json theme={null} { "version": "v1", "statements": [ { "resources": "srn2:cluster#*:table#*", "effect": "allow", "actions": "*" }, { "resources": "srn2:cluster#*", "effect": "allow", "actions": [ "Get*", "*Task", "RecommendConfig" ] } ] } ``` ### Table Reader A table reader policy allows any read-only operation on a given table using the Query Console UI. Unlike the [QueryTable](../security/custom-policies#query-table) policy, this policy includes additional permissions that allow the UI to retrieve additional helpful information about the table. ```json theme={null} { "version": "v1", "statements": [ { "description": "Allow any read-only operation on a Pinot table", "resources": "srn2:cluster#pinot:table#myTable", "effect": "allow", "actions": [ "GetState", "ValidateTableConfigs", "GetTableConfig", "GetSchema", "ValidateSchema", "Query" ] }, { "resources": "srn2:cluster#pinot", "effect": "allow", "actions": [ "GetTable" ] } ] } ``` ### **Query Table** This policy allows querying a given table. ``` { "version": "v1", "statements": [ { "description": "Allow querying a Pinot table", "resources": "srn2:cluster#pinot:table#myTable", "effect": "allow", "actions": "Query" } ] }RBAC Admin ``` ### RBAC Admin This policy allows viewing and making changes to the RBAC policies, roles, assignments, and API tokens. ```json theme={null} { "version": "v1", "statements": [ { "description": "Allow all actions on policies, role, and service tokens", "resources": [ "srn2:policy#*", "srn2:role#*", "srn2:service-token#*" ] "effect": "allow", "actions": [ "delete*", "create*", "update*", "detach*", "attach*" ] } ] } ``` ### Row-Level Access Control Policies > [!WARNING] > > Row-Level Access Control is currently in beta. Contact your customer success manager for a private preview. Row-Level Access Control policies allow you to restrict access to specific rows within a table using conditional filters. These policies include a `conditions` section with `rowFilters`: #### Single Condition ```json theme={null} { "version": "v1", "statements": [ { "description": "Allow access to specific user data only", "resources": ["srn2:cluster#pinot:table#pageviews"], "effect": "allow", "actions": ["query"], "conditions": { "rowFilters": ["user_id = 18314973"] } } ] } ``` #### Multi-Condition ```json theme={null} { "version": "v1", "statements": [ { "description": "Allow high-value sessions for specific user", "resources": ["srn2:cluster#pinot:table#pageviews"], "effect": "allow", "actions": ["query"], "conditions": { "rowFilters": ["user_id = 18314973 AND session_value > 1000"] } } ] } ``` Learn more about [Row-Level Access Control](/corecapabilities/security/row-level-access-control) for detailed implementation guidance. ### Using Wildcards Wildcards provide flexibility to leverage naming conventions used in your organization. In this example, the policy will: * Allow all actions on tables with the prefix "Test" * Allow queries on tables with the prefix "Prod" * Deny deletes for all tables ```json theme={null} { "version": "v1", "statements": [ { "description": "Query tables with the prefix 'Prod'", "resources": "srn2:cluster#*:table#Prod*", "effect": "allow", "actions": "Query" }, { "description": "All actions on tables with the prefix 'Test'", "resources": "srn2:cluster#*:table#Test*", "effect": "allow", "actions": "*" }, { "description": "Deny dropping tables and pausing consumption on all tables", "resources": "srn2:cluster#*:table#*", "effect": "deny", "actions": [ "Delete*", "PauseConsumption" ] } ] } ``` # Microsoft Entra ID Source: https://docs.startree.ai/corecapabilities/security/idp/azure Configure Microsoft Entra ID as your Identity Provider (IdP) with StarTree. Microsoft Entra ID enables centralized authentication, enhanced security, and streamlined user management, simplifying access to StarTree. ## Prerequisites 1. Admin access to the [Azure portal](https://portal.azure.com/). 2. Access to a StarTree environment. 3. Obtain the redirect URI from StarTree. ## Steps ### Register the StarTree App 1. Navigate to the Microsoft Azure portal. 2. Select **Microsoft Entra ID** from the portal menu. ![idp-azure-entra-id.png](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/security/images/azure-entra-id.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=6c053d587341a1cf91e7d05f527b1ce6) 3. Click on **Add** and select **App registration** to register a new app. 4. Specify a name for the application and select the supported account types option that fits your needs.  5. Under **Redirect URI**, select **Web** as the platform, and enter the URL that was provided by StarTree. ![idp-azure-register-application.png](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/security/images/azure-register-application.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=89ae34f5d0eaa7220b895884206de6b4) 6. **Register** the application. 7. Copy the **Application (client) ID** and the **Directory (tenant) ID**. You will need to provide these to StarTree. 8. Click **Add a certificate or secret** and create a new client secret. Save the client secret **Value**. You will need to provide this to StarTree. 9. Make sure the users have the email address field populated. ### Provide the following details to StarTree * Application (client) ID. * Directory (tenant) ID. * Client secret. * Issuer URL: [https://login.microsoftonline.com//v2.0](https://login.microsoftonline.com/\{tenantid}/v2.0) (replace the tenant ID with the actual value). ### \[Optional] Enable Groups Claim To enable groups claim for an application in Azure AD for OpenID Connect, complete the following steps: 1. Select **Azure Active Directory** from the left navigation menu. 2. Select **App registrations**.  3. Select the application you just created. 4. Select **Token configuration** from the left navigation menu  ![idp-azure-token-config.png](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/azure-token-config.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=9617da92fec46e8398221a93e1bfe36b) 5. Click the **+ Add groups claim** button and select the types of groups you want to include in the claim. 6. Once you save the groups claim, click **API permissions**. ![idp-azure-api-permissions.png](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/security/images/azure-api-permissions.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=b17c26940a42635b3ab8dc6706ba93e2) 7. Click on **+ Add a permission.** 8. Select **Microsoft Graph** and then **Delegated permissions.** 9. Scroll down to the **Directory** category, expand it, then check **Directory.Read.All** ![idp-azure-api-permission-1.png](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/corecapabilities/security/images/azure-api-permission-1.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=74bcff10700a1b92fbb92171ed63096c) 10. Click **Add permissions.** 11. When prompted, click **Yes** to grant admin consent. ### Granting User and Group Access to StarTree 1. Use the Azure portal search bar to search for **Enterprise applications.** 2. Select the application you created. 3. In the left navigation menu, expand the **Manage** menu and click on **Users and Groups.** ![idp-azure-users-and-groups.png](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/azure-users-and-groups.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=b2c844f4c0e8d85e95526826abdc1fe1) 4. Click **+ Add user/group** and add the users and groups that you want provide access to StarTree. 5. Click on **Assign.** # Google Source: https://docs.startree.ai/corecapabilities/security/idp/google Configure Google as your Identity Provider (IdP) with StarTree. Leverage your existing Google Workspace or Cloud Identity for centralized authentication, enhanced security, and streamlined user management for StarTree. ## Prerequisites 1. Admin access to Google Cloud Console. 2. Access to a StarTree environment. 3. Obtain the redirect URI from StarTree. ## Steps ### Create an OAuth client 1. Access your Google Cloud Console. 2. Create a new project or open an existing project. 3. Create an Oauth app: a. Open **APIs & Services.** b. Click on **Credentials.** c. Click on **Create Credentials** > **Oauth client ID.** d. Select **Web application** as the application type. e. Enter a name for the application. f. Enter the authorized redirect URI provided by StarTree (this will be in the form of `https://identity...startree.cloud/callback`) ![idp-google-authorized-redirect-uri.png](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/google-authorized-redirect-uri.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=ba73e97d3cc14d0398927cf53e3a7617) g. Copy the **Client ID** under **Additional information**. You will need to provide this to StarTree. h. Click **Create.** ### Fetching Groups from Google To allow StarTree to fetch group information from Google, configure a service account. This account needs Domain-Wide Delegation and permission to access the `https://www.googleapis.com/auth/admin.directory.group.readonly` API scope. > [!NOTE] > > To complete these steps, you’ll need an administrator account for a managed Google service, such as Google Workspace or Cloud Identity. [Learn more about Google accounts](https://support.google.com/a/answer/6375836?hl=en). To set up group fetching: 1. Follow [this guide](https://developers.google.com/workspace/guides/create-credentials#service-account) to set up a **service account** with Domain-Wide Delegation. Make sure that you: a. Generate the key as a JSON file. You will need to share the key file with StarTree. b. When setting up domain-wide delegation of authority for the service account, set the OAuth scope to [https://www.googleapis.com/auth/admin.directory.group.readonly](https://www.googleapis.com/auth/admin.directory.group.readonly). Only include the scope shown above, otherwise the setup will not work. 2. Enable the [Admin SDK](https://console.developers.google.com/apis/library/admin.googleapis.com/). 3. In the Google Admin Console, create or identify a Google Workspace user with a minimum of the Groups Reader role assigned, and provide this user ID to StarTree. The service account you created earlier will impersonate this user when making calls to the admin API. A valid user should be able to retrieve a list of groups when [testing the API](https://developers.google.com/admin-sdk/directory/v1/reference/groups/list#try-it). # Okta Source: https://docs.startree.ai/corecapabilities/security/idp/okta Configure Okta as your Identity Provider (IdP) with StarTree. Integrating Okta enables centralized authentication, enhanced security, and streamlined user management, simplifying access to StarTree. ## Prerequisites 1. Admin access to an Okta account \ (For example: `https://dev-1234567-admin.okta.com/admin`) 2. Access to a StarTree environment. 3. Obtain the sign-in and sign-out redirect URLs from StarTree. ## Steps 1. Sign in to Okta as an admin. 2. Click on **Applications** in the left navigation menu. 3. Create a new Application: a. Click **Create App Integration**. b. Select **OIDC - OpenID Connect** as the sign-in method. c. Select **Web Application** as the application type. d. Click **Next**. e. Enter the **App integration** **name**. f. Fill in the **sign-in** and **sign-out redirect URLs** provided to you by StarTree. ![okta-new-web-app-integration.png](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/okta-new-web-app-integration.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=228281ad55b30265607252fc5aaf7609) g. Choose the **controlled access** method that best fits your needs. If you’re not sure, you can select the option to skip group assignments for now h. Click **Save** to create the application 4. Obtain the **Client ID** and **Client Secret**. You will need to provide these to StarTree. ![okta-credentials](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/okta-credentials.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=702cd581d51498ce41dc33a29e7ac7d2) 5. Configure the groups claim:\ a. Click on the **Sign-on** tab. \ b. Click **Edit** in the **OpenID Connect ID Token** section.\ c. Click the **Issuer** dropdown and select **Okta URL**. \ d. Click **Save.** \ e. Copy the **Issuer URL** (example: `https://dev-1234567.okta.com`) . You will need to provide it to StarTree. \ f. If groups are managed in Okta: \ To have Okta return the list of all groups that a signed-in user belongs to, change the **Groups claim filter** to **Matches regex** and set the expression as needed (use `.*` to always include all groups that a user belongs to). ![okta-group-claims-in-okta.png](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/okta-group-claims-in-okta.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=1f6fd8294505a5dba9979784722445d9) g. If groups are managed outside of Okta (for example, in Active Directory): \ Set the “Groups claim type” to “Expression”, and enter the appropriate expression (e.g. **Groups.startsWith("active\_directory”,”my-group-name”,100)**).\ For more info, refer to the [Okta Help Center](https://support.okta.com/help/s/article/Why-isnt-my-Groups-claim-returning-Active-Directory-groups?language=en_US). ![okta-group-claims-outside-okta.png](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/okta-group-claims-outside-okta.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=6631f0e5359d52d8f6060044bf9f2357) 6. \[Optional] If you are using Okta’s [Custom Authorization Server](https://developer.okta.com/docs/concepts/auth-servers/#custom-authorization-server), then you will need to add the groups claim to the ID token.\ a. Navigate to the **Authorization Servers** section.\ b. Open the relevant custom authorization server.\ c. Click on the **Claims** tab.\ d. Under the ID claim type, enter the following claim name: `https://schemas.startree.ai/identity/claims/groups`, and set the value as needed. ![okta-group-claims-id-token.png](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/okta-group-claims-id-token.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=f1f403649391978233d0c931f7d53c6c) 7. To complete the configuration of Okta IdP in StarTree, provide the following details that you obtained in the previous steps: * **App integration name.** * **Client ID** and **Client Secret.** * **Issuer URL.** ### Granting User and Group Access to StarTree * You can assign users and groups directly to the application by clicking the **Assignments** tab. * Alternatively, to create new groups or manage user and application assignments within a group, go to **Directory** > **Groups** in the left navigation menu. You may also need to grant your app permission to manage and read groups: 1. In your app, click on the **Okta API Scopes** tab. 2. Find the relevant groups permission and click **Grant.** # Managing Access Source: https://docs.startree.ai/corecapabilities/security/manage-access Manage secure, fine-grained access to StarTree resources with Role-Based Access Control (RBAC), enabling administrators to define custom policies, create roles, and assign them to users or groups through the Security Manager interface integrated with your organization's Identity Provider (IDP). Role-Based Access Control (RBAC) enables granular control over data access within the StarTree environment. RBAC ensures that access is restricted to authorized users. The StarTree Security Manager provides a powerful mechanism to craft policies that match the needs of any data team. When you enable access control, StarTree integrates with your organization's Identity Provider (IDP) to authenticate users and groups. After successful authentication, StarTree enforces access control policies based on the roles you assign. ![](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/rbac_diagram.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=4caf548444312fdd2b4ab76d2bdac471) The diagram shows a typical RBAC system containing users/groups, their roles, and the associated policies. Together, these attributes form the set of rules that define who can access a resource. > [!NOTE] > > Access Control features require initial configuration and may involve integration with your existing identity provider. Contact StarTree Support or your account team to discuss your specific requirements and enable these features. Using the Security Manager you can: * **Define and manage custom policies:** Create and modify fine-grained access control rules based on specific conditions and requirements. * **Create and manage custom roles:** Group together related permissions and policies into easily manageable roles. * **Assign roles to users and groups:** Granularly control access to StarTree resources for different entities within the organization. ## Access Control Capabilities StarTree provides two levels of access control to meet different security requirements: ### Table-Level Access Control (RBAC) Traditional Role-Based Access Control provides all-or-nothing access to resources: * **Resource-level granularity**: Control access to entire tables, databases, clusters, and environments * **Pre-seeded roles**: Ready-to-use roles like `table-reader`, `system-admin`, and `rbac-admin` * **Custom role definition**: Create roles tailored to your organization's needs * **Multi-IDP support**: Works seamlessly with Okta, Azure AD, Google Workspace, and Auth0 * **API & UI management**: Manage policies through both web interface and REST APIs ### Row-Level Access Control > [!WARNING] > > Row-Level Access Control is currently in beta. Contact your customer success manager for a private preview. Advanced access control that restricts access to specific rows within tables using conditional policies and SQL expressions. Learn more about implementing [Row-Level Access Control](/corecapabilities/security/row-level-access-control) for fine-grained data access. ## Policies, Roles, and Assignments StarTree utilizes Role-Based Access Control (RBAC) to manage data access. RBAC is a security model where access rights are defined by assigning roles to users, users groups, and API tokens. Roles are collections of policies that define permissions for specific resources. ### Policies A policy defines the rules that govern access to specific resources. It outlines which actions (like reading, writing, or deleting) are permitted or prohibited on a particular resource, such as a table. Policies are the building blocks for granular control over data access within the StarTree platform. In the Policies screen, you can: * View existing policies: Easily browse and review all defined policies within the system. * Create new policies: Define custom policies with specific conditions and permissions, such as: * Resource-level access: Control access to specific tables, workspaces, and environments. * Operation-level access: Control allowed operations on resources (e.g., read, write, delete, query). ### Roles A Role represents a collection of one or more policies. It acts as a container for a set of permissions that define what actions a user or group is authorized to perform on specific resources. Think of it as a job title or a set of responsibilities. For example, a 'Data Analyst' role might include policies granting read access to certain tables, the ability to run queries, and potentially limited write access for specific purposes. Roles simplify user access management by grouping together relevant policies, making it easier to assign permissions to users based on their job functions or responsibilities. The Roles screen allows you to: * View existing roles: Easily browse and review all defined roles within the system. * Create new roles: Define custom roles by grouping together relevant policies and permissions. * Edit roles: Add or remove policies associated with an existing custom role. ### Role Assignments Role assignment is the process of associating a specific role with a particular user or group. This effectively grants the assigned entity the permissions defined within the associated role. Think of it as assigning a job title to an employee. When you assign the 'Data Analyst' role to a user, they automatically inherit all the permissions associated with that role, such as read access to specific datasets and the ability to run queries. Role Assignment is the crucial step that connects users and groups to the specific permissions defined within Roles, enabling effective and granular control over data access within the StarTree environment. > [!NOTE] > > By default, the first user in an environment is assigned the `system-admin` role, which grants them full access to all resources and actions in their StarTree environment. When inviting new users, you can easily control user access by assigning roles. You can also update or remove roles at any time to ensure that users have the correct level of access. In the Role Assignments screen, you can: * View existing assignments: Easily browse and review all assignments within the system. * Create new assignments: Grant permission by assigning a role to a user or a group. * Assign roles to users: Grant roles to individual users based on their email. * Assign roles to groups: Groups are defined and managed within your organization's Identity Provider (IDP). This could be a system like [Okta](https://help.okta.com/asa/en-us/content/topics/adv_server_access/docs/setup/create-a-group.htm), [Azure AD](https://learn.microsoft.com/en-us/entra/fundamentals/groups-view-azure-portal), or [Google Workspace](https://support.google.com/a/answer/9400082?hl=en\&ref_topic=25840\&sjid=9852883180017174836-NC). * Delete assignments: Delete an assignment to revoke access from a user or a group. > [!NOTE] > > Role assignments can't be edited. To change an assignment, delete the existing assignment and create a new one with a different role. ## Create a custom policy You can create a custom policy to define fine-grained access to specific resources. For example, you can define a policy that allows read-only access to a specific table. To create a custom policy, follow these steps: 1. Open the Security Manager by navigating to [https://dp.your\_environment\_id.startree.cloud/security-manager](https://dp.your_environment_id.startree.cloud/security-manager). 2. Click **Policies** in the left navigation menu. 3. Click **Create policy**. 4. Enter a name and a description for the new policy. 5. Provide the policy configuration in JSON format. See our [Custom policies](/corecapabilities/security/custom-policies) page to learn more about writing your own policy configuration. 6. Click **Create policy** to save the new policy. ## Testing a custom policy Due to potential propagation delays of up to an hour, we recommended testing your policy before saving it. StarTree provides an API that allows you to simulate access requests and see how your policies would be applied. You can use the following `curl` command to send a request to the Simulate API: ```bash theme={null} curl --location 'https://rbac-manager./api/v1/rbac-manager/simulate-access' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "policyDocumentJson": "", "subjectTypeToSubject": { "": "" }, "resourceSrn2String": "", "action": "" }' ``` * `policyDocumentJson` is the JSON content of the policy you want to test. * `subjectTypeToSubject` identifies the subject (user email, group name, or API token) that the API will use to simulate the policy with. The values can be one of the following pairs: | subject-type | subject-id | | - | - | | `email` | the user's email address (e.g., `john.doe@example.com`) | | `group` | The name of the group the user belongs to, as defined in your organization's IDP | | `service-token` | The username portion of the service token (see how to obtain a service token's username ) | * `resourceSrn2String `specifies the resource that the you want to simulate access to. * `action` is the action that you want to simulate on the resource (e.g., `Query`, `DeleteTable`, `UpdateSchema`). > [!NOTE] > > The simulate access API takes into account all roles assigned to the specified subject(s). This means that if a user belongs to multiple groups or has multiple roles directly assigned, the API will evaluate the cumulative effect of all associated policies to determine access. > [!NOTE] > > If you saved the policy without testing it using the simulate API, and you don't want to wait for the changes to propagate, you can still test the new policy immediately by using a private browsing window, or by clearing your browser's cache. After clearing your cache or using a private window, you should be able to see the effects of your policy changes immediately. ## Create a custom role You can create a custom role to group together policies, allowing for precise control over what actions users can perform on specific resources. Creating a custom role involves the following steps: 1. Open the Security Manager by navigating to [https://dp.your\_environment\_id.startree.cloud/security-manager](https://dp.your_environment_id.startree.cloud/security-manager). 2. Click “Roles” in the left navigation menu. 3. Click **Create role**. 4. Enter a name and a description for the new role. 5. Select the policies to be included in the new role. You can click the **Policy Details** button to view the detailed configuration of a policy. 6. Click **Create role** to save the new role. ## Assign roles to users and groups In the role assignments screen you can attach roles to individual users and groups. * Individual users are identified using their email address. * Groups are defined and managed within your organization's Identity Provider (IDP). This could be a system like [Okta](https://help.okta.com/asa/en-us/content/topics/adv_server_access/docs/setup/create-a-group.htm), [Azure AD](https://learn.microsoft.com/en-us/entra/fundamentals/groups-view-azure-portal), or [Google Workspace](https://support.google.com/a/answer/9400082). To assign a role to users or groups, follow these steps: 1. Access the Security Manager by navigating to [https://dp.your\_environment\_id.startree.cloud/security-manager](https://dp.your_environment_id.startree.cloud/security-manager). 2. Click **Role Assignments** in the left navigation menu. 3. Click **Assign role**. 4. Select the role that you want to assign. 5. Select the appropriate **Subject type**, depending on whether the subjects are individual users or groups. 6. Provide the list of user emails or group names, separated by commas 7. Click **Assign role** to save the new assignment. ## Guidance for Customer Admins For each StarTree Environment, a customer admin is designated. The responsibility of this person is to govern access to users/services in the customer organization via Security Manager. For a newly created environment, only the customer admin will have the ability to log into StarTree data plane. Rest of the users will be denied access until they're explicitly assigned a role by the customer admin. **Default workspace access control** In most environments, users prefer to create and manage datasets in the default workspace. This is the workspace that comes default with the environment. In order to grant access to this default workspace, customer admin can choose one of the following roles to assign to their users/services: * owner-default: This is a predefined role that grants workspace owner permissions for the ***default*** workspace to the subject. This will allow the subject to create/modify/delete datasets and do other workspace operations. * reader-default: This is a predefined role that grants workspace read permissions for the ***default*** workspace to the subject. This will allow the subject to list and query any datasets in the workspace. > [!NOTE] > > Note that subject must be assigned a role before getting access to data plane including things like Data Portal / Query console. If the above 2 roles are too generic or not restrictive enough, customer admin can create a custom role+policy to fine tune access. # API Token Management Source: https://docs.startree.ai/corecapabilities/security/manage-api-tokens This page describes how to create and manage API tokens within the StarTree environment. API tokens provide programmatic access to StarTree resources through role-based permissions. They consist of an access key and a secret key, formatted as `st--`. ## Generating an API token 1. Navigate to the Security Manager by browsing to [https://dp.your\_environment\_id.startree.cloud/security-manager](https://dp.your_environment_id.startree.cloud/security-manager). 2. Click **API Tokens** in the left navigation menu. 3. Click **Generate API token**. 4. Select the desired role for the token. This will determine the permissions that will be granted to the token. * For more information on roles and how they are defined, refer to the [Manage Access](/corecapabilities/security/manage-access) page. 5. Provide a description for the token. This will help you identify the token going forward. 6. Click **Generate API token** to generate the new token. 7. Copy the token and store it someplace safe. > [!WARNING] > > Copy the generated token immediately. This token will only be displayed once. ## Managing API tokens * **View Existing Tokens:** The API Tokens screen displays a list of all existing tokens, including their description, creation date, and their assigned role. * **Delete a Token:** To delete an API token, locate the token in the API Tokens list, click the actions menu (three vertical dots to the right of the token description) and click **Delete**. ## Security considerations * **Treat API tokens like passwords:** Store API tokens securely and never share them with unauthorized individuals. * **Use strong security practices:** Implement appropriate security measures to protect your API tokens, such as: * **Assign least privilege:** Grant only the minimal necessary permissions to API tokens. * **Ensure regular rotation:** Regularly rotate API tokens to minimize the risk of unauthorized access. * **Use secure storage:** Store API tokens securely and avoid hardcoding them directly into applications. ## Important notes * API tokens provide programmatic access to StarTree resources. Exercise caution when granting permissions to API tokens. * Revoking an API token will immediately disable its access to StarTree resources. ## Test the API token You can test the token using one of the following methods: ### Using CURL To test the API token using CURL, include the token in the `--header` (`-H`) authorization parameter. ``` curl --location --request GET 'https://pinot..cloud/appconfigs' \ --header 'Authorization: Bearer st-XkQBXKr652MV1VF9-5gkhGWNaGAdA1NU5yrHXRSPKMCcNRlRg' ``` ### Using the Swagger API 1. In Pinot UI, click on **Swagger REST API**. 2. Click **Authorize**. 3. In the "Available authorizations" dialog, scroll up to see the `oauth` authorization option. 4. Enter `Bearer ` (including the space), followed by the token in the **Value** textbox. ![image](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/swagger_ui_authorize_dialog.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=5dfa2f76cb91b283d3d6ebcebba4353c) 5. Click **Authorize**. 6. Find the API call you want to try and click the related **Try it out** button. ![image](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/security/images/swagger_try_it_out.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=6a66ff834fd54406bbc764cad276b1cd) ## Extracting Username and Password from a StarTree Bearer Token Certain integrations or tools require basic authentication (username and password) rather than bearer token authentication. In such cases, you can extract the equivalent username and password from your StarTree bearer token. StarTree bearer tokens are formatted as `st--`. To extract the username and password: 1. Remove the `st-` prefix from the token. 2. The 16 characters immediately following the prefix (before the hyphen) constitute the username. 3. The 32 characters after the hyphen constitute the password. **Example:** Given the bearer token: `st-XkQBXKr652MV1VF9-5gkhGWNaGAdA1NU5yrHXRSPKMCcNRlRg` * Username: `XkQBXKr652MV1VF9` * Password: `5gkhGWNaGAdA1NU5yrHXRSPKMCcNRlRg` Use these credentials where basic username/password authentication is required for your Pinot connection. # Security Manager Source: https://docs.startree.ai/corecapabilities/security/manage-security The StarTree Security Manager provides a centralized location for managing access control within your StarTree environment. Use the Security Manager to: * [Manage access for users and groups](/corecapabilities/security/manage-access) * [Generate and manage API tokens](/corecapabilities/security/manage-api-tokens) * [Write custom policies](/corecapabilities/security/custom-policies) # Row-Level Access Control Source: https://docs.startree.ai/corecapabilities/security/row-level-access-control Restrict access to specific rows within tables using conditional policies, enabling secure data sharing while maintaining data isolation for different users and groups. > [!WARNING] > > Row-Level Access Control is currently in beta. Contact your customer success manager for a private preview. Row-Level Access Control allows you to restrict access to a subset of rows within a table for specific entities (users, groups, or tokens). This provides more granular control than traditional table-level permissions. > [!NOTE] > > Row-Level Access Control builds upon StarTree's Role-Based Access Control (RBAC) system. Ensure you have [RBAC enabled](/corecapabilities/security/manage-access) before implementing row-level policies. ## How Row-Level Access Control Works Unlike table-level RBAC which provides all-or-nothing access to a table, Row-Level Access Control allows you to define conditions that determine which rows a user can access within a table. When a user with row-level policies queries a table, StarTree automatically: 1. **Identifies** the user's assigned roles and associated row-level policies 2. **Extracts** the row filter conditions from the policies 3. **Applies** these conditions as additional WHERE clauses to the query 4. **Returns** only the rows that match both the original query and the row filter conditions ## Policy Configuration Row-level policies use the same structure as standard RBAC policies but include a `conditions` section with `rowFilters`: ```json theme={null} { "version": "v1", "statements": [ { "description": "Allow access to specific user data only", "resources": ["srn2:cluster#pinot:table#pageviews"], "effect": "allow", "actions": ["query"], "conditions": { "rowFilters": ["user_id = 18314973"] } } ] } ``` ## Row Filter Syntax ### Single Condition Filters Row filters are valid SQL expressions that can be used in WHERE clauses: ```json theme={null} { "conditions": { "rowFilters": ["user_id = 12345"] } } ``` ```json theme={null} { "conditions": { "rowFilters": ["region = 'US'"] } } ``` ### Multi-Condition Filters Combine multiple conditions using SQL operators: ```json theme={null} { "conditions": { "rowFilters": ["user_id = 18314973 AND session_value > 1000"] } } ``` ```json theme={null} { "conditions": { "rowFilters": ["user_id IN (18314973, 18314974, 18314975) AND total_time_spent >= 10000"] } } ``` ```json theme={null} { "conditions": { "rowFilters": ["data_region = 'EU' AND compliance_flag = 'GDPR'"] } } ``` ## Dynamic Row Filters (available from Startree release 0.17) Static row filters work well when you can enumerate every value up front — for example, writing one policy per vendor ID. For multi-tenant applications where the tenant (or user, region, account, etc.) isn't known until query time, you can write **one policy** with a placeholder, and have StarTree fill it in automatically from a value the caller supplies with the query. > [!NOTE] > > Dynamic row filters use the same policy structure as static row filters — the only difference is the filter text contains a placeholder instead of a literal value. ### Placeholder Syntax A placeholder has the form `${parameterName}` or `${parameterName:TYPE}`: ```json theme={null} { "conditions": { "rowFilters": ["tenant_id = ${tenant_id}"] } } ``` ```json theme={null} { "conditions": { "rowFilters": ["account_id = ${account_id:LONG}"] } } ``` If no type is given, the placeholder defaults to `STRING`. Supported types: | Type | Accepts | | - | - | | `STRING` | Any text up to 1024 characters (default) | | `INT` | A 32-bit integer | | `LONG` | A 64-bit integer | | `FLOAT` / `DOUBLE` | A numeric value, with or without a decimal point | | `BOOLEAN` | `true` or `false` | A placeholder must stand in for a whole value being compared to a column — for example on the right-hand side of `=`, `>`, `BETWEEN`, or inside an `IN (...)` list: ```json theme={null} { "conditions": { "rowFilters": ["tenant_id = ${tenant_id} AND signup_date >= ${min_date:LONG}"] } } ``` ### Supplying Values at Query Time The caller supplies each placeholder's value as a **URL query parameter on the HTTP request to the query endpoint** — it is not part of the SQL text itself. For example, with the policy above (`tenant_id = ${tenant_id}`), the query request becomes: ```bash theme={null} curl -X POST "https:///query/sql?tenant_id=acme_corp" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"sql": "SELECT * FROM orders LIMIT 10"}' ``` StarTree reads `tenant_id=acme_corp` off the request URL, substitutes it into the policy's row filter as a properly typed, escaped literal, and applies it the same way a static row filter is applied — the caller never needs to (and cannot) write the filter into the SQL itself. This is particularly useful for embedded or white-labeled dashboards, where a single widget or service account queries on behalf of many tenants, and the tenant identifier is known to the embedding application at query time but shouldn't be hard-coded per-tenant into a policy. ### Validation and Error Handling Every value is validated before the query runs; nothing partial is ever returned: * A required parameter that is missing or empty fails the query. * A parameter supplied more than once (ambiguous) fails the query. * A value that doesn't match its declared type (for example, non-numeric text for an `INT` placeholder) fails the query. A failed validation returns an error response identifying which parameter was the problem, so the caller can correct the request. ## Implementation Steps Row-Level Access Control follows the same implementation process as any other RBAC policy: 1. **Create a policy** with row filter conditions in the Security Manager 2. **Create a role** and attach the policy to it 3. **Assign the role** to users or groups The process is identical to standard RBAC - the only difference is adding the `conditions` section with `rowFilters` to your policy configuration. You can use the StarTree Security Manager UI to apply these policies. ## Use Cases ### Vendor Data Isolation Allow vendors to see only their own orders in a shared table: ```json theme={null} { "version": "v1", "statements": [ { "description": "Vendor can only see their own orders", "resources": ["srn2:cluster#pinot:table#orders"], "effect": "allow", "actions": ["query"], "conditions": { "rowFilters": ["vendor_id = 'VENDOR_123'"] } } ] } ``` ### Sales Territory Management Restrict sales reps to view only their assigned accounts: ```json theme={null} { "version": "v1", "statements": [ { "description": "Sales rep can only access assigned territories", "resources": ["srn2:cluster#pinot:table#customer_accounts"], "effect": "allow", "actions": ["query"], "conditions": { "rowFilters": ["sales_rep_id = 'REP_456' AND territory = 'West'"] } } ] } ``` ### Regional Data Compliance Enforce data residency by region within a global table: ```json theme={null} { "version": "v1", "statements": [ { "description": "EU users can only access EU data", "resources": ["srn2:cluster#pinot:table#user_analytics"], "effect": "allow", "actions": ["query"], "conditions": { "rowFilters": ["data_region = 'EU' AND compliance_flag = 'GDPR'"] } } ] } ``` ### Multi-Tenant Embedded Dashboards Serve every tenant from one policy, with the embedding application supplying the tenant at query time: ```json theme={null} { "version": "v1", "statements": [ { "description": "Each embedded dashboard sees only its own tenant's data", "resources": ["srn2:cluster#pinot:table#events"], "effect": "allow", "actions": ["query"], "conditions": { "rowFilters": ["tenant_id = ${tenant_id}"] } } ] } ``` ## Limitations * Row-level policies may take up to one hour to propagate and become effective after creation or modification. * Dynamic row filter values must be supplied as URL query parameters on the HTTP request to the query endpoint. This is not available on access paths that don't expose a URL query string. * Exactly one value is required per placeholder on each request — zero, empty, or multiple values for the same parameter name fail the query. * Placeholder values are strictly typed (`STRING`, `INT`, `LONG`, `FLOAT`, `DOUBLE`, `BOOLEAN`); a value that doesn't match its declared type fails the query rather than being coerced. * String placeholder values are limited to 1024 characters and may not contain control characters. * The parameter names `sql`, `trace`, `getcursor`, `numrows`, and `offset` are reserved by the query API and cannot be used as placeholder names. ## Next Steps * Learn about [Custom Policy Configuration](/corecapabilities/security/custom-policies) * Explore [RBAC API usage](/corecapabilities/security/using-rbac-api) * Review [Security Actions](/corecapabilities/security/actions) available for policies # Using the StarTree Role-Based Access Control (RBAC) API Source: https://docs.startree.ai/corecapabilities/security/using-rbac-api Configure using the RBAC Manager API. RBAC allows you to define granular permissions for users, groups, and service tokens. ## Prerequisites * You need an Authorization Bearer token to invoke these APIs. See [Generating an API Token](./manage-api-tokens#generating-an-api-token). * Once RBAC is enabled, all access is governed by the roles and policies you set up. Ensure you assign appropriate admin privileges if needed. > [!NOTE] > > Looking for request and response schemas? Every endpoint on this page is in the [RBAC Manager API reference](/api-reference/rbac/introduction), with a playground you can call from the browser. ## Core RBAC Setup Process Setting up RBAC generally involves these four steps: 1. **Create a Policy:** Define *what* actions are allowed or denied on *which* resources. 2. **Create a Role:** Create a role entity that will group permissions. 3. **Attach Policy to Role:** Link the policy (permissions) to the role. 4. **Assign Role to Subject:** Grant the role (and its associated permissions) to users, groups, domains, or service tokens. Once these steps are completed, access control will be enforced based on your configuration. ## Listing and Viewing Policies Before creating a new policy, it's often helpful to see which policies already exist in your environment. 1. Start by [retrieving the list of policies](#listing-all-policies) using the `GET /api/v1/rbac-manager/policies` endpoint. 2. For any policy that you want to examine in more detail, [get the policy definition](#viewing-a-specific-policy) using the `GET /api/v1/rbac-manager/policies/{policySrn}` API. You can then decide whether to [update an existing policy](#updating-a-policy) or [create a new policy](#creating-a-policy). ## Working with Policies This section covers how to list, view, create, update, and delete policies. ### Listing All Policies To get a list of all defined policies, use the following API endpoint: **API Endpoint:** ``` GET /api/v1/rbac-manager/policies ``` **Example CURL Command:** ```bash theme={null} curl --location --request GET 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/policies' \ --header 'Authorization: Bearer ' ``` **Response:** A list of policy summaries, each including the policy's name and its SRN (StarTree Resource Name). ### Viewing a Specific Policy To view the full details of a single policy, including its description and the policy document JSON, use this endpoint: **API Endpoint:** ``` GET /api/v1/rbac-manager/policies/{policySrn} ``` **Example CURL Command:** ```bash theme={null} curl --location --request GET 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/policies/srn2:policy%23' \ --header 'Authorization: Bearer ' ``` > *`Replace with the name part of the policy's SRN`* **Response:** The complete details for the specified policy, including its name, description, the policy document JSON, creation timestamp, and last update timestamp. ### Creating a Policy Use this endpoint to create a policy: **API Endpoint:** ``` POST /api/v1/rbac-manager/policies ``` **Example CURL Command:** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/policies' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "name": "my-read-only-policy", "description": "Policy allowing query on myTable", "policyDocumentJson": "{ \"version\": \"v1\", \"statements\": [ { \"effect\": \"allow\", \"actions\": [\"query\"], \"resources\": [\"srn2:cluster#*:table#myTable\"] } ] }" }' ``` **Policy Document:** This JSON defines the permissions. It contains statements with effect (allow/deny), actions (e.g., "query", "\*"), and resources (e.g., "srn2:cluster#\*", "srn2:cluster#\*:table#myTable"). See the [custom policies documentation](/corecapabilities/security/custom-policies) for syntax details, examples, and additional information. ### Updating a Policy Use this endpoint to update an existing policy: **API Endpoint:** ``` PUT /api/v1/rbac-manager/policies/{policySrn} ``` **Example CURL Command:** ```bash theme={null} curl --location --request PUT 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/policies/srn2:policy%23' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "description": "Updated policy description", "policyDocumentJson": "{ \"version\": \"v1\", \"statements\": [ { \"effect\": \"allow\", \"actions\": [\"query\", \"read\"], \"resources\": [\"srn2:cluster#*:table#myTable\"] } ] }" }' ``` ### Deleting a Policy Use this endpoint to delete a policy: **API Endpoint:** ``` DELETE /api/v1/rbac-manager/policies/{policySrn} ``` **Example CURL Command:** ```bash theme={null} curl --location --request DELETE 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/policies/srn2:policy%23' \ --header 'Authorization: Bearer ' ``` > [!NOTE] > > A policy cannot be deleted if it is associated with a role. ## Working with Roles This section covers how to create, list, view, and delete roles. ### Creating a Role Use this endpoint to create a role: **API Endpoint:** ``` POST /api/v1/rbac-manager/roles ``` **Example CURL Command:** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "name": "table-reader-role", "description": "Provides read-only access to specific tables." }' ``` ### Listing All Roles Use this endpoint to list all roles: **API Endpoint:** ``` GET /api/v1/rbac-manager/roles ``` **Example CURL Command:** ```bash theme={null} curl --location --request GET 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles' \ --header 'Authorization: Bearer ' ``` ### Getting Role Details Use this endpoint to get details for a specific role: **API Endpoint:** ``` GET /api/v1/rbac-manager/roles/{roleSrn} ``` **Example CURL Command:** ```bash theme={null} curl --location --request GET 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23' \ --header 'Authorization: Bearer ' ``` ### Deleting a Role Use this endpoint to delete a role: **API Endpoint:** ``` DELETE /api/v1/rbac-manager/roles/{roleSrn} ``` **Example CURL Command:** ```bash theme={null} curl --location --request DELETE 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23' \ --header 'Authorization: Bearer ' ``` > [!NOTE] > > A role cannot be deleted if any subjects are currently assigned to it. ## Attaching Policies to Roles This section covers how to attach policies to roles, list attached policies, and detach policies. ### Attach a Policy to a Role Use this endpoint to link a policy to a role: **API Endpoint:** ``` POST /api/v1/rbac-manager/roles/{roleSrn}/attach-policy ``` **Example CURL Command:** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23table-reader-role/attach-policy' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "policySrn": "srn2:policy#my-read-only-policy" }' ``` You can attach multiple policies to a single role by calling this API multiple times. ### List Policies Attached to a Role Use this endpoint to list all policies attached to a role: **API Endpoint:** ``` GET /api/v1/rbac-manager/roles/{roleSrn}/policies ``` **Example CURL Command:** ```bash theme={null} curl --location --request GET 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23/policies' \ --header 'Authorization: Bearer ' ``` ### Detach a Policy from a Role Use this endpoint to detach a policy from a role: **API Endpoint:** ``` POST /api/v1/rbac-manager/roles/{roleSrn}/detach-policy ``` **Example CURL Command:** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23/detach-policy' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "policySrn": "srn2:policy#" }' ``` ## Role Assignments This section covers how to assign roles to subjects, list assignments, and delete assignments. ### Assign a Role to a Subject Use this endpoint to grant a role to a subject: **API Endpoint:** ``` POST /api/v1/rbac-manager/roles/{roleSrn}/create-assignment ``` **Subject Types:** * **user-email**: An individual user's email address (e.g., "[john.doe@acme.com](mailto:john.doe@acme.com)") * **group**: The name of a group managed in your Identity Provider (IDP) (e.g., "data-analysts") * **domain**: An email domain (e.g., "acme.com") * **service-token**: A service token identified by its accessKey, which is the 16 character string between `st-` and the following `-` (dash) **Example Request (assigning to a user):** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23table-reader-role/create-assignment' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "subject": "", "subjectType": "user-email" }' ``` **Example Request (assigning to a group):** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23/create-assignment' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "subject": "", "subjectType": "group" }' ``` **Example Request (assigning to a domain):** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23/create-assignment' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "subject": "", "subjectType": "domain" }' ``` **Example Request (assigning to a service token):** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23/create-assignment' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "subject": "", "subjectType": "service-token" }' ``` ### List Role Assignments Use this endpoint to list all assignments for a role: **API Endpoint:** ``` GET /api/v1/rbac-manager/roles/{roleSrn}/assignments ``` **Example CURL Command:** ```bash theme={null} curl --location --request GET 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23/assignments' \ --header 'Authorization: Bearer ' ``` ### Delete a Role Assignment Use this endpoint to delete a role assignment: **API Endpoint:** ``` POST /api/v1/rbac-manager/roles/{roleSrn}/delete-assignment ``` **Example CURL Command:** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/roles/srn2:role%23/delete-assignment' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "subject": "joe@someorg.com", "subjectType": "user-email" }' ``` ## Service Tokens Service tokens act as bearer tokens for programmatic access to StarTree services (like Pinot). They consist of an access key and a secret key, formatted as st-\-\. ### Creating and Using Service Tokens 1. **Create Token:** Use the following API endpoint: **API Endpoint:** ``` POST /api/v1/rbac-manager/service-tokens ``` **Example CURL Command:** ```bash theme={null} curl --location --request POST 'https://rbac-manager..startree.cloud/api/v1/rbac-manager/service-tokens' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "description": "" }' ``` The response contains the accessKey, secretKey, and the full bearerToken. > [!WARNING] > > Securely save the bearerToken and secretKey immediately; the secret key cannot be retrieved again. The accessKey is used as the identifier. 2. **Assign Role:** Use the create-assignment endpoint (detailed in the [Assign a Role to a Subject](#assign-a-role-to-a-subject) section) with subjectType: "service-token" and the token's accessKey as the subject. A token is unusable until assigned to a role. 3. **Use Token:** Use the full bearerToken (e.g., st-accesskey-secretkey) in the Authorization: Bearer \ header for API calls. ### Managing Service Tokens * **Describe:** **API Endpoint:** ``` GET /api/v1/rbac-manager/service-tokens/{tokenSrn} ``` (using the SRN which includes the access key) * **Revoke/Delete:** **API Endpoint:** ``` DELETE /api/v1/rbac-manager/service-tokens/{tokenSrn} ``` You can also temporarily disable a token by detaching all its roles. # Connect Grafana to StarTree Cloud Source: https://docs.startree.ai/corecapabilities/visualize_data/connect-grafana-to-startree-cloud Use the StarTree Pinot data source plugin to visualize StarTree Cloud data in Grafana. [StarTree](https://startree.ai) publishes an official [Pinot data source plugin for Grafana](https://grafana.com/grafana/plugins/startree-pinot-datasource/) that lets you query and visualize data in StarTree Cloud directly from Grafana dashboards. ## Prerequisites * Grafana version 9.1.1 or later. * Access to a StarTree Cloud cluster, including its Controller and Broker URLs. * An API Token for authenticating to your cluster (see [Generating an API Token](../security/manage-api-tokens#generating-an-api-token)). ## Install the plugin 1. In Grafana, go to **Administration** > **Plugins and data** > **Plugins**. 2. Search for **StarTree Pinot**. 3. Select the plugin and click **Install**. You can also install it from the command line with the [Grafana CLI](https://grafana.com/docs/grafana/latest/cli/): ```bash theme={null} grafana-cli plugins install startree-pinot-datasource ``` Restart Grafana after a CLI install so the plugin is picked up. ## Get your Controller and Broker URLs 1. Visit the apps page of your StarTree Cloud cluster. 2. Note the Pinot Controller URL and Broker URL for your environment, in the form of `pinot...startree.cloud` for the Controller and `broker...startree.cloud` for the Broker. ## Create the data source in Grafana 1. Go to **Connections** > **Data sources** and click **Add new data source**. 2. Select **StarTree Pinot** from the list. 3. Under **Connection**, enter the **Controller URL** and **Broker URL** for your cluster. 4. Under **Auth**, enable token authentication and paste the API token you generated as a prerequisite. 5. Click **Save & test** to confirm Grafana can reach your cluster. ## Query data The plugin supports two ways to build queries against your tables: * **Builder mode** — pick a **Table**, **Time Column**, and **Metric Column** to produce a time series panel without writing SQL. * **Code editor mode** — write a custom Pinot SQL query, then choose a format: * **Table** — renders results directly, with an optional time column conversion. * **Time series** — requires a time column alias, time format, and metric column to plot results over time. * **Annotations** — maps result columns to annotation events on a panel. The plugin also provides template macros (for example, time range filtering and group-by helpers) to simplify writing dashboard queries, and supports trace-to-log correlation for linking trace IDs across panels. For the full list of macros and advanced configuration options, see the [plugin documentation](https://grafana.com/grafana/plugins/startree-pinot-datasource/). # Connect Superset to StarTree Cloud Source: https://docs.startree.ai/corecapabilities/visualize_data/connect-superset-to-startree-cloud Connect Superset to StarTree Cloud to visualize your data. ## Prerequisites * Access to a StarTree Cloud cluster. * Access to a Superset instance. \ For more information about how to install and deploy Superset, see [What is Apache Superset?](https://superset.apache.org/docs/intro) * Superset instance has the Pinot database driver installed, which is compatible with StarTree Cloud. For more information, see how to [install a database driver for Superset](https://superset.apache.org/docs/databases/installing-database-drivers). ## Obtain the SQLAlchemy URI to connect to StarTree 1. Visit the apps page of your cluster. 2. Click on the **Pinot API Tokens** button. 3. On the API Tokens page, select **Clients** from the left hand sidebar. 4. Click on **Superset** in the client options. 5. Click **Create a new API Token**. 6. Copy the *SQLAlchemy URI*. ## Create the Database Connection in Superset 1. In Superset, click **Settings** in the top-right corner and select **Database Connections** under the Data section. 2. Click the **+ Database** button. 3. Open the **Choose a Database** dropdown and select *Apache Pinot*. \ If Apache Pinot is not listed, see the [instructions](https://superset.apache.org/docs/configuration/databases#apache-pinot) to configure the Apache Pinot database driver. 4. Specify a name for the database connection 5. Paste the *SQLAlchemy URI* that you copied from StarTree. 6. Test the connection. # Connect Tableau to StarTree Cloud Source: https://docs.startree.ai/corecapabilities/visualize_data/connect-tableau-to-startree-cloud Connect either Tableau Desktop or Tableau Server to your StarTree Cloud instance. ## Prerequisites * JDBC driver (JAR file) * Tableau connector (TACO file) * An API Token (see [Generating an API Token](../security/manage-api-tokens#generating-an-api-token)) ## Install the Apache Pinot Driver 1. Build or download the **JDBC driver**.\ You can build the driver from the [source code](https://github.com/apache/pinot/tree/master/pinot-clients/pinot-jdbc-client) or you can [download](https://repo1.maven.org/maven2/org/apache/pinot/pinot-jdbc-client/) one that is already built. In either case, you must use the shaded jar, which will have a filename that includes `shaded`, as in `pinot-jdbc-client-1.0.0-shaded.jar`. 2. Copy the JDBC driver JAR to the Tableau drivers directory. * On Mac: `~/Library/Tableau/Drivers/` * On Windows: `C:\Program Files\Tableau\Drivers` 3. Download the **StarTree Tableau Connector** TACO file from the [Tableau Exchange](https://exchange.tableau.com/products/1033), and copy it to the connectors directory. * On Mac: `/Users/[user]/Documents/My Tableau Repository/Connectors` * On Windows: `C:\Users\[user]\Documents\My Tableau Repository\Connectors` > [!NOTE] > > You can skip this step if the `Apache Pinot by Startree` connector is already installed (see step 2 below). 4. Launch Tableau. ## Connect Tableau to Apache Pinot 1. On the left side panel, under the **To a Server** section, click the **More...** button. 2. Click on the Apache Pinot by StarTree connector. ![Tableau Connectors](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/visualize_data/images/tableau_connector_ui.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=59d70c8003f7e68f9eba9eb289d0385f) 3. Configure the connection to Pinot:\ a. **Controller Host:** Enter the URL to your Pinot cluster, in the form of `pinot...startree.cloud`\ b. **Port:** This would typically be `9000` or `443`\ c. **Authentication:** Leave this set to `None` (we will handle authentication in the ‘Advanced’ tab). ![Apache Pinot Connection Dialog](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/visualize_data/images/tableau_connector_dialog.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=27e942547cc5df8ae21c12e057043d2a) d. Click on the **Advanced** tab.\ e. Set the **Scheme** to HTTPS.\ f. Enable TLS for Controller and Brokers.\ g. Using the API token that you obtained in the prerequisites, enter `headers.Authorization=Bearer ` in the **Additional Properties** text box (e.g. `headers.Authorization=Bearer st-N5zM7kPqW8xR3aVb-H1gJ4fD6sA9zX2cC5vB8nL0kM7pQ2wE4rT6`). ![Apache Pinot Connection Dialog](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/visualize_data/images/tableau_connector_dialog_advanced.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=965533083e0cbc69762d44d0ec67c212) h. click **Sign In**. 4. Once you are connected, you will be able to see the data from your data source and can begin to query. ![Fetch data from Pinot](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/corecapabilities/visualize_data/images/tableau_tables_ui.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=ce58c37874ac5728c1cbf95bede1eb46) # Architecture Source: https://docs.startree.ai/getstarted/deployment/components_deployed The StarTree Cloud architecture integrates a managed Apache Pinot cluster with advanced ingestion, storage, and query capabilities. It supports diverse data sources, offers tiered storage for cost-effective performance, and utilizes Kubernetes-based orchestration for scalable, cloud-native deployments. ## Managed Pinot Cluster At the core of StarTree Cloud is Apache Pinot, an open source, distributed analytics database designed for low latency, high throughput OLAP queries. It is packaged with StarTree plugins that improve the scalability, cost efficiency and usability of the database. ![Startree Cloud Architecture Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/getstarted/images/startree-cloud-architecture.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=b2d74102a1246bf57e4cde06264ec84f) The StarTree Cloud cluster manages the ingestion, storage, and query functions of Apache Pinot. ### Ingestion There's a wide range of connectors to ingest data into StarTree Cloud. * **Streaming connectors**\ Continuously ingest data in realtime (milliseconds latency) from various streaming sources, such as: Apache Kafka, Confluent Cloud, Amazon Kinesis, and others. * **Batch connectors**\ One-time or recurring ingestion from sources such as Amazon S3 and Google Cloud Storage (GCS). * **SQL connectors**\ Import data from SQL sources such as Snowflake and BigQuery. The ingestion layer also provides a rich library of pre-processing functions and decorators to enhance and transform data as it's ingested into Pinot. ### Storage StarTree Cloud offers tiered data storage (hot, warm, cold) to enable efficient table management and a balance of performance and cost. This means users benefit from millisecond query responses on actively used, recent data. For less frequent queries on historical data, while latency may be higher, the storage is more cost-effective. Complementing this, data is highly compressed using a combination of columnar layout and advanced compression and encoding strategies. ### Query StarTree Cloud enables users to query their data in Pinot via SQL, executing both common OLAP queries and complex multi-stage queries that incorporate joins and window functions. StarTree Cloud stands out for its exceptional query performance, consistently ranked among the fastest analytical databases. Users can leverage a range of indexing options for any column, alongside the powerful StarTree Index – an intelligent materialized view. StarTree Cloud customers run tens of thousands of queries per seconds with p99 latency in the order of milliseconds. For more information on query performance, [read this blog post](https://startree.ai/resources/what-makes-apache-pinot-fast-chapter-ii). ## Cloud Infrastructure StarTree Cloud is a fully cloud-native platform, available on AWS, Google Cloud, and Azure Cloud. Built for modern, scalable applications, StarTree Cloud is built upon: * **Kubernetes-Based Orchestration:** Ensures portability, scalability, and high availability. * **Containerized Microservices:** Modular and resilient system design. * **Flexible Networking Support:** VPC peering, private link, transit gateway, * **Decoupled Architecture:** Agents can pull in configurations from the StarTree Control plane and act accordingly. For more information see: [https://startree.ai/deployment-options](https://startree.ai/deployment-options) # Deployment Options Source: https://docs.startree.ai/getstarted/deployment/deployment_models StarTree Cloud provides a fully managed experience for deploying and managing real-time analytics. Leverage the Shared SaaS option for fast setup, or use your own cloud for more control. StarTree Cloud provides three deployment options built for varying requirements and allowing you to focus on the most important part — elevating the experience for your customers. * **Shared SaaS (Public SaaS):** A traditional SaaS edition of StarTree cloud is great for customers who want a low-touch experience and prefer a hassle-free, convenient SaaS service. * **Bring Your Own Cloud (BYOC):** Also called Private SaaS (Dedicated SaaS), this edition of StarTree Cloud caters to customers who want additional control over how data is managed and stored. * **Bring Your Own Kubernetes (BYOK):** Available in preview, this deployment option gives customers complete control over the StarTree Cloud infrastructure within their own Kubernetes environments, and is ideal for security-sensitive customers who require strict data controls. > [!NOTE] > > **Note**: StarTree also offers a 'StarTree Distribution' variant for deployment. This includes all the components in the data plane. These components need to be deployed and configured independently. Some users use this variant for on-prem (non cloud) installation of StarTree. There are two main components in each deployment model: * **Data Plane:** All the components required for data collection, management, and query processing along with the value-added services and tools. * **Control Plane:** Responsible for deployment, management, and coordination across all StarTree data planes. ![Public SaaS and BYOC Deployment Models](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/getstarted/images/startree-cloud-byoc-saas-comparison.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=1984af5a7e59ba5a7c11b69742fc0e66) In a **Public SaaS** deployment, both the StarTree control plane and data plane are deployed in the cloud network managed by StarTree. In this case, data and queries from your cloud network are sent to the StarTree cloud network. In a **BYOC** deployment, the StarTree data plane is deployed in your cloud network so that it is co-located with your other data systems and applications. In this case, the StarTree control plane remotely manages the data plane. A **BYOK** deployment is similar to BYOC, with the added security of network isolation. The environment is managed by a local StarTree agent, which is responsible for pulling upgrade packages and configuration metadata. ## Benefits ### Public SaaS * Fully hands-free: You don’t need to be involved at all in administering the data plane * Simpler setup: You just focus on your data, your schema, and queries. * Isolation guarantee: Your SaaS instance is isolated from an infrastructure as well as the Pinot cluster point of view. ### BYOC * Compliance & Data Security: Data never leaves the customer’s security perimeter Customers have full control over who gets access to the data as well as adhere to their data compliance and governance standards. * Fully managed: Customers get a comprehensive SaaS-like experience without the need for any in-house expertise. They don’t need to worry about deployment, upgrades, and routine cluster maintenance. * SLA: Customers get the same SLA guarantees as that of a traditional SaaS deployment * Flexibility: Since infrastructure is deployed in the customer’s environment, they can choose from a variety of supported compute and storage instance types and advanced customizations. * Cost savings: Allows the customer’s cloud savings plan to be applied towards this infrastructure. * No persistent connection is needed between the control and the data plane. ### BYOK * Provides a way to completely air-gap the Pinot cluster for enhanced security. * Compliance & Data Security: Data never leaves the customer’s security perimeter Customers have full control over who gets access to the data as well as adhere to their data compliance and governance standards. * Flexibility: Since infrastructure is deployed in the customer’s environment, they can choose from a variety of supported compute and storage instance types and advanced customizations. * Cost savings: Allows the customer’s cloud savings plan to be applied towards this infrastructure. * No inbound access from Startree Control plane. ![Deployment Flexibility](https://mintcdn.com/startree/qZwmUU4Se8wDV-BE/concepts/images/deployment_flexibility.png?fit=max\&auto=format\&n=qZwmUU4Se8wDV-BE\&q=85\&s=089ad273c419af289da9402597f1ca6f) ## Feature Checklist | Feature | Public SaaS | BYOC | BYOK | | - | - | :- | :- | | **Fully Managed** | Yes | Yes | Partial | | **SLA Guarantees** | Yes | Yes | Partial | | **24x7 Support** | Yes | Yes | Yes | | **Provisioning Complexity** | Simple | Simple | Moderate | | **Authentication and Authorization** | Supported | Supported | Supported | | **Isolation Guarantees** | Supported | Supported | Supported | | **Data management and Security** | Stored securely in StarTree operated clusters | Data never leaves customer’s compliance boundary | Data never leaves customer’s compliance boundary | | **Data Governance and Compliance** | Standard | Flexible: customer dictates the policies | Flexible: customer dictates the policies | | **Certifications** | Soc2 Type2 ISO27001 HIPAA BAA | Soc2 Type2 ISO27001 HIPAA BAA | Soc2 Type2 ISO27001 HIPAA BAA | | **Infrastructure Customization** | Standard compute and storage instance types | Broader range of compute and storage types to choose from | Broader range of compute and storage types to choose from | | **Cost Optimizations** | Standard: Uses StarTree Cloud account for cloud provider | Can apply customer cloud account savings for additional TCO reduction | Can apply customer cloud account savings for additional TCO reduction | | **Software Upgrades** | Supported | Supported | Supported | | **Infrastructure Upgrades** | Supported | Supported | Customer owned | | **Monitoring and Observability** | Supported | Supported | Supported | # Security and Compliance Source: https://docs.startree.ai/getstarted/deployment/security_certifications StarTree Cloud ships with enterprise grade security and has industry-leading certifications such as SOC2 and HIPAA. At StarTree, we are deeply committed to protecting customer and internal data. This commitment encompasses robust **security measures** for devices, networks, and systems, as well as strict adherence to **regulatory compliance** to ensure data integrity and trust. ## Security StarTree has implemented features in the following areas to protect data. ### Authentication and Authorization * Controlled access on all company assets, using the principle of least privilege * Strong password requirements * Dedicated customer instances with controlled access ### Encryption * AES disk encryption for data at rest * SSL/TLS for data encryption in transit * Support for encryption options in AWS S3, Google Cloud Storage (GCS), and Amazon Elastic Block Storage (EBS) ### Networking * Antivirus * Anti-malware * Endpoint detection and response (EDR) solutions * Monitoring, including for file integrity * Intrusion detection * Firewalls ### Audit Trails and Data Loss Prevention * Logging * Offsite backups * Regular updates to operating system and application software * Testing and peer review and approval prior to pushing changes to production Your real-time analytics data is stored in your Apache Pinot cluster. Depending on your account, this could be running in StarTree Cloud or in your own cloud account for BYOC deployments. Configuration details and access management secrets are stored in secure locations. Sensitive data is encrypted. StarTree Cloud security features include: * Authentication via OIDC compatible Identity Providers * Authorization via fine-grained Role Based Access Control (RBAC) policies * Encryption in transit via TLS 1.2+ * Encryption at rest: All stored data (including logs and metadata) is encrypted using industry-standard AES-256 encryption. ## Compliance StarTree Cloud has achieved the following certifications: * SOC2 Type 2 * ISO 27001 * HIPAA readiness ![Compliance Pn](https://mintcdn.com/startree/xe5mTwlEdZc68KYh/getstarted/images/compliance.png?fit=max\&auto=format\&n=xe5mTwlEdZc68KYh\&q=85\&s=c74d1f270ae5b9d000b47dd20b8cd490) For more details, see: [https://startree.ai/security-compliance](https://startree.ai/security-compliance) # Products Source: https://docs.startree.ai/getstarted/products StarTree Cloud offers applications and utilities that increase usability and reduce the time to value for users. ## StarTree Data Portal ### Overview The StarTree Data Portal is the single, unified interface for managing and exploring Apache Pinot clusters on StarTree Cloud. It combines enterprise-grade features with Apache Pinot OSS UI capabilities into one seamless platform, eliminating the need to navigate between multiple tools. You can access the StarTree Data Portal directly at: [https://dp.your-environment.startree.cloud](https://dp.your-environment.startree.cloud) The Data Portal makes it easy to: * Ingest data into Pinot tables from a variety of streaming and batch sources via a guided, visual workflow. * Transform data with minimal complexity, reducing the risk of format mismatches, quality issues, and connectivity errors. * Explore and query real-time and offline data interactively. * Monitor cluster health and troubleshoot operational issues. * Secure access to data and resources with fine-grained RBAC controls. By merging all of this into one interface, the Data Portal serves as the one-stop solution for all things Pinot. ### Key Features #### Table Creation & Data Ingestion * Visual, guided workflows for creating real-time and offline tables. * Ingest from streaming sources (e.g., Confluent, Kafka, Kinesis) and batch sources (e.g., S3, GCS, ADLS). * Apply transformations to clean or enrich data before ingestion. * Detect and resolve issues early, such as schema mismatches, data quality problems, or connectivity errors. #### Query Console * Interactive query editor with syntax highlighting and auto-complete. * Live data preview with sub-second latency for fresh data. * Table browser to explore datasets, schemas, and metadata. * Query templates for saving, reusing, and sharing frequently used queries. * Performance insights with query latency and scanned record statistics. #### Cluster Health * Real-time views of broker, server, and controller status. #### Observability StarTree Cloud provides out-of-the-box support for system monitoring through Grafana dashboards. * Users can view various system metrics such as CPU and memory usage. * Monitor ingestion and query traffic patterns directly in Grafana. * Export both metrics and system logs to external destinations. #### Security Manager * The Security Manager configures fine-grained Role-Based Access Control (RBAC) roles and policies. * Assign roles and policies to Individual email IDs, Groups, and Service accounts * StarTree Cloud admins can govern access control for multiple business units through a single interface. #### Integrated Apache Pinot OSS UI Tools * **Cluster Manager** – Inspect and manage cluster nodes, broker, server, minions, tenants, and segments. Advanced capabilities to manage Pinot Tables, like reloading and rebalancing. * **Zookeeper Browser** – Navigate and view cluster metadata in Zookeeper. * **Swagger API Explorer** – Test and interact with Pinot REST APIs directly from the UI. ## StarTree ThirdEye With StarTree ThirdEye, users can create multi-dimensional real-time alerts on timeseries data stored in Pinot to detect anomalies in business events. For example, using ThirdEye you could detect if there is a sudden spike in incoming orders. ThirdEye also enables root cause analysis on the alert by identifying which dimension caused this spike. For example, the spike could originate from a certain country or demographic which will be automatically detected. **StarTree ThirdEye is not supported in the SaaS trial** For more information and to request a demo see: [https://startree.ai/products/startree-thirdeye](https://startree.ai/products/startree-thirdeye) # StarTree Documentation Source: https://docs.startree.ai/index Guides, concepts, and reference for StarTree Cloud, the real-time analytics platform built on Apache Pinot. # StarTree Documentation Got a question? Explore our knowledge base Search or ask... Popular Topics: [Kafka Ingestion](/corecapabilities/ingestdata/dataportal/streaming/kafka), [Transformations](/corecapabilities/ingestdata/dataportal/data-modeling/concepts#transform-data-with-functions), [Iceberg Tables](/corecapabilities/external-table/), [Release Notes](/reference/startree-release-notes) Ingest and Query Data > **[Ingestion](/corecapabilities/ingestdata/ingest_data_overview)** > > Ingest data from streaming and batch sources- Kafka, Confluent, S3, GCS, and many more > **[Data Modeling](/corecapabilities/ingestdata/dataportal/data-modeling/overview)** > > Structure data from the source through data transformation and by customizing columns > **[Query Data](/corecapabilities/query_data/query_interfaces/query_console)** > > Query data through the console, client libraries, or REST APIs Store and Manage Data > **[Indexes](/corecapabilities/manage-data/indexes/overview)** > > Use indexes to accelerate queries and reduce resource usage > **[Tiered Storage](/corecapabilities/manage-data/set-up-tiered-storage/architecture)** > > Save costs and improve performance by automatically optimizing data storage > **[Manage Security](/corecapabilities/security/manage-security)** > > Manage access to your environment through the Security Manager References and Releases #### [API Reference](/api-reference) StarTree Cloud APIs to manage tables and query data #### [Releases](/reference/startree-release-notes) What’s included in each StarTree Cloud release, bug fixes #### [Apache Pinot Recipes](/recipes/recipes) Example implementations of common tasks in Apache Pinot Explore more Learn more about StarTree #### [Community](https://inviter.co/startree-community) Join our Slack community for support and updates #### [Blogs](https://startree.ai/resources/category/blog) Latest insights from the real-time analytics world #### [Videos](https://www.youtube.com/startreedata) Quick tutorials, concepts, feature walkthroughs © 2025 StarTree Inc. All rights reserved. [Trademark Policy](https://startree.ai/startree-trademark-policy) [Privacy Policy](https://startree.ai/privacy-policy) [Terms of Use](https://startree.ai/terms-conditions) [Responsible Disclosure](https://startree.ai/responsible-disclosure) [https://www.linkedin.com/company/startreedata/](https://www.linkedin.com/company/startreedata/) [https://www.youtube.com/startreedata](https://www.youtube.com/startreedata) [https://twitter.com/startreedata](https://twitter.com/startreedata) # Batch Sources Source: https://docs.startree.ai/integrations/batch Batch sources copied into Pinot segments. Batch loads copied into Pinot segments. > **[Amazon S3](/corecapabilities/ingestdata/dataportal/batch/s3)** > > Create a connection to ingest files stored in Amazon S3 buckets. > **[Google Cloud Storage](/corecapabilities/ingestdata/dataportal/batch/gcs)** > > Create a connection to ingest files from Google Cloud Storage. > **[Azure ADLS](/corecapabilities/ingestdata/dataportal/batch/adls)** > > Create a connection to ingest files from Azure Data Lake Storage. > **[Delta Lake](/corecapabilities/ingestdata/dataportal/batch/deltalake)** > > Ingest from Delta Lake hosted in S3 or GCS using a custom connection. > **[Snowflake](/corecapabilities/ingestdata/dataportal/batch/snowflake)** > > Create a connection to load batch data from Snowflake into StarTree Cloud. > **[BigQuery](/corecapabilities/ingestdata/dataportal/batch/bigquery)** > > Create a connection to load data from Google BigQuery tables and views. > **[File Upload](/corecapabilities/ingestdata/dataportal/batch/upload)** > > Upload a file up to 30MB in AVRO, CSV, JSON, Parquet, ProtoBuf, or ORC. > **[Custom Connection](/corecapabilities/ingestdata/dataportal/custom-connection)** > > Connect to any batch or streaming data source using the custom connector. # Language Clients & APIs Source: https://docs.startree.ai/integrations/clients Query StarTree Cloud from your application. Query StarTree Cloud from your application. > **[Java](/corecapabilities/query_data/query_interfaces/connect-via-java)** > > Query your cluster with Pinot's native, tenant-aware Java client via Maven or Gradle. > **[Python](/corecapabilities/query_data/query_interfaces/connect-via-python)** > > Query StarTree Cloud from Python applications using the pinotdb client library. > **[Node.js](/corecapabilities/query_data/query_interfaces/connect-via-nodejs)** > > Query Pinot from Node.js applications using a sample axios-based client driver. > **[REST API](/corecapabilities/query_data/query_interfaces/query-using-rest-API)** > > Run SQL over HTTP by POSTing queries to the Pinot broker's /query/sql endpoint. > **[Pinot Proxy](/corecapabilities/query_data/query_interfaces/pinot-proxy)** > > Route queries to the right broker tenant and access advanced query, cursor, routing, and debug APIs. > **[StarTree MCP](/corecapabilities/query_data/mcp/startree-mcp)** > > Deploy an MCP server for agentic workflows and natural-language queries on Pinot tables. # Export Source: https://docs.startree.ai/integrations/export Write Pinot data back to your lake. Write Pinot data back to your lake. > **[Data Export Task](/corecapabilities/external-table/data-export-task)** > > Export completed Pinot REALTIME segments as Parquet to a filesystem or Iceberg REST catalog. # External Tables Source: https://docs.startree.ai/integrations/external-tables Iceberg catalogs and raw Parquet queried in place. Query Iceberg catalogs and raw Parquet in place — no copy, no ETL. > **[AWS Glue](/corecapabilities/external-table/glue/onboarding-data-portal)** > > Connect an AWS Glue Iceberg catalog as an External Table with the Data Portal wizard — no API calls. > **[Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-data-portal)** > > Connect an Amazon S3 Tables bucket as an External Table with the point-and-click Data Portal wizard. > **[Unity Catalog](/corecapabilities/external-table/unity/onboarding-data-portal)** > > Connect Databricks or OSS Unity Catalog, including Delta tables with UniForm, via the Data Portal wizard. > **[Apache Polaris & Snowflake Open Catalog](/corecapabilities/external-table/polaris/onboarding-data-portal)** > > Connect Polaris or Snowflake Open Catalog Iceberg tables via the wizard using OAuth2 principal credentials. > **[Snowflake Horizon Catalog](/corecapabilities/external-table/horizon/onboarding-data-portal)** > > Connect Snowflake-managed Iceberg tables (Horizon) via the Data Portal wizard, authenticating with a PAT. > **[S3 Data Lake](/corecapabilities/external-table/s3/onboarding-data-portal)** > > Query raw Parquet files in an S3 bucket as an External Table — no catalog service required. > **[GCS Data Lake](/corecapabilities/external-table/gcs/onboarding-data-portal)** > > Connect raw Parquet files on GCS via the S3-compatible interop endpoint using the Data Portal wizard. > **[Support Matrix](/corecapabilities/external-table/support-matrix)** > > See supported catalogs, table and file formats, auth methods, features, indexes, and limits in one place. # Message Formats & Decoders Source: https://docs.startree.ai/integrations/formats Payload formats on the streaming ingestion path. Payload formats supported on the streaming ingestion path. > **[JSON](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/json)** > > Decode JSON-formatted messages from Kafka topics into Pinot rows during real-time ingestion. > **[Avro](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/avro)** > > Ingest Avro-encoded Kafka messages using schemas managed in Confluent Schema Registry. > **[Protobuf](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/protobuf)** > > Ingest Protobuf-encoded Kafka messages with schemas from Confluent Schema Registry. > **[Debezium CDC](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/debezium)** > > Consume Debezium CDC data from MySQL or Postgres in real time via Kafka or Kinesis. > **[DynamoDB](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/dynamodb)** > > Ingest DynamoDB Streams CDC data in real time without complex transformations. > **[Prometheus](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/prometheus)** > > Ingest Prometheus-formatted metrics data natively into Pinot tables. > **[AWS Glue Schema Registry](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/glue)** > > Decode Avro messages from Kafka or MSK with schemas managed in AWS Glue Schema Registry. # Integrations Source: https://docs.startree.ai/integrations/home Connect StarTree to your streaming and batch sources, lakehouse catalogs, language clients, and BI tools — every integration in one place. Connect StarTree to your sources, lakehouse, clients, and BI tools. Every tile links to the full guide in its home section of the docs. ## Streaming Sources Real-time ingestion into native Pinot tables. > **[Apache Kafka](/corecapabilities/ingestdata/dataportal/streaming/kafka)** > > Create a connection to stream real-time events from Apache Kafka into StarTree. > **[Confluent Cloud](/corecapabilities/ingestdata/dataportal/streaming/confluent)** > > Connect to fully managed Kafka in Confluent Cloud and ingest with API key authentication. > **[AWS MSK](/corecapabilities/ingestdata/dataportal/streaming/aws_msk)** > > Connect to Amazon MSK using IAM-based SASL authentication over TLS. > **[Amazon Kinesis](/corecapabilities/ingestdata/dataportal/streaming/kinesis)** > > Create a connection to ingest real-time data from Amazon Kinesis streams. > **[Redpanda](/corecapabilities/ingestdata/dataportal/streaming/redpanda)** > > Create a connection for high-performance streaming ingestion from Redpanda. > **[StreamNative](/corecapabilities/ingestdata/dataportal/streaming/streamnative)** > > Ingest from fully managed Kafka in StreamNative using SASL\_SSL with PLAIN authentication. > **[Aiven](/corecapabilities/ingestdata/dataportal/streaming/aiven)** > > Create a connection to ingest data from Aiven's managed Apache Kafka service. > **[WarpStream](/corecapabilities/ingestdata/dataportal/streaming/warpstream)** > > Stream data into StarTree using WarpStream's Kafka-compatible API. ## Batch Sources Batch loads copied into Pinot segments. > **[Amazon S3](/corecapabilities/ingestdata/dataportal/batch/s3)** > > Create a connection to ingest files stored in Amazon S3 buckets. > **[Google Cloud Storage](/corecapabilities/ingestdata/dataportal/batch/gcs)** > > Create a connection to ingest files from Google Cloud Storage. > **[Azure ADLS](/corecapabilities/ingestdata/dataportal/batch/adls)** > > Create a connection to ingest files from Azure Data Lake Storage. > **[Delta Lake](/corecapabilities/ingestdata/dataportal/batch/deltalake)** > > Ingest from Delta Lake hosted in S3 or GCS using a custom connection. > **[Snowflake](/corecapabilities/ingestdata/dataportal/batch/snowflake)** > > Create a connection to load batch data from Snowflake into StarTree Cloud. > **[BigQuery](/corecapabilities/ingestdata/dataportal/batch/bigquery)** > > Create a connection to load data from Google BigQuery tables and views. > **[File Upload](/corecapabilities/ingestdata/dataportal/batch/upload)** > > Upload a file up to 30MB in AVRO, CSV, JSON, Parquet, ProtoBuf, or ORC. > **[Custom Connection](/corecapabilities/ingestdata/dataportal/custom-connection)** > > Connect to any batch or streaming data source using the custom connector. ## External Tables Query Iceberg catalogs and raw Parquet in place — no copy, no ETL. > **[AWS Glue](/corecapabilities/external-table/glue/onboarding-data-portal)** > > Connect an AWS Glue Iceberg catalog as an External Table with the Data Portal wizard — no API calls. > **[Amazon S3 Tables](/corecapabilities/external-table/s3tables/onboarding-data-portal)** > > Connect an Amazon S3 Tables bucket as an External Table with the point-and-click Data Portal wizard. > **[Unity Catalog](/corecapabilities/external-table/unity/onboarding-data-portal)** > > Connect Databricks or OSS Unity Catalog, including Delta tables with UniForm, via the Data Portal wizard. > **[Apache Polaris & Snowflake Open Catalog](/corecapabilities/external-table/polaris/onboarding-data-portal)** > > Connect Polaris or Snowflake Open Catalog Iceberg tables via the wizard using OAuth2 principal credentials. > **[Snowflake Horizon Catalog](/corecapabilities/external-table/horizon/onboarding-data-portal)** > > Connect Snowflake-managed Iceberg tables (Horizon) via the Data Portal wizard, authenticating with a PAT. > **[S3 Data Lake](/corecapabilities/external-table/s3/onboarding-data-portal)** > > Query raw Parquet files in an S3 bucket as an External Table — no catalog service required. > **[GCS Data Lake](/corecapabilities/external-table/gcs/onboarding-data-portal)** > > Connect raw Parquet files on GCS via the S3-compatible interop endpoint using the Data Portal wizard. > **[Support Matrix](/corecapabilities/external-table/support-matrix)** > > See supported catalogs, table and file formats, auth methods, features, indexes, and limits in one place. ## Message Formats & Decoders Payload formats supported on the streaming ingestion path. > **[JSON](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/json)** > > Decode JSON-formatted messages from Kafka topics into Pinot rows during real-time ingestion. > **[Avro](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/avro)** > > Ingest Avro-encoded Kafka messages using schemas managed in Confluent Schema Registry. > **[Protobuf](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/protobuf)** > > Ingest Protobuf-encoded Kafka messages with schemas from Confluent Schema Registry. > **[Debezium CDC](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/debezium)** > > Consume Debezium CDC data from MySQL or Postgres in real time via Kafka or Kinesis. > **[DynamoDB](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/dynamodb)** > > Ingest DynamoDB Streams CDC data in real time without complex transformations. > **[Prometheus](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/prometheus)** > > Ingest Prometheus-formatted metrics data natively into Pinot tables. > **[AWS Glue Schema Registry](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/glue)** > > Decode Avro messages from Kafka or MSK with schemas managed in AWS Glue Schema Registry. ## Language Clients & APIs Query StarTree Cloud from your application. > **[Java](/corecapabilities/query_data/query_interfaces/connect-via-java)** > > Query your cluster with Pinot's native, tenant-aware Java client via Maven or Gradle. > **[Python](/corecapabilities/query_data/query_interfaces/connect-via-python)** > > Query StarTree Cloud from Python applications using the pinotdb client library. > **[Node.js](/corecapabilities/query_data/query_interfaces/connect-via-nodejs)** > > Query Pinot from Node.js applications using a sample axios-based client driver. > **[REST API](/corecapabilities/query_data/query_interfaces/query-using-rest-API)** > > Run SQL over HTTP by POSTing queries to the Pinot broker's /query/sql endpoint. > **[Pinot Proxy](/corecapabilities/query_data/query_interfaces/pinot-proxy)** > > Route queries to the right broker tenant and access advanced query, cursor, routing, and debug APIs. > **[StarTree MCP](/corecapabilities/query_data/mcp/startree-mcp)** > > Deploy an MCP server for agentic workflows and natural-language queries on Pinot tables. ## Visualization & BI Connect BI tools to StarTree Cloud. > **[Tableau](/corecapabilities/visualize_data/connect-tableau-to-startree-cloud)** > > Connect Tableau Desktop or Tableau Server to StarTree Cloud using the JDBC driver and TACO connector. > **[Superset](/corecapabilities/visualize_data/connect-superset-to-startree-cloud)** > > Connect Apache Superset to StarTree Cloud with a SQLAlchemy URI to visualize your data. ## Export Write Pinot data back to your lake. > **[Data Export Task](/corecapabilities/external-table/data-export-task)** > > Export completed Pinot REALTIME segments as Parquet to a filesystem or Iceberg REST catalog. # Streaming Sources Source: https://docs.startree.ai/integrations/streaming Real-time sources that feed native Pinot tables. Real-time ingestion into native Pinot tables. > **[Apache Kafka](/corecapabilities/ingestdata/dataportal/streaming/kafka)** > > Create a connection to stream real-time events from Apache Kafka into StarTree. > **[Confluent Cloud](/corecapabilities/ingestdata/dataportal/streaming/confluent)** > > Connect to fully managed Kafka in Confluent Cloud and ingest with API key authentication. > **[AWS MSK](/corecapabilities/ingestdata/dataportal/streaming/aws_msk)** > > Connect to Amazon MSK using IAM-based SASL authentication over TLS. > **[Amazon Kinesis](/corecapabilities/ingestdata/dataportal/streaming/kinesis)** > > Create a connection to ingest real-time data from Amazon Kinesis streams. > **[Redpanda](/corecapabilities/ingestdata/dataportal/streaming/redpanda)** > > Create a connection for high-performance streaming ingestion from Redpanda. > **[StreamNative](/corecapabilities/ingestdata/dataportal/streaming/streamnative)** > > Ingest from fully managed Kafka in StreamNative using SASL\_SSL with PLAIN authentication. > **[Aiven](/corecapabilities/ingestdata/dataportal/streaming/aiven)** > > Create a connection to ingest data from Aiven's managed Apache Kafka service. > **[WarpStream](/corecapabilities/ingestdata/dataportal/streaming/warpstream)** > > Stream data into StarTree using WarpStream's Kafka-compatible API. # Visualization & BI Source: https://docs.startree.ai/integrations/visualization Connect BI tools to StarTree Cloud. Connect BI tools to StarTree Cloud. > **[Tableau](/corecapabilities/visualize_data/connect-tableau-to-startree-cloud)** > > Connect Tableau Desktop or Tableau Server to StarTree Cloud using the JDBC driver and TACO connector. > **[Superset](/corecapabilities/visualize_data/connect-superset-to-startree-cloud)** > > Connect Apache Superset to StarTree Cloud with a SQLAlchemy URI to visualize your data. # Backfill offline segment Source: https://docs.startree.ai/recipes/backfill In this Apache Pinot recipe we'll learn how to backfill a segment moved from a real-time to offline table using the real-time to offline job. In this recipe we'll learn how to backfill a segment moved from a real-time to offline table using the [real-time to offline job](/recipes/real-time-offline-job). | Pinot Version | 0.9.3 | | - | - | | Code | [startreedata/pinot-recipes/backfill](https://github.com/startreedata/pinot-recipes/tree/main/recipes/backfill) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, run the following command to navigate to the recipe: ```sh theme={null} cd pinot-recipes/recipes/backfill ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` or (if you're using a Mac M1/M2) ```bash theme={null} docker-compose -f docker-compose-m1.yml up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Pinot Minion, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/backfill/docker-compose.yml) file on GitHub. ## Pinot Schema and Tables Now let's create a Pinot Schema, as well as real-time and offline tables. Pinot is going to take care of populating data into the offline table, but it still expects us to configure the table. ### Schema Our schema is going to capture some order events, and looks like this: ```json theme={null} { "schemaName": "orders", "dimensionFieldSpecs": [{ "name": "order_id", "dataType": "INT" }, { "name": "customer_id", "dataType": "INT" }, { "name": "order_status", "dataType": "STRING" } ], "metricFieldSpecs": [{ "name": "amount", "dataType": "FLOAT" }], "dateTimeFieldSpecs": [{ "name": "ts", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` *config/orders\_schema.json* ### Real-Time Table And the real-time table is defined below: ```json theme={null} { "tableName": "orders", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "timeType": "MILLISECONDS", "segmentPushType": "APPEND", "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy", "schemaName": "orders", "replicasPerPartition": "1" }, "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bucketTimePeriod": "2h", "bufferTimePeriod": "1m" } } }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowLevel", "stream.kafka.topic.name": "orders", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9093", "realtime.segment.flush.threshold.rows": 5 } }, "metadata": { "customConfigs": {} }, "routing": { "instanceSelectorType": "strictReplicaGroup" } } ``` *config/orders\_table.json* > [!WARNING] > > The `realtime.segment.flush.threshold.rows` config is intentionally set to an extremely small value so that the segment will be committed after 5 records have been ingested. > In a production system this value should be set much higher, as described in the [configuring segment threshold](/recipes/configuring-segment-threshold) guide. You can create the table and schema by running the following command: ```bash theme={null} docker run \ --network backfill \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/orders_schema.json \ -tableConfigFile /config/orders_table.json \ -controllerHost "pinot-controller" \ -exec ``` ### Offline Table The offline table config is defined below: ```json theme={null} { "tableName": "events", "tableType": "OFFLINE", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "HOURLY" } }, "tableIndexConfig": { "loadMode": "MMAP" }, "tenants": {}, "metadata": {} } ``` *config/orders\_offline\_table.json* You can create this table by running the following command: ```bash theme={null} docker run \ --network backfill \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/orders_schema.json \ -tableConfigFile /config/orders_offline_table.json \ -controllerHost "pinot-controller" \ -update -exec ``` ## Ingesting Data Let's ingest data into the `events` Kafka topic, by running the following: ```bash theme={null} docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server kafka:9092 --topic orders {"order_id":1,"customer_id":101,"order_status":"OPEN","amount":13.29,"ts":"1632463351000"} {"order_id":2,"customer_id":102,"order_status":"IN_TRANSIT","amount":209.35,"ts":"1632463361000"} {"order_id":3,"customer_id":103,"order_status":"COMPLETED","amount":199.35,"ts":"1632463391000"} {"order_id":4,"customer_id":105,"order_status":"COMPLETED","amount":3.24,"ts":"1632467065000"} {"order_id":5,"customer_id":103,"order_status":"OPEN","amount":9.77,"ts":"1632467066000"} {"order_id":6,"customer_id":104,"order_status":"OPEN","amount":55.52,"ts":"1632467068000"} {"order_id":7,"customer_id":104,"order_status":"CANCELLED","amount":52.54,"ts":"1632467070000"} {"order_id":8,"customer_id":105,"order_status":"OPEN","amount":13.29,"ts":"1632667070000"} {"order_id":9,"customer_id":105,"order_status":"IN_TRANSIT","amount":2.92,"ts":"1632667170000"} {"order_id":10,"customer_id":105,"order_status":"COMPLETED","amount":12.22,"ts":"1632677270000"} {"order_id":11,"customer_id":106,"order_status":"OPEN","amount":13.94,"ts":"1632677270400"} {"order_id":12,"customer_id":107,"order_status":"OPEN","amount":20.32,"ts":"1632677270403"} {"order_id":13,"customer_id":108,"order_status":"OPEN","amount":45.11,"ts":"1632677270508"} {"order_id":14,"customer_id":109,"order_status":"OPEN","amount":129.22,"ts":"1632677270699"} ``` Data will make its way into the real-time table. We can see how many records have been ingested by running the following query: ```sql theme={null} select * from orders order by order_id limit 10 ``` ![Orders query results](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/recipes/images/backfill-initial-query.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=ad7b297460ef2bb96f7742a580271368) *Orders query results* ## Scheduling the RT2OFF Job The Real-Time to Offline Job can be scheduled automatically via the real-time table config or manually via the REST API. We can trigger it manually by running the following command: ```bash theme={null} table="orders_REALTIME" curl -X POST \ "http://localhost:9000/tasks/schedule?taskType=RealtimeToOfflineSegmentsTask&tableName=${table}" \ -H "accept: application/json" 2>/dev/null | jq '.' ``` **Output** ```json theme={null} { "RealtimeToOfflineSegmentsTask": "Task_RealtimeToOfflineSegmentsTask_1647620599577" } ``` We can then check the Pinot Controller logs to see that it's been triggered: ```bash theme={null} docker exec -it pinot-controller grep -i "\[RealtimeToOff" logs/pinot-all.log ``` **Output** ```text theme={null} 2022/03/21 10:58:29.746 INFO [RealtimeToOfflineSegmentsTaskGenerator] [grizzly-http-server-1] Start generating task configs for table: orders_REALTIME for task: RealtimeToOfflineSegmentsTask 2022/03/21 10:58:29.763 INFO [RealtimeToOfflineSegmentsTaskGenerator] [grizzly-http-server-1] Finished generating task configs for table: orders_REALTIME for task: RealtimeToOfflineSegmentsTask ``` Now let's navigate to [localhost:9000/#/tables](http://localhost:9000/#/tables). You'll see the following: ![](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/rt2off-backfill.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=e06546e93a651817bd8385630a645034) Real-Time and Offline Tables You can see that a segment has been created in the offline table: ## Viewing offline segment You can list all the segments for a table by making the following request to the HTTP API: ```bash theme={null} table="orders" curl -X GET "http://localhost:9000/segments/${table}" \ -H "accept: application/json" 2>/dev/null | jq '.' ``` **Output** ```json theme={null} [ { "OFFLINE": [ "orders_1632463351000_1632467070000_0" ] }, { "REALTIME": [ "orders__0__0__20220321T0941Z", "orders__0__1__20220321T0941Z", "orders__0__2__20220321T0941Z" ] } ] ``` We have one offline segment - `orders_1632463351000_1632467070000_0`. You can check which records it contains by running the following query: ```sql theme={null} select * from orders_OFFLINE order by order_id limit 10 ``` ![Orders offline table query results](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/recipes/images/backfill-offline.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=ef6e3f5dd68182583f1ff29e245e1a1f) *Orders offline table query results* The offline table contains only the first 7 records that we ingested. ## Replacing offline segment Let's now backfill those 7 records to increase the value in the amount column by 20%. The documents with the updated amount value are in `data/orders.json`, shown below: ```json theme={null} {"order_id": 1, "customer_id": 101, "order_status": "OPEN", "amount": 15.947999999999999, "ts": "1632463351000"} {"order_id": 2, "customer_id": 102, "order_status": "IN_TRANSIT", "amount": 251.21999999999997, "ts": "1632463361000"} {"order_id": 3, "customer_id": 103, "order_status": "COMPLETED", "amount": 239.21999999999997, "ts": "1632463391000"} {"order_id": 4, "customer_id": 105, "order_status": "COMPLETED", "amount": 3.888, "ts": "1632467065000"} {"order_id": 5, "customer_id": 103, "order_status": "OPEN", "amount": 11.723999999999998, "ts": "1632467066000"} {"order_id": 6, "customer_id": 104, "order_status": "OPEN", "amount": 66.624, "ts": "1632467068000"} {"order_id": 7, "customer_id": 104, "order_status": "CANCELLED", "amount": 63.047999999999995, "ts": "1632467070000"} ``` *data/orders.json* We'll ingest this file using the following ingestion spec: ```yaml {5-8} theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentNameGeneratorSpec: type: fixed configs: segment.name: ${segmentName} jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/orders.json' outputDirURI: '/opt/pinot/data/orders/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: 'orders' pinotClusterSpecs: - controllerURI: '${pinotController}' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/job-spec.yml* Now let's run the job to replace segment `orders_1632463351000_1632467070000_0`: ```bash theme={null} docker run \ --network backfill \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml \ -values segmentName='orders_1632463351000_1632467070000_0' \ -values pinotController=http://pinot-controller:9000 ``` Once we've done that, we need to update the time boundary so that it starts from the latest time in the offline table: ```bash theme={null} curl -X POST \ "http://localhost:9000/tables/orders/timeBoundary" \ -H "accept: application/json" ``` We can re-run the query on the `orders` table: ```sql theme={null} select $segmentName, * from orders ORDER BY order_id limit 20 ``` And we'll see that the `amount` column for the first 7 orders has been updated: | \$segmentName | amount | customer\_id | order\_id | order\_status | ts | | - | - | - | - | - | - | | orders\_1632463351000\_1632467070000\_0 | 15.948 | 101 | 1 | OPEN | 1632463351000 | | orders\_1632463351000\_1632467070000\_0 | 251.22 | 102 | 2 | IN\_TRANSIT | 1632463361000 | | orders\_1632463351000\_1632467070000\_0 | 239.22 | 103 | 3 | COMPLETED | 1632463391000 | | orders\_1632463351000\_1632467070000\_0 | 3.888 | 105 | 4 | COMPLETED | 1632467065000 | | orders\_1632463351000\_1632467070000\_0 | 11.724 | 103 | 5 | OPEN | 1632467066000 | | orders\_1632463351000\_1632467070000\_0 | 66.624 | 104 | 6 | OPEN | 1632467068000 | | orders\_1632463351000\_1632467070000\_0 | 63.048 | 104 | 7 | CANCELLED | 1632467070000 | | orders\_\_0\_\_1\_\_20230321T1038Z | 13.29 | 105 | 8 | OPEN | 1632667070000 | | orders\_\_0\_\_1\_\_20230321T1038Z | 2.92 | 105 | 9 | IN\_TRANSIT | 1632667170000 | | orders\_\_0\_\_1\_\_20230321T1038Z | 12.22 | 105 | 10 | COMPLETED | 1632677270000 | | orders\_\_0\_\_2\_\_20230321T1038Z | 13.94 | 106 | 11 | OPEN | 1632677270400 | | orders\_\_0\_\_2\_\_20230321T1038Z | 20.32 | 107 | 12 | OPEN | 1632677270403 | | orders\_\_0\_\_2\_\_20230321T1038Z | 45.11 | 108 | 13 | OPEN | 1632677270508 | | orders\_\_0\_\_2\_\_20230321T1038Z | 129.22 | 109 | 14 | OPEN | 1632677270699 | *Query Results* # Chaining Transformation Functions Source: https://docs.startree.ai/recipes/chaining-transformation-functions In this recipe we'll learn how to chain transformation functions during the data ingestion process. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/chaining-transformation-functions](https://github.com/startreedata/pinot-recipes/tree/main/recipes/chaining-transformation-functions) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/chaining-transformation-functions ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/chaining-transformation-functions/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following JSON file: ```json theme={null} {"payload": {"userId": "3287651__David Smith"}} {"payload": {"userId": "4987622__Jenny Jones"}} {"payload": {"userId": "1965900__Stephen Davis"}} ``` *data/import.json* We're going to pull apart the numeric and string parts of the `userId` and store them in individual columns. ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName":"people", "dimensionFieldSpecs":[ { "name":"userIdString", "dataType":"STRING" }, { "name":"id", "dataType":"INT" }, { "name":"name", "dataType":"STRING" } ] } ``` *config/schema.json* The `userId` column will store the `userId` value from the JSON document. The `name` and `id` columns will store values extracted from the `userId`. We'll also have the following table config: ```json {9-22} theme={null} { "tableName":"people", "tableType":"OFFLINE", "segmentsConfig":{ "replication":1, "schemaName":"people" }, "ingestionConfig":{ "transformConfigs":[ { "columnName":"userId", "transformFunction":"jsonPathString(payload, '$.userId')" }, { "columnName":"id", "transformFunction":"Groovy({Long.valueOf(userId.split('__')[0])}, userId)" }, { "columnName":"name", "transformFunction":"Groovy({userId.split('__')[1]}, userId)" } ], "batchIngestionConfig":{ "segmentIngestionType":"APPEND", "segmentIngestionFrequency":"DAILY" } }, "tenants":{ "broker":"DefaultTenant", "server":"DefaultTenant" }, "tableIndexConfig":{ "loadMode":"MMAP" }, "metadata":{} } ``` *config/table.json* In this config we define transform configs (`ingestionConfig.transformConfigs`) that do the following: * Extract `payload.userId` using the [jsonPathString](https://docs.pinot.apache.org/configuration-reference/functions/jsonPathString) function. * Split the corresponding string on `__` and extracting the `id` and `name` using Groovy transformation functions. You can create the table and schema by running the following command:\` ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-json" \ -exec ``` You should see a message similar to the following if everything is working correctly: ```text theme={null} 2022/02/28 10:08:40.333 INFO [AddTableCommand] [main] Executing command: AddTable -tableConfigFile /config/table.json -schemaFile /config/schema.json -controllerProtocol http -controllerHost 172.31.0.3 -controllerPort 9000 -user null -password [hidden] -exec 2022/02/28 10:08:40.747 INFO [AddTableCommand] [main] {"status":"Table people_OFFLINE succesfully added"} ``` ## Ingestion Job Now we’re going to import the JSON file into Pinot. We'll do this with the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/import.json' outputDirURI: '/opt/pinot/data/people/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: 'people' pinotClusterSpecs: - controllerURI: 'http://pinot-controller-json:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/job-spec.yml* The import job will map fields in each JSON document to a corresponding column in the `people` schema. If one of the fields doesn't exist in the schema it will be skipped. In this case our JSON documents only have one top level field, `payload`, which doesn't have a corresponding column in the schema. Instead, transformation functions extract the `payload.userId` field and then store parts of it in different columns. You can run the following command to run the import: ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `people` table or copy/paste the following query: ```sql theme={null} select * from people limit 10 ``` You will see the following output: | id | name | userId | | - | - | - | | 3287651 | David Smith | 3287651\_\_David Smith | | 4987622 | Jenny Jones | 4987622\_\_Jenny Jones | | 1965900 | Stephen Davis | 1965900\_\_Stephen Davis | *Query Results* # How to Combine Source Fields into one Column Source: https://docs.startree.ai/recipes/combine-fields In this recipe we'll learn how to combine the data from fields in our data source into a single column in Apache Pinot. | Pinot Version | 0.10.0 | | - | - | | Code | [startreedata/pinot-recipes/combine-fields](https://github.com/startreedata/pinot-recipes/tree/main/recipes/combine-fields) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/combine-fields ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/json-unnest/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following JSON file: ```json theme={null} {"name":"Pete", "surname": "Smith"} {"name":"John", "surname": "Jones"} ``` *data/movies.json* ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "people", "dimensionFieldSpecs": [ { "name": "fullName", "dataType": "STRING" } ] } ``` *config/schema.json* You can create the schema by running the following command: ```bash theme={null} docker exec -it pinot-controller-json bin/pinot-admin.sh AddSchema \ -schemaFile /config/schema.json \ -exec ``` We'll also have the following table config: ```json {9-14} theme={null} { "tableName": "people", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1, "schemaName": "people" }, "ingestionConfig": { "transformConfigs": [ { "columnName": "fullName", "transformFunction": "concat(name, surname, ' ')" } ], "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "metadata": {} } ``` *config/table.json* The highlighted section contains a [transformation function](https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations) that concatenates the `name` and `surname` fields, separated by a space. You can create the table by running the following command:\` ```bash theme={null} docker exec -it pinot-controller-json bin/pinot-admin.sh AddTable \ -tableConfigFile /config/table.json \ -exec ``` ## Ingestion Job Now we’re going to import the JSON file into Pinot. We'll do this with the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/import.json' outputDirURI: '/opt/pinot/data/people/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: 'people' pinotClusterSpecs: - controllerURI: 'http://pinot-controller-combine:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/job-spec.yml* You can run the following command to run the import: ```bash theme={null} docker exec -it pinot-controller-json bin/pinot-admin.sh LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `people` table or copy/paste the following query: ```sql theme={null} select * from people ``` You will see the following output: | fullName | | - | | Pete Smith | | John Jones | *Query Results* We can see that the `name` and `surname` fields from our JSON file have been combined into a single `fullName` column for each person. # Configuring the Segment Threshold Source: https://docs.startree.ai/recipes/configuring-segment-threshold In this guide we'll learn how to configure the segment threshold for flushing data in Apache Pinot real-time tables. To learn how to configure the segment threshold for real-time tables, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/qBMv3CcKVsI) | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/configuring-segment-threshold](https://github.com/startreedata/pinot-recipes/tree/main/recipes/configuring-segment-threshold) | > [!NOTE] > > To learn what the segment threshold is and why it's important, see [Segment Threshold](https://dev.startree.ai/docs/pinot/concepts/segment-threshold). ## Configuration parameters The segment threshold is configured using the following parameters: | Property | Description | | - | - | | `realtime.segment.flush.threshold.rows` | Row count flush threshold. | | `realtime.segment.flush.threshold.time` | Time threshold that will keep a segment open for it is flushed. | | `realtime.segment.flush.threshold.segment.size` | The desired size of a completed segment. | There are two main approaches for configuring these parameters: ### Desired row threshold We can define a manual row threshold by specifying a value for `realtime.segment.flush.threshold.rows`. Pinot will complete/flush segments as soon as the consuming segment contains the specified number of rows. This will generally result in each segment having the same number of rows. However, if the time threshold defined by `realtime.segment.flush.threshold.time` is reached, a segment will be completed even if the row count flush threshold has not yet been reached. > [!NOTE] > > If *realtime.segment.flush.threshold.rows* is set to a value greater than 0, *realtime.segment.flush.threshold.segment.size* is ignored. ### Desired segment size Alternatively we can set `realtime.segment.flush.threshold.rows` to `0`, in which case Pinot will instead attempt to make sure that every segment has the desired size defined by `realtime.segment.flush.threshold.segment.size`. When configuring the segment threshold this way, the minimum number of rows in a segment is 10,000. The first segment for a new partition will have 100,000 rows. For subsequent segments Pinot will slowly adjust the number of rows to get closer to the desired segment size. This means that the first few segments might differ in size, but over time the segment size will approach the desired size. > [!NOTE] > > The algorithm used in this approach is described in more detail in the [Auto-tuning Pinot real-time consumption](https://engineering.linkedin.com/blog/2019/auto-tuning-pinot) blog post. ## A worked example Let's see how this works with a worked example. ### Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ### Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/configuring-segment-threshold ``` ### Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/configuring-segment-threshold/docker-compose.yml) file on GitHub. ### Pinot Schema and Table Let's create a Pinot Schema and Table. The schema is defined below: ```json theme={null} { "schemaName":"events", "dimensionFieldSpecs":[ { "name":"uuid", "dataType":"STRING" } ], "metricFieldSpecs":[ { "name":"count", "dataType":"INT" } ], "dateTimeFieldSpecs":[ { "name":"ts", "dataType":"TIMESTAMP", "format":"1:MILLISECONDS:EPOCH", "granularity":"1:MILLISECONDS" } ] } ``` *config/schema.json* And the table config below: ```json {20-22} theme={null} { "tableName":"events", "tableType":"REALTIME", "segmentsConfig":{ "timeColumnName":"ts", "schemaName":"events", "replication":"1", "replicasPerPartition":"1" }, "tableIndexConfig":{ "loadMode":"MMAP", "streamConfigs":{ "streamType":"kafka", "stream.kafka.topic.name":"events", "stream.kafka.broker.list":"kafka-segment:9093", "stream.kafka.consumer.type":"lowlevel", "stream.kafka.consumer.prop.auto.offset.reset":"smallest", "stream.kafka.consumer.factory.class.name":"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name":"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows":"0", "realtime.segment.flush.threshold.time":"1h", "realtime.segment.flush.threshold.segment.size":"5M" } }, "ingestionConfig":{ "batchIngestionConfig":{ "segmentIngestionType":"APPEND", "segmentIngestionFrequency":"DAILY" } }, "tenants":{ }, "metadata":{ } } ``` *config/table.json* Pinot will try to ensure that the size of our segments is 5MB. It will complete a segment after 1 hour if it hasn't already been flushed. Create the table and schema by running the following command: ```bash theme={null} docker run \ --network segment \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-segment" \ -exec ``` ### Ingesting Data Next, we're going to ingest some data into Kafka: ```bash theme={null} python datagen.py | kcat -P -b localhost:9092 -t events ``` ### Inspecting segment sizes After the ingestion script has run for a while we can inspect the size of the segments that have been completed. You can return the size of each completed segment by running the following command: ```bash theme={null} curl -X GET "http://localhost:9000/tables/events/size?detailed=true" \ -H "accept: application/json" 2>/dev/null | jq -r '.realtimeSegments .segments | .[] | .serverInfo | .[]' | jq -sc 'sort_by(.segmentName) | .[]' ``` **Output** ```json theme={null} {"segmentName":"events__0__0__20220301T1457Z","diskSizeInBytes":5144515} {"segmentName":"events__0__1__20220301T1504Z","diskSizeInBytes":5242744} {"segmentName":"events__0__2__20220301T1511Z","diskSizeInBytes":5242744} {"segmentName":"events__0__3__20220301T1518Z","diskSizeInBytes":5242744} {"segmentName":"events__0__4__20220301T1525Z","diskSizeInBytes":5242744} {"segmentName":"events__0__5__20220301T1532Z","diskSizeInBytes":5242744} {"segmentName":"events__0__6__20220301T1539Z","diskSizeInBytes":5242797} {"segmentName":"events__0__7__20220301T1546Z","diskSizeInBytes":5242797} ``` We can see that all the segments have a size of just over 5 million bytes, which is close to 5 MB. The first segment had a size of 4.9 MB, but after that the size increased to 4.9998 MB until the last two segments where it's increased to 4.9999 MB. Presumably adding an extra row would then take us further away from 5 MB. Now let's see how many rows are contained in each segment. You can do that for one of the segments by running the following command: ```bash theme={null} curl -X GET "http://localhost:9000/segments/events/events__0__1__20220301T1504Z/metadata" \ -H "accept: application/json" 2>/dev/null ``` **Output** ```json theme={null} { "segment.crc": "1090576205", "segment.creation.time": "1646147045318", "segment.end.time": "1646147463422", "segment.flush.threshold.size": "101912", "segment.index.version": "v3", "segment.name": "events__0__1__20220301T1504Z", "segment.realtime.download.url": "http://192.168.0.3:9000/segments/events/events__0__1__20220301T1504Z", "segment.realtime.endOffset": "201912", "segment.realtime.numReplicas": "1", "segment.realtime.startOffset": "100000", "segment.realtime.status": "DONE", "segment.start.time": "1646147042226", "segment.table.name": "events", "segment.time.unit": "MILLISECONDS", "segment.total.docs": "101912", "segment.type": "REALTIME" } ``` The `segment.flush.threshold.size` property indicates that this segment contains 101,912 rows. We can check how many rows are stored in all segments by running the following script: ```bash theme={null} segments=$( curl "http://localhost:9000/tables/events/size?detailed=true" 2>/dev/null | jq -r '.realtimeSegments .segments | .[] | .serverInfo | .[]' | jq -sc 'sort_by(.segmentName)' ) for segment in $(echo $segments | jq -c '.[]'); do segmentName=$(echo $segment | jq -r '.segmentName') rows=$( curl "http://localhost:9000/segments/events/${segmentName}/metadata" 2>/dev/null | jq '."segment.flush.threshold.size" | tonumber' ) echo $segment | jq -c --arg rows $rows '. + {"rows": $rows}' done ``` **Output** ```json theme={null} {"segmentName":"events__0__0__20220301T1457Z","diskSizeInBytes":5144515,"rows":"100000"} {"segmentName":"events__0__1__20220301T1504Z","diskSizeInBytes":5242744,"rows":"101912"} {"segmentName":"events__0__2__20220301T1511Z","diskSizeInBytes":5242744,"rows":"101912"} {"segmentName":"events__0__3__20220301T1518Z","diskSizeInBytes":5242744,"rows":"101912"} {"segmentName":"events__0__4__20220301T1525Z","diskSizeInBytes":5242744,"rows":"101912"} {"segmentName":"events__0__5__20220301T1532Z","diskSizeInBytes":5242744,"rows":"101912"} {"segmentName":"events__0__6__20220301T1539Z","diskSizeInBytes":5242797,"rows":"101913"} {"segmentName":"events__0__7__20220301T1546Z","diskSizeInBytes":5242797,"rows":"101913"} ``` It looks like Pinot has settled in on a size of 101,913 rows per segment. The number of rows would be reduced if we imported much bigger strings into the `uuid` column, but for now it looks fairly stable. # Importing CSV files with columns containing spaces Source: https://docs.startree.ai/recipes/csv-files-spaces-column-names Pinot can transform data at ingestion. In this recipe, we'll learn how to use a transformation to change the name of a column. We will ingest a CSV file with a column containing spaces in its name. We will then use a transformation function to remove the space while the data is being ingested into Pinot. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/csv-files-spaces-column-names](https://github.com/startreedata/pinot-recipes/tree/main/recipes/csv-files-spaces-column-names) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In the terminal, navigate to this recipe's directory: ```bash theme={null} cd pinot-recipes/recipes/csv-files-spaces-column-names ``` ## Launch Pinot Cluster Launch a Pinot Cluster: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find and examine the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/csv-files-spaces-column-names/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following CSV file, in which the `Case Number` column heading contains a space: | ID | Case Number | | - | - | | 10224738 | HY411648 | | 10224739 | HY411615 | | 11646166 | JC213529 | | 10224740 | HY411595 | *data/import.csv* ## Pinot Schema and Table Next we create a Pinot Schema and Table. A common pattern when creating a schema is to create columns that map directly to the names of the fields in our data source. We can't do that in this case since column names can't contain spaces, so instead we'll use the following: ```json theme={null} { "schemaName": "crimes", "dimensionFieldSpecs": [ { "name": "ID", "dataType": "INT" }, { "name": "CaseNumber", "dataType": "STRING" } ] } ``` *config/schema.json* We'll also have the following table config: ```json theme={null} { "tableName": "crimes", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1 }, "tenants": { "broker":"DefaultTenant", "server":"DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" }, "transformConfigs": [ {"columnName": "CaseNumber", "transformFunction": "\"Case Number\"" } ] }, "metadata": {} } ``` *config/table.json* > [!NOTE] > > The entry under `ingestionConfig.transformConfigs` makes sure that data in the `Case Number` field in the data source is ingested into the `CaseNumber` column of the table. To learn more about writing these functions, see the [ingestion transformation](https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations) documentation. Create the table and schema by running the following command: ```bash theme={null} docker run \ --network csv \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-csv" \ -exec ``` You should see a message similar to the following if everything is working correctly: ```text theme={null} 2021/11/25 12:02:04.606 INFO [AddTableCommand] [main] Executing command: AddTable -tableConfigFile /config/table.json -schemaFile /config/schema.json -controllerProtocol http -controllerHost 192.168.144.3 -controllerPort 9000 -user null -password [hidden] -exec 2021/11/25 12:02:05.084 INFO [AddTableCommand] [main] {"status":"Table crimes_OFFLINE succesfully added"} ``` ## Ingestion Job Next, we import the CSV file into Pinot. We'll do this with the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/import.csv' outputDirURI: '/opt/pinot/data/crimes/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'crimes' pinotClusterSpecs: - controllerURI: 'http://pinot-controller-csv:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/job-spec.yml* Run the following command to run the import: ```bash theme={null} docker run \ --network csv \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `crimes` table or copy/paste the following query: ```sql theme={null} select * from crimes limit 10 ``` You will see the following output: | CaseNumber | ID | | | - | - | - | | HY411648 | 10224738 | | | HY411615 | 10224739 | | | JC213529 | 11646166 | | | HY411595 | 10224740 | | *Query Results* # Converting DateTime Strings to Timestamps Source: https://docs.startree.ai/recipes/datetime-string-to-timestamp In this recipe we'll learn how to import DateTime strings into Pinot. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/datetime-string-to-timestamp](https://github.com/startreedata/pinot-recipes/tree/main/recipes/datetime-string-to-timestamp) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/datetime-string-to-timestamp ``` ## Launch Pinot Cluster and Kafka You can spin up a Pinot Cluster and Kafka Broker by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/datetime-string-to-timestamp/docker-compose.yml) file on GitHub. ## DateTime Strings in Pinot Pinot automatically creates Timestamps from DateTime strings that are in the following format: * `yyyy-mm-dd hh:mm:ss[.fffffffff]` * Millis since epoch If we have a DateTime value in another format we'll need to convert that value using a transformation function. We're going to import the following message into Kafka: ```json theme={null} { "timestamp1": "2019-10-09 22:25:25", "timestamp2": "1570656325000", "timestamp3": "10/09/2019T22:25:25" } ``` ## Add Schema and Table Now let's create a Pinot Schema and Table. ```json theme={null} { "schemaName": "dates", "dimensionFieldSpecs": [ {"name": "Date1", "dataType": "TIMESTAMP"}, {"name": "Date2", "dataType": "TIMESTAMP"}, {"name": "Date3", "dataType": "TIMESTAMP"} ], "dateTimeFieldSpecs": [ { "name": "Date4", "dataType": "TIMESTAMP", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` *config/schema.json* ```json {15-20} theme={null} { "tableName":"dates", "tableType":"REALTIME", "segmentsConfig":{ "timeColumnName":"Date4", "schemaName":"dates", "replication":"1", "replicasPerPartition":"1" }, "ingestionConfig":{ "batchIngestionConfig":{ "segmentIngestionType":"APPEND", "segmentIngestionFrequency":"DAILY" }, "transformConfigs":[ {"columnName":"Date1", "transformFunction":"timestamp1"}, {"columnName":"Date2", "transformFunction":"timestamp2"}, {"columnName":"Date3", "transformFunction": "FromDateTime(timestamp3, 'MM/dd/yyyy''T''HH:mm:ss')"}, {"columnName":"Date4", "transformFunction":"timestamp1"} ] }, "tableIndexConfig":{ "loadMode":"MMAP", "streamConfigs":{ "streamType":"kafka", "stream.kafka.topic.name":"dates", "stream.kafka.broker.list":"kafka-datetime:9093", "stream.kafka.consumer.type":"lowlevel", "stream.kafka.consumer.prop.auto.offset.reset":"smallest", "stream.kafka.consumer.factory.class.name":"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name":"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder" } }, "tenants":{}, "metadata":{} } ``` *config/table.json* This table has multiple transform configs defined that: * Map `timestamp1` from the data source into the `Date1` column * Map `timestamp2` from the data source into the `Date2` column * Map `timestamp3` from the data source into the `Date3` column, with help from the function. * Map `timestamp4` from the data source into the `Date1` column You can create the schema and table by running the following command: ```bash theme={null} docker run \ --network datetime \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -tableConfigFile /config/table.json \ -schemaFile /config/schema.json \ -controllerHost "pinot-controller-datetime" \ -exec ``` ## Ingesting Data into Kafka We can ingest a message into Kafka by running the following command: ````bash theme={null} ```bash printf '{"timestamp1": "2019-10-09 22:25:25", "timestamp2": "1570656325000", "timestamp3": "10/09/2019T22:25:25"}\n' | kcat -P -b localhost:9092 -t dates ```` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the dates table or copy/paste the following query: ```sql theme={null} select * from dates limit 10 ``` You will see the following output: | Date1 | Date2 | Date3 | Date4 | | - | - | - | - | | 2019-10-09 22:25:25.0 | 2019-10-09 21:25:25.0 | 2019-10-09 22:25:25.0 | 2019-10-09 22:25:25.0 | *Query Results* # Ingest CDC Data using PostgreSQL, Kafka, and Debezium Source: https://docs.startree.ai/recipes/debezium-cdc This recipe downloads Postgres change data capture (CDC) about `dvdrentals` and connects Kafka to Debezium. CDC data is written to Kafka, and then consumed by Apache Pinot. To understand how this recipe processes data, examine the commands executed in the Makefile. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/debezium-cdc](https://github.com/startreedata/pinot-recipes/tree/main/recipes/debezium-cdc) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In the terminal, navigate to this recipe's directory: ```bash theme={null} cd pinot-recipes/recipes/debezium-cdc ``` 1. Spin up a Pinot cluster using the Makefile, which uses Docker compose: ```bash theme={null} make recipe ``` 1. Navigate to [localhost:9000/#/query](http://localhost:9000/#/query) to see the data in Apache Pinot. ## Clean up ```bash theme={null} make clean ``` ## Troubleshooting To clean up old Docker installations that may be interfering with your testing of this recipe, run the following command: ```bash theme={null} docker system prune ``` # Replicating DynamoDB to Apache Pinot Source: https://docs.startree.ai/recipes/dynamodb-pinot-cdc In this developer guide, you'll learn how to ingest DynamoDB formatted change data capture data into Pinot. ## Introduction Imagine having the robust storage capabilities of DynamoDB combined with the lightning-fast analytics of Apache Pinot. ## What You'll Need * AWS account (with DynamoDB and Kinesis access) * Apache Pinot cluster * Your favorite code editor ## Setting Up the Replication Pipeline ### Step 1: Create a DynamoDB Table Let's start by creating our source of truth - a DynamoDB table. ![DynamoDBUI](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/recipes/images/create_dynamo_table.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=9e3fa806ae3c4fbeeacbc46ceffdeb7b) ### Step 2: Create a Kinesis Data Stream Time to create a highway for our data - Kinesis stream where dynamo will push its CDC. ![KinesisUI](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/recipes/images/create_kinesis_stream.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=f098e2f1936c1ed46fde4967a1bf3ef0) ### Step 3: Enable DynamoDB-Kinesis stream Now, let's turn on the data faucet by connecting dynamodb to kinesis ![DynamoDBUI](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/recipes/images/connect_kinesis_stream.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=cef6dc84bf6463d87c0118d61c44598f) ### Step 4: Create Pinot Schema Let's tell Pinot what our data looks like: ```json theme={null} { "schemaName": "pinot_dynamo_ingestion", "dimensionFieldSpecs": [ { "name": "eventName", "dataType": "STRING", "notNull": false }, { "name": "venue_name", "dataType": "STRING", "notNull": false }, { "name": "meetup_name", "dataType": "STRING", "notNull": false }, { "name": "meetup_id", "dataType": "STRING", "notNull": false }, { "name": "group_city", "dataType": "STRING", "notNull": false }, { "name": "group_country", "dataType": "STRING", "notNull": false }, { "name": "group_id", "dataType": "LONG", "notNull": false }, { "name": "group_name", "dataType": "STRING", "notNull": false }, { "name": "group_lat", "dataType": "DOUBLE", "notNull": false }, { "name": "group_lon", "dataType": "DOUBLE", "notNull": false }, { "name": "location", "dataType": "BYTES", "transformFunction": "toSphericalGeography(stPoint(group_lon,group_lat))", "notNull": false }, { "name": "is_delete", "dataType": "BOOLEAN", "transformFunction": "strcmp(eventName, 'REMOVE') = 0", "notNull": false } ], "metricFieldSpecs": [ { "name": "rsvp_count", "dataType": "INT", "notNull": false } ], "dateTimeFieldSpecs": [ { "name": "mtime", "dataType": "TIMESTAMP", "notNull": false, "format": "TIMESTAMP", "granularity": "1:MILLISECONDS" }, { "name": "ApproximateCreationDateTime", "dataType": "TIMESTAMP", "notNull": false, "format": "TIMESTAMP", "granularity": "1:MILLISECONDS" } ], "primaryKeyColumns": ["meetup_id"] } ``` ### Step 5: Create Pinot Table Configuration Now, let's set the table for our data feast! ```json theme={null} { "tableName": "pinot_dynamo_ingestion", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "mtime", "timeType": "MILLISECONDS", "retentionTimeUnit": "DAYS", "retentionTimeValue": "7", "schemaName": "pinot_dynamo_ingestion", "replication": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kinesis", "stream.kinesis.topic.name": "pinot-ingestion-stream", "region": "us-east-1", "accessKey": "XXXX", "secretKey": "XXXX", "shardIteratorType": "AFTER_SEQUENCE_NUMBER", "stream.kinesis.consumer.type": "lowlevel", "stream.kinesis.fetch.timeout.millis": "120000", "stream.kinesis.decoder.class.name": "ai.startree.pinot.plugin.inputformat.dynamodb.DynamoDbMessageDecoder", "stream.kinesis.decoder.prop.dynamodb.timeColumnName": "created_at_timestamp", "stream.kinesis.decoder.prop.dynamodb.deleteColumnName": "is_deleted", "stream.kinesis.decoder.prop.dynamodb.envelope.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory", "realtime.segment.flush.threshold.rows": "1000000", "realtime.segment.flush.threshold.time": "6h" }, "nullHandlingEnabled": true }, "routing": { "segmentPrunerTypes": ["time"], "instanceSelectorType": "strictReplicaGroup" }, "ingestionConfig": { "transformConfigs": [ ] }, "upsertConfig": { "mode": "PARTIAL", "deleteRecordColumn": "is_delete", "partialUpsertStrategies":{}, "comparisonColumns": ["ApproximateCreationDateTime"] }, "metadata": { "customConfigs": {} } } ``` ### Why do we have so many configurations? Let's try to understand which of these configs are necessary. When you enable CDC on dynamoDB table, it starts sending the data in the following format ```json theme={null} { "awsRegion": "us-east-2", "eventID": "e5c2d473-43e0-4dc3-bf28-cdcb575a83aa", "eventName": "INSERT", "userIdentity": null, "recordFormat": "application/json", "tableName": "pinot-ingestion-demo", "dynamodb": { "ApproximateCreationDateTime": 1719039879869591, "Keys": { "event_id": { "S": "badbe259-82d3-4869-b197-e935b0e942f2" }, "mtime": { "N": "1719039879651" } }, "NewImage": { "group_city": { "S": "New York" }, "location": { "B": "Y0d4aFkyVm9iMnhrWlhKZllubDBaWE09" }, "group_lon": { "N": "-97.169068" }, "event_time": { "S": "2024-07-10T12:34:39.651036" }, "group_id": { "N": "8032" }, "mtime": { "N": "1719039879651" }, "rsvp_count": { "N": "45" }, "group_country": { "S": "UK" }, "group_lat": { "N": "-41.13523" }, "event_id": { "S": "badbe259-82d3-4869-b197-e935b0e942f2" }, "venue_name": { "S": "Venue 1" }, "group_name": { "S": "Group 10" }, "event_name": { "S": "Event 3" } }, "SizeBytes": 313, "ApproximateCreationDateTimePrecision": "MICROSECOND" }, "eventSource": "aws:dynamodb" } ``` ```json theme={null} { "awsRegion": "us-east-1", "eventID": "1020b6ed-fc46-4767-8819-55c440041264", "eventName": "MODIFY", "userIdentity": null, "recordFormat": "application/json", "tableName": "pinot-ingestion-demo", "dynamodb": { "ApproximateCreationDateTime": 1719082124026, "Keys": { "mtime": { "N": "1719080907442" }, "meetup_id": { "S": "4a60c849-d475-4ad5-adbc-776cbf7ca5df" } }, "NewImage": { "group_city": { "S": "New York" }, "mtime": { "N": "1719080907442" }, "group_lon": { "N": "74.891677" }, "rsvp_count": { "N": "23" }, "group_lat": { "N": "-59.493753" }, "group_country": { "S": "UK" }, "meetup_id": { "S": "4a60c849-d475-4ad5-adbc-776cbf7ca5df" }, "venue_name": { "S": "Updated Venue 3" }, "group_id": { "N": "4673" }, "meetup_name": { "S": "Meetup 6" }, "group_name": { "S": "Group 8" } }, "OldImage": { "group_city": { "S": "New York" }, "mtime": { "N": "1719080907442" }, "group_lon": { "N": "-130.035739" }, "rsvp_count": { "N": "22" }, "group_lat": { "N": "66.888498" }, "group_country": { "S": "UK" }, "meetup_id": { "S": "4a60c849-d475-4ad5-adbc-776cbf7ca5df" }, "venue_name": { "S": "Updated Venue 2" }, "group_id": { "N": "4673" }, "meetup_name": { "S": "Meetup 6" }, "group_name": { "S": "Group 8" } }, "SizeBytes": 467 }, "eventSource": "aws:dynamodb" } ``` ```json theme={null} { "awsRegion": "us-east-1", "eventID": "78b55ff0-82ab-4fec-9f03-6312c5d2f6eb", "eventName": "REMOVE", "userIdentity": null, "recordFormat": "application/json", "tableName": "pinot-ingestion-demo", "dynamodb": { "ApproximateCreationDateTime": 1719081613486, "Keys": { "mtime": { "N": "1719080908912" }, "meetup_id": { "S": "735fd994-2071-4145-8f35-7e69e537aaba" } }, "OldImage": { "group_city": { "S": "London" }, "mtime": { "N": "1719080908912" }, "rsvp_count": { "N": "26" }, "group_lon": { "N": "-19.773637" }, "group_country": { "S": "Germany" }, "group_lat": { "N": "23.60754" }, "meetup_id": { "S": "735fd994-2071-4145-8f35-7e69e537aaba" }, "venue_name": { "S": "Venue 2" }, "group_id": { "N": "8524" }, "meetup_name": { "S": "Meetup 2" }, "group_name": { "S": "Group 6" } }, "SizeBytes": 257 }, "eventSource": "aws:dynamodb" } ``` #### Decoder Configuration To help pinot understand the dynamodb data format, we need to add decoder configs to our table ```json theme={null} "stream.kinesis.decoder.class.name": "ai.startree.pinot.plugin.inputformat.dynamodb.DynamoDbMessageDecoder", "stream.kinesis.decoder.prop.dynamodb.timeColumnName": "created_at_timestamp", "stream.kinesis.decoder.prop.dynamodb.deleteColumnName": "is_deleted", "stream.kinesis.decoder.prop.dynamodb.envelope.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder" ``` The `decoder.class.name` specifies our primary decoder. The `timeColumnName` specifies the column that should be filled with the `ApproximateCreationDateTime` from dynamodb json record. the `deleteColumnName` specifies the column that should be set to `true` in case we receive a `REMOVE` record from dynamodb Finally, the `envelope.decoder.class.name` simply specifies the vanilla decoder that should be used to parse the message. Since them dynamodb messages come in json format, we specify the `JSONMessageDecoder` here #### Upserts Configuration To handle updates properly, you need to enable upserts in Pinot. This is done in the `upsertConfig` section of the table configuration: ```json theme={null} "upsertConfig": { "mode": "PARTIAL", "deleteRecordColumn": "is_delete", "partialUpsertStrategies": {}, "comparisonColumns": ["ApproximateCreationDateTime"] } ``` Key points: * `mode`: Set to "PARTIAL" for partial updates. * `deleteRecordColumn`: Specifies the column that indicates if a record should be deleted. * `comparisonColumns`: Uses `ApproximateCreationDateTime` to determine the order of changes. #### Derived Column for Deletions A new derived column `is_delete` is created in the schema to signify whether a key needs to be removed from the upsert metadata: ```json theme={null} { "name": "is_delete", "dataType": "BOOLEAN", "transformFunction": "strcmp(eventName, 'REMOVE') = 0", "notNull": false } ``` This column is set to true when the `eventName` in the DynamoDB stream event is "REMOVE". #### Handling Different Event Types The configuration handles different event types as follows: 1. INSERT: New records are added to Pinot. 2. MODIFY: Existing records are updated using the upsert configuration. 3. REMOVE: Records are marked for deletion using the `is_delete` column. #### ApproximateCreationDateTime Usage The `ApproximateCreationDateTime` from the DynamoDB payload is used in the `comparisonColumns` of the upsert configuration. This ensures that changes are applied in the correct order, as it represents the sequence of events in DynamoDB. ```json theme={null} "comparisonColumns": ["ApproximateCreationDateTime"] ``` A corresponding column is added to the schema: ```json theme={null} { "name": "ApproximateCreationDateTime", "dataType": "TIMESTAMP", "notNull": false, "format": "TIMESTAMP", "granularity": "1:MILLISECONDS" } ``` ### Step 6: Create Pinot Table Let's bring our table to life! ```bash theme={null} bin/pinot-admin.sh AddTable \ -schemaFile /tmp/pinot/schema.json -tableConfigFile /tmp/pinot/table_config.json -exec ``` ## Insert, Update, Delete ### Insert Let's add some data to our DynamoDB table: ```bash theme={null} python publish_data.py Publish item with meetup_id: cc3fbaca-445c-4ebb-8f60-7b2a5eab3530 and mtime = 1719124016204 Publish item with meetup_id: c1731e4b-6726-40f5-8c8f-b34bc34a0744 and mtime = 1719124017046 Publish item with meetup_id: 9c4b139b-26aa-4d45-8efe-5d1ad2d4e1e3 and mtime = 1719124017265 Publish item with meetup_id: cfcb025b-7547-4002-9b52-95ad0b0a4ac8 and mtime = 1719124017488 Publish item with meetup_id: be08dc25-42a4-4fd3-8b6f-0bdaaeb4caf7 and mtime = 1719124017787 Publish item with meetup_id: bdc86183-a63a-4bc2-9901-22be89eb9774 and mtime = 1719124018010 Publish item with meetup_id: 0b27b577-dce5-43d1-b23f-8372b1882571 and mtime = 1719124018232 Publish item with meetup_id: 4d959048-796a-464f-ab6c-5afe174ec015 and mtime = 1719124018506 Publish item with meetup_id: 8a2e679c-3044-4526-bd9d-a2de908dbc62 and mtime = 1719124018723 Publish item with meetup_id: 98b3c314-75c2-420b-a154-d891a43e6efa and mtime = 1719124018945 DynamoDB insertion completed. ``` Check your Pinot table, and you'll see these rows magically appear! ![Pinot UI](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/recipes/images/all_pinot_data.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=7e12c28ef74cf20d796ec413d6e16ef1) ### Update Let's update a row: ```bash theme={null} python update_data.py --meetup_id cc3fbaca-445c-4ebb-8f60-7b2a5eab3530 --mtime 1719124016204 Updated item in DynamoDB with meetup_id: cc3fbaca-445c-4ebb-8f60-7b2a5eab3530 and mtime: 1719124016204 UpdatedAttributes: {'group_lon': Decimal('28.454082'), 'rsvp_count': Decimal('84'), 'venue_name': 'Updated Venue 1', 'group_lat': Decimal('11.194998')} ``` Query Pinot, and witness the transformation! #### Row before update ![Pinot UI](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/data_before_update.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=d79985e48e7303f625e8243ca8cafd4b) #### Row after update ![Pinot UI](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/recipes/images/data_after_update.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=80510d9bd750a86eba9cb9fc23063ba5) ### Delete To remove a row: ```bash theme={null} python delete_data.py --meetup_id cc3fbaca-445c-4ebb-8f60-7b2a5eab3530 --mtime 1719124016204 Deleted item with meetup_id: cc3fbaca-445c-4ebb-8f60-7b2a5eab3530 and mtime: 1719124016204 ``` Check Pinot: ![Pinot UI](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/data_deleted.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=a37a95b49eb35395396ff1a192f44697) ## Behind the Scenes: Viewing Operation Order Use the following in your Pinot queries: ```sql theme={null} SELECT * FROM pinot_dynamo_ingestion WHERE meetup_id = 'cc3fbaca-445c-4ebb-8f60-7b2a5eab3530' ORDER BY ApproximateCreationDateTime DESC OPTION(skipUpsert=True) ``` This will reveal the entire history of your data's journey. ![Pinot UI](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/skip_upsert.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=b738b1051287666b5421bbd44dc3adb7) ## Conclusion You've successfully created a real-time replication pipeline from DynamoDB to Apache Pinot. Your data is now ready for lightning-fast analytics while maintaining the reliability of DynamoDB. # Extract Parts of Date Source: https://docs.startree.ai/recipes/extract-parts-of-date In this recipe we'll learn how to extract parts of a date from a timestamp column. ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command, which starts up the Hybrid QuickStart: ```bash theme={null} docker run \ -p 9000:9000 \ apachepinot/pinot:1.0.0 QuickStart \ -type hybrid ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper all in one container. ## The Hybrid Quickstart The Hybrid Quickstart contains an offline and real-time table called `airlineStats`. These tables contain details of various flights from 2014. We can view the schema of the table by calling the `/schemas/` command. To get a list of the dimension fields, we can filter the results using jq like this: ```bash theme={null} curl -X GET "http://localhost:9000/schemas/airlineStats" 2>/dev/null | jq -c '.dimensionFieldSpecs[]' ``` Output: ```json theme={null} {"name":"ActualElapsedTime","dataType":"INT"} {"name":"AirTime","dataType":"INT"} {"name":"AirlineID","dataType":"INT"} {"name":"ArrDel15","dataType":"INT"} {"name":"ArrDelay","dataType":"INT"} ... ``` To get a list of date time fields, we can filter the results using jq like this: ```bash theme={null} curl -X GET "http://localhost:9000/schemas/airlineStats" 2>/dev/null | jq -c '.dateTimeFieldSpecs[]' ``` Output: ```json theme={null} {"name":"DaysSinceEpoch","dataType":"INT","format":"1:DAYS:EPOCH","granularity":"1:DAYS"} {"name":"ts","dataType":"TIMESTAMP","format":"1:MILLISECONDS:TIMESTAMP","granularity":"1:SECONDS"} {"name":"tsRaw","dataType":"TIMESTAMP","format":"1:MILLISECONDS:TIMESTAMP","granularity":"1:SECONDS"} ``` `ts` and `tsRaw` are the timestamp columns in this table. ## Extracting date components We can extract various components of the `ts` column using the `EXTRACT` function that was added in Apache Pinot 0.12. An example of all the available components is shown below: ```sql theme={null} select ts, EXTRACT(YEAR FROM ts) AS year, EXTRACT(MONTH FROM ts) AS month, EXTRACT(DAY FROM ts) AS day, EXTRACT(HOUR FROM ts) AS hour, EXTRACT(MINUTE FROM ts) AS min, EXTRACT(SECOND FROM ts) AS sec from airlineStats order by ts DESC limit 10 ``` | ts | year | month | day | hour | min | sec | | - | - | - | - | - | - | - | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | | 2014-02-14 00:00:00.0 | 2014 | 2 | 14 | 0 | 0 | 0 | *Query Results* We can also group by the extracted values, as shown below: ```sql theme={null} select EXTRACT(MONTH FROM ts) AS month, EXTRACT(DAY FROM ts) AS day, count(*) from airlineStats group by month, day order by count(*) DESC limit 10 ``` | month | day | count(\*) | | - | - | - | | 1 | 5 | 422 | | 1 | 3 | 409 | | 1 | 2 | 403 | | 1 | 6 | 354 | | 1 | 24 | 343 | | 1 | 9 | 340 | | 1 | 10 | 338 | | 1 | 30 | 333 | | 1 | 29 | 322 | | 1 | 26 | 319 | *Query Results* # Filtering records during ingestion Source: https://docs.startree.ai/recipes/filtering-ingestion To learn how to filter records when ingesting data into Pinot, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/-tK71ka6xMw) | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/filtering](https://github.com/startreedata/pinot-recipes/tree/main/recipes/filtering) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/filtering ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/filtering/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following JSON file: ```json theme={null} {"title": "Valentine's Day", "genre": "Comedy", "year": 2010, "id": 361248901147483647} {"title": "The Ugly Truth", "genre": "Comedy", "year": 2009, "id": 332567813147483648} {"title": "P.S. I Love You", "genre": "Romance", "year": 2007, "id": 346905752147483649} {"title": "Dear John", "genre": "Drama", "year": 2010, "id": 300441473147483650} {"title": "The Curious Case of Benjamin Button", "genre": "Fantasy", "year": 2008, "id": 394030854147483651} ``` *data/import.json* ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "movies", "dimensionFieldSpecs": [ { "name": "id", "dataType": "LONG" }, { "name": "title", "dataType": "STRING" }, { "name": "genre", "dataType": "STRING" }, { "name": "year", "dataType": "INT" } ] } ``` *config/schema.json* We'll also have the following table config: ```json {16-18} theme={null} { "tableName": "movies", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1, "schemaName": "movies" }, "tenants": { "broker":"DefaultTenant", "server":"DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "filterConfig": { "filterFunction": "year >= 2010" }, "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "metadata": {} } ``` *config/table.json* Our filtering function ensures that any records with a `year` property with a value of 2010 or more are **not imported**. You can create the table and schema by running the following command:\` ```bash theme={null} docker run \ --network filtering\ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-filtering" \ -exec ``` You should see a message similar to the following if everything is working correctly: ```text theme={null} 2022/02/25 09:39:00.839 INFO [AddTableCommand] [main] Executing command: AddTable -tableConfigFile /config/table.json -schemaFile /config/schema.json -controllerProtocol http -controllerHost 172.29.0.3 -controllerPort 9000 -user null -password [hidden] -exec 2022/02/25 09:39:01.308 INFO [AddTableCommand] [main] {"status":"Table movies_OFFLINE succesfully added"} ``` ## Ingestion Job Now we’re going to import the JSON file into Pinot. We'll do this with the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/import.json' outputDirURI: '/opt/pinot/data/movies/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: 'movies' pinotClusterSpecs: - controllerURI: 'http://pinot-controller-filtering:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` The import job will map fields in each JSON document to a corresponding column in the `movies` schema. If one of the fields doesn't exist in the schema it will be skipped. > [!NOTE] > > You can also apply transformation functions to JSON documents during the ingestion process. > For more details, see the [JSON Transformation Functions](/recipes/json-transformation-functions) guide. You can run the following command to run the import: ```bash theme={null} docker run \ --network filtering \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `movies` table or copy/paste the following query: ```sql theme={null} select * from movies limit 10 ``` You will see the following output: | genre | id | title | year | | - | - | - | - | | Comedy | 332567813147483648 | The Ugly Truth | 2009 | | Romance | 346905752147483649 | P.S. I Love You | 2007 | | Fantasy | 394030854147483651 | The Curious Case of Benjamin Button | 2008 | *Query Results* # How to fix host names Source: https://docs.startree.ai/recipes/fixed-hostname In this recipe, we'll learn how to fix the host name of components in an Apache Pinot cluster. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/fixed-hostname](https://github.com/startreedata/pinot-recipes/tree/main/recipes/fixed-hostname) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/fixed-hostname ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Pinot Minion, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/fixed-hostname/docker-compose.yml) file on GitHub. Navigate to the [Pinot UI](http://localhost:9000) and you should see the following: ![Pinot UI showing fixed host names](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/pinot-ui-fixed-hosts.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=53d8a850b560179d90c14759a3978b8d) Fixed hostnames ## Configuring fixed host names Let's have a look at the [Docker Compose file](https://github.com/startreedata/pinot-recipes/blob/main/recipes/fixed-hostname/docker-compose.yml) to see how we did this. The commands to start the container for each component are described below: **Controller** ```yml theme={null} StartController -zkAddress zookeeper-fixedhost:2181 -controllerHost pinot-controller-fixedhost ``` **Broker** ```yml theme={null} StartBroker -zkAddress zookeeper-fixedhost:2181 -brokerHost pinot-broker-fixedhost ``` **Server** ```yml theme={null} StartServer -zkAddress zookeeper-fixedhost:2181 -serverHost pinot-server-fixedhost ``` **Minion** ```yml theme={null} StartMinion -zkAddress zookeeper-fixedhost:2181 -minionHost pinot-minion-fixedhost ``` # How to Force Commit Consuming Segments Source: https://docs.startree.ai/recipes/force-commit In this recipe we'll learn how to use the `forceCommit` API to immediately commit consuming segments. This API is usually used when we've made stream compatible changes to our table config, like changing `segment.threshold` parameters. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/force-commit](https://github.com/startreedata/pinot-recipes/tree/main/recipes/force-commit) | ## Prerequisites You will need to [install Docker](https://docs.docker.com/get-docker/) to follow the code examples in this guide. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/force-commit ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/force-commit/docker-compose.yml) file on GitHub. ## Data generator This recipe contains a data generator that creates events with a timestamp, count, and UUID. You can generate data by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "ts": 1680171022742, "uuid": "5328f9b8-83ff-4e4c-8ea1-d09524866841", "count": 260 } ``` ## Kafka ingestion We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t events -Kø ``` We can check that Kafka has some data by running the following command: ```bash theme={null} docker exec -it kafka-querysegment kafka-run-class.sh \ kafka.tools.GetOffsetShell \ --broker-list localhost:9092 \ --topic events ``` We'll see something like the following: ```bash theme={null} events:0:19960902 ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [{"name": "uuid", "dataType": "STRING"}], "metricFieldSpecs": [{"name": "count", "dataType": "INT"}], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Now for the table config: ```json {13} theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "realtime.segment.flush.threshold.rows":"2500000", "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-forcecommit:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.time":"1h" } }, "ingestionConfig": { }, "tenants": {}, "metadata": {} } ``` *config/table.json* This highlighted section indicates that we're going to create new segments after every 2.5 million rows. We'll create the table by running the following: ```bash theme={null} docker run \ --network forcecommit \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-forcecommit" \ -exec ``` ## Querying by segment Once that's been created, we can head over to the [Pinot UI](http://localhost:9000/#/query) and run some queries. Pinot has several built-in virtual columns inside every schema that can be used for debugging purposes: | Column Name | Column Type | Data Type | Description | | - | - | - | - | | \$hostName | Dimension | STRING | Name of the server hosting the data | | \$segmentName | Dimension | STRING | Name of the segment containing the record | | \$docId | Dimension | INT | Document id of the record within the segment | The one that's useful for us is `$segmentName`, which we can use like this to count the number of records in each segment: ```sql theme={null} select $segmentName, ToDateTime(max(ts), 'YYYY-MM-dd HH:mm:ss') as maxTs, count(*) from events group by $segmentName order by maxTs desc limit 100 ``` | \$segmentName | maxTs | count(\*) | | - | - | - | | events\_\_0\_\_1\_\_20230602T1305Z | 2023-06-02 13:04:34 | 1423028 | | events\_\_0\_\_0\_\_20230602T1304Z | 2023-06-02 13:04:27 | 2500000 | *Query Results* Now, let's say that we decide we want to reduce the segment threshold to 1m rows. We can do that with the following table config: ```json {12} theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "realtime.segment.flush.threshold.rows":"1000000", "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-forcecommit:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.time":"1h" } }, "ingestionConfig": { }, "tenants": {}, "metadata": {} } ``` *config/table-newthreshold.json* Let's update the table config: ```bash theme={null} docker run \ --network forcecommit \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table-newthreshold.json \ -controllerHost "pinot-controller-forcecommit" \ -exec -update ``` We now have to wait until the current consuming segment reaches 2.5 million rows before this config will be used. Or we can have it start immediately if we call the `forceCommit` API: ```bash theme={null} curl -X POST "http://localhost:9000/tables/events/forceCommit" -H "accept: application/json" ``` After a few seconds the consuming segment will be committed and segments will be committed every 1m rows. If we run the query again, we'll see this output: | \$segmentName | maxTs | count(\*) | | - | - | - | | events\_\_0\_\_5\_\_20230602T1305Z | 2023-06-02 13:04:55 | 1000000 | | events\_\_0\_\_4\_\_20230602T1305Z | 2023-06-02 13:04:50 | 1000000 | | events\_\_0\_\_3\_\_20230602T1305Z | 2023-06-02 13:04:45 | 1000000 | | events\_\_0\_\_2\_\_20230602T1305Z | 2023-06-02 13:04:39 | 1000000 | | events\_\_0\_\_1\_\_20230602T1305Z | 2023-06-02 13:04:34 | 1423028 | | events\_\_0\_\_0\_\_20230602T1304Z | 2023-06-02 13:04:27 | 2500000 | *Query Results* # Real-Time RAG Pinot Source: https://docs.startree.ai/recipes/genai This repository is a Retrieval-Augmented Generation (RAG) example using Apache Pinot, LangChain, and OpenAI. The use case is to load documentation and allow an LLM to answer questions provided by a user. This approach enables you to generate AI responses that are fresh and in real time. This RAG example uses LangChain's `RecursiveUrlLoader`. It accepts a URL, recursively loads pages, and converts them into `documents`. These documents are converted into embeddings, submitted to a Kafka topic, and consumed by Apache Pinot. To understand how this recipe processes data, examine the commands executed in the Makefile. | Pinot Version | 1.1.0 | | - | - | | Code | [startreedata/pinot-recipes/genai](https://github.com/startreedata/pinot-recipes/tree/main/recipes/genai) | ## Prerequisites To follow the code examples in this guide, do the following: * [Install Docker](https://docs.docker.com/get-docker/) locally. * Download recipes ## Docker This repo builds the Apache Pinot project. You may get an error `No space left on device` when building the container. Execute the command below to free resources before building. ```bash theme={null} docker system prune --all --force ``` > [!NOTE] > > Building the Pinot image will take about 25 minutes to finish. ## Makefile To start the example, run the command below. ```bash theme={null} make all ``` This will start Pinot and Kafka. ## Load Documentation Run the command below to load Pinot with embeddings from your document site by providing a URL. The loader will recursively read the document site, generate embeddings, and write them into Pinot. ```bash theme={null} make loader URL=https://dev.startree.ai/docs ``` If you have a large document site, this loader will take longer. You will see confirmations on the screen as each embedding is sent to Kafka and Pinot. This loader creates an embedding per page so that we can perform an UPSERT in Pinot. If you have larger pages and depending on the AI model you are using, you may get this error: ``` This model's maximum context length is 8192 tokens ``` Alternatively, you can chunk the pages into smaller sizes and UPSERT those records in Pinot by URL + ChunkId. The implementation in this repository does not do that. ## Ask Your Questions Run the command below and ask a question that the documentation you loaded can answer. ```bash theme={null} make question ``` ## Clean up ```bash theme={null} make clean ``` ## Troubleshooting To clean up old Docker installations that may be interfering with your testing of this recipe, run the following command: ```bash theme={null} docker system prune ``` # Storing Geospatial Objects Source: https://docs.startree.ai/recipes/geospatial To learn how to store Geospatial data in Apache Pinot, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/ThRHkxn2bSo) To learn about geospatial indexing, see the [Geospatial indexing developer guide](/recipes/geospatial-indexing). | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/geospatial](https://github.com/startreedata/pinot-recipes/tree/main/recipes/geospatial) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/geospatial ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/geospatial/docker-compose.yml) file on GitHub. ## Generating Geospatial data This recipe contains a data generator that produces JSON documents that contain various geospatial objects in [Well-known text (WKT)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) format. You'll need to first install the following dependencies: ```bash theme={null} pip install geofactory faker geojson shapely ``` Once that's done you can run the data generator and grab just the first generated document, by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "tsString": 1678119130444, "uuid": "77a67adb-f9db-4e07-afa0-a59c2e27b3d0", "count": 908, "polygonString": "POLYGON ((62.00015 -32.338507, 128.785854 44.676913, 45.782208 72.58999, 62.00015 -32.338507))", "multiPolygonString": "MULTIPOLYGON (((-156.911813 53.429702, -41.450202 -21.951131, 7.988808 -0.515531, -33.662937 -41.782691, -104.939426 46.981135, -166.669848 -1.230373, 142.162691 -31.02204, -156.911813 53.429702)), ((55.485853 -15.824756, 49.151331 15.311564, -19.514604 41.509102, -118.56698 82.180807, 23.312518 -44.361666, -119.523485 87.590606, 86.006041 -75.833495, -123.597585 60.50805, 26.552532 -25.661127, 125.905494 45.329903, 55.485853 -15.824756)), ((-97.775848 23.71736, -12.923769 6.144659, -157.132715 84.533912, -65.101377 -48.554279, 130.157698 -43.126393, 82.894933 31.505981, 88.876909 -80.368662, -97.775848 23.71736)), ((77.334307 18.595837, 120.613052 -63.95572, -144.775823 47.697498, -22.514493 73.205054, 179.842676 -19.196558, 77.334307 18.595837)), ((-22.966058 -83.846852, 79.822455 72.20039, -4.942435 89.554717, 97.447234 -79.204027, -20.398943 18.112453, 43.921047 73.510537, -15.824053 -52.55715, -63.75579 -11.576211, 30.079239 -23.282294, -22.966058 -83.846852)), ((-78.440918 14.778834, -100.955781 9.000371, -126.927461 -10.596274, -101.563431 -65.955052, 144.488354 84.141029, -0.517424 -68.166871, 143.073386 -58.519274, -87.063505 -8.517407, 52.864293 -39.634933, 71.382693 -23.895989, -157.117149 48.370833, -78.440918 14.778834)), ((129.485682 -37.023947, 146.354911 -32.264661, 38.373808 53.792189, 163.996137 76.326083, 150.2438 68.221784, 118.460847 -28.845576, -83.330716 2.700953, 0.302329 -17.033323, 129.485682 -37.023947)), ((109.0075 31.381868, -119.15326 -88.312268, 68.191132 42.722532, 9.471264 -77.189739, 48.404414 50.660832, -59.398768 -82.253626, 37.209834 13.802957, 1.361337 17.779388, 130.120281 -27.930155, -51.280126 -82.602895, 109.0075 31.381868)), ((81.324979 64.682666, 125.893131 -17.517728, 149.267106 84.919949, -16.241389 -69.639628, 16.296239 32.932744, 81.324979 64.682666)), ((158.236415 15.068143, -19.047701 -43.09086, -92.883841 -55.723258, 90.540215 88.231732, -152.310498 62.972025, -60.100767 25.862447, -81.475891 9.735632, 83.042314 36.722952, 158.236415 15.068143)))", "pointString": "POINT (-124.782406 84.460566)", "lineStringString": "LINESTRING (0.169988 29.481278, 131.871624 -3.540022)", "geometryCollectionString": "GEOMETRYCOLLECTION (LINESTRING (-31.27193 -56.023134, -165.88589 -44.849634, -44.706247 -32.393057, 162.162153 28.42024, -110.441037 14.389923, 44.199914 75.002883, 177.923367 -74.590689, -58.204393 -83.820673, -52.530859 78.735928, -56.632839 74.800328, -71.255105 -67.45854, 114.825914 -9.889731, 44.228169 83.936279, -149.849434 -48.782353))" } ``` You can see from this output that we have three geospatial objects - a polygon, a multi polygon, and a point. Pinot also supports line strings, multi points, multi line strings, and geometry collections. You can read more about this in the [Geospatial documention](https://docs.pinot.apache.org/basics/indexing/geospatial-support#geospatial-data-types). ## Kafka ingestion We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t events -Kø ``` We can check that Kafka has some data by running the following command: ```bash theme={null} docker exec -it kafka-geospatial kafka-run-class.sh \ kafka.tools.GetOffsetShell \ --broker-list localhost:9092 \ --topic events ``` We'll see something like the following: ```bash theme={null} events:0:138780 ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [ {"name": "uuid", "dataType": "STRING"}, {"name": "polygon", "dataType": "BYTES"}, {"name": "multiPolygon", "dataType": "BYTES"}, {"name": "point", "dataType": "BYTES"}, {"name": "lineString", "dataType": "BYTES"}, {"name": "geometryCollection", "dataType": "BYTES"} ], "metricFieldSpecs": [{"name": "count", "dataType": "INT"}], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Note that the columns for `polygon`, `multiPolygon`, and `point` all have have a data type of `BYTES`. Geospatial columns must use the `BYTES` type because Pinot will serialize the Geospatial objects into bytes for storage purposes. Now for the table config: ```json {11-32} theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "ingestionConfig": { "transformConfigs": [ { "columnName": "polygon", "transformFunction": "toSphericalGeography(ST_GeomFromText(polygonString))" }, { "columnName": "multiPolygon", "transformFunction": "toSphericalGeography(ST_GeomFromText(multiPolygonString)" }, { "columnName": "point", "transformFunction": "toSphericalGeography(ST_GeomFromText(pointString))" }, { "columnName": "lineString", "transformFunction": "toSphericalGeography(ST_GeomFromText(lineStringString))" }, { "columnName": "geometryCollection", "transformFunction": "toSphericalGeography(ST_GeomFromText(geometryCollectionString))" } ] }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-geospatial:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows":"100000", "realtime.segment.flush.threshold.time":"1h" } }, "tenants": {}, "metadata": {} } ``` The most important lines in this file are the highlighted transform config functions. The `ST_GeomFromText` function creates a geometry object from a WKT representation. We then use `toSphericalGeography` to convert the geometry object to a geography object. We do this conversion because some of the other spatial functions only work on geography objects. We'll create the table by running the following: ```bash theme={null} docker run \ --network geospatial \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-geospatial" \ -exec ``` ## Geospatial Querying Once that's been created, we can head over to the [Pinot UI](http://localhost:9000/#/query) and run some queries. The following query returns three records where the point geometry object fits inside the polygon: ```sql theme={null} select ST_AsText(polygon) AS polygon, ST_AsText(point) AS point, ST_AsText(lineString) AS lineString, ST_AsText(multiPolygon) AS multiPolygon from events limit 1 ``` You will see something like the following output: | polygon | point | lineString | multiPolygon | | - | - | - | - | | POLYGON ((176.322421 81.384135, -72.119398 -70.951972, -58.286871 65.822307, 176.322421 81.384135)) | POINT (-172.593476 -21.260499) | LINESTRING (-39.605948 62.059305, 34.073705 0.799658, -139.772395 -1.703213, -77.278367 -71.676314, 63.493656 -85.863652, -161.998698 86.05301, 103.930496 -81.78901, -141.297058 26.328405, -157.828781 -63.650237, 49.545413 66.653053, 114.912326 -29.891357, 173.958216 -42.149873, -116.359889 60.393419, 85.314076 -11.976479, -120.473831 66.697589, 150.728768 -39.856015, -31.338929 -84.900021, 74.282083 74.694805, 49.396736 -26.537324, 71.561692 -50.287069, -119.012218 -77.801035, 127.597265 -81.655447, -107.114193 22.611314, 94.178538 43.505075, 118.342472 -16.803067, -97.814012 57.180657, 110.73994 -78.122044, 16.147591 71.993343, 50.29945 -86.283186, 168.445289 -67.303486, -116.9526 53.27939, 158.212888 -42.378283, -37.157991 -32.764193, -164.889449 13.440015, -134.571688 57.62997, -89.047944 -2.201528, -52.243972 67.624054, 52.416225 -45.269797, 176.506387 86.991437, -149.196303 70.011257, 70.088744 -78.29957, -44.795508 -58.590711, -47.21003 9.665805, -96.362498 -20.783698, 141.348796 34.565632, -156.840115 75.323202, 134.096607 71.191333, -81.516549 48.578975, 158.280571 -5.490793, 97.363266 9.227137, -85.523465 73.474778, -111.555913 10.648888, -36.575199 -28.657333, 171.439977 -66.52512, 151.746077 70.033849, -39.321975 27.239663, 29.730991 -40.406582, 160.594528 -89.762512, -5.265885 -41.629327, 41.617094 66.55888, -98.592913 49.369489, -144.211071 24.420265, 52.189571 -68.837367, -9.914865 61.997966, 10.23137 -46.383373, 93.297765 -32.987757, 82.796126 -47.822649, 61.55297 -82.747522, -19.677935 -33.215264, -98.717133 -55.313131, 34.170884 24.717411, -28.364093 65.403633, 37.893254 21.678849, 19.42964 -59.832252, -175.414992 -25.013375, 27.075554 -52.557481, 100.274289 24.866428, 90.267502 56.376479, -48.667261 -81.214369, -142.76731 16.573549, -115.703053 81.464063, 153.528885 -20.667554, 90.225929 -25.672213, -128.323393 34.743791, 15.638979 86.634385, 163.744803 82.119113, -45.423247 -6.762204, -16.503844 -19.213761, 22.50076 -53.717847, -76.784034 -28.383604, -132.753228 -11.004929, 80.753417 63.365668, 44.35864 87.425101, 142.581096 79.079986, -58.56832 -62.495883, 89.614403 41.707819, -175.216828 -25.967614, -84.488974 50.059492, 150.978167 -30.021803, 141.519166 -29.815106, 174.976466 -10.401967, 68.75459 -39.021284, -13.610707 -7.158904, 121.922659 -4.276183, -48.599251 49.250129, 66.579595 4.110557, 153.471355 42.813721, -30.784459 10.318573, 74.242803 -54.440399, 37.685038 -25.763434, -84.29051 -26.80077, -1.141665 55.231691, 32.181186 -57.471082, -160.010391 -26.870889, -117.149801 -88.057862, -110.921811 18.714086, 176.905592 44.707417, -70.471534 -47.305341, 113.045871 70.339783, 52.964186 -15.06033, 34.265453 53.469926, -94.55798 18.381975, -140.96694 45.645717, 154.467162 -52.987886, -101.26078 -6.048333, -157.652147 56.970663, -161.747241 21.669436, 70.128824 -85.958748, -60.92666 43.865177, 113.600366 -87.019599, 28.601477 43.628361, -102.80591 69.239108, -127.880824 67.67308, -134.371475 -11.070328, 76.01444 5.430101, -27.636749 52.231239, 102.615728 -14.548348, -110.381851 -5.8436, -127.101877 12.506771, 34.996208 74.271107, -47.585258 -85.519551, -4.221139 22.972952, -45.483676 65.396761, -127.201279 30.50113, 30.584315 -12.690909, 108.639983 18.753667, 134.095324 -5.896064, -95.481747 -85.337925, 63.284157 -87.960994, -46.447019 -53.642246, -164.194369 -51.739874, -126.18765 -8.284258, -74.243517 45.440134, -6.940857 88.354842, 111.649492 55.265647, -92.972516 -26.816927, 163.056917 7.907828, 37.253209 -84.507217, 145.648711 22.596968, 11.208132 33.22198, -22.596546 24.2137, -163.936167 9.185828, -21.910874 -49.467879, 176.547395 -17.462788, 165.401513 15.75875, 94.469871 -13.71385, 106.292617 -27.721893, 27.436531 84.897763, 52.564905 -82.264782, -118.699199 59.657506, 53.014269 -3.351515, 179.592392 -79.052068, 65.636512 -43.162616, -138.54432 63.320941, 178.198024 -42.023431, -22.471663 -14.467816, 151.883527 86.665156, 93.329297 -57.429819, 40.924958 -9.829078, -16.473208 39.395481, -101.394642 -8.31141, -161.775103 -52.022231, -3.065189 -8.15484, -43.10517 77.503457, 5.008822 18.41627, -135.194192 -56.203634, -79.08717 25.626562, -78.92879 -41.49407, 34.759373 -18.882311, 39.707508 32.435879, -169.920577 52.037722, 177.623278 77.037348, -24.202141 -68.593328, -94.187542 -20.794741, -92.325811 -86.284098, -33.13556 -83.712563, -32.04029 2.446138, -3.35398 -57.230626, 110.402786 -21.628998, 99.217551 -81.759833, 7.528128 -22.2746, -12.94194 -69.696166, 94.078346 62.177684, -49.462503 -50.364804, 85.281119 67.68178, 97.210298 -80.073917, -168.435434 20.45679, -47.903143 -13.328633, 114.763862 -18.981894, 162.954226 -56.041624, -20.17656 73.308346, -8.226225 -25.990478, 173.664107 -61.546799, 53.378885 -72.671294, 72.675493 -81.961164, -58.659954 -61.727779, 84.163521 30.43018, -54.920828 -52.330514, -108.42375 -40.516123, 35.000006 -7.492885, 148.098097 10.773416, -80.569735 21.697039, -46.564841 -1.058029, -21.721257 57.72145, 116.442123 5.820636, 35.100812 27.161244, 149.07338 53.212614, -46.369173 18.640516, 70.425198 -14.140541, 124.334374 28.70613, 18.992737 30.906371, -158.070867 72.596182, -114.381232 69.155047, -36.36565 59.985806, -133.83716 74.928327, 153.825315 26.518936, -92.649884 15.67211, 143.053284 -26.435581, -3.454669 18.631335, 78.38646 89.938578, -172.892697 -11.763195, -103.942842 -63.382503, 102.301659 5.696929, -62.556128 -70.739013, -126.826924 75.169294, 34.820952 30.189624, -99.807633 -60.073439, -13.745802 6.295363, 10.873219 -3.536484, -119.406271 63.473495, -48.107288 8.873623, 47.126602 59.708142, 166.355645 -28.889928, -19.536452 -74.622536, 108.560391 -60.184906, -56.080668 47.4497, 50.574122 -68.705208, 141.169488 -18.809269, 11.130543 -4.43483, 16.557308 -67.988467, 32.349421 86.635435, -113.366747 -2.887927, 43.460338 32.373973, 99.994242 70.631828, -147.818243 -88.373823, -117.216945 -38.925164, -90.892343 -81.196641, 35.154482 55.040215, 65.816648 -21.869375, 97.692292 -19.957612, 166.718869 -54.158266, 29.110551 -13.632212, -124.287976 -9.607919, 68.175629 65.44049, -37.287023 -38.880369, -28.054498 0.419048, -161.913089 -66.222916, 59.515031 -71.673546, 31.375715 -3.447172, 55.344786 -68.45671, 157.92012 -26.944477, -119.966836 -19.054635, -173.860055 36.790733, 119.995082 4.111344, -165.032131 -19.809698, 137.045843 -56.991861, -139.77337 -34.192811, -124.423552 -54.214506, -142.064943 63.929144, 76.857867 -37.475197, -175.726286 75.506758, 165.058833 -75.242941, -43.314247 12.232717, 92.26026 49.517266, 128.353324 18.686215, -173.669048 23.745093, -176.59101 -83.372717, -73.098719 -53.419438, -128.726266 75.207587, -113.192774 -39.168939, -128.089968 -81.157911, -100.897411 74.64299, 82.960021 25.424494, 41.139373 -3.155689, 85.467937 77.57565, -23.459712 -43.734892, -176.156855 -88.697419, -121.427807 87.37934, -128.11411 -10.831903, -151.468655 -18.94763, -31.413276 -37.700218, 20.897062 2.515821, -64.052651 44.414892, 55.808771 -14.750928, 45.962241 41.142575, 177.838597 33.748472, 172.904818 61.705984, -130.812715 -20.374941, 169.761311 56.764636, -166.799536 -35.71725, 22.440631 53.019796, 155.767456 6.994134, -130.600671 28.911237, -155.854725 -26.713577, -147.140814 -13.054319, 0.782785 -38.266728, 126.42647 6.470491, -19.511867 81.218572, 106.382149 69.527747, 38.084962 -53.939124, 107.483797 -38.087513, 11.710841 47.248047, -38.991485 -61.261603, 148.485777 88.985646, 143.717362 -89.72035, 160.688311 14.286745, 166.134534 7.077319, 27.602555 58.020542, -10.476567 67.040875, -60.054115 52.800669, 29.642927 -3.94563, -168.556162 -44.065381, -124.012889 60.432089, -20.876791 19.475738, 16.029474 -69.257664, -11.144071 46.744078, -162.933583 33.241045, -15.179625 74.099368, -14.318465 -6.939889, 151.051414 9.017413, 171.99381 -44.906023, -80.859959 73.159007, -142.792435 8.18011, -140.976409 52.723868, -127.47381 -23.340656, 86.772039 -52.227518, -65.745762 32.440371, -166.923014 62.678728, 172.886104 -63.145035, 159.774644 -87.06665, 115.725745 54.26459, -11.647147 51.512377, -31.974062 -85.63315, -50.565026 66.940239, -69.511127 59.672705, 123.488879 -74.304506, -150.132103 -87.458302, -18.47177 23.196824, 99.016123 29.163321, 147.91177 -65.645829, -145.478865 45.226061, -60.066758 42.891977, -147.671947 84.938849, 2.794951 -85.370273, 11.093209 31.124251, -107.252385 77.657456, -46.548666 68.553585, -55.814377 -30.023989, -155.913369 -32.20961, 23.941898 84.537624, -165.888227 27.793287, 149.148846 -9.787427, 166.623875 -43.676539, 136.318263 3.352695, -114.308953 77.883691, 86.043547 66.028605, 78.361733 71.441873, -41.203153 -10.193338, 98.067307 54.038317, -109.00711 -75.670806, -121.323229 -6.723581, -112.353298 30.429805, -159.690727 9.520088, 47.321877 -23.60118, -56.147693 -33.552735, -116.991696 38.778968, 118.881544 21.949385, -161.067974 78.627968, 88.803174 82.778036, 4.748978 -49.699477, 152.86819 -68.192031, -104.219466 31.21515, 69.976485 -9.182857, -72.369362 -53.629506, 16.160483 -12.771051, -88.28866 78.348621, -37.006021 -25.851402, 8.886839 -49.453527, 175.794783 -74.61257, 172.113645 38.283994, -146.950977 -64.959145, 69.367783 65.585728, -44.864421 -34.193042, -20.594424 22.735414, 24.056957 74.212979, -20.929085 55.510341, -128.137054 1.998671, 98.837962 9.501703, -32.932672 34.730879, -173.914798 76.88654, -104.695177 -30.067951, 1.249288 80.466789, 28.135459 -24.382456, 167.881075 88.070917, 12.97549 18.839277, 34.062114 -31.609371, 18.138164 -19.841583, 137.451056 22.919457, 4.313829 22.642922, 20.009889 16.324082, -159.655664 -46.564397, -92.481555 43.398373, 161.039174 -47.933591, 6.641751 -71.690922, -57.538665 40.920878, -67.995105 -86.098549, 101.75818 35.789859, -55.362315 -89.170121, 83.921659 17.38134, -45.803745 -67.722691, -99.43627 2.184392, 5.294237 -7.204079, -37.946044 83.46461, 57.065937 -24.182964, 3.621356 67.282066, 72.357067 20.306202, 23.563789 -74.439909, -107.622748 -66.538578, -45.904435 5.56731, -52.269034 58.105472, 24.177395 6.887253, 88.613931 65.298503, -76.016508 40.582501, -161.874686 50.778205, 34.27361 -3.308462, 144.510238 -51.738667, -99.657585 -60.44952, -113.113152 -12.333524, 28.003655 -29.897205, -53.997717 -9.133845, -103.558446 -30.785175, -122.533364 -83.483486, -73.033271 -12.146721, -152.803983 38.280836, 23.628879 51.667085, 103.093343 -31.196026, 120.334224 30.965174, -0.616269 36.401016, 73.084527 -12.009683, -143.279591 -18.205344, -118.030258 -73.844598, -130.344 4.361144, -62.475549 -85.423046, -59.405182 56.51623, -118.271827 62.856226, 120.099665 -0.839696, 50.709207 -89.016496, -37.712643 46.15153, -86.294708 -2.244355, -167.068308 -44.072243, -149.431393 40.372074, 4.627398 18.56288, 125.841489 54.979284, 135.882672 36.945496, -17.892891 50.816449, 112.646516 28.70489, 106.829812 -69.085286, 79.38991 -27.12574, 135.816373 8.440255, -92.940445 22.045166, 122.84331 28.337848, -69.74764 70.89657, 100.896755 21.030607, 173.1541 -40.162954, 117.926312 -49.238772, 129.990801 66.098744, -21.896191 54.385665, -43.641986 34.094825, -61.470852 -7.860776, -173.546487 -5.077445, 85.72068 -17.498026, -160.583992 -62.411389, -128.263524 -42.051014, 19.501394 -73.508098, 72.401021 -63.824531, -49.621791 -27.161031, -75.119549 -27.496253, -63.32588 8.380343, -23.613898 -17.003545, 173.374833 1.562424, 80.352584 -59.280009, -109.381399 -50.55484, -109.032478 37.346156, -169.605913 29.893478, 55.863456 -68.867174, -130.750144 -46.571815, 126.319961 35.395255, -100.171253 77.498002, 93.151856 74.458816, 38.948207 -69.120969, 175.838447 -57.356315, -89.771826 -88.766139, 26.958895 51.675663, 154.200495 13.811027, 61.365623 -25.955579, -106.330825 45.492816, 111.476066 -22.589074, -126.871069 -12.200052, 86.903995 5.190294, 106.298229 -25.945498, -76.928571 30.545851, 81.057478 -70.884687, 31.000843 -71.107159, -21.729561 64.567399, -64.066302 -36.250074, -85.298096 83.918144, -115.721055 21.906689, -97.71014 83.246844, 110.161527 10.777, -134.07631 40.116966, -13.191494 8.763722, 25.010343 8.507088, 37.654915 -32.549785, -24.747328 -73.335893, 85.717725 -63.881628, 166.868101 -72.086692, 83.762913 -68.902958, -106.751841 -14.968426, 49.285616 -54.545852, 26.645822 28.28981, -46.540048 44.187205, 69.306479 21.263316, 160.492736 1.118264, 70.932013 -89.956174, -95.726051 -16.49625, -87.3876 70.01841, 77.224777 -48.719265, 109.383701 55.655501, 49.132967 -21.321772, -132.666533 -35.311585, 167.012858 13.834435, 4.916061 -39.698582, 160.904434 -29.723021, -115.470477 -89.316922, 131.295548 39.326639, 132.970503 -72.888801, -177.914336 -86.001063, 42.258197 89.366265, -131.666232 -13.893237, -48.348907 -9.49313, -131.353794 88.706052, 125.493104 -35.080884, 122.151019 -74.918886, -128.326233 -47.437449, 50.728225 -32.972187, 76.528858 89.304653, -133.229121 -88.656607, -2.804844 -28.28229, 30.91049 -35.576649, -46.742461 -33.353068, -26.555062 49.317571, 59.430729 37.88883, -163.39316 30.09422, 112.054022 -77.561769, -93.63942 31.197075, 0.122914 61.245938, 138.923203 -30.477557, 100.339118 -39.919613, 171.604843 -38.067686, -8.488512 89.275135, -41.791682 41.789816, -127.16586 -37.828658, 178.142484 66.674578, -121.601972 -14.103024, -140.442892 -68.174977, -115.753077 -48.202574, 144.830848 17.357141, 50.63297 -67.536675, 126.659103 -89.266282, -9.617554 -80.479532, 178.992147 -63.729826) | MULTIPOLYGON (((-157.138803 19.18181, 177.653922 78.908605, 73.865228 -46.498039, 29.089036 -32.451709, -113.274383 35.257241, -157.138803 19.18181)), ((137.611215 -63.422384, -11.67216 -9.776203, 104.398197 -64.689331, -126.055342 31.100282, 158.788515 65.457857, -65.819031 -54.702393, -146.544439 72.677723, 82.282191 -84.675245, 124.150138 3.74385, 160.457713 -37.333328, 13.133476 13.493448, 137.611215 -63.422384)), ((-154.215233 86.954676, 88.565588 -62.92212, -40.124738 -47.491918, -154.215233 86.954676)), ((-79.928151 -67.374268, 20.941302 -9.103939, 128.858873 39.527307, -79.928151 -67.374268)), ((60.032965 47.065862, 174.341177 7.45859, -15.993429 42.459224, 107.840963 -38.456336, 23.50499 -39.973941, 60.032965 47.065862)), ((179.573616 -62.99732, -105.100176 -54.553241, 33.50347 65.307462, 154.868816 70.518507, 137.55974 38.791215, 62.541493 -43.301786, 164.520115 -56.401969, 16.971155 81.202442, -104.242781 -31.761278, 179.573616 -62.99732)), ((-77.705909 -48.978714, 57.147473 75.757949, 7.2753 -33.73846, -107.590355 27.262775, -77.705909 -48.978714)), ((-150.351053 -6.213762, -165.185977 13.382215, -24.536012 -2.610698, -150.351053 -6.213762)), ((-42.264055 -26.62393, -66.313869 61.632505, 66.043358 58.651193, 74.838775 47.873168, 44.824576 -0.363304, 67.416108 11.688783, -101.268148 -33.536187, -125.398473 -3.515584, 109.80077 20.884828, -25.961502 -39.772112, -148.813282 68.189077, -42.264055 -26.62393)), ((36.031235 65.385556, 52.041906 -50.553593, -161.286029 -74.420833, -25.593221 -66.459849, -152.194805 5.569134, -113.5458 45.46695, -130.464924 15.134347, -128.232822 39.355035, 36.031235 65.385556))) | *Query Results* We could also write another query to find the points that are within 500km of the centre of San Francisco: ```sql theme={null} SELECT uuid, ST_DISTANCE(point, ST_Point(-122, 37, 1)) FROM events WHERE ST_DISTANCE(point, ST_Point(-122, 37, 1)) < 500000 LIMIT 10 ``` You will see something like the following output: | uuid | st\_distance(point,'80c05e8000000000004042800000000000') | | - | - | | c58586a1-8ab1-4e41-b3b1-c6e3ed916cb7 | 394531.2325195099 | | 302f2ea0-df0f-4b59-9fd8-e73dae562d0f | 160515.7426762354 | | 23135371-8727-4a31-8179-922d725c9e7c | 326358.86843014683 | | 6373e83e-7b1c-4bd5-afaa-f44c9ee220c2 | 224860.30769621156 | | 3790ea3b-51a8-41a2-8077-6fa19cb4f779 | 354395.6844787827 | | 8717046e-9218-4a1f-8372-9555fd8fbe90 | 423105.8274571361 | | 28aafa33-b113-436a-b388-f6413f953abe | 257472.01145501286 | | 47a4a0eb-c88b-4721-a898-c336638b8ab7 | 435853.70174941793 | *Query Results* # Default Value for Geospatial Columns Source: https://docs.startree.ai/recipes/geospatial-default-value In this recipe we'll learn how to set a default value for a Geospatial point column. To learn more about using Geospatial in Apache Pinot, see the [Geospatial objects developer guide](/recipes/geospatial) or [Geospatial indexing developer guide](/recipes/geospatial-indexing). | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/geospatial-default](https://github.com/startreedata/pinot-recipes/tree/main/recipes/geospatial-default) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/geospatial-default ``` ## Launch Pinot Cluster Spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/geospatial-default/docker-compose.yml) file on GitHub. ## Generating Geospatial data This recipe contains a data generator that produces JSON documents that contain Geospatial points but occassionally null values instead. You'll need to first install the following dependencies: ```bash theme={null} pip install geofactory faker geojson shapely ``` Once that's done you can run the data generator and grab just the first generated document, by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n2 | jq ``` Output is shown below: ```json theme={null} { "tsString": 1679996635152, "uuid": "23251ad1-2748-4203-a5ab-8113788d426b", "count": 621, "pointString": null } { "tsString": 1679996637331, "uuid": "ed51e74f-851d-4245-8003-efb453bea090", "count": 378, "pointString": "POINT (165.139969 -79.693387)" } ``` You can see from this output that we have a null value in the first event and a geospatial point in the secone one. ## Kafka ingestion We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t events -Kø ``` We can check that Kafka has some data by running the following command: ```bash theme={null} docker exec -it kafka-geospatial kafka-run-class.sh \ kafka.tools.GetOffsetShell \ --broker-list localhost:9092 \ --topic events ``` We'll see something like the following: ```bash theme={null} events:0:138780 ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [ {"name": "uuid", "dataType": "STRING"}, {"name": "point", "dataType": "BYTES", "defaultNullValue": "003fe5f4a42008f90c4054e004d205fbe4"} ], "metricFieldSpecs": [{"name": "count", "dataType": "INT"}], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Note that the column for `point` has a data type of `BYTES`. Geospatial columns must use the `BYTES` type because Pinot will serialize the Geospatial objects into bytes for storage purposes. We are also passing in a `defaultNullValue` which must be a Hex encoded representation of a point. In this case the point is a location in the Arctic. > [!NOTE] > > You can get back a Hex encoded representation of a Geospatial object by running a query that returns the object. > For example: > > ```sql theme={null} > SELECT ST_GeomFromText('POINT (0.6861134172138761 83.5002942140996)') > FROM ignoreMe > ``` > > | 003fe5f4a42008f90c4054e004d205fbe4 | > | - | > | 003fe5f4a42008f90c4054e004d205fbe4 | > > *Query Results* Now for the table config: ```json theme={null} { "tableName": "events_geo", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events_geo", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-geospatial:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows":"100000", "realtime.segment.flush.threshold.time":"1h" } }, "ingestionConfig": { "transformConfigs": [ { "columnName": "point", "transformFunction": "toSphericalGeography(ST_GeomFromText(pointString))" } ] }, "tenants": {}, "metadata": {} } ``` When using a default value for a `BYTES` column we'll need to create the schema and table separately, rather than using the `AddTable` command. If we try to use the `AddTable` command, we'll end up with double decoding of the `defaultNullValue`, resulting in Pinot trying to store an invalid value. Instead, we'll create the schema with the `AddSchema` command: ```bash theme={null} docker run \ --network geospatial \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddSchema \ -schemaFile /config/schema.json \ -controllerHost "pinot-controller-geospatial" \ -exec ``` And then we'll create the table via the REST API: ```bash theme={null} curl -X POST http://localhost:9000/tables --data @config/table.json ``` ## Querying for defaults We can then the following query to check how many times the default value has been used: ```sql theme={null} SELECT STDistance( ST_GeomFromText('POINT (0.6861134172138761 83.5002942140996)'), toGeometry(point) ) AS distance, COUNT(*) FROM events GROUP BY distance ORDER BY COUNT(*) DESC, distance DESC LIMIT 10 ``` | distance | count(\*) | | - | - | | 0 | 3147529 | | 250.4421880885498 | 1 | | 250.35032602023736 | 1 | | 250.3445873086931 | 1 | | 250.32023987252305 | 1 | | 250.28500644222484 | 1 | | 250.26623501229875 | 1 | | 250.26271972684384 | 1 | | 250.25918510024783 | 1 | | 250.2511511361528 | 1 | *Query Results* # Indexing Geospatial Points Source: https://docs.startree.ai/recipes/geospatial-indexing To learn how to index and query geospatial points in Apache Pinot, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/J-4iHPolZz0) To learn about geospatial objects, see the [Geospatial objects developer guide](/recipes/geospatial). | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/geospatial-indexing](https://github.com/startreedata/pinot-recipes/tree/main/recipes/geospatial-indexing) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/geospatial-indexing ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/geospatial/docker-compose.yml) file on GitHub. ## Generating Geospatial data This recipe contains a data generator that produces JSON documents that contain various geospatial objects in [Well-known text (WKT)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) format. You'll need to first install the following dependencies: ```bash theme={null} pip install geofactory faker geojson shapely ``` Once that's done you can run the data generator and grab just the first generated document, by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "tsString": 1677845139617, "uuid": "362ec33c-6bd3-4928-8123-11ebe6a50c07", "count": 552, "pointString": "POINT (108.249223 5.520676)" } ``` You can see from this output that we have a geospatial point. Pinot also supports polygons, mutli polygons, line strings, multi points, multi line strings, and geometry collections. You can read more about this in the [Geospatial documention](https://docs.pinot.apache.org/basics/indexing/geospatial-support#geospatial-data-types). ## Kafka ingestion We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t events -Kø ``` We can check that Kafka has some data by running the following command: ```bash theme={null} docker exec -it kafka-geospatial kafka-run-class.sh \ kafka.tools.GetOffsetShell \ --broker-list localhost:9092 \ --topic events ``` We'll see something like the following: ```bash theme={null} events:0:138780 ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "events_geo", "dimensionFieldSpecs": [ {"name": "uuid", "dataType": "STRING"}, {"name": "point", "dataType": "BYTES"} ], "metricFieldSpecs": [{"name": "count", "dataType": "INT"}], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Note that the column for `point` has a data type of `BYTES`. Geospatial columns must use the `BYTES` type because Pinot will serialize the Geospatial objects into bytes for storage purposes. Now for the table config: ```json {10-19,36-41} theme={null} { "tableName": "events_geo", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events_geo", "replication": "1", "replicasPerPartition": "1" }, "fieldConfigList": [ { "name": "point", "encodingType": "RAW", "indexType": "H3", "properties": { "resolutions": "5" } } ], "tableIndexConfig": { "loadMode": "MMAP", "noDictionaryColumns": ["point"], "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-geospatial:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows":"100000", "realtime.segment.flush.threshold.time":"1h" } }, "ingestionConfig": { "transformConfigs": [ { "columnName": "point", "transformFunction": "toSphericalGeography(ST_GeomFromText(pointString))" } ] }, "tenants": {}, "metadata": {} } ``` The highlighted lines define a Geospatial index on the `point` column. If you scroll down a bit, under `ingestionConfig.transformConfigs` you can see transformation functions that converts the WKT string from our Kafka streams into a Geospatial object. We'll create the table by running the following: ```bash theme={null} docker run \ --network geospatial \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-geospatial" \ -exec ``` ## When is the Geospatial index used? Now that we've created a table that contains a column with a Geospatial index, it'd be useful to know when that index will be used. This index is used when a variety of predicates appear in the `WHERE` clause of a query. ### `ST_Distance` The `ST_Distance` function computes the distance in meters between two sets of coordinates. When used with this function, one of the arguments must be an identifer (i.e. column name) and the other a literal value. The result of calling `ST_Distance` must then be used in a range query i.e. it should be less than or greater than a specified distance. Examples are shown below: ```sql theme={null} WHERE ST_Distance(point, '') > 5000 WHERE ST_Distance('', point) < 2000` ``` ### `ST_Within` `ST_Within` checks the containment of two geospatial objects. It returns true if and only if the first geometry is completely inside the second geometry. When used with this function, the first argument must be an identifier (i.e. column name) and the other a literal value. An example is shown below: ```sql theme={null} WHERE ST_Within(point, '') ``` ### `ST_Contains` `ST_Contains` also checks the containment of two geospatial objects. More specifically, it returns true if and only if no points of the second geometry lie in the exterior of the first geometry, and at least one point of the interior of the first geometry lies in the interior of the second geometry. When used with this function, the first argument must be a literal and the other an identifier (i.e. column name). An example is shown below: ```sql theme={null} WHERE ST_Contains('', point) ``` ## Geospatial Querying Now let's head on over to the [Pinot UI](http://localhost:9000/#/query) and write some queries that use the Geospatial index. The following query counts how many points fit inside a polygon that [covers an area from Delaware to West Virginia](https://clydedacruz.github.io/openstreetmap-wkt-playground/). ![A polygon covering an area from Delaware to West Virginia](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/geospatial-polygon.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=af003cfe1415886d079309526377e540) *A polygon covering an area from Delaware to West Virginia* ```sql theme={null} select count(*) from events_geo WHERE ST_Within( point, toSphericalGeography(ST_GeomFromText('POLYGON(( -79.68778610229492 39.475226764883985, -76.83970928192137 40.48289486417028, -75.6193685531616 38.75281151479021, -77.61510372161864 37.51568305958246, -81.04884624481201 38.86621021801801, -79.68778610229492 39.475226764883985))')) ) = 1 limit 1 ``` | count(\*) | | - | | 747 | *Query Results* We can flip that query around to use `ST_Contains` like this: ```sql theme={null} select count(*) from events_geo WHERE ST_Contains( toSphericalGeography(ST_GeomFromText('POLYGON(( -79.68778610229492 39.475226764883985, -76.83970928192137 40.48289486417028, -75.6193685531616 38.75281151479021, -77.61510372161864 37.51568305958246, -81.04884624481201 38.86621021801801, -79.68778610229492 39.475226764883985))')), point ) = 1 limit 1 ``` This query will return the same results as the previous one. Finally, we can find some of the points within 50km of Washington: ```sql theme={null} select uuid, ST_AsText(point), ts, ST_Distance(point, toSphericalGeography(ST_GeomFromText( 'POINT(-77.39327430725099 38.93217314143698)'))) AS distance from events_geo WHERE distance < 50000 limit 10 ``` | uuid | stastext(point) | ts | distance | | - | - | - | - | | 1a9a7314-7550-4327-9fb5-911f636b869c | POINT (-77.729246 39.255027) | 2023-03-06 14:27:59.218 | 46146.138375698916 | | f6d3260b-93d7-478c-b7b1-2dfc214a09dc | POINT (-77.796573 39.060015) | 2023-03-06 14:40:17.591 | 37640.39151993725 | | 51693e80-cdd3-48d6-af53-78156357a5dc | POINT (-77.751046 39.115125) | 2023-03-06 14:40:17.591 | 37000.762022801944 | | fafe2a9c-ca39-4dd2-bf55-37013a1e2719 | POINT (-77.569928 39.292096) | 2023-03-06 14:04:46.509 | 42825.54468861071 | | d5329e92-edf7-4733-816e-9f2e2c580c2d | POINT (-77.188892 39.187191) | 2023-03-06 14:04:46.509 | 33399.262939756285 | | 9b7cba1f-a2e4-4690-bba9-48b71f50b915 | POINT (-77.053764 38.896884) | 2023-03-06 14:04:46.509 | 29635.04964656463 | | 1b5d5a0e-1a6b-46fb-85b4-42e5c0e208e5 | POINT (-77.618947 38.881897) | 2023-03-06 13:38:18.331 | 20311.553153107612 | | 1639f1eb-fb60-4f94-9c59-d7e595c8f416 | POINT (-76.960903 38.772332) | 2023-03-06 14:12:30.667 | 41445.58284663461 | *Query Results* ## How do I check that the Geospatial index is being used? We can check that the Geospatial index is being used by prefixing a query with `EXPLAIN PLAN FOR`, which will return the query plan. To get the query plan for the `ST_Distance` function, we'd write the following: ```sql theme={null} EXPLAIN PLAN FOR select uuid, ST_AsText(point), ts, ST_Distance(point, toSphericalGeography(ST_GeomFromText( 'POINT(-77.39327430725099 38.93217314143698)'))) AS distance from events_geo WHERE distance < 50000 limit 10 ``` | Operator | Operator\_Id | Parent\_Id | | - | - | - | | BROKER\_REDUCE(limit:10) | 1 | 0 | | COMBINE\_SELECT | 2 | 1 | | PLAN\_START(numSegmentsForThisPlan:71) | -1 | -1 | | SELECT(selectList:uuid, stastext(point), ts, st\_distance(point,'80c053592b680000014043775173125738')) | 3 | 2 | | TRANSFORM(st\_distance(point,'80c053592b680000014043775173125738'), stastext(point), ts, uuid) | 4 | 3 | | PROJECT(uuid, point, ts) | 5 | 4 | | DOC\_ID\_SET | 6 | 5 | | FILTER\_H3\_INDEX(indexLookUp:h3\_index,operator:RANGE, predicate:st\_distance( point,'80c053592b680000014043775173125738') \< '50000') | 7 | 6 | *Query Results* We must see `FILTER_H3_INDEX` as one of the operators, otherwise the index isn't being used. We can do the same thing for the `ST_Within` and `ST_Contains` queries. We should see the `INCLUSION_FILTER_H3_INDEX` operator for both these queries: For `ST_Within`, we'll see the following plan: | Operator | Operator\_Id | Parent\_Id | | - | - | - | | BROKER\_REDUCE(limit:1) | 1 | 0 | | COMBINE\_AGGREGATE | 2 | 1 | | PLAN\_START(numSegmentsForThisPlan:75) | -1 | -1 | | AGGREGATE(aggregations:count(\*)) | 3 | 2 | | TRANSFORM\_PASSTHROUGH() | 4 | 3 | | PROJECT() | 5 | 4 | | DOC\_ID\_SET | 6 | 5 | | INCLUSION\_FILTER\_H3\_INDEX(inclusionIndex:h3\_index,operator:EQ, predicate:stwithin(point,'84000000010000000600000000c053ec04b00000004043bcd43b0aae 28c05335bdcbffffff40443dcf7fb88244c052e7a3bbffffff4043605c20b209c0c053675ddb ffffff4042c201e70a0102c05443204c00000040436edff9f6ec98c053e c04b00000004043bcd43b0aae28') = '1') | 7 | 6 | *Query Results* Notice that the `INCLUSION_FILTER_H3_INDEX` operator contains the `stwithin` predicate. And for `ST_Contains`, we'll see this plan: | Operator | Operator\_Id | Parent\_Id | | - | - | - | | BROKER\_REDUCE(limit:1) | 1 | 0 | | COMBINE\_AGGREGATE | 2 | 1 | | PLAN\_START(numSegmentsForThisPlan:75) | -1 | -1 | | AGGREGATE(aggregations:count(\*)) | 3 | 2 | | TRANSFORM\_PASSTHROUGH() | 4 | 3 | | PROJECT() | 5 | 4 | | DOC\_ID\_SET | 6 | 5 | | INCLUSION\_FILTER\_H3\_INDEX(inclusionIndex:h3\_index,operator:EQ, predicate:stcontains('84000000010000000600000000c053ec04b00000004043bcd43b0aae 28c05335bdcbffffff40443dcf7fb88244c052e7a3bbffffff4043605c20b209c0c053675ddb ffffff4042c201e70a0102c05443204c00000040436edff9f6ec98c053e c04b00000004043bcd43b0aae28',point) = '1') | 7 | 6 | *Query Results* And notice on this one that the `INCLUSION_FILTER_H3_INDEX` operator contains the `stcontains` predicate. # Ingest GitHub API Events using Kinesis Source: https://docs.startree.ai/recipes/github-events-stream-kinesis ## Pull Request Merged Events Stream In this recipe, we will do the following: 1. Set up a Localstack Kinesis cluster (Optional) 2. Install AWS CLI 3. To set up a Pinot cluster, do the following: a. Start Zookeeper b. Start controller c. Start broker d. Start server 4. Create a Kinesis stream with name **pullRequestMergedEvents** 5. Create a real-time table and schema for **pullRequestMergedEvents** 6. Start a task which reads from [GitHub events API](https://developer.github.com/v3/activity/events/) and publishes events about merged pull requests to the stream. 7. Query the real-time data ## Launch localstack Kinesis This step is needed only when you don't want to use official Kinesis in AWS. It creates a mock Kinesis cluster via [Localstack](https://docs.localstack.cloud/get-started/). ```bash theme={null} docker run \ --name pinot-kinesis \ -p 127.0.0.1:4566:4566 \ localstack/localstack:0.12.15 ``` ## Install AWS CLI We need to interact with Kinesis throughout this recipe. For this purpose, we'll be using official AWS CLI to process the commands. The AWS CLI also works with localstack seamlessly. You can follow the [official AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-prereqs.html) to get started with CLI. You need to ensure the credentials are properly configured for your AWS account. In case of localstack cluster, the default credentials are as follows: ```text theme={null} accessKey: access secretKey: secret ``` ## Setup Pinot Cluster and Kinesis Tables There are multiple ways to set up the cluster. Here we'll consider only Docker and Launcher scripts. For Kubernetes and other set ups, you can check out our official documentation. **Docker** #### Pull docker image Get the latest Docker image. ``` export PINOT_VERSION=latest export PINOT_IMAGE=apachepinot/pinot:${PINOT_VERSION} docker pull ${PINOT_IMAGE} ``` ### With Quickstart Utility You can use the following single-command utility to run all the previous steps. Make sure to stop any previous running Pinot services. ```bash theme={null} $ docker run --rm -ti \ --network=pinot-demo \ --name pinot-github-events-quick-start \ ${PINOT_IMAGE} GitHubEventsQuickStart \ -personalAccessToken \ -kinesisEndpoint http://pinot-kinesis:4566 \ -sourceType kinesis \ -awsRegion us-east-1 ``` ### Without Quickstart #### Set up the Pinot cluster Follow the instructions in [Advanced Pinot Setup](https://docs.pinot.apache.org/getting-started/advanced-pinot-setup#start-pinot-components-using-docker) to set up the Pinot cluster with the components: 1. Zookeeper 2. Controller 3. Broker 4. Server #### Create a Kinesis stream Create a Kinesis stream called `pullRequestMergedEvents` for the demo. ```bash theme={null} aws kinesis create-stream \ --stream-name pullRequestMergedEvents \ --shard-count 3 \ --region us-east-1 \ --endpoint-url http://localhost:4566 ``` #### Add Pinot table and schema The schema is present at `examples/stream/githubEvents/pullRequestMergedEvents_schema.json` and is also pasted below ```json theme={null} { "schemaName": "pullRequestMergedEvents", "dimensionFieldSpecs": [ { "name": "title", "dataType": "STRING", "defaultNullValue": "" }, { "name": "labels", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "userId", "dataType": "STRING", "defaultNullValue": "" }, { "name": "userType", "dataType": "STRING", "defaultNullValue": "" }, { "name": "authorAssociation", "dataType": "STRING", "defaultNullValue": "" }, { "name": "mergedBy", "dataType": "STRING", "defaultNullValue": "" }, { "name": "assignees", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "authors", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "committers", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "requestedReviewers", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "requestedTeams", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "reviewers", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "commenters", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "repo", "dataType": "STRING", "defaultNullValue": "" }, { "name": "organization", "dataType": "STRING", "defaultNullValue": "" } ], "metricFieldSpecs": [ { "name": "count", "dataType": "LONG", "defaultNullValue": 1 }, { "name": "numComments", "dataType": "LONG" }, { "name": "numReviewComments", "dataType": "LONG" }, { "name": "numCommits", "dataType": "LONG" }, { "name": "numLinesAdded", "dataType": "LONG" }, { "name": "numLinesDeleted", "dataType": "LONG" }, { "name": "numFilesChanged", "dataType": "LONG" }, { "name": "numAuthors", "dataType": "LONG" }, { "name": "numCommitters", "dataType": "LONG" }, { "name": "numReviewers", "dataType": "LONG" }, { "name": "numCommenters", "dataType": "LONG" }, { "name": "createdTimeMillis", "dataType": "LONG" }, { "name": "elapsedTimeMillis", "dataType": "LONG" } ], "dateTimeFieldSpecs": [ { "name": "mergedTimeMillis", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:TIMESTAMP", "granularity": "1:MILLISECONDS" } ] } ``` *pullRequestMergedEvents\_schema.json* The table config is present at `examples/stream/githubEvents/docker/pullRequestMergedEvents_kinesis_realtime_table_config.json` and is also pasted below. > [!NOTE] > > If you're using official Kinesis on your AWS account, you can remove the `endpoint` property from the table config. ```json theme={null} { "tableName": "pullRequestMergedEvents", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "mergedTimeMillis", "retentionTimeUnit": "DAYS", "retentionTimeValue": "60", "schemaName": "pullRequestMergedEvents", "replication": "1", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [ "organization", "repo" ], "streamConfigs": { "streamType": "kinesis", "stream.kinesis.consumer.type": "lowlevel", "stream.kinesis.topic.name": "pullRequestMergedEvents", "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory", "realtime.segment.flush.threshold.time": "12h", "realtime.segment.flush.threshold.size": "100000", "stream.kinesis.consumer.prop.auto.offset.reset": "smallest", "region": "us-east-1", "shardIteratorType": "TRIM_HORIZON", "endpoint" : "http://pinot-kinesis:4566", "accessKey" : "access", "secretKey": "secret" } }, "metadata": { "customConfigs": {} } } ``` *pullRequestMergedEvents\_kinesis\_realtime\_table\_config.json* Add the table and schema using the following command ```bash theme={null} $ docker run \ --network=pinot-demo \ --name pinot-streaming-table-creation \ ${PINOT_IMAGE} AddTable \ -schemaFile examples/stream/githubEvents/pullRequestMergedEvents_schema.json \ -tableConfigFile examples/stream/githubEvents/docker/pullRequestMergedEvents_kinesis_realtime_table_config.json \ -controllerHost pinot-controller \ -controllerPort 9000 \ -exec ``` **Output** ```text theme={null} Executing command: AddTable -tableConfigFile examples/stream/githubEvents/docker/pullRequestMergedEvents_kinesis_realtime_table_config.json -schemaFile examples/stream/githubEvents/pullRequestMergedEvents_schema.json -controllerHost pinot-controller -controllerPort 9000 -exec Sending request: http://pinot-controller:9000/schemas to controller: 20c241022a96, version: Unknown {"status":"Table pullRequestMergedEvents_REALTIME succesfully added"} ``` #### Publish events Start streaming GitHub events into the Kinesis Stream **Prerequisites** Generate a [personal access token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line) on GitHub. ```bash theme={null} $ docker run --rm -ti \ --network=pinot-demo \ --name pinot-github-events-into-kinesis \ -d ${PINOT_IMAGE} StreamGitHubEvents \ -schemaFile examples/stream/githubEvents/pullRequestMergedEvents_schema.json \ -topic pullRequestMergedEvents \ -personalAccessToken \ -kinesisEndpoint http://pinot-kinesis:4566 \ -sourceType kinesis \ -awsRegion us-east-1 ``` **Launcher Scripts** #### Download Pinot Follow instructions in [Build from source](https://docs.pinot.apache.org/getting-started/running-pinot-locally#build-from-source-or-download-the-distribution) to get the latest Pinot code ### With Quickstart Utility For a single command to set up all the above steps ```bash theme={null} $ bin/pinot-admin.sh GitHubEventsQuickStart \ -personalAccessToken -sourceType kinesis \ -awsRegion us-east-1 \ -kinesisEndpoint http://localhost:4566 ``` ### Without Quickstart #### Set up the Pinot cluster Follow the instructions in [Advanced Pinot Setup](https://docs.pinot.apache.org/getting-started/advanced-pinot-setup#start-pinot-components-via-launcher-scripts) to set up the Pinot cluster with the components: 1. Zookeeper 2. Controller 3. Broker 4. Server #### Create a Kinesis Stream Create a Kinesis stream called `pullRequestMergedEvents` for the demo. ``` aws kinesis create-stream \ --stream-name pullRequestMergedEvents \ --shard-count 3 \ --region us-east-1 \ --endpoint-url http://localhost:4566 ``` #### Add Pinot table and schema Schema can be found at `/examples/stream/githubevents/` in the release, and is also pasted below: ```bash theme={null} { "schemaName": "pullRequestMergedEvents", "dimensionFieldSpecs": [ { "name": "title", "dataType": "STRING", "defaultNullValue": "" }, { "name": "labels", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "userId", "dataType": "STRING", "defaultNullValue": "" }, { "name": "userType", "dataType": "STRING", "defaultNullValue": "" }, { "name": "authorAssociation", "dataType": "STRING", "defaultNullValue": "" }, { "name": "mergedBy", "dataType": "STRING", "defaultNullValue": "" }, { "name": "assignees", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "authors", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "committers", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "requestedReviewers", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "requestedTeams", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "reviewers", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "commenters", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "" }, { "name": "repo", "dataType": "STRING", "defaultNullValue": "" }, { "name": "organization", "dataType": "STRING", "defaultNullValue": "" } ], "metricFieldSpecs": [ { "name": "count", "dataType": "LONG", "defaultNullValue": 1 }, { "name": "numComments", "dataType": "LONG" }, { "name": "numReviewComments", "dataType": "LONG" }, { "name": "numCommits", "dataType": "LONG" }, { "name": "numLinesAdded", "dataType": "LONG" }, { "name": "numLinesDeleted", "dataType": "LONG" }, { "name": "numFilesChanged", "dataType": "LONG" }, { "name": "numAuthors", "dataType": "LONG" }, { "name": "numCommitters", "dataType": "LONG" }, { "name": "numReviewers", "dataType": "LONG" }, { "name": "numCommenters", "dataType": "LONG" }, { "name": "createdTimeMillis", "dataType": "LONG" }, { "name": "elapsedTimeMillis", "dataType": "LONG" } ], "timeFieldSpec": { "incomingGranularitySpec": { "timeType": "MILLISECONDS", "timeFormat": "EPOCH", "dataType": "LONG", "name": "mergedTimeMillis" } } } ``` The table config is present at `examples/stream/githubEvents/docker/pullRequestMergedEvents_kinesis_realtime_table_config.json` and is also pasted below. If you're using official Kinesis on your AWS account, you can remove the `endpoint` property from the table config. ```json theme={null} { "tableName": "pullRequestMergedEvents", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "mergedTimeMillis", "retentionTimeUnit": "DAYS", "retentionTimeValue": "60", "schemaName": "pullRequestMergedEvents", "replication": "1", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [ "organization", "repo" ], "streamConfigs": { "streamType": "kinesis", "stream.kinesis.consumer.type": "lowlevel", "stream.kinesis.topic.name": "pullRequestMergedEvents", "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory", "realtime.segment.flush.threshold.time": "12h", "realtime.segment.flush.threshold.size": "100000", "stream.kinesis.consumer.prop.auto.offset.reset": "smallest", "region": "us-east-1", "shardIteratorType": "TRIM_HORIZON", "endpoint" : "http://localhost:4566", "accessKey" : "access", "secretKey": "secret" } }, "metadata": { "customConfigs": {} } } ``` *pullRequestMergedEvents\_kinesis\_realtime\_table\_config.json* Add the table and schema using the command ``` $ bin/pinot-admin.sh AddTable \ -tableConfigFile $PATH_TO_CONFIGS/examples/stream/githubEvents/pullRequestMergedEvents_kinesis_realtime_table_config.json \ -schemaFile $PATH_TO_CONFIGS/examples/stream/githubEvents/pullRequestMergedEvents_schema.json \ -exec ``` #### Publish events Start streaming GitHub events into the Kinesis Stream **Prerequisites** Generate a [personal access token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line) on GitHub. ``` $ bin/pinot-admin.sh StreamGitHubEvents \ -topic pullRequestMergedEvents \ -personalAccessToken \ -schemaFile $PATH_TO_CONFIGS/examples/stream/githubEvents/pullRequestMergedEvents_schema.json \ -sourceType kinesis \ -awsRegion us-east-1 \ -kinesisEndpoint http://localhost:4566 ``` ## Query Head over to the [Query Console](http://localhost:9000/) to check the data. ![](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/github_events_query_console.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=1a8a67685af1ca1cb7f57bfb599153ed) ### Visualizing Data You can use Superset or Tableau to visualize this data. To integrate with Superset you can check out the [Superset Integrations](https://docs.pinot.apache.org/integrations/superset) page. You can also use our [JDBC driver](https://docs.pinot.apache.org/users/clients/jdbc) to connect Tableau to Pinot. Here are some insights captured via Tableau- #### Most Active organizations in last 1 hour ![](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/github_events_bubble_chart_tableau.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=cf3856389d515cb95d2f8c9a25f0e07a) #### Total commits happening every minute ![](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/github_events_line_chart_tableau.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=55d3f8db72525f0b37d0dd285d493633) ## Resharding Kinesis Stream Pinot's Kinesis plugin has been designed to handle resharding in a Kinesis stream gracefully. Pinot ensures data in parent shards is consumed before children shards. Pinot creates segment per partition. Currently, partitions are mapped 1:1 with the shards. We take the last index of the shardID as the partition number. e.g. `shardId-000000000000` is partition 0, `shardId-000000000001` is partition 1, `shardId-000000000002` is partition 2, and so on. Each of the Pinot's data segment contains partition id as part of their name. e.g. segment name `pullRequestMergedEvents__5__0__20220315T2036Z` is composed of * tableName * partitionId * segmentNumber in the current partition * current timestamp in `yyyyMMddTHHmm` format This allows us to easily check if data from new shards after split/merge is being consumed or not. Let's test it out in our current setup. The segments for the stream without split look as follows ![](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/github_kinesis_presplit_segments.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=b4d944d9e7122a0678eb5c7b63ae4ecc) > [!NOTE] > > New shards are detected by `RealtimeSegmentValidationManager` which is a periodic task that runs in Controller. You can also trigger this task manually to check new segments instead of waiting for configured interval. > > ```bash theme={null} > curl -X GET "http://localhost:9000/periodictask/run?taskname=RealtimeSegmentValidationManager&tableName=pullRequestMergedEvents" -H "accept: application/json" > ``` ### Split a Shard Let's first list out all the shards ```bash theme={null} aws kinesis list-shards --stream-name pullRequestMergedEvents --region us-east-1 ``` ```json theme={null} { "Shards": [ { "ShardId": "shardId-000000000000", "HashKeyRange": { "StartingHashKey": "0", "EndingHashKey": "113427455640312821154458202477256070484" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660285068355537287018713152919383991364343226695682" } }, { "ShardId": "shardId-000000000001", "HashKeyRange": { "StartingHashKey": "113427455640312821154458202477256070485", "EndingHashKey": "226854911280625642308916404954512140969" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660285090656282485549336294455102264012704732676114" } }, { "ShardId": "shardId-000000000002", "HashKeyRange": { "StartingHashKey": "226854911280625642308916404954512140970", "EndingHashKey": "340282366920938463463374607431768211455" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660285112957027684079959435990820536661066238656546" } } ] } ``` Now let's split the `shardId-000000000000` in the middle. ```bash theme={null} aws kinesis split-shard --stream-name pullRequestMergedEvents --shard-to-split shardId-000000000000 --new-starting-hash-key 56713727820156410577229101238628035242 ``` This should result in following shards. `shardId-000000000003` and `shardId-000000000004` are two new shards. Also, `shardId-000000000000` is closed now since it contains an `EndingSequenceNumber`. ```json theme={null} { "Shards": [ { "ShardId": "shardId-000000000000", "HashKeyRange": { "StartingHashKey": "0", "EndingHashKey": "113427455640312821154458202477256070484" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660285068355537287018713152919383991364343226695682", "EndingSequenceNumber": "49627660285079505909886284024722478317308158351041888258" } }, { "ShardId": "shardId-000000000001", "HashKeyRange": { "StartingHashKey": "113427455640312821154458202477256070485", "EndingHashKey": "226854911280625642308916404954512140969" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660285090656282485549336294455102264012704732676114" } }, { "ShardId": "shardId-000000000002", "HashKeyRange": { "StartingHashKey": "226854911280625642308916404954512140970", "EndingHashKey": "340282366920938463463374607431768211455" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660285112957027684079959435990820536661066238656546" } }, { "ShardId": "shardId-000000000003", "ParentShardId": "shardId-000000000000", "HashKeyRange": { "StartingHashKey": "0", "EndingHashKey": "56713727820156410577229101238628035241" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660724013923279965274008000462144529148390395609138" } }, { "ShardId": "shardId-000000000004", "ParentShardId": "shardId-000000000000", "HashKeyRange": { "StartingHashKey": "56713727820156410577229101238628035242", "EndingHashKey": "113427455640312821154458202477256070484" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660724036224025163804631141997862801796751901589570" } } ] } ``` Let's check if data is being consumed in Pinot from these new shards. ![](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/github_kinesis_postsplit_segments.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=762c07c2094e0b89b58f5f6b536a00d6) As we can see, segments with partitionIDs 3 and 4 are now visible. e.g. `pullRequestMergedEvents__3__0__20220315T2028Z` and `pullRequestMergedEvents__4__0__20220315T2028Z` ### Merge two shards Now let's merge `shardId-000000000001` and `shardId-000000000002 `to create a new shard `shardId-000000000005`. ```bash theme={null} aws kinesis merge-shards --stream-name pullRequestMergedEvents --shard-to-merge shardId-000000000001 --adjacent-shard-to-merge shardId-000000000002 ``` New shard ids should look as follows - ```json theme={null} { "Shards": [ { "ShardId": "shardId-000000000000", "HashKeyRange": { "StartingHashKey": "0", "EndingHashKey": "113427455640312821154458202477256070484" }, "SequenceNumberRange": { "ShardId": "shardId-000000000000", "HashKeyRange": { "StartingHashKey": "0", "EndingHashKey": "113427455640312821154458202477256070484" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660285068355537287018713152919383991364343226695682", "EndingSequenceNumber": "49627660285079505909886284024722478317308158351041888258" } }, { "ShardId": "shardId-000000000001", "HashKeyRange": { "StartingHashKey": "113427455640312821154458202477256070485", "EndingHashKey": "226854911280625642308916404954512140969" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660285090656282485549336294455102264012704732676114", "EndingSequenceNumber": "49627660285101806655084814647864014035580838735824027666" } }, { "ShardId": "shardId-000000000002", "HashKeyRange": { "StartingHashKey": "226854911280625642308916404954512140970", "EndingHashKey": "340282366920938463463374607431768211455" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660285112957027684079959435990820536661066238656546", "EndingSequenceNumber": "49627660285124107400283345271005549753853487097330008098" } }, { "ShardId": "shardId-000000000003", "ParentShardId": "shardId-000000000000", "HashKeyRange": { "StartingHashKey": "0", "EndingHashKey": "56713727820156410577229101238628035241" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660724013923279965274008000462144529148390395609138" } }, { "ShardId": "shardId-000000000004", "ParentShardId": "shardId-000000000000", "HashKeyRange": { "StartingHashKey": "56713727820156410577229101238628035242", "EndingHashKey": "113427455640312821154458202477256070484" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660724036224025163804631141997862801796751901589570" } }, { "ShardId": "shardId-000000000005", "ParentShardId": "shardId-000000000001", "AdjacentParentShardId": "shardId-000000000002", "HashKeyRange": { "StartingHashKey": "113427455640312821154458202477256070485", "EndingHashKey": "340282366920938463463374607431768211455" }, "SequenceNumberRange": { "StartingSequenceNumber": "49627660890332880970606661397573849021940660525273710674" } } ] ``` Let's check if data is being consumed in Pinot from new shardID `shardId-000000000005` ![](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/github_kinesis_postmerge_segments.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=1443134c357243fbec50501fde7afffa) As we can see, segments with partitionID 5 such as `pullRequestMergedEvents__5__1__20220315T2036Z` are now visible. # How to use Google Cloud Storage as a Deep Store Source: https://docs.startree.ai/recipes/google-cloud-storage-deep-store In this recipe we'll learn how to use Google Cloud Storage as a Deep Store for Apache Pinot segments. The [deep store](https://docs.pinot.apache.org/basics/components/deep-store) (or deep storage) is the permanent store for segment files and is used for backup and restore operations. | Pinot Version | 0.9.3 | | - | - | | Code | [startreedata/pinot-recipes/google-cloud-storage](https://github.com/startreedata/pinot-recipes/tree/main/recipes/google-cloud-storage) | ## Prerequisites To follow the code examples in this guide, do the following: * [Install Docker](https://docs.docker.com/get-docker/) and the [Google Cloud CLI](https://cloud.google.com/sdk/docs/install) locally. * Create a GCP project and a user or service account that has permission to list and create buckets, and then navigate to [https://console.cloud.google.com/storage/browser](https://console.cloud.google.com/storage/browser) and create a bucket, for example `pinot-deepstore.yourdomain.com` * Download recipes ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/google-cloud-storage ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/google-cloud-storage/docker-compose.yml) file on GitHub. ## Controller configuration We need to provide configuration parameters to the Pinot Controller to configure MinIO as the Deep Store. This is done in the following section of the Docker Compose file: ```yml theme={null} pinot-controller: image: apachepinot/pinot:0.10.0 command: "StartController -zkAddress zookeeper-gcs:2181 -config /config/controller-conf.conf" ``` The configuration is specified in `/config/controller-conf.conf`, the contents of which are shown below: ```conf theme={null} controller.access.protocols.http.port=9000 controller.zk.str=zookeeper-gcs:2181 controller.helix.cluster.name=PinotCluster controller.host=pinot-controller-gcs controller.port=9000 controller.data.dir=gs:// controller.local.temp.dir=/tmp/pinot-tmp-data pinot.controller.segment.fetcher.protocols=file,http,gs pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.controller.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS pinot.controller.storage.factory.gs.projectId= pinot.controller.storage.factory.gs.gcpKey=/config/service-account.json ``` */config/controller-conf.conf* Let's go through some of these properties: * `controller.data.dir` contains the name of our bucket. * `pinot.controller.storage.factory.gs.projectId` contains the name of our GCP project. * `pinot.controller.storage.factory.gs.gcpKey` contains the path to our GCP JSON key file. You'll need to update the following lines: ``` controller.data.dir=gs:// pinot.controller.storage.factory.gs.projectId= ``` * Replace `` with the name of your bucket. * Replace `` with the name of your GCP project. You should also paste the contents of your GCP JSON key file into `config/service-account.json`. ## Pinot Schema and Tables Now let's create a Pinot Schema and real-time table. ### Schema Our schema is going to capture some simple events, and looks like this: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [ { "name": "uuid", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "count", "dataType": "INT" } ], "dateTimeFieldSpecs": [{ "name": "ts", "dataType": "TIMESTAMP", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` *config/schema.json* You can create the schema by running the following command: ```bash theme={null} docker exec -it pinot-controller-gcs bin/pinot-admin.sh AddSchema \ -schemaFile /config/schema.json \ -exec ``` ### Real-Time Table And the real-time table is defined below: ```json theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1", "retentionTimeUnit": "DAYS", "retentionTimeValue": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-gcs:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "10000", "realtime.segment.flush.threshold.time": "1h", "realtime.segment.flush.threshold.segment.size": "5M" } }, "tenants": {}, "metadata": {}, "task": { "taskTypeConfigsMap": { } } } ``` *config/table-realtime.json* > [!WARNING] > > The `realtime.segment.flush.threshold.rows` config is intentionally set to an extremely small value so that the segment will be committed after 10,000 records have been ingested. In a production system this value should be set much higher, as described in the [configuring segment threshold](/recipes/configuring-segment-threshold) guide. You can create the table by running the following command: ```bash theme={null} docker exec -it pinot-controller-gcs bin/pinot-admin.sh AddTable \ -tableConfigFile /config/table-realtime.json \ -exec ``` ## Ingesting Data Let's ingest data into the `events` Kafka topic, by running the following: ```bash theme={null} while true; do ts=`date +%s%N | cut -b1-13`; uuid=`cat /proc/sys/kernel/random/uuid | sed 's/[-]//g'` count=$[ $RANDOM % 1000 + 0 ] echo "{\"ts\": \"${ts}\", \"uuid\": \"${uuid}\", \"count\": $count}" done | docker exec -i kafka-minio /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 \ --topic events ``` Data will make its way into the real-time table. We can see how many records have been ingested by running the following query: ```sql theme={null} SELECT count(*) FROM events ``` ## Exploring Deep Store Now we're going to check what segments we have and where they're stored. You can get a list of all segments by running the following: ``` curl -X GET \ "http://localhost:9000/segments/events" \ -H "accept: application/json" 2>/dev/null | jq '.[] | .REALTIME[]' ``` The output is shown below: **Output** ```json theme={null} "events__0__0__20220505T1339Z" "events__0__1__20220505T1342Z" "events__0__2__20220505T1342Z" "events__0__3__20220505T1343Z" "events__0__4__20220505T1344Z" ``` Let's pick one of these segments, `events__0__3__20220505T1343Z` and get its metadata, by running the following: ```bash theme={null} tableName="events" segmentName="events__0__3__20220505T1343Z" curl -X GET \ "http://localhost:9000/segments/${tableName}/${segmentName}/metadata" \ -H "accept: application/json" 2>/dev/null | jq '.' ``` The output is shown below: **Output** ```json {4} theme={null} { "segment.crc": "532660340", "segment.creation.time": "1651758198369", "segment.download.url": "gs://pinot-events/events/events__0__3__20220505T1343Z", "segment.end.time": "1651758238283", "segment.flush.threshold.size": "10000", "segment.index.version": "v3", "segment.realtime.endOffset": "40000", "segment.realtime.numReplicas": "1", "segment.realtime.startOffset": "30000", "segment.realtime.status": "DONE", "segment.start.time": "1651758188443", "segment.time.unit": "MILLISECONDS", "segment.total.docs": "10000" } ``` We can see from the highlighted line that this segment is persisted at `gs://pinot-events/events/events__0__3__20220505T1343Z`. Let's go back to the terminal and return a list of all the segments in the bucket: ```bash theme={null} bucketName="pinot-events" gsutil ls -l gs://${bucketName}/events/ ``` The output is shown below: **Output** ```text theme={null} 256712 2022-05-05T13:42:07Z gs://pinot-events/events/events__0__0__20220505T1339Z 256817 2022-05-05T13:42:32Z gs://pinot-events/events/events__0__1__20220505T1342Z 257174 2022-05-05T13:43:15Z gs://pinot-events/events/events__0__2__20220505T1342Z 257224 2022-05-05T13:44:05Z gs://pinot-events/events/events__0__3__20220505T1343Z TOTAL: 4 objects, 1027927 bytes (1003.83 KiB) ``` # Groovy Transformation Functions Source: https://docs.startree.ai/recipes/groovy-transformation-functions In this recipe we'll learn how to use Groovy transformation functions to ingest a CSV file with column names contain spaces. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/groovy-transformation-functions](https://github.com/startreedata/pinot-recipes/tree/main/recipes/groovy-transformation-functions) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. Navigate to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/groovy-transformation-functions ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster and Kafka Broker by running the following command: ```bash theme={null} docker compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/groovy-transformation-functions/docker-compose.yml) file on GitHub. ## Controller configuration We need to provide configuration parameters to the Pinot Controller to enable Groovy in transformation functions. This is done in the following section of the Docker Compose file: ```yml theme={null} pinot-controller: image: apachepinot/pinot:1.0.0 command: "StartController -zkAddress zookeeper-grooy:2181 -config /config/controller-conf.conf" ``` The configuration is specified in `/config/controller-conf.conf`, the contents of which are shown below: ```conf theme={null} controller.access.protocols.http.port=9000 controller.zk.str=zookeeper-groovy:2181 controller.helix.cluster.name=PinotCluster controller.host=pinot-controller-groovy controller.port=9000 controller.data.dir=/tmp/data/PinotController controller.disable.ingestion.groovy=false ``` */config/controller-conf.conf* ## Dataset We're going to import a couple of JSON documents into Kafka and then from there into Pinot. ```json theme={null} {"timestamp": "2019-10-09 21:25:25", "payload": {"firstName": "James", "lastName": "Smith", "before": {"id": 2}, "after": { "id": 3}}} {"timestamp": "2019-10-10 21:33:25", "payload": {"firstName": "John", "lastName": "Gates", "before": {"id": 2}}} ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. Only the *timestamp* field from our data source maps to a schema column name - we'll be using transformation functions to populate the *id* and *name* columns. ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [ { "name": "id", "dataType": "INT" }, { "name": "name", "dataType": "STRING" } ], "dateTimeFieldSpecs": [{ "name": "timestamp", "dataType": "TIMESTAMP", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` *config/schema.json* The table config indicates that data will be ingested from the Kafka `events` topic: ```json theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestamp", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "query" : { "disableGroovy": false }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-groovy:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder" } }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" }, "transformConfigs": [ { "columnName": "id", "transformFunction": "Groovy({def jsonSlurper = new groovy.json.JsonSlurper(); def object = jsonSlurper.parseText(new groovy.json.JsonBuilder(payload).toPrettyString()); def result = object.after == null ? Long.valueOf(object.before.id) : Long.valueOf(object.after.id); return result}, payload)" }, { "columnName": "name", "transformFunction": "Groovy({def jsonSlurper = new groovy.json.JsonSlurper(); def object = jsonSlurper.parseText(new groovy.json.JsonBuilder(payload).toPrettyString()); return object.firstName + ' ' + object.lastName}, payload)" } ] }, "metadata": {} } ``` *config/table.json* Let's dive into the [transformation functions](https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations) defined under `ingestionConfig.transformConfigs`: * The *id* one extracts `payload.after.id` if the `after` property exists, otherwise it uses `payload.before.id` * The *name* one concatenates `payload.firstName` and `payload.lastName` They both use Groovy's JSON parser to create an object from the payload, before using logic from the programming language to return the desired out. > [!NOTE] > > If you only need to do simple data transformation, you can use the [built-in transformation functions](https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#inbuilt-pinot-functions). We can add the table and schema by running the following command: ```bash theme={null} docker run \ --network groovy \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-groovy" \ -exec ``` ## Ingest Data into Kafka We can run the following command to import a couple of documents into Kafka: ```bash theme={null} printf '{"timestamp": "2019-10-09 21:25:25", "payload": {"firstName": "James", "lastName": "Smith", "before": {"id": 2}, "after": { "id": 3}}} {"timestamp": "2019-10-10 21:33:25", "payload": {"firstName": "John", "lastName": "Gates", "before": {"id": 2}}}\n' | kcat -P -b localhost:9092 -t events ``` Let's check those documents have been imported by running the following command: ```bash theme={null} kcat -C -b localhost:9092 -t events -c2 ``` ```json theme={null} {"timestamp": "2019-10-09 21:25:25", "payload": {"firstName": "James", "lastName": "Smith", "before": {"id": 2}, "after": { "id": 3}}} {"timestamp": "2019-10-09 21:25:25", "payload": {"firstName": "James", "lastName": "Smith", "before": {"id": 2}}} ``` *Output* Looks good so far. ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `events` table or copy/paste the following query: ```sql theme={null} select * from events limit 10 ``` You will see the following output: | id | name | timestamp | | - | - | - | | 3 | James Smith | 2019-10-09 21:25:25.0 | | 2 | John Gates | 2019-10-10 21:33:25.0 | *Query Results* # How to Handle Null Values Source: https://docs.startree.ai/recipes/handle-null-values In this recipe we'll learn how to handle null or missing values in Apache Pinot tables. | Pinot Version | 1.10.0 | | - | - | | Code | [startreedata/pinot-recipes/null-values](https://github.com/startreedata/pinot-recipes/tree/main/recipes/null-values) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/null-values ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/json-unnest/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following JSON file: ```json theme={null} {"title": "Valentine's Day", "genre": "Comedy", "year": 2010, "id": 361248901147483647} {"title": "The Ugly Truth", "year": 2009, "id": 332567813147483648} {"title": "P.S. I Love You", "genre": "Romance", "year": 2007, "id": 346905752147483649} {"title": "Dear John", "year": 2010, "id": 300441473147483650} {"title": "The Curious Case of Benjamin Button", "genre": "Fantasy", "year": 2008, "id": 394030854147483651} ``` *data/import.json* `The Ugly Truth` and `Dear John` are both missing a value for `genre`, which we'll explore in this guide. ## Pinot Schema and Table Now let's create a couple of Pinot Schemas and Table. We're going to create one table where nulls are allowed and one where they aren't. We need to create one schema per table even though they will be identical except for the name. ```json theme={null} { "schemaName": "movies_nulls", "dimensionFieldSpecs": [ { "name": "id", "dataType": "LONG" }, { "name": "title", "dataType": "STRING" }, { "name": "genre", "dataType": "STRING" }, { "name": "year", "dataType": "INT" } ] } ``` *config/schema.json* And the schema for the no-nulls table: ```json theme={null} { "schemaName": "movies_no_nulls", "dimensionFieldSpecs": [ { "name": "id", "dataType": "LONG" }, { "name": "title", "dataType": "STRING" }, { "name": "genre", "dataType": "STRING" }, { "name": "year", "dataType": "INT" } ] } ``` *config/schema.json* We're also going to create two tables: one that allows null values and one that doesn't. This table doesn't handle null values: ```json theme={null} { "tableName": "movies_no_nulls", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1, "schemaName": "movies_no_nulls" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "metadata": {} } ``` *config/table\_no\_nulls.json* You can create the table by running the following command:\` ```bash theme={null} docker run \ --network null-values \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml \ -values tableName='movies_no_nulls' ``` And this table allows null values: ```json {6} theme={null} { "tableName": "movies_nulls", "tableType": "OFFLINE", "tableIndexConfig": { "loadMode": "MMAP", "nullHandlingEnabled": "true" }, "segmentsConfig": { "replication": 1, "schemaName": "movies_nulls" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "metadata": {} } ``` *config/table\_nulls.json* The highlighted config is how we indicate that we want this table to have null values. You can create the table by running the following command:\` ```bash theme={null} docker run \ --network null-values \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml \ -values tableName='movies_nulls' ``` ## Ingestion Job Now we’re going to import the JSON file into these tables. We'll do this with the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/import.json' outputDirURI: '/opt/pinot/data/${tableName}' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: ${tableName} pinotClusterSpecs: - controllerURI: 'http://pinot-controller-nulls:9000' pushJobSpec: pushParallelism: 2 pushAttempts: 2 ``` *config/job-spec.yml* The import job will map fields in each JSON document to a corresponding column in the `movies` schema. You can run the following command to run the import on the `movies_no_null` table: ```bash theme={null} docker run \ --network null-values \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml \ -values tableName='movies_no_nulls' ``` And the following to run the import on the `movies_nuls` table: ```bash theme={null} docker run \ --network null-values \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml \ -values tableName='movies_nulls' ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and run the following query to return the rows that have a genre and year in the `movies_no_nulls` table: ```sql theme={null} select * from movies_no_nulls WHERE genre IS NOT NULL AND year IS NOT NULL; ``` You will see the following output: | genre | id | title | year | | - | - | - | - | | Comedy | 361248901147483647 | Valentine's Day | 2010 | | null | 332567813147483648 | The Ugly Truth | 2009 | | Romance | 346905752147483649 | P.S. I Love You | 2007 | | null | 300441473147483650 | Dear John | 2010 | | Fantasy | 394030854147483651 | The Curious Case of Benjamin Button | -2147483648 | *Query Results* We can see from the results that the null genres and null years haven't been filtered out for the `movies_no_nulls` table. The `genre` and `year` columns in the `movies_nulls` table, on the other hand, supports null values, which we can see by running the following query: ```sql theme={null} select * from movies_nulls WHERE genre IS NOT NULL AND year IS NOT NULL; ``` You will see the following output: | genre | id | title | year | | - | - | - | - | | Comedy | 361248901147483647 | Valentine's Day | 2010 | | Romance | 346905752147483649 | P.S. I Love You | 2007 | *Query Results* Pinot still stores the same default values for those columns, but when null handling is enabled the query engine filters them out of the result set when evaluating the `IS NOT NULL` clause. # Import data files from different directories Source: https://docs.startree.ai/recipes/import-data-files-different-directories In this recipe we'll learn how to import CSV files from different directories. | Pinot Version | 0.9.3 | | - | - | | Code | [startreedata/pinot-recipes/import-data-files-different-directories](https://github.com/startreedata/pinot-recipes/import-data-files-different-directories) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/import-data-files-different-directories ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/import-data-files-different-directories/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following CSV files: ### input/2000\_2009/2000\_2009.csv ```csv theme={null} title,genre,year,id "The Da Vinci Code","Thriller",2006,331567813147483648 "Ice Age: The Meltdown","Children",2006,337567813147483648 "P.S. I Love You","Romance",2007,346905752147483649 "The Curious Case of Benjamin Button","Fantasy",2008,394030854147483651 "Harry Potter and the Half-Blood Prince","Fantasy",2009,323030454547483651 "The Ugly Truth","Comedy",2009,332567813147483648 ``` ### input/2010\_2019/2010\_2019.csv ```csv theme={null} title,genre,year,id "Dear John","Drama",2010,300441473147483650 "Valentine's Day","Comedy",2010,361248901147483647 "Minions","Comedy",2015,333567813147483648 ``` ### input/2020\_present/2020\_present.csv ```csv theme={null} title,genre,year,id "Uncharted","Action",2022,493030854147483651 ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. A common pattern when creating a schema is to create columns that map directly to the names of the fields in our data source. We can't do that in this case since column names can't contain spaces, so instead we'll have the following: ### config/schema.json ```json theme={null} { "schemaName": "movies", "dimensionFieldSpecs": [{ "name": "id", "dataType": "LONG" }, { "name": "title", "dataType": "STRING" }, { "name": "genre", "dataType": "STRING" }, { "name": "year", "dataType": "INT" } ] } ``` We'll also have the following table config: ### config/table.json\* ```json theme={null} { "tableName": "movies", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1 }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" }, "transformConfigs": [] }, "metadata": {} } ``` You can create the table and schema by running the following command: ```bash theme={null} docker run \ --network csv \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -tableConfigFile /config/table.json \ -schemaFile /config/schema.json \ -controllerHost "pinot-controller-csv" \ -exec ``` You should see a message similar to the following if everything is working correctly: ```text theme={null} 2021/11/25 12:02:04.606 INFO [AddTableCommand] [main] Executing command: AddTable -tableConfigFile /config/table.json -schemaFile /config/schema.json -controllerProtocol http -controllerHost 192.168.144.3 -controllerPort 9000 -user null -password [hidden] -exec 2021/11/25 12:02:05.084 INFO [AddTableCommand] [main] {"status":"Table movies_OFFLINE succesfully added"} ``` ## Ingestion Job Now we’re going to import the CSV file into Pinot. We'll do this with the following ingestion spec: ### config/job-spec.yml ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentNameGeneratorSpec: configs: use.global.directory.sequence.id: true jobType: SegmentCreationAndTarPush inputDirURI: '/input' includeFileNamePattern: 'glob:**/*.csv' outputDirURI: '/data' pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'movies' pinotClusterSpecs: - controllerURI: 'http://pinot-controller-csv:9000' pushJobSpec: pushParallelism: 2 pushAttempts: 2 ``` The part of this file that we're most interested in is the following section: ```yaml theme={null} segmentNameGeneratorSpec: configs: use.global.directory.sequence.id: true ``` This config makes sure that the segment id (that's used in the segment name) is incremented globally across all directories, rather than being incremented on a directory by directory basis. If we don't set this config, every segment that's created from a CSV file will be replaced by the segment created from the next CSV file. We would end up with only one segment that contained the rows from the last CSV that got processed. > [!NOTE] > > We only need to use this config when importing data for tables that don't have a timestamp field. If a timestamp field is specified, timestamps are used in the segment name and the likelihood of segments being overridden is reduced. You can run the following command to run the import: ```bash theme={null} docker run \ --network csv \ -v $PWD/config:/config \ -v $PWD/data:/data \ -v $PWD/input:/input \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `crimes` table or copy/paste the following query: ```sql theme={null} select * from movies limit 10 ``` You will see the following output: | genre | id | title | year | | - | - | - | - | | Action | 493030854147483651 | Uncharted | 2022 | | Drama | 300441473147483650 | Dear John | 2010 | | Comedy | 361248901147483647 | Valentine's Day | 2010 | | Comedy | 333567813147483648 | Minions | 2015 | | Thriller | 331567813147483648 | The Da Vinci Code | 2006 | | Children | 337567813147483648 | Ice Age: The Meltdown | 2006 | | Romance | 346905752147483649 | P.S. I Love You | 2007 | | Fantasy | 394030854147483651 | The Curious Case of Benjamin Button | 2008 | | Fantasy | 323030454547483651 | Harry Potter and the Half-Blood Prince | 2009 | | Comedy | 332567813147483648 | The Ugly Truth | 2009 | *Query Results* # Infer Pinot Schema from Avro Schema Source: https://docs.startree.ai/recipes/infer-schema-avro In this recipe, we'll learn how to infer a Pinot schema from an Avro schema. | Pinot Version | 0.9.0 | | - | - | | Code | [startreedata/pinot-recipes/infer-schema-avro-data](https://github.com/startreedata/pinot-recipes/tree/main/recipes/infer-schema-avro-data) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In the terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/infer-schema-avro-data ``` ## JSON Data We're going to infer a Pinot schema from the following Avro schema: ```json theme={null} { "type":"record", "namespace":"com.simonaubury", "name":"MastodonDetails", "fields":[ {"name":"m_id","type":"long","default":0}, {"name":"created_at","type": "int","logicalType":"date"}, {"name":"created_at_str","type":"string","default":"unknown"}, {"name":"app","type":"string","default":"unknown"}, {"name":"url","type":"string","default":"null"}, {"name":"base_url","type":"string","default":"null"}, {"name":"language","type":"string","default":"null"}, {"name":"favourites","type":"int","default":0}, {"name":"username","type":"string","default":""}, {"name":"bot","type":"boolean"}, {"name":"tags","type":"int","default":0}, {"name":"characters","type":"int","default":0}, {"name":"words","type":"int","default":0}, {"name":"mastodon_text","type":"string","default":"null"} ] } ``` */avro/mastodon-topic-value.avsc* ## Infer schema Now we're going to infer a schema for this input file. We can do this using the `AvroSchemaToPinotSchema` command. You can generate a schema file that creates a dimension column field per Aro field, by running the following command: ```bash theme={null} docker run \ -v ${PWD}/config:/config \ -v ${PWD}/avro:/avro \ apachepinot/pinot:1.0.0 AvroSchemaToPinotSchema \ -avroSchemaFile /avro/mastodon-topic-value.avsc \ -pinotSchemaName="mastodon" \ -outputDir="/config" \ -dimensions="" \ -timeColumnName "created_at" \ -metrics "favourites,words,characters,tags" ``` A JSON file will be written to `./config/mastodon.json`, the contents of which are shown below: ```json theme={null} { "schemaName": "mastodon", "dimensionFieldSpecs": [ {"name": "m_id", "dataType": "LONG"}, {"name": "created_at_str", "dataType": "STRING"}, {"name": "app", "dataType": "STRING"}, {"name": "url", "dataType": "STRING"}, {"name": "base_url", "dataType": "STRING"}, {"name": "language", "dataType": "STRING"}, {"name": "username", "dataType": "STRING"}, {"name": "bot", "dataType": "BOOLEAN"}, {"name": "mastodon_text", "dataType": "STRING"} ], "metricFieldSpecs": [ {"name": "favourites", "dataType": "INT"}, {"name": "tags", "dataType": "INT"}, {"name": "characters", "dataType": "INT"}, {"name": "words", "dataType": "INT"} ], "dateTimeFieldSpecs": [ { "name": "created_at", "dataType": "INT", "format": "1:DAYS:EPOCH", "granularity": "1:DAYS" } ] } ``` *./config/mastodon.json* # Infer Pinot Schema from JSON Data Source: https://docs.startree.ai/recipes/infer-schema-json-data In this recipe we'll learn how to infer a Pinot schema from a JSON input file. | Pinot Version | 0.9.0 | | - | - | | Code | [startreedata/pinot-recipes/infer-schema-json-data](https://github.com/startreedata/pinot-recipes/tree/main/recipes/infer-schema-json-data) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/infer-schema-json-data ``` ## JSON Data We're going to infer a Pinot schema from the following input file: ```json theme={null} { "id":"7044874109", "type":"PushEvent", "actor":{ "id":18542751, "login":"LimeVista", "display_login":"LimeVista", "gravatar_id":"", "url":"https://api.github.com/users/LimeVista", "avatar_url":"https://avatars.githubusercontent.com/u/18542751?" }, "repo":{ "id":115911530, "name":"LimeVista/Tapes", "url":"https://api.github.com/repos/LimeVista/Tapes" }, "payload":{ "push_id":2226018068, "size":1, "distinct_size":1, "ref":"refs/heads/master", "head":"c5fc8b32a9ead1eba315d97410cb4ac1e6ca1774", "before":"892d872c5d3f24cc6837900c9f4618dc2fe92930", "commits":[ { "sha":"c5fc8b32a9ead1eba315d97410cb4ac1e6ca1774", "author":{ "name":"Lime", "email":"4cc153d999e24274955157fc813e6f92f821525d@outlook.com" }, "message":"Merge branch 'master' of https://github.com/LimeVista/Tapes\\n\\n# Conflicts:\\n#\\t.gitignore", "distinct":true, "url":"https://api.github.com/repos/LimeVista/Tapes/commits/c5fc8b32a9ead1eba315d97410cb4ac1e6ca1774" } ] }, "public":true, "created_at":"2018-01-01T11:00:00Z" } ``` */data/github.json* ## Infer schema Now we're going to infer a schema for this input file. We can do this using the `JsonToPinotSchema` command. ### Schema with only dimension fields You can generate a schema file that creates a dimension column field per JSON field, by running the following command: ```bash theme={null} docker run \ -v ${PWD}/data/github.json:/data/github.json \ -v ${PWD}/config:/config \ apachepinot/pinot:0.9.3 JsonToPinotSchema \ -jsonFile /data/github.json \ -pinotSchemaName="github" \ -outputDir="/config" \ -dimensions="" ``` A JSON file will be written to `./config/github.json`, the contents of which are shown below: ```json theme={null} { "schemaName" : "github", "dimensionFieldSpecs" : [ { "name" : "id", "dataType" : "STRING" }, { "name" : "type", "dataType" : "STRING" }, { "name" : "actor.id", "dataType" : "INT" }, { "name" : "actor.login", "dataType" : "STRING" }, { "name" : "actor.display_login", "dataType" : "STRING" }, { "name" : "actor.gravatar_id", "dataType" : "STRING" }, { "name" : "actor.url", "dataType" : "STRING" }, { "name" : "actor.avatar_url", "dataType" : "STRING" }, { "name" : "repo.id", "dataType" : "INT" }, { "name" : "repo.name", "dataType" : "STRING" }, { "name" : "repo.url", "dataType" : "STRING" }, { "name" : "payload.push_id", "dataType" : "LONG" }, { "name" : "payload.size", "dataType" : "INT" }, { "name" : "payload.distinct_size", "dataType" : "INT" }, { "name" : "payload.ref", "dataType" : "STRING" }, { "name" : "payload.head", "dataType" : "STRING" }, { "name" : "payload.before", "dataType" : "STRING" }, { "name" : "payload.commits", "dataType" : "STRING" }, { "name" : "public", "dataType" : "BOOLEAN" }, { "name" : "created_at", "dataType" : "STRING" } ] } ``` *./config/github.json* ### Schema with time column We should probably specify the `created_at` field as a date time field, which we can do using the following command: ```bash theme={null} docker run \ -v ${PWD}/data/github.json:/data/github.json \ -v ${PWD}/config:/config \ apachepinot/pinot:0.9.3 JsonToPinotSchema \ -jsonFile /data/github.json \ -pinotSchemaName="github_with_ts" \ -outputDir="/config" \ -timeColumnName=created_at ``` A JSON file will be written to `./config/github.json`, the contents of which are shown below: ```json theme={null} { "schemaName" : "github_with_ts", "dimensionFieldSpecs" : [ { "name" : "id", "dataType" : "STRING" }, { "name" : "type", "dataType" : "STRING" }, { "name" : "actor.id", "dataType" : "INT" }, { "name" : "actor.login", "dataType" : "STRING" }, { "name" : "actor.display_login", "dataType" : "STRING" }, { "name" : "actor.gravatar_id", "dataType" : "STRING" }, { "name" : "actor.url", "dataType" : "STRING" }, { "name" : "actor.avatar_url", "dataType" : "STRING" }, { "name" : "repo.id", "dataType" : "INT" }, { "name" : "repo.name", "dataType" : "STRING" }, { "name" : "repo.url", "dataType" : "STRING" }, { "name" : "payload.push_id", "dataType" : "LONG" }, { "name" : "payload.size", "dataType" : "INT" }, { "name" : "payload.distinct_size", "dataType" : "INT" }, { "name" : "payload.ref", "dataType" : "STRING" }, { "name" : "payload.head", "dataType" : "STRING" }, { "name" : "payload.before", "dataType" : "STRING" }, { "name" : "payload.commits", "dataType" : "STRING" }, { "name" : "public", "dataType" : "BOOLEAN" } ], "dateTimeFieldSpecs" : [ { "name" : "created_at", "dataType" : "STRING", "format" : "1:DAYS:EPOCH", "granularity" : "1:DAYS" } ] } ``` *./config/github\_with\_ts.json* ### Schema with unnested fields At the moment the `payload.commits` JSON array is stored as a string. We can unnest each of the values in the documents in the array, by running the following command: ```bash theme={null} docker run \ -v ${PWD}/data/github.json:/data/github.json \ -v ${PWD}/config:/config \ apachepinot/pinot:0.9.3 JsonToPinotSchema \ -jsonFile /data/github.json \ -pinotSchemaName="github_unnest" \ -outputDir="/config" \ -timeColumnName=created_at \ -fieldsToUnnest=payload.commits ``` A JSON file will be written to `./config/github_unnest.json`, the contents of which are shown below: ```json theme={null} { "schemaName" : "github_unnest", "dimensionFieldSpecs" : [ { "name" : "id", "dataType" : "STRING" }, { "name" : "type", "dataType" : "STRING" }, { "name" : "actor.id", "dataType" : "INT" }, { "name" : "actor.login", "dataType" : "STRING" }, { "name" : "actor.display_login", "dataType" : "STRING" }, { "name" : "actor.gravatar_id", "dataType" : "STRING" }, { "name" : "actor.url", "dataType" : "STRING" }, { "name" : "actor.avatar_url", "dataType" : "STRING" }, { "name" : "repo.id", "dataType" : "INT" }, { "name" : "repo.name", "dataType" : "STRING" }, { "name" : "repo.url", "dataType" : "STRING" }, { "name" : "payload.push_id", "dataType" : "LONG" }, { "name" : "payload.size", "dataType" : "INT" }, { "name" : "payload.distinct_size", "dataType" : "INT" }, { "name" : "payload.ref", "dataType" : "STRING" }, { "name" : "payload.head", "dataType" : "STRING" }, { "name" : "payload.before", "dataType" : "STRING" }, { "name" : "payload.commits.sha", "dataType" : "STRING" }, { "name" : "payload.commits.author.name", "dataType" : "STRING" }, { "name" : "payload.commits.author.email", "dataType" : "STRING" }, { "name" : "payload.commits.message", "dataType" : "STRING" }, { "name" : "payload.commits.distinct", "dataType" : "BOOLEAN" }, { "name" : "payload.commits.url", "dataType" : "STRING" }, { "name" : "public", "dataType" : "BOOLEAN" } ], "dateTimeFieldSpecs" : [ { "name" : "created_at", "dataType" : "STRING", "format" : "1:DAYS:EPOCH", "granularity" : "1:DAYS" } ] } ``` *./config/github\_unnest.json* # Ingesting Avro Messages Source: https://docs.startree.ai/recipes/ingest-avro In this recipe we'll learn how to ingest Avro messages from Apache Kafka into Apache Pinot. The Avro schema will be stored in the Confluent Schema Registry, so we'll learn how to integrate with that as well. Watch the following video about ingesting Avro encoded messages into Apache Pinot, or follow the tutorial below. [YouTube video player](https://www.youtube.com/embed/u2DdOZDKnQw) | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/ingest-avro](https://github.com/startreedata/pinot-recipes/tree/main/recipes/ingest-avro) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/ingest-avro ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, Zookeeper, and the Confluent Schema Registry. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/ingest-avro/docker-compose.yml) file on GitHub. ## Data Generator This recipe contains a data generator that creates events with data about people. It uses the [Faker library](https://faker.readthedocs.io/en/master/), so you'll first need to install that: ```bash theme={null} pip install faker ``` You can generate data by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "ts": 1680614486181, "person": { "id": "43e44826-44f5-4474-9a12-6740bfc0e7d8", "name": "Elizabeth Morales", "email": "qmyers@example.com", "age": 40, "address": { "street_address": "2555 Garza Trafficway", "city": "Denisefurt", "state": "Nebraska", "country": "Mauritania" }, "phone_number": "001-901-538-1610x7283", "job": { "company": "Reynolds, Hansen and Alexander", "position": "Psychotherapist, dance movement", "department": "harness robust eyeballs" }, "interests": [ "Music" ], "friend_ids": [ "34ba8e1f-4d94-4624-9920-8c32bb2e0e2e" ] } } ``` ## Avro Schema The Avro schema for our messages is described below: ```json theme={null} { "type": "record", "name": "PersonMessage", "namespace": "ai.startree", "fields": [ {"name": "ts", "type": "long"}, { "name": "person", "type": { "type": "record", "name": "Person", "fields": [ {"name": "id", "type": "string"}, {"name": "name", "type": "string"}, {"name": "email", "type": "string"}, {"name": "age", "type": "int"}, { "name": "address", "type": { "type": "record", "name": "Address", "fields": [ {"name": "street_address", "type": "string"}, {"name": "city", "type": "string"}, {"name": "state", "type": "string"}, {"name": "country", "type": "string"} ] } }, {"name": "phone_number", "type": "string"}, { "name": "job", "type": { "type": "record", "name": "Job", "fields": [ {"name": "company", "type": "string"}, {"name": "position", "type": "string"}, {"name": "department", "type": "string"} ] } }, {"name": "interests", "type": [{"type": "array", "items": "string"}, "null"]}, {"name": "friend_ids", "type": [{"type": "array", "items": "string"}, "null"]} ] } } ] } ``` *avro/person-topic-value.avsc* ## Kafka ingestion We're going to ingest the stream of people into Kafka using the Kafka Python client. We'll need to install the following libraries: ```bash theme={null} pip install avro confluent-kafka ``` We're going to stream the messages produced by `datagen.py` into the following script: ```python theme={null} import json import sys from confluent_kafka.avro import AvroProducer from confluent_kafka import avro def kafka_producer(schema_name): producer_config = { 'bootstrap.servers': 'localhost:9092', 'schema.registry.url': 'http://localhost:8081', 'broker.address.family': 'v4' } value_schema = avro.load(f"avro/{schema_name}-value.avsc") producer = AvroProducer(producer_config, default_value_schema=value_schema) return schema_name, producer if __name__ == "__main__": topic_name, producer = kafka_producer(schema_name="person-topic") while True: line = sys.stdin.readline().rstrip('\n') if not line: break event = json.loads(line) producer.produce(topic = topic_name, value = event) producer.flush() ``` *kafkaproducer.py* This script first creates a Kafka producer that knows about the Avro schema and schema registry. It then infinitely ingests an infinite stream from `stdin` and writes Avro messages to Kafka. We can combine the data generator script with this one by running the following code: ```bash theme={null} python datagen.py | python kafkaproducer.py ``` Once we've done that, let's check the messages are being ingested into the `person-topic` using kcat: ```bash theme={null} kcat -C -b localhost:9092 -t person-topic \ -r localhost:8081 -s value=avro \ -o end -c1 | jq ``` The output is shown below: ```json theme={null} { "ts": 1680615014367, "person": { "id": "04da5a02-4984-46d7-83e4-1e769235e2ab", "name": "Wayne Jacobs", "email": "austinpittman@example.org", "age": 22, "address": { "street_address": "571 Shaun Corner", "city": "Port Marcfort", "state": "Florida", "country": "Argentina" }, "phone_number": "595.121.1530x308", "job": { "company": "Gentry-Castillo", "position": "Broadcast journalist", "department": "innovate enterprise channels" }, "interests": null, "friend_ids": null } } ``` Next, we're going to ingest the stream into Pinot. ## Pinot schema and table Our Pinot schema is shown below: ```json theme={null} { "schemaName": "people", "dimensionFieldSpecs": [ {"name": "person.id", "dataType": "STRING"}, {"name": "person.name", "dataType": "STRING"}, {"name": "person.email", "dataType": "STRING"}, {"name": "person.age", "dataType": "INT"}, {"name": "person.address.street_address", "dataType": "STRING"}, {"name": "person.address.city", "dataType": "STRING"}, {"name": "person.address.state", "dataType": "STRING"}, {"name": "person.address.country", "dataType": "STRING"}, {"name": "person.phone_number", "dataType": "STRING"}, {"name": "person.job.company", "dataType": "STRING"}, {"name": "person.job.position", "dataType": "STRING"}, {"name": "person.job.department", "dataType": "STRING"}, {"name": "person.interests", "dataType": "STRING", "singleValueField": false}, {"name": "person.friend_ids", "dataType": "STRING", "singleValueField": false} ], "metricFieldSpecs": [ ], "dateTimeFieldSpecs": [ {"name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS"} ] } ``` *schema.json* And the table config is below: ```json {18-21} theme={null} { "tableName": "people", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "timeType": "MILLISECONDS", "schemaName": "people", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowLevel", "stream.kafka.topic.name": "person-topic", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "stream.kafka.decoder.prop.format": "AVRO", "stream.kafka.decoder.prop.schema.registry.rest.url": "http://schema-registry:8081", "stream.kafka.decoder.prop.schema.registry.schema.name": "person-topic-value", "stream.kafka.broker.list": "kafka:9093", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} }, "ingestionConfig": { "complexTypeConfig": { "delimiter": "." } }, "routing": { "instanceSelectorType": "strictReplicaGroup" } } ``` *table.json* The highlighted section defines a decoder for processing Avro messages via the schema registry. The URL for the schema registry is defined along with the Avro schema name. We can create the table and schema by running the following command: ```bash theme={null} docker run \ --network ingestavro \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller" \ -exec ``` ## Querying Pinot We can then run the following query via the [Pinot UI](http://localhost:9000/#/query?query=select+person.id%2C+person.interests%2C+person.age%2C+person.address.country%0Afrom+people+%0Awhere+person.friend_ids+%3C%3E+%27null%27%0AAND+person.interests+%3C%3E+%27null%27%0Aand+ARRAYLENGTH%28person.interests%29+%3E+1%0A%0Alimit+10\&tracing=false\&useMSE=false): ```sql theme={null} select person.id, person.interests, person.age, person.address.country from people where person.friend_ids <> 'null' AND person.interests <> 'null' and ARRAYLENGTH(person.interests) > 1 limit 10 ``` The output will look something like this: | person.id | person.interests | person.age | person.address.country | | - | - | - | - | | 311b5e5e-060a-45f3-b050-e7b9e268dceb | \["Photography","Cooking","Fashion","Running"] | 45 | Zambia | | d29e52eb-41d8-4902-92fb-8ed6f51115b1 | \["Cycling","Dancing","Music"] | 60 | Qatar | | b1a0cf13-8daf-4120-865f-7ac3ae51c3c4 | \["Yoga","Painting","Swimming"] | 48 | Sierra Leone | | f774187e-ee0a-4a2f-abbb-e73ba86a1dfb | \["Music","Gardening"] | 67 | Belize | | c47cf80b-8104-480f-a9de-046f10f4c3f0 | \["Art","Meditation"] | 31 | Mayotte | | c62586ff-24d9-43cb-9bb5-50b7e9a4a297 | \["Painting","Yoga"] | 57 | Azerbaijan | | 175a0032-af11-4e58-8ab5-9bfbed70eabf | \["Baking","Fashion","Cooking","Baking","Gardening"] | 73 | Wallis and Futuna | | 82940700-00e1-4d3a-8333-d30165fec35a | \["Fashion","Fashion","Fishing"] | 35 | Saint Martin | | f27d06e9-686e-42f3-a6d8-742c046ea3d7 | \["Sports","Reading","Reading","Cycling","Traveling"] | 27 | Russian Federation | | 0f1189ce-77ae-46cb-9f78-bf841b9f7e86 | \["Sports","Reading","Baking","Reading"] | 74 | Sri Lanka | *Query Results* # Ingest CSV files from a S3 bucket Source: https://docs.startree.ai/recipes/ingest-csv-files-from-s3 In this recipe, we will learn how Apache Pinot can be configured to ingest CSV files from an AWS S3 bucket. ## Prerequisites You will need a running Pinot cluster locally to follow the code examples in this guide. Make sure you have the following configured in the machine that you are going to deploy Apache Pinot. * A valid account in AWS. * A local installation of AWS CLI along with your [credentials configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html). * An AWS [access key](https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/) for having programmatic access to S3. ## Create an S3 bucket Let's create an S3 bucket called `pinot-demo` to keep the source CSV files. You can use the AWS CLI to do that. ```bash theme={null} aws s3 mb s3://pinot-demo ``` ## Copy CSV files Create a CSV file called `transcript.csv` with the following content. ```csv theme={null} studentID,firstName,lastName,gender,subject,score,timestampInEpoch 200,Lucy,Smith,Female,Maths,3.8,1570863600000 200,Lucy,Smith,Female,English,3.5,1571036400000 201,Bob,King,Male,Maths,3.2,1571900400000 202,Nick,Young,Male,Physics,3.6,1572418800000 ``` Now, copy that file into S3 bucket using the CLI. ```bash theme={null} aws s3 cp transcript.csv s3://pinot-demo/rawdata/transcript.csv ``` ## Configure Pinot Now that we have the CSV file in the S3 bucket. Let's configure Pinot to ingest it and create a segment out of it. First, let's create a schema and a table definition for the transcript data set. Create the `transcript_schema.json` as follows. ```json theme={null} { "schemaName": "transcript", "dimensionFieldSpecs": [ { "name": "studentID", "dataType": "INT" }, { "name": "firstName", "dataType": "STRING" }, { "name": "lastName", "dataType": "STRING" }, { "name": "gender", "dataType": "STRING" }, { "name": "subject", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "score", "dataType": "FLOAT" } ], "dateTimeFieldSpecs": [{ "name": "timestampInEpoch", "dataType": "LONG", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` Create the `transcript_table.json` as follows. ```json theme={null} { "tableName": "transcript", "segmentsConfig" : { "timeColumnName": "timestampInEpoch", "timeType": "MILLISECONDS", "replication" : "1", "schemaName" : "transcript" }, "tableIndexConfig" : { "invertedIndexColumns" : [], "loadMode" : "MMAP" }, "tenants" : { "broker":"DefaultTenant", "server":"DefaultTenant" }, "tableType":"OFFLINE", "metadata": {} } ``` Upload the table using following command. ```bash theme={null} bin/pinot-admin.sh AddTable \ -tableConfigFile transcript_table.json \ -schemaFile transcript_schema.json -exec ``` ## Create the ingestion job spec file Create a file called `job-spec.yml` and add the following content to it. ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: 's3://pinot-demo/rawdata/' includeFileNamePattern: 'glob:**/*.csv' outputDirURI: 's3://pinot-demo/segments' overwriteOutput: true pinotFSSpecs: - scheme: s3 className: org.apache.pinot.plugin.filesystem.S3PinotFS configs: region: us-east-1 recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'transcript' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' ``` Let's break down this specification further for better understanding. The following configuration block instructs Pinot to use S3 as the underlying file system implementation, with the `className` pointing to the implementation class. ```yaml theme={null} pinotFSSpecs: - scheme: s3 className: org.apache.pinot.plugin.filesystem.S3PinotFS configs: region: us-east-1 ``` Below, `inputDirURI` specifies the S3 bucket location where Pinot should ingest the data from. If you remember, we copied the `transcript.csv` file into that folder. The directive `includeFileNamePattern` filters all CSV files in that folder. Once the ingestion is completed, Pinot writes the segments into the location specified by `outputDirURI` ```yaml theme={null} inputDirURI: 's3://pinot-demo/rawdata/' includeFileNamePattern: 'glob:**/*.csv' outputDirURI: 's3://pinot-demo/segments' ``` We are reading CSV files. Hence, the following configuration block uses the CSV record reader format. ```yaml theme={null} recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' ``` ### Initiate the ingestion job Now we have everything in place. Let's go ahead and kick off the ingestion by running: ```bash theme={null} bin/pinot-admin.sh LaunchDataIngestionJob \ -jobSpecFile job-spec.yml ``` Once it is completed, you should see the `transcript` table populated with data in the [Query Console](http://localhost:9000/#/query). # Ingesting JSON files Source: https://docs.startree.ai/recipes/ingest-json-files Learn how to ingest JSON documents from a newline-delimited JSON (jsonlines) file. Watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/zuuQwqkXNK8) | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/ingest-json-files](https://github.com/startreedata/pinot-recipes/tree/main/recipes/ingest-json-files) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. Clone this repository and navigate to this recipe: ```bash theme={null} git clone git@github.com:startreedata/pinot-recipes.git cd pinot-recipes/recipes/ingest-json-files ``` ## Run the recipe Spin up a Pinot cluster using Docker Compose: ```bash theme={null} docker compose up ``` Open another tab to add the `movies` table: ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -tableConfigFile /config/table.json \ -schemaFile /config/schema.json \ -controllerHost "pinot-controller-json" \ -exec ``` Import `data/ingest.json` into Pinot: ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml ``` Navigate to [http://localhost:9000/#/query](http://localhost:9000/#/query) and run the following query: ```sql theme={null} select * from movies limit 10 ``` You will see the following output: | genre | id | title | year | | - | - | - | - | | Drama | 300441473147483650 | Dear John | 2010 | | Comedy | 332567813147483648 | The Ugly Truth | 2009 | | Romance | 346905752147483649 | P.S. I Love You | 2007 | | Comedy | 361248901147483647 | Valentine's Day | 2010 | | Fantasy | 394030854147483651 | The Curious Case of Benjamin Button | 2008 | # Ingesting JSON files from Kafka Source: https://docs.startree.ai/recipes/ingest-json-files-kafka In this recipe, we'll learn how to ingest JSON documents from Apache Kafka into Apache Pinot. To learn how to ingest JSON files from Apache Kafka into Pinot, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/OCV13UMQrgA) [*Mark Needham shows how to ingest JSON files*](https://twitter.com/markhneedham) If you have complex JSON documents with a nested structure, see how to ingest [complex JSON documents with a nested structure from Kafka into Pinot](https://www.youtube.com/watch?v=Jc03u8rXc2w). | Pinot Version | 0.10.0 | | - | - | | Code | [startreedata/pinot-recipes/ingest-json-files-kafka](https://github.com/startreedata/pinot-recipes/tree/main/recipes/ingest-json-files-kafka) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/ingest-json-files-kafka ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/ingest-json-files-kafka/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following JSON files: ```json theme={null} {"title": "Valentine's Day", "genre": "Comedy", "year": 2010, "releaseDate": "2010-02-12 00:00:00", "budget": 52000000, "boxOffice": 216500000} {"title": "The Ugly Truth", "genre": "Comedy", "year": 2009, "releaseDate": "2010-04-14 00:00:00", "budget": 38000000, "boxOffice": 205300000} {"title": "P.S. I Love You", "genre": "Romance", "year": 2007, "releaseDate": "2007-12-21 00:00:00", "budget": 30000000, "boxOffice": 156800000} {"title": "Dear John", "genre": "Drama", "year": 2010, "releaseDate": "2010-04-14 00:00:00", "budget": 25000000, "boxOffice": 115000000} {"title": "The Curious Case of Benjamin Button", "genre": "Fantasy", "year": 2008, "releaseDate": "2008-12-25 00:00:00", "budget": 167000000, "boxOffice": 335800000} ``` *data/import1.jsonl* ```json theme={null} {"title": "Pirates of the Caribbean: Salazar's Revenge", "genre": "Action", "year": 2017, "releaseDate": "2017-05-26 00:00:00", "budget": 230000000, "boxOffice": 794881442} {"title": "The Hunger Games", "genre": "Action", "year": 2012, "releaseDate": "2012-03-23 00:00:00", "budget": 78000000, "boxOffice": 694394724} {"title": "Pride & Prejudice", "genre": "Romance", "year": 2005, "releaseDate": "2005-09-16 00:00:00", "budget": 28000000, "boxOffice": 121616555} ``` *data/import2.jsonl* ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "movies", "dimensionFieldSpecs": [ { "name": "title", "dataType": "STRING" }, { "name": "genre", "dataType": "STRING" }, { "name": "year", "dataType": "INT" } ], "metricFieldSpecs": [ { "name": "budget", "dataType": "INT" }, { "name": "boxOffice", "dataType": "INT" } ], "dateTimeFieldSpecs": [ { "name": "releaseDate", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` *config/schema.json* We'll also have the following table config: ```json {13-16,19} theme={null} { "tableName": "movies", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "releaseDate", "timeType": "MILLISECONDS", "schemaName": "movies", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "streamConfigs": { "streamType": "kafka", "stream.kafka.broker.list": "kafka-json:9093", "stream.kafka.consumer.type": "lowLevel", "stream.kafka.topic.name": "events", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" }, "loadMode": "MMAP" }, "task": { "taskTypeConfigsMap": {} }, "metadata": { "customConfigs": {} } } ``` *config/table.json* We need to tell Pinot where our Kafka cluster lives as well as the topic that we wish to pull events from. Finally, we need to specify an offset value, which indicates where Pinot should start pulling data in each topic partition. A value of `smallest` means it will start from the earliest offset. A value of `largest` means it will start from the latest offset. You can create the table and schema by running the following command:\` ```bash theme={null} docker exec -it pinot-controller-json bin/pinot-admin.sh AddTable \ -tableConfigFile /config/table.json \ -schemaFile /config/schema.json \ -exec ``` ## Importing data Now we're going to import the JSON files into Kafka: ```bash theme={null} docker exec -i kafka-json kafka-console-producer.sh \ --bootstrap-server kafka-json:9092 \ --topic events < data/import1.jsonl ``` ```bash theme={null} docker exec -i kafka-json kafka-console-producer.sh \ --bootstrap-server kafka-json:9092 \ --topic events < data/import2.jsonl ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `movies` table or copy/paste the following query: ```sql theme={null} select * from movies limit 10 ``` You will see the following output: | boxOffice | budget | genre | releaseDate | title | year | | - | - | - | - | - | - | | 216500000 | 52000000 | Comedy | 2010-02-12 00:00:00.0 | Valentine's Day | 2010 | | 205300000 | 38000000 | Comedy | 2010-04-14 00:00:00.0 | The Ugly Truth | 2009 | | 156800000 | 30000000 | Romance | 2007-12-21 00:00:00.0 | P.S. I Love You | 2007 | | 115000000 | 25000000 | Drama | 2010-04-14 00:00:00.0 | Dear John | 2010 | | 335800000 | 167000000 | Fantasy | 2008-12-25 00:00:00.0 | The Curious Case of Benjamin Button | 2008 | | 794881442 | 230000000 | Action | 2017-05-26 00:00:00.0 | Pirates of the Caribbean: Salazar's Revenge | 2017 | | 694394724 | 78000000 | Action | 2012-03-23 00:00:00.0 | The Hunger Games | 2012 | | 121616555 | 28000000 | Romance | 2005-09-16 00:00:00.0 | Pride & Prejudice | 2005 | *Query Results* # Ingest Parquet files from an S3 Bucket into Pinot Using Spark Source: https://docs.startree.ai/recipes/ingest-parquet-files-from-s3-using-spark In this recipe we'll learn how to ingest Parquet formatted data from an AWS S3 bucket into a Pinot cluster. The ingestion will be executed as an Apache Spark job. Once the data is ingested, we will run ad hoc queries against Pinot to derive valuable insights with the lowest latency. These are the plugins we will be using throughout the tutorial: * pinot-batch-ingestion-spark * pinot-s3 * pinot-parquet | Pinot Version | 0.9.3 | | - | - | | Code | [startreedata/pinot-recipes/ingest-parquet-files-from-s3-using-spark](https://github.com/startreedata/pinot-recipes/tree/main/recipes/ingest-parquet-files-from-s3-using-spark) | ## Prerequisites This tutorial requires you have these on your local workstation. * JDK version 11 or higher * [Apache Spark 2.4.8 with Hadoop 2.7](https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz) * [Apache Pinot 0.9.3 binary distribution](https://downloads.apache.org/pinot/apache-pinot-0.9.3/apache-pinot-0.9.3-bin.tar.gz) * An [AWS S3](https://aws.amazon.com/s3/) account with reading and write permissions * [AWS CLI](https://aws.amazon.com/cli/) installation with properly configured [access key credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html). ## Overview To understand how these plugins work together, let’s take an example of a fictitious mobile application. The company Foo recently launched a mobile application. Now, the engineering team is curious to learn the user engagement statistics of the application. They are planning to calculate the daily active users (DAU) metrics to quantify that information. The mobile app emits an event similar to this whenever a user performs an activity in the application. ```json theme={null} { "eventId":"950ce51e-2f14-406a-a17f-27279f181ef3", "userId":1234, "eventType":"Comment", "ts":1633114228000 } ``` These events are sent to the backend and subsequently accumulated into an S3 bucket in Parquet format. Our goal is to ingest those events into Pinot and then write a SQL query to calculate the DAU. ![Architecture](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/pinot-parquet-overview.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=599cf531e4b6e153ebacd63f9d2eaa09) *Architecture* ## Launch Pinot Cluster Once you have everything ready, follow [this guide](https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally#batch) to spin up a single-node Pinot cluster. You can start Pinot using the `quick-start-batch` launcher script. ```bash theme={null} bin/quick-start-batch.sh ``` ## Prepare the input data set To simulate the mobile app events, we will use a sample CSV file that you can download from [here](https://github.com/startreedata/pinot-recipes/blob/main/recipes/ingest-parquet-files-from-s3-using-spark/data.csv). Having this file in CSV format helps us to spot errors and correct them quickly. ### Convert the CSV file into Parquet Once you have downloaded the file, we will convert it into Parquet format using Apache Spark. To do that, open up a terminal and navigate to the location where you’ve extracted the Spark distribution. Then execute the following command to launch the Spark-scala shell. We are using Spark shell here as we don’t want to write a fully-fledged Spark script to get this done. ```bash theme={null} /bin/spark-shell ``` Once the shell is running, execute the below commands in the shell. ```scala theme={null} val df = spark.read.format("csv").option("header", true).load("/path/to/events.csv") df.write.option("compression","none").mode("overwrite").parquet("/path/to/output") ``` The first command above creates a Spark data frame out of the CSV file. Make sure to provide the exact location of the CSV file. The second command writes the data frame as a Parquet file into the path specified. If the Spark job was successful, you should see .parquet files inside the /path/to/output directory. ### Upload the Parquet file to S3 Now we have our Parquet file in place. Let’s go ahead and upload that into an S3 bucket. You can use AWS CLI or AWS console for that based on your preference. We will use the AWS CLI to upload the Parquet files into an S3 bucket called `pinot-spark-demo`: ```bash theme={null} aws s3 cp /path/to/output s3://pinot-spark-demo/rawdata/ --recursive ``` ## Create a table and schema We need to create a table to query the data that will be ingested. All tables in Pinot are associated with a schema. You can check out the [Table configuration](https://docs.pinot.apache.org/configuration-reference/table) and [Schema configuration](https://docs.pinot.apache.org/configuration-reference/schema) for more details. Create a file called `events_schema.json` with a text editor and add the following content. That represents the schema definition to capture events. ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [ { "name": "eventId", "dataType": "STRING" }, { "name": "userId", "dataType": "INT" }, { "name": "eventType", "dataType": "STRING" } ], "metricFieldSpecs": [], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` *events\_schema.json* Notice that the attribute ts has been defined under the date-time section. We will be filtering events using the timestamp attribute. Hence, we explicitly declared ts as a data-time field candidate. Create another file called `events_table.json` with the following content. That represents the events table. ```json theme={null} { "tableName": "events", "segmentsConfig": { "timeColumnName": "ts", "timeType": "MILLISECONDS", "replication": "1", "schemaName": "events" }, "tableIndexConfig": { "invertedIndexColumns": [], "loadMode": "MMAP" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableType": "OFFLINE", "metadata": {} } ``` *events\_table.json* On a terminal, navigate to the location where Pinot distribution is running. Execute the following command from the `/bin` directory to define events schema and the table. ```bash theme={null} ./pinot-admin.sh AddTable \ -tableConfigFile /path/to/events_table.json \ -schemaFile /path/to/events_schema.json -exec ``` If the above command is successful, you should see the events schema and table that has been created inside Pinot data explorer, which you can access over at [http://localhost:9000/](http://localhost:9000/). ![Table Overview](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/parquet-query.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=fa45695ccbbfbdbf7b3d65431f43f35a) *Table Overview* ## Ingest data from S3 Now that we have an empty table inside Pinot. Let’s go ahead and ingest the Parquet files in the S3 bucket into the running Pinot instance. Batch Data ingestion in Pinot involves the following steps. * Read data and generate compressed segment files from the input * Upload the compressed segment files to the output location * Push the location of the segment files to the controller This process can be illustrated as follows. ![Spark Data Ingestion](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/parquet-spark.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=b1306d299e3265d66981fae72eea1401) *Spark Data Ingestion* Once the location is available to the controller, it can notify the servers to download the segment files and populate the tables. The above steps can be performed using any distributed executor of your choice, such as Hadoop, Spark, Flink, etc. For this tutorial, we will be using Apache Spark to execute the steps. Pinot provides runners for Spark out of the box. As a user, you don’t need to write a single line of code. You can write runners for any other executor using our provided interfaces. Let’s create a specification file for the data ingestion job and name it as events-s3.yml. Add the following content to it. ```yml theme={null} executionFrameworkSpec: name: spark segmentGenerationJobRunnerClassName: >- org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentTarPushJobRunner segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentUriPushJobRunner jobType: SegmentCreationAndMetadataPush inputDirURI: 's3://pinot-spark-demo/rawdata' outputDirURI: 's3://pinot-spark-demo/output/' overwriteOutput: true pinotFSSpecs: - scheme: s3 className: org.apache.pinot.plugin.filesystem.S3PinotFS configs: region: us-east-1 recordReaderSpec: dataFormat: parquet className: org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader tableSpec: tableName: events schemaURI: 'http://localhost:9000/tables/events/schema' tableConfigURI: 'http://localhost:9000/tables/events' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' pushJobSpec: pushParallelism: 2 pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *events-s3.yml* We have kept the execution framework as Spark in the job specification and configured the appropriate runners for each of our steps. Also we set the ingestion `jobType` as `SegmentCreationAndMetadataPush`, which is more performant and lightweight for the controller. When using this jobType, make sure that controllers and servers have access to the outputDirURI so that they can download segments from directly. Other supported jobTypes are: * SegmentCreation * SegmentTarPush * SegmentUriPush * SegmentCreationAndTarPush * SegmentCreationAndUriPush * SegmentCreationAndMetadataPush We also provide the S3 Filesystem and Parquet reader implementation in the config to use. You can refer to [Ingestion Job Spec](https://docs.pinot.apache.org/configuration-reference/job-specification) for a complete list of configurations. Once everything is in place, let’s kick off the ingestion job by executing the below command. ```bash theme={null} export PINOT_VERSION=0.8.0 export PINOT_DISTRIBUTION_DIR=/path/to/apache-pinot-0.8.0-bin export SPARK_HOME=/path/to/spark-2.4.8-bin-hadoop2.7 cd ${PINOT_DISTRIBUTION_DIR} ${SPARK_HOME}/bin/spark-submit \ --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \ --master "local[2]" \ --deploy-mode client \ --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \ --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/pinot-batch-ingestion-spark-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-${PINOT_VERSION}-shaded.jar" \ local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \ -jobSpecFile '/Users/dunith/Work/ingestion-specs/events-s3.yml' ``` Make sure to replace the values for `PINOT_DISTRIBUTION_DIR` and `SPARK_HOME` to match with your local environment. The above command includes the JARs of all the required plugins in Spark’s driver classpath. In practice, you only need to do this if you get a ClassNotFoundException. The command will take a few seconds to complete based on your machine’s performance. If the ingestion job doesn’t throw out any errors to the console, that means you have successfully populated the events table with data coming from the S3 bucket. The data explorer shows the populated events table as follows. ## Querying Pinot to calculate the daily active users (DAU) Here comes the exciting part. Now we have data ingested into Pinot. Let’s write some SQL queries to calculate DAU based on the user engagement events. The mobile app has been released to the public. Its audience will grow daily. The first time users install the app, some will continue to use it regularly or from time to time, and some will quickly lose interest. However, these user segments are considered active because they had at least one session in the app at a particular time. Let’s use SQL to calculate the number of unique users in the last three months. Copy and paste the following query into the Query console and run. ```sql theme={null} select ToDateTime(ts, 'yyyy-MM-dd') as t_date, count(distinct userId) as DAU from events where ts > now() - 86400000*30*3 group by t_date order by 1 asc ``` That should result in the following output: ![Query Results](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/parquet-query-result.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=58695ea20ba83140d7c9cc25d4413932) *Query Results* ## What next? Now we have seen the power of Pinot’s pluggable architecture. It enables you to mix and match different data source implementations and input formats to optimize your data ingestion strategy. In a data-driven organization, that flexibility allows data engineers to build data pipelines that cater to various data sources and workloads. For example, you can switch from an on-premise HDFS storage to a cloud data lake like S3 just by changing a few lines in the ingestion specification. Data analysts and data scientists will also benefit from Pinot’s ability to ingest vast amounts of data from data lakes and give them low-latency access to exploratory analysis. You can take this tutorial to the next level by configuring a data visualization like Apache Superset, which complements the interactive analysis by giving you more visualization capabilities. # How to flatten objects in JSON documents Source: https://docs.startree.ai/recipes/json-flatten In this recipe we'll learn how to flatten nested objects when ingesting JSON documents into Apache Pinot. | Pinot Version | 0.10.0 | | - | - | | Code | [startreedata/pinot-recipes/json-flatten](https://github.com/startreedata/pinot-recipes/tree/main/recipes/json-flatten) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/json-flatten ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/json-flatten/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following JSON file: ```json theme={null} {"id":1,"name":"Leanne Graham","username":"Bret","email":"Sincere@april.biz","address":{"street":"Kulas Light","suite":"Apt. 556","city":"Gwenborough","zipcode":"92998-3874","geo":{"lat":"-37.3159","lng":"81.1496"}},"phone":"1-770-736-8031 x56442","website":"hildegard.org","company":{"name":"Romaguera-Crona","catchPhrase":"Multi-layered client-server neural-net","bs":"harness real-time e-markets"}} {"id":2,"name":"Ervin Howell","username":"Antonette","email":"Shanna@melissa.tv","address":{"street":"Victor Plains","suite":"Suite 879","city":"Wisokyburgh","zipcode":"90566-7771","geo":{"lat":"-43.9509","lng":"-34.4618"}},"phone":"010-692-6593 x09125","website":"anastasia.net","company":{"name":"Deckow-Crist","catchPhrase":"Proactive didactic contingency","bs":"synergize scalable supply-chains"}} {"id":3,"name":"Clementine Bauch","username":"Samantha","email":"Nathan@yesenia.net","address":{"street":"Douglas Extension","suite":"Suite 847","city":"McKenziehaven","zipcode":"59590-4157","geo":{"lat":"-68.6102","lng":"-47.0653"}},"phone":"1-463-123-4447","website":"ramiro.info","company":{"name":"Romaguera-Jacobson","catchPhrase":"Face to face bifurcated interface","bs":"e-enable strategic applications"}} {"id":4,"name":"Patricia Lebsack","username":"Karianne","email":"Julianne.OConner@kory.org","address":{"street":"Hoeger Mall","suite":"Apt. 692","city":"South Elvis","zipcode":"53919-4257","geo":{"lat":"29.4572","lng":"-164.2990"}},"phone":"493-170-9623 x156","website":"kale.biz","company":{"name":"Robel-Corkery","catchPhrase":"Multi-tiered zero tolerance productivity","bs":"transition cutting-edge web services"}} {"id":5,"name":"Chelsey Dietrich","username":"Kamren","email":"Lucio_Hettinger@annie.ca","address":{"street":"Skiles Walks","suite":"Suite 351","city":"Roscoeview","zipcode":"33263","geo":{"lat":"-31.8129","lng":"62.5342"}},"phone":"(254)954-1289","website":"demarco.info","company":{"name":"Keebler LLC","catchPhrase":"User-centric fault-tolerant solution","bs":"revolutionize end-to-end systems"}} {"id":6,"name":"Mrs. Dennis Schulist","username":"Leopoldo_Corkery","email":"Karley_Dach@jasper.info","address":{"street":"Norberto Crossing","suite":"Apt. 950","city":"South Christy","zipcode":"23505-1337","geo":{"lat":"-71.4197","lng":"71.7478"}},"phone":"1-477-935-8478 x6430","website":"ola.org","company":{"name":"Considine-Lockman","catchPhrase":"Synchronised bottom-line interface","bs":"e-enable innovative applications"}} {"id":7,"name":"Kurtis Weissnat","username":"Elwyn.Skiles","email":"Telly.Hoeger@billy.biz","address":{"street":"Rex Trail","suite":"Suite 280","city":"Howemouth","zipcode":"58804-1099","geo":{"lat":"24.8918","lng":"21.8984"}},"phone":"210.067.6132","website":"elvis.io","company":{"name":"Johns Group","catchPhrase":"Configurable multimedia task-force","bs":"generate enterprise e-tailers"}} {"id":8,"name":"Nicholas Runolfsdottir V","username":"Maxime_Nienow","email":"Sherwood@rosamond.me","address":{"street":"Ellsworth Summit","suite":"Suite 729","city":"Aliyaview","zipcode":"45169","geo":{"lat":"-14.3990","lng":"-120.7677"}},"phone":"586.493.6943 x140","website":"jacynthe.com","company":{"name":"Abernathy Group","catchPhrase":"Implemented secondary concept","bs":"e-enable extensible e-tailers"}} {"id":9,"name":"Glenna Reichert","username":"Delphine","email":"Chaim_McDermott@dana.io","address":{"street":"Dayna Park","suite":"Suite 449","city":"Bartholomebury","zipcode":"76495-3109","geo":{"lat":"24.6463","lng":"-168.8889"}},"phone":"(775)976-6794 x41206","website":"conrad.com","company":{"name":"Yost and Sons","catchPhrase":"Switchable contextually-based project","bs":"aggregate real-time technologies"}} {"id":10,"name":"Clementina DuBuque","username":"Moriah.Stanton","email":"Rey.Padberg@karina.biz","address":{"street":"Kattie Turnpike","suite":"Suite 198","city":"Lebsackbury","zipcode":"31428-2261","geo":{"lat":"-38.2386","lng":"57.2232"}},"phone":"024-648-3804","website":"ambrose.net","company":{"name":"Hoeger LLC","catchPhrase":"Centralized empowering task-force","bs":"target end-to-end models"}} ``` *data/users.json* Each document has a nested `address` and `company` that we're going to flatten so that we can import all the fields into Pinot. ## Pinot Schema Working out all the column names and their types is tedious with complex JSON documents, but luckily Pinot provides a tool that [infers the schema](/recipes/infer-schema-json-data) for us. We'll call it by running the following command: ```bash theme={null} docker exec -it pinot-controller-json \ bin/pinot-admin.sh JsonToPinotSchema \ -dimensions "" \ -jsonFile /data/users.json \ -pinotSchemaName users \ -outputDir /data/schema \ -delimeter '_' ``` The generated schema file, which gets written to `/data/schema/users.json`, looks like this: ```json theme={null} { "schemaName" : "users", "dimensionFieldSpecs" : [ { "name" : "id", "dataType" : "INT" }, { "name" : "name", "dataType" : "STRING" }, { "name" : "username", "dataType" : "STRING" }, { "name" : "email", "dataType" : "STRING" }, { "name" : "address_street", "dataType" : "STRING" }, { "name" : "address_suite", "dataType" : "STRING" }, { "name" : "address_city", "dataType" : "STRING" }, { "name" : "address_zipcode", "dataType" : "STRING" }, { "name" : "address_geo_lat", "dataType" : "STRING" }, { "name" : "address_geo_lng", "dataType" : "STRING" }, { "name" : "phone", "dataType" : "STRING" }, { "name" : "website", "dataType" : "STRING" }, { "name" : "company_name", "dataType" : "STRING" }, { "name" : "company_catchPhrase", "dataType" : "STRING" }, { "name" : "company_bs", "dataType" : "STRING" } ] } ``` *config/schema.json* We can see that those nested properties have been converted into individual columns e.g. `address_street`, `address_geo_lat`, `company_name`. > [!NOTE] > > The `.` character is used as the default delimiter, but we have overriden that by passing in the `-delimiter` parameter. > We'll need to make sure that we pass in this delimiter to our table config later on in this guide. We'll need to create two copies of the schema file as Pinot requires each table to have its own schema. We also need to update the `schemaName` to be `users_flatten` in one and `users_no_flatten` in the other. ## Pinot Table Now let's create Pinot tables that uses the above schemas. We're going to create two different tables so that we can see what happens if we don't specify the config that flattens JSON documents. First, the table config that doesn't do any flattening: ```json theme={null} { "tableName": "users_no_flatten", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1, "schemaName": "users_no_flatten" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "metadata": {} } ``` *config/table-no-flatten.json* And now one that has the flattening config: ```json {13-15} theme={null} { "tableName": "users_flatten", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1, "schemaName": "users_flatten" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" }, "complexTypeConfig": { "delimeter": "_" } }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "metadata": {} } ``` *config/table-flatten.json* The highlighted config flattens the nested JSON fields. We need to specify `delimiter` if our schema doesn't use the `.` character to separate nested field names. You can create the tables by running the following commands: ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema-no-flatten.json \ -tableConfigFile /config/table-no-flatten.json \ -controllerHost "pinot-controller-json" \ -exec ``` ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema-flatten.json \ -tableConfigFile /config/table-flatten.json \ -controllerHost "pinot-controller-json" \ -exec ``` ## Ingestion Job Now we’re going to import the JSON file into Pinot. We'll do this with the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:users.json' outputDirURI: '/opt/pinot/data/${tableName}/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: ${tableName} pinotClusterSpecs: - controllerURI: 'http://pinot-controller-json:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/job-spec.yml* The import job will map fields in each JSON document to a corresponding column in the `users` schema. You can run the following command to run the import: ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml \ -values tableName='users_no_flatten' ``` ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml \ -values tableName='users_flatten' ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and we'll run some queries see how the JSON flattening works. First, let's query `users_no_flatten`: ```sql theme={null} select id, name, username, company_name, company_bs, address_street, address_geo_lat, address_geo_lng from users_no_flatten limit 5 ``` You will see the following output: | id | name | username | company\_name | company\_bs | address\_street | address\_geo\_lat | address\_geo\_lng | | - | - | - | - | - | - | - | - | | 1 | Leanne Graham | Bret | null | null | null | null | null | | 2 | Ervin Howell | Antonette | null | null | null | null | null | | 3 | Clementine Bauch | Samantha | null | null | null | null | null | | 4 | Patricia Lebsack | Karianne | null | null | null | null | null | | 5 | Chelsey Dietrich | Kamren | null | null | null | null | null | *Query Results* We can see that the nested fields all have null values. Now let's try the same query on the `users_flatten` table: ```sql theme={null} select id, name, username, company_name, company_bs, address_street, address_geo_lat, address_geo_lng from users_flatten limit 5 ``` You will see the following output: | id | name | username | company\_name | company\_bs | address\_street | address\_geo\_lat | address\_geo\_lng | | - | - | - | - | - | - | - | - | | 1 | Leanne Graham | Bret | Romaguera-Crona | harness real-time e-markets | Kulas Light | -37.3159 | 81.1496 | | 2 | Ervin Howell | Antonette | Deckow-Crist | synergize scalable supply-chains | Victor Plains | -43.9509 | -34.4618 | | 3 | Clementine Bauch | Samantha | Romaguera-Jacobson | e-enable strategic applications | Douglas Extension | -68.6102 | -47.0653 | | 4 | Patricia Lebsack | Karianne | Robel-Corkery | transition cutting-edge web services | Hoeger Mall | 29.4572 | -164.2990 | | 5 | Chelsey Dietrich | Kamren | Keebler LLC | revolutionize end-to-end systems | Skiles Walks | -31.8129 | 62.5342 | *Query Results* On this table the flattened fields are all hydrated with values from the source data. # How to index a JSON column Source: https://docs.startree.ai/recipes/json-index To learn how to configure an index for a JSON column, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/MVFCLivbuLA) | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/json-index](https://github.com/startreedata/pinot-recipes/tree/main/recipes/json-index) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/json-index ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/json-index/docker-compose.yml) file on GitHub. ## Data generator This recipe contains a data generator that creates events with data about people. It uses the [Faker library](https://faker.readthedocs.io/en/master/), so you'll first need to install that: ```bash theme={null} pip install faker ``` You can generate data by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "ts": 1680172162791, "person": { "id": "ead572cb-c8bc-457a-b331-4f380c5ebe18", "name": "Melissa Howard", "email": "daniel75@example.com", "age": 33, "address": { "street_address": "974 Michael Hollow", "city": "Johnstonport", "state": "South Dakota", "country": "Latvia" }, "phone_number": "+1-697-077-6893x6562", "job": { "company": "Ramirez-Hunt", "position": "Communications engineer", "department": "iterate efficient e-markets" }, "interests": [ "Swimming" ], "friend_ids": [ "88a696de-34d1-4300-ac3c-3084a6bdceeb" ] } } ``` ## Kafka ingestion We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t people -Kø ``` We can check that Kafka has some data by running the following command: ```bash theme={null} docker exec -it kafka-querysegment kafka-run-class.sh \ kafka.tools.GetOffsetShell \ --broker-list localhost:9092 \ --topic events ``` We'll see something like the following: ```bash theme={null} events:0:19960902 ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "people", "dimensionFieldSpecs": [ { "name": "person", "dataType": "JSON" } ], "metricFieldSpecs": [], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Our schema has only two columns - one for the timestamp and another one that stores the person. Now for the table config: ```json {12-21} theme={null} { "tableName": "people", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "people", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "jsonIndexConfigs": { "person": { "maxLevels": 2, "excludeArray": false, "disableCrossArrayUnnest": true, "includePaths": null, "excludePaths": null, "excludeFields": ["age"] } }, "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "people", "stream.kafka.broker.list": "kafka-jsonindex:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows":"100000", "realtime.segment.flush.threshold.time":"1h" } }, "ingestionConfig": { }, "tenants": {}, "metadata": {} } ``` This highlighted section contains the config for the JSON index. An explanation of each of the config parameters is shown below: | Config Key | Description | Type | Default | | - | - | - | - | | **maxLevels** | Max levels to flatten the json object (array is also counted as one level) | `int` | -1 (unlimited) | | **excludeArray** | Whether to exclude array when flattening the object | `boolean` | false (include array) | | **disableCrossArrayUnnest** | Whether to not unnest multiple arrays (unique combination of all elements) | `boolean` | false (calculate unique combination of all elements) | | **includePaths** | Only include the given paths, e.g. "*.a.c\[\*]*" (mutual exclusive with **excludePaths**). Paths under the included paths will be included, e.g. "*.a.b*" is configured to be included. | `Set` | null (include all paths) | | **excludePaths** | Exclude the given paths, e.g. "*.a.c\[\*]*" (mutual exclusive with **includePaths**). Paths under the excluded paths will also be excluded, e.g. "*.a.b*" is configured to be excluded. | `Set` | null (include all paths) | | **excludeFields** | Exclude the given fields, e.g. "*b*", "*c*", even if it is under the included paths. | `Set` | null (include all fields) | So we are including all fields except `age` in the JSON index. > [!NOTE] > > Fields included in the JSON index can be filtered using the `JSON_MATCH` function. > If you use this function with a field that isn't included in the index, it won't return any records. We'll create the table by running the following: ```bash theme={null} docker run \ --network jsonindex \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-jsonindex" \ -exec ``` ## Querying by JSON index Let's now try to query this table using the JSON index. JSON indexes support the following predicates: `=`, `<>`, `IN`, and `NOT IN` ### State = Kentucky ```sql theme={null} select ts, person from people WHERE JSON_MATCH(person, '"$.address.state" = ''Kentucky''') LIMIT 3 ``` | ts | person | | - | - | | 2023-03-30 09:40:59.788 | `{"address":{"street_address":"8173 Diaz Hollow Apt. 175","country":"British Indian Ocean Territory (Chagos Archipelago)","city":"East Carlos","state":"Kentucky"},"friend_ids":["de27ec51-2895-4044-98bc-5643853d7d0d","84080eb3-1c17-4b4a-ada7-a764ed44cef7"],"name":"Robert Alvarado","phone_number":"085.136.9039x6810","id":"874a4543-f528-49e2-85a8-d31a6cbae269","job":{"company":"Gray, Huerta and Moyer","position":"Chemical engineer","department":"transition collaborative networks"},"interests":["Swimming","Meditation","Reading","Cycling","Reading"],"email":"jamiebarker@example.net","age":52}` | | 2023-03-30 09:40:59.823 | `{"address":{"street_address":"277 Robert Crossroad Apt. 622","country":"Haiti","city":"Port Karenshire","state":"Kentucky"},"friend_ids":["2285322d-0049-4386-b41c-9f96a0ed5038"],"name":"Diana Thomas","phone_number":"171-966-4122","id":"b996770e-478c-412d-89bd-d3a085140016","job":{"company":"Diaz LLC","position":"Scientist, research (maths)","department":"deploy granular convergence"},"interests":["Sports","Photography","Art","Photography","Reading"],"email":"ncook@example.org","age":40}` | | 2023-03-30 09:40:59.843 | `{"address":{"street_address":"85628 William Curve","country":"Djibouti","city":"Gregoryborough","state":"Kentucky"},"friend_ids":["1a7ee9e4-6daa-4994-a077-3170bffab04f"],"name":"Monica Harrell","phone_number":"(120)253-7632","id":"c71df52f-49c4-4d61-b6d6-d94134c92e8b","job":{"company":"Larson-Mcdaniel","position":"Ranger/warden","department":"seize leading-edge web services"},"interests":["Traveling","Music","Art"],"email":"allen60@example.net","age":33}` | *Query Results* ### State `<\>` Kentucky ```sql theme={null} select count(*) from people WHERE JSON_MATCH(person, '"$.address.state" <> ''Kentucky''') ``` | count(\*) | | - | | 319584 | *Query Results* ### State IN Kentucky or Alabama ```sql theme={null} select json_extract_scalar(person, '$.address.state', 'STRING') AS state, count(*) from people WHERE JSON_MATCH(person, '"$.address.state" IN (''Kentucky'', ''Alabama'')') GROUP BY state ORDER BY count(*) DESC ``` | state | count(\*) | | - | - | | Kentucky | 6682 | | Alabama | 6414 | *Query Results* ### State NOT IN Kentucky or Alabama ```sql theme={null} select json_extract_scalar(person, '$.address.state', 'STRING') AS state, count(*) from people WHERE JSON_MATCH(person, '"$.address.state" NOT IN (''Kentucky'', ''Alabama'')') GROUP BY state ORDER BY count(*) DESC ``` | state | count(\*) | | - | - | | Montana | 6743 | | Kansas | 6718 | | Maryland | 6693 | | Ohio | 6677 | | Connecticut | 6662 | | Mississippi | 6639 | | Rhode Island | 6630 | | Arkansas | 6621 | *Query Results* ### Unindexed field What if we try to query by the `age` column, which hasn't been indexed? ```sql theme={null} select count(*) from people WHERE JSON_MATCH(person, '"$.age"=59') ``` | count(\*) | | - | | 0 | *Query Results* We get back no results! We can even run the following query to confirm that the table contains records with an age of 59: ```sql theme={null} select count(*) from people WHERE JSON_EXTRACT_SCALAR(person, '$.age', 'STRING') = 59 ``` | count(\*) | | - | | 5018 | *Query Results* # JSON Transformation Functions Source: https://docs.startree.ai/recipes/json-transformation-functions In this recipe we'll learn how to use JSON transformation functions to extract values from nested JSON documents during the data ingestion process. | Pinot Version | 0.9.3 | | - | - | | Code | [startreedata/pinot-recipes/json-transformation-functions](https://github.com/startreedata/pinot-recipes/tree/main/recipes/json-transformation-functions) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/json-transformation-functions ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/json-transformation-functions/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following JSON file: ```json theme={null} {"name":"Pete", "meta": {"age":24}, "subjectsAndGrades":[{"name":"Maths", "grade":"A"}, {"name":"English", "grade":"B"}]} {"name":"John", "meta": {"age":28}, "subjectsAndGrades":[{"name":"Maths", "grade":"A"}, {"name":"Computer Science", "grade":"C"}]} ``` *data/import.json* ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName":"people", "dimensionFieldSpecs":[ { "name":"name", "dataType":"STRING" }, { "name":"age", "dataType":"INT" }, { "dataType":"STRING", "name":"subjects", "singleValueField":false }, { "dataType":"STRING", "name":"grades", "singleValueField":false } ] } ``` *config/schema.json* The `subjects` and `grades` columns will both contains arrays of values, which we can configure by setting `"singleValueField":false`. We'll also have the following table config: ```json {9-22} theme={null} { "tableName":"people", "tableType":"OFFLINE", "segmentsConfig":{ "replication":1, "schemaName":"people" }, "ingestionConfig":{ "transformConfigs":[ { "columnName":"subjects", "transformFunction":"jsonPathArray(subjectsAndGrades, '$.[*].name')" }, { "columnName":"grades", "transformFunction":"jsonPathArray(subjectsAndGrades, '$.[*].grade')" }, { "columnName":"age", "transformFunction":"JSONPATHLONG(meta, '$.age')" } ], "batchIngestionConfig":{ "segmentIngestionType":"APPEND", "segmentIngestionFrequency":"DAILY" } }, "tenants":{ "broker":"DefaultTenant", "server":"DefaultTenant" }, "tableIndexConfig":{ "loadMode":"MMAP" }, "metadata":{} } ``` *config/table.json* In this config we define transform configs (`ingestionConfig.transformConfigs`) to extract the subject names and grades from the `subjectAndGrades` property, using the [jsonPathArray](https://docs.pinot.apache.org/configuration-reference/functions/jsonpatharray) function. We also define one to extract the `age` from the `meta` property using the [JSONPATHLONG](https://docs.pinot.apache.org/configuration-reference/functions/jsonpathlong) function. You can create the table and schema by running the following command: ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-json" \ -exec ``` You should see a message similar to the following if everything is working correctly: ```text theme={null} 2022/02/25 13:21:11.963 INFO [AddTableCommand] [main] Executing command: AddTable -tableConfigFile /config/table.json -schemaFile /config/schema.json -controllerProtocol http -controllerHost 172.30.0.3 -controllerPort 9000 -user null -password [hidden] -exec 2022/02/25 13:21:13.337 INFO [AddTableCommand] [main] {"status":"Table people_OFFLINE succesfully added"} ``` ## Ingestion Job Now we’re going to import the JSON file into Pinot. We'll do this with the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/import.json' outputDirURI: '/opt/pinot/data/movies/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: 'people' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/table.json* The import job will map fields in each JSON document to a corresponding column in the `people` schema. If one of the fields doesn't exist in the schema it will be skipped. In this case the `name` field will be automatically mapped to the `name` column. The `subjectAndGrades` field is processed by transformation functions and the values are imported into the `subjects` and `grades` columns. The `meta` field is processed by a transformation function to extract the `age` property, which is stored in the `age` column. You can run the following command to run the import: ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `people` table or copy/paste the following query: ```sql theme={null} select * from people limit 10 ``` You will see the following output: | age | grades | name | subjects | | - | - | - | - | | 24 | A,B | Pete | Maths,English | | 28 | A,C | John | Maths,Computer Science | *Query Results* # How to unnest arrays in JSON documents Source: https://docs.startree.ai/recipes/json-unnest In this recipe we'll learn how to unnest or explode values when ingesting JSON documents into Apache Pinot. When we unnest an array it means that we'll create one row in Pinot for each item in the array rather than one row for the whole JSON document. | Pinot Version | 0.10.0 | | - | - | | Code | [startreedata/pinot-recipes/json-unnest](https://github.com/startreedata/pinot-recipes/tree/main/recipes/json-unnest) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/json-unnest ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/json-unnest/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following JSON file: ```json showLineNumbers theme={null} {"Title":"The Matrix","Year":"1999","Rated":"R","Released":"31 Mar 1999","Runtime":"136 min","Genre":"Action, Sci-Fi","Director":"Lana Wachowski, Lilly Wachowski","Writer":"Lilly Wachowski, Lana Wachowski","Actors":"Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss","Plot":"When a beautiful stranger leads computer hacker Neo to a forbidding underworld, he discovers the shocking truth--the life he knows is the elaborate deception of an evil cyber-intelligence.","Language":"English","Country":"United States, Australia","Awards":"Won 4 Oscars. 42 wins & 51 nominations total","Ratings":[{"Source":"Internet Movie Database","Value":"8.7/10"},{"Source":"Rotten Tomatoes","Value":"88%"},{"Source":"Metacritic","Value":"73/100"}],"Metascore":"73","imdbRating":"8.7","imdbVotes":"1,851,767","imdbID":"tt0133093","Type":"movie","DVD":"15 May 2007","BoxOffice":"$172,076,928","Production":"N/A","Website":"N/A","Response":"True"} {"Title":"Avatar","Year":"2009","Rated":"PG-13","Released":"18 Dec 2009","Runtime":"162 min","Genre":"Action, Adventure, Fantasy","Director":"James Cameron","Writer":"James Cameron","Actors":"Sam Worthington, Zoe Saldana, Sigourney Weaver","Plot":"A paraplegic Marine dispatched to the moon Pandora on a unique mission becomes torn between following his orders and protecting the world he feels is his home.","Language":"English, Spanish","Country":"United States","Awards":"Won 3 Oscars. 89 wins & 131 nominations total","Ratings":[{"Source":"Internet Movie Database","Value":"7.8/10"},{"Source":"Rotten Tomatoes","Value":"82%"},{"Source":"Metacritic","Value":"83/100"}],"Metascore":"83","imdbRating":"7.8","imdbVotes":"1,189,326","imdbID":"tt0499549","Type":"movie","DVD":"22 Apr 2010","BoxOffice":"$760,507,625","Production":"N/A","Website":"N/A","Response":"True"} {"Title":"The Boss Baby","Year":"2017","Rated":"PG","Released":"31 Mar 2017","Runtime":"97 min","Genre":"Animation, Adventure, Comedy","Director":"Tom McGrath","Writer":"Michael McCullers, Marla Frazee","Actors":"Alec Baldwin, Steve Buscemi, Jimmy Kimmel","Plot":"A suit-wearing, briefcase-carrying baby pairs up with his 7-year old brother to stop the dastardly plot of the CEO of Puppy Co.","Language":"English, Spanish","Country":"United States","Awards":"Nominated for 1 Oscar. 4 wins & 21 nominations total","Ratings":[{"Source":"Internet Movie Database","Value":"6.3/10"},{"Source":"Rotten Tomatoes","Value":"53%"},{"Source":"Metacritic","Value":"50/100"}],"Metascore":"50","imdbRating":"6.3","imdbVotes":"122,684","imdbID":"tt3874544","Type":"movie","DVD":"25 Jul 2017","BoxOffice":"$175,003,033","Production":"N/A","Website":"N/A","Response":"True"} {"Title":"Tall Girl","Year":"2019","Rated":"TV-PG","Released":"13 Sep 2019","Runtime":"101 min","Genre":"Comedy, Drama, Family","Director":"Nzingha Stewart","Writer":"Sam Wolfson","Actors":"Ava Michelle, Griffin Gluck, Sabrina Carpenter","Plot":"Jodi, the tallest girl in her high school, has always felt uncomfortable in her own skin. But after years of slouching, being made fun of, and avoiding attention at all costs, Jodi finally decides to find the confidence to stand t...","Language":"English","Country":"United States","Awards":"2 wins & 1 nomination","Ratings":[{"Source":"Internet Movie Database","Value":"5.2/10"},{"Source":"Rotten Tomatoes","Value":"38%"}],"Metascore":"N/A","imdbRating":"5.2","imdbVotes":"22,990","imdbID":"tt9252508","Type":"movie","DVD":"13 Sep 2019","BoxOffice":"N/A","Production":"N/A","Website":"N/A","Response":"True"} {"Title":"Love Hard","Year":"2021","Rated":"TV-MA","Released":"05 Nov 2021","Runtime":"104 min","Genre":"Comedy, Romance","Director":"Hernan Jimenez","Writer":"Daniel Mackey, Rebecca Ewing","Actors":"Nina Dobrev, Jimmy O. Yang, Darren Barnet","Plot":"An LA girl, unlucky in love, falls for an East Coast guy on a dating app and decides to surprise him for the holidays, only to discover that she's been catfished. This lighthearted romantic comedy chronicles her attempt to reel in lo","Language":"English","Country":"United States","Awards":"1 win & 1 nomination","Ratings":[{"Source":"Internet Movie Database","Value":"6.3/10"},{"Source":"Rotten Tomatoes","Value":"57%"},{"Source":"Metacritic","Value":"42/100"}],"Metascore":"42","imdbRating":"6.3","imdbVotes":"48,458","imdbID":"tt10752004","Type":"movie","DVD":"05 Nov 2021","BoxOffice":"N/A","Production":"N/A","Website":"N/A","Response":"True"} ``` *data/movies.json* We're particularly interested in the `Ratings` property of each movie. We're going to unnest or explode the ratings so that we have one row per rating in our database after the data is ingested. ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json {12-19} theme={null} { "schemaName": "movie_ratings", "dimensionFieldSpecs": [ { "name": "Title", "dataType": "STRING" }, { "name": "Year", "dataType": "INT" }, { "name": "Ratings.Source", "dataType": "STRING" }, { "name": "Ratings.Value", "dataType": "STRING" }, { "name": "Rated", "dataType": "STRING" } ] } ``` *config/schema.json* When we unnest a JSON list in a document, column names will be prefixed with the parent property name, in this case `Ratings`. We'll also have the following table config: ```json {9-13} theme={null} { "tableName": "movie_ratings", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1, "schemaName": "movie_ratings" }, "ingestionConfig": { "complexTypeConfig": { "fieldsToUnnest": [ "Ratings" ] }, "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" }, }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "metadata": {} } ``` *config/table.json* The highlighted config specifies which field in the data source should be unnested. The `Ratings` field contains an array of JSON objects and that array will be exploded to create one row for each item in the array. > [!NOTE] > > If you want to rename the unnested fields, see [How to rename fields when unnesting arrays in JSON documents](/recipes/json-unnest-rename-fields) You can create the table and schema by running the following command:\` ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-json" \ -exec ``` You should see a message similar to the following if everything is working correctly: ```text theme={null} 2022/05/04 09:11:35.397 INFO [AddTableCommand] [main] Executing command: AddTable -tableConfigFile /config/table.json -schemaFile null -controllerProtocol http -controllerHost 192.168.96.3 -controllerPort 9000 -user null -password [hidden] -exec 2022/05/04 09:11:35.614 INFO [AddTableCommand] [main] {"status":"Table movie_ratings_OFFLINE succesfully added"} ``` ## Ingestion Job Now we’re going to import the JSON file into Pinot. We'll do this with the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/movies.json' outputDirURI: '/opt/pinot/data/movies/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: 'movie_ratings' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/job-spec.yml* The import job will map fields in each JSON document to a corresponding column in the `movie_ratings` schema. If one of the fields doesn't exist in the schema it will be skipped. You can run the following command to run the import: ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:0.12.0-arm64 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `movie_ratings` table or copy/paste the following query: ```sql theme={null} select * from movie_ratings limit 10 ``` You will see the following output: | Rated | Ratings.Source | Ratings.Value | Title | | - | - | - | - | | R | Internet Movie Database | 8.7/10 | The Matrix | | R | Rotten Tomatoes | 88% | The Matrix | | R | Metacritic | 73/100 | The Matrix | | PG-13 | Internet Movie Database | 7.8/10 | Avatar | | PG-13 | Rotten Tomatoes | 82% | Avatar | | PG-13 | Metacritic | 83/100 | Avatar | | PG | Internet Movie Database | 6.3/10 | The Boss Baby | | PG | Rotten Tomatoes | 53% | The Boss Baby | | PG | Metacritic | 50/100 | The Boss Baby | | TV-PG | Internet Movie Database | 5.2/10 | Tall Girl | | TV-PG | Rotten Tomatoes | 38% | Tall Girl | | TV-MA | Internet Movie Database | 6.3/10 | Love Hard | | TV-MA | Rotten Tomatoes | 57% | Love Hard | | TV-MA | Metacritic | 42/100 | Love Hard | *Query Results* We can see that each movie has multiple rows, one for each rating given to that movie. # How to rename fields when unnesting arrays or flattening objects in JSON documents Source: https://docs.startree.ai/recipes/json-unnest-rename-fields In this recipe we'll learn how to rename fields when [unnesting/exploding values](/recipes/json-unnest) or [flattening objects](/recipes/json-flatten) while ingesting JSON documents into Apache Pinot. | Pinot Version | 0.10.0 | | - | - | | Code | [startreedata/pinot-recipes/json-unnest-rename-fields](https://github.com/startreedata/pinot-recipes/tree/main/recipes/json-unnest-rename-fields) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/json-unnest-rename-fields ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/json-unnest-rename-fields/docker-compose.yml) file on GitHub. ## Dataset We're going to import the following JSON file: ```json theme={null} {"Title":"The Matrix","Year":"1999","Rated":"R","Meta": {"Released":"31 Mar 1999","Runtime":"136 min"},"Genre":"Action, Sci-Fi","Director":"Lana Wachowski, Lilly Wachowski","Writer":"Lilly Wachowski, Lana Wachowski","Actors":"Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss","Plot":"When a beautiful stranger leads computer hacker Neo to a forbidding underworld, he discovers the shocking truth--the life he knows is the elaborate deception of an evil cyber-intelligence.","Language":"English","Country":"United States, Australia","Awards":"Won 4 Oscars. 42 wins & 51 nominations total","Ratings":[{"Source":"Internet Movie Database","Value":"8.7/10"},{"Source":"Rotten Tomatoes","Value":"88%"},{"Source":"Metacritic","Value":"73/100"}],"Metascore":"73","imdbRating":"8.7","imdbVotes":"1,851,767","imdbID":"tt0133093","Type":"movie","DVD":"15 May 2007","BoxOffice":"$172,076,928","Production":"N/A","Website":"N/A","Response":"True"} {"Title":"Avatar","Year":"2009","Rated":"PG-13","Meta": {"Released":"18 Dec 2009","Runtime":"162 min"},"Genre":"Action, Adventure, Fantasy","Director":"James Cameron","Writer":"James Cameron","Actors":"Sam Worthington, Zoe Saldana, Sigourney Weaver","Plot":"A paraplegic Marine dispatched to the moon Pandora on a unique mission becomes torn between following his orders and protecting the world he feels is his home.","Language":"English, Spanish","Country":"United States","Awards":"Won 3 Oscars. 89 wins & 131 nominations total","Ratings":[{"Source":"Internet Movie Database","Value":"7.8/10"},{"Source":"Rotten Tomatoes","Value":"82%"},{"Source":"Metacritic","Value":"83/100"}],"Metascore":"83","imdbRating":"7.8","imdbVotes":"1,189,326","imdbID":"tt0499549","Type":"movie","DVD":"22 Apr 2010","BoxOffice":"$760,507,625","Production":"N/A","Website":"N/A","Response":"True"} {"Title":"The Boss Baby","Year":"2017","Rated":"PG","Meta": {"Released":"31 Mar 2017","Runtime":"97 min"},"Genre":"Animation, Adventure, Comedy","Director":"Tom McGrath","Writer":"Michael McCullers, Marla Frazee","Actors":"Alec Baldwin, Steve Buscemi, Jimmy Kimmel","Plot":"A suit-wearing, briefcase-carrying baby pairs up with his 7-year old brother to stop the dastardly plot of the CEO of Puppy Co.","Language":"English, Spanish","Country":"United States","Awards":"Nominated for 1 Oscar. 4 wins & 21 nominations total","Ratings":[{"Source":"Internet Movie Database","Value":"6.3/10"},{"Source":"Rotten Tomatoes","Value":"53%"},{"Source":"Metacritic","Value":"50/100"}],"Metascore":"50","imdbRating":"6.3","imdbVotes":"122,684","imdbID":"tt3874544","Type":"movie","DVD":"25 Jul 2017","BoxOffice":"$175,003,033","Production":"N/A","Website":"N/A","Response":"True"} {"Title":"Tall Girl","Year":"2019","Rated":"TV-PG","Meta": {"Released":"13 Sep 2019","Runtime":"101 min"},"Genre":"Comedy, Drama, Family","Director":"Nzingha Stewart","Writer":"Sam Wolfson","Actors":"Ava Michelle, Griffin Gluck, Sabrina Carpenter","Plot":"Jodi, the tallest girl in her high school, has always felt uncomfortable in her own skin. But after years of slouching, being made fun of, and avoiding attention at all costs, Jodi finally decides to find the confidence to stand t...","Language":"English","Country":"United States","Awards":"2 wins & 1 nomination","Ratings":[{"Source":"Internet Movie Database","Value":"5.2/10"},{"Source":"Rotten Tomatoes","Value":"38%"}],"Metascore":"N/A","imdbRating":"5.2","imdbVotes":"22,990","imdbID":"tt9252508","Type":"movie","DVD":"13 Sep 2019","BoxOffice":"N/A","Production":"N/A","Website":"N/A","Response":"True"} {"Title":"Love Hard","Year":"2021","Rated":"TV-MA","Meta": {"Released":"05 Nov 2021","Runtime":"104 min"},"Genre":"Comedy, Romance","Director":"Hernan Jimenez","Writer":"Daniel Mackey, Rebecca Ewing","Actors":"Nina Dobrev, Jimmy O. Yang, Darren Barnet","Plot":"An LA girl, unlucky in love, falls for an East Coast guy on a dating app and decides to surprise him for the holidays, only to discover that she's been catfished. This lighthearted romantic comedy chronicles her attempt to reel in lo","Language":"English","Country":"United States","Awards":"1 win & 1 nomination","Ratings":[{"Source":"Internet Movie Database","Value":"6.3/10"},{"Source":"Rotten Tomatoes","Value":"57%"},{"Source":"Metacritic","Value":"42/100"}],"Metascore":"42","imdbRating":"6.3","imdbVotes":"48,458","imdbID":"tt10752004","Type":"movie","DVD":"05 Nov 2021","BoxOffice":"N/A","Production":"N/A","Website":"N/A","Response":"True"} ``` *data/movies.json* We're particularly interested in the `Ratings` property of each movie. We're going to unnest or explode the ratings so that we have one row per rating in our database after the data is ingested. We'll also pull out the `Meta.Released` property. ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "movie_ratings", "dimensionFieldSpecs": [ { "name": "Title", "dataType": "STRING" }, { "name": "Released", "dataType": "STRING" }, { "name": "Rated", "dataType": "STRING" }, { "name": "Source", "dataType": "STRING" }, { "name": "Rating", "dataType": "STRING" } ] } ``` *config/schema.json* You can create the schema by running the following command:\` ```bash theme={null} docker exec -it pinot-controller-json bin/pinot-admin.sh AddSchema \ -schemaFile /config/schema.json \ -exec ``` We'll also have the following table config: ```json {9-25} theme={null} { "tableName": "movie_ratings", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1, "schemaName": "movie_ratings" }, "ingestionConfig": { "complexTypeConfig": { "fieldsToUnnest": ["Ratings"] }, "transformConfigs": [ { "columnName": "Released", "transformFunction": "Meta.Released" }, { "columnName": "Source", "transformFunction": "Ratings.Source" }, { "columnName": "Rating", "transformFunction": "Ratings.Value" } ], "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "metadata": {} } ``` *config/table.json* The existence of the `complexTypeConfig` section means that nested JSON objects will be flattened into individual fields. It also which field in the data source should be unnested. The `Ratings` field contains an array of JSON objects and that array will be exploded to create one row for each item in the array. The `transformConfigs` section maps the following values: * `Ratings.Source` to the `Source` column * `Ratings.Value` to the `Rating` column * `Meta.Released` to the `Released` column. You can create the table by running the following command:\` ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-json" \ -exec ``` ## Ingestion Job Now we’re going to import the JSON file into Pinot. We'll do this with the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/movies.json' outputDirURI: '/opt/pinot/data/movies/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: 'movie_ratings' pinotClusterSpecs: - controllerURI: 'http://pinot-controller-json:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/job-spec.yml* The import job will map fields in each JSON document to a corresponding column in the `movie_ratings` schema. If one of the fields doesn't exist in the schema it will be skipped. You can run the following command to run the import: ```bash theme={null} docker run \ --network json \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `movie_ratings` table or copy/paste the following query: ```sql theme={null} select * from movie_ratings limit 10 ``` You will see the following output: | Rated | Rating | Released | Source | Title | | - | - | - | - | - | | R | 8.7/10 | 31 Mar 1999 | Internet Movie Database | The Matrix | | R | 88% | 31 Mar 1999 | Rotten Tomatoes | The Matrix | | R | 73/100 | 31 Mar 1999 | Metacritic | The Matrix | | PG-13 | 7.8/10 | 18 Dec 2009 | Internet Movie Database | Avatar | | PG-13 | 82% | 18 Dec 2009 | Rotten Tomatoes | Avatar | | PG-13 | 83/100 | 18 Dec 2009 | Metacritic | Avatar | | PG | 6.3/10 | 31 Mar 2017 | Internet Movie Database | The Boss Baby | | PG | 53% | 31 Mar 2017 | Rotten Tomatoes | The Boss Baby | | PG | 50/100 | 31 Mar 2017 | Metacritic | The Boss Baby | | TV-PG | 5.2/10 | 13 Sep 2019 | Internet Movie Database | Tall Girl | *Query Results* We can see that each movie has multiple rows, one for each rating given to that movie. # How to update a JSON index Source: https://docs.startree.ai/recipes/json-update-index In this recipe we'll learn how to update the configuration of a [JSON index](/recipes/json-index). | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/update-json-index](https://github.com/startreedata/pinot-recipes/tree/main/recipes/update-json-index) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/update-json-index ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/query-by-segment/docker-compose.yml) file on GitHub. ## Data generator This recipe contains a data generator that creates events with data about people. It uses the [Faker library](https://faker.readthedocs.io/en/master/), so you'll first need to install that: ```bash theme={null} pip install faker ``` You can generate data by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "ts": 1680172162791, "person": { "id": "ead572cb-c8bc-457a-b331-4f380c5ebe18", "name": "Melissa Howard", "email": "daniel75@example.com", "age": 33, "address": { "street_address": "974 Michael Hollow", "city": "Johnstonport", "state": "South Dakota", "country": "Latvia" }, "phone_number": "+1-697-077-6893x6562", "job": { "company": "Ramirez-Hunt", "position": "Communications engineer", "department": "iterate efficient e-markets" }, "interests": [ "Swimming" ], "friend_ids": [ "88a696de-34d1-4300-ac3c-3084a6bdceeb" ] } } ``` ## Kafka ingestion We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t people -Kø ``` We can check that Kafka has some data by running the following command: ```bash theme={null} docker exec -it kafka-querysegment kafka-run-class.sh \ kafka.tools.GetOffsetShell \ --broker-list localhost:9092 \ --topic events ``` We'll see something like the following: ```bash theme={null} events:0:19960902 ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "people", "dimensionFieldSpecs": [ { "name": "person", "dataType": "JSON" } ], "metricFieldSpecs": [], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` *schema.json* Our schema has only two columns - one for the timestamp and another one that stores the person. Now for the table config: ```json {12-21} theme={null} { "tableName": "people", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "people", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "jsonIndexConfigs": { "person": { "maxLevels": 2, "excludeArray": false, "disableCrossArrayUnnest": true, "includePaths": null, "excludePaths": null, "excludeFields": null } }, "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "people", "stream.kafka.broker.list": "kafka-jsonindex:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows":"100000", "realtime.segment.flush.threshold.time":"1h" } }, "ingestionConfig": { }, "tenants": {}, "metadata": {} } ``` *table.json* This highlighted section contains the config for the JSON index. An explanation of each of the config parameters is shown below: | Config Key | Description | Type | Default | | - | - | - | - | | **maxLevels** | Max levels to flatten the json object (array is also counted as one level) | `int` | -1 (unlimited) | | **excludeArray** | Whether to exclude array when flattening the object | `boolean` | false (include array) | | **disableCrossArrayUnnest** | Whether to not unnest multiple arrays (unique combination of all elements) | `boolean` | false (calculate unique combination of all elements) | | **includePaths** | Only include the given paths, e.g. "*.a.c\[\*]*" (mutual exclusive with **excludePaths**). Paths under the included paths will be included, e.g. "*.a.b*" is configured to be included. | `Set` | null (include all paths) | | **excludePaths** | Exclude the given paths, e.g. "*.a.c\[\*]*" (mutual exclusive with **includePaths**). Paths under the excluded paths will also be excluded, e.g. "*.a.b*" is configured to be excluded. | `Set` | null (include all paths) | | **excludeFields** | Exclude the given fields, e.g. "*b*", "*c*", even if it is under the included paths. | `Set` | null (include all fields) | So we are including all fields in the JSON index. > [!NOTE] > > Fields included in the JSON index can be filtered using the `JSON_MATCH` function. > If you use this function with a field that isn't included in the index, it won't return any records. We'll create the table by running the following: ```bash theme={null} docker run \ --network jsonindex \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-jsonindex" \ -exec ``` ## Querying by JSON index Let's now try to query this table using the JSON index. JSON indexes support the following predicates: `=`, `<>`, `IN`, and `NOT IN` ```sql theme={null} select json_extract_scalar(person, '$.address.state', 'STRING') AS state, count(*) from people WHERE JSON_MATCH(person, '"$.age" = 59') GROUP BY state ORDER BY count(*) DESC ``` | state | count(\*) | | - | - | | New Hampshire | 34 | | Pennsylvania | 32 | | Kansas | 32 | | New Jersey | 32 | | Maryland | 31 | | Vermont | 30 | | Rhode Island | 30 | | Virginia | 30 | | Ohio | 30 | | Washington | 29 | *Query Results* And one more: ```sql theme={null} select count(*) from people WHERE JSON_MATCH(person, '"$.interests[0]" = ''Swimming''') ``` | count(\*) | | - | | 3319 | *Query Results* ## Updating JSON index Now let's say we want to update the config of the JSON index so that not all fields are indexed. At the moment this must be done in the following steps: 1. Update the table config to remove the index completely. 2. Refresh all segments. 3. Update the table config with the new config. 4. Refresh all segments. Let's first remove the index by calling the `AddTable` command with the following table config: ```json theme={null} { "tableName": "people", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "people", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "people", "stream.kafka.broker.list": "kafka-jsonindex:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows":"100000", "realtime.segment.flush.threshold.time":"1h" } }, "ingestionConfig": { }, "tenants": {}, "metadata": {} } ``` *table-no-index.json* Next, let's run the command ```bash theme={null} docker run \ --network jsonindex \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table-no-index.json \ -controllerHost "pinot-controller-jsonindex" \ -exec -update ``` And let's refresh all segments: ```bash theme={null} curl -X 'POST' 'http://localhost:9000/segments/people_REALTIME/reload?type=REALTIME' ``` If we now re-run our last query, we'll get the following output: ```text theme={null} [ { "message": "QueryExecutionError:\njava.lang.IllegalStateException: Cannot apply JSON_MATCH on column: person without json index\n\tat org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518)\n\tat org.apache.pinot.core.plan.FilterPlanNode.constructPhysicalOperator(FilterPlanNode.java:285)\n\tat org.apache.pinot.core.plan.FilterPlanNode.run(FilterPlanNode.java:92)\n\tat org.apache.pinot.core.plan.AggregationPlanNode.buildNonFilteredAggOperator(AggregationPlanNode.java:178)", "errorCode": 200 } ] ``` Let's now add the updated index, based on the table config below: ```json {12-21} theme={null} { "tableName": "people", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "people", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "jsonIndexConfigs": { "person": { "maxLevels": 2, "excludeArray": true, "disableCrossArrayUnnest": true, "includePaths": null, "excludePaths": ["$.address.street_address"], "excludeFields": ["age"] } }, "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "people", "stream.kafka.broker.list": "kafka-jsonindex:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows":"100000", "realtime.segment.flush.threshold.time":"1h" } }, "ingestionConfig": { }, "tenants": {}, "metadata": {} } ``` *table-updated-index.json* This time we're not going to index `age`, any array fields (i.e `interests` or `friend_ids`), or `address.street_address`. Let's apply that config ```bash theme={null} docker run \ --network jsonindex \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table-updated-index.json \ -controllerHost "pinot-controller-jsonindex" \ -exec -update ``` Refresh all segments again: ```bash theme={null} curl -X 'POST' 'http://localhost:9000/segments/people_REALTIME/reload?type=REALTIME' ``` It's query time! Let's start with the swimming query: ```sql theme={null} select count(*) from people WHERE JSON_MATCH(person, '"$.interests[0]" = ''Swimming''') ``` | count(\*) | | - | | 0 | *Query Results* This field isn't indexed anymore, so we don't get any results. Same thing if we search by street address: ```sql theme={null} select count(*) from people WHERE JSON_MATCH(person, '"$.address.street_address" = ''862 Mercado Mountain Suite 529''') ``` | count(\*) | | - | | 0 | *Query Results* But we can still search by country: ```sql theme={null} select count(*) from people WHERE JSON_MATCH(person, '"$.address.country" = ''Croatia''') ``` | count(\*) | | - | | 308 | *Query Results* # Connecting to Kafka with SASL Authentication Source: https://docs.startree.ai/recipes/kafka-sasl In this guide we'll learn how to configure Apache Pinot to work with Kafka using SASL authentication. In this guide we'll learn how to ingest data into Apache Pinot from an [Apache Kafka cluster configured with SASL authentication](https://docs.confluent.io/platform/current/kafka/authentication_sasl/index.html). | Pinot Version | 0.10.0 | | - | - | | Code | [startreedata/pinot-recipes/kafka-sasl](https://github.com/startreedata/pinot-recipes/tree/main/recipes/kafka-sasl) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/kafka-sasl ``` ### Launch Pinot and Kafka Clusters You can spin up Pinot and Kafka clusters by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/kafka-sasl/docker-compose.yml) file on GitHub. The Kafka cluster is launched with the following JAAS (Java Authentication and Authorization Service) for SASL configuration ```text theme={null} KafkaServer { org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret" user_admin="admin-secret" user_alice="alice-secret"; }; Client{}; ``` We have two users: * `admin` with the password `admin-secret` * `alice` with the password `alice-secret` ### Pinot Schema and Table Let's create a Pinot Schema and Table. The schema is defined below: ```json theme={null} { "schemaName":"events", "dimensionFieldSpecs":[ { "name":"uuid", "dataType":"STRING" } ], "metricFieldSpecs":[ { "name":"count", "dataType":"INT" } ], "dateTimeFieldSpecs":[ { "name":"ts", "dataType":"TIMESTAMP", "format":"1:MILLISECONDS:EPOCH", "granularity":"1:MILLISECONDS" } ] } ``` *config/schema.json* And the table config below: ```json {16-18} theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-sasl:9093", "security.protocol": "SASL_PLAINTEXT", "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"admin\" password=\"admin-secret\";", "sasl.mechanism": "PLAIN", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder" } }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "tenants": {}, "metadata": {} } ``` *config/table.json* The part of this configuration that we're interested in is highlighted. The credentials that we want to use are specified in the `sasl.jaas.config` property. > [!NOTE] > > If our Kafka cluster has SSL enabled, we would need to specify `security_protocol` as `SASL_SSL` instead of `SASL_PLAINTEXT`. For an example of using SSL and SASL, see [Connecting to Kafka with SSL and SASL authentication](/recipes/kafka-ssl-sasl) Create the table and schema by running the following command: ```bash theme={null} docker exec -it pinot-controller-sasl bin/pinot-admin.sh AddTable \ -tableConfigFile /config/table.json \ -schemaFile /config/schema.json \ -exec ``` ### Ingesting Data Next, we're going to ingest some data into Kafka: ```bash theme={null} while true; do ts=`date +%s%N | cut -b1-13`; uuid=`cat /proc/sys/kernel/random/uuid | sed 's/[-]//g'` count=$[ $RANDOM % 1000 + 0 ] echo "{\"ts\": \"${ts}\", \"uuid\": \"${uuid}\", \"count\": $count}" done | docker exec -i kafka-sasl /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 \ --producer.config /etc/kafka/kafka_client.conf \ --topic events; ``` We need to pass in a configuration file with SASL credentials when producing events as well. The contents of the configuration file are shown below: ```conf theme={null} security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="alice" \ password="alice-secret"; ``` */etc/kafka/kafka\_client.conf* ## Querying Now let's navigate to [localhost:9000/#/query](http://localhost:9000/#/query?query=select+count%28*%29%2C+sum%28count%29+%0Afrom+events\&tracing=false\&pqlSyntax=false) and copy/paste the following query: ```sql theme={null} select count(*), sum(count) from events ``` You will see the following output: | count(\*) | sum(count) | | - | - | | 64209 | 31917036 | *Query Results* # Connecting to Kafka with SSL and SASL Authentication Source: https://docs.startree.ai/recipes/kafka-ssl-sasl In this guide we'll learn how to ingest data into Apache Pinot from an [Apache Kafka cluster configured with SSL and SASL authentication](https://docs.confluent.io/platform/current/kafka/authentication_sasl/index.html). | Pinot Version | 0.10.0 | | - | - | | Code | [startreedata/pinot-recipes/kafka-ssl-sasl](https://github.com/startreedata/pinot-recipes/tree/main/recipes/kafka-ssl-sasl) | ## Prerequisites To follow the code examples in this guide, do the following: * [Install Docker](https://docs.docker.com/get-docker/) locally. * Ensure you have a Kafka cluster with SSL enabled. [Confluent Cloud](https://confluent.cloud/) offers a hosted Kafka service with free credits to get started. * Download recipes ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/kafka-ssl-sasl ``` ### Launch Pinot Cluster You can spin up Pinot cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/kafka-ssl-sasl/docker-compose.yml) file on GitHub. ### Pinot Schema and Table Let's create a Pinot Schema and Table. The schema is defined below: ```json theme={null} { "schemaName":"events", "dimensionFieldSpecs":[ { "name":"uuid", "dataType":"STRING" } ], "metricFieldSpecs":[ { "name":"count", "dataType":"INT" } ], "dateTimeFieldSpecs":[ { "name":"ts", "dataType":"TIMESTAMP", "format":"1:MILLISECONDS:EPOCH", "granularity":"1:MILLISECONDS" } ] } ``` *config/schema.json* And the table configuration below: ```json {15-18} theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "", "security.protocol": "SASL_SSL", "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";", "sasl.mechanism": "PLAIN", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder" } }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "tenants": {}, "metadata": {} } ``` *config/table.json* You'll need to replace `` with the host and port of your Kafka cluster. The credentials that we want to use are specified in the `sasl.jaas.config` property. You'll need to replace `` and `` with your own credentials. > [!NOTE] > > If our Kafka cluster does not have SSL enabled, we would need to specify `security_protocol` as `SASL_PLAINTEXT` instead of `SASL_SSL`. For an example of using SASL without SSL, see [Connecting to Kafka with SASL authentication](/recipes/kafka-sasl) Create the table and schema by running the following command: ```bash theme={null} docker exec -it pinot-controller-ssl-sasl bin/pinot-admin.sh AddTable \ -tableConfigFile /config/table.json \ -schemaFile /config/schema.json \ -exec ``` ### Ingesting Data Ingest a few messages into your Kafka cluster: ```json theme={null} {"ts": "1649757242937", "uuid": "fc43b2fafbf64d9e8dff8d6be75d881d", "count": 308} {"ts": "1649757242941", "uuid": "11f2500386ec42be84debba1d5bfd2f7", "count": 515} {"ts": "1649757242945", "uuid": "f2dcf496957146eaa12605c5d8c005a0", "count": 142} ``` If you're using Confluent Cloud you can ingest these messages via the UI. ## Querying Now let's navigate to [localhost:9000/#/query](http://localhost:9000/#/query?query=select+count%28*%29%2C+sum%28count%29+%0Afrom+events\&tracing=false\&pqlSyntax=false) and copy/paste the following query: ```sql theme={null} select count(*), sum(count) from events ``` You will see the following output: | count(\*) | sum(count) | | - | - | | 3 | 965 | *Query Results* # Lookup-Based Join Support Source: https://docs.startree.ai/recipes/lookup-joins In this recipe we'll learn how to use Pinot's lookup-based join support to perform joins between fact and dimension tables. | Pinot Version | 0.9.0 | | - | - | | Code | [startreedata/pinot-recipes/lookup-joins](https://github.com/startreedata/pinot-recipes/tree/main/recipes/lookup-joins) | ## What are lookup joins? Pinot was designed with querying single tables in mind, but sometimes we'll have queries where we want to bring in data from other tables. Large-scale joins of Pinot tables can be done using the [Presto Pinot connector](https://www.startree.ai/blog/real-time-analytics-with-presto-apache-pinot-part-1). In this guide we'll focus on joins based on lookup tables that are small in size. For example, to enrich the final result with the customer name, you can join an orders table with a customers table, which can be used as a lookup table. Joining a fact table with a small lookup table is made possible by Pinot's [Lookup UDF function](https://docs.pinot.apache.org/users/user-guide-query/lookup-udf-join). The Lookup UDF supports decoration join functionality by getting additional dimensional data via the primary key from a dimension table. ## Use Case To understand how lookup joins work, let’s take a simple example. Assume we have a fact table of orders and a dimension table of customers with the following schemas. ![Customers and Orders table schemas](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/lookup-use-case.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=767b04e3f69ae8578b55b5216790930c) *Customers and Orders table schemas* The goal is to answer the following questions. 1. Find all the orders placed by Gold customers. 2. Find the total number of orders by customer’s tier. 3. Find the top five countries by sales. The orders table alone can’t answer the questions above. Hence, we need to do a lookup join with the customers table to enrich the result with additional dimensions such as tier and country. That requires us to define customers as a dimension table and join with orders table using the customer\_id field. ## Solution Architecture The solution is based on two Pinot tables. * Orders table – This table captures every fact about sales orders and qualifies as a fact table. We can implement this as a real-time table as new orders constantly arrive. Data ingestion takes place from a Kafka topic called the orders. * Customers table – This captures customer dimensions and has a primary key of customer\_id.\ We can model this as a dimension table of OFFLINE type.\ Since customer information is not frequently updated as orders, a daily batch job can be used to populate this table. ![Solution architecture](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/lookup-solution-architecture.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=743ff50a030b663f757069df0844b3bc) *Solution architecture* ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/lookup-joins ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Zookeeper, and Kafka. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/lookup-joins/docker-compose.yml) file on GitHub. ## Create orders Kafka topic Execute the following command from the project’s root directory to create a new topic called orders to represent the incoming stream of orders. ```bash theme={null} docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \ --create --bootstrap-server kafka:9092 --topic orders ``` ## Define orders table and ingest order records Let’s first define a schema and table for the orders table. You can find the schema definition in the [orders\_schema.json](https://github.com/startreedata/pinot-recipes/blob/main/recipes/lookup-joins/config/orders_schema.json) file and table definition in the [orders\_table.json](https://github.com/startreedata/pinot-recipes/blob/main/recipes/lookup-joins/config/orders_table.json) file. They are both located in the [/config](https://github.com/startreedata/pinot-recipes/tree/main/recipes/lookup-joins/config) directory. In terms of configurations, they look similar to any other real-time table, so we won't go through them here. Go ahead and execute the following command from the project’s root directory. ```bash theme={null} docker run \ --network lookup-join \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/orders_schema.json \ -tableConfigFile /config/orders_table.json \ -controllerHost "pinot-controller" \ -exec ``` The above command connects to the container that runs the Pinot controller and executes `pinot-admin.sh` to create the schema and a table for orders. ## Define the customers table and ingest customer records Let’s create the customers dimension table next. You can also find the schema and table definitions inside the [/config](https://github.com/startreedata/pinot-recipes/tree/main/recipes/lookup-joins/config) directory. You’ll notice that the `customer_id` field has been marked as a primary key in the [customers\_schema.json](https://github.com/startreedata/pinot-recipes/blob/main/recipes/lookup-joins/config/customers_schema.json) file. ```json theme={null} { "schemaName": "customers", "primaryKeyColumns": [ "customer_id" ], "dimensionFieldSpecs": [ { "dataType": "INT", "name": "customer_id" }, { "dataType": "STRING", "name": "name" }, { "dataType": "STRING", "name": "tier" }, { "dataType": "STRING", "name": "country" }, { "dataType": "STRING", "name": "dob" } ] } ``` *customers\_schema.json* Also, you will notice the following unique configurations in the customers\_table.json file. ```json theme={null} { "tableName": "customers", "tableType": "OFFLINE", "isDimTable": true, "segmentsConfig": { "segmentPushType": "REFRESH", "replication": "1" }, "tenants": { }, "tableIndexConfig": { "loadMode": "MMAP" }, "metadata": { "customConfigs": { } }, "quota": { "storage": "200M" } } ``` *customers\_table.json* The *isDimTable* directive instructs Pinot to mark this table as a dimension table. Pinot will then replicate a copy of this table on all the servers. Execute the following command to create the schema and table definition for customers. ```bash theme={null} docker run \ --network lookup-join \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/customers_schema.json \ -tableConfigFile /config/customers_table.json \ -controllerHost "pinot-controller" \ -exec ``` At this point, you should be able to see two tables, orders, and customers have been created inside Pinot Query Console. ## Ingest sample records Now that we have the tables created inside Pinot. Before writing any join queries, let’s ingest some sample records into both tables. ### Produce sample orders to Kafka You can find two sample data files inside the [/data](https://github.com/startreedata/pinot-recipes/tree/main/recipes/lookup-joins/data) directory. The [orders.json](https://github.com/startreedata/pinot-recipes/blob/main/recipes/lookup-joins/data/orders.json) file contains thousand of JSON formatted orders. A sample would look like this: ```json theme={null} { "order_id":1, "customer_id":40, "order_status":"PROCESSING", "amount":7075.2, "ts":"1619116750" } ``` Execute the following command to publish those events into the orders topic we created earlier. ```` ```bash cat data/orders.json | kcat -P -b localhost:9092 -t orders ```` If you check the orders table after a few seconds, you will see it has been populated with data, as shown below: ![Orders table](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/orders-table.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=baa0757a651971ae5479da6b6539ddd9) *Orders table* ## Ingest the customers CSV file In the same /data directory, you can find the customers.csv file containing several customer records. Execute the following command to ingest them into the customers table. ```bash theme={null} docker run \ --network lookup-join \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/customers_job-spec.yml ``` You can find the ingestion spec file inside the /config directory. The above command launches an ingestion job inside the Pinot controller. The /data directory has been mounted as a volume there to read the customers.csv file and write the generated segments back to the same folder. Upon successful ingestion, you should see that the customers table will be populated with data. ## Write the lookup join queries Now that we have got both tables created and populated with data. What remains is to write SQL queries that answer the questions stated above. The signature of the lookup UDF function looks similar to this: lookUp('dimTableName', 'dimColToLookUp', 'dimJoinKey1', factJoinKeyVal1, 'dimJoinKey2', factJoinKeyVal2 ... ) Where: * *dimTableName* - Name of the dimension table to perform the lookup on. * *dimColToLookUp* - The column name of the dimension table to be retrieved to decorate our result. * *dimJoinKey* - The column name on which we want to perform the lookup, i.e., the join column name for the dimension table. * *factJoinKeyVal* - The value of the dimension table join column will retrieve the dimColToLookUp for the scope and invocation. The return type of the UDF will be that of the dimColToLookUp column type. There can also be multiple primary keys and corresponding values. Let’s walk through a couple of queries that answer the questions that we initially had. ### Find all the orders placed by Gold customers What are the orders placed by Gold customers? To answer this problem, we need to write a SQL query that performs a lookup join between orders and customers table, based on customer\_id. The joined result can be further filtered down on the ‘tier’ field. ```sql theme={null} SELECT orders.order_id, lookup('customers','name','customer_id',customer_id) as name, lookup('customers','tier','customer_id',customer_id) as tier, orders.amount FROM orders WHERE tier='Gold' LIMIT 10 ``` The query returns the following result: ![Orders placed by Gold customers](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/lookups-gold.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=dde6427eb1518202d4f68452ad6e4ba8) *Orders placed by Gold customers* ### Find the total number of orders by customer’s tier Which customer tier has made the most sales? The query looks similar to the above, but we need to aggregate the orders.amount by customers.tier. ```sql theme={null} SELECT lookup('customers','tier','customer_id',customer_id) as tier, SUM(orders.amount) as sales_total FROM orders GROUP BY tier ORDER By sales_total DESC ``` The above returns this: ![Sales by tier](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/lookups-tier.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=44606475ec4ee52208f136fa806212de) *Sales by tier* ### Find the top 5 countries by sales Who are the top five countries that have contributed to the most sales? The query will be the same as the above, but we will use the customers.country field to enrich the results further. ```sql theme={null} SELECT lookup('customers','country','customer_id',customer_id) as country, COUNT(*) as total_orders, SUM(orders.amount) as sales_total FROM orders GROUP BY country ORDER By sales_total DESC LIMIT 5 ``` ![Sales by country](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/lookups-country.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=c178c2ff9336e614c683e4f0eddcf094) *Sales by country* ## Things to consider When a table is marked as a dimension table, it will be replicated on all the hosts. Hence, it is necessary that the size of the dimension table has to be small. Pinot allows you to control the size of a dimension table to prevent oversized tables. The maximum size quota for a dimension table in a cluster is controlled by controller.dimTable.maxSize controller property. Table creation will fail if the storage quota exceeds this maximum size. ## Conclusion The lookup UDF function in Pinot enables you to join a table with a small lookup table to enrich the final result of a query. These lookup tables must be modeled as dimension tables of type OFFLINE and possess a primary key. Dimension tables are replicated across all servers of a Pinot cluster, taking advantage of data locality to perform fast and scalable lookups. You can find more information about Lookup joins in the following video. [YouTube video player](https://www.youtube.com/embed/HryANqHnTQk) # Managed Offline Flow Source: https://docs.startree.ai/recipes/managed-offline-flow Pinot is most commonly used to provide real-time analytics based on streaming data, which can be achieved using a real-time table. However, after running these systems for a while, we'll want to update the data ingested into this table. Perhaps the name of a value in a column has been updated, or we want to remove some duplicate records. Segments in real-time tables can't be replaced, but we can replace those in offline tables. Managed offline flow is the way that Pinot handles the process of moving the data from real-time to offline tables. In this recipe we'll learn how to use Pinot offline managed flow. | Pinot Version | 0.9.3 | | - | - | | Code | [startreedata/pinot-recipes/managed-offline-flow](https://github.com/startreedata/pinot-recipes/tree/main/recipes/managed-offline-flow) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. Clone this repository and navigate to this recipe: ```bash theme={null} git clone git@github.com:startreedata/pinot-recipes.git cd pinot-recipes/recipes/ingest-json-files ``` ## Makefile ```bash theme={null} make recipe ``` Running this recipe will build the foundation and start producing data into Kafka. Run the next Make task: ## Managed Offline Flow ```bash theme={null} make manage_offline_flow ``` The Make command above will perform these tasks: * Sets the necessary properties in the Pinot Controller to enable the managed offline flow task: `RealtimeToOfflineSegmentsTask`.`timeoutMs` and `.numConcurrentTasksPerInstance`. * Schedules the task to run. * Prints logs related to the task. * Updates the hybrid table's time boundary so that you can see records that have been move to offline. ## View realtime and offline segments Navigate to [http://localhost:9000/#/query](http://localhost:9000/#/query) and run the following query: ```sql theme={null} select $segmentName, count(*) cnt from events group by $segmentName order by cnt desc ``` Run the statement above to see records migrate from REALTIME to OFFLINE by running `make realtime` to generate more data and `make manage_offline_flow` to migrate older data to OFFLINE. See the [README on GitHub for this recipe](https://github.com/startreedata/pinot-recipes/blob/main/recipes/managed-offline-flow/README.md) for sample output. ## Clean up ```bash theme={null} make clean ``` ## Troubleshooting To clean up old Docker installations that may be interfering with your testing of this recipe, run the following command: ```bash theme={null} docker system prune ``` # Merge Real-Time Segments in Real-Time Tables Source: https://docs.startree.ai/recipes/merge-segments-realtime To improve disk storage and query performance, we recommend merging segments in real-time Pinot tables using the [Minion merge rollup task](https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task). To learn how to merge segments in real-time Pinot tables, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/6Ck1d5iswm4) If your use case supports aggregating data, you may also want to rollup segments in real-time tables. For more information, check out the following video. [YouTube video player](https://www.youtube.com/embed/jvP64G4kQAo) | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/merge-small-segments-realtime](https://github.com/startreedata/pinot-recipes/tree/main/recipes/merge-small-segments-realtime) | > [!NOTE] > > You can also merge segments in offline tables. For more information see the [merge small segments in offline tables guide](/recipes/merge-small-segments) ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/merge-small-segments-realtime ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/ingest-json-files/docker-compose.yml) file on GitHub. ## Controller configuration The Pinot controller is launched with the following custom configuration: ```properties {8-9} theme={null} controller.access.protocols.http.port=9000 controller.zk.str=zookeeper-mergesegments:2181 controller.helix.cluster.name=PinotCluster controller.host=pinot-controller-mergesegments controller.port=9000 controller.data.dir=/data controller.task.scheduler.enabled=true controller.task.frequencyPeriod=5m ``` *controller-conf.conf* This configuration is needed to enabled Pinot's task scheduler, which we'll use to automatically merge segments. > [!NOTE] > > We've configured the task scheduler to run every 5 minutes, but you'd set that to an hour or more in a production system. ## Dataset We've got the following data generator that generates data in bursts depending on the minute of the hour: ```python theme={null} import datetime as dt import uuid import random import json import time while True: now = dt.datetime.now() ts = int(now.timestamp() * 1000) id = str(uuid.uuid4()) count = random.randint(0, 1000) print( json.dumps({"ts": ts, "uuid": id, "count": count}) ) sleep_time = 0.01 if now.minute % 2 == 0 else 0.00005 time.sleep(sleep_time) ``` *datagen.py* If the minute is divisible by 2, we generate around 20,000 messages per second and if it isn't we generate around 100 messages per second. An example of a message produced by this script is shown below: ```json theme={null} { "ts": 1680260357275, "uuid": "0d4c5cc7-cdc7-42d0-924d-27babc18ad94", "count": 126 } ``` We can write that data into Kafka by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t events -Kø ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [{"name": "uuid", "dataType": "STRING"}], "metricFieldSpecs": [{"name": "count", "dataType": "INT"}], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` *config/schema.json* We'll also have the following table config: ```json {20-21,25-28,36-40} theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-mergesegments:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows":"1000000", "realtime.segment.flush.threshold.time":"1m" } }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" }, "transformConfigs": [] }, "tenants": {}, "metadata": {}, "task": { "taskTypeConfigsMap": { "MergeRollupTask": { "5m_2m.mergeType": "concat", "5m_2m.bucketTimePeriod": "5m", "5m_2m.bufferTimePeriod": "2m" } } } } ``` *config/table.json* There are a few bits of config that we're interested in. First up is segment threshold, which is defined like so: ```json theme={null} "realtime.segment.flush.threshold.rows":"1000000", "realtime.segment.flush.threshold.time":"1m" ``` This means that a segment will be committed once it contains 1m records or every minute, whichever comes first. For more on configuring the segment threshold, see the [segment threshold guide](/recipes/configuring-segment-threshold). Next is the `batchIngestionConfig`: ```json theme={null} "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } ``` > [!WARNING] > > This shouldn't be strictly necessary because it's a config that's usually used for offline tables, but the merge/roll-up logic in the current version (0.12.1) relies on it being there. The reliance on this config existing has [already been fixed](https://github.com/apache/pinot/commit/dadd42cff16cb8f4b7ec510ad73455a198b60e37) in the main branch and therefore this config won't be required when 0.13.0 is released. We're also interested in the `MergeRollupTask`, which is extracted below: ```json theme={null} "task": { "taskTypeConfigsMap": { "MergeRollupTask": { "5m_2m.mergeType": "concat", "5m_2m.bucketTimePeriod": "5m", "5m_2m.bufferTimePeriod": "2m" } } } ``` This configuration will bucket records from the same 5 minute period and will only process records with a timestamp from more than 2 minutes ago. > [!NOTE] > > We are intentionally using very small values for the `bucketTimePeriod` and `bufferTimePeriod` for the purposes of this example. You'll want to use larger values for production systems. You can create the table and schema by running the following command: ```bash theme={null} docker run \ --network mergesegments \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-mergesegments" \ -exec ``` Remove the `-arm64` suffix if you're not using a Mac M1/M2. ## Viewing segments We can navigate to the [Pinot UI](http://localhost:9000) and run the following query to see the segments that have been created and the number of records that they contain: ```sql theme={null} select $segmentName, count(*), ToDateTime(min(ts), 'YYYY-MM-dd HH:mm:ss') AS minDate, ToDateTime(max(ts), 'YYYY-MM-dd HH:mm:ss') AS maxDate from events group by $segmentName order by max(ts) DESC limit 50 ``` We'll see something like the following output: | segmentName | count(\*) | minDate | maxDate | | - | - | - | - | | events\_\_0\_\_6\_\_20230331T1122Z | 15137 | 2023-03-31 11:21:58 | 2023-03-31 11:22:14 | | events\_\_0\_\_5\_\_20230331T1120Z | 778623 | 2023-03-31 11:20:54 | 2023-03-31 11:21:58 | | events\_\_0\_\_4\_\_20230331T1119Z | 110868 | 2023-03-31 11:19:51 | 2023-03-31 11:20:54 | | events\_\_0\_\_3\_\_20230331T1118Z | 691993 | 2023-03-31 11:18:46 | 2023-03-31 11:19:51 | | events\_\_0\_\_2\_\_20230331T1117Z | 182076 | 2023-03-31 11:17:46 | 2023-03-31 11:18:46 | | events\_\_0\_\_1\_\_20230331T1116Z | 602990 | 2023-03-31 11:16:40 | 2023-03-31 11:17:46 | | events\_\_0\_\_0\_\_20230331T1115Z | 118298 | 2023-03-31 11:15:51 | 2023-03-31 11:16:40 | *Query Results* ## Merge segments The job to merge segments runs every 5 minutes, so if we wait a little while it will eventually start. > [!NOTE] > > If you want to manually trigger the merge segments job, see the [merge segments section](/recipes/merge-small-segments) of the merge small segments guide. We can check the Pinot Controller logs to see that it's been triggered: ```bash theme={null} docker exec -it pinot-controller-mergesegments \ grep -ri --color "\[MergeRollupTaskGenerator" logs/pinot-all.log ``` **Output** ```text theme={null} 2023/03/31 11:19:15.651 INFO [MergeRollupTaskGenerator] [pool-10-thread-5] Start generating task configs for table: events_REALTIME for task: MergeRollupTask 2023/03/31 11:19:15.658 INFO [MergeRollupTaskGenerator] [pool-10-thread-5] Creating the gauge metric for tracking the merge/roll-up task delay for table: events_REALTIME and mergeLevel: 5m_2m.(watermarkMs=1680261300000, bufferTimeMs=120000, bucketTimeMs=300000, taskDelayInNumTimeBuckets=0) 2023/03/31 11:19:15.658 INFO [MergeRollupTaskGenerator] [pool-10-thread-5] Bucket with start: 1680261300000 and end: 1680261600000 (table : events_REALTIME, mergeLevel : 5m_2m) cannot be merged yet 2023/03/31 11:19:15.663 INFO [MergeRollupTaskGenerator] [pool-10-thread-5] Finished generating task configs for table: events_REALTIME for task: MergeRollupTask, numTasks: 0 2023/03/31 11:24:15.672 INFO [MergeRollupTaskGenerator] [pool-10-thread-3] Start generating task configs for table: events_REALTIME for task: MergeRollupTask 2023/03/31 11:24:15.682 INFO [MergeRollupTaskGenerator] [pool-10-thread-3] Update watermark for table: events_REALTIME, mergeLevel: 5m_2m from: 1680261300000 to: 1680261300000 2023/03/31 11:24:15.696 INFO [MergeRollupTaskGenerator] [pool-10-thread-3] Finished generating task configs for table: events_REALTIME for task: MergeRollupTask, numTasks: 1 ``` And we can check the Pinot Minion logs to see if the job has run: ```bash theme={null} docker exec -it pinot-minion-mergesegments \ grep -ri --color "INFO \[MergeRollup" logs/pinot-all.log ``` **Output** ```text theme={null} 2023/03/31 11:24:16.718 INFO [MergeRollupTaskExecutor] [TaskStateModelFactory-task_thread-0] Starting task: MergeRollupTask with configs: {maxNumRecordsPerSegment=null, partitionBucketTimePeriod=5m, segmentNamePrefix=merged_5m_2m_1680261855683_0_events, authToken=null, downloadURL=http://pinot-controller-mergesegments:9000/segments/events/events__0__0__20230331T1115Z,http://pinot-controller-mergesegments:9000/segments/events/events__0__1__20230331T1116Z,http://pinot-controller-mergesegments:9000/segments/events/events__0__2__20230331T1117Z,http://pinot-controller-mergesegments:9000/segments/events/events__0__3__20230331T1118Z,http://pinot-controller-mergesegments:9000/segments/events/events__0__4__20230331T1119Z, mergeType=concat, mergeLevel=5m_2m, 5m_2m.bufferTimePeriod=2m, 5m_2m.bucketTimePeriod=5m, push.mode=TAR, segmentName=events__0__0__20230331T1115Z,events__0__1__20230331T1116Z,events__0__2__20230331T1117Z,events__0__3__20230331T1118Z,events__0__4__20230331T1119Z, tableName=events_REALTIME, uploadURL=http://pinot-controller-mergesegments:9000/segments, enableReplaceSegments=true, push.controllerUri=http://pinot-controller-mergesegments:9000, roundBucketTimePeriod=null, 5m_2m.mergeType=concat, TASK_ID=Task_MergeRollupTask_cd467be0-472d-44a9-99d3-6e6fcdf69d54_1680261855713_0, overwriteOutput=false} 2023/03/31 11:24:22.497 INFO [MergeRollupTaskExecutor] [TaskStateModelFactory-task_thread-0] Finished task: MergeRollupTask with configs: {maxNumRecordsPerSegment=null, partitionBucketTimePeriod=5m, segmentNamePrefix=merged_5m_2m_1680261855683_0_events, authToken=null, downloadURL=http://pinot-controller-mergesegments:9000/segments/events/events__0__0__20230331T1115Z,http://pinot-controller-mergesegments:9000/segments/events/events__0__1__20230331T1116Z,http://pinot-controller-mergesegments:9000/segments/events/events__0__2__20230331T1117Z,http://pinot-controller-mergesegments:9000/segments/events/events__0__3__20230331T1118Z,http://pinot-controller-mergesegments:9000/segments/events/events__0__4__20230331T1119Z, mergeType=concat, mergeLevel=5m_2m, 5m_2m.bufferTimePeriod=2m, 5m_2m.bucketTimePeriod=5m, push.mode=TAR, segmentName=events__0__0__20230331T1115Z,events__0__1__20230331T1116Z,events__0__2__20230331T1117Z,events__0__3__20230331T1118Z,events__0__4__20230331T1119Z, tableName=events_REALTIME, uploadURL=http://pinot-controller-mergesegments:9000/segments, enableReplaceSegments=true, push.controllerUri=http://pinot-controller-mergesegments:9000, roundBucketTimePeriod=null, 5m_2m.mergeType=concat, TASK_ID=Task_MergeRollupTask_cd467be0-472d-44a9-99d3-6e6fcdf69d54_1680261855713_0, overwriteOutput=false}. Total time: 5779ms ``` Let's now run the segments query again: | segmentName | count(\*) | minDate | maxDate | | - | - | - | - | | events\_\_0\_\_8\_\_20230331T1124Z | 109890 | 2023-03-31 11:23:59 | 2023-03-31 11:25:08 | | events\_\_0\_\_7\_\_20230331T1123Z | 743572 | 2023-03-31 11:23:04 | 2023-03-31 11:23:59 | | events\_\_0\_\_6\_\_20230331T1122Z | 72745 | 2023-03-31 11:21:58 | 2023-03-31 11:23:04 | | events\_\_0\_\_5\_\_20230331T1120Z | 778623 | 2023-03-31 11:20:54 | 2023-03-31 11:21:58 | | merged\_5m\_2m\_1680261855683\_0 \_events\_1680261600000\_1680261654662\_1 | 4567 | 2023-03-31 11:20:00 | 2023-03-31 11:20:54 | | merged\_5m\_2m\_1680261855683\_0 \_events\_1680261351086\_1680261599999\_0 | 1701658 | 2023-03-31 11:15:51 | 2023-03-31 11:19:59 | We can see that the merge job has rolled up segments 0-4 into two larger segments, containing data from 5 minute windows. If we wait a few more minutes, the next time it runs it will roll up the rest of the data for the 11:20 to 11:25 window, as well as some of the next window. We can see the output of the next few runs below: | segmentName | count(\*) | minDate | maxDate | | - | - | - | - | | events\_\_0\_\_18\_\_20230331T1134Z | 124044 | 2023-03-31 11:34:34 | 2023-03-31 11:35:09 | | events\_\_0\_\_17\_\_20230331T1133Z | 343966 | 2023-03-31 11:33:33 | 2023-03-31 11:34:34 | | events\_\_0\_\_16\_\_20230331T1132Z | 443483 | 2023-03-31 11:32:26 | 2023-03-31 11:33:33 | | events\_\_0\_\_15\_\_20230331T1131Z | 439137 | 2023-03-31 11:31:26 | 2023-03-31 11:32:26 | | events\_\_0\_\_14\_\_20230331T1130Z | 355319 | 2023-03-31 11:30:19 | 2023-03-31 11:31:26 | | merged\_5m\_2m\_1680262455984\_0 \_events\_1680262200000\_1680262219170\_1 | 1591 | 2023-03-31 11:30:00 | 2023-03-31 11:30:19 | | merged\_5m\_2m\_1680262455984\_0 \_events\_1680261900010\_1680262199999\_0 | 2368619 | 2023-03-31 11:25:00 | 2023-03-31 11:29:59 | | merged\_5m\_2m\_1680262155835\_0 \_events\_1680261600000\_1680261899994\_0 | 1604525 | 2023-03-31 11:20:00 | 2023-03-31 11:24:59 | | merged\_5m\_2m\_1680261855683\_0 \_events\_1680261351086\_1680261599999\_0 | 1701658 | 2023-03-31 11:15:51 | 2023-03-31 11:19:59 | # Merge Small Segments in Offline Tables Source: https://docs.startree.ai/recipes/merge-small-segments In this recipe we'll learn how to merge small segments into larger ones. By doing this, Pinot can benefit from disk storage and query performance. This is done using the [Minion merge rollup task](https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task). | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/merge-small-segments](https://github.com/startreedata/pinot-recipes/tree/main/recipes/merge-small-segments) | > [!WARNING] > > You can only merge segments in offline tables that have a time column. > [!NOTE] > > You can also merge segments in real-time tables. For more information see the [merge segments in real-time tables guide](/recipes/merge-segments-realtime) ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/merge-small-segments ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/ingest-json-files/docker-compose.yml) file on GitHub. ## Dataset We're going to import a couple of CSVs that contain results from the Australian Open tennis tournament that was held in January 2022. The contents of the files are shown below: ```csv theme={null} round,winner,loser,score,matchTime "R128","Miomir Kecmanovic","Salvatore Caruso","6-4 6-2 6-1","2022-01-17 11:00:00" "R128","Tommy Paul","Mikhail Kukushkin","6-3 6-4 6-2","2022-01-17 11:10:00" "R128","Oscar Otte","Chun Hsin Tseng","6-4 6-3 6-2","2022-01-17 12:00:00" "R128","Lorenzo Sonego","Sam Querrey","7-5 6-3 6-3","2022-01-17 13:14:00" "R128","Gael Monfils","Federico Coria","6-1 6-1 6-3","2022-01-17 15:32:00" ``` *input/matches0.csv* ```csv theme={null} round,winner,loser,score,matchTime "R128","Sebastian Korda","Cameron Norrie","6-3 6-0 6-4","2022-01-17 11:10:00" "R128","Corentin Moutet","Lucas Pouille","3-6 6-3 6-4 6-3","2022-01-17 11:03:00" "R128","Tallon Griekspoor","Fabio Fognini","6-1 6-4 6-4","2022-01-17 13:08:00" "R128","Pablo Carreno Busta","Tomas Martin Etcheverry","6-1 6-2 7-6(2)","2022-01-17 18:14:00" "R128","Carlos Alcaraz","Alejandro Tabilo","6-2 6-2 6-3","2022-01-17 16:51:00" ``` *input/matches1.csv* ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "matches", "dimensionFieldSpecs": [{ "name": "round", "dataType": "STRING" }, { "name": "winner", "dataType": "STRING" }, { "name": "loser", "dataType": "STRING" }, { "name": "score", "dataType": "STRING" } ], "dateTimeFieldSpecs": [{ "name": "matchTime", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` *config/schema.json* We'll also have the following table config: ```json theme={null} { "tableName": "matches", "tableType": "OFFLINE", "segmentsConfig": { "timeColumnName": "matchTime", "schemaName": "matches", "replication": "1", "replicasPerPartition": "1" }, "task": { "taskTypeConfigsMap": { "MergeRollupTask": { "1day.mergeType": "concat", "1day.bucketTimePeriod": "1d", "1day.bufferTimePeriod": "5m" } } }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" }, "transformConfigs": [] }, "metadata": {} } ``` *config/table.json* > [!WARNING] > > Our table must specify `segmentsConfig.timeColumnName`, otherwise the merge process won't merge any segments. The main thing that we're interested in is the `MergeRollupTask`, which is extracted below: ```json theme={null} "task": { "taskTypeConfigsMap": { "MergeRollupTask": { "1day.mergeType": "concat", "1day.bucketTimePeriod": "1d", "1day.bufferTimePeriod": "5m" } } } ``` This configuration will bucket records from the same 1 day period into the same segment. It will only process records with a timestamp from more than 5 minutes ago. You can create the table and schema by running the following command:\` ```bash theme={null} docker run \ --network mergesegments \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-mergesegments" \ -exec ``` Remove the `-arm64` suffix if you're not using a Mac M1/M2. ## Import Data Now let's import those CSV files into Pinot, using the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/input' includeFileNamePattern: 'glob:**/*.csv' outputDirURI: '/data' pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'matches' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/job-spec.yml* You can run the following command to run the import: ```bash theme={null} docker run \ --network mergesegments \ -v $PWD/config:/config \ -v $PWD/data:/data \ -v $PWD/input:/input \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml \ -values pinotController=http://pinot-controller-mergesegments:9000 ``` Once this job has run, we can list the created segments by running the following command: ```bash theme={null} curl -X GET "http://localhost:9000/segments/matches?type=OFFLINE" \ -H "accept: application/json" 2>/dev/null | jq '.' ``` **Output** ```json theme={null} [ { "OFFLINE": [ "matches_OFFLINE_1642417200000_1642433520000_0", "matches_OFFLINE_1642417380000_1642443240000_1" ] } ] ``` Let's wrap this command in a function so that we can use it again later: ```bash theme={null} show_segments () { table=${1:="matches"} table_type=${2:="OFFLINE"} curl -X GET "http://localhost:9000/segments/${table}?type=${table_type}" \ -H "accept: application/json" 2>/dev/null | jq '.' } ``` We could then call the function like this: ```bash theme={null} show_segments ``` We can check the contents of these segments by writing the following function: ```bash theme={null} segments_breakdown () { table=${1:="matches"} table_type=${2:="OFFLINE"} segments=$(curl -X GET "http://localhost:9000/segments/${table}?type=${table_type}" 2>/dev/null | jq -r '.[] []') for segment in $(echo $segments | jq '.[]'); do metadata=`curl -X GET "http://localhost:9000/segments/${table}/$(echo ${segment} | jq -r)/metadata" \ -H "accept: application/json" 2>/dev/null | jq '.'` docs=`echo $metadata | jq '."segment.total.docs" | tonumber'` startTime=`echo $metadata | jq '."segment.start.time" | tonumber'` endTime=`echo $metadata | jq '."segment.end.time" | tonumber'` echo "$segment,$docs,$startTime,$endTime" done } ``` We can call it like this: ```bash theme={null} segments_breakdown ``` **Output** ```text theme={null} matches_OFFLINE_1642417200000_1642433520000_0,5,1642417200000,1642433520000 matches_OFFLINE_1642417380000_1642443240000_1,5,1642417380000,1642443240000 ``` ## Merge segments Now we're going to merge these segments using the [Minion merge rollup task](https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task). The configuration that we defined in the `matches` table is going to bucket records from a 1 day period into the same bucket. Since our events all happened on the same day, we would expect that all records will be merge into a single segment. We can run the merge rollup task by running the following: ```bash theme={null} tableName="matches_OFFLINE" curl -X POST "http://localhost:9000/tasks/schedule?taskType=MergeRollupTask&tableName=${tableName}" \ -H "accept: application/json" 2>/dev/null | jq '.' ``` **Output** ```json theme={null} { "MergeRollupTask": "Task_MergeRollupTask_1646909665695" } ``` We can then check the Pinot Controller logs to see that it's been triggered: ```bash theme={null} docker exec -it pinot-controller-csv grep -ri --color "\[MergeRollup" logs/pinot-all.log ``` **Output** ```text theme={null} 2022/03/10 10:51:37.383 INFO [TaskGeneratorRegistry] [main] Initialized TaskGeneratorRegistry with 4 task generators: [MergeRollupTask, RealtimeToOfflineSegmentsTask, ConvertToRawIndexTask, SegmentGenerationAndPushTask] in 716ms 2022/03/10 10:54:25.628 INFO [MergeRollupTaskGenerator] [grizzly-http-server-25] Start generating task configs for table: matches_OFFLINE for task: MergeRollupTask 2022/03/10 10:54:25.666 INFO [MergeRollupTaskGenerator] [grizzly-http-server-25] Creating the gauge metric for tracking the merge/roll-up task delay for table: matches_OFFLINE and mergeLevel: 1day.(watermarkMs=1642377600000, bufferTimeMs=86400000, bucketTimeMs=86400000, taskDelayInNumTimeBuckets=0) 2022/03/10 10:54:25.667 INFO [MergeRollupTaskGenerator] [grizzly-http-server-25] Update watermark for table: matches_OFFLINE, mergeLevel: 1day from: 1642377600000 to: 1642377600000 2022/03/10 10:54:25.695 INFO [MergeRollupTaskGenerator] [grizzly-http-server-25] Finished generating task configs for table: matches_OFFLINE for task: MergeRollupTask, numTasks: 1 ``` And we can check the Pinot Minion logs to see if the job has run: ```bash theme={null} docker exec -it pinot-minion-csv grep -ri --color "INFO \[MergeRollup" logs/pinot-all.log ``` **Output** ```text theme={null} 2022/03/10 10:54:37.768 INFO [MergeRollupTaskExecutor] [TaskStateModelFactory-task_thread-0] Starting task: MergeRollupTask with configs: {maxNumRecordsPerSegment=null, partitionBucketTimePeriod=1d, segmentNamePrefix=merged_1day_1646909665668_0_matches, authToken=null, downloadURL=http://192.168.96.3:9000/segments/matches/matches_OFFLINE_1642417200000_1642433520000_0,http://192.168.96.3:9000/segments/matches/matches_OFFLINE_1642417380000_1642443240000_1, mergeType=concat, mergeLevel=1day, segmentName=matches_OFFLINE_1642417200000_1642433520000_0,matches_OFFLINE_1642417380000_1642443240000_1, tableName=matches_OFFLINE, uploadURL=http://192.168.96.3:9000/segments, enableReplaceSegments=true, roundBucketTimePeriod=null} 2022/03/10 10:54:38.149 INFO [MergeRollupTaskExecutor] [TaskStateModelFactory-task_thread-0] Finished task: MergeRollupTask with configs: {maxNumRecordsPerSegment=null, partitionBucketTimePeriod=1d, segmentNamePrefix=merged_1day_1646909665668_0_matches, authToken=null, downloadURL=http://192.168.96.3:9000/segments/matches/matches_OFFLINE_1642417200000_1642433520000_0,http://192.168.96.3:9000/segments/matches/matches_OFFLINE_1642417380000_1642443240000_1, mergeType=concat, mergeLevel=1day, segmentName=matches_OFFLINE_1642417200000_1642433520000_0,matches_OFFLINE_1642417380000_1642443240000_1, tableName=matches_OFFLINE, uploadURL=http://192.168.96.3:9000/segments, enableReplaceSegments=true, roundBucketTimePeriod=null}. Total time: 381ms ``` Let's now check the list of segments again: ```bash theme={null} show_segments ``` **Output** ```json theme={null} [ { "OFFLINE": [ "matches_OFFLINE_1642417200000_1642433520000_0", "matches_OFFLINE_1642417380000_1642443240000_1", "merged_1day_1646909665668_0_matches_1642417200000_1642443240000_0" ] } ] ``` We can see the new segment, but the initial segments are still there as well. The Pinot broker knows to use the new segment when it processes queries, so this isn't a problem. We can confirm this by running the following query on the [Pinot UI](http://localhost:9000): ```sql theme={null} select $segmentName, * from matches ``` | \$segmentName | loser | matchTime | round | score | winner | | - | - | - | - | - | - | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Salvatore Caruso | 2022-01-17 11:00:00.0 | R128 | 6-4 6-2 6-1 | Miomir Kecmanovic | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Mikhail Kukushkin | 2022-01-17 11:10:00.0 | R128 | 6-3 6-4 6-2 | Tommy Paul | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Chun Hsin Tseng | 2022-01-17 12:00:00.0 | R128 | 6-4 6-3 6-2 | Oscar Otte | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Sam Querrey | 2022-01-17 13:14:00.0 | R128 | 7-5 6-3 6-3 | Lorenzo Sonego | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Federico Coria | 2022-01-17 15:32:00.0 | R128 | 6-1 6-1 6-3 | Gael Monfils | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Cameron Norrie | 2022-01-17 11:10:00.0 | R128 | 6-3 6-0 6-4 | Sebastian Korda | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Lucas Pouille | 2022-01-17 11:03:00.0 | R128 | 3-6 6-3 6-4 6-3 | Corentin Moutet | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Fabio Fognini | 2022-01-17 13:08:00.0 | R128 | 6-1 6-4 6-4 | Tallon Griekspoor | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Tomas Martin Etcheverry | 2022-01-17 18:14:00.0 | R128 | 6-1 6-2 7-6(2) | Pablo Carreno Busta | | merged\_1day\_1680183805991\_0\_matches\_1642417200000\_1642443240000\_0 | Alejandro Tabilo | 2022-01-17 16:51:00.0 | R128 | 6-2 6-2 6-3 | Carlos Alcaraz | Also, Pinot's retention manager will take care of removing the old segments the next time that it runs. We'll see the following messages in the Pinot Controller's logs when the retention manager has run: ```bash theme={null} docker exec -it pinot-controller-csv grep -ri --color "\[SegmentDeletionManager\|\[RetentionManager" logs/pinot-all.log ``` **Output** ```text theme={null} 2022/03/14 13:11:18.006 INFO [RetentionManager] [main] Starting RetentionManager with runFrequencyInSeconds: 21600, deletedSegmentsRetentionInDays: 7 2022/03/14 13:15:08.024 INFO [RetentionManager] [pool-7-thread-4] Start managing retention for table: matches_OFFLINE 2022/03/14 13:15:08.032 WARN [RetentionManager] [pool-7-thread-4] Invalid retention time: null null for table: matches_OFFLINE, skip 2022/03/14 13:15:08.034 INFO [RetentionManager] [pool-7-thread-4] Start cleaning up segment lineage for table: matches_OFFLINE 2022/03/14 13:15:08.043 INFO [RetentionManager] [pool-7-thread-4] Finished cleaning up segment lineage for table: matches_OFFLINE, deleted segments: matches_OFFLINE_1642417200000_1642433520000_0,matches_OFFLINE_1642417380000_1642443240000_1 in 9ms 2022/03/14 13:15:08.043 INFO [RetentionManager] [pool-7-thread-4] Segment lineage metadata clean-up is successfully processed for table: matches_OFFLINE 2022/03/14 13:15:08.043 INFO [RetentionManager] [pool-7-thread-4] Removing aged (more than 7 days) deleted segments for all tables 2022/03/14 13:15:10.053 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Moved segment matches_OFFLINE_1642417200000_1642433520000_0 from file:/data/matches/matches_OFFLINE_1642417200000_1642433520000_0 to file:/data/Deleted_Segments/matches/matches_OFFLINE_1642417200000_1642433520000_0 2022/03/14 13:15:10.053 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Moved segment matches_OFFLINE_1642417380000_1642443240000_1 from file:/data/matches/matches_OFFLINE_1642417380000_1642443240000_1 to file:/data/Deleted_Segments/matches/matches_OFFLINE_1642417380000_1642443240000_1 2022/03/14 13:15:10.053 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 2 segments from table matches_OFFLINE:[matches_OFFLINE_1642417200000_1642433520000_0, matches_OFFLINE_1642417380000_1642443240000_1] ``` We can then check the contents of that segment by running the following: ```bash theme={null} segments_breakdown ``` **Output** ```text theme={null} merged_1day_1646909665668_0_matches_1642417200000_1642443240000_0,10,1642417200000,1642443240000 ``` We now have a single segment that contains all the records from the two CSV files that we ingested. ## Automatic scheduling We can also automatically schedule the merge task by adding the following configuration to the Pinot Controller: ```properties theme={null} controller.task.scheduler.enabled=true controller.task.frequencyPeriod=3600 ``` *Controller configuration* # Using MinIO as Deep Store for an Offline Table Source: https://docs.startree.ai/recipes/minio-deep-store In this recipe we'll learn how to use [MinIO](https://docs.min.io/docs/aws-cli-with-minio) as a Pinot deep store. To understand how this recipe processes data, examine the commands executed in the Makefile. | Pinot Version | 1.1.0 | | - | - | | Code | [startreedata/pinot-recipes/minio](https://github.com/startreedata/pinot-recipes//tree/main/recipes/minio) | ## Prerequisites To follow the code examples in this guide, do the following: * [Install Docker](https://docs.docker.com/get-docker/). * Download recipes ## Makefile Clone this repository and navigate to this recipe: ```bash theme={null} make recipe ``` ## Validate Check that minio has the segment in the deep store. You can also log into the minio console and check. [http://localhost:9001/browser/deepstore](http://localhost:9001/browser/deepstore). ``` docker exec minio mc ls myminio/deepstore/transcript ``` ## Clean up ```bash theme={null} make clean ``` ## Troubleshooting To clean up old Docker installations that may be interfering with your testing of this recipe, run the following command: ```bash theme={null} docker system prune ``` # How to pause and resume ingestion of a stream Source: https://docs.startree.ai/recipes/pause-resume-consumption Pause and resume ingestion of a stream for the following reasons: * There’s a problem with the underlying stream, and we need to restart the server, reset offsets, or recreate a topic * We want to ingest data from different streams into the same table. * Discover a mistake in the Pinot ingestion configuration, and Pinot is throwing exceptions and you're not able to ingest any more data. To learn how to use the `pauseConsumption` and `resumeConsumption` APIs, which are used to pause and resume ingestion of a stream. watch the following video, or complete the tutorial below, starting with the Prerequites. [YouTube video player](https://www.youtube.com/embed/u9CwDpMZRog) For changes to the table configuration, you don't need to pause and resume; use the [`forceCommit API`](/api-reference/table/force-commit-the-current-consuming-segments). | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/pause-resume](https://github.com/startreedata/pinot-recipes/tree/main/recipes/pause-resume) | ## Prerequisites You will need to [install Docker](https://docs.docker.com/get-docker/) to follow the code examples in this guide. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/pause-resume ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/pause-resume/docker-compose.yml) file on GitHub. ## Data generator This recipe contains a data generator that creates events with a timestamp, count, and UUID. You can generate data by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "tsString": "2022-11-23T12:08:44.127481Z", "uuid": "e1c58795-a009-4e21-ae76-cdd66e090797", "count": 203 } ``` ## Kafka ingestion We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t events -Kø ``` We can check that Kafka has some data by running the following command: ```bash theme={null} docker exec -it kafka-querysegment kafka-run-class.sh \ kafka.tools.GetOffsetShell \ --broker-list localhost:9092 \ --topic events ``` We'll see something like the following: ```bash theme={null} events:0:19960902 ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [{"name": "uuid", "dataType": "STRING"}], "metricFieldSpecs": [{"name": "count", "dataType": "INT"}], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Now for the table config: ```json {14} theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "ingestionConfig":{ "transformConfigs": [ { "columnName": "ts", "transformFunction": "FromDateTime(tsString, 'YYYY-MM-dd''T''HH:mm:ss.SS''Z''')" } ] }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-pauseresume:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder" } }, "tenants": {}, "metadata": {} } ``` *config/table.json* This highlighted section highlights a transformation function that has a subtle error. The second parameter passed to the FromDateTime function describes the format of the DateTime string, which we defined as: ``` YYYY-MM-dd''T''HH:mm:ss.SS''Z'' ``` But tsString has values in the following format: ``` 2022-11-23T12:08:44.127550Z ``` i.e., we don’t have enough S values - there should be 5 rather than 2. We'll create the table by running the following: ```bash theme={null} docker run \ --network forcecommit \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-forcecommit" \ -exec ``` If we navigate to the Pinot UI, we'll notice that no records have been ingested. Instead our server logs are full of the following error: ```text theme={null} 2023/06/02 15:00:31.705 ERROR [LLRealtimeSegmentDataManager_events__0__0__20230602T1459Z] [events__0__0__20230602T1459Z] Caught exception while transforming the record: org.apache.pinot.spi.stream.StreamDataDecoderResult@19717934 java.lang.RuntimeException: Caught exception while evaluation transform function for column: ts at org.apache.pinot.segment.local.recordtransformer.ExpressionTransformer.transform(ExpressionTransformer.java:126) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.recordtransformer.CompositeTransformer.transform(CompositeTransformer.java:90) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.segment.creator.TransformPipeline.processPlainRow(TransformPipeline.java:97) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.segment.creator.TransformPipeline.processRow(TransformPipeline.java:92) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:559) [pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:434) [pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:629) [pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: java.lang.RuntimeException: Caught exception while executing function: fromDateTime(tsString,'YYYY-MM-dd'T'HH:mm:ss.SS'Z'') at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator$FunctionExecutionNode.execute(InbuiltFunctionEvaluator.java:231) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator.evaluate(InbuiltFunctionEvaluator.java:106) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.recordtransformer.ExpressionTransformer.transform(ExpressionTransformer.java:123) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] ... 7 more Caused by: java.lang.IllegalStateException: Caught exception while invoking method: public static long org.apache.pinot.common.function.scalar.DateTimeFunctions.fromDateTime(java.lang.String,java.lang.String) with arguments: [2023-06-02T15:59:58.186685Z, YYYY-MM-dd'T'HH:mm:ss.SS'Z'] at org.apache.pinot.common.function.FunctionInvoker.invoke(FunctionInvoker.java:130) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator$FunctionExecutionNode.execute(InbuiltFunctionEvaluator.java:229) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator.evaluate(InbuiltFunctionEvaluator.java:106) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.recordtransformer.ExpressionTransformer.transform(ExpressionTransformer.java:123) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] ... 7 more Caused by: java.lang.reflect.InvocationTargetException at jdk.internal.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.apache.pinot.common.function.FunctionInvoker.invoke(FunctionInvoker.java:127) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator$FunctionExecutionNode.execute(InbuiltFunctionEvaluator.java:229) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator.evaluate(InbuiltFunctionEvaluator.java:106) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.recordtransformer.ExpressionTransformer.transform(ExpressionTransformer.java:123) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] ... 7 more Caused by: java.lang.IllegalArgumentException: Invalid format: "2023-06-02T15:59:58.186685Z" is malformed at "6685Z" at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.common.function.DateTimePatternHandler.parseDateTimeStringToEpochMillis(DateTimePatternHandler.java:38) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.common.function.scalar.DateTimeFunctions.fromDateTime(DateTimeFunctions.java:271) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at jdk.internal.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.apache.pinot.common.function.FunctionInvoker.invoke(FunctionInvoker.java:127) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator$FunctionExecutionNode.execute(InbuiltFunctionEvaluator.java:229) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.function.InbuiltFunctionEvaluator.evaluate(InbuiltFunctionEvaluator.java:106) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] at org.apache.pinot.segment.local.recordtransformer.ExpressionTransformer.transform(ExpressionTransformer.java:123) ~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf] ... 7 more ``` ## The Pause/Resume workflow We're going to run the following workflow to sort this out: * Pause ingestion for the table * Fix the transformation function * Resume ingestion Pause consumption by running the following command: ```bash theme={null} curl -X POST \ "http://localhost:9000/tables/events/pauseConsumption" \ -H "accept: application/json" ``` Then we need to fix the config, with the following one: ```json {14} theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "ingestionConfig":{ "transformConfigs": [ { "columnName": "ts", "transformFunction": "FromDateTime(tsString, 'YYYY-MM-dd''T''HH:mm:ss.SSSSSS''Z''')" } ] }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-pauseresume:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder" } }, "tenants": {}, "metadata": {} } ``` *config/table.json* Let's apply that: ```bash theme={null} docker run \ --network pauseresume \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table-fixed.json \ -controllerHost "pinot-controller-pauseresume" \ -exec -update ``` And resume consumption: ```bash theme={null} curl -X POST \ "http://localhost:9000/tables/events/resumeConsumption?consumeFrom=smallest" \ -H "accept: application/json" ``` If we navigate back to the Pinot UI, it will now be ingesting data. And we have no more errors in our server logs either. # Ingest from Apache Pulsar Source: https://docs.startree.ai/recipes/pulsar In this Apache Pinot Recipe, we'll learn how to ingest data from Apache Pulsar. Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo! | Pinot Version | 1.1.0 | | - | - | | Code | [startreedata/pinot-recipes/pulsar](https://github.com/startreedata/pinot-recipes/tree/main/recipes/pulsar) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. Clone this repository and navigate to this recipe: ```bash theme={null} git clone git@github.com:startreedata/pinot-recipes.git cd pinot-recipes/recipes/ingest-json-files ``` ## Makefile ```bash theme={null} make recipe ``` To produce data into Pulsar, use the Python code below. ```python theme={null} import pulsar import json import time import random import uuid client = pulsar.Client('pulsar://localhost:6650') producer = client.create_producer('events') message = { "ts": int(time.time() * 1000.0), "uuid": str(uuid.uuid4()).replace("-", ""), "count": random.randint(0, 1000) } payload = json.dumps(message, ensure_ascii=False).encode('utf-8') producer.send(payload) client.close() ``` ## See the data Navigate to [localhost:9000/#/query](http://localhost:9000/#/query) to see the data in Apache Pinot. ## Clean up ```bash theme={null} make clean ``` ## Troubleshooting To clean up old Docker installations that may be interfering with your testing of this recipe, run the following command: ```bash theme={null} docker system prune ``` # How to Query a Table by Segment Source: https://docs.startree.ai/recipes/query-by-segment In this recipe we'll learn how to query a table to find out which records are in a particular segment. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/query-by-segment](https://github.com/startreedata/pinot-recipes/tree/main/recipes/query-by-segment) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/query-by-segment ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/query-by-segment/docker-compose.yml) file on GitHub. ## Data generator This recipe contains a data generator that creates events with a timestamp, count, and UUID. You can generate data by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "ts": 1680171022742, "uuid": "5328f9b8-83ff-4e4c-8ea1-d09524866841", "count": 260 } ``` ## Kafka ingestion We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t events -Kø ``` We can check that Kafka has some data by running the following command: ```bash theme={null} docker exec -it kafka-querysegment kafka-run-class.sh \ kafka.tools.GetOffsetShell \ --broker-list localhost:9092 \ --topic events ``` We'll see something like the following: ```bash theme={null} events:0:19960902 ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [{"name": "uuid", "dataType": "STRING"}], "metricFieldSpecs": [{"name": "count", "dataType": "INT"}], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` Now for the table config: ```json {13} theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "realtime.segment.flush.threshold.rows":"100000", "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-querysegment:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.time":"1h" } }, "ingestionConfig": { }, "tenants": {}, "metadata": {} } ``` This highlighted section indicates that we're going to create new segments after every 100,000 rows. We'll create the table by running the following: ```bash theme={null} docker run \ --network querysegment \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/schema.json \ -tableConfigFile /config/table.json \ -controllerHost "pinot-controller-querysegment" \ -exec ``` ## Querying by segment Once that's been created, we can head over to the [Pinot UI](http://localhost:9000/#/query) and run some queries. Pinot has several built-in virtual columns inside every schema that can be used for debugging purposes: | Column Name | Column Type | Data Type | Description | | - | - | - | - | | \$hostName | Dimension | STRING | Name of the server hosting the data | | \$segmentName | Dimension | STRING | Name of the segment containing the record | | \$docId | Dimension | INT | Document id of the record within the segment | The one that's useful for us is `$segmentName`, which we can use like this to count the number of records in each segment: ```sql theme={null} select $segmentName, count(*) from events group by $segmentName limit 10 ``` | \$segmentName | count(\*) | | - | - | | events\_\_0\_\_142\_\_20230330T1004Z | 100000 | | events\_\_0\_\_27\_\_20230330T1003Z | 100000 | | events\_\_0\_\_123\_\_20230330T1004Z | 100000 | | events\_\_0\_\_15\_\_20230330T1003Z | 100000 | | events\_\_0\_\_185\_\_20230330T1004Z | 100000 | | events\_\_0\_\_96\_\_20230330T1003Z | 100000 | | events\_\_0\_\_169\_\_20230330T1004Z | 100000 | | events\_\_0\_\_160\_\_20230330T1004Z | 100000 | | events\_\_0\_\_77\_\_20230330T1003Z | 100000 | | events\_\_0\_\_71\_\_20230330T1003Z | 100000 | *Query Results* We can then pick one of those segments and see what records are stored in that segment: ```sql theme={null} select * from events -- Change the segment name to match one that's returned by the previous query WHERE $segmentName = 'events__0__142__20230330T1004Z' limit 10 ``` | count | ts | uuid | | - | - | - | | 666 | 2023-03-30 10:03:04.498 | 8e4f5bb3-ad5c-45ec-89cb-946acee52625 | | 298 | 2023-03-30 10:03:04.498 | f439c3ad-86a1-4043-895e-681d5497cda0 | | 128 | 2023-03-30 10:03:04.498 | 9e29317f-c8fd-44d8-8313-aa28a80f1e5e | | 40 | 2023-03-30 10:03:04.498 | 697e2dbc-aee5-466e-b24e-763b18aff003 | | 659 | 2023-03-30 10:03:04.498 | 8d981789-7e61-4a25-81d1-b62616e877e8 | | 436 | 2023-03-30 10:03:04.498 | f4a883f3-3e6a-4169-a7ca-2469a5100ef2 | | 408 | 2023-03-30 10:03:04.498 | dc21499e-cdb3-4fae-93c1-0b9bd0233ae0 | | 457 | 2023-03-30 10:03:04.498 | 0de9a860-5af3-4155-93c9-d396df3cd0b3 | | 197 | 2023-03-30 10:03:04.498 | c4ddd15a-814f-4036-9eee-3121f59783f8 | | 362 | 2023-03-30 10:03:04.498 | 11b00463-2587-4838-9f05-089c05ad9fea | *Query Results* # Clickstream Analytics Dashboard with StarTree Cloud Free Tier and Streamlit Source: https://docs.startree.ai/recipes/real-time-clickstream-dashboard-guide This guide will walk you through creating a real-time clickstream analytics dashboard using StarTree Cloud (Apache Pinot) and Streamlit. ## Use Case An e-commerce company wants to understand user behavior on their website to improve engagement, streamline the customer journey, and increase conversions. They need real-time insights into how users interact with their website's content and design. ## Solution Use the following technologies: * Apache Pinot: A real-time OLAP datastore to capture and analyze clickstream data. * StarTree Cloud: A managed service for Apache Pinot, providing a free tier for experimentation and development. [Sign up for a Trial](https://startree.ai/trial/) * Streamlit: A Python framework for building interactive web applications, perfect for creating our dashboard. ## Dashboard Features The dashboard will provide the following insights: * Event Counts by Type: A bar chart showing how often different user actions occur (e.g., page views, product clicks, add to carts, purchases). * Average Duration by Event Type: A table displaying the average time users spend on each type of event. ## Benefits * Content Optimization: Identify underperforming pages and improve them based on user engagement. * Conversion Funnel Enhancement: Analyze the user journey and find areas where users drop off. * Personalization: Tailor website content and recommendations in real-time. * A/B Testing: Measure the impact of website changes by comparing clickstream data. * Marketing Campaign Effectiveness: Track the success of marketing campaigns. ## Prerequisites StarTree Cloud Free Tier Account: Sign up for a free account at [StarTree.ai](https://www.startree.ai/) * Python and Streamlit: Install Python and Streamlit on your local machine. * Sample Data: Prepare sample clickstream data or use the provided sample data. ## Steps ### Generate an Authentication Token: * Log in to your StarTree Cloud account. * Navigate to the "API Tokens" section. * Generate a new token with appropriate permissions. ### Update Broker URL and Token: * Replace placeholders in the Python script with your actual StarTree Cloud broker URL and authentication token. ### Install Required Packages: * Run pip install streamlit pandas altair requests in your terminal. ### Run the Streamlit App: * Save the provided Python script as streamlit\_app.py. * Execute streamlit run streamlit\_app.py. ### Access the Dashboard: * Open your web browser and go to [http://localhost:8501](http://localhost:8501). ### Code #### \[streamlit\_app.py] For the StarTree Free Tier - Serverless use the following script Note: Before running the script make sure to update the Pinot Broker URL and Bearer Token ```Python theme={null} import streamlit as st import requests import pandas as pd import altair as alt # StarTree Cloud Broker URL and Bearer Auth Token PINOT_BROKER_URL = 'https://broker.pinot.XXXX:443/query/sql' # Ensure the URL ends with /query/sql BEARER_AUTH_TOKEN = 'Bearer XXXX' # Replace with your actual Bearer Auth token # Function to query StarTree Cloud Pinot def query_pinot(query): headers = { 'Content-Type': 'application/json', 'Accept': 'application/json', 'Authorization': BEARER_AUTH_TOKEN, 'database':'test_workspace' } st.write("Headers being sent:") st.json(headers) # Log headers to ensure they are correctly formatted response = requests.post( PINOT_BROKER_URL, json={'sql': query}, headers=headers ) if response.status_code != 200: st.error(f"Error querying Pinot: {response.status_code}{response.reason}") st.text(response.text) # Display the response text for additional debugging return [] try: result = response.json() if 'resultTable' not in result or 'rows' not in result['resultTable']: st.error("Unexpected response format") st.json(result) # Display the unexpected JSON for debugging return [] return result['resultTable']['rows'] except ValueError: st.error("Failed to parse response as JSON") st.text(response.text) # Display the raw response text for debugging return [] # Streamlit app st.title('Clickstream Metrics Dashboard') # Query event counts by type event_counts_query = """ SELECT event_type, COUNT(*) as event_count FROM clickstream GROUP BY event_type """ event_counts_data = query_pinot(event_counts_query) if event_counts_data: event_counts_df = pd.DataFrame(event_counts_data, columns=['event_type', 'event_count']) # Display event counts st.write("## Event Counts by Type") st.write(event_counts_df) # Bar chart of event counts bar_chart = alt.Chart(event_counts_df).mark_bar().encode( x='event_type', y='event_count' ) st.altair_chart(bar_chart, use_container_width=True) # Query average duration by event type avg_duration_query = """ SELECT event_type, AVG(duration) as avg_duration FROM clickstream GROUP BY event_type """ avg_duration_data = query_pinot(avg_duration_query) if avg_duration_data: avg_duration_df = pd.DataFrame(avg_duration_data, columns=['event_type','avg_duration']) # Display average duration st.write("## Average Duration by Event Type") st.write(avg_duration_df) # Line chart of average duration line_chart = alt.Chart(avg_duration_df).mark_line().encode( x='event_type', y='avg_duration' ) st.altair_chart(line_chart, use_container_width=True) ``` #### \[streamlit\_app.py] For Dedicate Pinot Instance use this script Note: Before running the script make sure to replace the Pinot Broker URL and Basic Auth Token ```Python theme={null} import streamlit as st import requests import pandas as pd import altair as alt # StarTree Cloud Broker URL and Basic Auth Token PINOT_BROKER_URL = 'https://broker.pinot.XXXX:443/query/sql' # Replace with your actual broker URL BASIC_AUTH_TOKEN = 'Basic your-basic-auth-token' # Replace with your actual Basic Auth token # Function to query StarTree Cloud Pinot def query_pinot(query): headers = { 'Content-Type': 'application/json', 'Accept': 'application/json', 'Authorization': BASIC_AUTH_TOKEN } response = requests.post( f'{PINOT_BROKER_URL}/query/sql', json={'sql': query}, headers=headers ) if response.status_code != 200: st.error(f"Error querying Pinot: {response.status_code}{response.reason}") return [] try: result = response.json() if 'resultTable' not in result or 'rows' not in result['resultTable']: st.error("Unexpected response format") st.json(result) # Display the unexpected JSON for debugging return [] return result['resultTable']['rows'] except ValueError: st.error("Failed to parse response as JSON") st.text(response.text) # Display the raw response text for debugging return [] # Streamlit app st.title('Clickstream Metrics Dashboard') # Query event counts by type event_counts_query = """ SELECT event_type, COUNT(*) as event_count FROM clickstream GROUP BY event_type """ event_counts_data = query_pinot(event_counts_query) if event_counts_data: event_counts_df = pd.DataFrame(event_counts_data, columns=['event_type','event_count']) # Display event counts st.write("## Event Counts by Type") st.write(event_counts_df) # Bar chart of event counts bar_chart = alt.Chart(event_counts_df).mark_bar().encode( x='event_type', y='event_count' ) st.altair_chart(bar_chart, use_container_width=True) ``` # How to manually schedule real-time to offline job Source: https://docs.startree.ai/recipes/real-time-offline-job In this recipe we'll learn how to manually schedule Apache Pinot's Real-Time to Offline job. This job is used to transition data from real-time to offline tables. For background reading on the use case for doing this, see [Managed Offline Flow](/recipes/managed-offline-flow). | Pinot Version | 0.9.3 | | - | - | | Code | [startreedata/pinot-recipes/managed-offline-flow](https://github.com/startreedata/pinot-recipes/tree/main/recipes/managed-offline-flow) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/managed-offline-flow ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Pinot Minion, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/managed-offline-flow/docker-compose.yml) file on GitHub. ## Pinot Schema and Tables Now let's create a Pinot Schema, as well as real-time and offline tables. Pinot is going to take care of populating data into the offline table, but it still expects us to configure the table. ### Schema Our schema is going to capture some simple events, and looks like this: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [ { "name": "uuid", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "count", "dataType": "INT" } ], "dateTimeFieldSpecs": [{ "name": "ts", "dataType": "TIMESTAMP", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` *config/schema.json* You can create the schema by running the following command: ```bash theme={null} docker exec -it pinot-controller-rt bin/pinot-admin.sh AddSchema \ -schemaFile /config/schema.json \ -exec ``` ### Offline Table The offline table config is defined below: ```json theme={null} { "tableName": "events", "tableType": "OFFLINE", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "HOURLY" } }, "tableIndexConfig": { "loadMode": "MMAP" }, "tenants": {}, "metadata": {} } ``` *config/table-offline.json* You can create the table by running the following command: ```bash theme={null} docker exec -it pinot-controller-rt bin/pinot-admin.sh AddTable \ -tableConfigFile /config/table-offline.json \ -exec ``` ### Real-Time Table And the real-time table is defined below: ```json theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1", "retentionTimeUnit": "DAYS", "retentionTimeValue": "1" }, "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bufferTimePeriod": "1m", "bucketTimePeriod": "5m", "roundBucketTimePeriod": "1m", "schedule": "0 * * * * ?", "mergeType": "rollup", "count.aggregationType": "max", "maxNumRecordsPerSegment": "100000" } } }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-rt:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "10000", "realtime.segment.flush.threshold.time": "1h", "realtime.segment.flush.threshold.segment.size": "5M" } }, "tenants": {}, "metadata": {} } ``` *config/table-realtime.json* > [!WARNING] > > The `realtime.segment.flush.threshold.rows` config is intentionally set to an extremely small value so that the segment will be committed after 10,000 records have been ingested. > In a production system this value should be set much higher, as described in the [configuring segment threshold](/recipes/configuring-segment-threshold) guide. The main thing that we're interested in is the `RealtimeToOfflineSegmentsTask`, which is extracted below: ```json theme={null} "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bufferTimePeriod": "1m", "bucketTimePeriod": "5m", "roundBucketTimePeriod": "1m", "mergeType": "rollup", "count.aggregationType": "max", "maxNumRecordsPerSegment": "100000" } } } ``` This configuration enables the job and defines what should happen when it runs. Let's understand what each of the parameters does: * `bufferTimePeriod` - Tasks won't be scheduled unless the time window is older than this buffer. * `bucketTimePeriod` - The time window size/amount of data processed for each run. > [!WARNING] > > The values for `bufferTimePeriod` and `bucketTimePeriod` are intentionally set to very low values so that it's easier to see how they work. > In a production setup, these values would usually be set to a time of 1 day or more. * `roundBucketTimePeriod` - Round the time value before merging rows in the offline table. The value of `1m` that we've used means that values in the `ts` column will be rounded to the nearest column. * `mergeType` - The type of aggregation to apply when creating rows in the offline table. Valid values are: * `concat` - Don't aggregate anything. * `rollup` - Perform metric aggregations across common dimensions and time. * `dedup` - Deduplicate rows that have the same values. * `{metricName}.aggregationType` - Aggregation function to apply to the metric for aggregations. Only applicable for `rollup`. Allowed values are `sum`, `max`, and `min`. We are selecting the `max` value for the `count` column. * `maxNumRecordsPerSegment` - The number of records to include in a segment. You can create the table by running the following command: ```bash theme={null} docker exec -it pinot-controller-rt bin/pinot-admin.sh AddTable \ -tableConfigFile /config/table-realtime.json \ -exec ``` ## Ingesting Data Let's ingest data into the `events` Kafka topic, by running the following: ```bash theme={null} while true; do ts=`date +%s%N | cut -b1-13`; uuid=`cat /proc/sys/kernel/random/uuid | sed 's/[-]//g'` count=$[ $RANDOM % 1000 + 0 ] echo "{\"ts\": \"${ts}\", \"uuid\": \"${uuid}\", \"count\": $count}" done | docker exec -i kafka-rt /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 \ --topic events ``` Data will make its way into the real-time table. We can see how many records have been ingested by running the following query: ```sql theme={null} SELECT count(*) FROM events ``` ## Scheduling the RT2OFF Job The Real-Time to Offline Job can be scheduled automatically via the real-time table config or manually via the REST API. We can trigger it manually by running the following command: ```bash theme={null} curl -X POST "http://localhost:9000/tasks/schedule?taskType=RealtimeToOfflineSegmentsTask&tableName=events_REALTIME" \ -H "accept: application/json" 2>/dev/null | jq '.' ``` **Output** ```json theme={null} { "RealtimeToOfflineSegmentsTask": "Task_RealtimeToOfflineSegmentsTask_1646672575118" } ``` We can then check the Pinot Controller logs to see that it's been triggered: ```bash theme={null} docker exec -it pinot-controller-rt grep -rni --color "\[RealtimeToOff" logs/pinot-all.log ``` **Output** ```text theme={null} 2022/03/07 17:02:54.999 INFO [RealtimeToOfflineSegmentsTaskGenerator] [grizzly-http-server-8] Start generating task configs for table: events_REALTIME for task: RealtimeToOfflineSegmentsTask 2022/03/07 17:02:55.118 INFO [RealtimeToOfflineSegmentsTaskGenerator] [grizzly-http-server-8] Finished generating task configs for table: events_REALTIME for task: RealtimeToOfflineSegmentsTask ``` Let's run the job a few times and then navigate to [localhost:9000/#/tables](http://localhost:9000/#/tables). You'll see the following: ![Real-Time and Offline Tables](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/rt2off.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=061efc5f2427ae52e0c6200ea5222c1f) Real-Time and Offline Tables You can see that some segments have been created in the offline table: ## Time Boundary The data written to segments in offline tables won't immediately be seen in query results. When a query is received by Pinot Brokers, the broker sends a time boundary annotated version of the query to the offline and real-time tables. > [!NOTE] > > For more explanation about how the time boundary is computed, see [Time Boundary](/recipes/time-boundary-hybrid-table). The time boundary is based on the value of `ingestionConfig.batchIngestionConfig.segmentIngestionFrequency` in our offline table: * If it's set to `HOURLY`, then `timeBoundary = Maximum end time of offline segments - 1 HOUR` * Otherwise, `timeBoundary = Maximum end time of offline segments - 1 DAY` Our offline table had the following configuration: ```json theme={null} "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "HOURLY" } } ``` This means that the time boundary will be an hour before the maximum end time in offline segments. > [!NOTE] > > You can check the time boundary of a table by querying the HTTP API on the Pinot Broker, as shown in [How to work out the time boundary in hybrid tables](/recipes/time-boundary-hybrid-table) In our case the time boundary is set to `1646661240000` or `2022-03-07T13:54:00` in a more friendly format. This value will increase in increments of 5 minutes every time that the RT2OFF job runs as we set `bucketTimePeriod` to `5m`. You can increase this value in the real-time table config to have the job process more data. If we temporarily pause the ingestion, we can check what queries are actually getting executed on each table. Let's say we run the following query to count the number of records: ```sql theme={null} select count(*) from events ``` | count(\*) | | - | | 1450197 | *Query Results* Behind the scenes, the broker sends the following queries to the server: ```sql theme={null} select count(*) from events_OFFLINE WHERE ts <= 1646661240000 ``` > [!NOTE] > > The first time that you run this query it may return **no records** since the time stamp will be at least an hour ago and the data in the offline segments will probably be more recent than that. | count(\*) | | - | | 26595 | *Query Results* ```sql theme={null} select count(*) from events_REALTIME WHERE ts > 1646661240000 ``` | count(\*) | | - | | 1423602 | *Query Results* So it does seem like the query results are combining the offline and real-time tables. This becomes clearer if we write a query that returns the most common timestamps: ```sql theme={null} select ts, count(*) from events GROUP by ts ORDER BY count(*) DESC LIMIT 5 ``` | ts | count(\*) | | - | - | | 2022-03-04 15:54:00.0 | 17166 | | 2022-03-04 15:53:00.0 | 9429 | | 2022-03-07 16:04:56.059 | 1 | | 2022-03-07 15:29:59.193 | 1 | | 2022-03-07 14:55:02.075 | 1 | *Query Results* Recall that when we created offline segments we were rolling up timestamps to the nearest minute, which means that we now have a lot of events with the same timestamp. ## Summary In this guide we've learnt how to move data from a real-time table to an offline table using the Real-Time to Offline job. While doing this transformation we rolled up timestamp data to have a minute instead of millisecond granularity. # Automatically schedule real-time to offline job Source: https://docs.startree.ai/recipes/real-time-offline-job-automatic-scheduling In this recipe we'll learn how to automatically schedule Apache Pinot's Real-Time to Offline job. This job is used to transition data from real-time to offline tables. For background reading on the use case for doing this, see [Managed Offline Flow](/recipes/managed-offline-flow). | Pinot Version | 0.9.3 | | - | - | | Code | [startreedata/pinot-recipes/managed-offline-flow-automatic-scheduling](https://github.com/startreedata/pinot-recipes/tree/main/recipes/managed-offline-flow-automatic-scheduling) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/managed-offline-flow-automatic-scheduling ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Pinot Minion, Kafka, and Zookeeper. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/managed-offline-flow-automatic-scheduling/docker-compose.yml) file on GitHub. ## Controller configuration We need to provide configuration parameters to the Pinot Controller to have the Real-Time to Offline job (RT2OFF) run automatically. This is done in the following section of the Docker Compose file: ```yml theme={null} pinot-controller: image: apachepinot/pinot:0.9.3 command: "StartController -zkAddress zookeeper-rt:2181 -config /config/controller-conf.conf" ``` The configuration is specified in `/config/controller-conf.conf`, the contents of which are shown below: ```conf {8-10} theme={null} controller.access.protocols.http.port=9000 controller.zk.str=zookeeper-rt:2181 controller.helix.cluster.name=PinotCluster controller.host=pinot-controller-rt controller.port=9000 controller.data.dir=/data controller.task.scheduler.enabled=true controller.task.frequencyPeriod=5m ``` */config/controller-conf.conf* We're particularly interested in the last two lines: * `controller.task.scheduler.enabled=true` enables the automatic running of the RT2OFF job * `controller.task.frequencyPeriod=5m` configures it to run every 5 minutes ## Pinot Schema and Tables Now let's create a Pinot Schema, as well as real-time and offline tables. Pinot is going to take care of populating data into the offline table, but it still expects us to configure the table. ### Schema Our schema is going to capture some simple events, and looks like this: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [ { "name": "uuid", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "count", "dataType": "INT" } ], "dateTimeFieldSpecs": [{ "name": "ts", "dataType": "TIMESTAMP", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` *config/schema.json* You can create the schema by running the following command: ```bash theme={null} docker exec -it pinot-controller-rt bin/pinot-admin.sh AddSchema \ -schemaFile /config/schema.json \ -exec ``` ### Offline Table The offline table config is defined below: ```json theme={null} { "tableName": "events", "tableType": "OFFLINE", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "HOURLY" } }, "tableIndexConfig": { "loadMode": "MMAP" }, "tenants": {}, "metadata": {} } ``` *config/table-offline.json* You can create the table by running the following command: ```bash theme={null} docker exec -it pinot-controller-rt bin/pinot-admin.sh AddTable \ -tableConfigFile /config/table-offline.json \ -exec ``` ### Real-Time Table And the real-time table is defined below: ```json theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replication": "1", "replicasPerPartition": "1", "retentionTimeUnit": "DAYS", "retentionTimeValue": "1" }, "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bufferTimePeriod": "1m", "bucketTimePeriod": "5m", "roundBucketTimePeriod": "1m", "schedule": "0 * * * * ?", "mergeType": "rollup", "count.aggregationType": "max", "maxNumRecordsPerSegment": "100000" } } }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.broker.list": "kafka-rt:9093", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "10000", "realtime.segment.flush.threshold.time": "1h", "realtime.segment.flush.threshold.segment.size": "5M" } }, "tenants": {}, "metadata": {} } ``` *config/table-realtime.json* > [!WARNING] > > The `realtime.segment.flush.threshold.rows` config is intentionally set to an extremely small value so that the segment will be committed after 10,000 records have been ingested. > In a production system this value should be set much higher, as described in the [configuring segment threshold](/recipes/configuring-segment-threshold) guide. The main thing that we're interested in is the `RealtimeToOfflineSegmentsTask`, which is extracted below: ```json theme={null} "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bufferTimePeriod": "1m", "bucketTimePeriod": "5m", "schedule": "0 * * * * ?" } } } ``` This configuration enables the job and defines what should happen when it runs. The `schedule` parameter indicates when this task will be run. The value is a [Quartz Cron expression](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) and in this case we have the job running once a minute. > [!WARNING] > > The values for `bufferTimePeriod` and `bucketTimePeriod`, and `schedule` are intentionally set to very low values so that it's easier to see how they work. > > In a production setup, `bufferTimePeriod` and `bucketTimePeriod` would usually be set to a time of 1 day or more, and `schedule` could be set to run once a day. > [!NOTE] > > For a breakdown of all the parameters that can be defined for the `RealtimeToOfflineSegmentsTask`, see the [Manually scheduling real-time to offline job](/recipes/real-time-offline-job) guide. You can create the table by running the following command: ```bash theme={null} docker exec -it pinot-controller-rt bin/pinot-admin.sh AddTable \ -tableConfigFile /config/table-realtime.json \ -exec ``` ## Ingesting Data Let's ingest data into the `events` Kafka topic, by running the following: ```bash theme={null} while true; do ts=`date +%s%N | cut -b1-13`; uuid=`cat /proc/sys/kernel/random/uuid | sed 's/[-]//g'` count=$[ $RANDOM % 1000 + 0 ] echo "{\"ts\": \"${ts}\", \"uuid\": \"${uuid}\", \"count\": $count}" done | docker exec -i kafka-rt /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 \ --topic events ``` Data will make its way into the real-time table. We can see how many records have been ingested by running the following query: ```sql theme={null} SELECT count(*) FROM events ``` ## Checking that the job is running We can check that the job is running by querying the logs on the Pinot Controller, as shown below: ```bash theme={null} docker exec -it pinot-controller-rt grep -ri --color "\[RealtimeToOff" logs/pinot-all.log ``` We should see the following output: ```text theme={null} 2022/03/08 13:21:00.009 INFO [RealtimeToOfflineSegmentsTaskGenerator] [DefaultQuartzScheduler_Worker-8] Start generating task configs for table: events_REALTIME for task: RealtimeToOfflineSegmentsTask 2022/03/08 13:21:00.099 INFO [RealtimeToOfflineSegmentsTaskGenerator] [DefaultQuartzScheduler_Worker-8] Finished generating task configs for table: events_REALTIME for task: RealtimeToOfflineSegmentsTask ``` When the job can't create any new segments for the offline table it will instead output the following: ```text theme={null} 2022/03/08 13:24:00.008 INFO [RealtimeToOfflineSegmentsTaskGenerator] [DefaultQuartzScheduler_Worker-1] Start generating task configs for table: events_REALTIME for task: RealtimeToOfflineSegmentsTask 2022/03/08 13:24:00.076 INFO [RealtimeToOfflineSegmentsTaskGenerator] [DefaultQuartzScheduler_Worker-1] Window with start: 1646745600000 and end: 1646745900000 is not older than buffer time: 60000 configured as 1m ago. Skipping task generation: RealtimeToOfflineSegmentsTask ``` or ```text theme={null} 2022/03/08 13:30:00.201 INFO [RealtimeToOfflineSegmentsTaskGenerator] [DefaultQuartzScheduler_Worker-1] Start generating task configs for table: events_REALTIME for task: RealtimeToOfflineSegmentsTask 2022/03/08 13:30:00.212 INFO [RealtimeToOfflineSegmentsTaskGenerator] [DefaultQuartzScheduler_Worker-1] No realtime-completed segments found for table: events_REALTIME, skipping task generation: RealtimeToOfflineSegmentsTask ``` If you see these messages often, it might make sense to reduce the frequency of the RT2OFF job. # Download Recipes Source: https://docs.startree.ai/recipes/recipes Recipes help you learn how to solve common problems with Apache Pinot. To download Apache Pinot recipes, do one of the following: * Clone the repository using SSH * Clone the repository using HTTPS * If you **don't have a Git client** or **don't want to clone** the repository, [click to download a zip file](https://github.com/startreedata/pinot-recipes/archive/refs/heads/main.zip) that contains the recipes. ### Clone the `pinot-recipes` repository using SSH To clone the `pinot-recipes` repository using SSH, do the following: 1. Set up SSH keys associated with your GitHub account on your local machine. For more information, see [Connecting to GitHub with SSH](https://docs.github.com/en/authentication/connecting-to-github-with-ssh). 2. Run the following command: ```bash theme={null} git clone git@github.com:startreedata/pinot-recipes.git ``` ### Clone the `pinot-recipes` repository using HTTPS To clone the `pinot-recipes` repository using HTTPS, do the following: 1. Run the following command: ```bash theme={null} git clone https://github.com/startreedata/pinot-recipes.git ``` 2. At the prompt, enter your GitHub username and a personal access token. If you forgot your GitHub username, see [Remembering your GitHub username or email](https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-email-preferences/remembering-your-github-username-or-email). If you need a personal access token, see [Creating a personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token). # How to Remove a Server Source: https://docs.startree.ai/recipes/removing-server In this recipe we'll learn how to remove a server from a Pinot cluster. ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/removing-server ``` ## Launching Pinot Cluster Create Kubernetes cluster: ```bash theme={null} kind create cluster ``` You can spin up a Pinot Cluster by running the following command: ```bash theme={null} helm repo add pinot https://raw.githubusercontent.com/apache/pinot/master/kubernetes/helm kubectl create ns pinot-quickstart helm install pinot pinot/pinot \ -n pinot-quickstart \ --set cluster.name=pinot \ --set server.replicaCount=4 ``` Port Forward Pinot UI on port `9000` ```bash theme={null} kubectl port-forward service/pinot-controller 9000:9000 -n pinot-quickstart ``` ```bash theme={null} helm repo add kafka https://charts.bitnami.com/bitnami helm install -n pinot-quickstart kafka kafka/kafka --set replicas=1,zookeeper.image.tag=latest ``` Port forward Kafka on port 9090 ```bash theme={null} kubectl port-forward service/kafka-headless 9092:9092 -n pinot-quickstart ``` Add following line to `/etc/hosts` ``` 127.0.0.1 kafka-0.kafka-headless.pinot-quickstart.svc.cluster.local ``` Next, we're going to deploy a Kafka pod and connect to it: ```bash theme={null} kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.4.0-debian-11-r22 --namespace pinot-quickstart --command -- sleep infinity kubectl exec --tty -i kafka-client --namespace pinot-quickstart -- bash ``` We're going to create the `events` topic with 5 partitions, by running the following command: ```bash theme={null} kafka-topics.sh \ --create --bootstrap-server localhost:9092 \ --replication-factor 1 \ --partitions 5 \ --topic events ``` ## Generating data This recipe contains a data generator that creates events with a timestamp, count, and UUID. You can generate data by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "ts": 1680171022742, "uuid": "5328f9b8-83ff-4e4c-8ea1-d09524866841", "count": 260 } ``` ## Ingesting data into Kafka We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t events -Kø ``` ## Adding Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [{"name": "uuid", "dataType": "STRING"}], "metricFieldSpecs": [{"name": "count", "dataType": "INT"}], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` *schema.json* Now for the table config: ```json theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "timeType": "DAYS", "retentionTimeUnit": "DAYS", "retentionTimeValue": "3650", "segmentPushType": "APPEND", "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy", "schemaName": "events", "replication": "2", "replicasPerPartition": "2" }, "tenants": {}, "ingestionConfig":{ "transformConfigs": [ { "columnName": "ts", "transformFunction": "FromDateTime(tsString, 'YYYY-MM-dd''T''HH:mm:ss.SSSSSS''Z''')" } ] }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "simple", "stream.kafka.topic.name": "events", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.hlc.zk.connect.string": "kafka-zookeeper:2181", "stream.kafka.zk.broker.url": "kafka-zookeeper:2181", "stream.kafka.broker.list": "kafka:9092", "realtime.segment.flush.threshold.time": "3600000", "realtime.segment.flush.threshold.size": "50000", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} } } ``` *table.json* ```bash theme={null} kubectl apply -f config/pinot-events.yml ``` ## Checking segment assignment As soon as this config has been applied, Pinot will start ingesting the data from Kafka. We'll let it run for a little while and then run the following script to check segment assignment: ```bash theme={null} python segments_to_server.py \ --segments-to-servers false \ --partitions-to-servers false \ --servers-to-partitions true ``` **Output** ```text theme={null} Server_pinot-server-0 OrderedSet(['1', '3']) Server_pinot-server-1 OrderedSet(['0', '2', '4']) Server_pinot-server-2 OrderedSet(['0', '2', '4']) Server_pinot-server-3 OrderedSet(['1', '3']) ``` We can see that the partitions are spread across servers 0-3. ## Removing server from cluster Next, we're going to remove a server from the Pinot cluster: ```bash theme={null} kubectl scale statefulset.apps/pinot-server -n pinot-quickstart --replicas=3 ``` After we've done this, be should see one 'dead' server on the Pinot UI: ![List of Pinot servers with one dead](https://mintcdn.com/startree/SwAzqeuInnw8J52u/recipes/images/dead-server.png?fit=max\&auto=format\&n=SwAzqeuInnw8J52u\&q=85\&s=cf34bf7e0ac2ab29bf5c6c833147b5cc) List of Pinot servers with one dead In this example server 3 has been removed, but your mileage may vary. So, the underlying server has been removed, but Pinot still thinks its there. Pinot has an API that lets us remove instances, so let's try that out: Try to remove the instance: ```bash theme={null} instance="Server_pinot-server-3.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098" curl -X DELETE \ "http://localhost:9000/instances/${instance}" \ -H "accept: application/json" ``` ```json title="Output (shortened for brevity)" theme={null} { "code": 409, "error": "Failed to drop instance - Instance exists in ideal state for events_REALTIME" } ``` It didn't work because the server is still part of the ideal state for the table. ## Removing tags from server To fix this, we need to first remove all the tags from that server, by running the following: ```bash theme={null} instance="Server_pinot-server-3.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098" curl -X PUT \ "http://localhost:9000/instances/${instance}/updateTags?tags=&updateBrokerResource=false" \ -H "accept: application/json" ``` **Output** ```json theme={null} { "status":"Updated tags: [] for instance: Server_pinot-server-3.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098" } ``` This will prevent any new segments being assigned to this server, but we still have one more step to do. ## Rebalancing segments The next is to rebalanae the segments so that any assigned to server 3 will be moved elsewhere. The API that does this takes in a lot of parameters, so we've wrapped it in the `rebalance.py` script, which you can download from GitHub. The contents are shown below: ```python theme={null} import requests import click import json @click.command() @click.option('--table_name', default="events", help='Table name') def run_rebalance(table_name): base_url = f"http://localhost:9000/tables/{table_name}/rebalance" params = { 'type': 'realtime', 'dryRun': 'false', 'reassignInstances': 'true', 'includeConsuming': 'false', 'bootstrap': 'false', 'downtime': 'false', 'minAvailableReplicas': '1', 'bestEfforts': 'false', 'externalViewCheckIntervalInMs': '1000', 'externalViewStabilizationTimeoutInMs': '3600000', 'updateTargetTier': 'false', } response = requests.post(base_url, params=params) print(response.url) print(response.status_code) print(json.dumps(response.json(), indent=4)) if __name__ == '__main__': run_rebalance() ``` *rebalance.py* Adjust those parameters accordingly before you run the script. You can run the script like this: ```bash theme={null} python rebalance.py ``` If you then wait a few seconds or maybe longer depending on how many segments you have, the segments will be rebalanced. At the time we're writing this guide there isn't an API to check on the progress of rebalancing, but this will be added in Pinot 0.13. You can manually check if it's completed by running the following script again: ```bash theme={null} python segments_to_server.py \ --segments-to-servers false \ --partitions-to-servers false \ --servers-to-partitions true ``` Once rebalancing has completed, you shouldn't see server 3 in the output: **Output** ```text theme={null} Server_pinot-server-0 OrderedSet(['1', '2', '4']) Server_pinot-server-1 OrderedSet(['0', '1', '3', '4']) Server_pinot-server-2 OrderedSet(['0', '2', '3']) ``` ## Removing instance from Pinot And once that's the case, we can retry the API that removes the instance from Pinot: ```bash theme={null} instance="Server_pinot-server-3.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098" curl -X DELETE \ "http://localhost:9000/instances/${instance}" \ -H "accept: application/json" ``` **Output** ```json theme={null} { "status":"Successfully dropped instance" } ``` # How to Check Segment to Server Assignment Source: https://docs.startree.ai/recipes/segment-assignment In this recipe we'll learn how to check to which servers our segments have been assigned. ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/segment-assignment ``` ## Launch Pinot Cluster Create Kubernetes cluster: ```bash theme={null} kind create cluster ``` You can spin up a Pinot Cluster by running the following command: ```bash theme={null} helm repo add pinot https://raw.githubusercontent.com/apache/pinot/master/kubernetes/helm kubectl create ns pinot-quickstart helm install pinot pinot/pinot \ -n pinot-quickstart \ --set cluster.name=pinot \ --set server.replicaCount=4 ``` Port Forward Pinot UI on port `9000` ```bash theme={null} kubectl port-forward service/pinot-controller 9000:9000 -n pinot-quickstart ``` ```bash theme={null} helm repo add kafka https://charts.bitnami.com/bitnami helm install -n pinot-quickstart kafka kafka/kafka --set replicas=1,zookeeper.image.tag=latest ``` Port forward Kafka on port 9090 ```bash theme={null} kubectl port-forward service/kafka-headless 9092:9092 -n pinot-quickstart ``` Add following line to `/etc/hosts` ``` 127.0.0.1 kafka-0.kafka-headless.pinot-quickstart.svc.cluster.local ``` Next, we're going to deploy a Kafka pod and connect to it: ```bash theme={null} kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.4.0-debian-11-r22 --namespace pinot-quickstart --command -- sleep infinity kubectl exec --tty -i kafka-client --namespace pinot-quickstart -- bash ``` We're going to create the `events` topic with 5 partitions, by running the following command: ```bash theme={null} kafka-topics.sh \ --create --bootstrap-server kafka.pinot-quickstart.svc.cluster.local:9092 \ --replication-factor 1 \ --partitions 5 \ --topic events ``` ## Data generator This recipe contains a data generator that creates events with a timestamp, count, and UUID. You can generate data by running the following command: ```bash theme={null} python datagen.py 2>/dev/null | head -n1 | jq ``` Output is shown below: ```json theme={null} { "ts": 1680171022742, "uuid": "5328f9b8-83ff-4e4c-8ea1-d09524866841", "count": 260 } ``` ## Kafka ingestion We're going to ingest this data into an Apache Kafka topic using the [kcat](https://github.com/edenhill/kcat) command line tool. We'll also use `jq` to structure the data in the `key:payload` structure that Kafka expects: ```bash theme={null} python datagen.py --sleep 0.0001 2>/dev/null | jq -cr --arg sep ø '[.uuid, tostring] | join($sep)' | kcat -P -b localhost:9092 -t events -Kø ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "events", "dimensionFieldSpecs": [{"name": "uuid", "dataType": "STRING"}], "metricFieldSpecs": [{"name": "count", "dataType": "INT"}], "dateTimeFieldSpecs": [ { "name": "ts", "dataType": "TIMESTAMP", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` *schema.json* Now for the table config: ```json theme={null} { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "timeType": "DAYS", "retentionTimeUnit": "DAYS", "retentionTimeValue": "3650", "segmentPushType": "APPEND", "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy", "schemaName": "events", "replication": "2", "replicasPerPartition": "2" }, "tenants": {}, "ingestionConfig":{ "transformConfigs": [ { "columnName": "ts", "transformFunction": "FromDateTime(tsString, 'YYYY-MM-dd''T''HH:mm:ss.SSSSSS''Z''')" } ] }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "simple", "stream.kafka.topic.name": "events", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.hlc.zk.connect.string": "kafka-zookeeper:2181", "stream.kafka.zk.broker.url": "kafka-zookeeper:2181", "stream.kafka.broker.list": "kafka:9092", "realtime.segment.flush.threshold.time": "3600000", "realtime.segment.flush.threshold.size": "50000", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} } } ``` *table.json* You can add the schema and table config by running the following command: ```bash theme={null} kubectl apply -f config/pinot-events.yml ``` As soon as this config has been applied, Pinot will start ingesting the data from Kafka. ## Segment assignment Next, we're going to have a look at segment assignment across our 4 servers. Pinot is using balanced segment assignment, which is the default. This means that new segments are assigned to the server with the least segments already assigned. First, install some dependencies: ```bash theme={null} pip install click ordered-set requests ``` We can then run the script to see how partititons have been assigned: ```bash theme={null} python segments_to_server.py \ --segments-to-servers true \ --partitions-to-servers false \ --servers-to-partitions false ``` This will return the following (truncated) output: **Output** ```bash theme={null} Segment to Servers events__0__0__20230605T1334Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__0__1__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__0__2__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__0__3__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__0__4__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__0__5__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__1__0__20230605T1334Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__1__1__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__1__2__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__1__3__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__1__4__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__1__5__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__2__0__20230605T1334Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__2__1__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__2__2__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__2__3__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__2__4__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__2__5__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__3__0__20230605T1334Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__3__1__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__3__2__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__3__3__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__3__4__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__3__5__20230605T1335Z ['Server_pinot-server-2', 'Server_pinot-server-3'] events__4__0__20230605T1334Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__4__1__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__4__2__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__4__3__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__4__4__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] events__4__5__20230605T1335Z ['Server_pinot-server-0', 'Server_pinot-server-1'] ``` We can visually see that segments with the same prefix are being assigned to the same pair of servers. e.g. segments that begin with `events__4` are assigned to servers 0 and 1. The script also returns mappings of partitions to servers: ```bash theme={null} python segments_to_server.py \ --segments-to-servers false \ --partitions-to-servers true \ --servers-to-partitions false ``` **Output** ```bash theme={null} Partition to Servers 0 OrderedSet(['Server_pinot-server-0', 'Server_pinot-server-1']) 1 OrderedSet(['Server_pinot-server-2', 'Server_pinot-server-3']) 2 OrderedSet(['Server_pinot-server-0', 'Server_pinot-server-1']) 3 OrderedSet(['Server_pinot-server-2', 'Server_pinot-server-3']) 4 OrderedSet(['Server_pinot-server-0', 'Server_pinot-server-1']) ``` This gives us a tidier view of how data is being assigned to servers. Partition 0 is assigned to servers 0 and 1, partition 1 is assigned to servers 2 and 3, and so on. We can also flip it around and see the mapping of servers to partitions: ```bash theme={null} python segments_to_server.py \ --segments-to-servers false \ --partitions-to-servers false \ --servers-to-partitions true ``` **Output** ```bash theme={null} Server to Partitions Server_pinot-server-0 OrderedSet(['0', '2', '4']) Server_pinot-server-1 OrderedSet(['0', '2', '4']) Server_pinot-server-2 OrderedSet(['1', '3']) Server_pinot-server-3 OrderedSet(['1', '3']) ``` # Using a Star-Tree Index Source: https://docs.startree.ai/recipes/startree-index In this recipe we'll learn how to use the Star-tree index. To understand how this recipe processes data, examine the commands executed in the Makefile. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/startree-index](https://github.com/startreedata/pinot-recipes/tree/main/recipes/startree-index) | ## Makefile Run this recipe using make. ```bash theme={null} make recipe ``` This will build up the infrastructure: Pinot and Kafka, create the tables, and produce streaming data. Three tables will be generated: * webtraffic - a real-time table with any Pinot indexes. * webstraffic\_inverted - a real-time table with an `inverted index` on the columns: country, browserType, and deviceBrand. ```json theme={null} "invertedIndexColumns": [ "country", "browserType", "deviceBrand" ], ``` * webtraffic\_startree - a real-time table with a `startree index`. ```json theme={null} "starTreeIndexConfigs": [ { "dimensionsSplitOrder": [ "country", "browserType", "deviceBrand" ], "skipStarNodeCreationForDimensions": [], "functionColumnPairs": [ "COUNT__*", "SUM__timeSpent", "AVG__timeSpent" ], "maxLeafRecords": 10000 } ], ``` Open your browser to the [Pinot console](http://localhost:9000) and execute the SQL statements below. ```sql theme={null} select browserType, count(*) from webtraffic WHERE country = 'Uruguay' GROUP BY browserType limit 10 ``` ```sql theme={null} select country, sum(timeSpent) AS totalTime from webtraffic group by country order by totalTime DESC limit 10 ``` ```sql theme={null} select count(*), sum(timeSpent) AS totalTime from webtraffic where country = 'United Kingdom' order by totalTime DESC limit 10 ``` ```sql theme={null} select browserType, count(*) from webtraffic WHERE country IN ('Germany', 'United Kingdom', 'Spain') GROUP BY browserType limit 10 ``` Try changing `FROM webtraffic` to `FROM webtraffic_inverted` or `FROM webtraffic_stree` ## Clean up ```bash theme={null} make clean ``` ## Troubleshooting To clean up old Docker installations that may be interfering with your testing of this recipe, run the following command: ```bash theme={null} docker system prune ``` # How to Get the Time Boundary for a Hybrid Table Source: https://docs.startree.ai/recipes/time-boundary-hybrid-table Hybrid tables consist of real-time and offline tables with the same name. When querying these tables, the Pinot broker uses the time boundary to determine which records to read from the offline table and which to read from the real-time table. To learn how to get the time boundary for a hybrid table, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/lB3RaKJ0Hbs) | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/time-boundary-hybrid-table](https://github.com/startreedata/pinot-recipes/tree/main/recipes/time-boundary-hybrid-table) | > [!NOTE] > > For more information on time boundaries, see [Concepts: Time Boundary](https://dev.startree.ai/docs/pinot/concepts/time-boundary) ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/time-boundary-hybrid-table ``` ## Makefile The Makefile contains all of the commands need to start up Pinot and Kafka. Run the make command below. `make recipe` This command will also: * Create a Kafka topic called `events`. * Create a hybrid Pinot `events` table. * Batch load data into the offline `events` table in Pinot. * Generate stream data using the Pinot schema to Kafka and ultimately into the realtime `events` table in Pinot. When you go to the [table list](http://localhost:9000/#/tables) in Pinot, you will see an `events_REALTIME` and an `events_OFFLINE` table. When you go to the [query console](http://localhost:9000/#/query) in Pinot, you will only see one table: `events`. ## Select Pinot Segments The stream data generator will generate 1000 records into the `events_REALTIME` table. The batch loader will load 10 records into the `events_OFFLINE` table for a total of `1010` records. If you count the number of records in this table, you will only get `1000`. ```sql theme={null} select count(*) from events ``` If you look at the `query response stats` you'll see `1000` documents scanned from `1010` totalDocs. When querying hybrid tables, the Pinot Broker must decide which records to read from the offline table and which to read from the real-time table. If you run the SQL below, you'll see that there are no OFFLINE segments. They are only realtime segments. ```sql theme={null} select $segmentName, count(*) from events group by $segmentName ``` Check the current time boundary: ```bash theme={null} curl "http://localhost:8099/debug/timeBoundary/events" -H "accept: application/json" 2>/dev/null | jq '.' ``` Execute the API call below to force Pinot to update the `event` table's time boundary. ```sql theme={null} curl -X POST \ "http://localhost:9000/tables/events/timeBoundary" \ -H "accept: application/json" | \ jq ``` Run the query again below. This time, you should see both offline and realtime segments. ```sql theme={null} select $segmentName, count(*) from events group by $segmentName ``` # Full Upserts in Pinot Source: https://docs.startree.ai/recipes/upserts-full In this recipe we'll learn how to use the upsert functionality. Upsert is a term used to describe inserting a record into a database if it does not already exist or update it if it does exist. Practically, events arriving in a stream can have updated attributes at a later point in time. For example, an e-commerce order can transition between multiple states of a state machine, including `CREATED`, `PROCESSING`, `IN_TRANSIT`, and `DELIVERED`. A [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) stream capturing an orders table may emit change events containing different values for the order status. But, from an analytics perspective, you may only interested in the most up-to-date version and state for each order. For example, consider writing a SQL query to retrieve orders that took more than two days for the delivery. To enable that, we need to merge all change events belonging to a particular order to its latest value. Apache Pinot supports that by enabling upserts on a real-time table. ## Understanding upserts in Pinot Pinot, by default, allows querying all events ingested from a Kafka topic by a particular primary key (a dimension). Revisiting our e-commerce example above, that kind of a query will return all the state changes for all orders. In some cases, we need to get back the most up-to-date version and state for each order. Pinot is an immutable datastore, which means that there is no genuine concept of upsert as you stream data into it from Kafka. For the upsert implementation, it’s essential to understand that an individual record is not updated via a write; instead, updates are appended to a log and a pointer maintains the most recent version of a record. Pinot upserts work in two modes: 1. Full upserts 2. [Partial upserts](/recipes/upserts-partial) We will focus on full upserts in this recipe. To learn how to use full upserts on a real-time Pinot table, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/UVPszTFl2jE) | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/full-upserts](https://github.com/startreedata/pinot-recipes/tree/main/recipes/full-upserts) | ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/upserts-full ``` ## Start Pinot and learn Spin up a Pinot cluster using Docker Compose: ```bash theme={null} docker compose \ -f ../pinot-compose.yml \ -f ../kafka-compose.yml up -d ``` Add table and schema: ```bash theme={null} docker cp config/orders_schema.json pinot-controller:/opt/pinot/ docker cp config/orders_table.json pinot-controller:/opt/pinot/ docker exec pinot-controller ./bin/pinot-admin.sh AddTable \ -schemaFile /opt/pinot/orders_schema.json \ -tableConfigFile /opt/pinot/orders_table.json \ -exec ``` Open a tab to import messages into Kafka: ```bash theme={null} docker exec -it kafka kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders ``` Paste the following: ```json theme={null} {"order_id":1,"customer_id":104,"order_status":"IN_TRANSIT","amount":29.35,"ts":"1632467063"} {"order_id":2,"customer_id":105,"order_status":"COMPLETED","amount":3.24,"ts":"1618931459"} {"order_id":3,"customer_id":103,"order_status":"OPEN","amount":9.77,"ts":"1626484196"} {"order_id":4,"customer_id":104,"order_status":"COMPLETED","amount":90.35,"ts":"1623066325"} {"order_id":5,"customer_id":105,"order_status":"OPEN","amount":55.52,"ts":"1635543905"} ``` Query Pinot: ```sql theme={null} select * from orders limit 10 ``` Go back to the Kafka tab and paste the following: ```json theme={null} {"order_id":5,"customer_id":105,"order_status":"CANCELLED","amount":55.52,"ts":"1635543948"} ``` Run the following query in Pinot to see that the `order_status` is for `order_id` 5 is now set to `CANCELLED`. ```sql theme={null} select * from orders where order_id=5 ``` # Partial Upserts in Pinot Source: https://docs.startree.ai/recipes/upserts-partial To learn how to perform partial upserts on a real-time table, watch the following video, or complete the tutorial below. [YouTube video player](https://www.youtube.com/embed/V7rQ_pP1GtI) To get a better understanding of upserts and how they work, see the [Full Upserts](/recipes/upserts-full) documentation. | Pinot Version | 1.0.0 | | - | - | | Code | [startreedata/pinot-recipes/partial-upserts](https://github.com/startreedata/pinot-recipes/tree/main/recipes/partial-upserts) | > [!NOTE] > > JSON key merging with [MERGE\_JSON](/corecapabilities/manage-data/merge-json) will be available starting with the > **StarTree Cloud 0.17.0 release**. That guide uses StarTree's custom merger; > keep this Apache Pinot recipe's built-in strategy configuration when following the tutorial below. ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/upserts-partial ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker-compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Zookeeper, and Kafka. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/partial-upserts/docker-compose.yml) file on GitHub. ## Create `meetup_rsvp` Kafka topic This recipe explores capturing RSVPs for meetup events. A meetup can be hosted by multiple groups, at multiple venues. Let's create the `meetup_rsvp` topic in Kafka to record the RSVPs. ```bash theme={null} docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \ --create --bootstrap-server kafka:9092 --topic meetup_rsvp ``` ## Pinot Schema and Table Now let's create a Pinot Schema and Table. First, the schema: ```json theme={null} { "schemaName": "meetup_rsvp", "primaryKeyColumns": [ "event_id" ], "dimensionFieldSpecs": [ { "name": "event_id", "dataType": "INT" }, { "name": "venue_name", "dataType": "STRING", "singleValueField": false }, { "dataType": "STRING", "name": "group_name", "singleValueField": false } ], "metricFieldSpecs": [ { "name": "rsvp_count", "dataType": "INT" } ], "dateTimeFieldSpecs": [ { "name": "mtime", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` *config/meetup\_rsvp\_schema.json* Note that, the `event_id` column is appointed as the primary key, which is mandatory for upserts in Pinot. ```json theme={null} "primaryKeyColumns": [ "event_id" ] ``` We'll also have the following table config: ```json theme={null} { "tableName": "meetup_rsvp", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "mtime", "timeType": "MILLISECONDS", "retentionTimeUnit": "DAYS", "retentionTimeValue": "1", "segmentPushType": "APPEND", "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy", "schemaName": "meetup_rsvp", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowLevel", "stream.kafka.topic.name": "meetup_rsvp", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.hlc.zk.connect.string": "zookeeper:2181/kafka", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.zk.broker.url": "zookeeper:2181/kafka", "stream.kafka.broker.list": "kafka:9093", "realtime.segment.flush.threshold.size": 30, "realtime.segment.flush.threshold.rows": 30 }, "nullHandlingEnabled": true }, "metadata": { "customConfigs": {} }, "routing": { "instanceSelectorType": "strictReplicaGroup" }, "upsertConfig": { "mode": "PARTIAL", "partialUpsertStrategies": { "rsvp_count": "INCREMENT", "group_name": "UNION", "venue_name": "APPEND" } } } ``` *config/orders\_table.json* In this table configuration, we only enable upserts on three columns: `rsvp_count`, `group_name`, and `venue_name`. Hence, the `mode` is set to `PARTIAL`. ```json theme={null} "upsertConfig": { "mode": "PARTIAL", "partialUpsertStrategies": { "rsvp_count": "INCREMENT", "group_name": "UNION", "venue_name": "APPEND" } } ``` > [!NOTE] > > When using the `APPEND` strategy, you must make sure that the corresponding column can accept multiple values, by specifying the following config: > > ``` > "singleValueField": false > ``` When RSVP events are ingested with unique `event_id` values, `rsvp_count` will be incremented. The name of the group will added to the `group_name` column, if not exists. Also, the venue will be appended to the `venue_name`. You can create the table and schema by running the following command:\` ```bash theme={null} docker run \ --network upserts \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -tableConfigFile /config/meetup_rsvp_table.json \ -controllerHost "pinot-controller" \ -schemaFile /config/meetup_rsvp_schema.json \ -exec ``` ## Produce some RSVP events We will simulate a few RSVPs by publishing the following events to the `meetup_rsvp` Kafka topic. ```bash theme={null} echo - ' {"event_id":3,"venue_name":"Indonesia","group_name":"C","rsvp_count":1,"mtime":"1635140709"} {"event_id":3,"venue_name":"China","group_name":"C","rsvp_count":1,"mtime":"1646067689"} {"event_id":2,"venue_name":"France","group_name":"C","rsvp_count":1,"mtime":"1616646138"} {"event_id":1,"venue_name":"Myanmar","group_name":"B","rsvp_count":1,"mtime":"1632930567"} {"event_id":1,"venue_name":"Hungary","group_name":"A","rsvp_count":1,"mtime":"1643574332"} {"event_id":1,"venue_name":"China","group_name":"B","rsvp_count":1,"mtime":"1645779637"} ' | kcat -P -b localhost:9092 -t meetup_rsvp ``` ## Querying Once that's completed, navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `meetup_rsvp` table or copy/paste the following query: ```sql theme={null} select event_id, sum(rsvp_count) as total_rsvp from meetup_rsvp group by event_id order by total_rsvp desc limit 10 ``` # Upserts and the real-time to offline job Source: https://docs.startree.ai/recipes/upserts-real-time-offline-job In this guide, we'll learn how to use the combination of Pinot's upsert and real-time to offline job features. Let's quickly recap these features: * Upsert is usually used when we are capturing events that contain a state transition. This could be the location of a driver or the status of an order. When querying these events, we want to return the latest state, grouped by a primary key. * The real-time to offline job is used to move segments from a real-time table to an offline table. > [!WARNING] > > The main thing to keep in mind when combining these features is that upsert functionality only applies to real-time tables. > > As soon as those segments are moved to an offline table, the upsert logic is no longer applied at query time. > We will need to backfill the offline segments created by the real-time to offline job to achieve upsert-like queries. ## Prerequisites To follow the code examples in this guide, you must [install Docker](https://docs.docker.com/get-docker/) locally and download recipes. ## Navigate to recipe 1. If you haven't already, download recipes. 2. In terminal, go to the recipe by running the following command: ```bash theme={null} cd pinot-recipes/recipes/upserts-real-time-offline-job ``` ## Launch Pinot Cluster You can spin up a Pinot Cluster by running the following command: ```bash theme={null} docker compose up ``` This command will run a single instance of the Pinot Controller, Pinot Server, Pinot Broker, Pinot Minion, Zookeeper, and Kafka. You can find the [docker-compose.yml](https://github.com/startreedata/pinot-recipes/blob/main/recipes/upserts-real-time-offline-job/docker-compose.yml) file on GitHub. ## Pinot Schema and Tables Now let's create a Pinot Schema, as well as real-time and offline tables. Pinot is going to take care of populating data into the offline table, but it still expects us to configure the table. ### Schema Our schema is going to capture some order events, and looks like this: ```json theme={null} { "schemaName": "orders", "primaryKeyColumns": [ "order_id" ], "dimensionFieldSpecs": [{ "name": "order_id", "dataType": "INT" }, { "name": "customer_id", "dataType": "INT" }, { "name": "order_status", "dataType": "STRING" } ], "metricFieldSpecs": [{ "name": "amount", "dataType": "FLOAT" }], "dateTimeFieldSpecs": [{ "name": "ts", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` *config/orders\_schema.json* ### Offline Table The offline table config is defined below: ```json theme={null} { "tableName": "orders", "tableType": "OFFLINE", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "orders", "replication": "1", "replicasPerPartition": "1" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "HOURLY" } }, "tableIndexConfig": { "loadMode": "MMAP" }, "tenants": {}, "metadata": {} } ``` *config/orders\_offline\_table.json* You can create the table and schema by running the following command: ```bash theme={null} docker run \ --network upserts \ -v $PWD/config:/config \ apachepinot/pinot:1.0.0 AddTable \ -schemaFile /config/orders_schema.json \ -tableConfigFile /config/orders_offline_table.json \ -controllerHost "pinot-controller" \ -exec ``` ### Real-Time Table And the real-time table is defined below: ```json theme={null} { "tableName": "orders", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "timeType": "MILLISECONDS", "segmentPushType": "APPEND", "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy", "schemaName": "orders", "replicasPerPartition": "1" }, "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bucketTimePeriod": "2h", "bufferTimePeriod": "1m" } } }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowLevel", "stream.kafka.topic.name": "orders", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.hlc.zk.connect.string": "zookeeper:2181/kafka", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.zk.broker.url": "zookeeper:2181/kafka", "stream.kafka.broker.list": "kafka:9093", "realtime.segment.flush.threshold.rows": 5 } }, "metadata": { "customConfigs": {} }, "routing": { "instanceSelectorType": "strictReplicaGroup" }, "upsertConfig": { "mode": "FULL" } } ``` *config/orders\_table.json* > [!WARNING] > > The `realtime.segment.flush.threshold.rows` config is intentionally set to an extremely small value so that the segment will be committed after 10,000 records have been ingested. > In a production system this value should be set much higher, as described in the [configuring segment threshold](/recipes/configuring-segment-threshold) guide. Create the table by running the following command: ```bash theme={null} wget -q --method=POST "http://localhost:9000/tables?validationTypesToSkip=ALL" \ --header="accept: application/json" \ --header="Content-Type: application/json" \ --body-file=config/orders_table.json \ -O - ``` You **must** pass in `validationTypesToSkip=All` when calling this API endpoint. ## Ingesting Data Let's ingest data into the `events` Kafka topic, by running the following: ```bash theme={null} echo -e ' {"order_id":1,"customer_id":104,"order_status":"OPEN","amount":29.35,"ts":"1632463351000"} {"order_id":1,"customer_id":104,"order_status":"IN_TRANSIT","amount":29.35,"ts":"1632463361000"} {"order_id":1,"customer_id":104,"order_status":"COMPLETED","amount":29.35,"ts":"1632463391000"} {"order_id":2,"customer_id":105,"order_status":"COMPLETED","amount":3.24,"ts":"1632467065000"} {"order_id":3,"customer_id":103,"order_status":"OPEN","amount":9.77,"ts":"1632467066000"} {"order_id":4,"customer_id":104,"order_status":"OPEN","amount":55.52,"ts":"1632467068000"} {"order_id":4,"customer_id":104,"order_status":"CANCELLED","amount":55.52,"ts":"1632467070000"} {"order_id":5,"customer_id":105,"order_status":"OPEN","amount":12.22,"ts":"1632667070000"} {"order_id":5,"customer_id":105,"order_status":"IN_TRANSIT","amount":12.22,"ts":"1632667170000"} {"order_id":5,"customer_id":105,"order_status":"COMPLETED","amount":12.22,"ts":"1632677270000"} {"order_id":6,"customer_id":106,"order_status":"OPEN","amount":13.94,"ts":"1632677270400"} {"order_id":7,"customer_id":107,"order_status":"OPEN","amount":20.32,"ts":"1632677270403"} {"order_id":8,"customer_id":108,"order_status":"OPEN","amount":45.11,"ts":"1632677270508"} {"order_id":9,"customer_id":109,"order_status":"OPEN","amount":129.22,"ts":"1632677270699"} ' | kcat -P -b localhost:9092 -t orders ``` Data will make its way into the real-time table. Let's query the `orders` table: ```sql theme={null} select * from orders order by order_id limit 10 ``` | amount | customer\_id | order\_id | order\_status | ts | | - | - | - | - | - | | 29.35 | 104 | 1 | COMPLETED | 1632463391000 | | 3.24 | 105 | 2 | COMPLETED | 1632467065000 | | 9.77 | 103 | 3 | OPEN | 1632467066000 | | 55.52 | 104 | 4 | CANCELLED | 1632467070000 | | 12.22 | 105 | 5 | COMPLETED | 1632677270000 | | 13.94 | 106 | 6 | OPEN | 1632677270400 | | 20.32 | 107 | 7 | OPEN | 1632677270403 | | 45.11 | 108 | 8 | OPEN | 1632677270508 | | 129.22 | 109 | 9 | OPEN | 1632677270699 | *Query Results* As expected, we have the latest event for each order. ## Scheduling the RT2OFF Job The Real-Time to Offline Job can be scheduled [automatically through the real-time table config](/recipes/real-time-offline-job-automatic-scheduling) or manually via the REST API. We can trigger it manually by running the following command: ```bash theme={null} tableName="orders_REALTIME" wget -q --method=POST \ "http://localhost:9000/tasks/schedule?taskType=RealtimeToOfflineSegmentsTask&tableName=${tableName}" \ --header="accept: application/json" -O - ``` You can see which records are available in the offline segment by running the following query: ```sql theme={null} select * from orders_OFFLINE ``` | amount | customer\_id | order\_id | order\_status | ts | | - | - | - | - | - | | 29.35 | 104 | 1 | OPEN | 1632463351000 | | 29.35 | 104 | 1 | IN\_TRANSIT | 1632463361000 | | 29.35 | 104 | 1 | COMPLETED | 1632463391000 | | 3.24 | 105 | 2 | COMPLETED | 1632467065000 | | 9.77 | 103 | 3 | OPEN | 1632467066000 | | 55.52 | 104 | 4 | OPEN | 1632467068000 | | 55.52 | 104 | 4 | CANCELLED | 1632467070000 | *Query Results* We have duplicate records for `order_id=1` and `order_id=4`, which we'll need to sort out. The [time boundary](/recipes/time-boundary-hybrid-table) that indicates where records should be read from is `1632463470000`. This means that records with a timestamp less than or equal to this value will come from the offline table and records with a timestamp greater than this value will come from the real-time table. You can see which records will be returned from our newly created offline segment by running the following query: ```sql theme={null} select * from orders_OFFLINE WHERE ts <= 1632463470000 ``` | amount | customer\_id | order\_id | order\_status | ts | | - | - | - | - | - | | 29.35 | 104 | 1 | OPEN | 1632463351000 | | 29.35 | 104 | 1 | IN\_TRANSIT | 1632463361000 | | 29.35 | 104 | 1 | COMPLETED | 1632463391000 | *Query Results* This means that although we need to fix the records with `order_id=1` and `order_id=4`, we'll only see an impact on queries against the `orders` table for `order_id=1` until the time boundary increases. You can see which records are returned from the `orders` table by running the following query: ```sql theme={null} select * from orders where order_id = 1 or order_id = 4 ``` | amount | customer\_id | order\_id | order\_status | ts | | - | - | - | - | - | | 29.35 | 104 | 1 | OPEN | 1632463351000 | | 29.35 | 104 | 1 | IN\_TRANSIT | 1632463361000 | | 29.35 | 104 | 1 | COMPLETED | 1632463391000 | | 55.52 | 104 | 4 | CANCELLED | 1632467070000 | *Query Results* ## Replacing offline segment Let's now backfill the records in the offline segment with the documents in `data/orders.json` that only contain the most recent event for each order: ```json theme={null} {"order_id":1,"customer_id":104,"order_status":"COMPLETED","amount":29.35,"ts":"1632463391000"} {"order_id":2,"customer_id":105,"order_status":"COMPLETED","amount":3.24,"ts":"1632467065000"} {"order_id":3,"customer_id":103,"order_status":"OPEN","amount":9.77,"ts":"1632467066000"} {"order_id":4,"customer_id":104,"order_status":"CANCELLED","amount":55.52,"ts":"1632467070000"} ``` *data/orders.json* We'll ingest this file using the following ingestion spec: ```yaml theme={null} executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentNameGeneratorSpec: type: fixed configs: segment.name: ${segmentName} jobType: SegmentCreationAndTarPush inputDirURI: '/data' includeFileNamePattern: 'glob:**/orders.json' outputDirURI: '/opt/pinot/data/orders/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' tableSpec: tableName: 'orders' pinotClusterSpecs: - controllerURI: 'http://pinot-controller:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` *config/job-spec.yml* Now let's run the job to replace the offline segment, which has the name `orders_1632463351000_1632467070000_0`: ```bash theme={null} docker run \ --network upserts \ -v $PWD/config:/config \ -v $PWD/data:/data \ apachepinot/pinot:1.0.0 LaunchDataIngestionJob \ -jobSpecFile /config/job-spec.yml \ -values segmentName='orders_1632463351000_1632467070000_0' ``` Once we've done this we can go back and query the `orders` table: ```sql theme={null} select * from orders where order_id = 1 or order_id = 4 ``` | amount | customer\_id | order\_id | order\_status | ts | | - | - | - | - | - | | 29.35 | 104 | 1 | COMPLETED | 1632463391000 | | 55.52 | 104 | 4 | CANCELLED | 1632467070000 | *Query Results* We can also confirm that `order_id=4` has been fixed by querying the offline table: ```sql theme={null} select * from orders_OFFLINE ``` | amount | customer\_id | order\_id | order\_status | ts | | - | - | - | - | - | | 29.35 | 104 | 1 | COMPLETED | 1632463391000 | | 3.24 | 105 | 2 | COMPLETED | 1632467065000 | | 9.77 | 103 | 3 | OPEN | 1632467066000 | | 55.52 | 104 | 4 | CANCELLED | 1632467070000 | *Query Results* # Benefits of StarTree V2 Cloud Architecture Source: https://docs.startree.ai/reference/cloud-v2-benefits Since the last couple years, we've evolved the StarTree Cloud architecture to enhance security, enable modular updates, and support faster feature iteration. This new architecture introduces several advancements, including improved security through our Uber Agent model, a centralized config-driven approach, and enhanced data plane features. ## Key Features ### Stronger Security In the new design, we’re introducing a new component called *Uber Agent* that does most of the heavy lifting. Once deployed, the Uber agent can fetch the corresponding environment configuration from the control plane and take necessary actions. * Uber agent does the heavy lifting and sits in the Data Plane * It pulls instructions and configs from control plane and do the needful * No inbound access needed for day-day managing the cluster ### Centralized Configs All changes to an environment (including first deployment) now go through a “Config Service” hosted in the StarTree Control plane. There are well defined templates for various components which are under version control and managed in a centralized fashion. This makes it very easy for the StarTree Ops team to make changes, specify environment specific overrides - directly through our Admin portal. Each change is reviewed, versioned and safely applied. This avoids any manual tinkering in the customer environment. * All changes versioned, audited, and templated * Any customisation is auto synced with the dataplane * No overrides of the configs going forward ### Safer upgrades The new cloud architecture is more modular which also helps in upgrades, We can now update/upgrade individual components (Pinot, DM, Query console, Platform) in a safe manner. We’ve also improved the component release cadence across the entire fleet. For instance, Pinot, Data Manager and other software components can now be upgraded at a monthly cadence. ### Enhanced Automation A lot of the routine operataional tasks are handled via automation with minimal human effort. This includes enhanced pre-checks on Pinot to catch any issues early on and not during the upgrades. ### Enhanced data plane features In addition to changing the way infrastructure is deployed and managed, we’re also releasing new features in the data plane such as * **Heterogeneous Pinot clusters**: We can now create multiple server pools in the same Pinot cluster with ease (each pool can be customized in terms of the instance SKU) * **Secret management service**: store your credentials in a secure manner in standard systems (eg: AWS Secrets Manager) and reference it when creating datasets in StarTree Cloud * **Graviton instance support**: New environments can now be configured with Graviton instances which are roughly 10-20% cheaper * **Custom AMI support**  * **Custom Grafana Dashboards** and custom alert rules * Support for **Kafka AZ awareness** * **Workspaces** support: Users can now provision workspaces for ensuring logical separation of their Pinot cluster. # Glossary Source: https://docs.startree.ai/reference/glossary Definitions for real-time user-facing analytics, StarTree products, and platforms [A](#a) | [B](#b) | [C](#c) | [D](#d) | [E](#e) | [F](#f) | [G](#g) | [H](#h) | [I](#i) | [J](#j) | [K](#k) | [L](#l) | [M](#m) | [N](#n) | [O](#o) | [P](#p) | [Q](#q) | [R](#r) | [S](#s) | [T](#t) | [U](#u) | [V](#v) | [W](#w) [X](#x) |[Y](#y) |[Z](#z) ## A ### Apache Helix Apache Helix is an open-source, dynamic cluster management framework that manages and automatically distributes resources in a cluster. Helix maintains optimal system performance and reliability and ensures orderly and efficient operations amidst changing system states. In Apache Pinot, Helix manages topology changes for both brokers and servers, and optimizes query loads across the cluster. Helix uses Zookeeper to store cluster state and metadata. ### Apache ZooKeeper Pinot uses [Apache ZooKeeper](https://zookeeper.apache.org/) for cluster management and storing cluster state information. ZooKeeper is an open source project that manages the connection between various servers in a distributed system, synchronizing server states. ZooKeeper maintains configuration information, provides distributed synchronization, and facilitates group services to ensure cluster consistency, reliability, and orderly execution of processes. ### Adaptive routing Adaptive routing is used in Apache Pinot to optimize query routing and improve performance and latency. It dynamically selects a subset of servers to participate in a query based on factors such as server availability, workload balancing, and segment distribution. Adaptive routing distributes the workload evenly while minimizing tail latency and maximizing parallelism. Adaptive routing is a strategy used in query routing to dynamically select the most suitable server based on various factors, such as server statistics, workload, and performance metrics. Supports efficient distribution of queries across multiple servers to optimize query response time and resource utilization. ### Aggregation An aggregation combines multiple data values into a single value. Often used to derive summary statistics or metrics. Aggregations reduce the amount of data by grouping and summarizing the data based on specific criteria, such as time intervals or dimensions. Aggregations can be used to calculate metrics like sum, count, average, maximum, or minimum. In Apache Pinot, [ingestion aggregations](https://docs.pinot.apache.org/developers/advanced/ingestion-level-aggregations) are used to aggregate data during real-time data ingestion, resulting in improved query performance and reduced storage requirements. #### See also * [Supported Pinot aggregation functions](https://docs.pinot.apache.org/users/user-guide-query/query-syntax/supported-aggregations) * [How to write custom aggregation functions](https://docs.pinot.apache.org/developers/developers-and-contributors/extending-pinot/custom-aggregation-function) ### anomaly detection Anomaly detection identifies patterns or data points that deviate significantly from the expected or normal behavior. It involves analyzing data to detect unusual or unexpected events, outliers, or patterns that may indicate potential anomalies or abnormalities. Anomaly detection techniques are commonly used in various fields such as cybersecurity, fraud detection, network monitoring, and predictive maintenance. ### Apache Pinot Provides fast, scalable, and real-time analytics capabilities. Originally designed and developed at LinkedIn, and then donated to the Apache Software Foundation. [Apache Pinot](https://docs.pinot.apache.org/) is optimized for querying and aggregating large volumes of data in real-time, making it suitable for use cases such as monitoring, anomaly detection, personalization, recommendation systems, and interactive data exploration. Key features include real-time ingestion, columnar storage, distributed architecture, real-time indexing, SQL-like query language, and integration with ecosystem tools. ### Apache Pulsar [Apache Pulsar](https://pulsar.apache.org/) is a real-time distributed messaging and streaming platform that provides high-performance, durable messaging, and event streaming capabilities. It's designed to handle large-scale, high-throughput, and low-latency data streaming use cases. Pulsar offers features such as pub-sub messaging, message replay, geo-replication, and built-in support for schema enforcement and data retention policies. Pulsar is highly scalable and fault-tolerant. ### Apache ZooKeeper [Apache ZooKeeper](https://zookeeper.apache.org/) is a centralized service used to maintain configuration information. ZooKeeper provides distributed synchronization and group services. Pinot uses ZooKeeper for cluster management and storing cluster state information. ### Amazon Kinesis Amazon Kinesis is a fully managed service provided by Amazon Web Services (AWS) for real-time streaming data ingestion and processing. It allows you to collect, process, and analyze large amounts of data in real-time from various sources, including websites, mobile applications, IoT devices, and more. Kinesis provides capabilities for data streaming, data analytics, and data processing, enabling you to build real-time applications and gain insights from your streaming data. ### analytics Analytics is a field that investigates and interprets data to better understand the domain under analysis, and effectively communicate and make decisions based on the data. Two main types of data analytics consist of batch analytics and real-time analytics. ### array An array is a data structure that stores a collection of elements of the same type. It allows you to store multiple values in a single variable and access them using an index. ### authentication Authentication access in Apache Pinot supports HTTP basic authentication, and follows the established standards for HTTP basic authentication, where credentials are provided via an HTTP Authorization header. Pinot components, such as the Pinot controller and broker, can be configured to require authentication information (credentials) for API access. Users can provide their credentials through either dedicated username and password arguments or tokens. The Pinot Controller UI dynamically adapts to the authentication configuration and displays a login prompt when basic authentication is enabled. Restricted users are shown all available UI functions, but their operations will fail with an error message if access control lists (ACL) prohibit access. ### ## B ### backfill data Backfill data refers to the process of updating or filling in historical data in a system. In Apache Pinot, backfilling makes changes to the raw data and reflects these changes in a Pinot offline table. Typically, backfilling data is performed manually and requires writing custom flows to update the offline data. ### Balanced segment assignment Pinot assigns each segment to the server with the least segments already assigned by default. This balanced segment assigment strategy, ensures each server in a cluster has balanced query load, and each query is routed to all the servers in a cluster. ### Basic authentication A method for an HTTP user agent to provide a username and password when making an HTTP request. Contains a header field `Authorization: Basic `, where the credentials are the Base64 encoding of the username and password joined by a colon `:`. ### Batch data Batch ingestion or batch import is a data analytics method where data is collected over a period of time, and then processed all at once. Contrasts with streaming data which is processed continuously and immediately as it's generated. Handling data in batches saves resources and manages the workload effectively. Batch jobs may be throttled to a steady rate of processing, or jobs paused and resumed to prevent overrunning system resources. Used when immediate processing isn't crucial and when handling large volumes of data efficiently is a priority. Some architectures also use “microbatching,” where data is processed in small, discrete quantities—usually less than one minute of accumulated data. Pinot supports uploading data from standalone file systems, Hadoop, and Spark. For more information, see [Batch ingestion](https://docs.pinot.apache.org/basics/data-import/batch-ingestion). ### Bitmap inverted index A bitmap inverted index is a type of Pinot inverted index that maintains a mapping from each value to a bitmap of rows. This design allows for efficient value lookup operations, providing improved querying capabilities. ### Bloom filter In Apache Pinot, Bloom filters use a probabilistic data structure to determine which segments do not contain specific data. Makes scanning on disk more efficient by minimizing the amount of data that needs to be scanned. ### Broker The Pinot broker optimizes query processing, data retrieval, and enhances data-driven applications. The broker accepts queries from clients, forwards the queries to appropriate servers, collects results, and consolidates results into a single response to send back to the client. The broker uses [Helix](#helix) to find the location of each segment and route requests to the appropriate server. In [hybrid tables](#hybrid-table), the broker ensures that the overlap between real-time and offline segment data is queried exactly once by performing offline and real-time federation. ### Built-in virtual columns A built-in virtual columns are automatically generated and available in the schema for debugging purposes. Built-in virtual columns include $hostName, $ segmentName, and \$docId. ### Broker Query API Use the [Broker Query API](https://docs.pinot.apache.org/users/api/querying-pinot-using-standard-sql) to query data from Apache Pinot. It's typically used to retrieve data from Pinot and consolidate the results into a single response. Responsible for routing the query to the appropriate Pinot servers and merging the results before returning them to the client. ### Bloom filter A Bloom filter is a probabilistic data structure used to definitively determine if an element is not present in a dataset because the filter never yields false negatives. Because Bloom filters may produce false positives, a Bloom filter cannot determine with certainty whether an element is present in the dataset. ## C ### cluster An Apache Pinot cluster includes multiple nodes or servers working together to store and process data, including [brokers](#broker), [controllers](#controller) and [minions](#minion). The controller node coordinates data distribution, load balancing, and query routing across the cluster. Each node in the cluster is responsible for storing a portion of the data and executing query operations. Clusters are horizontally scalable, so you can add or remove nodes depending on your workload and data growth. Pinot cluster data is partitioned and distributed across the nodes to ensure high availability and parallel processing capabilities. ### column Pinot data is stored in tables within rows and columns. ### compaction Compressing data to optimize disk usage. ### controller Certain computer nodes in a cluster are set aside for cluster metadata, and cluster configuration and orchestration tasks. Pinot controller nodes use [Apache Helix](#apache-helix) and \[Apache Zookeeper]. ## D ### dashboard Visualize real-time analytics data. Dashboards let you query and graph data. ### Data Manager The Data Manager is a StarTree application used to ingest data into StarTree Cloud. ### data model A Pinot data model refers to the way data is organized and structured in Pinot, which includes concepts such as segments, tables, schemas, and tenants. Data is stored in a columnar format and adds additional indexes to enable fast filtering, aggregation, and group by operations. Raw data is divided into small data shards called segments, and one or more segments together form a table. Tables are associated with a schema that defines the columns and their data types. Tenants in Pinot prevent sharing ownership of database tables across microservice teams. ### data service Stores time series data and handles writes and queries. ### data source A source where data is collected from. Examples include Kafka, Hadoop Distributed File System (HDFS), Amazon S3, Google Cloud Storage (GCS), Azure Blob Storage, local file system, and other sources via custom connectors or integrations. ### data type A data type is defined by the values it can take, the programming language used, or the operations that can be performed on it. StarTree supports the following data types: | Data type | Alias/annotation | | :- | :- | | string | | | boolean | | | float | double | | integer | int, long | | unsigned integer | uint, unsignedLong | | time | dateTime | ## E ### event Metrics gathered at irregular time intervals. #### Exponential Time Smoothing (ETS) An enhanced form of the [Holt Winters](#holt-winters) exponential smoothing algorithm. ### expression A combination of one or more constants, variables, operators, and functions. ## F ### float A real number written with a decimal point dividing the integer and fractional parts (`1.0`, `3.14`, `-20.1`). ## G ### geometrycollection A spatial data type that represents a collection of different geometries, such as points, lines, polygons, or other geometry collections. Used to store and manipulate complex spatial structures in geospatial databases. Pinot supports geospatial data types including POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION. Pinot also supports various data types, including STRING, LONG, INT, and BOOLEAN. ### geospatial data types Geospatial data types abstract and encapsulate spatial structures such as boundary and dimension that represent shapes. Pinot supports geospatial data types such as POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION using Well-Known Text (WKT) and Well-Known Binary (WKB) forms of geospatial objects. Pinot also supports various data types, including STRING, LONG, INT, and BOOLEAN. ## H ### Helix Helix is a generic cluster management framework used to manage partitions and replicas in a distributed system. Helix is embedded in Apache Pinot components, such as the controller, broker, and server. Helix drives the state of a Pinot cluster and ensures cluster stability by coordinating state transitions and maintaining consistency. Helix uses Zookeeper to store cluster state and metadata. ### Helix partition A Helix partition represents a subset of data in a Pinot table. Used to divide data into smaller segments for efficient processing and distribution across Pinot servers. Each partition can have multiple replicas, which are copies of the same data, to ensure fault tolerance and high availability. ### histogram A visual representation of statistical information that uses rectangles to show the frequency of data items in successive, equal intervals or bins. ### Holt Winters A forecasting method that uses the following three components to predict future values based on historical data: * Average value of time series * Trend (historical upward or downward movement of data) * Seasonality (fluctuations or patterns that occur at fixed intervals) A visual representation of statistical information that uses rectangles to show the frequency of data items in successive, equal intervals or bins. ### hybrid table A hybrid table comprises one offline table and one real-time table that share the same name. An offline table is used to store data that is pushed periodically. A real-time table is used to store data consumed as it arrives. The offline table can have a high retention period than a real-time table. When an offline segment is pushed to cover a recent time period, the Pinot broker automatically switches to use the offline table for segments for that time period. ## I ### int A numeric data type that represents integers (whole numbers). Pinot supports various data types, including STRING, LONG, INT, and geospatial data types such as POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION. ### instance An entity comprising data on a server (or virtual server in cloud computing). ### instance owner A type of admin role for a user. Instance owners have read/write permissions for all resources within the instance. ## J ### Jaeger Open source tracing used in distributed systems to monitor and troubleshoot transactions. ### JSON JavaScript Object Notation (JSON) is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute–value pairs and array data types. ## K ### keyword A keyword is reserved by a program because it has special meaning. Every programming language has a set of keywords (reserved names) that cannot be used as an identifier. ## L ### linestring Represents a sequence of connected line segments in geospatial data. A type of geospatial object used to represent paths or lines on a map. Pinot supports geospatial data types including POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION. Pinot also supports various data types, including STRING, LONG, INT, and BOOLEAN. ### literal A literal is value in an expression, a number, character, string, function, record, or array. Literal values are interpreted as defined. ### long A numeric data type that can store whole numbers ranging from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807. It’s typically used to represent large integers. Pinot supports various data types, including STRING, LONG, INT, and geospatial data types such as POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION. ### load balancing Improves workload distribution across multiple computing resources in a network. Load balancing optimizes resource use, maximizes throughput, minimizes response time, and avoids overloading a single resource. Using multiple components with load balancing instead of a single component may increase reliability and availability. If requests to any server in a network increase, requests are forwarded to another server with more capacity. Load balancing can also refer to the communications channels themselves. ## M ### metastore Contains internal information about the system and status of the system. ### Minions Minions are lightweight, background workers that execute asynchronous tasks on Pinot segments. They are orchestrated by the Pinot Controller and coordinated using Apache Helix, which handles task assignments, execution tracking, and failure recovery. Minions enable Pinot to run ingestion and data lifecycle operations, such as segment generation, merging, transformation, and movement, without interrupting query-serving processes. They provide a scalable and fault-tolerant mechanism to handle ingestion and processing tasks in a distributed manner. ### multilinestring A geospatial data type that represents a collection of linestrings. Stores multiple linestrings as a single entity. Each linestring in a multilinestring is defined by a sequence of coordinates. Pinot supports geospatial data types including POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION. Pinot also supports various data types, including STRING, LONG, INT, and BOOLEAN. ### multipolygon A geospatial data type that represents a collection of polygons. Can consist of multiple polygons that are not connected to each other. Each polygon within a multipolygon can have its own set of coordinates and can represent a separate area or shape. Pinot supports geospatial data types including POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION. Pinot also supports various data types, including STRING, LONG, INT, and BOOLEAN. ## N ### node A server that is part of a cluster. Related entries: [server](#server) ### null A data type that represents a missing or unknown value. ## O ## P ### parameter A key-value pair used to pass information to functions. ### Pinot components A Pinot cluster consists of multiple distributed system components, linearly scalable across an unbounded number of nodes. Pinot includes the following components: * **Controller**: Manages, allocates, and schedules Pinot cluster resources. Drives consistency and routing in a Pinot cluster. Controllers are horizontally scaled as an independent component (container) and have visibility of the state of all other components in a cluster. The controller reacts and responds to state changes in the system and schedules the allocation of resources for tables, segments, or nodes. Also serves as the HTTP gateway for the REST API administration of a Pinot deployment. Includes a web-based query console to quickly and easily run queries. * **Broker**: Receives queries from a client and routes its execution to one or more Pinot servers before returning a consolidated response. * **Server**: Servers host segments that are allocated across multiple nodes and routed on an assignment to a tenant (single-tenant by default). Containers that scale horizontally are notified by Helix through state changes driven by the controller. A server can either be a real-time server or an offline server. A real-time and offline server have very different resource usage requirements. Because of this, resource isolation can be used to prioritize high-throughput real-time data streams that are ingested and made available for query through a broker. * **Minion** (optional): See [minion](#minions). ### Pinot storage model The Pinot storage model and infrastructure components include segments, tables, tenants, and clusters. Pinot has a distributed systems architecture that scales horizontally across multiple nodes as the size of a table grows over time. Pinot breaks data into segments (similar to shards/partitions in high-availability (HA) relational databases). Pinot tables are a logical abstraction that refers to a collection of related data in columns and rows (documents). Schemas associated with tables define the columns in a table and their data types. Multiple Pinot tables can share a single schema. Tables are independently configured for concerns such as indexing strategies, partitioning, tenants, data sources, or replication. Pinot supports multi-tenancy. Every Pinot table is associated with a tenant, so tables may belong to a particular logical namespace, grouped under a single tenant name, and isolated from other tenants. The isolation between tenants provides different namespaces for applications and teams to prevent sharing tables or schemas. Development teams building applications don't need to operate an independent deployment of Pinot. An Pinot organization can operate a single cluster and scale it out as new tenants increase the overall volume of queries. Manage your own schemas and tables without being impacted by other tenants on a cluster. By default, all tables belong to a default tenant named "default". Tenants satisfy the architectural principle of a "database per service/application" without having to operate many independent data stores. Tenants schedule resources so that segments are able to restrict a table's data to reside only on a specified set of nodes. Similar to the kind of isolation that is ubiquitously used in Linux containers, compute resources in Pinot can be scheduled to prevent resource contention between tenants. A Pinot cluster is a group of tenants, and a set of compute nodes. Typically, there is only one cluster per environment or data center. Multiple clusters aren't necessary givens Pinot supports multiple tenants. Pinot cluster may consist of 1000+ nodes distributed across a data center. Cluster nodes can be added to linearly increase performance and availability of queries. The number of nodes and the compute resources per node reliably predicts the query per second (QPS) for a Pinot cluster. Capacity planning is easily achieved using service-level agreements (SLAs) that assert performance expectations for end-user applications. ### point A point data type is a geospatial data type that represents a single point in space. Used to store and manipulate coordinates in a two-dimensional space. In Pinot, a point data type can be represented using the Well-Known Text (WKT) or Well-Known Binary (WKB) formats. Pinot supports geospatial data types including POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION. Pinot also supports various data types, including STRING, LONG, INT, and BOOLEAN ### polygon A geospatial data type that represents a closed shape with straight sides. In Pinot, a polygon can be defined using the Well-Known Text (WKT) or Well-Known Binary (WKB) format, and consists of a series of coordinates that define the vertices of the polygon. A polygon can have any number of sides and can be used to represent various geographic features such as land boundaries or building footprints. Pinot supports geospatial data types including POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION. Pinot also supports various data types, including STRING, LONG, INT, and BOOLEAN. ### predicate expression A predicate expression compares two values and returns `true` or `false` based on the relationship between the two values. A predicate expression is comprised of a left operand, a comparison operator, and a right operand. ### process A set of predetermined rules. A process can refer to instructions being executed by the computer processor or refer to the act of manipulating data. ## Q ### query A request that returns real-time analytics data. ## R ### Real-time analytics Real-time analytics includes collecting, analyzing, and interpreting data in real-time or almost real-time. Helps organizations make data-driven decisions in time-sensitive situations. Useful in multiple areas, including fraud detection, discovering anomalies, optimizing operational efficiency, personalizing customer experiences, and enabling proactive response to changing conditions. ### REPL A Read-Eval-Print Loop (REPL) is an interactive programming environment where you type a command and immediately see the result. ### record A tuple of named values represented using a record type. ### regular expressions Regular expressions (regex or regexp) are patterns used to match character combinations in strings. ### replication factor An attribute that determines how many copies of the data are stored in the cluster. ### RFC3339 timestamp A timestamp that uses the human-readable DateTime format proposed in [RFC 3339](https://tools.ietf.org/html/rfc3339) (for example: `2020-01-01T00:00:00.00Z`). ### RFC3339Nano timestamp A [Golang representation of the RFC 3339 DateTime format](https://go.dev/src/time/format.go) that uses nanosecond resolution--for example: `2006-01-02T15:04:05.999999999Z07:00`. ## S ### schema Defines the structure, names, and data types used in the columns in a Pinot table. A schema also defines the type of column, including dimension columns, metric columns, and time columns. The Pinot schema is stored in Zookeeper along with the table configuration, and is used for efficient data processing and analysis in Pinot. ### secret A secret is a key-value pairs that contains information you want to control access to, such as API keys, passwords, or certificates. ### segment A segment is a time-based partition used for efficient data storage and querying in Pinot clusters. A horizontal shard that represents a chunk of table data with some number of rows. A segment stores data for all table columns in the table and has a columnar format for efficient memory mapping and query serving. ### shard Shards are referred to as segments in Pinot. A shared represents a chunk of table data. Stores data for all columns of the table in a columnar format, and can be directly mapped into memory for serving queries. ### stream A stream is a continuous flow of data that is generated and consumed in real-time. Streams let you process and analyze data as it's produced, giving you real-time insights and actions. Streams are commonly used in systems like Apache Kafka to handle high volumes of data and enable real-time data ingestion and processing. ### stream ingestion Stream ingestion is the process of ingesting data from streaming services, such as Kafka, into a database, such as Apache Pinot. Stream ingestion lets you query data in real-time as soon as it's ingested into the database. Ingesting a stream of data provides support for checkpoints to prevent data loss. Stream ingestion can be configured to throttle the consumption rate for better performance management. Custom stream ingestion plugins can also be developed to support other streaming platforms. ### string A data type used to represent text. Pinot supports various data types, including STRING, LONG, INT, and geospatial data types such as POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION. ## T ### TCP StarTree uses Transmission Control Protocol (TCP) port 8086 for client-server communication over the StarTree HTTP API.? ### table Tables store real-time analytics data. ### technical preview A new feature released to gather feedback from customers and the community. Send feedback to our [StarTree Community Slack](https://startreecommunity.slack.com/). ### tenant Pinot supports multiple tenants. Because each Pinot table is associated with a tenant, tables may belong to a particular logical namespace, grouped under a single tenant name, and isolated from other tenants. The isolation between tenants provides different namespaces for applications and teams to prevent sharing tables or schemas. Development teams building applications don't need an independent deployment of Pinot. ### time series data Sequence of data points typically consisting of successive measurements made from the same source over a time interval. Time series data shows how data evolves over time. On a time series data graph, one of the axes is always time. Time series data may be regular or irregular. Regular time series data changes in constant intervals. Irregular time series data changes at non-constant intervals. ### token Tokens (or API tokens) verify user and organization permissions. ### transformation A function that returns a value or a set of values calculated from specified points. ## U ### UDP User Datagram Protocol is a packet of information. When a request is made, a UDP packet is sent to the recipient. The sender doesn't verify the packet is received. The sender continues to send the next packets. This means computers can communicate more quickly. This protocol is used when speed is desirable and error correction is not necessary. ### User-facing real-time analytics Analytical tools exposed to the end users of your product. In a user-facing analytics application, users receive personalized analytics on their devices, resulting in hundreds of thousands of queries per second. Queries triggered by apps may grow quickly in proportion to the number of active users on the app, as many as millions of events per second. Data generated in Pinot is immediately available for analytics in latencies under one second. User-facing real-time analytics requires ingesting data in real time, support for high-velocity, high-dimensional event data from multiple sources, low latency query results for hundreds of thousands of queries per second. Also requires reliability and high availability, scalability, and low cose to serve. ## V ### variable A storage location (identified by a memory address) paired with an associated symbolic name (an identifier). A variable contains some known or unknown quantity of information referred to as a value. ## W ### windowing Grouping data based on specified time intervals. ## Z ### ZooKeeper [Apache ZooKeeper](https://zookeeper.apache.org/) is a centralized service used to maintain configuration information. ZooKeeper provides distributed synchronization and group services. Pinot uses ZooKeeper for cluster management and storing cluster state information. # StarTree Release Notes Source: https://docs.startree.ai/reference/startree-release-notes Every StarTree Cloud platform release and Data Portal release in one place, with highlights and links to the full notes. StarTree ships on two tracks. **StarTree Cloud platform releases** carry the StarTree distribution of Apache Pinot along with StarTree extensions, and they arrive as numbered versions every few months. **Data Portal releases** update the web console you use to onboard data, run queries, and manage tables, and they ship every few weeks under a release date. This page lists every release on both tracks. Use it to catch up on what changed since your last upgrade, or to find the release that introduced a feature. ## Latest releases > **[StarTree Cloud 0.16.0](/reference/startree-release-notes/0.16.0)** > > **July 2026.** External Tables scale to large, frequently updated Iceberg datasets with segment groups and deletion vectors. The StarTree Execution Engine gains new join types, spill to disk, and `BIG_DECIMAL` support. > **[Data Portal: September 22, 2026](/reference/startree-release-notes/data-portal#september-22-2026)** > > **September 2026.** Actionable error messages with a public error code reference, Query Insights, and connection editing on by default. ## What to read > **[Upgrading a cluster](#startree-cloud-platform-releases)** > > Read every platform release between your current version and the target. Behavior changes add up across versions. > **[Using Data Portal](#data-portal-releases)** > > Data Portal updates roll out automatically. Check the latest dates to see new screens and workflows. > **[Breaking changes](/reference/startree-release-notes/0.16.0)** > > Recent platform releases list breaking changes in their own section near the end of each page. ## StarTree Cloud platform releases Each version bundles a StarTree Pinot build with the Apache Pinot changes it picks up and the StarTree extensions built on top. Version 0.13 was not released. **2025 to 2026** | Version | Released | Highlights | | - | - | - | | [0.16.0](/reference/startree-release-notes/0.16.0) | July 2026 | Segment groups and Iceberg deletion vectors for External Tables, StarTree Execution Engine maturity, Minion Task Orchestration and audit logging on by default | | [0.15.0](/reference/startree-release-notes/0.15.0) | April 2026 | StarTree Iceberg Tables GA with Iceberg catalog support, Query Log, replica scale down, Parquet and complex type correctness | | [0.14.0](/reference/startree-release-notes/0.14.0) | March 2026 | External Tables for Iceberg (Glue and S3 Tables) and S3 Parquet, Columnar Segment Processing, Minion Task Orchestration Framework | | [0.12.0](/reference/startree-release-notes/0.12.0) | February 2026 | Segment backfill and purge, consolidated preload for tiered storage, Table Storage Usage API, embedded schema extraction for Parquet | | [0.11.1](/reference/startree-release-notes/0.11.1) | September 2025 | Tenant rebalance, row level RBAC, Pinot audit logs, text index pinning for tiered storage | | [0.11.0](/reference/startree-release-notes/0.11.0) | July 2025 | Graviton GA on AWS, AZ aware Kafka consumer, Alter Table Task GA, Calcite 1.39 upgrade | | [0.10.1](/reference/startree-release-notes/0.10.1) | February 2025 | Pauseless ingestion (Beta), Alter Table Task (Beta), CTE reuse (Alpha), snapshots for off heap dedup | **2023 to 2024** | Version | Released | Highlights | | - | - | - | | [0.10.0](/reference/startree-release-notes/0.10.0) | November 2024 | Upsert backfill with externally partitioned segments, OOM protection in the multi-stage engine, Delta Lake 3.0 connector, DynamoDB CDC ingestion | | [0.9.0](/reference/startree-release-notes/0.9.0) | June 2024 | Faster upserts, re-streaming to recreate tables from streams, cursor pagination for large query results | | [0.8.1](/reference/startree-release-notes/0.8.1) | April to May 2024 | Kubernetes 1.29 upgrade for AWS and GCP deployments, platform stability fixes | | [0.8.0](/reference/startree-release-notes/0.8.0) | February 2024 | StarTree Upsert on by default, Cluster Health Dashboard, role based access control for tables | | [0.7.2](/reference/startree-release-notes/0.7.2) | December 2023 | Kubernetes 1.25 upgrade on AWS, IMDSv2 by default | | [0.7.1](/reference/startree-release-notes/0.7.1) | November 2023 | File ingestion task improvements, size based tuning for segment refresh, separate RocksDB logs | | [0.7.0](/reference/startree-release-notes/0.7.0) | August 2023 | Segment purge for GDPR deletes, tiered storage with sparse indexes (Beta), faster restarts for large upsert tables | | [0.6.1](/reference/startree-release-notes/0.6.1) | April 2023 | Tiered storage performance, EKS 1.23 upgrade, S3 bucket versioning on by default | | [0.6.0](/reference/startree-release-notes/0.6.0) | February 2023 | Multi-stage engine joins and clauses, multi volume support, consumer record lag, upsert enhancements | **2022** | Version | Released | Highlights | | - | - | - | | [0.5.0](/reference/startree-release-notes/0.5.0) | June to August 2022 | Null value support, real-time deduplication, upsert improvements, query cancellation | | [0.4.0](/reference/startree-release-notes/0.4.0) | April to May 2022 | Segment push to real-time tables, server failure detector, timestamp index, Spark 3.x support | ## Data Portal releases Data Portal releases are named by date and apply to every environment without an upgrade. Full notes for every release are on the [Data Portal release notes](/reference/startree-release-notes/data-portal) page. **2026** | Released | Highlights | | - | - | | [September 22, 2026](/reference/startree-release-notes/data-portal#september-22-2026) | Actionable error messages and error code reference, Query Insights, connection editing on by default | | [August 27, 2026](/reference/startree-release-notes/data-portal#august-27-2026) | Table Health Dashboard Overview tab, ZSTANDARD compression by default in generated table configs | | [August 11, 2026](/reference/startree-release-notes/data-portal#august-11-2026) | Monitoring dashboard for External Tables, Apache Polaris, Snowflake Open Catalog, and Horizon Catalog support | | [July 20, 2026](/reference/startree-release-notes/data-portal#july-20-2026) | Unity Catalog and GCS Parquet External Tables | | [July 10, 2026](/reference/startree-release-notes/data-portal#july-10-2026) | Table Health Dashboard and alerting, query history in Query Console | | [July 1, 2026](/reference/startree-release-notes/data-portal#july-1-2026) | Task status summary in the task manager, trigger Purge and File Ingestion tasks from Data Portal | | [June 12, 2026](/reference/startree-release-notes/data-portal#june-12-2026) | In-product documentation and tooltips, dictionary encoded indexes on External Tables | | [May 6, 2026](/reference/startree-release-notes/data-portal#may-6-2026) | IAM role authentication for External Tables, RBAC aware Data Portal | | [April 20, 2026](/reference/startree-release-notes/data-portal#april-20-2026) | Scheduled External Table ingestion, table storage usage | | [March 25, 2026](/reference/startree-release-notes/data-portal#march-25-2026) | Query Analyzer in the Query Console, shareable saved queries | | [March 11, 2026](/reference/startree-release-notes/data-portal#march-11-2026) | Raw S3 Parquet and Iceberg REST catalog support for AWS Glue and S3 Tables (Beta) | | [February 18, 2026](/reference/startree-release-notes/data-portal#february-18-2026) | Tiered storage configuration, broker and server tenant configuration in the UI | | [January 15, 2026](/reference/startree-release-notes/data-portal#january-15-2026) | Protobuf descriptor files for real-time tables, cancel queries from the editor | | [January 9, 2026](/reference/startree-release-notes/data-portal#january-9-2026) | Query observability, richer Table List view | **2025** | Released | Highlights | | - | - | | [December 17, 2025](/reference/startree-release-notes/data-portal#december-17-2025) | Visual query execution stats and EXPLAIN PLAN, dedicated JSON tab for execution details | | [December 10, 2025](/reference/startree-release-notes/data-portal#december-10-2025) | Stage level stats for multi-stage queries, segment builder defaults based on schema width | | [December 4, 2025](/reference/startree-release-notes/data-portal#december-4-2025) | Folder selection for batch sources, partial results in Query Console | | [November 11, 2025](/reference/startree-release-notes/data-portal#november-11-2025) | Tableau as a dedicated client, Clients section moved into Data Portal | # Release Version 0.10.0: November 2024 Source: https://docs.startree.ai/reference/startree-release-notes/0.10.0 ## Apache Pinot updates since the last StarTree release * Enhanced support for backfilling data in an upsert enabled table. Users can now upload externally partitioned segments to an upsert-enabled table. [\[link\]](https://github.com/apache/pinot/pull/13107) * Added several improvements in UI load times to improve user experience, especially in environments where there are a large number of tables and/or very large tables. [\[link\]](https://github.com/apache/pinot/pull/13296) * Added support for OOM protection in multi-stage query engine. This feature will prevent servers from crashing by killing expensive queries. This ensures that other applications are not disrupted. [\[link\]](https://docs.pinot.apache.org/operators/operating-pinot/oom-protection-using-automatic-query-killing) [\[link\]](https://github.com/apache/pinot/pull/13598) * Improved scalability by allowing multiple segments to be uploaded together instead of one by one. This is especially useful in cases where a large number of segments are added during initial ingestion or data backfill. [\[link\]](/corecapabilities/manage-data/segment-refresh-task) [\[link\]](https://github.com/apache/pinot/pull/13646) * Improved observability and stability by providing an API for checking the segment state. Users can now determine if some segments need to be reloaded to ensure all replicas are in the correct state. [\[link\]](https://github.com/apache/pinot/pull/13789) * Introduced query rate limiter at the database level, which will apply to all the tables in the database at the aggregate level. [\[link\]](https://github.com/apache/pinot/pull/13544) [\[link\]](https://docs.pinot.apache.org/users/user-guide-query/query-quotas#database-level-query-quota) * Added MAP type support with string keys and typed values along with the MapItem function, which can extract map values using a key. The support for map type is also added to the Pinot UI. [\[link\]](https://docs.pinot.apache.org/configuration-reference/schema#complexfieldspec) [\[link\]](https://github.com/apache/pinot/pull/14245) * Added implementations for comparison (=, !=, >, >=, \<, \<=, BETWEEN) and binary arithmetic scalar functions for multi-stage query engine. This resolves issues like string comparison failure due to the lack of polymorphism support and incorrect result types for numeric arithmetic. [\[link\]](https://github.com/apache/pinot/pull/13711) * Added parameters to support aggregation functions like DISTINCTCOUNTHLL (log2m) in startree index. [\[link\]](https://github.com/apache/pinot/pull/13835) [\[link\]](https://docs.pinot.apache.org/basics/indexing/star-tree-index#aggregation) * Added Lookup Join strategy as a hint to improve performance when the right table in the JOIN is a Dimension table. [\[link\]](https://docs.pinot.apache.org/users/user-guide-query/multi-stage-query/optimizing-joins#lookup-join) [\[link\]](https://github.com/apache/pinot/pull/13966) * Introduced a more detailed query execution plan that also provides detailed information about the physical operators being used in the multi-stage query engine. [\[link\]](https://github.com/apache/pinot/pull/13733) [\[link\]](https://docs.pinot.apache.org/users/user-guide-query/multi-stage-query/understanding-multi-stage-explain) ## StarTree Cloud ### StarTree extensions for Apache Pinot * Added flexibility for users to provide the number of retries in case of failures when atomic sync is configured. This allows users to also upgrade their StarTree Cloud environment while data is being ingested. * Improved performance while ingesting data from Delta Lake or using SegmentImportTask by changing the default value of parameter "push.mode" to "metadata". * Added several improvements to Delta Lake 3.0 connector to support Delta Protocol Reader version 3 and Writer version 7. * Added support for ingesting data from DynamoDB CDC streams using the DynamoDB message decoder. [\[link\]](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/dynamodb) * Added native support for ingesting Prometheus-formatted metrics data into tables in StarTree Cloud. Users can now leverage the price/performance of StarTree Cloud for their metrics solution built on Prometheus. [\[link\]](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/prometheus) * Added the ability to merge smaller segments into large segments to improve performance in an upsert enabled table leveraging SegmentRefreshTask. [\[link\]](/corecapabilities/manage-data/segment-refresh-task) * Added TTL for metadata and deleted keys for upsert-enabled tables using Offheap upsert. This will improve scalability and manageability by reducing the size of managed keys and metadata. * Added data consistency guarantees when running queries while upserts are being processed. Sometimes the result set would not be consistent without this guarantee. [\[link\]](https://docs.pinot.apache.org/basics/data-import/upsert#data-consistency-when-queries-and-upserts-happen-concurrently) * Improved the server restart time, when needed, by preloading a snapshot of primary keys in an upsert enabled table. In absence of this feature, the primary keys will be built during the server restart, resulting in long restart times. [\[link\]](/corecapabilities/manage-data/offheap-upsert) * Improved scalability and reliability for Dedup by moving the metadata from on-heap implementation to off-heap implementation, similar to off-heap upsert. * Added several health checks to ensure tables in StarTree Cloud are always optimized for best performance. The list of health checks includes a check to ensure no table in production is running with a single replica of data. [\[link\]](/corecapabilities/observability/health-checks) ### Data Manager * Added the ability for users to modify the schema and table configuration even after a table has been created, enabling greater flexibility. Users can optimize their table for better performance using Data Manager. * Added enhanced validation to ensure accurate field type and data type configurations during table creation, reducing errors and improving data integrity. ### ThirdEye * Improved onboarding with the new alert creation flow. Creating alerts is now simpler and faster. Creating dimension exploration alerts is now possible in no-code. * Added new Impact dashboard. This dashboard provides managers and alert owners a clear and intuitive understanding of the health and performance of all monitored metrics. # Release Version 0.10.1: February 2025 Source: https://docs.startree.ai/reference/startree-release-notes/0.10.1 ## Key Product Features ### Pauseless Ingestion *(Beta)* This feature makes real-time ingestion in Pinot truly pauseless, thus improving the data freshness even further. In the original design, there was a small pause in real-time ingestion (partition level) when the consuming segment was being committed. Pauseless consumption removes this constraint, thereby allowing to resume the process of consuming the next segment while the current consuming segment is being committed. **Current caveats:** Dedup and partial upserts are not currently supported (support for these is planned in a future release). ### Alter Table Task (ATT / SRT) *(Beta)* The StarTree [Alter Table Task](/corecapabilities/manage-data/alter-table-task) is an improved alternative to the [Segment Refresh Task](/corecapabilities/manage-data/segment-refresh-task), designed for more efficient, batch-wise segment refreshes. Unlike the Segment Refresh Task, where the map phase and segment generation are local, leading to inefficiencies due to redundant segment creation and merging, the alter table task optimizes this process by writing all mapper output files to a single location, allowing segments to be generated directly from the output. This eliminates the extra merge step, reducing overhead and improving performance. The task follows a structured Map → Reduce → Upload workflow, ensuring better resource utilization, fault tolerance, and deep store integration. With built-in retries that dynamically adjust parameters, it provides a more scalable and reliable approach for large table refreshes. ### CTE Reuse *(Alpha/Experimental)* In the Multi-Stage Query Engine (MSQE), there are several cases in which the same table is scanned twice or the same join is executed again. Some of these parts can be quite expensive (eg: table scan) and therefore doubles the cost of the query. This feature allows parts of the query subtree to be reused, resulting in optimized query execution. More details in [this PR](https://github.com/apache/pinot/issues/14196). This feature needs a hint to enable. Described in [this PR](https://github.com/apache/pinot/pull/14507) **Example:** ``` SET useSpools = true; select * from userAttributes as a1 join userGroups as a2 on a1.userUUID = a2.userUUID join userAttributes as a3 on a1.userUUID = a3.userUUID limit 10 ``` ### Snapshot support for offheap Dedup in real-time tables Improved the offheap deduplication feature in Pinot real-time tables by adding support for snapshots. This refers to the metadata snapshot that can be preloaded into the server, thus dramatically improving the restart time. These snapshots are handled by a minion task called *DedupSnapshotCreationTask*. For more details, see the [documentation](/corecapabilities/manage-data/offheap-dedup). ## New Stability Features * Group by trimming in MSQE: Parity with v1 engine regarding group by trimming support (needs to be explicitly configured in the query). * Improve performance of MSQE in the presence of large IN clause. * Added support for using the failure detector module in broker with the Multi-Stage Engine (MSE). Previously this was only enabled for the V1 engine. * Rebalance Hardening: * Added a dry-run mode with summary that provides important stats such as total segments to be moved, number of servers and so on (see [this PR](https://github.com/apache/pinot/pull/15050) for details). * Added several pre-checks (eg: minimizeDataMovement, needsReload). * Correctly handle the case where the number of partitions in the input stream (eg: Kafka) have been reduced since the real-time table was created. * Trigger an alert if the streaming consumer is not updating the ingestion delay metric correctly. This will help detect data freshness issues proactively. * Fixed broken alerts in Cluster Health Check (eg: Query Health) * Clean up verbose logging in case of virtual columns * Bug fixes: * Ensuring segments in the deep store are cleaned up properly (post retention). * Prevent creating tables with a replication factor > number of available servers. This was allowed this earlier, which resulted in ingestion pause issues. * Remove snapshot lock on upsert tables which otherwise might cause ingestion issues. * Cleanup stale state of dimension tables on servers post deletion * Use Ideal State (IS) for computing segment list in consistent push (previously it relied on segment Zookeeper metadata which can get stale during failures). ## **Now() Function in Multi-Stage Engine Returns TIMESTAMP** > [!WARNING] > > As part of our ongoing efforts to align Pinot SQL more closely with the widely adopted Postgres SQL standard, the return type for the NOW() function in the Multi-Stage Engine (MSE) has been upgraded from LONG/BIGINT to TIMESTAMP. > > This change delivers two key benefits: > > * Postgres Compatibility: You can expect NOW() to behave just as it does in Postgres, making your SQL experience more seamless across systems. > * Enhanced Functionality: NOW() can be directly used with functions like TIMESTAMPADD, and anywhere a TIMESTAMP type is expected in the multi-stage engine, improving usability and reducing friction. > > **What does this mean for you?** > > If you have existing queries that perform arithmetic with NOW() as a LONG/BIGINT, you’ll want to update them to use CAST() for compatibility. For example: > > * `CAST(NOW() AS BIGINT) - 86400000` > * `WHERE longCol <= CAST(NOW() AS BIGINT)` > > **Note:** As a matter of convenience, the MSE engine will automatically add type coercion (implicit cast) rules for TIMESTAMP / LONG so that the user doesn't have to add such cast functions explicitly. ## Default Configuration Flags for Pinot | Configuration | Old Value | New Value | Description | Impact | | - | - | - | - | - | | `Server starter Freshness checker` | Disabled Timeout = 10 minutes Freshness threshold = 10 seconds | Enabled Timeout = 70 minutes Freshness threshold = 1 minute | Enable the [freshness checker](https://docs.pinot.apache.org/operators/operating-pinot/server-startup-status-checkers#freshness-based-segment-checker) by default. The server will wait for the specified timeout for it to catch up with the corresponding real-time stream so that freshness is below a certain threshold. | This will avoid servers to start serving queries before they’re fully caught up and avoid stale results. One caveat is this will lead to longer server restart time. | | `Server storage capacity (to pause ingestion)` | \< None > | 95% of server storage capacity | Block ingestion for all relevant tables when a given server in the tenant reaches 95% storage capacity. For real-time tables this means pausing the ingestion. For offline tables, this means stop scheduling new ingestion tasks. **Note:** This is a safeguard to protect the cluster. The user should be already alerted at this point (disk full threshold of 90%). | Reduces the bytes written to the ZK log, especially useful in clusters where a large number of segments exist. This in turn, reduces the load on Zookeeper. | | `V4 Forward Index` | rawIndexWriterVersion = 2 | rawIndexWriterVersion = 4 | TEnable v4 forward index by default which is more scalable. Earlier version (v2) had a hard limit on the number of documents per segment. | Prevent segment build failures (eg: out of bounds exception) especially in cases where there are a lot of values per segment. In addition, V4 is also more memory efficient (Direct Memory). | | `Index rebuild throttle (Server level)` | \< None > | Index throttle: 25% CPU cores StarTree Index throttle: 1 | Control how many indexes we can build in parallel on the server (eg: during startup). There are different limits imposed for the StarTree index as compared to other indexes. **Note:** All the configs related to segment level throttling can be changed dynamically by updating Cluster config. [Documentation](https://docs.pinot.apache.org/operators/tutorials/segment-operations-throttling) | These limits will help avoid excessive load on the Pinot server during segment rebuilds. One caveat is this will lead to a longer server restart time. | | `Max parallel reloads (Server level)` | 1 | Max refresh threads: Max(10, 25% CPU cores) | Controls how many parallel segment reloads can be done on a given server. | This limit has been raised, which will speed up the total segment reload time. However, also added limits to how fast indexes can be built during reload. In general, with all these settings, server restart time will be longer. | | `Segment download throttle (Server level)` | \< None > | 25% of CPU cores | Controls how many segments can be downloaded in parallel on a given server. Works in conjunction with the existing table level throttle config (pinot.server.instance.table. level.max.parallel. segment.downloads). **Note:** All the configs related to segment level throttling can be changed dynamically by updating Cluster config. [Documentation](https://docs.pinot.apache.org/operators/tutorials/segment-operations-throttling) | This helps prevent an excessive number of segments being downloaded on a given server, thus preventing failures. One caveat is this will lead to a longer server restart time. | | `MSE thread throttling` | \< None > | pinot.beta.multistage.engine.max.server.query.threads = 1000 | This config controls the number of threads that can be spawned on a Pinot server for executing MSE queries. | This prevents a server from crashing in case an expensive MSE query ends up spawning too many threads. | ## Zookeeper Configuration Changes | Configuration | Old Value | New Value | Description | Impact | | - | - | - | - | - | | `Limit state transitions per instance ZK` | 100000 | 1000 | Whenever a Helix state transition is sent to a server, writes are performed on ZooKeeper (current state Znode update and new state transition Znode). This config limits the number of state transitions sent to any instance at once, which helps limit the number of state transition Znodes and current state updates at any given time. | Helps throttle the writes to ZooKeeper during operations on tables with a large number of segments. This reduces the load on ZooKeeper and hence reduces session timeouts. | | `Compression threshold` | 1 MB | 30 KB | The size threshold beyond which Znodes will be compressed before being written to ZK. | Reduces the bytes written to the ZK log, especially useful in clusters where a large number of segments exist. This in turn, reduces the load on Zookeeper. | | `ZK Session timeout (ZK Client)` | 30 seconds | 300 seconds | The period after which a client will timeout when establishing a session with ZK. | ZK client session timeouts can be very expensive. This increase in timeout will help avoid that, especially when Zookeeper is under heavy load. | | `PV Size increase` | 100 GB | 200 GB | The disk attached with every Zookeeper instance. | A larger disk size will allow more room to grow ZooKeeper logs, and prevent outages related to disk failures. | | `jute.maxbuffer` | 10 MB | 60 MB | This will help in cases where there are a large number of segments and to prevent ZK client exceptions. | No negative impact - it will help in use cases with a large number of segments. For example, in the case of tiered storage. | ### Default Configuration Flags for Cluster * publishNotReadyAddresses is set to True by default. This helps address DNS resolution issues during restarts/upgrades. # Release Version 0.11.0: July 2025 Source: https://docs.startree.ai/reference/startree-release-notes/0.11.0 # Summary This release brings critical enhancements across Operational cost savings, query performance, system observability, and operational resilience for both StarTree Data Platform and Cloud. ### **Highlights** * **Graviton GA**: We are announcing full Support for the Graviton line of CPUs for our customers in AWS for Pinot components. These SKUs are typically 15-20% cheaper. * **AZ aware Kafka consumer**: Users can now configure AZ aware Kafka consumer for optimizing data transfer costs across zones. Depending on usage, this can bring about 50-75% reduction in cross AZ data transfer costs. * **Calcite Upgrade (v1.39.0):** Improves MSE query performance, especially with large IN clauses. * **Alter Table Task (ATT) GA:** Now production-ready, replacing SRT for efficient table updates; includes deep store sync (Alpha) and direct tiered storage uploads. * **Data Portal** GA: Data Portal is now GA offering a unified interface that brings together Apache Pinot OSS tools and enterprise-grade features, making it easier than ever to manage, query, and monitor Pinot clusters — all from a single place. * **Rebalance Enhancements:** Smarter defaults, risk checks, reduced data movement, and improved UI and observability. * **Snowflake Connector:** Adds secure key-based authentication for ingestion. ### **Key Improvements** * **MSE Logging & Stats:** Better error diagnostics, reduced noise, and stats on failed queries. * **Pinot Proxy Optimization:** Lower CPU usage with improved broker routing. * **Segment Import Task (SIT):** Fixes data loss edge cases, optimizes memory use. * **GRPC & Memory Handling:** Adds throttling and auto-recovery to prevent OOM crashes. * **Faster Cleanup & Validations:** Batched table deletions, better error messaging, and safer ingestion setups. ### **Stability Fixes** * Fixes around Kinesis ingestion, stream partition handling, table config updates without minions, and better JSON responses for edge queries. * Improved MSE thread management reduces contention and enhances throughput. * Additional validation (e.g., bloom filter on boolean types) and safety checks across ingestion and query paths. ### **Observability & Monitoring** * **New Dashboards:** For upserts, rebalances, SRT/ATT tasks, and MSE query load. * **Expanded Metrics & Alerts:** More visibility into ingestion errors, ZK node size, broker failures, and query health. * **Alert Refinements:** Better thresholds for CPU, GC, and system anomalies. Following sections go through this in detail # StarTree Data ### **New Features** * **AZ aware Kafka consumer support (Beta)** Users can now configure AZ aware Kafka consumers in their environment. This assumes server pools have been setup in the environment. This can be enabled by specifying this additional property inside the ingestion part of table config: ```javascript theme={null} "client.rack": "${CLOUD_AZ}" ``` where, *CLOUD\_AZ* is an environment variable that's automatically configured on each Pinot server. On startup, each server will automatically get the right zone value for this environment variable. You can read more about this [in this doc](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/realtime/az-aware-kafka). * **Calcite upgrade to 1.39.0** Upgraded the internal Calcite dependency version to 1.39.0. The major changes to Calcite are documented as part of [this PR](https://github.com/apache/pinot/pull/15263). The primary motivation is to address performance issue in the Multi-Stage Engine (MSE) in the presence of large IN clause (details in this [issue](https://github.com/apache/pinot/issues/13617)). * **Correlation ID for Query Tracking**\ Introduced support for correlation ID, enabling better tracking and observability across multi-stage engine (MSE) queries. This helps users debug issues with MSE easily by collecting all relevant logs from brokers and servers in one place for a given query. Previously, it was difficult to track down relevant logs for a given query in question, making it harder to identify the root cause of failures / timeouts. Ability to collect all the relevant logs for a given correlation ID improves this debugging process.  By default, Pinot assigns a random correlation ID to the query as soon as the broker receives it, but clients can provide their ID by using the clientQueryId query option as shown below ``` set clientQueryId='myCustomCid'; select *  from userAttributes limit 10 ``` For more details, please [refer to this doc](https://docs.pinot.apache.org/users/user-guide-query/query-correlation-id). * **asof join** support Added support for executing asof joins in Pinot which enables user to join 2 Pinot tables based on timestamp values (specified in match condition) that follow each other, precede each other, or match exactly. For more details, please [refer to this doc](https://docs.pinot.apache.org/users/user-guide-query/query-syntax/joins#asof-join). * **Deep Store Sync via Alter Table Task (Alpha Release)**\ Introduced a new mode in Alter Table Task (ATT) to rebuild any segments in the deep store that are out of sync with the corresponding local segments in Pinot Servers, following a table config or schema change. This runs as a background task and hence relieves the Pinot servers from doing this during a restart / upgrade operation, thus reducing overall turnaround time. Note: This background task is optional and intended in scenarios wherein we want to reduce the overhead on Pinot servers. Configure the ATT task config as given below on the table config to enable it ```json theme={null} "task": {   "taskTypeConfigsMap": {     "StarTreeAlterTableTask": {       "reloadOnly": "true",       "forceReload": "true"     }   } } ``` > [!NOTE] > > **This is an experimental mode and not GA yet.** For more details, [refer to this doc](https://docs.startree.ai/corecapabilities/manage-data/deep-store-sync). * **Rebalance Phase 2 Enhancements** * *minimizeDataMovement* option added to Rebalance parameters to minimize network overhead ([doc](https://docs.pinot.apache.org/operators/operating-pinot/rebalance/rebalance-servers#minimal-data-movement-algorithm)).  * UI enhancements * Basic and advanced options * Highlight risky rebalance parameters * "Dry Run", Status tracking buttons * New defaults:  * reassignInstances: true,  * includeConsuming: true,  * minAvailableReplicas: -1 * Pre-check improvements * Display status of PASS / WARN / FAIL and a message * Check Disk utilization during and after a rebalance and flag servers that might be at risk (default threshold of 90%) * Flag risky & missed rebalance parameters * *needReload* pre-check now checks all servers * Summary improvements: * Always on * Add tag level summary * Add summary about the consuming segments and track the top 10 based on most offsets to catch up and oldest segment creation time * Observability improvements * Always include the jobId and table name for easier debuggability * New segment throttle metrics for threshold and current count ([doc](https://docs.pinot.apache.org/operators/tutorials/segment-operations-throttling#metrics)) * Improve rebalance progress stats tracking in ZK for better observability into how the rebalance is progressing * Update state model to prioritize DROPPED over other states, followed by OFFLINE. Modify the state model to allow direct state transitions to DROPPED from ONLINE / CONSUMING rather than the two step process that happens today (this also reduces the number of state transitions overall for DROPPED) - we enabled the rollout of this only in STP though (in OSS users will need to set a config to update the state model) * **Key Based authentication support for Snowflake connector**\ Added key-pair authentication support when ingesting data from Snowflake (previously we only supported username and password). This provides a more secure way to connect and ingest data from Snowflake. Here’s a sample config to use this new feature: ```json theme={null} {  "sql.snowflake.account": "",  "sql.snowflake.user": "",  "sql.snowflake.privateKey": "BASE64_ENCODED_PRIVATE_KEY",  "sql.snowflake.privateKeyPassword": "BASE64_ENCODED_PASSWORD",  "sql.snowflake.db": "",  "sql.className": "ai.startree.connectors.sql.snowflake.SnowflakeConnector",  "sql.snowflake.schema": ""   ... } ``` * **Alter Table Task (ATT) - GA**\ With this release, we’re making the Alter Table Task (ATT) globally available for production. ATT will replace the older SRT (Segment Refresh Task) going ahead and is designed to be more efficient and scalable. More information on ATT can be found [in this doc.](https://docs.startree.ai/corecapabilities/manage-data/alter-table-task#alter-table-task) * **Support for altering table config with cloud tiered storage in an efficient manner**\ Users can now easily make changes to tables with cloud tiered storage (eg: add/remove indexes) in an efficient manner using SRT/ATT jobs. Previously, these jobs would generate new segments based on the specified change and upload to the deep store, which was then uploaded to the remote tier via the Pinot server. Now, this is done directly by the minion job, thus reducing server overhead. Required configs to enable this feature:\ **Cluster Config:** ```json theme={null} pinot.server.instance.table.data.manager.provider.class : "ai.startree.pinot.data.tier.table.StarTreeDefaultTableDataManagerProvider" ``` **Task Config:** ```json theme={null} "directUploadToTier": "true" ``` should be added in the task Configs list for the table, by default it is false. For more information, please refer [to this doc](https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/setup#minion-tasks) ### **Improvements** * **Indexing improvements (Text, Json)** We've made the following improvements: * **Text Index (Lucene)**: TEXT\_MATCH is now extended to support Lucene query parser options at query time. For example, now we're able to do the following ```javascript theme={null} SELECT * FROM myTable WHERE TEXT_MATCH(myCol, 'term1 term2', 'parser=STANDARD, defaultOperator=AND') ``` For more information, please [refer to the documentation](https://docs.startree.ai/corecapabilities/manage-data/indexes/text-index#text-match-query-options). * **Composite Json index**: Introduced a composite JSON index to accelerate complex JSON queries by enabling efficient filtering and retrieval of nested fields. This now offers greater control in path selection / inclusion, configure internal range or text index. For more information, [please refer to the documentation](https://docs.startree.ai/corecapabilities/manage-data/indexes/composite-json-index). * **Add stats on errors in MSE**  The new MSE engine now shows detailed query stats even when the query fails (which was not true previously). This helps in debugging MSE query errors / timeouts and pinpoint where the failure occurred. Please see [this doc for more details](https://docs.pinot.apache.org/users/user-guide-query/multi-stage-query/understanding-stage-stats) * **Ease of schema evolution**  In [this PR](https://github.com/apache/pinot/pull/15350) we've made it easier for users to evolve the Pinot schema on a live Pinot table. Previously, schema changes needed a reload operation to be complete before the changes are reflected correctly in the query results. Previously, in some cases users also noticed query failures during this transition period. Now, with the on the fly virtual column feature, users can safely run queries as soon as the schema change is done. The new behaviour is captured in the table below: | **Scenario** | **Before** | **After** | | :- | :- | :- | | Rolling addition of a new column (newCol) | Queries fail with schema mismatch | Queries succeed; newCol is exposed as a virtual column | | Add a new column without segment reload | Empty results due to server‑side pruning | Queries succeed and return all columns, with newCol filled with default | | Queries over existing columns only | No change | No change | | Queries referencing a partially loaded column | Merge errors or empty responses | Queries succeed with consistent results, including the new virtual column | * **Merge and compaction for Upsert enabled tables (via SegmentRefreshTask)** The SegmentRefreshTask (SRT) has now been improved to support compaction and merge of segments in a real-time upsert enabled table. This included some bug fixes, along with guardrails for CPU overhead and optimizations to speed up the process. For more details, please refer to [this doc](https://docs.startree.ai/corecapabilities/manage-data/segment-refresh-task#upsert-table). * **Reducing noisy logs in MSE on error**    MSE engine will now reduce the log volume during query errors by reclassifying certain errors and adjusting how query errors are logged, minimizing redundant stack traces. Previously this caused a lot of logs to be generated during query issues, leading to excessive disk usage. * **Making colocated joins more efficient** We can now join a left table and right table with different partitions in a colocated manner (0 data shuffle across nodes). This requires the tables to be setup in a certain way (i.e. using same partition function upstream and tables must use the [same instance assignment strategy](https://docs.pinot.apache.org/basics/releases/1.1.0#preconfiguration-based-mirror-instance-assignment-11578)). More information on this can be found [in this PR](https://github.com/apache/pinot/pull/15764). * **Pinot Proxy CPU optimization**\ Pinot Proxy allows users to send queries easily to multiple broker tenants. The proxy will determine the relevant set of brokers for the requested table(s) and route accordingly. More information on Pinot proxy can be [found here](https://docs.startree.ai/corecapabilities/query_data/query_interfaces/pinot-proxy#using-the-pinot-proxy). Earlier, this was adding some CPU overhead which was not ideal.  In this release, we’ve added a bunch of optimizations to reduce the CPU overhead when queries are routed via the Pinot proxy. This includes the ability to pass a list of table names in the header (previously this was done by parsing the query), preferring the local broker and other optimizations. * **Always on Upsert related background tasks**\ Automatic task scheduling for upsert-enabled tables with built-in support for: * UpsertSnapshotCreationTask * **Improved Logging for catching bad queries**\ Queries are now logged as they begin execution—helping ops teams detect and cancel problematic queries in real-time before they degrade the cluster. Earlier this was done only on query completion which was insufficient. * **SegmentImportTask (SIT) Improvements**\ In StarTree Cloud, users can use the SIT task to move data between realtime and offline table (hybrid mode) after a certain age. We made a bunch of improvements to SIT in this release including: * Data loss bug fix: There was a scenario in which data from the realtime table can be deleted (due to retention) before its moved to the offline table. This is now fixed.  * Segment size improvements: Improved the algorithm to create appropriately sized segments in the offline table.  * Minion OOM issue: Minion tasks now initialize the record readers in a lazy manner, thus preventing all the segments to be loaded at once - which in turn causes memory overhead and may lead to OOM failures. * Improved runbook related to SIT.    SIT docs can be [found here](https://docs.startree.ai/corecapabilities/manage-data/segment-import-task#segment-import-task). * **GRPC & Trino Error Handling Enhancements**\ Added support for better error feedback when segments are unavailable during GRPC or Trino-based query execution. * **Improved Error Messages for Groovy Transform Failures**\ Enhanced visibility into Groovy-based transform failures during table creation flows with clearer error logging. * **Optimized Table Deletion (including deep store)**\ Optimized table deletion by deleting segment files in batches (which was done sequentially before). * **Create an alert for missing upsert snapshots**\ Created alert for missing upsert snapshot config. This will help prevent scenarios wherein server goes into a crash loop because the snapshots were stale and the disk wasn’t enough for the table to generate the snapshots. This is tracked using new metric:\ CLUSTER\_HEALTH\_TASK\_UPSERT\_PREBUILT\_SNAPSHOT\_DISABLED\_TABLES\_COUNT ### **Bug Fixes & Stability** * **Fix for Table Config Updates in the absence of minion pod**\ Prevented table config update failures when no minion pod is up, by avoiding minion dependency for unrelated config changes. * **Comprehensive clean up of historical segments in deep store**\ Fixed issue where .tar.gz segments created via SRT tasks were skipped by the Retention Manager. * **Better validation for FileIngestion mode**\ Disallowed switching between sync and append ingestion modes post table creation to ensure state consistency. * **Direct Memory OOM Auto-Recovery**\ Enhanced Pinot servers to recover from out-of-memory (OOM) situations for direct memory without needing manual pod restarts. * **Upsert Snapshot Task Monitoring**\ Added failure/success tracking and alerting for snapshot creation jobs not running within 72 hours. * **Query OOM Risk Visibility**\ Log queries that generate an excessive number of groups beyond a configurable threshold - as soon as it happens (instead of waiting for the query to complete). This helps in triaging failures in the presence of such expensive queries. * **Bloom filter validation**\ Prevent adding bloom filter on boolean type column at the time of table creation (earlier this caused failures during segment load). * **Improved handling of stream partition unavailability**\ Fixed an issue in cases when a stream partition is no longer available (reached end of life), trying to create a consuming segment for it can cause NPE. Earlier this caused ingestion from Kinesis to come to a halt when the corresponding partition/shard was unavailable. * **Fix for ingestion issues from Kinesis during resharding + pause and resume**\ Fixed issue where ingestion completely stopped from Kinesis when pause/resume was invoked post resharding of Kinesis stream . * **Fix the JSON response payload for always false queries**\ Always false queries eg: (select … where 1=0) now return the correct empty JSON response, something like this: ```json theme={null} "resultTable": { "dataSchema": null, "rows": [] } ``` Previously, no response was returned which caused issues in the invoking application * **Improved Thread Management in MSE**  Improved thread handling by eliminating unnecessary blocking in query submission and GRPC execution paths. The updated design simplifies the code and reduces reliance on intermediate thread pools, resulting in better performance and lower thread contention in production environments. With this, MSE servers can now use GRPC in direct mode, which is more efficient.  * **Prevent server OOM due to GRPC queries**  Introduced a memory based throttling mechanism for GRPC requests. When a GRPC query server's direct memory usage is high and exceeds a certain threshold, it will start rejecting further requests with RESOURCE\_EXHAUSTED status code until memory usage goes below the threshold.  ### Breaking Changes * New server instances will now be “untagged” In the past, when new servers were added to an existing cluster, they were automatically tagged with *DefaultTenant*, which caused issues in multi-tenant configurations. Starting 0.11.0, new servers will need to be manually tagged to assign them to a particular tenant. * Enable logical type in Avro schema by default This makes parsing Avro schema easier (eg: in case of BigDecimal which is encoded as a byte array). However, this could cause an issue if the user is trying to encode custom data in the raw format.  # StarTree Data Portal We’re excited to announce the StarTree Data Portal is now available! This unified interface brings together Apache Pinot OSS UI tools and enterprise-grade features to make managing, querying, and monitoring Pinot clusters easier than ever. ### Highlights * **Table Creation & Data Ingestion** – Guided workflows for creating real-time and offline tables, with built-in data validation. * **Query Console** – Interactive SQL editor with live data preview, table browser, and performance insights. * **Cluster Health** – Real-time health dashboard on Cluster, Table, Server, Controller, Broker, Minion, etc. * **Security Manager** – Fine-grained RBAC for users, groups, and service accounts. * **Integrated OSS Pinot UI** – Access Cluster Manager, Zookeeper Browser, and Swagger API Explorer directly within the Data Portal. With this release, the Data Portal becomes your **one-stop solution for all things Pinot in StarTree Cloud**. You can access the StarTree Data Portal directly at: [https://dp.your-environment.startree.cloud](https://dp.your-environment.startree.cloud) # StarTree Cloud Platform ## **New Features** * **Graviton GA** We're now able to provision Graviton instances for Pinot components (Server, Broker, Controller, Minion). Graviton instances are roughly 10-20% more cost effective. * **First class support for workspaces (Private Preview)**\ We’re now able to support workspaces in a given Startree Cloud environment (SaaS or BYOC). This is currently in private preview and needs a manual configuration to enable. Workspaces provide logical isolation in the same environment. In other words - users of one workspace can only see and query tables in that workspace and nothing more. This is a good mechanism in case you want to separate your data across business units or workloads. ### **Improvements** * **JWT Handling**: Added expiry checks for JWTs. Earlier this was not honored leading to the session not being terminated post expiry. ## **Dashboards & Metrics** * **Grafana Updates**: Added new dashboards and visual panels, including: * Dashboard for capturing various broker level query errors * New upsert/dedup dashboard * New dashboard for capturing rebalance related metrics * Segment Refresh Task (SRT), Alter Table Task (ATT) metrics dashboard * Estimated total server MSE query threads on broker dashboard * Pauseless GC dashboards * Improved CPU usage panels for brokers, servers, controllers * **Metrics Enhancements**: * Added comprehensive metrics for broker-level query errors (e.g. queryErrorBrokerSegmentUnavailable, queryErrorAccessDenied, queryErrorJsonParsing, etc.), improving diagnosis and alerting. * Ingestion metrics * Rows filtered due to filter functions * Rows dropped due to transform or decode exceptions * Znode size monitoring * Introduced metrics and alerts for ZNode sizes under  * /IDEALSTATES/ * /EXTERNALVIEW/ * /PROPERTYSTORE/SEGMENTS/. * Fixed Prometheus metric name for Pinot server Netty metrics * Improved metrics related to Authentication performance ## **Alerts & Monitoring** * **New Alerts**: * Snapshot creation failure * Pinot Znode size near ZK buffer limit * Query error code rate * **Alert Improvements**: Updated CPU usage alert definitions * Increased duration for HighGCPercent and refined threshold # Release version 0.11.1: September 2025 Source: https://docs.startree.ai/reference/startree-release-notes/0.11.1 # Summary This release introduces **major new capabilities** for tenant operations, security, and auditing, along with important performance enhancements, guardrails, and stability improvements across Apache Pinot and StarTree components. ## **🚀 New Features** ### **Tenant Rebalance** You can now rebalance resources at the **tenant level**. * Applies automatically to all tables within a tenant. * Internally orchestrates table-level rebalances to provide a seamless experience. * Helps optimize resource distribution across large multi-tenant clusters. 📖 [Read the documentation](https://docs.pinot.apache.org/operators/operating-pinot/rebalance/rebalance-tenant) ### **Row-level RBAC** Row-level access control (RLAC) enables **fine-grained data security**. * Define row-level filters using **SQL expressions**. * Access is restricted to roles authorized for the corresponding filter. * Enforce per-user or per-team visibility rules without duplicating datasets. 📖 [Read the documentation](https://docs.startree.ai/corecapabilities/security/row-level-access-control) ### **Pinot Audit Logs** Pinot now supports **auditing of cluster operations**. * Capture audit events from **Controller** and **Broker** components. * Persist logs in Pinot’s **deep store** for long-term archival. * Configurable options for logging **headers** and **payloads** to meet compliance and debugging needs. 📖 [Read the documentation](https://docs.startree.ai/corecapabilities/security/audit/pinot-audit-logs) ## **⚡ Improvements** ### **Text Index Pinning for Tiered Storage** Users can now configure a text index when using cloud tiered storage and be able to pin it locally for accelerating query performance. Read full documentation here: [https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/use-text-index](https://docs.startree.ai/corecapabilities/manage-data/set-up-tiered-storage/use-text-index) ### **Table Deletion & Creation** Enhanced validations and cleanup improve operational reliability: * **Deletion**: * Automatically removes Minion tasks that are not IN\_PROGRESS. * Fails gracefully with clear error messages if tasks are still running. * **Creation**: * Detects dangling task data from previous tables. * Provides actionable error messages guiding users to retry deletion or manually clean up. ### **Table Rebalance Manager** * Introduces a single manager to handle rebalance jobs across the cluster. * Prevents multiple jobs from running simultaneously on the same table. * Tracks and reports progress of ongoing rebalances. *** ### **Pinot Server Restart Speedup** Server restart time is now **3x faster or more**: * Reuses **on-disk upsert metadata** instead of rebuilding from snapshots. * Typical restart time is now **minutes per server**, even with billions of primary keys. * Improves operational ease for **StarTree Upserts**. * **Note**: Requires clean shutdown. In case of unclean shutdown, servers fall back to snapshots. *** ## **🛠 Bug Fixes & Stability** ### **Pinot Guardrails** New safeguards to protect clusters against ingestion and query instability: * Block ingestion if primary key count exceeds thresholds. * Validate conflicting indexing configurations that can cause ingestion failures. * Validate multi-value column compatibility. * Add throttling to prevent CPU spikes during ingestion or compaction. * Prevent index build failures caused by malformed or bad data. *** ### **Stability Improvements** * Increased frequency of **Upsert snapshot tasks** for faster recovery. * Improved filesystem (FS) call resilience: **timeouts and retries** added for GCS, ADLS, and S3. * Changed default max length strategy: no more silent string truncation (previously caused ingestion failures with JSON columns). # Release version 0.12.0: February 2026 Source: https://docs.startree.ai/reference/startree-release-notes/0.12.0 # **StarTree Cloud highlights** ## **New Features** ### **Composite JSON index enhancements** Added support for the following indexes within the Composite JSON index * FST Index * Text index This enables users to configure the Composite Json index on a JSON column and configure these sub-indexes. Also added tiered storage support for these sub-indexes. For more details, please refer to the [documentation](https://docs.startree.ai/corecapabilities/manage-data/indexes/composite-json-index). ### **Segment Backfill & Purge** * **Segment Backfill Dry Run Mode**\ A new capability was added to preview backfill actions before execution. It allows us to see which segments will be purged and what data will be ingested without making actual changes, reducing the risk of data loss. * **Support Segment Purge for Upsert Tables**\ Segment purge tasks now work with upsert tables by marking rows as deleted, rather than removing them. ### **Consolidated Preload for Tiered Storage**  New mode to download all indexes into a single consolidated mmap file.\ Config: ``` preload.enable.index.consolidation (default: false) ``` Preload index maintains one mmap mapping per index per column per segment. Usually Linux kernel has a limit of 65536, when the amount of segments and configured pinned indexes grows close to the number, this feature could be turned on to reduce the mappings to one mapping per segment. Consolidated preload index puts all preloaded indexes in one file, and slices the buffers of different indexes on one mmap buffer. The config is dynamic, the preloaded index would be fetched from the S3 after the change on `preload.enable.index.consolidation` (either change to false or to true) ### **New Table Storage Usage API** Unexpected growth in table storage can increase infrastructure and object storage costs. Pinot stores table data across multiple locations, such as server disk, deep store, and remote object stores used by tiered storage. The Table Storage Usage API reports table size with a breakdown by storage location and highlights mismatches between the expected size and the actual size. For more info, please see the [doc](https://docs.startree.ai/corecapabilities/manage-data/table-storage-usage). ### **Tiered Storage: Ease of table evolution** Evolving the table schema and config requires running of minion tasks (SegmentRefreshTask or AlterTableTask). This release makes it easy to run these tasks and directly update the remote tier (without going through Pinot servers). This will be a lot more efficient and faster to make changes. ### **Embedded Schema Extraction for Parquet Files** While creating a table, Data Portal now automatically extracts schema from Parquet file metadata, eliminating manual schema inference. Supports primitive types, logical type annotations, and timestamp handling with a new priority system: Provided → Embedded → Inferred schemas. ## **Improvements** ### **Minion Task Execution enhancements & Guardrails** * **Adaptive Disk Usage**\ Stops mapper phase when disk usage exceeds maxDiskUsagePercentage (default: 85%). This prevents disk full failures during large ingestion jobs. * **Faster reduce phase task generation in Alter Table Task** Added support to parallelize metadata file Downloads in Alter Table Task Reduce phase. This was single threaded before and caused a bottleneck. This is controlled by the table level config flag (`numMetadataFileDownloadThreads`, default: 4). * **Default Max Concurrent Tasks per Minion Based on Memory**\ Automatically derives safe concurrency levels from system memory. This is crucial in preventing out of memory errors. * **Soft File Count Limit for Delta Ingestion**\ Enforces a \~200k soft limit on file count to prevent excessive segment generation. * **Conflict prevention** Prevent conflicting scenarios such as running refresh / alter task during Delta ingestion or running SIT/Delta/SRT during an ongoing backfill task. * **Improve File Ingestion Task (FIT) Documentation, Defaults, and Validations**\ Includes new defaults for consistent push retries and validation of critical config fields. * **Cluster-Level Max Subtask Limit Enforcement**\ Ensures subtask count respects cluster safety thresholds. * **New Metrics** New metrics added to capture consistent push failure, skipped tasks due to conflict and improved task metric accuracy * **Added Tenant rebalance cancellation** - Added the ability to cancel tenant rebalance * **Allow manual or ad-hoc trigger of the controller's periodic task** - Added support for triggering controller tasks on demand (one-time execution), rather than requiring them to be scheduled ### **Performance & Stability** * **Controller Thread Pool Defaults**\ Sets bounded defaults (general executor: 1000, rebalance: 200) to avoid stability issues. * **Tiered Storage Enhancements** * **Cleanup dangling deep store sessions and stale reduce outputs**\ New configs: * `bufferDaysToPurgeOutputSegments` (default: 3 days) * `cleanupDanglingIntermediateFiles` (default: true) * **Avoid Prefetching Forward Index When JSON or Text Index Exists**\ Reduces unnecessary I/O for JSON\_EXTRACT\_INDEX and text\_match operations. ## **Bug Fixes** * Fix IndexOutOfBounds in Backfill for Empty Predicates * Improved backwards compatibility in dangling file cleanup * Fixes related to task limit validation, config overrides, inconsistent retry configs * \*\*Segment Import Task: \*\*Allows changing bucket duration (e.g., 1h → 6h) without data loss or duplicates. # **Apache Pinot (OSS) highlights** ## **New Features** * **Robust OOM Protection**\ Unified lifecycle/metadata for all query execution threads → safer cancellation, better resource tracking, improved observability. For more information, please see the [doc](https://docs.pinot.apache.org/operators/operating-pinot/oom-protection-using-automatic-query-killing) * **Apache Arrow Decoder (Experimental)**\ Adds initial Arrow-format ingestion support. This is intended for improving ingestion efficiency by reducing the processing overhead. * **N-gram Filtering Index (Experimental)**\ Added support for Realtime n-gram index to pre-filter non-matching strings efficiently. * **Kafka Client Default Upgraded to Kafka 3** * **IP Address Functions**\ Added helper functions such as ipPrefix, ipSubnetMin, ipSubnetMax, etc. * **Array Manipulation Functions**\ Added helper functions to push elements to front/back for all primitive/string array types. ## **Improvements** * **Support MAP Type for Derived Columns During Reload** * **Partial Upserts Stability**\ Disable reload on consuming segments; force commit to avoid corruption. * **Segment Reload Failure Tracking**\ This release adds in-memory tracking for failed reloads. * **Automatic Rewrite of MIN/MAX/SUM on Long/String Types**\ Rewrites to type-correct variants to avoid precision loss. * **Star-tree Index Build Robustness** * Skip star-tree creation if index build fails * Roll back to existing index when updates fail * New metrics to track failures * **Startree MV Aggregation Support**\ The following functions: SUMMV, COUNTMV, AVGMV are now supported on multi-value columns * **Async Segment Refresh Message Processing**\ Enabled by default in StarTree Cloud. * **Audit logging filtering improvements** - [URL based filtering support while collecting audit log](https://github.com/apache/pinot/pull/16807) * **Perf Improvements in MSE (join optimization)** Improved query performance via optimizing hash function usage in the query planner ([https://github.com/apache/pinot/pull/16830](https://github.com/apache/pinot/pull/16830)) # Release version 0.14.0: March 2026 Source: https://docs.startree.ai/reference/startree-release-notes/0.14.0 # **Summary** This release focuses on making StarTree **more lakehouse-native, faster, and more reliable in production**. * **Lakehouse Integration:** Deeper support for Iceberg and Parquet (including complex types and improved readers) makes it easier to run analytics directly on data lakes. * **Performance & Query Execution:** Enhancements like replica-aware routing, broker-based execution, and improved caching reduce query latency and improve efficiency. * **Streaming & CDC:** Expanded Debezium support strengthens real-time ingestion pipelines. * **Observability & Reliability:** New metrics, retry mechanisms, and safeguards improve stability and operational visibility. **Bottom line:** Faster queries, better lakehouse integration, and stronger production readiness. # **StarTree Cloud highlights** ## **New Features** ### **External Tables: Iceberg (Glue & S3 Tables)** StarTree Cloud can now query Apache Iceberg tables in place — no re-ingestion required. Iceberg tables registered in Nessie, AWS Glue or S3 Tables are queried directly using Pinot's query engine. For more details, see the [External Tables documentation](https://docs.startree.ai/corecapabilities/external-table/onboarding-data-portal). ### **External Tables: S3 Remote Parquet Files** StarTree Cloud can now register an S3 location as a Pinot table and query Parquet files in place without ingestion. For more details, see the [External Tables documentation](https://docs.startree.ai/corecapabilities/external-table/onboarding-data-portal). ### **Query Engine & Execution** Added 2 new features in the query engine for improving performance and isolation * Added **replica-group-aware query routing**, enabling better isolation and workload-aware query execution. [Read the docs](https://docs.startree.ai/corecapabilities/query_data/advanced_operations/replica-group-based-workload-isolation). * Introduced support for **brokers acting as intermediate stage workers** in distributed query execution. This helps in improving elasticity for multi-stage engine queries given that broker scalability is simpler as compared to the servers. The number of brokers used and which brokers are selected can be controlled via Helix tags and query options. For more information please [read the docs](https://docs.startree.ai/corecapabilities/query_data/advanced_operations/intermediate-stage-on-broker). ### **Faster segment maintenance with Columnar Segment Processing** Operations like reload, refresh, alter table, and import now use a new columnar processing engine that works column by column instead of row by row. This aligns the processing path with how data is physically stored in Pinot, which means significantly less CPU usage and lower memory pressure during maintenance windows. For teams running frequent schema changes or large scale segment operations, the impact is a noticeably faster turnaround with fewer resource spikes on your cluster. It also supports expression transformations, sorting, sanitization, and time column handling. ### **Minion Task Orchestration Framework** A new plan-based orchestration framework for managing complex, multi-step Minion task workflows. Task plans define sequences of tasks with dependencies, enabling safer and more predictable execution. Includes REST APIs for managing task plans and support for ad-hoc (one-time) task triggers. File Ingestion Task and the Segment Purge Task are onboarded on this framework. [Read the docs](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration). [Read the docs](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration). ### **Segment Purge Task Enhancements** The segment purge capability received major improvements across the board: * **Flexible purge criteria** — Supports column-level predicates and SQL query-based selectors for targeting rows to purge * **Dry run mode** — Preview which segments and rows are affected before executing, with verbose reporting of skipped segments * **Ad-hoc API** — Trigger purge tasks on demand without a scheduled task config * **Performance** — Replaced `DISTINCT` with `GROUP BY` in purge queries for significantly better performance on large tables ### **Composite JSON Index Enhancements** * Added **FST and Text index** as sub-indexes within the Composite JSON index, with tiered storage support * Added **partitioned inverted index** — the dictionary and posting lists are split into N sub-indexes by JSON path, reducing per-partition memory pressure and enabling read parallelism * Added **PromQL label query support** via the JSON index * `TEXT_MATCH` predicates now work correctly on consuming (real-time) segments ### **Query Analyzer** [Query Analyzer](https://docs.startree.ai/corecapabilities/ai/query-analyzer/overview) is a new AI-powered tool embedded in the Data Portal Query Console that helps you understand and optimize Apache Pinot multi-stage engine (MSE) queries. It analyzes your SQL query alongside table metadata, explain plans, and execution statistics to produce prioritized, evidence-backed optimization recommendations — without requiring deep Pinot expertise. This is a beta feature, disabled by default. Contact your StarTree account team to have it enabled for your environment. ### **Ingestion: New Decoder Support** * **Debezium CDC** — Added Debezium decoder support for all Confluent Schema Registry formats (JSON, Protobuf, Avro), enabling CDC pipelines to ingest directly into StarTree without custom transformation * **AWS Glue Schema Registry** — New stream decoder for Kafka topics encoded with the AWS Glue Schema Registry Avro wire format * **Kafka 3.0 Confluent Consumer** — Added `ConfluentKafkaConsumerFactory` for Kafka 3.x clients ### **Additional New Features** * **Tiered Storage Caching Integration** — Prefetched tiered storage segments are now stored in the Parquet disk cache, reducing repeated S3 access for hot segments * **Table Storage Usage API** — Added percentile-based segment size stats (p50, p90, p99) and verbose level controls * **Deep Store Stale Segment Detection** — New API `GET /tables/{tableName}/deepstoreStaleSegmentInfo` to estimate segments out of sync between servers and the deep store * **Cluster Cloner Controls** — Added options to conditionally skip deep store copying, table deletion, and schema/config change checks during cluster migration * **Delta Ingestion Reliability** — Upfront config validation, auto-default S3/GCS parameters, and correct failure propagation when files fail during segment generation * **Batch File Listing Optimizations** — Narrowed S3 listing scope using glob pattern prefixes; paginated PinotFS listing with filter push-down in the Preview API * **gRPC Authentication & Authorization** — RBAC enforcement for gRPC requests in broker access control * Introduced **Arrow-based Parquet column reader** for improved performance. * Added support for: * **Null handling in remote Parquet tables** * **Complex types (Struct, List)** in Parquet ingestion ## **Improvements** * **MSE Engine leaf server routing (default)**: Sets`pinot.broker.mse.use.leaf.server.for.intermediate.stage` to `true` by default. This will limit the fanout of the MSE intermediate stages and thus improve performance & efficiency of the queries. * **OOM Protection** — OOM resource accounting enabled by default with query kill. * **Upsert Stability** — Fixed SIGSEGV errors on upsert table startup/shutdown; prevented RocksDB state reuse for partial upsert tables * Improved query performance with **prefetchable forward index reader**. * Enhanced Parquet read efficiency with: * Page-level caching * Cache eviction mechanisms * **Audit Identity** — Added `StarTreeTokenResolver` for consistent identity attribution in audit logs * **Observability** — New metric for long segment replacement durations; Parquet page cache Prometheus metrics with per-layer hit/miss tracking; Preload cache size and buffer usage metrics are added. ## **Bug Fixes** * Purge task no longer uploads empty segments or throws exceptions at the max segment limit * Fixed Iceberg segment name normalization and segment name conflict on partition collisions * Fixed Delta ingestion silent success on file processing errors; fixed `preserveNullValues` not being honored consistently * Fixed timestamp column min/max values being incorrectly converted to milliseconds * Fixed Parquet reader errors: multi-value string `UnsupportedOperationException`, INT96 timestamp `ClassCastException`, and incorrect cache lookups * Fixed gRPC connection failures — now falls back to HTTP gracefully * Fixed Parquet disk cache write position bug on reload; disabled snapshot recovery by default * Fixed `GlobPrefixExtractor` URI scheme restoration for S3 and GCS paths * Fixed `filterColumnsForRow` dropping incomplete and sanitized flags in dedup processing * Fixed `InstancePoolsNReplicaGroupsCheck` health check failure for tables using `instancePartitionsMap` * Fixed controller crash-loop on restart when schemas are missing ## Breaking Changes **Window/Join Row Limit Error After Upgrade** Symptom:\ A query that worked before the upgrade now fails with one of:\   - `Cannot build in memory window cache for WINDOW operator, reach number of rows limit: …`\   - `Cannot build in memory hash table for HASH JOIN operator, reach number of rows limit: …`\   Resolution: Add one or both of these to your failing queries: *   `SET useLeafServerForIntermediateStage = false;` *   `SET maxRowsInWindow = ` / `SET maxRowsInJoin = ` **What changed ?**\ The default for`pinot.broker.mse.use.leaf.server.for.intermediate.stage` changed in this release (set to true by default). Starting 0.14 release, intermediate stages (including window and join) can run on fewer servers than before. Reusing leaf servers for intermediate stages avoids an extra network shuffle of data to other servers, which generally improves query throughput (and latency in some cases), and reduces cluster network load. However, in some cases, each server might process more rows and can hit the per-server row limit. Setting`useLeafServerForIntermediateStage` to false restores the old query fanout behavior; raising the row limits gives more headroom. # **Apache Pinot Highlights** Following changes are made to the open source Apache Pinot project with respect to the last release timeline. ## New Features * Enhancements to **multi-stage query engine (v2)**, including better stage execution and scalability. * Improved **query routing and server selection**, enabling more efficient distributed execution. * Expanded **JSON and text indexing capabilities** for richer query support. * Added improvements to **stream ingestion (Kafka/CDC)** for better handling of real-time pipelines. * New/updated **minion tasks and table management APIs** for maintenance workflows (e.g., purge, dedup). ## Improvements * Faster queries via improvements in **filter pushdown, segment pruning, and index utilization**. * Enhanced **upsert and dedup performance**, including better handling of edge cases. * Improved **Parquet/deep storage integration**, making lakehouse-style querying more efficient. * Better **metrics and observability** for query execution, segment lifecycle, and system health. * Improved **broker/server memory and resource usage**. * General **stability and scalability improvements** in distributed query execution. ## Bug Fixes * Fixed **query correctness issues** in joins, aggregations, and edge-case filters. * Resolved **upsert inconsistencies and state management issues**. * Fixed **stream ingestion edge cases** (offset handling, schema mismatches). * Addressed **segment loading, replacement, and metadata inconsistencies**. * Fixed **NPEs, race conditions, and build/dependency issues**. # Release version 0.15.0: April 2026 Source: https://docs.startree.ai/reference/startree-release-notes/0.15.0 # **Summary** Release `0.15.0` is centered on one major milestone: **StarTree Iceberg Tables are production ready for broader customer adoption, with Iceberg catalog support and enhanced operational reliability**. * **StarTree Iceberg Tables:** Stronger validation, better status/error surfacing, and safer execution paths make Iceberg based external tables easier to run in production at scale. * **Reliability first:** Ingestion and task orchestration improvements reduce failure blast radius and improve recoverability for long running jobs. * **Data correctness and type handling:** Substantial work landed around Parquet readers, complex type mapping, and column metadata behavior to improve query correctness. * **Operational visibility and security:** Better auth metrics, request level tracing, and storage/query observability improve day 2 operations. # **StarTree Cloud highlights** ## **New Features** ### **StarTree Iceberg Table is now GA** StarTree Iceberg Table take a major step forward in this release, with a strong focus on production readiness for Iceberg catalogs (including AWS Glue and S3 Tables) and in-place query workflows. * Expanded validation and guardrails for external table onboarding. * Better operational controls when handling real-world catalog and table configuration edge cases. * Improved ingestion task behavior and lifecycle handling for external table pipelines. * IAM role based onboarding support for external Iceberg tables We published an internal performance benchmark comparing StarTree's external table offering with a similar offering in Trino and ClickHouse. StarTree is orders of magnitude faster and infrastructure footprint is 15x lower in comparison. Read full benchmark here: [https://startree.ai/resources/iceberg-query-benchmark-vs-trino-vs-clickhouse/](https://startree.ai/resources/iceberg-query-benchmark-vs-trino-vs-clickhouse/) For onboarding and usage details, see the [External Tables documentation](https://docs.startree.ai/corecapabilities/external-table/onboarding-data-portal). **Better reliability and Control** StarTree Iceberg Table onboarding now has stronger execution and recovery behavior for production workloads: * More robust task execution flow for external table ingestion runs. * Clearer status and root-cause error reporting to speed up incident triage. * Optional handling for file-level failures to keep ingestion progressing when appropriate. * REST API to clear page caches on server and controller ### **Query and type-system improvements for External Table** This release improves correctness when querying external Parquet and complex schemas: * Better handling for complex and nested type paths. * Improvements in multi-value reads and metadata interpretation. * More consistent behavior for column statistics and predicate handling. ### **Operational observability and auth telemetry** StarTree Cloud now surfaces better operational signals across query, cache, and security paths: * New/expanded metrics for auth plugin behavior. * Better reader and cache observability for Parquet query paths. * Improved request-level tracing for external table ingestion components. ### **Query Log** Built-in query log system table: Every executed query (single-stage and multi-stage) is now asynchronously captured to a managed system\_query\_log Pinot table. This table captures all relevant fields from the query response which is useful to slice and dice historical queries by different dimensions (latency, memory used, docs scanned etc). For more information, please refer to [Query Logger documentation](https://docs.startree.ai/corecapabilities/query_data/advanced_operations/query-logger). ### **Replica scale down** Ability to scale down a replica group on a specified schedule is now available. Users can use this feature to reduce the footprint during periods of low activity (for instance in many cases, query volume may scale down in non business hours). ## **Improvements** * **Default Kafka AZ aware ingestion**: Real-time tables now ingest from Kafka in an AZ aware fashion by default, thus reducing cross-AZ traffic (and hence reducing cost). * **Task and orchestration maturity:** Better task metadata tracking, cleaner status transitions, and stronger scheduling/runtime controls. * **Cluster and ingestion safeguards:** Additional checks and throttling paths help reduce overload during ingestion and segment operations. * **Parquet path performance tuning:** Safer concurrent read behavior and improved cache/prefetch handling for heavy query workloads. * **Reload hardening**: Ability to simulate reload operation (dry-run) and cancel in-progress reloads. * **Min-max index on sorted raw columns**: Min-max indexes can now be built on sorted columns that do not use dictionary encoding, extending pruning coverage. * **Hardening of offline upserts** and support for off heap (persistent) upsert metadata. * **enforceConsumptionInOrder** enabled by default: Partial upsert and dedup tables now enforce in-order consumption by default, preventing correctness issues from out-of-order message delivery ## **Bug Fixes** * Fixed multiple correctness issues in Parquet and complex-type readers that could affect query output under edge schemas. * Fixed reliability issues in upsert/dedup and RocksDB lifecycle paths to reduce race conditions and memory leaks. * Fixed several ingestion and controller edge cases, including null handling, status transitions, and flaky runtime behaviors. * Fixed platform and packaging issues that impacted build stability and release workflows. # **Apache Pinot Highlights** This section describes [**Apache Pinot**](https://github.com/apache/pinot) (open source) changes in the baseline that ships with StarTree Cloud **0.15.0** compared with **0.14.0**. It does not describe StarTree-only extensions. ## New Features * **Vector Search** — Full multi-phase vector search: IVF\_FLAT, IVF\_PQ compressed ANN, filtered ANN, SQL radius queries, HNSW efSearch, IVF\_ON\_DISK, and adaptive planner. * **MSE Enhancements** — Native MSE planning for SUM/AVG over MV column. Broker pruning for non-partitioned leaf paths. Lookup join support in physical optimizer * **Upsert derivations:** Post-partial-upsert transforms support derived columns after partial upsert merge. * **Arrow Batch Ingestion** — ArrowRecordReader for ingesting Arrow IPC files.`pinot-arrow` is included in the standard Pinot binary bundle for Arrow-based columnar read paths. * **Distinct Early Termination** — Support early termination in combine operator for predictable query latency. * **AI Metadata on Schema/TableConfig** — description and tags fields on Schema, FieldSpec, and TableConfig for capturing enhanced user context. ## Improvements * **MSE Observability** — Upstream/downstream stage ID MDC fields for debugging. Improved error propagation and logging. * **Adaptive Routing** — Export adaptive routing stats as broker metrics. * **Auth ordering:** Authentication runs before query validation so unauthorized requests fail earlier. * **Large response handling:** Cursor-style response lifecycle cleanup moves toward the broker with a batch delete API (review upstream notes if you integrate with response stores). * **Minion observability:** Minion task generation logs carry correlation context (MDC) keyed by task id. * **Dynamic Server Thread Pool** — Thread pool size can be modified at runtime without restart ## Bug Fixes * **Upsert snapshots:** Bitmap optimizations during upsert snapshot and segment commit paths. * **Kafka consumption:** Partition-level consumer avoids incorrect re-seeking past `read_committed`-filtered batches. * **Record extraction:** Map serialization is canonicalized with more reliable preservation of primitive types in record extractors. * **FUNNEL\_COUNT NPE** — Fix when WHERE clause filters out all rows ## Backwards Incompatible Changes * **Native text and FST index removed** — Migrate to Lucene-based text index. * **Inverted index always created during segment generation** — Previously optional # Release version 0.16.0: July 2026 Source: https://docs.startree.ai/reference/startree-release-notes/0.16.0 # **Summary** Release `0.16.0` centers on scaling **StarTree External Tables to large, frequently-updated Iceberg datasets** and maturing the **StarTree Execution Engine** (native multi-stage query runtime), alongside broad reliability and observability hardening across ingestion, upsert/dedup, and task orchestration. * **External Tables at scale:** Segment grouping and native Iceberg deletion-vector support let external tables handle far larger file counts and row-level updates/deletes without re-ingestion. * **StarTree Execution Engine maturity:** New join types, full `BIG_DECIMAL` support, spill-to-disk for memory-intensive queries, and transparent fallback to the legacy engine close major functional gaps. * **Operational reliability:** The Minion Task Orchestration Framework is now on by default, audit logging is on by default, and a significant query-correctness fix eliminates silent wrong results under certain distributed execution plans. * **Apache Pinot OSS:** The open-source baseline continues to track upstream Apache Pinot; see [Apache Pinot Highlights](#apache-pinot-highlights) below for the OSS-only feature summary. # **StarTree Cloud highlights** ## **New Features** ### **External Tables: Segment Groups & Deletion Vectors bring Iceberg to scale** External Tables take a major step forward this release with two foundational capabilities that make large, frequently-updated Iceberg tables production-ready: * **Segment groups:** Many small Iceberg-derived segments are now grouped into a single logical unit end-to-end (controller, broker, and server), including for realtime tables, dramatically cutting cluster metadata overhead so external tables can scale to far larger file counts. * **Deletion vectors:** Pinot now natively applies Iceberg's row-level delete/update markers at query time, with atomic snapshot synchronization, server-side loading, and broker-side read consistency. The underlying index moved from JSON to a scalable Parquet-backed format with lazy point lookups. * **Broadened catalog and auth support:** Unity Catalog, Google Cloud Storage (via S3 interoperability), and OAuth2 client-credentials authentication for Iceberg REST catalogs. * **Iceberg schema evolution:** Field-ID/alias-based column resolution keeps the Pinot table schema automatically in sync as the source Iceberg table's schema evolves. For onboarding and usage details, see the [External Tables documentation](https://docs.startree.ai/corecapabilities/external-table/overview), including new pages on [Segment Groups](https://docs.startree.ai/corecapabilities/external-table/segment-groups) and [Deletion Vectors](https://docs.startree.ai/corecapabilities/external-table/deletion-vectors). ### **StarTree Execution Engine matures significantly** * Added sort-merge join and native lookup-join operators, plus full `BIG_DECIMAL` support (order-preserving encoding, casts, arithmetic, and aggregation — double-precision, not arbitrary-precision), closing major functional gaps with the legacy Java engine. * Added spill-to-disk for memory-intensive aggregations, sorts, and `GROUP BY` operations, backed by an auto-sized shared memory pool, so large queries degrade gracefully instead of failing with out-of-memory errors. * The engine now falls back transparently to the Java engine for a wider range of previously-unsupported plans and functions, improving coverage without hard failures. * Expanded SQL surface: `NOT IN` predicates, native array construction, automatic type widening for mixed-precision arithmetic, and multi-value column exchanges. * Reduced per-query engine overhead through internal performance work on the native execution path. This is an opt-in, beta capability — see [StarTree Execution Engine](https://docs.startree.ai/corecapabilities/query_data/query_languages/startree-engine) for how to enable it and its join-hint/spill-to-disk configuration. ### **Additional New Features** * **[Multi-Stage Materialized Views](https://docs.startree.ai/corecapabilities/query_data/materialized-views/multi-stage-materialized-views):** A new StarTree-specific minion task materializes multi-stage SQL — including joins — into a regular table on a schedule, queried directly (no automatic query rewrite). * **Kafka 4.0 connector:** A new Kafka 4.0 stream connector plugin ships alongside the existing Kafka 3.0 plugin. ## **Improvements** * **Native / MSE query engine:** Always-on per-operator metrics, cleaner error messages (no more internal stack traces leaking to clients), configurable network runtime worker threads, independently configurable broker-as-worker behavior, and engine wait times now bounded by the query's actual timeout instead of a fixed value. * **External Tables & segment groups:** Concurrent per-column prefetch, coalesced S3 range reads, local-disk caching of Parquet footers, parallel segment-group member loading, uniform tier/partition validation across segment-group members, and hardened lifecycle handling with failure isolation. * **Ingestion:** Delta Lake ingestion now supports globally consistent push (atomic table swap with reconciliation); the Minion Task Orchestration Framework (MTOF) is enabled by default at the cluster level with new orchestration observability metrics; ingestion now integrates with Pinot's OOM-protection safeguards. * **Upsert & Dedup:** Snapshot watermarks and locations are now persisted to ZooKeeper for durability across restarts; TTL cleanup is capped at the minion snapshot watermark to avoid removing data still needed for consistency. * **Segment & Indexing:** Parquet reader performance improvements (thread-local buffer reuse, typed array stores, reduced redundant page lookups), support for `DATA_PAGE_V2`-encoded Parquet files, a bounded/shared decoded dictionary cache with orthogonal TTL eviction, and faster composite JSON index remaps. * **Security & Observability:** Audit logging is now enabled by default on the controller and broker — see [Breaking Changes](#breaking-changes) below, since this turns on request-payload capture by default; the base runtime container image was hardened for FIPS compliance; Query Log (`system_query_log`) gained a configurable tenant, a `queryHash` column, and a default 200 GiB storage cap (up from 100 GiB) with compression on heavy string columns. * **Table Preview:** Clearer error messages on datasource connection failures, and catalog table sampling is now parallelized at the file and column level for large catalogs. ## **Bug Fixes** * **Query correctness:** Fixed a bug where hash-distributed exchanges in the native engine could silently return wrong results — the most significant correctness fix this release. Also fixed several native-engine crashes, panics, and query hangs under specific plan shapes. * **External Tables & Segment Groups:** Fixed several correctness and reliability issues in Parquet/Iceberg readers, deletion-vector reads now fail closed rather than silently proceeding when snapshot state can't be read, and fixed catalog/task config drift and orphaned-fragment cleanup. * **Upsert & Dedup:** Fixed upsert segment cleanup skipping over-replicated empty segments and a double-acquire race during reload batching. * **Security:** Fixed a Regular-Expression-Denial-of-Service (ReDoS) vulnerability in query normalization, and removed a vulnerable embedded-Jetty dependency (CVE-2026-2332). * **Platform:** Fixed client-facing APIs incorrectly returning HTTP 500 instead of 400 for client errors, and fixed inconsistent updates to concurrency-limit configuration. ## **Breaking Changes** * **Audit logging is now on by default, including request-payload capture.** Both controller and broker seed `pinot.audit.{controller,broker}.enabled=true` and `capture.request.payload.enabled=true` (plus default header/exclude-pattern/JWT-claim settings) on the controller's next restart after upgrade — on both new and existing clusters. Any key you've already set explicitly is left untouched. If you need audit logging (or payload capture specifically) to stay off, set `pinot.audit.controller.enabled=false` / `pinot.audit.broker.enabled=false` via `POST /cluster/configs` **before** restarting the controller on 0.16.0. See [Pinot Audit Logging](https://docs.startree.ai/corecapabilities/security/audit/pinot-audit-logs#0160-defaults-and-how-to-opt-out) for the full default set. * **Minion Task Orchestration Framework (MTOF) is now enabled by default** at the cluster level (`controller.startree.task.manager.enableTaskOrchestration`, previously off by default). Set it to `false` to disable orchestration cluster-wide, or set `forceLegacyTaskFlow=true` on a specific task type to route only that task type to the legacy flow. See [Minion Task Orchestration](https://docs.startree.ai/corecapabilities/ingestdata/adv-concepts/batch/minion-task-orchestration). * **`EXPLAIN` output now includes segment-level query plans by default.** `pinot.query.multistage.explain.include.segment.plan` now defaults to `true` (was `false`), which may affect tooling that parses `EXPLAIN` output. Use `EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR ` for the old logical-plan-only output, or set the property back to `false` explicitly. * **Default RocksDB `max_open_files` for upsert/dedup storage was raised from 1024 to 4096.** This is the shared off-heap upsert/dedup RocksDB store's file-handle cap — clusters with tight file-descriptor `ulimit`s should verify headroom before upgrading. Any explicitly-set value is unaffected. * **The `ls` alias for the `LsDeepStore` admin CLI subcommand (`bin/startree-pinot-admin.sh`) was removed, with no deprecation period** — only the full `LsDeepStore` command name resolves now. This affects BYOC/BYOK environments with direct CLI access; update any scripts calling `... ls ` to `... LsDeepStore ` before upgrading. * **Default MSE metrics mode changed:** `pinot.metrics.mse.mode` now defaults to `MSE` on brokers and `DUAL` on servers (OSS default: `SERVER`). Brokers stop emitting `pinot.server.*`-namespaced MSE metrics and start emitting `pinot.mse.*` instead; servers emit both namespaces. Set `pinot.metrics.mse.mode=SERVER` explicitly to keep the old broker behavior. # **Apache Pinot Highlights** This section describes [**Apache Pinot**](https://github.com/apache/pinot) (open source) changes in the baseline that ships with StarTree Cloud **0.16.0** compared with **0.15.0**. It does not describe StarTree-only extensions. ## New Features * **New SQL functions** — `OVERLAY`, `isPrivateIp`, `arraySliceLong`, unary `+`/`-` operators, and multi-column `CORRELATE_BY` support in `FUNNEL_COUNT`. * **New REST/admin APIs** — Query-syntax validation without execution, table-config validation/auto-tuning, and cron-based periodic task scheduling. ## Improvements * **Broker-side segment pruning for the multi-stage engine is now on by default** — Multi-stage queries now prune segments at the broker during routing, using the table's configured segment pruners (time, partition, etc.), before fanning out leaf stages to servers — matching single-stage engine behavior. Support was added in 0.15.0 but off by default on the default planning path; 0.16.0 flips `pinot.broker.multistage.logical.planner.use.broker.pruning` to `true`. Override per query with the `useBrokerPruning` query option, or set the broker config to `false` to restore the previous behavior. * **MSE query optimizer** — New Calcite optimization rules, join-predicate pushdown re-enabled by default, aggregate pushdown through `UNION ALL`, and broker-side short-circuiting when all segments are pruned. * **MSE observability** — Configurable metrics modes, adaptive-routing in-flight-request stats, per-server stats surfaced on streaming query responses, and a hook to expose the compiled MSE query plan. * **Index and query performance** — Reduced FST regexp memory footprint with interruptible matching, avoided posting-list copying on `JSON_MATCH`, and reduced HLL dictionary sizes. * **Ingestion** — Faster JSON-heavy ingestion transforms, intermediate transform columns no longer need to exist in the schema, and upsert `validDocIds` consensus is now enforced across task generators. * **Reliability** — Table configs with unresolved environment-variable templates are now rejected at write time instead of failing later, and a backward-incompatibility check now protects upsert `deleteRecordColumn` changes. * **Apache Calcite upgraded to 1.42.0**, bringing newer SQL planner/optimizer capabilities from upstream. ## Bug Fixes * **Query correctness** — Fixed column-to-column comparisons in `WHERE` clauses, incorrect millisecond-precision truncation for `TIMESTAMP` comparisons in the multi-stage engine, and MSE null-handling with `OR` predicates. * **Index correctness** — Fixed an out-of-bounds error in the Lucene text-index reuse path on retried segment conversion, and a doc-ID mapping corruption on unpaired UTF-16 surrogates. * **Stability** — Fixed several null-pointer exceptions and race conditions across controller, server, and minion, plus instance-partition assignment bugs for multi-stream realtime tables. * **Client and API fixes** — The broker query API now returns HTTP 400 (not 500) for client errors, and numerous Java client contract bugs were fixed for endpoints including schema listing, rebalance, and task management. * **Security** — Excluded a vulnerable embedded Jetty dependency pulled in transitively via Hadoop (CVE-2026-2332). ## Backwards Incompatible Changes * **`TimeFieldSpec` deprecated** — New schemas can no longer declare a `TimeFieldSpec`; use a regular `DateTimeFieldSpec` for the time column instead. * **Batch message mode removed** — The already-dormant batch message mode has been removed and can no longer be enabled. # Release Version 0.4.0: April-May 2022 Source: https://docs.startree.ai/reference/startree-release-notes/0.4.0 ## Apache Pinot ### Allow pushing segments to real-time table Users can now push segments to a real-time table, thus simplifying onboarding when ingesting from a hybrid source (real-time and offline). This makes it very easy to bootstrap or backfill a real-time table. For more details, see [the GitHub issue](https://github.com/apache/pinot/issues/8283). ### Deduplication support in real-time Pinot table Added the ability to remove duplicates in the streaming data sources based on a primary key. For more information, see the [Stream Ingestion with Dedup](https://docs.pinot.apache.org/basics/data-import/dedup) documentation. ### Server Failure Detector Added a new Failure Detector module in the Pinot Broker that can take failed servers out of rotation in order to prevent further query failures. More details in the Github issue. For more details in the [Github issue](https://github.com/apache/pinot/issues/8490) ### Minion observability enhancements. Added health endpoint for minions for proactively identifying ingestion issues (offline ingestion). For more details, see [github.com/apache/pinot/pull/8669](https://github.com/apache/pinot/pull/8669). New ingestion minion tasks/endpoints (metadata) to enable ease of debugging for users. For more details, see [github.com/apache/pinot/pull/8551](https://github.com/apache/pinot/pull/8551). ### Smart Approximations Added smart functions to automatically switch to approximate data structure when cardinality is high for DISTINCT\_COUNT and PERCENTILE. Add broker config pinot.broker.use.approximate.function to turn the feature on (off by default) Add query config useApproximateFunction to override the broker level config. For more details, see [github.com/apache/pinot/pull/8189](https://github.com/apache/pinot/pull/8189). ### Timestamp index Added support to configure a new TIMESTAMP index on columns of type TIMESTAMP. This will automatically pre-aggregate column values based on the specified time granularities. For more details, see [docs.pinot.apache.org/basics/indexing/timestamp-index](https://docs.pinot.apache.org/basics/indexing/timestamp-index) ### Spark 3.x support Added support for running offline Pinot ingestion jobs in Spark 3.x. For more details, see [github.com/apache/pinot/pull/8560](https://github.com/apache/pinot/pull/8560) ### Real-Time text search support Added support for Mutable FST Index that enables text search use cases on real-time data. The older Lucene indexes were created on segment flush and hence not available for the most recent data hosted in consuming segments. For more details, see [github.com/apache/pinot/pull/8861](https://github.com/apache/pinot/pull/8861) ### Distinct on Multi-value columns Added support to use DISTINCT query operator on multi-value columns. More details in this [github.com/apache/pinot/issues/8850](https://github.com/apache/pinot/issues/8850) ### Enhanced aggregation support during ingestion Ingestion Pre-Aggregation is now supported for MIN, MAX, and COUNT, in addition to SUM. To enable the feature, add an aggregationConfig to the ingestionConfigs of a real-time table config. The format of the config is (with example) ```json theme={null} "aggregationConfigs": [ { "columnName": "destColumn", "aggregationFunction": "MIN(srcColumn)" } ], ``` The destColumn must be in the schema and the srcColumn must not be in the schema. Additionally, all destColumns must be noDictionaryColumns. For more details, see [github.com/apache/pinot/pull/8611](https://github.com/apache/pinot/pull/8611) ### GapFill function Added a new function to enable users to fill gaps in timeseries data using previous or default values. For more details, see [docs.pinot.apache.org/users/user-guide-query/gap-fill-functions](https://docs.pinot.apache.org/users/user-guide-query/gap-fill-functions) ### Support for building all indexes in batch ingestion job Ability to create all indexes during segment generation, reducing the processing during segment load on the server. For more details, see [github.com/apache/pinot/issues/8334](https://github.com/apache/pinot/issues/8334) ## StarTree Extensions for Apache Pinot > [!NOTE] > > Available only in StarTree Cloud ### Pinot Proxy *(ALPHA)* Added a new endpoint to enable external services like Presto to be able to connect to internal Pinot servers (not exposed outside the k8 cluster) in a secure manner. For more details, refer to [this doc](https://dev.startree.ai/docs/startree-enterprise-edition/startree-query-connectors/connecting-to-presto#prestodb-pinot-proxy) ### Offline Ingestion: Auto Infer source partition column on sub directory *(ALPHA)* Added ability to derive columns in Pinot schema from the source file path. This is very useful when the source directory is partitioned on a dimension (eg: time with day as the smallest bucket). This partition column present in the file path is then automatically treated as one of the Pinot columns. For more details, refer to [this doc](https://dev.startree.ai/docs/startree-extensions/file-ingestion-task#derive-columns-from-the-source-file-paths). ### Offline Ingestion: Auto partition source data on sub directory *(ALPHA)* Added ability to repartition source data on a particular sub-directory defined by its level in the path. This feature is useful to group data from different files into the same segment or set of segments. For more details, refer to [this doc](https://dev.startree.ai/docs/startree-extensions/file-ingestion-task#partition-data-by-sub-directory-level-in-the-source-file-paths). ### Google PubSub connector for Pinot improvements *(ALPHA)* Added improvements to the PubSub connector such as retry mechanism, configurable timeout, fixed bugs in stream recreation, improved reliability of snapshots on checkpoint. ## StarTree Cloud - includes BYOC (Bring Your Own Cloud) and SaaS ### Soc2 Type1 Certification (*GA*) Achieved Soc2 Type1 certification. For more information, see [the blog post](https://www.startree.ai/blogs/announcing-startree-soc-2-type-1-compliance). ### Authentication on Pinot APIs *(ALPHA)* Announcing Alpha availability of authenticated Pinot APIs. Customers can use a generated token to get secure access to pinot apis. ### OIDC Security provider *(ALPHA)* StarTree admins can now configure any OIDC compliant IDP e.g. Okta to provide authenticated access to their data plane. ## Data Manager: Self-Service Ingestion tool ### Enhanced user experience *(BETA)* Launched simplified ingestion flow for Data Manager with guided experience. Now users can upload large files and can configure schema with more customizations at ease. ### Confluent Schema Registry support *(ALPHA)* Added support for using Confluent Schema registry with basic auth during real-time Kafka based ingestion. ### Confluent Schema Registry Json data format support *(ALPHA)* Added support for Confluent Schema registry Json data format to be used during Kafka based real-time ingestion ### File upload size limit increased *(GA)* Previous version only supported uploading 1 MB of files. Increased this limit to 30 MB. ### Offline Ingestion Improvements *(GA)* Trigger offline ingestion job immediately after new dataset creation. Previously there was a considerable delay for the async ingestion job to start. ### Kafka upsert support self serve *(GA)* Added support for configuring upserts in your Kafka based real-time dataset through the UI. For more details, see [this doc](https://dev.startree.ai/docs/pinot/recipes/upserts-real-time-offline-job) ### IAM role based S3 ingestion *(ALPHA)* Added support to ingest data from S3 using IAM role based access. Previously, users had to enter the access key and secret key which was not ideal.\ For more details, see [this doc](https://dev.startree.ai/docs/startree-enterprise-edition/startree-dataset-manager/s3-role-based-ingestion) ## ThirdEye: Anomaly Detection and Root Cause Analysis Tool ### Timezone support *(ALPHA)* May 2022 Now Users can configure the timezone during alert creation. For more details, see [dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#timezone](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#timezone) ### Anomaly Summary and Investigate *(ALPHA)* May 2022 Now users can self-serve root-cause analysis,give feedback, add comments and save the investigation associated with a given anomaly. For more details, see [dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/perform-root-cause-analysis#find-anomalies](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/perform-root-cause-analysis#find-anomalies) ### In-app help and support *(ALPHA)* Now users can access helpful tips and documentation within the ThirdEye application for quicker task completion or onboarding to ThirdEye. ### HTTP Detector (API) *(ALPHA)* Now users can plug detection algorithms into ThirdEye platform to detect anomalies in near real-time. (Example: Prophet is now supported to detect anomalies using HTTP Detector (API). For more details, see [https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/http](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/http) ### Timezone and timeformat support for ThirdEye Anomaly Reports *(ALPHA)* ThirdEye anomaly reports are now sent on the local timezone. For more details, see [dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/troubleshooting/faq\_tips](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/troubleshooting/faq_tips) ### Anomaly status *(ALPHA)* Users can now save comments and update the status for each anomaly saying it is “unexpected” or not. ### Pre-configured anomaly detection techniques (low code) *(ALPHA)* User can now use pre-configured alert templates (low code) created using the existing anomaly detection techniques supported by ThirdEye to detect anomalies in the metrics data. For more details, see [dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/anomaly-detection-algorithms](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/anomaly-detection-algorithms). ### Slack notifications *(ALPHA)* Now supports integrations with different channels of notifications to users (Email, Slack and Webhook). For more details, see [dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/notification/](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/notification/) # Release Version 0.5.0: Jun-Aug 2022 Source: https://docs.startree.ai/reference/startree-release-notes/0.5.0 ## Apache Pinot ### Enhanced null value support Added proper null value support in all Pinot layers for raw and dictionary encoded single value columns, for all data types. For more information, see the [Github issue](https://github.com/apache/pinot/issues/8697) documentation. ### Deduplication support in real-time Pinot table Users can now push segments to a real-time table, thus simplifying onboarding when ingesting from a hybrid source (real-time and offline). This makes it very easy to bootstrap or backfill a real-time table. For more details, see [the GitHub issue](https://github.com/apache/pinot/issues/8283). ### Upsert Improvements Added the following enhancements: * Prevent data corruption when getting out-of-order messages. See [PR #9159](https://github.com/apache/pinot/pull/9159) * Improve performance when offloading segments. See [PR #8674](https://github.com/apache/pinot/pull/8674) and [PR #9132](https://github.com/apache/pinot/pull/9132) ### Multi-stage query engine enhancements Made further progress in implementing full JOIN support as well as supporting existing query constructs with multi-stage engine, including: * Distributed shuffle joins * Group by * Sub-query * Complex queries using a mix of the above See [GitHub issue #8260](https://github.com/apache/pinot/issues/8260) For more details, see [V2 Multi-Stage Query Engine documentation](https://docs.pinot.apache.org/developers/advanced/v2-multi-stage-query-engine). > [!NOTE] > > Requires [separate configuration](https://docs.pinot.apache.org/developers/advanced/v2-multi-stage-query-engine) to enable and is in beta release > Feature support is limited to those listed above > No concurrent multi-stage query support ### API for segment reload status Added a new controller API for checking status of segment reload operation. More details in [this PR](https://github.com/apache/pinot/pull/8828) and [docs](https://docs.pinot.apache.org/users/tutorials/schema-evolution). ### Helix version upgrade to 1.0.4 Upgraded Helix version to 1.0.4 in Pinot. ### Query cancellation support Added support for cancelling a submitted query by its ID. This is useful to cancel long running queries in order to conserve resources on the Pinot cluster. This is controlled by the following config: * pinot.server.enable.query.cancellation, false by default * pinot.broker.enable.query.cancellation, false by default More details [in this PR](https://github.com/apache/pinot/pull/9171) or read about it in the [docs](https://docs.pinot.apache.org/basics/getting-started/frequent-questions/query-faq#how-do-i-cancel-a-query). ### Declarative support for query options Added support for declaratively specifying query options in the SQL query string. Example: ```sql theme={null} SET skipUpsert = 'false'; SELECT count(*) FROM tbl ``` More details in [this PR](https://github.com/apache/pinot/pull/9017) and [docs](https://docs.pinot.apache.org/users/user-guide-query/query-options#after-release-0.11.0). ### Pause support for real-time tables Added support to pause / resume ingestion for a real-time Pinot table. More details in [this PR](https://github.com/apache/pinot/pull/8986). ### Bloom filter for no dictionary columns Added support for configuring bloom filter for no dictionary single valued columns in Pinot table config. ### Protobuf decoder Added Protobuf decoder for realtime ingestion with file or schema registry based descriptor support. More details in [this wiki](https://docs.pinot.apache.org/v/release-0.11.0/basics/data-import/pinot-input-formats#protocol-buffers). ### Kafka ingestion from specific period/timestamp Ability to begin consumption from a specific period (2d ago, 12h ago) or timestamp (in format 2022-08-09T12:31:38.222Z) string. More details in [PR](https://github.com/apache/pinot/pull/9193) and [docs](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka). ### Minion UI in the Pinot Admin UI Added Minion tab in Pinot UI, to show more details about task queues, tasks and sub tasks, along with exceptions. More details in [PR](https://github.com/apache/pinot/pull/8978). ### Hide Query Console from Pinot UI Added a cluster config to hide Query Console from Pinot UI. Cluster config to disable can be found [here](https://docs.pinot.apache.org/configuration-reference/cluster). ## StarTree Extensions for Apache Pinot > [!NOTE] > > Available only in StarTree Cloud ### Tier storage enhancements *(BETA)* Following enhancements were made to StarTree’s cloud tiered storage feature: Performance 1. Async S3 client for supporting higher fetch volume and parallelism Observability 1. API to check tier status and metrics Ease of use 1. Ability to use same bucket as deepstore with custom path 2. Ability to use same tenant for local and tier ### Pubsub Correctness and operational improvements *(ALPHA)* Following enhancements were made to the PubSub connector 1. Auto delete old snapshots to reduce cost 2. Handling message re-delivery for better correctness ## StarTree Cloud - includes BYOC (Bring Your Own Cloud) and SaaS ### Self serve Portal *(BETA)* Launched a self serve portal for creating a StarTree workspace (BYOC and SaaS) without any involvement from StarTree personnel. You can find the portal here: [https://startree.cloud/](https://startree.cloud/). The corresponding documentation can be found here: [https://dev.startree.ai/docs/startree-enterprise-edition/startree-cloud/getting-started/](https://dev.startree.ai/docs/startree-enterprise-edition/startree-cloud/getting-started/) ### Authentication Service *(BETA)* Created a new unified authentication mechanism across platform and services within a StarTree workspace. This new model makes it safer and easier for all components to implement authentication. ### Improved Pinot Token generation workflow *(ALPHA)* Created a new, simpler workflow for generating tokens required for securely accessing a Pinot cluster. The earlier workflow required a cluster restart (transparent to user). ### Support for custom storage profile *(ALPHA)* Added support for overriding the pre-configured storage spec (based on cluster size) with a custom spec. This is useful to customize Persistent Volume spec for different components like Pinot server, minion, Zookeeper and so on. ### Improved monitoring and alerting for StarTree Cloud components *(ALPHA)* Added new metrics and corresponding alerts for monitoring critical things such as Zookeeper health and I/O profile. ### SSL/TLS *(GA)* Announcing GA for encrypted network access to StarTree workspace via SSL/TLS ## Data Manager: Self-Service Ingestion tool ### BigQuery connector *(BETA)* Users can now self-serve data ingestion from BigQuery using Data Manager no-code experience with few clicks. See [documentation](https://dev.startree.ai/docs/startree-extensions/sql-connector). ### Dataset ingestion status *(BETA)* Users can now monitor the data ingestion status and view ingestion logs after submitting the ingestion jobs. This will help users to debug issues and fix them as needed. See [documentation](https://dev.startree.ai/docs/startree-enterprise-edition/startree-dataset-manager/ingestion-status). ### Kinesis connector *(BETA)* Users can now self-serve data ingestion from AWS Kinesis using Data Manager no-code experience with few clicks. See [this doc](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/amazon-kinesis) for more details on the connector itself. ### Automated segment name generator support for time columns with simple date format *(GA)* Users can now self-serve data ingestion from AWS Kinesis using Data Manager no-code experience with few clicks. See [this doc](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/amazon-kinesis) for more details on the connector itself. ### Automated segment name generator support for time columns with simple date format *(GA)* The segment name generator type is automatically updated for time columns in simple date format. Users no longer need to configure this manually. ### Support enableSync for offline batch ingestion *(GA)* Now Data Manager supports enableSync for offline batch ingestion to refresh the data in Pinot whenever needed or at certain schedule. ## ThirdEye: Anomaly Detection and Root Cause Analysis Tool ### StarTree ThirdEye Community Edition *(GA)* Users can now explore and learn or evaluate before purchasing for free using StarTree Community Edition. Useful links: * [Blog to learn more](https://www.startree.ai/blog/introducing-startree-thirdeye-anomaly-detection-and-insights-with-apache-pinot) * [Quick start guide](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/getting-started/QuickTrialStarTreeCommEdition) * [StarTree Community Channel](https://communityinviter.com/apps/startreedata/startree-community) * [StarTree ThirdEye Community vs Enterprise edition features](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/ThirdEyeCommEdVsEntEdition) ### Alert creation (Derived/transformed metrics support) *(GA)* Users can now create “derived or transformed metrics” during alert creation. See [documentation](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/alert/derived-metric-alert). ### Alert creation (Improvements to Timeout and Timezone support) *(GA)* During alert creation: * Users no longer need to update the time column format and the time column name during alert creation. It will be automatically derived based on Pinot configurations. Users can always overwrite it if needed. * Users of StarTree ThirdEye now have an increased timeout window and can use long-running aggregations during anomaly detection. ### Holiday pattern recognization *(GA)* Users can now identify holiday patterns during anomaly detection to improve detection accuracy. (See [Event fetcher](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher) and [StarTree ThirdEye template](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets)) ### Slack notification and subscription to slack channels *(GA)* Users can now subscribe to slack channels using StarTree ThirdEye subscription user interface to send anomaly notifications to the slack channels. See [documentation](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/notification/#step-1-create-a-slack-app) ### Authentication on StarTree ThirdEye APIs *(GA)* Users can now use a generated token to get secure access to StarTree ThirdEye APIs. These tokens have an extended life so users can use these tokens for automation such as bulk creation of alerts or ingestion of “Events” to StarTree ThirdEye using external scripts. See [documentation](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/use-the-api#obtaining-a-basic-token---without-expiration). ### Auto onboard datasets *(GA)* Users can now auto onboard all datasets from “Pinot” to ThirdEye with one click during data source creation. See [documentation](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/database/#steps-to-register-a-new-datasource). ### Bulk delete anomalies *(GA)* Users can now bulk delete anomalies from the anomalies list view. ### Duplicate and reset alerts *(GA)* Users can now create “Duplicate and Reset Alerts” using StarTree ThirdEye from the Alerts detail page or Alerts list view page. ### Create alert (No Code) *(BETA)* Users can now create alerts and detect anomalies in a few seconds without writing a single line of code. See [documentation](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/getting-started/create-your-first-alert). ### RCA events *(BETA)* Now users can perform root-cause analysis in ThirdEye using a self-serve events UI. Using this users can identify events that caused the anomalous events. Users can upload any type of event (ex: Holidays, Jira events etc using ThirdEye APIs (link)) Example of events that helps with “Root-cause analysis” of an anomalous event in a KPI are: * Custom events (Ex: A/B test-related specific events) * Public events (Ex: holiday or political event or competition etc) * Internal events (Ex: Feature ramp, software deployment, metric definition changes etc) ### Anomaly filtering *(BETA)* Now users can filter anomalies by alerts/dataset/metrics, etc. in the anomaly list/report view # Release Version 0.6.0: Feb 2023 Source: https://docs.startree.ai/reference/startree-release-notes/0.6.0 ## Apache Pinot ### Multi stage enhancements The multi stage query engine now supports the `HAVING`, `ORDER BY`, `IN`, and `NOT IN` clauses. It also supports left joins, semi joins, and inequality joins. All functions have now been registered with the Calcite catalog reader. ### V1 query engine enhancements Added support for isDistinctFrom and isNotDistinctFrom. More details in the [pull request](https://github.com/apache/pinot/pull/9312). ### Multi volume Added capability to tiering storage locally by attaching multiple disks (eg: SSD + HDD). More details in [the Pinot documentation](https://docs.pinot.apache.org/operators/operating-pinot/separating-data-storage-by-age/using-multiple-directories). ### Segment level debug Added segment level Debug API and UI. More details in the [pull request](https://github.com/apache/pinot/pull/9609). ### Consumer record lag Added a new API for exposing record based lag during real-time consumption. Currently this is only supported for Kafka data source. More details in the [pull request](https://github.com/apache/pinot/pull/9515). ### Custom time boundary for hybrid tables Added new APIs for configuring a custom time boundary for hybrid Pinot tables as well as a validation API to ensure all segments have finished uploading during offline push (check against ideal state). More details in the [pull request](https://github.com/apache/pinot/pull/9356). ### Upserts enhancements Added enableSnapshot flag in upsertConfig to use snapshot for upsert metadata recovery. This will help achieve TTL (Time To Live) support for primary keys. More details in the [Pinot documentation](https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#extract-record-headers-as-pinot-table-columns) and [pull request](https://github.com/apache/pinot/pull/9224). ### Extract metadata from stream event header Added support for using fields within the message envelope as columns in the Pinot schema (eg: key within a Kafka message envelope). More details in the [pull request](https://github.com/apache/pinot/pull/9062). ### Segment Reload enhancements Added ability to change compression type during segment reload. More details in this [PR](https://github.com/apache/pinot/pull/9454). In addition, added capability within the Pinot UI to track reload progress. More details in [PR-9521](https://github.com/apache/pinot/pull/9521) and [PR-9700](https://github.com/apache/pinot/pull/9700). ### Adaptive Server Selection **Warning**: This feature is experimental, under development, and turned off by default. We recommend using this feature for **testing purposes only**. When a query is received, we could use one of the implemented Adaptive Selectors (NumInFlightRequests, Latency, Hybrid) to efficiently route queries to the best server instead of using a naive round robin approach. More details can be found in the [Pinot documentation](https://docs.pinot.apache.org/operators/operating-pinot/tuning/query-routing-using-adaptive-server-selection). ### Frictionless Ingestion * Automatically infer parquet reader type based on file metadata in case of offline ingestion. More details in this [PR](https://github.com/apache/pinot/pull/9294). * Added Spark Job Launcher utility for offline table ingestion within the Pinot admin tool for ease of use. More details in this [PR ](https://github.com/apache/pinot/pull/9288). * Added continueOnError flag within the Pinot table config. If set to true, any errors from data type or expression transformations are ignored and null / default values are used instead. This is useful when users don’t want the ingestion to stop because of a few bad records. More details in [PR-9320](https://github.com/apache/pinot/pull/9320) and [PR-9376 ](https://github.com/apache/pinot/pull/9376). ### MergeRollup task on real-time tables Added support for merging / rolling up segments of a real-time Pinot table. More details in this [PR](https://github.com/apache/pinot/pull/9890). ### Force commit Added a new resetConsumption API in the controller to force the current consuming segment in a real-time Pinot table to be committed. More details in this [PR](https://github.com/apache/pinot/pull/9197). ### Seamless stream change Added the capability to modify stream properties (for eg: start consuming from a different Kafka topic) without disabling the table. More details in this [PR](https://github.com/apache/pinot/pull/9289). ### Logging enhancements * Added a /loggers API endpoint to change logging level at runtime. More details in this [PR](https://github.com/apache/pinot/pull/9180). * Added a new API to allow downloading logs from individual components (broker/server) as well as a new controller API to download any remote log. More details in this [PR](https://github.com/apache/pinot/pull/9259). ## StarTree Extensions for Apache Pinot: Available only in StarTree Cloud ### RocksDB backed Upsert *(BETA)* Added the capability of configuring RocksDB backend for managing upsert metadata in a Pinot server. This enables the server to handle a lot more primary keys than before (previously this was done in memory). ### Databricks Delta Ingestion *(ALPHA)* Added support for ingesting data from a Databricks Delta table into an offline Pinot table. ### SegmentRefresh for RT tables *(ALPHA)* Added support for performing a segment refresh operations for all completed segments within a real-time Pinot table. This enables users to ensure real-time table segments adhere to the latest Pinot table config. ### Debezium connector for MySQL *(ALPHA)* Added support for ingesting MySQL Debezium CDC format messages from a real-time stream. ### Tiered storage *(BETA)* * Improved server restart time when cloud tiered storage is enabled by persistently caching certain portions of all column indexes needed during restart. * Query performance improvements using selective columnar fetch based on query pattern and block level reads. ### File size based task planner in ingestion *(GA)* Added capability to configure minion tasks to ingest in a size based manner in addition to count based. This allows the user to ingest all files from a data source in a single round of tasks based on the total size. ## StarTree Cloud - includes BYOC (Bring Your Own Cloud) and SaaS ### Disaster Recovery for data plane *(ALPHA)* StarTree Admin is able to recover a given workspace from a region failure by recovering the StarTree cluster state (RTO: 24 hours) ### Release decoupling *(ALPHA)* StarTree admins can now release individual components like Pinot or Data Manager without requiring a full release ### Authentication service *(GA)* Authentication service for secure access to Startree environments is now GA ### Token generation for Try environments *(GA)* Token generation for secure access to Pinot cluster in trial environments is now GA. ## Data Manager: Self-Service Ingestion tool ### Improved AWS IAM Role based onboarding experience *(BETA)* Users now can check the AWS account id directly on DM instead of asking the StarTree customer support team. More enhancements to come in the next release. ### Dimension Table support *(GA)* Users can now create dimension tables from DM. ### Enhanced Datetime inference logic *(GA)* More accurate datetime column inference during data modeling in DM ### Support enhanced security mechanisms for Kafka SASL authentication *(GA)* Support added in DM for the following SASL mechanisms: * PLAINTEXT * SCRAM-SHA-256 * SRAM-SHA-512 ### Support for GZ file ingestion *(GA)* Gzipped files can be ingested via DM directly.. Other supported formats: Avro, Json, CSV, Parquet, ORC ### Enhanced Data ingestion *(GA)* Couple of improvements for robust data ingestion experience: 1. Improved dictionary inference logic 2. Data size configuration for Minion ### Support for schema registry in Kafka connector *(GA)* Users will only see the topics that are registered in the schema registry and the data format is no longer needed to be selected. ### Record reader config support *(GA)* Users can now pass the record reader specific configs via DM. (e.g. split delimiter for csv reader) ### BigQuery connector *(GA)* Users can now self-serve data ingestion from BigQuery using Data Manager no-code experience with a few clicks. For more information, see [https://dev.startree.ai/docs/startree-extensions/sql-connector](https://dev.startree.ai/docs/startree-extensions/sql-connector). ### Dataset ingestion status *(GA)* Users can now monitor the data ingestion status and view ingestion logs after submitting the ingestion jobs. This will help users to debug issues and fix them as needed. For more information, see [https://dev.startree.ai/docs/startree-enterprise-edition/startree-dataset-manager/ingestion-status](https://dev.startree.ai/docs/startree-enterprise-edition/startree-dataset-manager/ingestion-status). ### Kinesis connector *(GA)* Users can now self-serve data ingestion from AWS Kinesis using Data Manager no-code experience with few clicks. For more information, see [https://dev.startree.ai/docs/startree-enterprise-edition/startree-dataset-manager/kinesis](https://dev.startree.ai/docs/startree-enterprise-edition/startree-dataset-manager/kinesis). ## ThirdEye: Anomaly Detection and Root Cause Analysis Tool ### Cohort recommender *(GA)* Cohort recommender) will help users identify the cohorts of top contributors (single or a group of dimensions) contributing to a spike in a given metric of a dataset. Using this feature user can now monitor single or multiple time series data for a given time range for the in-scope dataset and metrics. For more information, see [https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/cohort-recommender](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/cohort-recommender). ### Guided onboarding and alert creation flow *(GA)* Users of ThirdEye can now use guided ThirdEye onboarding and sample alerts to quickly try, evaluate and get started with ThirdEye in a few clicks. ### Integrated flow for cohort recommender and multi-dimension alert creation *(GA)* Users of ThirdEye can now use cohort recommender to identify top contributors (single or group of dimensions) for a given metric and monitor single or multiple time series in two to three clicks. Using these features users need not manually configure alerts which can be time-consuming and prone to error. ### Bulk delete support for Alerts, anomalies and related entities *(GA)* Users can now bulk delete alerts, anomalies, and related entities to clean up all experiment data or static data which are no longer in use. ### Anomaly filters *(GA)* Now users can apply “Anomaly filters” (such as Threshold-based, Weekend, Holiday, and Cold\_Start filters) as part of the low-code alert configuration to fine-tune the alerts and improve accuracy by eliminating noise. ### Root-cause analysis (Dimension filter (include/exclude) *(GA)* Users can now perform root-cause analysis for a selected set of dimensions instead of entire list of dimensions. For more information, see [https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#rcaexcludeddimensions](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#rcaexcludeddimensions). ### Launched Dimension Exploration *(GA)* Users can now monitor multiple time series for single or group dimensions for a dataset and metric by configuring a single one-time (low-code/no-code) alert. For more information, see [https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/dimension-exploration](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/dimension-exploration). ### Load default alert templates *(GA)* Now users can load default alert templates from the Alert Templates Configuration screen instead of using API to load default alert templates. ### Alert creation (Derived/transformed metrics support) *(GA)* Users can now create “derived or transformed metrics” during alert creation. For more information, see [https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/alert/derived-metric-alert](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/alert/derived-metric-alert). # Release Version 0.6.1: Apr 2023 Source: https://docs.startree.ai/reference/startree-release-notes/0.6.1 ## Apache Pinot * HLL and Theta Aggregation support in star-tree index and Merge/Rollup * API to track the status/progress of segment rebalance * Pinot Spark 3 Connector * Bug fixes, metric improvement query performance improvement (both single-stage and multi-stage query engines) ## StarTree Extensions for Apache Pinot: Available only in StarTree Cloud * Tiered storage performance improvement ## StarTree Cloud - includes BYOC (Bring Your Own Cloud) and SaaS * EKS upgrade to Kubernetes version 1.23. * S3 bucket versioning is turned ON by default along with a lifecycle policy to expire all non-current object versions after 5 days. * Profiles (S,M,L) for prometheus and automated remote-write configuration ## Data Manager: Self-Service Ingestion tool * BigQuery SQL ingestion improvements: * Support for only ‘dataset’ level access instead of requiring access to the entire ‘project’ * Support for BigQuery specific lexical policies such as hyphens. * Support to read UNION type by extracting a non-null child field and using it as the schema. * New Data Manager usage metrics used for centralized monitoring. ## ThirdEye: Anomaly Detection and Root Cause Analysis Tool * Users can decide which [ThirdEye algorithms](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/anomaly-detection-algorithms#choosing-the-right-algorithm) and [related templates](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/templates/) to select and apply it on their data for accurate anomaly detection. * Users can detect anomalies in [mutable data](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/mutable-data) using StarTree ThirdEye. * Users can now [subscribe to anomaly notifications](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/how-tos/notification/#creating-a-subscription-group-using-apis) at dimension level granularity. # Release Version 0.7.0: August 2023 Source: https://docs.startree.ai/reference/startree-release-notes/0.7.0 ## Apache Pinot updates since last StarTree release * Delete records in upsert-enabled tables * Pipeline breaker and dynamic broadcast support for semi-joins * Null value support in aggregate and filters * Order null values * Support `UNION`, `INTERSECT`, and `EXCEPT` operators * Support ranking window functions for v2 engine: * `ROW_NUMBER` window function support * `RANK` * `DENSE_RANK` * Multistage query updates: * Table-level access validation * Queries per second (QPS) quota check for all tables * Phase timers for `REQUEST_COMPILATION`, `AUTHORIZATION`, and `QUERY_EXECUTION` ## StarTree Cloud The following updates apply to both bring-your-own-cloud (BYOC) and SaaS deployments. * Configure tags to add to all StarTree-managed AWS resources, which can be useful for auditing and compliance initiatives. * StarTree's Pinot deep store S3 bucket has additional default protections. The bucket policy now actively denies any non-SSL traffic. * All new and existing clusters will be upgraded to use Kubernetes version 1.24. * StarTree-provisioned Google Cloud Platorm (GCP) clusters are now VPC-native. Specify subnet CIDRs for nodes, pods, and services. * We've added automated certification renewal to TLS certiticates to improve the architecture's stability and reliability. * In a disaster, recover a workspace in from a region failure by recovering the StarTree cluster state. * Administrators can release individual components, like Pinot or Data Manager, without requiring a full release. ### StarTree extensions for Apache Pinot * Delete selected records from a Pinot table to meet GDPR requirements. See [Segment purge](/corecapabilities/manage-data/purge-task) for details. #### Beta extensions * *(BETA)* Pinot server restart time is optimized for large upsert-enabled tables. * *(BETA)* Tiered storage: * Tiered storage supports sparse indexes, leading to performance improvements for queries running against high cardinality columns. For details, see [Use sparse index](/corecapabilities/manage-data/set-up-tiered-storage/use-sparse-index) * Google Cloud Services (GCS) is available as a cloud tier. * Stale segments are automatically cleaned up from the object tier. * Trace remote reads with newly available statistics. * Overwrite index configurations for the object tier. #### Alpha extensions * *(ALPHA)* Push data into Pinot to create a table with the write API. Send data from any application to a REST API endpoint. ## Data Manager * Data Manager now masks connection credentials to improve security. * We've optimized minion segment creation to avoid out-of-memory (OOM) and out-of-disk (OOD) issues during ingestion, increasing batch ingestion reliability. * The segment refresh task is more stable and reliabile when users change table configuration of an existing table. * Integrate with Delta Lake and Databricks Lakehouse to ingest data and keep it in sync in Pinot. * Configure batch ingestion to be performed atomically (either all or none of the data is ingested), preventing inconsistency in query results. * Continuously backfill data from BigQuery with each periodic sync. * Connect to Amazon Web Services (AWS) Kinesis and S3 using an AWS cross-account IAM role. * Integrate with Confluent Cloud. * *(BETA)* Backfill data from SQL sources on an ad hoc basis. ## ThirdEye * We've simplified the workflow for creating and configuring new alerts. * ThirdEye alerts support data mutability, meaning they can handle recent changes in underlying data. For details, see [Streaming upsert use case](/thirdeye/concepts-data-alerts-notifications/concepts-mutable-data). * Configure subscription groups on a per-dimension basis within an alert to alert specific teams to for anomalies in different slices of the underlying dataset. For details, see [Create a subscription group](/thirdeye/how-tos/add_notification). * Filter and sort dimensions by name and latest anomaly from the UI. We've also improve performance for dimension exploration. * Anomaly detector algorithm updates: * [Exponential smoothing (ETS)](/thirdeye/operators/anomaly-detector/ets) performance has improved. * Find anomalies in seasonal patterns with the new [Matrix profile algorithm](/thirdeye/operators/anomaly-detector/matrix_profile). * [Default alert templates](/thirdeye/how-tos/alert/use-templates) are now automatically refreshed on new releases instead of requiring an API request. # Release Version 0.7.1: November 2023 Source: https://docs.startree.ai/reference/startree-release-notes/0.7.1 ## Significant Apache Pinot updates since last StarTree release For complete details on Pinot changes, see [Releases](https://docs.pinot.apache.org/basics/releases). * Skip unparseable records in the CSV reader. To enable, set the `skipUnParseableLines` flag to true ([pull request](https://github.com/apache/pinot/pull/11487)). * Protocol buffer ingestion supports null values with Proto 3 ([pull request](https://github.com/apache/pinot/pull/11553)) * Upgrade Confluent libraries from 5.5.3 to 7.2.6 ([pull request](https://github.com/apache/pinot/pull/11753)) * Faster real-time table ingestion with updates to the segment builder. To enable, edit the table configuration to set `realtime.segment.flush.enable_column_major` to `true` ([pull request](https://github.com/apache/pinot/pull/11776)) * Improve alias handling in single-stage engine with multiple fixes to column aliases ([pull request](https://github.com/apache/pinot/pull/11610)) * Enhance handling of new partitions when using to prevent "instance unavailable" exceptions ([pull request](https://github.com/apache/pinot/pull/11585)) * Optimize performance in the multi-stage engine: * For a single join key and group key scenario, operate directly on the key values without wrappers ([pull request](https://github.com/apache/pinot/pull/11630)) * Operate on column indexes in multi-stage aggregations to prevent extra conversion steps * Avoid converting unnecessary rows in aggregations ([pull request](https://github.com/apache/pinot/pull/11607)) * Enhance segment assignments for upsert tables with more checks to ensure that the conditions required for upsert functionality to work are not violated ([pull request](https://github.com/apache/pinot/pull/11628)) * Fix handling of literals used in aggregation for v2 engine ([pull request](https://github.com/apache/pinot/pull/11863)) ### Breaking changes * You must now specify the data type of literals in Pinot queries. Before this change, for example, `2022-02-02 22:22:22.123` was automatically treated as a timestamp data type. Now, following standard SQL behavior, use `CAST('2022-02-02 22:22:22.123' AS TIMESTAMP)` instead ([pull request](https://github.com/apache/pinot/pull/11763)). * Change the "forbidden" error to "unauthorized" ([pull request](https://github.com/apache/pinot/pull/11501)) * Table configurations that point to a different schema name no longer work ([pull request](https://github.com/apache/pinot/pull/11591)). * You can no longer change the table state using the `GET` call ([pull request](https://github.com/apache/pinot/pull/11621)). * You can no longer create a schema with `NaN` as the default value ([pull request](https://github.com/apache/pinot/pull/11661)). * `BigDecimal` responses are now stored as a string with double quotes instead of a number ([pull request](https://github.com/apache/pinot/pull/11716)). ## Dependencies * Upgrade to Parquet 1.12.3 ([pull request](https://github.com/apache/pinot/pull/11337)) * Upgrade to Hadoop 3.2.4 ([pull request](https://github.com/apache/pinot/pull/11369)) * Upgrade to AVRO 1.10.2 ([pull request](https://github.com/apache/pinot/pull/11698)) ### StarTree extensions for Apache Pinot The following updates are available only in StarTree Cloud. * Improvements to [file ingestion task](https://dev.startree.ai/docs/ingest-data/file-ingestion-task): * Enhancements to batch ingestion using minion to improve atomic ingestion and backfill operations * Control size-based segment creation with to improve performance * Automatically tune segment size for segment refresh task without configuring `maxNumRecordsPerTask` and `maxNumRecordsPerSegment`. Size-based tuning helps make predictable segment sizes and avoid memory- or size- related exceptions * Validation is stricter for using sync mode in conjunction with other tasks. You can no longer schedule the segment refresh task at the same time as sync mode. * Separate RocksDB log from server logs to improve debugging experience and allow you to set different retention and rollover policies * Improve Kafka logs by changing the following classes to error-level: * `KafkaConsumer` * `AppInfoParser` * `ConsumerConfig` * Enhancements to upsert tables: * Correctly track primary key count and add corresponding metrics * Improve stability during deletion * Improve performance and navigation in broker and server Grafana dashboards * Move to Google Trust Services Certificate Authority to improve certification management ## Data Manager * Improve data sampling from Kafka topics with large numbers of partitions by preventing "no data" error in preview * Automate Google Cloud Platform (GCP) credentials in Data Manager so you can ingest instead of having to contact StarTree support * Improve error messages to aid troubleshooting ## ThirdEye * Improve loading time for multi-dimension alerts and dashboard statistics * Simplified alert creation with advanced anomaly detection and tuning options, reducing complexity of data patterns and seasonality # Release Version 0.7.2: Dec 2023 Source: https://docs.startree.ai/reference/startree-release-notes/0.7.2 This release only impacts customers with StarTree Cloud deployments on AWS. ## Upgrade Kubernetes Release Version * Kubernetes 1.24 is reaching end-of-life soon. To support StarTree Cloud customers with deployments on AWS, we've upgraded to Kubernetes 1.25. For more information about Amazon EKS Kubernetes versions, see [the AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html). Additionally, customers' production environments updated to 0.7.2 will use IMDSv2 by default. IMDSv2 helps protect against potential security threats and unauthorized access to sensitive information within cloud environments. This is important for securing cloud-based applications and services. # Release Version 0.8.0: February 2024 Source: https://docs.startree.ai/reference/startree-release-notes/0.8.0 ## Apache Pinot updates since last StarTree release For details on Pinot changes, see [Releases](https://docs.pinot.apache.org/basics/releases). * Permit defining `NULL` handling at the table level or at each individual column level. [Link](https://github.com/apache/pinot/pull/11960) * Add `lastUsed` option in `resumeConsumption` endpoint in the broker API to improve UX. [Link](https://github.com/apache/pinot/pull/12200) * Improve ingestion validation with `TimeValidationTransformer` to mark a record as invalid if the primary time column is out of range (1971 inclusive to 2071 exclusive). [Link](https://github.com/apache/pinot/pull/11907) * Update the `tables` API endpoint to list only the dimension tables by specifying `dimension` as the type, for example: ``` curl -X 'GET' \ 'http://localhost:9000/tables?type=dimension; \ - H 'accept: application/json' ``` [Link](https://github.com/apache/pinot/pull/11859) * Enhance consuming segment handling to avoid an under-counting error with upsert tables. [Link](https://github.com/apache/pinot/pull/11978) * Improve the logic for taking snapshots by making them more atomic and in an order that permits correct table preloading. [Link](https://github.com/apache/pinot/pull/12232) * Add the following new metrics: * `pinot_server_tableRebalanceInProgress_Value{table=${tableName},tabletype=${tableType}}` indicates whether a table is being rebalanced. `1` indicates rebalancing is in progress and `0` when it's not. [Link](https://github.com/apache/pinot/pull/12270) * `pinot_server_tableDisabled_Value{table=${tableName},tableType=${tableType}}` indicates whether a table is disabled. It uses `1` to indicate the table is disabled and `0` when it is not. [Link](https://github.com/apache/pinot/pull/12000) * `pinot_server_tableConsumptionPaused_Value{table=${tableName},tableType=${tableType}}` indicates whether table consumption is paused. `1` indicates the table is consumption is paused and `0` when it's not. [Link](https://github.com/apache/pinot/pull/12000) * Add a set of catch-all regexes for JMX -> Prometheus Exporter for when a regex used does not match a metric. [Link](https://github.com/apache/pinot/pull/12073) * Add compression configuration for aggregation in a star-tree index. [Link](https://github.com/apache/pinot/pull/11744) * Add a new flag to indicate whether the query result is partial or full. [Link](https://github.com/apache/pinot/pull/11592) * Add `DATETIMECONVERTWINDOWHOP` transformation function. [Link](https://github.com/apache/pinot/pull/11773) * Enable tracking of out of order events in an upsert-enabled table using a new configuration `outOfOrderRecordColumn`. [Link](https://github.com/apache/pinot/pull/11877) * Enable support for leveraging a star-tree index in conjunction with filtered aggregations, including filtered group-by aggregations. [Link](https://github.com/apache/pinot/pull/11886) * Add a new MV dictionary-encoded forward index format that only stores the unique MV entries, reducing storage footprint for indexes. [Link](https://github.com/apache/pinot/pull/1199) * Introduce low disk mode to table rebalance, which is set to false by default. When set to true, the server will first offload segments before loading the new segments during rebalance. [Link](https://github.com/apache/pinot/pull/12072) * Introduce a new configuration `controller.realtime.segment.deepStoreUploadRetry.parallelism` (the default setting is `1`) to increase the size of the thread pool used for retrying segment uploads. Also the upload retry is now an asynchronous operation. [Link](https://github.com/apache/pinot/pull/12017) * Enable `SegmentGenerationAndPushTask` to push segment(s) to a realtime table, supporting bootstrapping an upsert enabled table. [Link](https://github.com/apache/pinot/pull/12084) * Add the ability to specify a custom Lucene analyzer used by text index for indexing and search on an individual column basis. [Link](https://github.com/apache/pinot/pull/12027) * Add murmur3 support as partition function * Enhance `DistinctCountThetaSketch` aggregation function by adding new parameters to give the end-user more control over how sketches are aggregated at query time. [Link](https://github.com/apache/pinot/pull/12042) * Add new configuration in upsert, `deletedKeysTTL`, which when set will remove deleted keys and mark the `validDocID` as invalid after the `deletedKeysTTL` threshold period, improving memory utilization. [Link](https://github.com/apache/pinot/pull/12037) * Add support for vector index using Hierarchical Navigable Small World (HNSW). [Link](https://github.com/apache/pinot/pull/11977) * Add the ability to initialize broker tags from configuration and automatically update the broker resource when broker joins the cluster for the first time. [Link](https://github.com/apache/pinot/pull/12175) * Enable partition level force-commit functionality, expanding the endpoint to accept a comma-separated list of partitions or consuming segment names. [Link](https://github.com/apache/pinot/pull/12088) * The following updates are specific to the multi-stage query engine: * Optimize partition-based query performance when using the multi-stage query engine. The engine is now able to determine table partitioning and apply the best data shuffle mechanism automatically. [Link](https://github.com/apache/pinot/issues/12015) * Enable the multi-stage query engine to run multiple operator chains, provided there is no requirement for distributed data shuffling. [Link](https://github.com/apache/pinot/pull/11937) * Enable multiple SEMI-JOINs in the multi-stage query engine to use index lookup within the same node for a left-table scan. [Link](https://github.com/apache/pinot/pull/12038) * Add support in the multi-stage query engine for early termination and direct error, warning, or stats return in the multi-stage query engine. [Link](https://github.com/apache/pinot/pull/11746) * Bug fix: Segments created in realtime tables are guided by the parameter `realtime.segment.flush.threshold.segment.size` if it is set. [Link](https://github.com/apache/pinot/pull/12016) ## StarTree Cloud ### StarTree Extensions for Apache Pinot * Enable bootstrapping of upsert-enabled tables by supporting batch ingestion using `fileingestiontask` into a realtime table. * StarTree Upsert on by default for all StarTree deployments, providing enhanced scalability and stability when using upsert. * Improved server restart times when using StarTree upserts * Ability to take snapshot for improved recoverability of upsert tables * Provide visibility into the health of various components (Server, Broker, Controller, Tables, etc.) using the Cluster Health Dashboard in Pinot Control Panel. The dashboard is updated every 20 minutes and can be triggered on-demand by using the `/periodictask/run` API call. * Ability to gate access to Pinot tables using a new Role Based Access Control (RBAC) system. Roles can be assigned to individual users, IDP groups or Pinot API tokens. Access can be controlled at a table-level granularity along with the ability to allow/deny specific APIs on Pinot clusters and tables. **Alpha Release** ### Data Manager * Add Custom Connector option that lets you create a dataset using a JSON connection configuration to a Google Cloud Storage (GCS) data source. * Enhance interface to select a directory or multiple directories in an AWS S3 bucket. * Add SSL certificate support for Kafka. Now, you can enter details to connect with Kafka under SSL Authentication Type in Kafka Source. * Enhance Delta Lake connector to support IAM role access in AWS. ### ThirdEye * Link anomaly alerts to PagerDuty for instant notification and efficient incident management. [Link](https://dev.startree.ai/docs/get-started-with-thirdeye/how-tos/add_notification#pagerduty-support) * Enable customizable bounds for precise anomaly detection, enhancing decision-making accuracy. [Link](https://dev.startree.ai/docs/get-started-with-thirdeye/templates/startree-mean-variance) * Protect sensitive information with automated data masking during the automated anomaly detection alert creation process.[Link](https://dev.startree.ai/docs/get-started-with-thirdeye/templates/startree-experimental-mean-variance-with-mask) * UI/UX improvements * View related tasks for each alert, including success, failure, and access to logs for troubleshooting. * View a list of notifications sent per subscription group for better insight into alert distribution. * Easily identify which subscription groups are receiving specific alerts. # Release Version 0.8.1: April-May 2024 Source: https://docs.startree.ai/reference/startree-release-notes/0.8.1 This release impacts customers with StarTree Cloud deployments on AWS and GCP. ## Upgrade Kubernetes Release Version * Kubernetes 1.25 is reaching end-of-life soon. To support StarTree Cloud customers with deployments on AWS and GCP, we've upgraded to Kubernetes 1.29. For more information about Amazon EKS Kubernetes versions, see [the AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html). For more information about Google Cloud GKE Kubernetes versions, see the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/release-schedule]). Additionally, there are a few platform stability improvements impacting the following: * Alerts configuration * Adding requests / limits to all platform components We will be doing this upgrade beginning in April and completing by the end of May 2024. We will begin with the Non-production (dev, staging, qa, etc.) environments, followed by Production environments. Note that all environments created after StarTree release 0.8.0 are already on Kubernetes version 1.29. **Downtime?** * There will be a rolling restart. For tables with replication > 1, there should not be any downtime though performance may be impacted in some cases. * For V2 Query Engine users, there will be some downtime that is dependent on how long it takes the servers to restart. This is generally dependent on factors such as the amount of data on a server. Questions? Please contact StarTree Support for any questions regarding release. # Release Version 0.9.0: June 2024 Source: https://docs.startree.ai/reference/startree-release-notes/0.9.0 ## Apache Pinot updates since last StarTree release For details on Pinot changes, see [Releases](https://docs.pinot.apache.org/basics/releases). * Pinot's database support leaps forward! A new way to include database context within API requests. [Link](https://github.com/apache/pinot/pull/12417) * Support ListAgg WITHIN GROUP clause [Link](https://github.com/apache/pinot/pull/13146) * Ordering Within Groups: Pinot now enables in-group sorting. Users can define the GROUP BY clause as usual and sort the data within each group based on specific columns. * Pinot empowers users with ListAgg, a powerful function that overcomes limitations. It allows for aggregating multiple values into a single, comma-separated list for each group defined by the GROUP BY clause. * Distinct Control for Lists: Want to ensure unique values in your aggregated lists? No problem! Pinot's ListAgg now supports a DISTINCT flag, allowing control of whether duplicate values are included when building lists. * StarTree indexes can now be leveraged for queries with NOT following a predicate, improving query performance. [Link](https://github.com/apache/pinot/pull/12988) * Replacing the default DynamicMessage decoder with an option to leverage code generated by protoc. This optimization can lead to 3-4x faster deserialization, enhancing overall query performance when dealing with Protobuf data. [Link](https://github.com/apache/pinot/pull/12980) * Empowering minion nodes with the ability to download segments directly from servers, enhancing data availability. This functionality is controlled by a new task-level configuration: allowDownloadFromServer (defaults to false). [Link](https://github.com/apache/pinot/pull/12960) * Pinot to Offer ValueWindowFunctions \[LEAD, LAG, FIRST\_VALUE, LAST\_VALUE) for Enhanced Analytics * Add support for the Postgres SQL [date\_bin](https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-BIN) function. [Link](https://github.com/apache/pinot/pull/12790) * Advanced phrase search capabilities for Lucene-indexed tables. [Link](https://github.com/apache/pinot/pull/12680) * Wildcard Matching: Find variations of phrases using wildcards (e.g., *pache pino* to match "Apache Pinot" or "apache pinot"). * Prefix Matching: Search for terms starting with a specific prefix (e.g., "pino" to find "Pinot" or "Pinot Noir"). * This feature is configurable through a new option in Lucene text-indexed columns (disabled by default). * Pinot now supports GZIP compression for raw forward indexes, allowing you to squeeze more data into less storage space in certain scenarios. [Link](https://github.com/apache/pinot/pull/12668) * Addressed inconsistencies in how exclusive predicates (e.g., !=, NOT IN) were handled, leading to more intuitive and predictable filtering behavior. [Link](https://github.com/apache/pinot/pull/13139) * Consistent Wildcard Matching: Previously, wildcard (\*) behavior differed between inclusive and exclusive predicates. Now, both types of predicates use the same logic, ensuring consistent results. * No More False Positives: Documents lacking a specific key will no longer be included in results for exclusive predicates, eliminating unexpected matches. * Nested Exclusive Predicates: Craft even more intricate filtering logic with the newfound support for nested exclusive predicates within JSON paths. * These enhancements deliver a significant boost to the accuracy and power of your JSON data filtering in Pinot. * Allows support for multi-value fields to jsonExtractIndex [Link](https://github.com/apache/pinot/pull/12532) * Introducing a trio of powerful UDFs (user-defined functions): These UDFs generate derived columns containing prefixes, postfixes, and n-grams. These derived columns persisted, allowing Pinot to leverage its efficient inverted indexes for faster filtering. [Link](https://github.com/apache/pinot/pull/12392) * Prefix & Postfix UDFs: Find terms starting/ending with specific characters (e.g., "data\*", "\*data"). * Ngram UDF: Extract character sequences (n-grams) for granular matching (e.g., 3-grams for "data\*"). * Pinot is adding minion tags to isolate tasks. This allows assigning specific tasks to designated minion groups for better resource management and control. [Link](https://github.com/apache/pinot/pull/12459) * The introduction of CLP compression for string columns simplifies data handling by hiding internal encoding details and providing a clean interface for string retrieval. [Link](https://github.com/apache/pinot/pull/12504) * Pinot now lets you define custom logic for merging data during partial updates. This goes beyond merging individual columns, allowing you to control how entire rows are combined for more complex update scenarios. [Link](https://github.com/apache/pinot/pull/11983) * Pinot now optimizes data distribution when adding new server pools. This ensures existing replica groups remain stable while leveraging the new pool for future deployments. [Link](https://github.com/apache/pinot/pull/11953) * Pinot empowers you to ingest data at high rates while maintaining a consistent view of your information. This ensures accurate queries by preventing issues like missing or duplicate primary keys. [Link](https://github.com/apache/pinot/pull/12976) * Pinot gains support for complex data structures during batch ingestion, allowing more flexible data processing for batch and minion tasks. [Link](https://github.com/apache/pinot/pull/12942) * Pinot swallows more data formats! This update allows Avro readers to process int96 data types commonly found in Parquet files, resolving compatibility issues. [Link](https://github.com/apache/pinot/pull/12484) * Pinot now simplifies secure connections by allowing keystore and truststore swaps on the fly. This eliminates the need to recreate entire security contexts, making certificate management smoother. [Link](https://github.com/apache/pinot/pull/12277) * Pinot now handles null values more intelligently when using "mode", “minmaxrange”, “first\_with\_time”, “last\_with\_time”, “percentilles“ (most frequent value) within groups. This update follows PostgreSQL logic, where nulls are ignored during calculation, ensuring more accurate aggregation results. [Link](https://github.com/apache/pinot/pull/12227) * Pinot introduces two new metrics(MULTI\_STAGE\_QUERIES\_EXECUTED, MULTI\_STAGE\_QUERIES\_BY\_TABLE) to monitor multi-stage queries [Link](https://github.com/apache/pinot/pull/12982) * Pinot's table consumption info endpoint is receiving an upgrade. A new field(serversFailingToRespond) now indicates servers that failed to respond during data gathering. This clarifies situations where information might be incomplete due to server issues. As a result, the endpoint provides a more comprehensive picture of data consumption. [Link](https://github.com/apache/pinot/pull/12523) * Pinot gets a serialization upgrade to Protobuf! This improves stability across versions and ensures compatibility for future changes. [Link](https://github.com/apache/pinot/pull/13221) ## StarTree Cloud ### StarTree Extensions for Apache Pinot * Improved Upsert Performance: * Upsert metadata cleanup is now faster, avoiding delays during critical table state transitions. * A new minion task UpsertSnapshotCreationTask accelerates server restarts and table rebalancing by building upsert metadata quickly. * Streamlined Data Management: Re-streaming allows for table recreation from streaming sources without impacting existing workloads. * Pagination support for large query results: Cursors empower clients to retrieve results in smaller portions and navigate forward/backward within the result set while querying data in Pinot. * Enhanced Monitoring and Control: * New metrics provide deeper insights into various ingestion tasks, including task duration, data ingested, input file size, and import failures (for alerting). * The desiredSegmentSize configuration for SegmentImportTask offers more control over segment granularity. * Optimized default configurations for SegmentRefreshTask and FileIngestionTask to improve performance. * Consistent push support for FileIngestionTask simplifies data ingestion workflows. * Advanced Functionality: * Complex data structures (ComplexTypes) can now be handled within the SegmentProcessorFramework. * FileIngestionTask can now skip checkpointing and watermarks for specific use cases. * A configuration-driven SegmentToRowsRatio for checkpoints to prevent unnecessary recalculations during restarts. * Segment refresh can be triggered based on StarTreeIndex configuration changes. * Bug Fixes and Stability Improvements: * FileIngestionTask now gracefully fails when encountering permission issues. * Improved validation for various ingestion tasks ensures data integrity. * Adhoc tasks in SqlConnectorBatchPushTask no longer silently drop tasks due to resource limitations. * Semaphore handling in the checkpoint manager is fixed. * MaxNumRecordsPerTask for DeltaTableIngestionTask is now correctly enforced. * Enhanced Health Checks: * New health checks monitor table-to-broker resource associations and instance pool configurations for large clusters, ensuring overall system health. * Security: * Pinot certificates can now be automatically updated in runtime without downtime, keeping your deployment secure. * Batch Restarts: Pinot receives a restart optimization for clusters with server pools and all tables using instance pools and replica groups. The Pinot operator will now restart servers in batches of 5, expediting the restart process and minimizing disruption during maintenance or upgrades. ### Data Manager * Data Manager gets Workspace with Serverless!: Pinot is integrated with a database management system. This update introduces the concept of "workspaces" for data access control. Workspaces in Data Manager have a one-to-one mapping with the Database in Pinot. * Upsert Efficiency Boost: Off-heap upsert processing becomes the default for all streams, accelerating upsert operations and maximizing resource utilization. On-demand mode in Kinesis is not supported. ### ThirdEye * Never Miss a Resolution: Receive timely Slack notifications when anomalies are fixed. * Personalized Tracking: Get alerts for every anomaly, keeping track of the issues that matter the most. * Organized and Actionable: Use tags to organize Slack notifications and quickly see who resolves each anomaly. * Custom Workspaces: Enjoy personalized workspaces in ThirdEye that combine alerts, notification groups, anomalies, data sources, and datasets for a streamlined workflow. # Data Portal Release Notes Source: https://docs.startree.ai/reference/startree-release-notes/data-portal This page tracks Data Portal release notes. Each entry covers new features and enhancements shipped in that release. ## September 22, 2026 ### New Features * **[Actionable error messages across Data Portal](/corecapabilities/observability/error-codes)**\ Errors now follow a single contract: a stable error code, a plain-language title, the specific cause, suggested remediation steps, and a correlation ID you can paste into a support ticket. Raw technical detail is preserved but collapsed by default, and the innermost cause is no longer dropped when an error is wrapped — so an S3 region redirect or an Iceberg authorization failure reads as exactly that instead of a generic indexing failure. Every error code maps to a published troubleshooting guide in the error code reference, so you can look up what a code means and how to resolve it without opening a ticket. * **Query Insights**\ A new view over the cluster's query log shows total queries, error count and error rate, and p50/p95/p99 latency for a selected time window, filterable by table and query engine. Top slow and top expensive queries are grouped by query template so parameterized queries collapse into a single row, errors break down by class — timeouts, execution failures, planning failures, quota rejections — and any logged query can be sent straight to Query Analyzer. A dedicated tab lists the queries currently running on the cluster. Available when Query Logger is enabled on the cluster. > [!NOTE] > > **Coming soon: AI Insights.** An AI-powered proactive query analyzer will review the queries run on this cluster in the background and surface meaningful insights, such as recurring slow patterns, missing indexes, and tuning opportunities across your workload, without you having to pick a query first. * **GA for Connection Lifecycle Management**\ Editing and deleting saved connections is now enabled for everyone rather than sitting behind a feature flag. Connection details also list every table using a connection, instead of showing only the first. ### Enhancements * **Query History with usage statistics**\ Query History now summarizes what has been running — volume, success and failure counts, and latency — rather than listing raw entries, making it faster to spot a regression in your own query workload. * **Reload checks explain why a reload is needed**\ Table health now reports the specific reasons a reload check is failing and what a reload would change, instead of only flagging that one is required. * **Star-tree index size and tiered storage in Segment Analyzer**\ Segment metadata now reports star-tree index size, and segments on tiered storage report their index information instead of showing gaps. * **Tasks moved under Monitoring**\ Tasks has moved from the Data & AI group to the Monitoring group in the main navigation, next to the other operational views. * **Cleaner state after table deletion**\ Deleting a table now clears its health check state and saved preferences, so a table recreated with the same name starts clean. ### Bug Fixes * **Table preview on mixed-type JSON columns**\ Schema inference now widens the inferred type across sampled values instead of locking to the type of the first one, fixing intermittent 500s when previewing JSON data with mixed types. * **Empty and duplicated error toasts**\ An error with no matching handler now renders a cause, a details control, and a reference instead of an empty toast, and Query History no longer shows a raw 403 toast on top of the in-page permission notice or renders failed statistics as zeros. *** ## August 27, 2026 ### New Features * **Overview tab on the Table Health Dashboard**\ The Table Health Dashboard now opens on an Overview tab summarizing the state of each component at a glance, so you can see where to look before drilling into individual tables and checks. ### Enhancements * **Better compression by default in generated table configs**\ Table configs generated during onboarding now default RAW (no dictionary) dimension columns to ZSTANDARD compression, which compresses variable-width strings, JSON, and bytes payloads materially better than the previous default — a direct win for logging-style tables and tiered storage. * **Storage Breakdown with no indexes**\ The storage breakdown now shows a "no index found" message instead of breaking when a table has no indexes. ### Bug Fixes * **IAM role details and sync status for External Table connections**\ Fixed connection lifecycle gaps where the IAM node role and ARN were not populated, and where associated-table lookups misreported External Table sync status. * **StarTree indexes on timestamp index derived columns**\ Table configs that reference timestamp-index derived columns in the StarTree index or in query expression overrides now save from Data Portal, instead of failing validation and forcing a REST API workaround. *** ## August 11, 2026 ### New Features * **Monitoring dashboard for External Tables**\ Table details now open with a Monitoring tab and an at-a-glance ingestion summary under the table title: sync status, data-synced progress, data synced till, files pending, records synced, and the next scheduled run, auto-refreshing every 15 seconds. External Tables get a clear sync-state model — healthy, lagging, running, initial sync, or failed — showing exactly how far queryable data trails the upstream catalog, with one-click Sync Now or Retry Sync and a "not ready to query" indicator until the first snapshot lands. * **Apache Polaris, Snowflake Open Catalog, and Horizon Catalog support for External Tables**\ External Table onboarding now supports Apache Polaris, Snowflake Polaris (Open Catalog), and Snowflake Horizon Catalog as first-class Iceberg REST catalog options, with no manual API workarounds. * **Segment-level storage drill-down**\ Building on Table Storage Usage from earlier releases, the table details page now includes segment and server instance views, so you can drill into which segments and indexes consume how much storage — and where they live — right from the Storage section. * **IAM role–based access for External Table metastore and storage**\ Configure metastore and storage access for External Tables using IAM roles across all AWS-backed catalogs and S3 data lake sources, instead of static credentials. Setup screens now display the External ID, StarTree account ID, and cluster node role you need when creating the trust policy. * **PromQL support in Query Console**\ Query Console now supports PromQL alongside SQL, and time-series query results render as interactive graphs — so you can explore metrics-style data visually without leaving Data Portal. * **In-product documentation for External Tables**\ The documentation panels opens inline in the External Table creation flow, so you can read about each step without leaving the UI. The panels include links to the full documentation for deeper dives. ### Enhancements * **Query History and Running Queries in the main navigation**\ Query History and Running Queries have moved out of the Query Console's inner tabs into the primary navigation under Query, making it much quicker to check query volume, success rates, and slow queries. * **Query Analyzer effort modes**\ Choose between Low and High effort when analyzing a query — Low for fast feedback, High for deeper analysis with query rewrites, rewrite repair, and scope reduction. * **More index types for External Tables**\ External Tables now support configuring FST, IFST, StarTree, and Timestamp indexes from the UI, extending the index coverage added in recent releases. * **Storage usage for External Tables**\ Table storage usage is now available for External Tables, following performance optimizations to the underlying storage-usage API. * **Full detail view for saved connections**\ Existing Connections now offers a per-connection details view listing every configured parameter with copy buttons and a JSON view. Credentials stay masked until you explicitly reveal them. * **Better-guided External Table onboarding**\ Building on last release's onboarding cleanup, setup screens now surface provider-side context such as AWS Glue database descriptions, documentation links appear throughout the creation flow, and system schemas like `information_schema` no longer clutter the namespace dropdown. * **Task failure causes surfaced first**\ Task details now open on the generation logs tab by default, so root causes like disk-usage breaches or segment-count limits are visible immediately instead of buried behind the task list. ### Bug Fixes * **Easier way to configure Cluster config keys**\ Adding and removing keys from the cluster configuration in Data Portal is now more intuitive, with a single Add button and a Remove icon for each key instead of a confusing two-step process. * **Grafana charts match task execution time**\ Grafana charts for completed tasks now show the task's actual execution window instead of the current time range. * **Faster, more reliable sign-in**\ Fixed an authentication issue where loading auth details could stall for over two minutes or return empty configuration, causing "Unable to authenticate" errors and retry loops at login. * **Delta Lake ingestion no longer fails with Delta Kernel enabled**\ Fixed a NullPointerException during Delta Lake ingestion when the Delta Kernel reader is enabled. * **Warnings instead of errors for valid queries**\ Syntactically correct queries no longer display a misleading error message before running — they now show a warning where relevant. *** ## July 20, 2026 ### New Features * **[Unity Catalog support for External Tables](/corecapabilities/external-table/unity/onboarding-data-portal)**\ External Tables can now connect to Databricks Unity Catalog — including Delta Lake tables exposed through it — broadening catalog coverage beyond the existing Iceberg REST, Glue, and S3 Tables options. * **[GCS Parquet External Tables](/corecapabilities/external-table/gcs/onboarding-data-portal)**\ Create External Tables over raw Parquet files in Google Cloud Storage, alongside the existing Amazon S3 data lake support. * **OAuth2 authentication for Iceberg REST catalogs**\ Connect to Iceberg REST catalogs like Databricks Unity Catalog, Polaris using OAuth2 client credentials, for least-privilege, service-principal access instead of static keys. ### Enhancements * **Glue and S3 Tables promoted to top-level catalog options**\ When creating an External Table, AWS Glue and Amazon S3 Tables now appear as top-level catalog choices instead of being nested under a generic Iceberg REST grouping, matching how you actually think about them. * **Cleaner External Table onboarding**\ Building on the onboarding cleanup from recent releases, terminology and labels have been tidied throughout the flow, and the bucket name and prefix fields now include placeholders that make the expected input format clear. * **Index hints in Query Console**\ Query Console now flags queries that run against tables with no indexes, or that scan a large share of rows, and points you to add indexes or run Query Analyzer for AI-powered recommendations. * **More accurate External Table ingestion progress**\ For Iceberg External Tables, ingestion progress and the last synced checkpoint are now derived from Pinot's byte-based lag signal for a truer progress reading. ### Bug Fixes * **Query Analyzer no longer discards in-progress analysis**\ Closing the query editor while an analysis is still running now prompts for confirmation first, so you don't lose results you're waiting on. *** ## July 10, 2026 ### New Features * **[Table Health Dashboard & Alerting](/corecapabilities/observability/table-health-and-alerting)**\ A new real-time health check dashboard shows every table's pass/fail status at a glance, with drill-down into which checks are failing and why, plus one-click remediation where available. Subscribe an email address to alerts for check failures and recoveries, scoped as broadly as every table or as narrowly as a single check. * **Query history in Query Console**\ Query Console now shows the historical run list for every query you've executed. Query Analyzer draws on that same history to surface slow or problematic queries automatically, without any extra instrumentation. * **Workspace-level control over AI features**\ Workspace and system admins can now enable or disable AI-powered features — including Query Analyzer — for an entire workspace, instead of relying on a single global switch. * **StarTree Index support in Data Portal**\ StarTree Index is now available as a configuration option when creating or editing an External Table, alongside other index types. * **[AWS Glue Schema Registry Avro decoder](/corecapabilities/ingestdata/adv-concepts/realtime/decoders/glue)**\ Real-time tables consuming Avro-encoded Kafka or Amazon MSK messages can now resolve schemas from the AWS Glue Schema Registry, with transparent handling of schema evolution across producer versions. ### Enhancements * **Sharper Query Analyzer findings** Building on last release's accuracy improvements, Query Analyzer findings are now more human-readable and factor in your cluster's actual topology (segment spread, server sizing). Suggested query rewrites are validated and repaired against the live cluster before being surfaced, and a new safety backstop automatically reduces query scope under resource pressure. * **Step-by-step in-product guides**\ Building on the in-product documentation introduced last release, Data Portal now includes step-by-step guided walkthroughs for key workflows, and the left navigation automatically collapses or expands to make room for them. * **Security Manager moved into Settings**\ Security Manager is no longer a separate top-level nav item — it now lives inside the new Settings page alongside notification preferences and other account-level configuration. ### Bug Fixes * **Consistent search filtering**\ Search filters across Data Portal no longer intermittently fail to filter results. * **Nested column names render in Query Console**\ Query results with nested column names now display correctly instead of failing to render. * **Kafka connection sampling no longer times out**\ Fixed a timeout error when sampling a Kafka stream without credentials during connection setup. *** ## July 1, 2026 ### New Features * **[Task status summary in the task manager](/corecapabilities/observability/minion-task-runs)**\ The task manager displays a count of tasks in each state — total, completed, running, waiting, error, and others — so operators can assess cluster-wide ingestion health at a glance. * **Trigger Purge and File Ingestion tasks from Data Portal**\ Run Purge and File Ingestion tasks directly from Data Portal, with progress tracking, failure highlighting, and minion resource usage linked to Grafana. ### Enhancements * **Locked-down data modeling for External tables**\ For External table-related workflows, the Data Modeling step no longer allows editing columns, adding transformations, or editing the ingestion configuration once a table is created, preventing changes that External tables don't support. ### Bug Fixes * **Clearer RBAC policy error messages**\ Deleting or updating a policy that's still attached to roles now returns the actual reason for the conflict instead of a generic "Request failed" error. * **Readable errors in Query Console**\ Query Console errors no longer get buried under long query text, making the actual error message easier to find. * **Complete, scrollable connection error messages**\ Errors when creating a new connection now display in full and are scrollable, instead of being cut off. *** ## June 12, 2026 ### New Features * **In-product documentation and tooltips**\ Tooltips and inline guidance have been added across Data Portal pages and actions, explaining key fields and workflows without leaving the UI. * **Dictionary-encoded indexes on External Tables** External Tables now support the full range of Pinot indexes — including dictionary-encoded types — and you can add and configure all of them on UI. * **More power to Query Analyzer**\ The Query Analyzer now prioritizes query correctness and draws on richer context — your SQL, table schema, execution plan, and many more guidance — to deliver sharper, more specific suggestions with every analysis. ### Enhancements * **Richer External Table detail view**\ The External Table detail view now shows the next scheduled ingestion run and the underlying connection details (data source and catalog), alongside existing status and progress indicators. * **Preload enabled by default for tiered External Tables**\ Tiered storage for External Tables now enables preload by default, improving query performance on cold-tier data without manual configuration after table creation. * **StarTree index hidden for upsert tables**\ The StarTree index option is no longer shown for upsert tables, where it is not supported. * **Clearer Query Analyzer responses**\ Query Analyzer output is less verbose, focusing on actionable optimization suggestions rather than excessive model detail. * **Analyze Query scoped to highlighted text**\ Analyze Query in the Query Console now runs only on highlighted text in the editor, so you can analyze a specific statement without sending the entire query buffer. ### Bug Fixes * **Cluster Manager config panels load for all cluster names**\ Fixed instance configuration panels returning 404 when the Helix cluster name was not `pinot`. *** ## May 6, 2026 ### New Features * **IAM role authentication for External Tables**\ External Table setup now supports IAM role authentication, so you can connect to AWS data sources using instance or cross-account roles without embedding long-lived credentials in the workflow. * **RBAC-aware Data Portal**\ Data Portal now enforces RBAC permissions across table creation and listing, table metadata views (schema, indexing, connections, and additional config), and Query Console (SQL editor and table list). Users only see and can act on resources their role allows. ### Enhancements * **Streamlined navigation for non-admin users**\ Security Manager, Pinot Components, and Zookeeper are hidden for non-admin users. Data Portal Swagger API docs are now accessible to all users. * **Manual segment reload after configuration changes**\ Data Portal no longer automatically reloads segments after every schema, config, or index change. Rapid concurrent reloads could cause race conditions and failed segment reloads; after making changes, reload segments manually when ready. The UI prompts you to reload once all changes are complete. * **Clearer authentication error messages**\ Authentication failures now show actionable guidance instead of opaque errors. The auth service retries up to three times on transient failures before surfacing a message. ### Bug Fixes * **Chronological sorting for timestamps**\ Fixed time-based sorting across Data Portal — including the Segment Reload Status popup — so entries sort in true chronological order. *** ## April 20, 2026 ### New Features * **Scheduling for External Table ingestion**\ Building on Run Now and Pause Ingestion from the previous release, External Table tasks now support configurable ingestion schedules via the minion task schedule API. Pause and resume ingestion by adding or removing the cron schedule from the task configuration. * **Visibility into Table Storage Usage** The interface displays table size, segment count, and historical growth patterns, helping operators monitor capacity and understand how tables are growing. This makes it easier to identify tables consuming significant storage and plan for capacity needs. ### Enhancements * **Input guardrails for External Table configurations**\ Added validation guardrails to protect against bad inputs and parameters during External Table setup, reducing the risk of misconfiguration. Includes safety checks for user parameters, detection of bad schemas or unsupported datasets, credential validation, and proper error codes with response messages. * **RBAC: GetTenant added to default roles**\ The `GetTenant` permission is now included in default RBAC roles, resolving an issue where table creation could fail at the final step due to missing tenant access. * **RBAC: GetRunningQuery permission missing from default read policy**\ Fixed an issue where users with read-data permissions could not view running queries due to a missing `GetRunningQuery` permission in the default policy. The authentication check now includes the workspace in the resource identifier as per the allow rule. ### Bug Fixes * **Unsaved changes lost during table creation**\ Fixed a bug where pending schema or configuration changes were silently discarded if not explicitly applied before proceeding to the next step in the table creation wizard. ## March 25, 2026 ### New Features * **[Query Analyzer](/corecapabilities/ai/query-analyzer/overview) integrated into the Query Console**\ The Query Analyzer is now embedded directly in the Data Portal Query Console. Click **Analyze Query** on any query to get AI-powered insights — optimization suggestions, stage-level execution details, and plan analysis — without leaving the console. This is a beta feature and is disabled by default. Contact your StarTree account team to have it enabled for your environment. * **Share saved queries via a reference URL**\ Saved queries can now be shared with teammates using a shareable reference URL. The query detail view also tracks and displays who created and last modified the query. * **Default instance role authentication for S3 Data Lake source**\ When an S3 bucket is in the same AWS account as the cluster and the cluster role already has direct access, you no longer need to configure explicit IAM keys or a cross-account IAM role. A new default authentication mode automatically falls back to the cluster's instance role, reducing setup overhead for same-account S3 configurations. ### Enhancements * **Ingestion observability for External Tables**\ The External Table detail view now surfaces rich ingestion state: status (`RUNNING`, `COMPLETED`, `FAILED`, `IDLE`), next scheduled run, ingestion progress (segments generated vs. total discovered files), the last synced file checkpoint, and any error messages. This makes it significantly easier to monitor and debug External Table ingestion without external tooling. * **Addition of StreamNative Kafka connector support**\ The StreamNative Kafka connector has been added to the Data Portal, allowing users to connect to Kafka clusters and ingest data into Pinot tables. * **Clearer error messages when task details are unavailable**\ When task details for a specific table cannot be found (for example, when no active task exists), the task manager now shows a meaningful, user-friendly message instead of surfacing a raw 404 error. *** ## March 11, 2026 ### New Features * **Raw S3 Parquet data lake support (Beta, enabled on demand)**\ Data Portal can now help connect Pinot Query Engine to S3 Parquet datasets as a S3 data lake source. Point to an S3 location and query it using Pinot Query Engine. * **Iceberg REST catalog support — AWS Glue and S3Tables (Beta, enabled on demand)**\ Query Iceberg tables using Pinot Query Engine. Data Portal now supports connecting to Iceberg REST catalogs backed by AWS Glue and AWS S3Tables, using REST endpoints with AWS SigV4 authentication. ### Enhancements * **Dark mode**\ Data Portal now supports dark mode, improving usability in low-light environments and for those who prefer a darker UI. * **Swagger UI for Data Portal APIs**\ The Data Portal Swagger UI is now exposed alongside Pinot's Swagger, giving you a single place to browse and test Data Manager APIs. You can now use this to manage connections to your data sources. * **Deleted segments no longer appear in Data Portal**\ Fixed a discrepancy where segments deleted from an offline table could still appear in Data Portal. Data Portal now correctly reflects the empty state after deletions. ## February 18, 2026 ### New Features * **Configure tiered storage directly in Data Portal**\ Introduced support for configuring Pinot tiered storage from Data Portal. Users can enable or disable tiered storage, configure tiers following [documented best practices](/corecapabilities/manage-data/set-up-tiered-storage/setup), and rely on sensible defaults where appropriate. * **Broker and server tenant configuration in the UI**\ Added the ability to configure broker and server tenants for tables directly from Data Portal, removing the need for manual edits to accomplish cluster-level configuration. * **Retention time configuration**\ Introduced controls to set and adjust table retention time (defaulting to 10 years), giving users direct control over data lifecycle policies without leaving the UI. * **MAP data type — end-to-end support**\ Added full support for the Pinot `MAP` data type across the stack. The Data Modeling UI lets users define MAP fields with key/value specs, validation rules, and index constraints (JSON index supported; some index types restricted). Query Console renders MAP column values as JSON for easier inspection. ### Enhancements * **Saved queries no longer auto-run on click**\ Saved queries are no longer executed automatically when clicked. Users can now open a saved query to review or edit it before choosing to run it, reducing accidental execution of heavy queries. * **Support for newer Kafka consumer implementation**\ Updated the default Kafka consumer implementation in Data Portal to use the newer `KafkaFactory` class from Apache Pinot. This aligns with current best practices and improves compatibility with newer Kafka and Pinot deployments. ### Bug Fixes * **Long double values no longer show as "Invalid Number"**\ Fixed an issue where certain long double values (for example, `97.60000000000001`) were incorrectly displayed as "Invalid Number" in query results. These values now render correctly. * **Stale ingestion status in the Data Portal UI**\ The table ingestion status could appear outdated even while ingestion was running. The table details view now refreshes more frequently, giving users an accurate, up-to-date status. * **Pinot Preview API hanging on S3 files**\ Resolved an issue where the Pinot Preview API could hang when previewing data from S3. The preview experience for S3-based sources is now more reliable. * **Clearer permission error for running queries**\ Improved error handling in Query Console when a user is missing the `getRunningQueries` permission. Queries continue to run successfully with clearer, less disruptive messaging instead of a confusing 403 popup. *** ## January 15, 2026 ### New Features * **Protobuf descriptor file support for real-time tables**\ When creating real-time tables with Protobuf data, users can now point to a descriptor file (for example, a file on S3) instead of requiring a schema registry URL. This aligns Data Portal with common Protobuf deployment patterns and removes the need for manual overrides. * **Cancel queries from the editor**\ The query editor now attaches a unique client query ID to each execution, enabling cancellation of that specific query mid-flight without going through a separate API call. ### Enhancements * **Text index support in index configuration**\ Extended index configuration to include text index options via Pinot APIs, improving query performance and search flexibility on free-text fields. *** ## January 9, 2026 ### New Features * **Query observability**\ Operators can now see all running queries alongside their metadata and total runtime via Pinot's query APIs, filling a key gap for managing long-running or expensive queries. ### Enhancements * **Richer Table List view**\ The Table List now surfaces operational details at a glance: ingestion status with a clear definition of "healthy", disabled table indicators, data source details (bucket, topic, or saved connection name), table size, and segment count. Real-time tables that have no scheduled next run show "N/A" rather than a misleading value. * **Inline IAM setup guidance for S3 and Kinesis**\ The S3/Kinesis connection flow now includes sample IAM and trust policies with placeholders for StarTree and bucket details, plus a link to the [full IAM role setup guide](/corecapabilities/ingestdata/recipes/iam-role-s3). Users get step-by-step guidance without leaving Data Portal. * **Group claims visible in the user menu**\ When RBAC is enabled, the user menu now shows the group claims from the authentication token alongside the user's email. A summary of groups is shown by default with an option to expand all. When RBAC is off or the token carries no group claims, the UI handles this gracefully. * **Stale error messages cleared on retry**\ When a user retries an action after a failure, the previous error toast is now dismissed before the new attempt resolves. This prevents stale errors from lingering and makes the outcome of each retry clear. *** ## December 17, 2025 ### New Features * **Visual query execution stats and EXPLAIN PLAN**\ Building on the stage-level stats introduced in the previous release, Query Console now surfaces EXPLAIN plan visualizations alongside execution stats for all query types, making performance debugging significantly easier. * **Cleaner execution details with a dedicated JSON tab**\ Execution details are now organized into tabs, with raw JSON in its own tab and the table widget standardized across the app. * **Type-ahead search in all dropdowns**\ Every dropdown in Data Portal now supports inline filtering — type a few characters to narrow a long list of options. Loading indicators appear inside the dropdown while data is being fetched. ### Bug Fixes * **Transformation rules consistently enforced across views**\ Fixed an inconsistency where transformation functions could be applied to existing fields in JSON View even though Table View blocked it. Validation is now enforced at the API level regardless of which view is used. * **Query execution metadata no longer returns empty values**\ Resolved an issue where the Query Console SQL API sometimes returned zero or empty values for query timings and stats. *** ## December 10, 2025 ### New Features * **Stage-level stats for multi-stage engine queries**\ Query Console now includes `stageStats` in the JSON response and a visual representation of the stage stats when running queries with the multi-stage engine. This gives users detailed per-stage execution stats for performance tuning and troubleshooting without needing a separate tool. ### Enhancements * **Smarter segment builder defaults based on schema width**\ Data Portal now automatically sets `columnMajorSegmentBuilderEnabled` based on table schema width — disabled for tables with fewer than 40 columns, enabled for wider schemas. Commit-time compaction defaults to off, with the option to enable it when turning on Upsert during table creation. This avoids a one-size-fits-all configuration and improves out-of-the-box performance. * **Consistent default indexes across all ingestion flows**\ The Upload File ingestion flow now applies the same default index configuration logic used in other ingestion paths. Tables created from uploaded files get sensible indexes automatically, reducing the need for manual tuning after setup. * **JSON and text columns default to no-dictionary**\ JSON and text columns are now marked as "no dictionary" by default, with JSON index enabled where appropriate. This prevents the segment bloat that occurred when these column types were incorrectly assigned a dictionary, and aligns default behavior with best practices for JSON and text workloads. *** ## December 4, 2025 ### New Features * **Folder selection for batch sources**\ When adding a batch source in Data Portal, you can now browse and select folders directly using a folder picker instead of typing paths manually. This makes source configuration faster and reduces the chance of path errors. * **Partial results in Query Console**\ When some segments are unavailable, Query Console now displays the partial results alongside a clear error message, rather than showing only an error. ### Enhancements * **Ingestion type renamed from "Standard" to "Append"**\ In the Additional Config section for real-time tables, the ingestion type previously labeled "Standard" is now called "Append". The new label better describes how the mode works and reduces confusion when choosing between Append, Upsert, and Dedupe. * **Ingestion failures surface explicitly** Ingestion flows now default to `"continueOnError": false`, so failures are surfaced immediately instead of being silently skipped. This makes it easier to catch and act on ingestion issues early. * **More reliable file pattern handling**\ The ingestion backend now correctly prioritizes `includeFilePatterns` when present, ensuring ingestion follows the configured patterns and avoids picking up unintended files. *** ## November 11, 2025 ### New Features * **Tableau available as a dedicated client**\ Tableau is now listed as a first-class client in the Clients page, with a tailored connection URL generation flow. This removes the need for manual URL construction when connecting Tableau to StarTree. * **Clients section moved into Data Portal**\ Client connection management has moved from MyApps into Data Portal, so all client setup and configuration is accessible from one place within the main UI. *** # StarTree Cloud Portal Source: https://docs.startree.ai/star-tree-cloud-portal The StarTree Cloud Console is your central hub for managing StarTree environments. From here you can monitor your StarTree cluster, access application endpoints, and configure identity providers. *** ## Getting Started Access to the Cloud Console is invitation-based. When a StarTree team member invites you, you'll receive an email with a link to create your account. Click the link, set a password, and you're in. If you already have an account, use **Log in** instead of creating a new one. *** ## Cluster Overview This section summarizes the current utilization across all the environments for this account. More specifically, it shows the vCPU utilization for each of the Pinot components (Server/Broker/Controller/...) ## Environments After logging in, you land on the **Environments** page — a grid of all environments your account can access. ![Screenshot 2026 09 03 At 12 26 46 PM](https://mintcdn.com/startree/l1COxfC0REaOaAgQ/images/Screenshot-2026-09-03-at-12.26.46-PM.png?fit=max\&auto=format\&n=l1COxfC0REaOaAgQ\&q=85\&s=db064e178e94e38e7b1bf2d6a164952b) Each card shows: | Field | Description | | - | - | | Environment type | How the environment is deployed (e.g., BYOC — Bring Your Own Cloud) | | Cloud provider | AWS, GCP or Azure | | Region and zone | The primary cloud region | | Availability zones | Specific zones the cluster spans | | Created at | When the environment was provisioned | Click any card (or its arrow) to open the environment detail page. *** ## Environment Detail The environment detail page has three tabs: **Pinot**, **App Links**, and **IDP**. ### Pinot Tab The Pinot tab shows all cluster components and their resource allocations. ![Screenshot 2026 09 03 At 12 30 36 PM](https://mintcdn.com/startree/l1COxfC0REaOaAgQ/images/Screenshot-2026-09-03-at-12.30.36-PM.png?fit=max\&auto=format\&n=l1COxfC0REaOaAgQ\&q=85\&s=fd59f4b34846a251f450240c4b1f4775) **Component groups** — Controller, Brokers, Server, Zookeeper, and Minions — are each shown as a summary card with total vCPU, RAM, and Disk. Below each summary card, individual node group cards show the instance count and per-node resources. Click the eye icon on any node group card to open a **View Details** drawer with: * **CPU / Memory / Disk** — exact specs for the node type * **Compute Profile** — the machine series and profile name (e.g., `i4i`, `m6a`) * **JVM Configuration** — heap size and direct memory size * **Autoscaling** (Minions only) — whether autoscaling is enabled, and the minimum and maximum instance counts ![Screenshot 2026 09 03 At 12 28 29 PM](https://mintcdn.com/startree/l1COxfC0REaOaAgQ/images/Screenshot-2026-09-03-at-12.28.29-PM.png?fit=max\&auto=format\&n=l1COxfC0REaOaAgQ\&q=85\&s=9162144b5f37083482e7f43ffc74eca8) ![Screenshot 2026 09 03 At 12 28 41 PM](https://mintcdn.com/startree/l1COxfC0REaOaAgQ/images/Screenshot-2026-09-03-at-12.28.41-PM.png?fit=max\&auto=format\&n=l1COxfC0REaOaAgQ\&q=85\&s=ecd0436800b7cce1520ddde5b3b27922) Minion groups with autoscaling enabled show an **AUTO** badge and a live instance range (e.g., `1 of 1–10`). ### App Links Tab ![Screenshot 2026 09 03 At 12 28 53 PM](https://mintcdn.com/startree/l1COxfC0REaOaAgQ/images/Screenshot-2026-09-03-at-12.28.53-PM.png?fit=max\&auto=format\&n=l1COxfC0REaOaAgQ\&q=85\&s=640a02d824906eb7c9bf88e43655a41d) The App Links tab lists all service endpoints for the environment. Use these to connect your tools and pipelines directly to the cluster. | Link type | What it's for | | - | - | | OSS Pinot | The Apache Pinot UI for query and schema management | | Data Portal | StarTree Data Portal for data set management, Query console and admin operations | | Grafana | Metrics and dashboards for the cluster | | Broker URL | Direct broker endpoint for queries | | Proxy Broker | Proxy broker endpoint for external query routing | ### IDP Tab The IDP (Identity Provider) tab lets you configure external authentication for your environment. Users will be able to log in to your Pinot cluster using your organization's existing identity system. ![Screenshot 2026 09 03 At 12 29 09 PM](https://mintcdn.com/startree/l1COxfC0REaOaAgQ/images/Screenshot-2026-09-03-at-12.29.09-PM.png?fit=max\&auto=format\&n=l1COxfC0REaOaAgQ\&q=85\&s=3aa4aa47327deaa027ea220606fee3e4) **Viewing configured providers** — Each configured provider is shown as a card with its type (e.g., GitHub), client ID, and redirect URI. **Adding a provider** — Click **+ Add Provider** and fill in the form: * **Provider type** — Choose OpenID Connect (OIDC) or GitHub * **Provider Name** — A friendly label for this provider * **Client ID** and **Client Secret** — From your identity provider's app registration * **Redirect URI** — Auto-populated; copy this into your identity provider's allowed callback URLs * **Issuer URI** (OIDC only) — The OIDC issuer URL (e.g., `https://accounts.google.com`) * **Groups Filter** (optional) — Restrict access to users in specific groups You can view, edit, or delete existing providers using the icons on each provider card. *** ## Coming Soon The following features are on the roadmap: **Self-service capacity management** — Scale your Pinot Servers, Brokers, and Minion pools directly from the console without opening a support ticket. Set instance counts and resource profiles on demand. **Usage dashboard** — View vCPU-hour consumption, memory usage, and storage trends over time — broken down by component and environment over time — to understand your resource footprint. # Access Control in ThirdEye Source: https://docs.startree.ai/thirdeye/access-control-in-thirdeye Access to alerts, anomalies, and other resources in ThirdEye can be restricted to particular users and groups of users. This document explains how authorization works in ThirdEye and how to write your own authorization policies for ThirdEye resources. For a specific use-case, see our [RBAC recipe](/thirdeye/how-tos/thirdeye_recipes/rbac-recipe). ## Writing authorization policies ThirdEye talks to a separate authorization service to determine if a user can access a particular resource. The authorization service matches users and resources to authorization policies in order to make that determination. In most cases, ThirdEye resources can be organized into namespaces. Access control policies are applied to all resources in a particular namespace. ![Items](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/access_control/namespace_separation.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=2706b744d221af82e8835929661d8262) You can create and manage authorization policies and view the authorization server's API documentation at `https://MY_ENV.MY_ORG.startree.cloud/auth/swagger-ui/index.html`. Authorization policies have 4 important fields: 1. `policyType` - This can be either ALLOW or DENY. 2. `namespaceSrn` - The namespace that this policy applies to. 3. `priority` - Policies are sorted in ascending order by priority. Smaller numbers take priority over higher numbers. 4. `rule` - The rule to match resources and users for this policy. This is where most of the action happens. ### Example policies: ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:default:europe", "priority": 1, "rule": "action='read'", "description": "Allow anyone to read resources in the europe namespace." } ``` ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:default:europe", "priority": 1, "rule": "action='write' and subject_user_groups CONTAIN 'Europe'", "description": "Allow anyone in the Europe group to write resources in the europe namespace." } ``` ### StarTree resource names The StarTree resource names (SRNs), are used to identify *who* may have access to *what*. The SRNs contain three relevant fields: resource type, namespace, and resource id. Field values are colon-separated and must match the regex `^[a-z0-9-_]+$`. Example SRN for an Alert: ``` srn:zone : thirdeye-alert : default : 15213 ^ ^ ^ ^ Start SRN Resource Type Namespace Resource ID ``` #### Namespaces If a resource is not assigned to a namespace, then the namespace is `default`. You can set the resource's namespace by adding `"auth": {"namespace": "my-namespace"}` to the resource's configuration. Anomalies are automatically assigned to the namespace of the alert or enumeration item that created it. Investigations are automatically assigned to the namespace of the anomaly it's related to. Example Alert: ```json theme={null} { "name": "Views-Threshold-DE", "template": { "name": "startree-threshold" }, "auth": { "namespace": "alerts-de" }, ...additional configuration } ``` #### SRN's for ThirdEye resources | Thirdeye Resource | SRN Resource Type | Example SRN | | - | - | - | | Data Source | thirdeye-data\_source | srn:zone:thirdeye-data\_source:default:105 | | Dataset | thirdeye-dataset | srn:zone:thirdeye-dataset:default:114 | | Alert Template | thirdeye-alert\_template | srn:zone:thirdeye-alert\_template:default:103 | | Alert | thirdeye-alert | srn:zone:thirdeye-alert:default:140 | | Enumeration Item | thirdeye-enumeration\_time | srn:zone:thirdeye-enumeration\_item:1776 | | Anomaly | thirdeye-anomaly | srn:zone:thirdeye-anomaly:default:1252 | | RCA Investigation | thirdeye-rca\_investigation | srn:zone:thirdeye-rca\_investigation:1343 | ### Policy matching When ThirdEye checks authorization, 3 pieces of information are sent to the authorization service: 1. Resource SRN 2. User SRN 3. Action #### Policy filtering First, the auth service searches for all policies in the resource's namespace and the default namespace. The policies are sorted by ascending priority. Policies in the `default` namespace are ordered first, followed by policies in the resource's namespace. For example, if the auth service is checking authorization for resource SRN `srn:zone:thirdeye-alert:regional_analysts_uk:140`, the policies are ordered like this: ```json theme={null} [ { "priority": 1, "namespaceSrn": "srn:zone:namespace:default:default", "rule": "..." }, { "priority": 100, "namespaceSrn": "srn:zone:namespace:default:default", "rule": "..." }, { "priority": 1, "namespaceSrn": "srn:zone:namespace:regional_analysts_uk:default", "rule": "..." }, { "priority": 100, "namespaceSrn": "srn:zone:namespace:regional_analysts_uk:default", "rule": "..." } ] ``` #### Attribute context Next, the auth service creates a context map of attributes used in the policy rules. The auth service maps the `User` SRN to the attributes sent by the OIDC provider when the user logged into StarTree cloud. The auth service puts the user attributes, all the fields of the user SRN and resource SRN, and the action into the context map. Example context map: ```json theme={null} { "action": "read", "resource_namespace_srn": "srn:zone:namespace:default:default", "resource_namespace_srn_entity": "namespace", "resource_namespace_srn_identity": "default", "resource_namespace_srn_namespace": "default", "resource_srn": "srn:zone:dataset:default:pageviews", "resource_srn_entity": "dataset", "resource_srn_identity": "pageviews", "resource_srn_namespace": "default", "subject_namespace_srn": "srn:zone:namespace:default:default", "subject_namespace_srn_entity": "namespace", "subject_namespace_srn_identity": "default", "subject_namespace_srn_namespace": "default", "subject_srn": "srn:zone:user:default:system", "subject_srn_entity": "user", "subject_srn_identity": "system", "subject_srn_namespace": "default", "subject_user_email": "jackson@startree.ai", "subject_user_groups": [ "my-okta-group" ], "subject_user_name": "jackson@startree.ai" } ``` #### Rule matching Finally, the policy rules are executed within the context. The policy rules use a SQL-like syntax where the columns are the context fields and the values are the context values. The first matching policy is applied. Example rule matching the above context: ``` action='read' AND subject_user_groups CONTAINS 'my-okta-group' ``` ## Namespaces for ThirdEye resources Most ThirdEye resources allow you to set the namespace in the resource's configuration, but in some cases the namespace is inherited from a parent resource. Example alert: ```json theme={null} { "name": "MyAlert", "prop1": "val1...", "auth": { "namespace": "my-namespace" } } ``` This table shows how each resource gets its namespace: | Resource | How is the namespace configured? | | - | - | | Data source | Namespace can only be set through the API. | | Dataset | Namespace can only be set through the API. | | Alert Template | Namespace is set in the UI by editing the config. | | Alert | Namespace is set in the UI by editing the config. | | Enumeration Item | Namespace is set in the Alert’s config block that creates the enumeration item. | | Anomaly | Namespace comes from the Alert or Enumeration Item that created it. If the anomaly was created by an enumeration item without an auth config, then the anomaly is assigned to the Alert’s namespace. Namespace can be overridden through the API. | | RCA Investigation | When created in the UI or without an auth config, the namespace is assigned to the Anomaly’s namespace. Namespace can be overridden through the API. | | Other resources | Always in the default namespace. | In all cases: * If there is no namespace, then the namespace is `default`. * You can check the namespace of a resource by GETing the resource. ```json theme={null} { "id": "12345", "prop1": "val1...", "auth": { "namespace": "my-namespace" } } ``` ## Policy enforcement in ThirdEye ### How does Thirdeye respond when someone doesn't have access? #### Unauthenticated requests If a user makes any unauthenticated request, ThirdEye responds with status 401 Unauthorized. #### Requests for a single entity If a user requests a single entity but does not have access to that entity, ThirdEye responds with status 403 Forbidden. #### How does ThirdEye respond to `get all` requests? If a user requests a list of resources, such as all alerts, ThirdEye only returns the alerts that the user can access. If the user cannot access any alert, then an empty list is returned. ### Specific permissions required for each API request | Request | Policy action | Notes | | - | - | - | | **All Resources** | | | | `GET /api/{resource}` | read | Only returns the resources that the user can read. | | `GET /api/{resource}/{id}` | read | | | `GET /api/{resource}/name/{name}` | read | | | `POST /api/{resource}` | write | Resource ID is always 0 when checking write permission for new resources. | | `PUT /api/{resource}` | write | The user must have write access for the resource's current state *and* the new state. | | `DELETE /api/{resource}/{id}` | write | | | **Alerts** | | | | `POST /api/alerts/evaluate` (existing alert) | read | If the request body contains an alert id, ThirdEye evaluates the alert as it is stored in the database. | | `POST /api/alerts/evaluate` (new alert) | write | If the request body does not contain an alert id, ThirdEye evaluates the alert in the request body. | | `POST /api/alerts/validate` | write | | | `POST /api/alerts/{id}/reset` | write | | | **Anomalies** | | | | `POST /api/anomalies/{id}/feedback` | write | | # Alert Configuration and Execution Source: https://docs.startree.ai/thirdeye/alert-configuration The alert configuration uses a JSON configuration that contains all the logic to fetch and analyze data in order to detect anomalies. Here's an example alert configuration from the [getting started tutorial](/thirdeye/getting-started/create-your-first-alert): ```json theme={null} { "name": "my-first-detection-configuration", "description": "", "cron": "0 0 5 ? * * *", "template": { "name": "startree-threshold" }, "templateProperties": { "dataSource": "pinot", "dataset": "complexWebsite", "aggregationFunction": "sum", "aggregationColumn": "views", "max": "15000", "min": "5000", "monitoringGranularity": "P1D" } } ``` *quickstart\_complexWebsite\_alert.json* Let’s examine this configuration key by key. ## Root structure ### Name ```json theme={null} "name": "my-first-detection-configuration" ``` The name of the alert. Keep it short. ### Description ```json theme={null} "description": "Detect anomalies in pageviews." ``` The description of the alert. ### Cron ```json theme={null} "cron": "0 0/1 * 1/1 * ? *" ``` The cron schedule defines how often and when the detection runs. The format used is the 6-7 fields Quartz cron format. See [Quartz documentation](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html#introduction).\ A cron expression builder [is available here](https://www.freeformatter.com/cron-expression-generator-quartz.html). ### Template ```json theme={null} "template": { "nodes": [ .. .. .. ], "metadata": { ... } } ``` The template contains the analysis logic. * `"nodes"` contains the detection logic: a list of operators that fetch, clean and analyze data. * `"metadata"` contains metadata used for root cause analysis, search, and data delay management. The field is called `"template"` because it is meant to be reused. Templates can be defined independently of alerts and reused by multiple alerts. See [create templates](/thirdeye/how-tos/alert/use-templates). Inside the template, you can use variables in the format `${myVariable}`. Variables are defined in the `templateProperties`. ### Template properties ```json theme={null} "templateProperties": { "key": "value", } ``` This section lists the values available for the template. You can see them as environment variables. For instance: ```json theme={null} "template": { "nodes": [ { "type": "AnomalyDetector", "params": { "type": "THRESHOLD", "component.max": "${max}", }}] }, "templateProperties": { "max": 15000 } ``` Will be replaced at runtime by: ```json theme={null} "template": { "nodes": [ { "type": "AnomalyDetector", "params": { "type": "THRESHOLD", "component.max": 15000, // REPLACED HERE }}] } ``` ## Nodes ### Execution Nodes are the tasks that form the detection plan. They are also called plan nodes. Nodes can have multiple inputs and outputs. The inputs of nodes are the outputs of other nodes. ![plan node](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_alert_plan_node.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=c3ca732a5e8735918567df9410bc7cf0) The list of nodes with their input and outputs form an execution DAG, as in this example: ```json theme={null} "nodes": [ { "name": "root", "type": "AnomalyDetector", "inputs": [ { "targetProperty": "current", "sourcePlanNode": "missingDataManager", "sourceProperty": "currentOutput" } ], "outputs": [] }, { "name": "missingDataManager", "type": "TimeIndexFiller", "inputs": [ { "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentOutput" } ], "outputs": [ { "outputName": "currentOutput" } ] }, { "name": "currentDataFetcher", "type": "DataFetcher", "inputs": [], "outputs": [ { "outputKey": "pinot", "outputName": "currentOutput" } ] } ] ``` An input is defined by: * a `sourcePlanNode`: the name of another node * a `sourceProperty`: the name of the output we want in the other node An output is defined by: * an `outputKey`: the name of the output to get from the node. Used when a Node can return multiple, distinct outputs. * an `outputName`: the name by which other nodes can call this output To understand how node dependencies are resolved, consider the 2 first nodes: * The **root** node requires one input. This input is from the plan node **missingDataManager** and has the name **currentOuput**. * The **missingDataManager** node exists, and it returns one output called **currentOuput**: the dependency is correct, **root** depends on **missingDataManager**. **missingDataManager** will be executed before root. The execution DAG is constructed using the following algorithm: 1. Start from the root. 2. Resolve the node dependencies. 3. The node dependencies can in turn have dependencies to be resolved. Repeat 2, recursively. 4. Stop when all node dependencies are resolved. In this example, nodes only have one input and one output. An example of a node with multiple inputs is the `AnomalyDetector` of type `"PERCENTAGE_CHANGE"`. It compares 2 inputs: a baseline table and a current table. See [multiple inputs alerts](/thirdeye/how-tos/alert/multiple-inputs-alerts). ### Nodes behavior Now that you know how the detection plan is built, let’s have a look at what’s happening. #### DataFetcher The plan starts with the `DataFetcher`. Take the time to read the SQL query. ```json theme={null} { "name": "currentDataFetcher", "type": "DataFetcher", "params": { "component.dataSource": "${dataSource}", "component.query": "SELECT __timeGroup(hoursSinceEpoch, '${timeColumnFormat}', ${monitoringGranularity}) as ts, ${metric} as met FROM ${dataset} WHERE __timeFilter(hoursSinceEpoch, '${timeColumnFormat}') GROUP BY ts ORDER BY ts LIMIT 10000" }, "inputs": [], "outputs": [ { "outputKey": "pinot", "outputName": "currentOutput" } ] } ``` The `DataFetcher` executes the given query and returns the result in the output named `currentOutput`. Notice the usage of 2 functions: ```sql theme={null} __timeGroup(hoursSinceEpoch, '${timeColumnFormat}', '${monitoringGranularity}') as ts __timeFilter(hoursSinceEpoch, '${timeColumnFormat}') ``` These functions are ThirdEye macros: they are translated to correct SQL, depending on the SQL engine. They make it easier to build detection pipelines. They are similar to [Grafana macros](https://grafana.com/docs/grafana/latest/datasources/mysql/). * `__timeGroup` transforms a timecolumn in bins of a given granularity, in milliseconds. It makes it easy to group by a timecolumn with a specific granularity. In our example, it transforms the column to bins of 1 day. If we ever want to monitor per hour, we can just change `monitoringGranularity` to `P1H`. * `__timeFilter` is a time range filter. Start and end dates are automatically generated depending on the detection runtime. Refer to the [macro documentation](./operators/data-fetcher.md#macros). #### TimeIndexFiller After data is fetched, it is consumed by the `TimeIndexFiller`: ```json theme={null} { "name": "missingDataManager", "type": "TimeIndexFiller", "params": { "component.timestamp": "ts" }, "inputs": [ { "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentOutput" } ], "outputs": [ { "outputName": "currentOutput" } ] } ``` Imagine you have no data on a Sunday. Because we group by a time column in the SQL DataFetcher, the Sunday line will be missing. This will cause problems in downstream nodes. `TimeIndexFiller` takes care of this kind of problem. Notice there is no parameter for this node. This is because macros used by the `DataFecher` add metadata to result, (granularity, time column, etc…). `TimeIndexFiller` uses this metadata to know how to fill missing points. If you don’t want to use macros, you can still configure the `TimeIndexFiller` manually. #### AnomalyDetector The data finally flows to the `AnomalyDetector` node. ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "THRESHOLD", "component.monitoringGranularity": "P1D", "component.timestamp": "ts", "component.metric": "met", "component.max": "${max}", "component.min": "${min}", "anomaly.metric": "${metric}" }, "inputs": [ { "targetProperty": "current", "sourcePlanNode": "missingDataManager", "sourceProperty": "currentOutput" } ], "outputs": [] } ``` The `AnomalyDetector` is of type `"THRESHOLD"`. There are many other types of detector, such as `PERCENTAGE_CHANGE`, `HOLT_WINTERS`, and so on. Parameters prefixed by `component.*` are specific to the detector execution. `component.timestamp` and `component.metric` tell the detector the column names of the time and the metric. See the documentation for [Anomaly detection algorithms](/thirdeye/concepts-data-alerts-notifications/concepts-anomaly-detection-algorithms) to learn more. Notice that there is no output. Anomaly detectors do output anomalies data, but this data is directly consumed by ThirdEye. ## Metadata This part contains metadata that helps ThirdEye make sense of your alert pipeline. Metadata is used for anomaly filtering in the UI, RCA, and pipeline execution. ThirdEye is designed to give you maximum control over your queries and transformations, but this also means it is not always possible for ThirdEye to infer important information from the pipeline. The metadata fields allow you to explicitly set important metadata. ```json theme={null} "metadata": { "datasource": { "name": "${dataSource}" }, "dataset": { "name": "${dataset}", "completenessDelay": "PT2H" }, "metric": { "name": "views", "where": "browser=chrome" }, "eventContext": { "types": ["HOLIDAY"], "sqlFilter" : "'US' member of dimensionMap['countryCode']" }, "granularity": "P1D", "timezone": "Europe/Paris" } ``` Most of the time, `metadata` values are already used in `nodes`, so it is good practice putting them as template properties like in this example. ### datasource #### name ```json theme={null} "name": "${dataSource}" ``` The name of the datasource to use for RCA. Also used for anomaly filtering. ### dataset #### name ```json theme={null} "name": "${dataset}" ``` The name of the dataset to use for RCA. Also used for anomaly filtering. #### completenessDelay ```json theme={null} "completenessDelay": "PT2H" ``` The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. For instance: if you have a streaming system that can ingest data in 2 minutes, but you often have some data points up to 1 hour late, your **completenessDelay** is 1 hour (`PT1H`). ThirdEye will only run detection on data that is complete. Default value is no delay. This is rarely the case in real life. #### mutabilityPeriod ```json theme={null} "mutabilityPeriod": "P3D" ``` Used to manage data mutability. [Learn more](/thirdeye/concepts-data-alerts-notifications/concepts-mutable-data). In ISO-8601 format. At each scheduled detection job, TE will re-run the detection on the data in the mutable period. This ensures TE detection results are up-to-date with the latest mutations of data. #### dimensions ```json theme={null} "dimensions": ["country", "browser"] ``` List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. #### rcaExcludedDimensions ```json theme={null} "rcaExcludedDimensions": ["os", "device"] ``` List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. `dimensions` and `rcaExcludedDimensions` arguments cannot be used at the same time. ### metric #### name ```json theme={null} "name": "views" ``` The name of the metric to use for RCA. Can be a column name `col1` or a derived metric expression `2*col1 - col2`. #### aggregationFunction ```json theme={null} "aggregationFunction": "count" ``` The aggregation function to use on the metric for RCA. If the metric name is known to ThirdEye, this parameter is optional. ThirdEye will use the default aggregation function of the metric. If the metric is not known to ThirdEye, this parameter must be passed. For instance, you can perform RCA on a count of a dimension column with ```json theme={null} "name": "myDimension" "aggregationFunction": "count" ``` #### where ```json theme={null} "where": "browser=chrome" ``` An SQL filter clause that will be applied for RCA. If you apply some custom filtering in your alert `DataFetcher` query, it is most of the time relevant to apply the same filtering for RCA. Notice that this field accepts `AND browser=chrome` instead of `browser=chrome`. This makes it easy to share custom filter clauses between `DataFetcher` queries and this field using a template property. ### eventContext #### types ```json theme={null} "types": ["HOLIDAY"], ``` A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. #### sqlFilter ```json theme={null} "sqlFilter": "'US' member of dimensionMap['countryCode']" ``` An SQL filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. See filter examples [here](/thirdeye/operators/event-fetcher). ### granularity ```json theme={null} "granularity": "P1D" ``` The granularity of the data when it is passed to the `AnomalyDetector` nodes, in ISO-8601 format. Consider a detection pipeline with a cron that runs every day at 3am. By default, the endTime of the detection pipeline will be 3am. This means the last 0am-3am data will be taken, and put in a day bucket, even though the day is not finished and the data for the day is not complete. This can be avoided by passing a granularity of 1 day (`P1D`). ThirdEye will not take the 0am-3am because it is not a complete day. With *granularity*, ThirdEye only runs detection for periods that are complete. This can be combined with the `completenessDelay` parameter. ### timezone ```json theme={null} "timezone": "Europe/Paris" ``` Timezone to use for the alert. In [TZ database format](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List). Defaults to `UTC`. The timezone is used: * To apply [`completenessDelay`](#completenessdelay) and [`granularity`](#granularity) * By the [DataFetcher](#datafetcher) macros `__timeGroup` and `__timeFilter`. * By the [TimeIndexFiller](/thirdeye/operators/time-index-filler) * By the [AnomalyDetectors](/thirdeye/operators/anomaly-detector), to compute the right boundary of a top-right anomaly * To merge anomalies > [!NOTE] > > Metadata fields are not mandatory. You can start without metadata and update the alert configuration later if you need RCA. RCA is available as soon as the metadata fields are set. # ThirdEye Dimensions Recommender Source: https://docs.startree.ai/thirdeye/cohort-recommender The ThirdEye dimensions recommender uses an API call to recommend a list of dimensions which are metrics drivers. ## Problem Context: Working with a large dataset containing various metrics (measurements) and dimensions (categories that classify the data). Understanding what factors most influence your metrics can be challenging. For example, one might want to know which user demographics (country, device type) have the biggest impact on website pageviews. ## Use Case: The ThirdEye Dimensions Recommender is a tool that helps you identify the key dimensions driving a specific metric. This can be valuable for: * Optimizing marketing campaigns: See which demographics or channels contribute most to conversions. * Troubleshooting performance issues: Identify dimensions associated with unexpected dips or spikes in metrics. * Understanding user behavior: Discover which user segments have the most significant influence on key metrics. ## Example: Analyze website traffic. One wants to know which user characteristics (dimensions) most affect total pageviews (metric). Here's how to use the ThirdEye Dimensions Recommender: * Metric Selection: Specify the metric you want to analyze, for example, by providing its ID or name (e.g., "pageviews"). * Filtering: Define a timeframe (start and end date) for the analysis. You can optionally set a threshold (minimum metric value) or a percentage contribution for the dimensions you're interested in. * Additional Criteria: You can add a "where" clause to filter data based on specific dimension values (e.g., only users from the US) or a "having" clause to filter based on the number of data points (e.g., only dimensions with over 1 million rows). Customization: * Max Depth: Set the maximum number of dimensions the recommender should consider when identifying contributing factors (e.g., country, device, browser version). * Limit: Specify the maximum number of top recommendations to return for each dimension combination. * Run the Analysis: The API will return a list of dimensions that significantly impact your chosen metric within the specified timeframe and criteria. ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/DIMENSIONRECOMMEND.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=a813117ffb503440f5e6187f469a0596) ## API reference The recommender API is accessible at `/api/rca/metrics/cohorts`. Here's an example: ```http theme={null} POST https://thirdeye.yournamespace.domain.com/api/rca/metrics/cohorts accept: application/json ``` This endpoint accepts a JSON payload and can be leveraged using the following parameters. ### Parameters | Name | Description | | - | - | | start | The starting time in epoch milliseconds (millis). Example: 1623110400000 means Tue Jun 08 2021 00:00:00 UTC | | end | The end time in epoch millis . Example: 1623283200000 means Thu Jun 10 2021 00:00:00 UTC | | threshold | Exact value of metric. Dimensions Recommender which contribute more than this will be included. | | percentage | If threshold is not provided, then percentage is used. Example: setting percentage to 10 means that threshold = 10% of overall aggregate. | | generateEnumerationItems (beta) | If set to true, ThirdEye will also try to generate the list of enumeration items in addition to dimensions recommender | | where | This is an additional where clause that can be added to the query in the form of a SQL expression. Example: `""where": "country LIKE 'US%' AND \"device\" = 'phone'""` | | having | Similar to where, having clause is a SQL expression that can be added to the query. Example: `"COUNT(*) > 1000000"` | | maxDepth | The max depth in dimensions that the recommendation engine will dive to. Example: maxDepth = 3 means all dimensions reported will have a max of 3 dimensions. like `country = 'US', device = 'phone', version='0.3.0'`. Default value is 10 | | dimensions | If set, this is the list of dimensions that the dimensions recommender will iterate through. Example: if the dataset has 5 dimensions and ` "dimensions": ["country", "device"]`. ThirdEye will only iterate on these 2 dimensions | | limit | If set, this behaves like top N. The number of results are trimmed to limit and the results are sorted in descending order by contribution. The default value is 100 | #### Metric A metric can be mentioned in multiple ways in the API. #### Using ID ```json theme={null} { "metric": { "id": 12746 } } ``` #### Using name ```json theme={null} { "metric": { "name": "pageviews" } } ``` #### Specifying the column name, aggregation function and table name ```json theme={null} { "metric": { "aggregationColumn": "views", "aggregationFunction": "SUM", "dataset": { "name": "pageviews" } } } ``` ## Payload examples Return recommended dimensions with more than 10% contribution to the overall metric having more than n rows in a given timeframe. ```json theme={null} { "metric": { "aggregationColumn": "views", "aggregationFunction": "SUM", "dataset": { "name": "pageviews" } }, "percentage": 10, "having": "COUNT(*) > 1000000", "start": "1623110400000", "end": "1623283200000" } ``` Return recommended dimensions that are above 40k page views showing only the top 5 for every dimension combination. ```json theme={null} { "metric": { "id": 12746 }, "threshold": 400000, "maxDepth": 3, "limit": 5, "start": "1623110400000", "end": "1623283200000" } ``` # Anomaly Detection Algorithms Source: https://docs.startree.ai/thirdeye/concepts-data-alerts-notifications/concepts-anomaly-detection-algorithms Before diving into algorithms, it is important to understand the principles of good alerting systems. If not done cautiously, alert pipelines can quickly get messy, generate alert spam, alert fatigue, or miss important anomalies. When an alert triggers, it should be easy to understand why. Similarly, if an alert doesn't fire, it should be easy to check and see what happened.\ The more complicated an alert condition becomes, the harder it is to understand and debug (^1) The above is theory that applies well to simple metric monitoring, like CPU usage in computers, but it can be harder to apply for operations and business monitoring, the core use cases of ThirdEye. Business monitoring is full of complex patterns, special events and seasonalities (^2) When designing alerts, you will have to find your place on the quadrant below. ![the more complex the less actionable](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_actionable_complex.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=d9f86fd8d63b0f592d3ce71a35269ca6) In ThirdEye, alert rules tend to be more complex than in other systems, and RCA algorithms help you make sense of the anomalies. This does not mean you should start by complex alerts. Good practices to apply: * Do not try to monitor everything. Think about what is important and what is actionable. * Start simple and iterate(^3) to finetune your alert. * Pareto principle applies. Root for the 80% before going for the very complex 20%. (^4) With this in mind, here is a review of the commonly used detector algorithms in ThirdEye. ## Detection algorithms > [!NOTE] > > Click on a detector name to go to its template documentation. ### [Threshold](/thirdeye/templates/startree-threshold) Simplest method. Detect an anomaly if a metric is above a maximum threshold or below a minimum threshold. Good for signals that are mostly flat, or should not go in certain range.\\ ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_threshold.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=727475bbced70ea72d6b65b0fd1c9135) Pros: easy to configure and understand. Cons: does not manage seasonalities. You have to estimate noise yourself. ### [Mean Variance](/thirdeye/templates/startree-mean-variance) Estimate the mean. Estimate the standard deviation, and consider the standard deviation is caused by noise only. If the value is above `mean + n*std` or below `mean + n*std`, detects an anomaly. Good if your signal is flat with a lot of noise. ![mean variance detection](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_mean_variance.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=8487c8f212480bd07c0520f70e6ed244) Pros: No need to estimate noise. Can adapt if noise changes. Cons: Does not manage seasonalities. ### [Percentage change](/thirdeye/templates/startree-percentage-rule) Compare current timeseries to a baseline. If the percentage change is above a certain threshold, detects an anomaly. A simple way of managing seasonalities. You define the baseline yourself. A common usage is to compare your current value to the value of last weeks. For instance, compare Thursday 8, 8pm to Thursday 1, 8pm. This way, you manage hourly and weekly seasonalities. Most of the time, percentage rule is easy to understand, but because percentage rules use division, it can be sensitive with values close to 0. It can also be sensitive to noise: if you define a percentage change limit of 20%, and the noise is such that it is common to do -+10% around the mean, you will have false positives when you have -10% in the baseline followed by +10% in the observed value. ![percentage change detection](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_percentage.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=f8e565ba6992d98817eb74a2f565947f) Pros: easiest way to manage seasonality. Easy to understand.\ Cons: Sensitive to noise(^5) Sensitive to big trends(^6) ### [Absolute change](/thirdeye/templates/startree-absolute-rule) Compares current time series to a baseline. If the absolute change is above a certain threshold, detect it as an anomaly. An alternative to percentage rule, that works better for noisy and small values. ![absolute change detection](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_absolute.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=71ee4ca99d1db5523847706beea7cd0f) Pros: alternative to avoid percentage rule pitfalls. Cons: the absolute change value is harder to set and understand than a percentage. ### [Exponential time smoothing (ETS) ](/thirdeye/templates/startree-ets) Exponential time smoothing (simple, double, Holt-Winters) methods(^7) are a set of statistical forecasting algorithm. Timeseries are extracted as a sum of trend, seasonality and noise(^8) The algorithm estimates these components. This algorithm performs well for daily data and hourly data. Holt-Winters method is faster than most other ML methods while keeping a very good short-term forecasting performance. The model detects an anomaly when the observed value is too far from the predicted value. The sensitivity can be finetuned. ![ets detection](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_ets.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=598d06bdce5a1bfd3f150c023193fab7) Pros: manages seasonality, trend and noise.\ Cons: as a model that heavily relies on historical data, it is sensitive to concept drift, false trends(^9), special events, bad data, etc. ### [MSTL](/thirdeye/templates/startree-mstl) The MSTL method(^10) decomposes a timeseries in trend, multiple seasonal patterns and residuals.\ Residuals can be seen as containing the noise and the outliers. If a residual goes outside the\ expected residual range, it is considered an anomaly. The complete algorithm is an adaptation of the `R` tsoutliers (^11) algorithm for online detection.\\ ![mstl detection](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_mstl.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=d1f59fb240b184bec4b1f8d40ade374e) Pros: Robust to anomalies in historical data. Easier to finetune than ETS. Can perform better than ETS when there are multiple seasonal patterns.\ Cons: Slower than ETS. Control on how the model behaves is limited. ### [Matrix Profile](/thirdeye/templates/startree-matrix-profile) The matrix profile method(^12) is a "direct" anomaly detection algorithm. It returns an anomaly score, it does not return upper and lower bounds. Points are detected as an anomaly if they exceed a given threshold. The matrix profile method only works well with seasonal patterns. It works well from minutely do daily data. The idea of the algorithm is to compare snippets of the timeseries between each other. If one snippet does not look like any other snippet, it has a high anomaly score. ![matrix profile detection](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_matrix_profile.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=9dd85c20e3e3c71bd85e80b6a5bdaa11) Pros: Very fast. Great at high-frequency monitoring. Robust to anomalies in historical data. Easy to finetune. Cons: only good for timeseries with seasonal patterns. No lower/upper bound are provided, this can hinder the interpretation of anomalies. ### [Remote HTTP](/thirdeye/operators/anomaly-detector) The Remote HTTP detector allows the anomaly detection to be performed by a remote HTTP service. The user can configure the alert to point to a REST endpoint. This endpoint must be able to accept the thirdeye detection payload and respond back with a specific response API. With this detector, you can use your favorite models in any language like R or Python. ## Choosing the right algorithm To pick the correct algorithm, you can follow the decision chart below. Make sure to look at your timeseries first. To display a timeseries in ThirdEye quickly, pick the threshold template. See [How to create an alert](/thirdeye/how-tos/alert/how-to-use-alert-creation-ui). You can change the template easily, ThirdEye has an interactive feedback loop that makes it easy to try different configurations. ```mermaid theme={null} graph TD q3{Type of \nanomalies \nto detect}--Illegal values, \nout of range values---->threshold threshold(Threshold template) q3--Spikes, deviations---->q4 q4{Historical data \n> 1 week?}--NO---->threshold q4--YES---->q5 q5{Historical data \n> 2 weeks?}--NO--->simple_rule simple_rule("Percentage |\nor\nAbsolute |\n") q5--YES--->q6 q6{Seasonal patterns \nin the data?}--NO--->simple_learning simple_learning("Mean-Variance | seasonality=P0D\nETS | seasonalMode=NONE") q6--YES--->advanced_learning advanced_learning{Timeseries granularity}--Weekly or more--->simple_learning advanced_learning--Daily--->daily_hourly advanced_learning--Hourly--->daily_hourly daily_hourly("ETS | seasonality=P7D\nor\nMatrix profile\n") advanced_learning--Minutely--->minutely minutely("Matrix profile | seasonality=P1D") ``` ### Auomated Anomaly Detection Algorithm Recommendation and tuning (Based on data pattern during alert creation) ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AlgorithmRecommender.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=7af533c7690bf45f5d75fa48946049da) ### Going further If you want to implement your own model in ThirdEye, see create a detector documentation. (^1): Similar discussion here [https://netflix.github.io/atlas-docs/asl/alerting-philosophy/#keep-it-simple](https://netflix.github.io/atlas-docs/asl/alerting-philosophy/#keep-it-simple) (^2): This is why RCA algorithms are made available. (^3): See [percentage rule and noise](https://medium.com/@cdecatheu/data-quality-anomaly-detections-that-work-with-thirdeye-8fbce5d2f34a) (^4): It is not a coincidence that the Pareto principle was developed in the context of quality control. See [https://en.wikipedia.org/wiki/Pareto\_principle](https://en.wikipedia.org/wiki/Pareto_principle) (^5): See [percentage rule and noise](https://medium.com/@cdecatheu/data-quality-anomaly-detections-that-work-with-thirdeye-8fbce5d2f34a#8e80) (^6): See [percentage rule and big trends](https://medium.com/@cdecatheu/data-quality-anomaly-detections-that-work-with-thirdeye-8fbce5d2f34a#86ca) (^7): See [https://otexts.com/fpp3/expsmooth.html](https://otexts.com/fpp3/expsmooth.html) (^8): See [https://otexts.com/fpp2/tspatterns.html#tspatterns](https://otexts.com/fpp2/tspatterns.html#tspatterns) (^9): See a [discussion about false trends](https://medium.com/@cdecatheu/data-quality-anomaly-detections-that-work-with-thirdeye-8fbce5d2f34a#cee8). (^10): See [MSTL publiation](https://arxiv.org/abs/2107.13462). (^11): See [tsouliers introduction](https://robjhyndman.com/hyndsight/tsoutliers/). (^12): See [references here](https://www.cs.ucr.edu/~eamonn/MatrixProfile.html) and [introduction here](https://www.cs.ucr.edu/~eamonn/Matrix_Profile_Tutorial_Part1.pdf) # Anomaly Merger Rules Source: https://docs.startree.ai/thirdeye/concepts-data-alerts-notifications/concepts-anomaly-merger-rules Consider an alert that runs with hourly granularity and an incident that last 3 hours. The alert will detect an anomaly 3 times, but it is actually the same incident. ThirdEye merges the anomalies in a single one. By default, the following rules are applied: 1. consecutive anomalies are merged 2. if an anomaly lasts for more than 7 days, a new anomaly is created ## Examples ### Anomaly merge 1. Default behavior ![merging anomalies](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_merge_anomalies_default.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=f40c58d511f153498aac0d231f0aa672) 2. Custom behavior\ You can configure ThirdEye to merge anomalies that have a small gap between them using [mergeMaxGap](/thirdeye/templates/startree-threshold#anomaly-merger).\ For instance, with `mergeMaxGap=PT2H`, anomalies that have less than 2 hours between them are merged together. ![custom merging anomalies](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_merge_anomalies_custom.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=615dae25f5cb9bd1195f0ba26b7df96b) 3. Disable anomaly merger\ You can disable anomaly merger by setting `mergeMaxGap=PT0S`. ![anomaly merger disabled](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_merge_anomalies_disabled.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=0c57f27f043f0ce11312c6811eb95ec3) ### Anomaly max length A new anomaly is created when the maximum length is reached. You can customize this maximum length with [mergeMaxDuration](/thirdeye/templates/startree-threshold#anomaly-merger). ![merging maximum](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_merge_max_length.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=5ae217cef9e62da3f58647c985bcbfc4) # Heatmaps Source: https://docs.startree.ai/thirdeye/concepts-data-alerts-notifications/concepts-heatmaps The dimension heatmap provides a visualization of how the metric - sliced by each dimension - changes when compared to a baseline. By default, the baseline is one week before the anomaly period. The figure below shows an example of dimension heatmaps. The metric has 15 dimensions. Each row represents one dimension, and each cell within the row is a dimension value. The size of the cell is proportional to the number of observations having the cell value. It is called *contribution*. For instance, for dimension `assignees`, around 90% of the observations have the value ``(^1) When there are too many small values, they are grouped into a “OTHER” category. ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_heatmap.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=fb7891e54ef1565cea153e985e531694) The color of the cell is determined by the change. The more important is a change, the more intense is the color. Blue means change up. Red means change down. The change is defined as the percentage point (pp) difference between the baseline contribution and the current contribution. For instance: for dimension `userType`, in the baseline, `Bot` dimension represents 20% of the traffic. In the current observations, `Bot` represents `50%` of the traffic. This is a +30%pp change. `Bot` is deep blue. ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_color_scale.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=f4492d2626f08814b3f2fe9a0758ab8a) (^1): Empty value 90% of time?? This is fine, the example dataset is GitHub pull requests. It just means around 90% of PRs are not assigned. # Mutable Data Source: https://docs.startree.ai/thirdeye/concepts-data-alerts-notifications/concepts-mutable-data In most event systems, data is regarded as immutable, meaning that once it's been written, it can't be altered. This is not always true with Pinot. Data can be mutated in 2 ways: * through [streaming upserts](https://docs.pinot.apache.org/basics/data-import/upsert) * through [automatic segement refresh](/corecapabilities/manage-data/segment-refresh-task) ThirdEye is compatible with these features. ## Principle - anomaly consolidation In ThirdEye, a detection job can be run multiple times on the same timeframe. New anomalies and existing anomalies are consolidated based on the following rules: * if the data has not changed, no action is taken. * if the data has changed and new anomalies are detected, they are created and notified. * if the data has changed and some anomalies no longer exist, they are marked as outdated. They won't be visible in the UI by default, but their investigations and feedbacks are preserved. * if the data has changed and an anomaly is detected on a timeframe where an anomaly already exists: * if the change is *significant enough* , the existing anomaly is marked as outdated, the new anomaly is created and notified. * if the change is not *significant enough*, the existing anomaly is updated with the latest value. It is not re-notified. The new anomaly is dropped. To determine what constitutes a *significant enough* change, you can adjust the `reNotifyPercentageThreshold`and `reNotifyAbsoluteThreshold` parameters. For more information, see the [AnomalyMerger documentation](/thirdeye/operators/post-processor/anomaly_merger). By default, **ThirdEye only runs detection jobs on data that it hasn't analyzed before. It does not consider potential mutations in previously analyzed data.** For example, if ThirdEye analyzed data up until Thursday, at the next scheduled detection job, it will analyze data from Friday. However, there are two methods to make ThirdEye re-process data that has already been analyzed: * Use the `/api/alerts/{id}/run` endpoint. This can be done programmatically or via the [Swagger interface](/thirdeye/how-tos/use-the-api). The detection timeframe can be configured. * Use the `mutabilityPeriod` [alert configuration](/thirdeye/alert-configuration#mutabilityperiod) parameter. At each scheduled detection job, ThirdEye will re-process the data that falls in the mutable period. For instance, if the mutable period is set to 3 days, and ThirdEye processed data up to Thursday, then at the next detection job ThirdEye will run from Tuesday (Friday - 3 days). The following day, ThirdEye will process data from Wednesday (Saturday - 3 days), and so on. ## Use cases ### Segment refresh Data is loaded in daily batch in Pinot. The batch system is complex and has many sources of data. Sometimes, a source of data is broken, resulting in missing data downstream in Pinot. When this happens: * ThirdEye detects the anomalies. ![missing data](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_data_mutability_missing_data.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=49338de9fb81a60c1ac9a0a301696fff) * A data engineer will fix the issue and reload the data. Pinot segments are refreshed, the data is fixed. ThirdEye anomalies are now outdated: ![\_data\_mutability\_te\_outdated.png](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_data_mutability_te_outdated.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=16b4382a31a48638db0989c9bcc6dcea) * The data engineer uses the `/api/alerts/{id}/run` endpoint to re-run the detection. ThirdEye is up-to-date. ![thirdeye up to date](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_data_mutability_te_up_to_date.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=7375fe0472636b479d08db16447a247a) In this case, there is still an anomaly! ThirdEye sends a notification for this new anomaly. ### Streaming upsert use case Data is ingested in streaming. Because upserts is enabled, some rows can be mutated multiple times, up to 2 days in the past. The data is monitored by ThirdEye at a 15-minute granularity. (4 points per hour).\ Sometimes, rows in Pinot are mutated after the detection job of ThirdEye. This means some anomalies can get slightly outdated. To avoid this, a user can set the mutabilityPeriod to 2 days: `mutabilityPeriod=P2D`. This way, at each detection job, the detection will be re-run on the last two days of data, and ThirdEye results will stay up-to-date with the mutated data. # Top contributor algorithm Source: https://docs.startree.ai/thirdeye/concepts-data-alerts-notifications/concepts-top-contributor-algorithm The **Top Contributors** algorithm identifies which combination of dimensions contributes the most to an anomaly. The figure below shows an example summary. ![top contributors table](https://user-images.githubusercontent.com/98617940/198924485-c1601ace-10ad-4aee-bfe3-9491644d4297.png) By looking at the summary, we immediately know that (browser="chrome") contribute most to the overall change. Dimension combinations are ranked by their contribution to the anomaly. ## Algorithm > [!NOTE] > > Below explains the algorithm used in StarTree Enterprise Edition. StarTree Community Edition uses a simpler algorithm that cannot combine multiple dimensions. Original publication from *Data cube algorithm* in [https://engineering.linkedin.com/blog/2020/analyzing-anomalies-with-thirdeye](https://engineering.linkedin.com/blog/2020/analyzing-anomalies-with-thirdeye) ### Selection of nodes from dimension hierarchies This section describes how dimension combinations are selected and used to generate the above summary table. As an example, we will use an additive metric with three dimensions: continent, country and state. The image below shows the dimension breakdown of these dimensions, which forms a tree structure. Note that this figure has omitted some nodes. For instance, FR is a child of Europe with the children of FR omitted. In other words, every leaf of the tree is located at the third level. ![image](https://user-images.githubusercontent.com/98617940/198924606-b738c054-bd65-4e26-8044-1159caf9c4b0.png) In the above tree structure, the name near the circle is a dimension value for the corresponding level. When we drill down to the deeper levels of the tree, the dimension values are appended to the previous ones. For example, the node "North America" represents the data segments whose (continent = "North America"); the node US represents the data segments whose (continent = "North America", country = "US"). The root node does not have any dimension value because it is the aggregation of all data segments. A baseline is chosen - by default 1 week - and two trees are created: one for the anomaly period, one for the baseline. These trees are then compared to find the significant changes. **Defining change significance score** StarTree ThirdEye computes the impact of the change with three factors: change ratio, change difference, and segment contribution. Intuitively, the change ratio measures how big the change is. Change difference measures the unexpected change compared to its parents. In other words, how much surprise the change is compared to its parent’s change. The segment contribution measures the physical impact of the node. More formally, given the baseline and current values of a node n and its parent, the change significance score is calculated as: ![image](https://user-images.githubusercontent.com/98617940/198924698-5c5c88f9-dedd-4ded-80b2-9eeaa1427f14.png) where vB and vC are the baselines and current node value, respectively; r is the expected change ratio between the baseline and current from its parent node, which is defined as *r = (v\_parentC) / (v\_parentB)*; *contribution C* is the contribution of the current node, and *contribution all* is the overall contribution. **Roll-up significance scores** The importance of a dimension d (e.g., country) is defined as the sum of the significance score of all its children (e.g., (country="US"), (country="FR"), and so on). Formally, the dimension importance of d is: ![image](https://user-images.githubusercontent.com/98617940/198924779-b39c5b34-9ed1-4b56-b197-97f290e05139.png) where m is the children's count of dimension d. Afterward, the dimension importance is used to determine the tree structure in the figure below, i.e., the root level is the most important dimension. Finally, each parent node picks the top k (i.e., the summary size) children nodes and we merge the result of parent nodes from the bottom to the top; each merge operation keeps only the top k nodes. **Present data cube changes** Finally with the nodes selected, we can summarize the change using a table. The input of our original problem is two data cubes, in which each row is a node in the 3rd level of the tree. ![image](https://user-images.githubusercontent.com/98617940/198924853-a5795af2-ac0c-45e2-9f51-56a37a93db94.png) Suppose that the above diagram shows the rolled-up results of our baseline and current data cubes, in which the bolded nodes are selected during the rolling-up. The following shows the table to represent the summary: | Continent | Country | State (Province) | Baseline traffic | Current traffic | | - | - | - | - | - | | OTHER | (ALL) | (ALL) | 55 | 77 | | Europe | (ALL) | (ALL) | 25 | 43 | | N. America | CA | (ALL) | 10 | 23 | | N. America | US | OTHER | 15 | 32 | | N. America | US | Calfiornia | 15 | 25 | A node of the tree in the above diagram is either a data segment (leaf node) or grouped data segment (non-leaf node). In the table, the row of a leaf node contains all three dimensions. The row of a non-leaf node contains only one or two dimensions and the remaining dimensions are either (ALL) or OTHER. (ALL) indicates all its children nodes are grouped together, while OTHER means that some children are not included. For example, the node (N. America, US, OTHER) does not include the child (N. America, US, California). # ThirdEye Architecture Source: https://docs.startree.ai/thirdeye/concepts-thirdeye-components/concepts-thirdeye-architecture **High level system architecture of ThirdEye** ![architecture](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/ThirdEyeArchitecture.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=42128c9a9e73f3111c69998cd925154e) ThirdEye Architecture ![Building blocks](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_architecture.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=85f5eae65595ecfba6cb480993e46cbe) ### UI This is the service that serves the web interface. It communicates with the backend. ### Coordinator The **Coordinator** is the entry point of the REST API. The request are (optionally) authenticated and authorized. The coordinator can perform CRUD operations on ThirdEye entities, simulate detection pipeline, and run interactive RCA. ### Scheduler The **Scheduler** runs the task crons and create the task objects. The tasks are persisted in the database and polled by the workers. The 2 main type of tasks are detection tasks and notification tasks. ### Worker **Workers** run the tasks. Here is the execution principle: * workers wait for tasks to be added to the queue * once a worker finds a new task, it assigns it to itself (gets a lock) * on successfully getting a lock, the worker executes the task * upon completion, the worker captures the results and persists them into the db ## Notes UI, coordinator, and worker can be scaled independently based on their respective load. The scheduler is not replicated. # ThirdEye Entities Source: https://docs.startree.ai/thirdeye/concepts-thirdeye-components/concepts-thirdeye-entities This page discuss key topics and concepts at a high level and provide useful background information and explanation to help you understand how ThirdEye works, and make it easier for you to use the platform. The diagram below shows how a new user would interact with ThirdEye: ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_workflow_high_level.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=07edb27ca73caa7df9a2cb98d7ceadda) Let's review the entities and terminology used in this workflow. In ThirdEye, there are 3 main families of entities: *data*, *alerts*, and *notifications*. ## Data entities * **Datasource**\ A connection to a data system. This is where ThirdEye gets the data you want to monitor.\ Example: an Apache Pinot instance. * **Dataset**\ A mapping to a table in a **datasource**.\ Example: the `orders` table. * **Metric**\ A column in a **dataset**, and an associated aggregation function.\ Example: the `revenue` column with the function `SUM` ## Alert entities * **Alert** - also called **detection pipeline**\ A complete anomaly detection rule configuration.\ Example: create an **anomaly** if `revenue` is bigger than `20000`. Check every hour. * **Alert template**\ A detection logic boilerplate that can be used to create *Alert*.\ Example: create an **anomaly** if `${metric}` is bigger than `{max_value}`. * **Anomaly**\ A problem detected by a **detection pipeline**\ This is what you look at when there is a problem. An anomaly has a start time and an end time.\ Example: `revenue` was `30000`, above the threshold of `20000`, on Thursday 3, between 9pm and 10pm. ## Notification entities * **Notification system**\ A system where to send alerts.\ Example: email, Slack. * **Subscription Group**\ A list of notification recipients and a list of alerts. Anomalies raised by the alerts are sent to the recipients.\ Example: email John and Jenny if revenueAlert or userAlert detect an anomaly. # ThirdEye Worker Management Source: https://docs.startree.ai/thirdeye/concepts-thirdeye-components/concepts-thirdeye-worker-management ThirdEye consists of 3 components * Coordinator : serves the APIs for UI * Scheduler : schedules tasks for detection, onboarding, notification, etc * Worker : carries out the execution of tasks scheduled by scheduler As Worker does most of the heavy lifting in terms of computations and resource usage it becomes necessary to have the provision of adding more workers to the setup to distribute the task load. In this section we will discuss how ThirdEye handles the multiple workers and what strategy is in place to distribute the load across all these workers. ## Overview All the components interact with each other indirectly through the MySQL database. For example coordinator creates an alert based on the API call from UI. Scheduler iterates on this alert and creates a job that runs at fixed intervals defined by the cron in the alert. These job runs creates tasks in the database which are then picked by workers to execute. When we introduce multiple workers we need to ensure that * workers don't rerun a task which is already picked by another worker for execution * workers don't overload the database with queries in order to get tasks * load is evenly distributed among the workers * each task is ran to completion and not left unpicked or stuck in bad state ## Why do we need this? To make ThirdEye scalable, consistent and efficient the system needs to be fine-tuned to get the most out of given resources. Thirdeye worker management tries to address: 1. Flexibility in vertically scaling an individual worker 2. Even load distribution among multiple workers 3. Minimal moving parts in the architecture to reduce maintenance and points of failure 4. Disaster recovery during and after worker failures 5. Ability to tune workers based on known load patterns ## Architecture Worker characteristics * Each worker has a unique worker id * The worker achieves parallelism internally using multi-threading * Each worker builds an internal virtual queue which is used to pull tasks for execution ![Worker Architecture](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/worker_arch.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=2094871b170f1cf47d7deee9cd83d1d8) Different knobs are exposed as configurations to fine tune the worker characteristics based on the given use cases. ## Configuring a worker ### Achieve unique worker id The configuration file (`server.yaml`) has a configuration ```yaml theme={null} taskDriver: id: ``` If the ThirdEye setup is deployed on bare metal VMs, we can simply provide unique ids to each worker instance through their configuration file. But if the setup is deployed on a Kubernetes cluster using helm charts, we need to enable the `randomWorkerIdEnabled` flag as the worker replicas will be identical and refer a common worker configuration file. ```yaml theme={null} taskDriver: randomWorkerIdEnabled: true ``` This can also be used in the bare metal setup ### Reduce DB overhead As each worker has parallel threads running internally, we need to set the parallelism based on the expected load on the worker. Overestimating the parallelism may lead to DB overhead as each thread will keep on querying the DB for tasks but won't get any as all the tasks would be picked up already. ```yaml theme={null} taskDriver: maxParallelTasks: ``` Sometimes it may be possible that the task load is not evenly distributed over time. There may be tasks that runs hourly, daily, etc. In such cases we need the parallelism as there is a task load after each hour. In such cases we can increase the sleep time for a thread when the thread is not able to fetch a task ```yaml theme={null} taskDriver: noTaskDelay: # 15 seconds (ISO 8601 standard) ``` We can set it to say PT15M (15 minutes) if we only have tasks being scheduled hourly. ### Avoid worker collision In multi-worker setup, it's possible that more than one worker picks up the same task simultaneously. Normally the conflict gets resolved based on which worker updates the task status first in the database, but ideally it's good to avoid this situation as this leads to unnecessary cpu cycles on worker and database queries on MySQL. This can be achieved by * increasing queue depth to accommodate more tasks -> reduce probability of collision * increase the range of random delay so that workers are out of sync while fetching tasks from db ```yaml theme={null} taskDriver: taskFetchSizeCap: randomDelayCap: # 15 seconds (ISO 8601 standard) ``` # Dimension exploration Source: https://docs.startree.ai/thirdeye/dimension-exploration # Dimension Exploration Dimension exploration gives users the ability to create alerts for every value of a certain dimension or combination of dimensions. This is similar to running a for loop on different sets of values on the same detection workflow. ## Problem description Let’s take a look at the page views example. `pageviews` is a metric which computes the views aggregated by sum over a given time period. You can create an alert that fires if `pageviews` drops by 5% using a single alert. However, what if the user wants to be alerted for a `pageviews` drop of 5% by country? The user would have to create a separate alert for each country. ![problem-statement](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/dx-problem-statement.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=fea57a2b11845015bfaffa96323a3fde) This is the problem that dimension exploration tries to solve by giving the user the ability to write a single alert to monitor a workflow for different values of dimensions, in this example, `country = { ‘US’, ‘IN’, ‘FR’ }`. ## Approach The approach here leverages the flexibility of the detection pipeline to be able to model a ‘loop’ within a workflow. ### Detection pipeline DAG The detection pipeline supports nodes that allow multiple named input and output streams. This section provides a simplified directed acyclic graph (DAG). ![plan node](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_alert_plan_node.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=c3ca732a5e8735918567df9410bc7cf0) This gives us the ability to introduce 3 new primitives/operators. * Enumerator Node * Fork Join Node * Combiner Node ![problem-statement](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/dx-dag1.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=09046a48c34b4827d01abb1dd163decf) ### Enumerator node An enumerator node outputs a set of key-value pairs. These key-value pairs will be used to determine the template variables used for running the ‘loop’. #### Output schema ```java theme={null} List> ``` Each of the items in a list is a map containing template variables and values. Therefore, an enumerator can emit multiple variables and values per iteration. Current template variables in ThirdEye are restricted to strings. This poses a design limitation since there is no way to feed structured data back into the template. An enumerator itself can be of different types. #### Dynamic dimensions using dataframe enumerator (Beta) ThirdEye supports enumerating an alert based on the results of a query. The query can be any valid SQL query that returns a dataframe. The dataframe is then used to enumerate the alert. Here is an example alert payload. ```json theme={null} { "name": "pageviews per country", "description": "This is a sample detection", "template": { "name": "startree-threshold-query-dx" }, "templateProperties": { "dataSource": "pinot", "dataset": "pageviews", "monitoringGranularity": "P1D", "aggregationColumn": "views", "aggregationFunction": "sum", "max": "900000", "min": "100000", "enumeratorQuery": "SELECT DISTINCT country FROM pageviews" }, "cron": "0 */15 * * * ?" } ``` In this case, the `enumeratorQuery` parameter takes a SQL query that returns a dataframe. #### Manual listing using default enumerator The default enumerator node could be configured to emit a series of values configured as-is. This allows the user to pass a set of template properties that can be fed into the child pipeline. In this example, different values of the `queryFilter` property will be fed into the pipeline for each run. ```json theme={null} { "name": "enumerator", "type": "Enumerator", "params": { "type": "default", "items": "${enumerationItems}" } } ``` where `enumerationItems` is a template variable that can be fed in using a JSON like this ```json theme={null} { "name": "my alert", ... other properties, "templateProperties": { "dataSource": "pinot", ... "enumerationItems": [ { "name":"US Only", "params": { "queryFilters": " AND country = 'US'", "max": 300000, "min": 100000 } }, { "name": "Overall", "params": { "queryFilters": "", "max": 900000, "min": 300000 } } ] } } ``` ### Combiner node The combiner node accepts the results of a fork operator and combines all the results. A combiner can have different types. Initial approach would be to have a combiner that combines anomalies from each of the enumerations and tags them accordingly. Example: The expectation here is to consume a list of outputs from each DAG and emit a single consolidated output. Note that each node can have multiple outputs. It is essentially flattening a set of lists into a single list. ```json theme={null} { "name": "combiner", "type": "Combiner" } ``` In this case, we use the default combiner, which should work fine for most use cases. ### Fork join node A fork join node follows a fork + join model which allows the user to input a set of enumerations. The fork join node forks the pipeline to execute a sub workflow DAG for each of the enumerations. #### Operation The Fork Join Node takes 3 parameters: * **Enumerator:** This is an enumerator node that will be used to get all the enumerations that need to be executed * **Root:** This is the root node of the sub DAG that the fork join is supposed to execute feeding in the enumerations * **Combiner:** This is the combiner node that will consume the list of outputs from each enumeration execution. The sub-DAG execution is done in parallel in a threaded execution model to leverage parallelization. Each execution of the fork join consumes an entry from the enumerator. This is what we call an *enumeration item*. ### Enumeration item The enumeration node generates a list of enumeration items. These contain a set of parameters that are fed in the sub pipeline which is executed in the fork join. They define the results of the pipeline—meaning, any anomalies which may be calculated in the sub pipeline need to be attributed to the corresponding enumeration item. For ease of access, filtering and other operations, enumeration items are also persisted in the database as entities. Here is a sample enumeration item entity ```json theme={null} { "name": "US, 1.0", "description": "slice for US only results", "params": { "queryFilter": " AND country='US' AND version='1.0'" } } ``` Anomalies generated with this pipeline is automatically tagged with the enumeration item: ```json theme={null} { // anomaly object "id": 1234, ... "enumerationItem": { "id": 1234 } } ``` # Alert Configuration Examples Source: https://docs.startree.ai/thirdeye/getting-started/alert-create-examples Here are some example configurations which you can copy, change parameters, and quickly create alert in few clicks. ## Example-1 - StarTree Threshold ```javascript theme={null} { "name": "Sample-alert-name", //change this before you try "description": "Using StarTree ThirdEye Threshold alert", "template": { "name": "startree-threshold" }, "templateProperties": { "dataSource": "Sample-datasource", //change this before you try "dataset": "sample-dataset", //change this before you try "timeColumn": "time-column-from-pinot-schema", //change this before you try "timeColumnFormat": "timezone-format-from-pinot-schema", //change this before you try. You can get this info from Pinot schema. Just copy and paste "aggregationFunction": "sample-metric-name", //change this before you try "aggregationColumn": "orders", "monitoringGranularity": "P1D", "max": "500", //Adjust this and preview to get best results "min": "100" //Adjust this and preview to get best results }, "cron": "0 0 0/2 1/1 * ? *" } ``` ## Example-2 - StarTree ETS ```javascript theme={null} { "name": "Sample-alert-name", //change this before you try "description": "Uses template startree-ets.", "template": { "name": "startree-ets" }, "templateProperties": { "dataSource": "Sample-datasource", //change this before you try "dataset": "sample-dataset", //change this before you try "timeColumn": "time-column-from-pinot-schema", //This is optional. change this to overwrite default value. "timeColumnFormat": "timezone-format-from-pinot-schema", //This is optional. change this to overwrite default value.You can get this info from Pinot schema. Just copy and paste "aggregationFunction": "sum", "aggregationColumn": "sample-metric-name", //change this before you try "seasonalityPeriod": "P7D", //Adjust this and preview to get best results "lookback": "P120D", //Adjust this and preview to get best results "monitoringGranularity": "P1D", //Adjust this and preview to get best results "sensitivity": "3" //Adjust this and preview to get best results }, "cron": "0 0 0 1/1 * ? *" } ``` ## Example-3 - StarTree ETS with dimension exploration ```javascript theme={null} { "name": "pageviewsdemo-ets-dx-withdifferentTuningParameters", "description": "", "template": { "name": "startree-ets-dx" }, "templateProperties": { "dataSource": "pinot", "dataset": "PageViewsData", "aggregationFunction": "sum", "seasonalityPeriod": "P7D", "lookback": "P20D", "monitoringGranularity": "P1D", "sensitivity": "${sensitivity}", "aggregationColumn": "views", "queryFilters": "${queryFilters}", "enumerationItems": [ { "name": "overall", "params": { "queryFilters": "", "sensitivity": "1" } }, { "name": "US", "params": { "queryFilters": " AND country = 'US'", "sensitivity": "1" } }, { "name": "chrome", "params": { "queryFilters": " AND browser = 'chrome'", "sensitivity": "3" } } ] }, "cron": "0 0 5 ? * * *" } ``` # Create your First Alert Source: https://docs.startree.ai/thirdeye/getting-started/create-your-first-alert > [!NOTE] > > You must install ThirdEye before you begin. Alerts are the rules you build to detect anomalies. An alert configuration contains all the information necessary to: * Fetch data from a datasource * Preprocess data * Compute baselines * Detect anomalies * Filter anomalies with business rules * Send alerts to your subscription groups In this guide, we'll create an alert that detects when a metric lies outside of a defined range. We will use StarTree Cloud, which includes Apache Pinot hosted as software-as-a-service (SaaS). ## Load a sample dataset To begin, load data into Apache Pinot. Use StarTree Data Manager to load data. 1. Go to your StarTree Data Manager at `data.your.instance.startree.cloud`. 2. Click **Create a Dataset**. 3. In Sample Datasets, click **Complex Website**. 4. Set the name to `complexWebsite`, and then click **Confirm**. The test dataset is now loaded into Pinot. ## Set up the datasource Configure ThirdEye to get access to the Pinot database. Connect ThirdEye to Pinot. In StarTree Cloud, ThirdEye connects to the Pinot database automatically. The first time ThirdEye is opened, a page will guide you through the onboarding of the dataset. Ask StarTree support for help if you cannot see Pinot datasets or fail to onboard. ## Create the alert configuration 1. In StarTree Cloud, open the Thirdeye app. 2. Click **Create Alert**. 3. Copy and paste this JSON in the configuration field: Use this JSON to configure your alert. ```json theme={null} { "name": "my-first-detection-configuration", "description": "", "cron": "0 0 5 ? * * *", "template": { "name": "startree-threshold" }, "templateProperties": { "dataSource": "pinot", "dataset": "complexWebsite", "aggregationFunction": "sum", "aggregationColumn": "views", "max": "15000", "min": "5000", "monitoringGranularity": "P1D" } } ``` *quickstart\_complexWebsite\_alert.json* This configuration uses the `startree-threshold` template. A template contains all the logic to perform a detection, and exposes configuration fields. Properties are configured in `templateProperties`. With this configuration, the following logic is applied: * An SQL query returns the sum of views, grouped by day, from the `complexWebSite` table. * A threshold rule is applied to the query results. If a data point falls outside the thresholds, it is flagged as an anomaly. > [!NOTE] > > For more information about alert configuration, see [understanding detection configuration](/thirdeye/alert-configuration). 4. Below the JSON configuration, in the preview panel, press the refresh button: you will see the result of the detection configuration. Notice that there are a few anomalies. ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_cloud_quickstart_alert_preview.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=8ee1740db1d967188b3c69d02b9e91f8) Try changing the dates on the timeframe selector in the top right corner and the min/max values in the JSON to see what happens. Click the legend fields at the bottom to show/hide timeseries. 5. (Optional) Use the notifications panel below the chart to send anomaly messages to external systems like Slack and email. To create a notification, click **Create Alert**. ThirdEye will guide you through the process, helping you find the right algorithm for your data. Once you get to the end and click **Create**, your first alert is created, and you will be redirected to the alert page. When an alert is created, the detection is rerun on past data. It may take a few seconds, so you may have to refresh the page. ## View an anomaly 1. Click an anomaly to open the anomaly page. ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_cloud_quickstart_alert_page.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=51bf5b4f0249e489f45c583070b8d7c1) 2. Click investigate to open the root-cause analysis page. ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_cloud_quickstart_anomaly_page.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=e37c8f792082caf24b4034db1e152b07) 3. Use the bottom tabs to explore and help you find the root-cause of the anomaly. ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_cloud_quickstart_rca_page.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=af3b5dd88346121859c78075e16abee7) ## Next Steps Now that you have successfully configured your first alert and inspected an anomaly, explore these links to learn more: * To learn how to use root cause analysis features in the anomaly pages, see [root cause analysis concepts](/thirdeye/concepts-data-alerts-notifications/concepts-heatmaps) * To learn more about alert configuration, see [Alert configuration and execution](/thirdeye/alert-configuration). * To see alert examples, see [create alerts how-tos](/thirdeye/how-tos/alert/how-to-use-alert-creation-ui). # ThirdEye Data Requirements Source: https://docs.startree.ai/thirdeye/getting-started/data-requirements StarTree ThirdEye searches through your data looking for data points that are unexpected. For anomaly detection to be effective, your data must be configured in a way that makes sense to the algorithm examining it. This page provides the data schema requirements and configuration to help you use ThirdEye effectively. Time series data may have different scales, ranges, and units. You must normalize your data for it to be balanced in how it affects anomaly detection algorithms. There are three types of columns that are required for ThirdEye to work; a timestamp, a metric, and a dimension. Further considerations are listed after a presentation of details for each of these followed by examples. ## Data readiness and validation checklist * Timestamp needs to be in epoch millis for ThirdEye to work efficiently. If there is a non-epoch millis column then convert it to epoch millis using the “derived column” and apply time range indexes based on time granularity used in ThirdEye (Daily, hourly, 15 mins, 1 min). * Cannot have data gaps [Link to completeness delay](/thirdeye/alert-configuration#metadata), [definition](/thirdeye/alert-configuration#completenessdelay), [Link to external blog how to do DQ check using ThirdEye](https://medium.com/the-ab-tasty-tech-blog/data-quality-timeseries-anomaly-detection-at-scale-with-thirdeye-468f771154e6) * Based on Query patterns from ThirdEye to Pinot - apply those specific indexing. * Joins are not supported in ThirdEye. Make sure derived columns pre-ingestion are created for those to support TE use cases. * ThirdEye needs a denormalized schema for the following reasons: * TE can run any SQL query Pinot can support for just running alerts. We can do this with a custom template. [Link to ThirdEye templates](/thirdeye/templates/startree-absolute-rule). One can clone this to create a custom template and update the dag in it (query against Pinot). However, all the templates need to be cloned to add this custom query) * The complexity is more on: how the metric is then defined, how people can choose different detection algorithms on it, how it is exposed in the UI, how that is shared in RCA, notifications, etc., alerts might work, but the entire experience is not great. RCA may break. If RCA top contributors, heatmap needs to provide accurate insights at a quicker speed then transformed data will be ideal. * If there are too many dimensions in the table how to make RCA more performant? Ans: Number of dimensions in a schema - indexes on dimensions (too many dimensions will slow down RCA UI) - rcaExcludedDimensions [Link to the doc](/thirdeye/alert-configuration#rcaexcludeddimensions) ## Timestamp This field is vital for detection accuracy. ThirdEye works with seasonal or trend-based patterns, but does not perform well with random or cyclic data. It also requires at least 30 days worth of data for granularity in the hourly/daily range. ThirdEye works with timestamps that have: * One column, with type of 'DateTime' * When this column isn't set, the timestamp is automatically set as the start time of each ingestion period * A format that Pinot can use, see [DateTime strings in Pinot](/recipes/datetime-string-to-timestamp#datetime-strings-in-pinot) for more information. ## Metric A metric is a quantifiable value. With ThirdEye, metrics can be: * One or more columns containing numeric or categorical values in the data feed. For each data feed, you can specify multiple metrics, but you must select at least one column as `metrics`. * Directly configured during alert configuration and need not be a column in the dataset, such as events streams on which you can use count metric. If you want to use `SUM` to aggregate your data, make sure your metrics are additive in each dimension. Fraction-based metrics are non-additive. This includes ratio, percentage, and so on. You can use pre-aggregated metrics or configure aggregation methods in ThirdEye and run it during anomaly detection at run time. If the Pinot data has correct indexes and data volume is not too high then you will get sub-seconds results for aggregated queries. ## Dimension A dimension is one or more categorical values. For ThirdEye: * A combination of different values identifies a particular single-dimension time series, for example: country, language, tenant. * You can select zero or more columns as dimensions. * The dimension columns can be of any data type. Avoid using large volumes of columns and values to prevent excessive numbers of dimensions from being processed, which is very slow. ## Further considerations If you are doing real-time anomaly detection, verify the logical steps done correctly. These are: * Get event * Write event * Query historic events (requires at least 30 days' worth of data) * Run anomaly detector You want to ensure missing data is handled well or use filters in anomaly detection configuration do avoid that becoming a problem. Find the right granularity of your data. * When data granularity is very high, you have more data, but it will become noisy * As your data granularity goes down, it becomes tough to detect anomalies * Pay attention to consistency, use the same granularity across data so you don't have one source being recorded with a granularity of 1 minute and another with a granularity of 5 minutes. ## Examples ### Revenue metrics | Timestamp | Category | Market | Revenue | | - | - | - | - | | 2020-6-1 | Food | US | 1000 | | 2020-6-1 | Apparel | US | 2000 | | 2020-6-2 | Food | UK | 800 | In this table, the: * Metric is Revenue * Dimensions are Category and Market * Timeseries pattern is daily ### Application errors | Timestamp | Application component | Region | Error count | | - | - | - | - | | 2020-6-1 | Employee database | WEST EU | 9000 | | 2020-6-1 | Message queue | EAST US | 1000 | | 2020-6-2 | Message queue | EAST US | 8000 | In this table, the: * Metric is Error count * Dimensions are Application component and Region * Timeseries pattern is Daily # Getting Started with ThirdEye Source: https://docs.startree.ai/thirdeye/getting-started/free-tier-tutorial This tutorial is a great place to start if this is your first time using ThirdEye. We’ll be using a sample PageViews dataset to explore all the different features and tools of ThirdEye. ### What you will accomplish in this tutorial 1. Adding a sample dataset to ThirdEye 2. Creating an alert on a metric in the sample dataset 3. Configuring anomaly notifications 4. Viewing an anomaly 5. Performing a root cause analysis ### Prerequisites To start, [contact StarTree](https://startree.ai/demo) for details on getting ThirdEye. ## Bringing your data into ThirdEye When you access ThirdEye for the first time, you’ll be taken to a welcome screen. This welcome flow will walk you through the two primary steps to get you started with Thirdeye. The first step is to connect your data source and onboard your datasets to ThirdEye. On the first screen select StarTree Cloud, as we’ll onboard a dataset from the Apache Pinot database from the trial. Next, select the ‘eCommerce Website PageViews’ dataset under the Sample Datasets section. ThirdEye will add this dataset to the StarTree Cloud Apache Pinot database for you. The dataset is a sample dataset of an ecommerce company that is recording versions and views of its checkout page. This is a key metric for the ecommerce company that uses this as a marker to track sales and revenue of the business. ## Creating your alert Next after onboarding your dataset, select the second step to create your first alert. You’re taken to an alert wizard that will guide you through the steps to create an alert. > [!NOTE] > > What is an alert?\ > Alerts are the rules you build to detect anomalies in your metrics. ThirdEye gives you a variety of > templates and algorithms to model your metric and its patterns. When the metric falls out of the range of the model you select, ThirdEye alerts you of the anomaly. Select the dataset we onboarded ‘demo\_pageviews’, then select ‘views’ as the metric. ‘views’ is one of the columns in the dataset that we will monitor as a metric for anomalies. > [!NOTE] > > What is a metric? > > In Startree ThirdEye, a Metric refers to a column in your dataset or a statistic derived from the data that is tracked within ThirdEye. > In this context, a Metric is typically a specific measure or KPI (Key Performance Indicator) that is important for monitoring the health > and performance of your business. For example, a Metric could represent things like: > > 1. Business KPIs (e.g., revenue, transaction volume) > 2. App Behaviors (e.g. log-ins, errors) > 3. Cloud spend > 4. Networking traffic Select `SUM` as the aggregation function, and select daily granularity. For this alert and the options we set so far, ThirdEye will sum the views coming into the table every day. ![ThirdEye Alert Creation Wizard](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/free_tier_getting_started/alert_creation_wizard.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=e89659a7bae49bce2ef5b331e63362d2) ## Selecting your detection algorithm Next, we need to select the model that will fit the views data in order to detect anomalies. Luckily we have a Detection Algorithm Recommender that will search through and fit all the possible algorithms ThirdEye offers to find the best one. ![ThirdEye Detection Algorithm Selection](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/free_tier_getting_started/detection_algorithm_selection.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=ca7bce86d714da7f440fec6871f6d723) The alert recommender looks through the historical data of the alert to see which of our models best fits this pattern of data. When the dimension recommender is ready, choose the StarTree-ETS option 1 from the recommended configurations. You can see how the model predicts the pattern, and where it detects the anomalies where the red dots are. On the legend of the graph, select the ‘Upper and Lower bound’ legend item to add the ranges of the model shown below. ![ThirdEye Detection Algorithm Recommender](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/free_tier_getting_started/detection_algorithm_selection_recommended.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=a675137f44523c6592396588dbc3d92c) ## Configure Notifications When an anomaly shows up in your dataset, it’s important to be notified when it happens. Let’s set up a notification group – which we can use to configure ThirdEye to send notifications when anomalies happen. Click on the toggle button to configure notifications, and select ‘Create a new notification group for this alert’. Let’s set up email notifications for the anomalies that occur. Fill out the form with a subscription group name and your email address that you want alerts to go to. By default, only new anomalies will be notified, but since this is a sample dataset from 2020 we will enable ‘Notify Historical Anomalies’ to receive the notifications for this dataset. For now let’s leave the other fields at their default values. ![Recommended detection algorithm](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/free_tier_getting_started/notification_configuration.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=8c50bf35538aaf648be2d2cfbe20b34b) Feel free to add more emails or configure more alerts to the other supported platforms like PagerDuty, Slack or to your custom applications. ## Finish creating your alert Click ‘Create Alert’, and on the modal, name your alert and keep the default schedule settings. These schedule settings means that ThirdEye will run the detection on the previous day’s data at 5 AM every day to report the anomalies that appeared. Let’s leave the default as is. ![Confirmation modal for alert creation](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/free_tier_getting_started/alert_finish.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=1674f9b5359ae9861933a14e21e30629) ## Viewing your anomalies After creating your alert, you’re taken to the Alert page. The page will update automatically with the anomalies you saw on the Alert Creation screen. When the page updates, click on the latest anomaly. This is the red point lower and further right than the other. ![Viewing anomalies on the alert page](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/free_tier_getting_started/alert_page.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=90bc39b96a79342be95857d04003d470) You should be taken to a page like this:\\ ![Anomaly page](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/free_tier_getting_started/anomaly_page.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=ff9aa79349e634c7178d9d60b8206fb4) On this page, we can confirm whether we have a true anomaly or not. Ultimately, ThirdEye can only estimate what the monitored metric should be, so sometimes ThirdEye might miss true anomalies or detect false positives. Let’s investigate this anomaly to see whether it's a true anomaly or not. Select ‘Investigate Anomaly’ to see further and perform a root cause analysis (RCA). > [!NOTE] > > What is root cause analysis?\ > Root cause analysis in anomaly detection aims to identify the underlying factor(s) responsible for > unusual patterns or behavior in data. It goes beyond simply flagging anomalies and seeks to > pinpoint why those anomalies occurred, guiding future corrective actions or system improvements. ## Perform a root cause analysis After selecting to investigate the anomaly, you should be taken to an Investigation page to perform a RCA on the anomaly. You should see a page like the one below: ![RCA investigation page](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/free_tier_getting_started/investigation_page.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=8dfe1352e121779b7d199927862424bd) On this page we have two RCA visualizations, Top Contributors and Heatmap and Dimension Drills. Top Contributors show what other fields and combinations of those fields in your dataset had drastic changes. Heatmap and Dimension Drills will show the distribution of different fields and their values and how those changed. You may be wondering what these two visualizations are comparing to? That’s a great question. ThirdEye compares the data at the time of the anomaly to the same time one week prior. If your data doesn’t have a weekly pattern, you can change the time interval to check against in the dropdown at the top of the page. The dropdown is shown below, and you can choose to compare the data from one day, week, month, or year ago against the data at the time of the anomaly. ![RCA investigation page baseline picker](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/free_tier_getting_started/rca_baseline_picker.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=c24a1a2375852e23e22fec213f7ba66d) For now, let’s stick with comparing the data from a week ago, and let’s take a look at the two visualizations. ## RCA Visualizations ![RCA top contributors](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/free_tier_getting_started/rca_top_contributors.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=46fa8652c116d44c0e75cd6bf3df2a09) Looking at the Top Contributors, we can see that, compared to a week ago, there was a drop in views on the chrome browser, on version 0.3, and on the mobile (phone) version. There was also a smaller drop in views on the same browser and version, but on desktops. Could this version, browser type and devices be the root causes of the anomaly? Take a look at the Heatmap and Dimension Drills. This takes a look of the data from a different angle and shows you for each column of your data, how did the makeup of the different values in that column change? Blue indicates increases in field values, red indicates decreases in field values, darker colors indicate larger changes, and lighter colors indicate smaller changes. ![RCA heatmap](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/free_tier_getting_started/rca_heatmap.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=e6ddcb58f0b8a71bff61bfea1de6c8f5) Looks like there was a 320,000 decrease in chrome browser views and a 40% decrease in chrome browser views compared to other browser views. Seems like we’re getting a better understanding now of what was causing this anomaly. ## Saving the Investigation Next, after taking a closer look at the two RCA visualizations, let’s save our investigation. Optionally, you can add dimensions to your Investigation Preview at the bottom. The Investigation Preview at the bottom will update with graphs of the dimensions you choose. ![RCA saving the investigation](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/free_tier_getting_started/saving_investigation.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=e015aa750b8523adf641628ce768a36d) Click next, and advance to the Preview Investigation page. Here you can optionally define anomalous events. These are special days that lead to irregular behavior of your metrics. For an eCommerce company, this could be Black Friday, Christmas Day, etc. We are going to skip this feature of ThirdEye for now and keep going to the next page. You should now be on the Review Investigation Page. Here let’s name our investigation and add some notes in the comments about what we found from the RCA page. Fill out something like what is shown below and save your investigation! ![RCA reviewing the investigation](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/free_tier_getting_started/reviewing_investigation.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=6af8c8a43f6952b31905001775e23850) ## Confirming your anomaly After saving the investigation, confirm to ThirdEye that an anomaly has been found. ThirdEye takes the feedback for true positive anomalies and false positive anomalies and tunes the model accordingly to give you more accurate results. ![RCA confirm anomaly](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/free_tier_getting_started/confirming_anomaly.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=9474e7e37827d12ba45b4be57689c072) ## Conclusion And that’s it! That concludes your ThirdEye tutorial. You went through setting up an alert on a metric, setting up notifications, finding anomalies, and finally investigating an anomaly to find out what the potential business issue is! Next, try connecting some live data and see for yourself how ThirdEye can save your business time and money. # StarTree ThirdEye Getting Started Checklist Source: https://docs.startree.ai/thirdeye/getting-started/getting-started-checklist To get started with [StarTree ThirdEye](https://startree.ai/products/startree-thirdeye), do the following: 1. Review [prerequisites](#prerequisites) 2. [Make your data accessible by ThirdEye](#make-your-data-accessible-by-thirdeye) 3. [Add a notification system](#add-a-notification-system) 4. [Create alerts](#create-alerts) ## Prerequisites 1. Apache Pinot must be installed and running. To install Pinot, do one of the following: * Install the open source version of [Apache Pinot](https://pinot.apache.org/) on your own infrastructure (called BYOC for *bring your own cloud*) * Use [StarTree Cloud](https://www.startree.ai/), which includes Pinot hosted as software-as-a-service (SaaS) along with a few other perks. 2. StarTree ThirdEye must be installed and running. To install ThirdEye, do one of the following: * Install [ThirdEye](/thirdeye/getting-started/install-thirdeye). For more information, see [infrastructure requirements](/thirdeye/getting-started/infrastructure-requirements). * Use StarTree Cloud, which includes an available instance of ThirdEye, ready to connect to data you've uploaded into your StarTree Cloud instance of Pinot. 3. Upload your data into Apache Pinot > [!NOTE] > > Data to be used must have the right data architecture in order to be useful. See [ThirdEye data requirements](/thirdeye/getting-started/data-requirements) for more information. 4. Work through the [ThirdEye use case planning template](/thirdeye/getting-started/use-case-template) to ensure you understand what intend to accomplish in the following checklist. 5. Data readiness and validation checklist: * Timestamp needs to be in epoch millis for ThirdEye to work efficiently. If there is a non-epoch millis column then convert it to epoch millis using the “derived column” and apply time range indexes based on time granularity used in ThirdEye (Daily, hourly, 15 mins, 1 min). * Cannot have data gaps [Link to completeness delay](/thirdeye/alert-configuration#metadata), [definition](/thirdeye/alert-configuration#completenessdelay), [Link to external blog how to do DQ check using ThirdEye](https://medium.com/the-ab-tasty-tech-blog/data-quality-timeseries-anomaly-detection-at-scale-with-thirdeye-468f771154e6) * Based on Query patterns from ThirdEye to Pinot - apply those specific indexing. * Joins are not supported in ThirdEye. Make sure derived columns pre-ingestion are created for those to support TE use cases. * ThirdEye needs a denormalized schema for the following reasons: * TE can run any SQL query Pinot can support for just running alerts. We can do this with a custom template. [Link to ThirdEye templates](/thirdeye/templates/startree-absolute-rule). One can clone this to create a custom template and update the dag in it (query against Pinot). However, all the templates need to be cloned to add this custom query) * The complexity is more on: how the metric is then defined, how people can choose different detection algorithms on it, how it is exposed in the UI, how that is shared in RCA, notifications, etc., alerts might work, but the entire experience is not great. RCA may break. If RCA top contributors, heatmap needs to provide accurate insights at a quicker speed then transformed data will be ideal. * If there are too many dimensions in the table how to make RCA more performant? Ans: Number of dimensions in a schema - indexes on dimensions (too many dimensions will slow down RCA UI) - rcaExcludedDimensions [Link to the doc](/thirdeye/alert-configuration#rcaexcludeddimensions) ## Make your data accessible by ThirdEye To make your data accessible by ThirdEye, you must [connect ThirdEye to a datasource](/thirdeye/how-tos/database). If you're using StarTree Cloud, see an example of how to [load a sample dataset section](/thirdeye/getting-started/create-your-first-alert#load-a-sample-dataset). If you are using StarTree Cloud, when you open ThirdEye, ThirdEye automatically connects to Pinot and guides you through the connection process. ## Add a notification system Create one or more groups for alerts to be sent to, each with one or more routes for sending the notifications, such as email, Slack, or PagerDuty. These groups are called *Subscription Groups* in ThirdEye. Notifications are sent to subscription groups. The entire set of subscription groups and routes is referred to as a *notification system*. [Create your notification system](/thirdeye/how-tos/add_notification) first so that you have a place to send alert notifications when you create the alerts. ## Create alerts Read [Create your first alert](/thirdeye/getting-started/create-your-first-alert) for a basic example you can use to guide you through the process. The [Create alerts](/thirdeye/how-tos/alert/how-to-use-alert-creation-ui) section of the docs contains some practical recipes and patterns you can use. To learn more about how to configure alerts, see [Alert configuration and execution](/thirdeye/alert-configuration). # ThirdEye Infrastructure Requirements Source: https://docs.startree.ai/thirdeye/getting-started/infrastructure-requirements ThirdEye is available as a managed platform with StarTree Cloud.\ You can also install ThirdEye from a binary file, from sources or using Docker. See [Install ThirdEye](/thirdeye/getting-started/install-thirdeye) for installation procedures. ## ThirdEye components ThirdEye has five different types of pods, which are the components that make up a ThirdEye deployment. * **Coordinator:** This is the API Server. It also powers root cause analysis (RCA) pipelines and ad hoc charting. * **Scheduler:** Runs all the cron jobs and creates tasks * **GUI:** Serves the React frontend for ThirdEye * **Worker:** Executes the tasks * **MySQL:** Provides state/metadata persistence for ThirdEye and serves the task queue Consider the following details in your capacity planning: * Coordinator and Worker pods do most of the heavy lifting * ThirdEye uses MySQL for task processing as well as for processing and storing anomalies, which can require significant DB resources * Memory Consumption * The task that demands the most memory is the detection pipeline as it will fetch large chunks of data to process in-memory * The highest memory consumption will occur in cases where: * The granularity of alerts is low. For example, a process with a granularity of one minute will process significantly more data points than one with a granularity of one hour. * A coordinator previews an alert on a large timeframe * At the moment when an alert is created, because in the worker the alert is processed across all of the historical data. * CPU Requirements * ThirdEye benefits greatly from parallel processing across multiple processors * You can configure the number of threads and workers available * More CPU greatly helps in executing multiple concurrent tasks and responding to API calls * The ThirdEye backend can be run entirely in a single virtual machine. This means that the coordinator, scheduler, and worker can run in a single JVM process ## Example deployments Here we provide examples of what might be used for different deployment needs, such as production or development. ### Minimal deployment for development We run the backend services: Coordinator, Scheduler and Worker in 1 single JVM. | Node | Component | Replicas | Mem | CPU | EBS/SSD | Type | | - | - | - | - | - | - | - | | Single node | ThirdEye Server | 1 | 8G | 2 | 0G | m5.large | | | ThirdEye UI | 1 | 1G | 1 | 0G | | | | MySQL | 1 | 8G | 2 | 100G | m5.large | ### Production examples This section includes two examples: the standard setup and the enterprise setup. Start with the standard setup. #### Standard setup The standard production setup involves running everything in separate pods, while keeping the coordinator, scheduled, and UI in the same node. | Node | Component | # pods | Mem | CPU | EBS/SSD | AWS Type | GCP Type | | - | - | - | - | - | - | - | - | | Node-1 | ThirdEye Coordinator | 1 | 8G | 1 | 0G | 1.0 x m5.large | 1 x n2-standard-2 | | Node-1 | ThirdEye Scheduler | 1 | 4G | 1 | 0G | 0.5 x m5.large | 0.5 x n2-standard-2 | | Node-1 | ThirdEye UI | 1 | 0.5G | 0.5 | 0G | 0.1 x m5.large | 0.1 x n2-standard-2 | | Node-2 | ThirdEye Worker | 1 | 8G | 2 | 0G | 1.0 x m5.large | 1 x n2-standard-2 | | Node-2 | MySQL | 1 | 8G | 2 | 100G | 1.0 x m5.large | 1 x n2-standard-2 | #### Enterprise setup This is identical to the standard setup, except that it has more workers. Adding workers enables running more tasks in parallel. We may not need to scale up the coordinator since UI requests may remain limited, but in this example we show how you might choose to, if needed. | Node | Component | # pods | Mem | CPU | EBS/SSD | AWS Type | GCP Type | | - | - | - | - | - | - | - | - | | Node-1 | ThirdEye Coordinator | 1 | 16G | 2 | 0G | 1 x m5.xlarge | 1 x n2-standard-4 | | Node-2 | ThirdEye UI | 1 | 0.5G | 1 | 0G | 0.1 x m5.xlarge | 0.1 x n2-standard-4 | | Node-2 | ThirdEye Scheduler | 1 | 8G | 2 | 0G | 0.5 x m5.xlarge | 0.5 x n2-standard-4 | | Node-3,4,5 | ThirdEye Worker | 3 | 16G | 2 | 0G | 3 x m5.xlarge | 3 x n2-standard-4 | | Node-6 | MySQL | 1 | 16G | 2 | 100G | 1 x m5.xlarge | 1 x n2-standard-4 | ## External resources * [AWS Instance Types](https://aws.amazon.com/ec2/instance-types/m5/) * GCP * [GCP Instance Types](https://cloud.google.com/compute/docs/general-purpose-machines) * [E2 Machine](https://cloud.google.com/compute/docs/general-purpose-machines#e2_machine_types_table) vs [C3 Machine](https://cloud.google.com/compute/docs/general-purpose-machines#c3_machine_types) types. (Note: E2 Machines are less expensive.) # Install ThirdEye Source: https://docs.startree.ai/thirdeye/getting-started/install-thirdeye ThirdEye is available as a managed platform with StarTree Cloud. It can also be installed from a binary file, from sources or using Docker. Before you start, read [ThirdEye Infrastructure Requirements](/thirdeye/getting-started/infrastructure-requirements) for details about capacity planning, components, and deployment planning. **StarTree Cloud** StarTree Cloud provides managed hosting for ThirdEye on all major cloud platforms, including AWS and GCP. To start using ThirdEye in StarTree Cloud, [**sign up for StarTree Cloud**](https://startree.ai/demo). **Helm** To install ThirdEye on Kubernetes using Helm, complete the steps in this doc. ## Prerequisites * Access to StarTree ThirdEye artifacts * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) installed * `kubectl` is connected to the k8s cluster where you want to install ThirdEye * [Helm 3.x](https://helm.sh/docs/intro/install/) installed ### Access to StarTree ThirdEye artifacts Contact [StarTree](https://startree.ai/contact-us) to obtain credentials to create a k8s secret that lets helm access ThirdEye source files. Use these to set up access to the Artifactory repo. ```bash theme={null} # This assumes you are installing Thirdeye in the "thirdeye" namespace namespace="thirdeye" username="" STARTREE_REPO_URL="" ARTIFACTORY_API_KEY="" kubectl create secret docker-registry startree -n $namespace \ --docker-server=$STARTREE_REPO_URL \ --docker-username=$username \ --docker-password=$ARTIFACTORY_API_KEY ``` ## ThirdEye Installation To install ThirdEye, complete the follow steps to set up a Helm repo and run the Helm install. ### Set up Helm repo Edit the following script to include your credentials. ```powershell theme={null} STARTREE_HELM_REPO_URL="" # Add stable helm repo helm repo add helmstable https://charts.helm.sh/stable helm repo add startree $STARTREE_HELM_REPO_URL --username $username --password $ARTIFACTORY_API_KEY # Update your local repo helm repo update # fetch dependencies. example: mysql. See Chart.yaml helm dependency update ``` Verify the `startree-thirdeye` chart is accessible. ```shell theme={null} helm search repo startree-thirdeye ``` Expected Output: ``` NAME CHART VERSION APP VERSION DESCRIPTION startree/startree-thirdeye 2.4.0 0.5.0-SNAPSHOT One Stop Shop For Anomaly Detection. ``` ### Running Helm Install Edit the following script to include your credentials.\`\`\` ```shell theme={null} # setup StarTree Docker repo export DOCKER_CONTAINER_URL="" # Your ThirdEye domain name export THIRDEYE_UI_PUBLIC_URL="http://thirdeye.internal.company.com" # Artifact Versions HELM_VERSION="2.6.0" TE_VERSION="0.154.0" TE_UI_VERSION="2.41.4" helm install thirdeye startree/startree-thirdeye \ --version ${HELM_VERSION} \ -n $namespace \ --set image.repository="${DOCKER_CONTAINER_URL}/startree-thirdeye" \ --set ui.image.repository="${DOCKER_CONTAINER_URL}/startree-thirdeye-ui" \ --set ui.publicUrl="${THIRDEYE_UI_PUBLIC_URL}" \ --set image.tag=${TE_VERSION} \ --set ui.image.tag=${TE_UI_VERSION} ``` ## Upgrading ThirdEye ```shell theme={null} # For example, This upgrades ThirdEye to the desired development image. helm upgrade --install thirdeye -n "${namespace}" . \ --set image.tag=${TE_VERSION} \ --set ui.image.tag=${TE_UI_VERSION} ``` ## Uninstalling ThirdEye ```shell theme={null} helm uninstall thirdeye -n $namespace ``` ## Configurations To specify ThirdEye configurations, do one of the following: * Set parameters by adding `--set key=value[,key=value]` arguments to the `helm install` command. * Create a YAML file to specify parameters values, like this: Alternatively, you can create a YAML file that specifies the values for the parameters, then load it like this: ```bash theme={null} helm install thirdeye . -f values.yaml ``` ### Dynamic Secrets ThirdEye has plugin infrastructure which lets you to create your own plugins. To avoid creating `Secret` resources for the sensitive data for each new plugin, create dynamic secrets. For example, add the following information to a `values.yaml` file: ```yaml theme={null} secrets: smtpUsername: env: SMTP_USER value: tobefedexternally smtpPassword: env: SMTP_PASSWORD value: tobefedexternally holidayLoaderKey: encoded: true value: ``` Here are some important details: 1. A single `Secret` resource is created with data fields corresponding to each entry in `secrets`. 2. A secret data field is injected as an environment variable. In our example, we see the key `env`. In the server pods, if we pass `env`, `smtpUsername` is injected as an environment variable with the variable name `SMTP_USER` and value `tobefedexternally`, while `holidayLoaderKey` won't be injected as environment variable. 3. The values are processed as plain text by default and encoded internally unless you provide `encoded: true`, then the value isn't encoded because Helm assumes the value is already encoded. 4. We recommend passing values other than simple strings (like the JSON payload) as `base64` encoded values to avoid parsing issues. ### Holiday Events ThirdEye lets you display events from external Google calendars. To enable this feature, provide a **base64 encoded** JSON key. See [https://docs.simplecalendar.io/google-api-key/](https://docs.simplecalendar.io/google-api-key/) for more details. To install this feature: ```bash theme={null} helm install thirdeye . \ --set secrets.holidayLoaderKey.value="" \ --set secrets.holidayLoaderKey.encoded=true ``` ### Custom Calendar List The ThirdEye Helm chart has a default list of the following calendars: * en.australian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.austrian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.brazilian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.canadian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.china#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.christian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.danish#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.dutch#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.finnish#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.french#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.german#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.greek#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.hong\_kong#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.indian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.indonesian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.irish#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.islamic#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.italian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.japanese#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.jewish#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.malaysia#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.mexican#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.new\_zealand#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.norwegian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.philippines#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.polish#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.portuguese#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.russian#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.singapore#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.sa#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.south\_korea#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.spain#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.swedish#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.taiwan#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.uk#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.usa#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) * en.vietnamese#[holiday@group.v.calendar.google.com](mailto:holiday@group.v.calendar.google.com) But if you want to pass a selected list of calendars, do this: ```bash theme={null} helm install thirdeye . \ --set config.calendars="{en.australian#holiday@group.v.calendar.google.com,en.austrian#holiday@group.v.calendar.google.com}" ``` ### SSL/TLS Support To enable SSL/TLS on ThirdEye components, contact [StarTree](https://startree.ai/contact-us). ### Basic Authentication Support To configure basic authentication, add the following to your `server.yaml` file: ```yaml theme={null} auth: enabled: true basic: enabled: true users: - username: admin - password: admin ``` Details | Property | Description | | - | - | | `enabled` | Flag to enable/disable auth | | `basic.enabled` | Flag to enable/disable Basic authentication filter | | `basic.users[].username` | Username for authentication | | `basic.users[].password` | Password for authentication | ### Other Customizations | Property | Description | | - | - | | `image.repository` | Docker repository where ThirdEye server image is present | | `image.tag` | Docker image tag of ThirdEye server image | | `ui.image.repository` | Docker repository where ThirdEye UI image is present | | `ui.image.tag` | Docker image tag of ThirdEye UI image | | `ui.port` | UI service port | | `ui.publicUrl` | Url on which ThirdEye UI is exposed publicly. All the notifications will use this url to share the anomaly page link | | `scheduler.enabled` | Flag to run a separate scheduler. If not enabled then coordinator itself will take care of scheduling tasks | | `worker.enabled` | Flag to run a separate worker. If not enabled then coordinator itself will take care of running tasks | | `worker.replicas` | Number of worker pods required | | `worker.randomWorkerIdEnabled` | Flag to enable assigning random worker ids to worker pods. Must be set true for multiple workers. | | `prometheus.enabled` | Flag to expose prometheus metrics and adding annotations for prometheus to scrape the metrics | | `mysql.enabled` | Flag to disable MySQL deployment if using external instance | | `mysql.url` | Database URL if using external instance | | `mysql.port` | Database port if using external instance | | `mysql.mysqlUser` | Database username | | `mysql.mysqlPassword` | Database password | | `mysql.persistence.size` | Size of persistent volume created for database storage | | `config.jdbcParameters` | Config to pass additional parameters to the jdbc connection string | | `[coordinator/worker/scheduler].strategy` | Specifies the strategy used to replace old Pods by new ones. | | `tls.[coordinator/worker/scheduler/ui].secretName` | When provided it will override the default secret names referred for tls keys | **From Source** * See [Build](https://github.com/startreedata/thirdeye#build) procedure on Github **Docker** The Docker image is not publicly available yet, but you can build it manually. * See [Build](https://github.com/startreedata/thirdeye#build) on Github * See [Docker](https://github.com/startreedata/thirdeye#docker) To be the first to know when public docker images are available, [join the StarTree Community slack](https://inviter.co/startree-community). **From Binary** Binaries are not publicly available yet, but you can build from the sources manually. * See [Build](https://github.com/startreedata/thirdeye#build) on Github * See [Running from distribution](https://github.com/startreedata/thirdeye#running-thirdeye-from-distribution) To be the first to know when public binaries are available, [join the StarTree Community slack](https://inviter.co/startree-community). ### Recommended ThirdEye Version | Build | Edition | Version | | - | - | - | | helm chart | Community | 2.6.0 | | thirdeye | Community | 1.199.0 | | thirdeye-ui | Community | 2.41.4 | | startree-thirdeye | Enterprise | 0.154.0 | # StarTree ThirdEye Use Case Planning Template Source: https://docs.startree.ai/thirdeye/getting-started/use-case-template This page exists to help you think through the details of your use case for anomaly detection with ThirdEye as part of the [ThirdEye getting started checklist](/thirdeye/getting-started/getting-started-checklist) prerequisites. This template asks you to collect your information in one place to make getting started with ThirdEye easier and more efficient. The page is split into the following sections: 1. [Problems](#problems) 2. [Data flow](#data-flow) 3. [Schema](#schema) 4. [Use case examples](#use-case-examples) Note that when you are directed to gather information, this only includes information pertinent to the data you will use for anomaly detection with ThirdEye. ## Problems Think through the following questions: * What key performance indicators (KPIs) will you be monitoring? * Who in the company cares about eack KPI? (For example: The CEO, end users, CIO, CTO, others?) * Which team/vertical is responsible for these metrics, such as sales, marketing, and so on? * How do you detect issues today? * What is your current gap when trying to detect issues? What problem are you trying to solve? * What do you do today when you detect an issue? Who is informed? What is their plan of action? This is typically where you want to write up your use case, so that you are working toward specific goals. See the included [Use case examples](#use-case-examples) for ideas. ## Data flow Where does your data come from? Where does it go? How is it processed? At what points are metrics being gathered and what are those metrics? What you especially want to know are all of your inputs and outputs as data flows through your system. You can draw a diagram, write it out in a list, or use whatever method is most useful and clear to you. As you do this, you may discover monitoring gaps and new opportunities for data gathering that can lead to system improvement. ## Schema Gather the following information about your data: * Schema * Dimensions * Metrics * Data source(s) * Key performance indicators (KPIs) and how you build and monitor them or intend to do so ## Use case examples Here are some samples of use case information to collect to help you make your thoughts and plans a bit more concrete as you get going. The first includes vital information while the second is more detailed and will make both the process smoother and your end result more useful. ### Use case 1: High Priority—Intake volume by input channel (*must have*) Make the following decisions and record your answers: * What is the business use-case and impact if a specific problem is not detected in a timely manner? (For example, a drop-off in monthly sales orders could indicate an issue with a recent pricing change.) * What to monitor (metrics and dimensions - average/sum/% changes etc) * How frequently to monitor? * What is the monitoring granularity needed? For example, daily, weekly, monthly? Remember to configure the timestamp in Pinot accordingly, if needed. * What is the pattern for training data? The seasonality (weekly/daily/monthly)? The data retention period in Pinot must be configured based on this, for example, if your intent is to compare daily trends against the previous seven calendar days. * Are there any filters to be applied to handle missing/incomplete data, for example, to look only at weekday patterns, impact thresholds, and so on. * Who is to be notified if this issue is found, and by what method, such as an API call, email, Slack, or SMS? * What data updates (upserts/updates) you are planning to do? ### Use case 2: High Priority—Intake volume by input channel (*nice to have*) Collect everything in Use case 1 and add: * Alert-related details * Alert granularity * Alert frequency * Data source * Data Volume to be processed * Collect a sample record or two (a screenshot will do) # Subscription Groups and Notification Systems Source: https://docs.startree.ai/thirdeye/how-tos/add_notification The following routes for notifications are currently supported: * Email * Slack * PagerDuty * Webhooks To create alert notifications, you first create a subscription group. Alerts are assigned to subscription groups, which can be reused across multiple alerts. The subscription group is assigned one or more notification routes, such as email or Slack. Subscription groups can be created from the [ThirdEye UI](#create-subscription-groups-using-the-thirdeye-ui) and from the [ThirdEye API](#create-subscription-groups-using-the-thirdeye-api). ## Create subscription groups using the ThirdEye UI Configure a subscription group to receive email notifications as follows: 1. Click **Create**. 2. Create **Subscription Group**. 3. Type in the **Name of the Subscription Group**. It's a good idea to name it something memorable and appropriately descriptive. 4. Enter schedule in cron job style (`cron -n|-p|-s|-m` mailcommand). The cron schedule defines when and at what frequency the subscription notifications will be sent. The format used is the 6-7 fields Quartz cron format. See the [Quartz documentation](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html#introduction) for details. A cron expression builder [is available here](https://www.freeformatter.com/cron-expression-generator-quartz.html). 5. Enter Email ID in **Subscribe Emails** section then click **Add**. 6. Select and add alerts (one by one) to be subscribed by clicking **->** next to the alert name in the **All alerts** section. (This is an optional step, but unless you perform it, the group won't be subscribed to receive any alert notifications. However, you can create subscription groups and then assign alerts later.) 7. Click **Next** and **Finish**. ![Select Notification Type](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/thirdeye-notification-subscription.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=83bf83eb5ed62dc09cccc41c2bc2b002) ![Email](https://user-images.githubusercontent.com/98617940/183694715-3d4ea15e-9071-4b9b-9601-c82d9d8c0ece.png) ![Slack](https://user-images.githubusercontent.com/98617940/183694866-b669651c-1663-4d52-8a25-b6a8d6b98e3e.png) ![Webhook](https://user-images.githubusercontent.com/98617940/183694986-baa73d77-a552-4ec6-9e3d-aad59be68b55.png) ![PagerDuty](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/PagerDutyConfiguration.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=77d4a4c8f85ed8abde015f989ed09791) Configure Subscription Groups to subscribe to alerts and receive notifications ## Create subscription groups using the ThirdEye API Use `/api/subscription-groups` to perform CRUD operations on subscription groups. ### Create a subscription group Use `POST /api/subscription-groups` to create a subscription group. The payload is a list of subscription groups. Here is a sample creating a subscription group with a single email notification channel. It is associated with a single alert. The alert is identified by its ID. ```json theme={null} [ { "name": "sg-with-alert", "alerts": [], "alertAssociations": [ { "alert": { "id": 247243 } } ], "cron": "0 */5 * * * ?", "specs": [ { "type": "email-smtp", "params": { "from": "no-reply@startree.ai", "smtp": { "host": "${SMTP_HOST}", "port": "${SMTP_PORT}", "user": "${SMTP_USER}", "password": "${SMTP_PASSWORD}" }, "to": [ "your.email@gmail.com" ] } } ] } ] ``` Here is the same example with a subscription group that is associated with an alert and an enumeration item. Here, in addition to the alert ID we also specify the enumeration item ID. ```json theme={null} { "id": 250958, "name": "sg-alert-and-enumeration-item", "alertAssociations": [ { "alert": { "id": 247243 }, "enumerationItem": { "id": 247397 } } ], "cron": "0 */5 * * * ?", "specs": [ { "type": "email-smtp", "params": { "emailRecipients": { "from": "no-reply@startree.ai", "to": [ "your.email@gmail.com" ] }, "smtp": { "host": "${SMTP_HOST}", "port": "${SMTP_PORT}", "user": "${SMTP_USER}", "password": "${SMTP_PASSWORD}" } } } ] } ``` ## Slack \[API Support] All notification channels are hosted within subscription groups. CRUD operations can be performed as usual on subscription group entities using the `/api/subscription-groups` endpoint. ### Step 1. Create a Slack app Create a Slack app for your workspace and add an incoming webhook. The webhook is tied to a specific channel and would look something like this. ```http theme={null} https://hooks.slack.com/services/T1234567/AAAAAAAA/ZZZZZZ ``` Note that one app can support multiple incoming webhooks. ### Optional: Subscription groups values can be populated via environment variables To prevent the incoming webhook from being visible in plain text to any user, you can create an environment variable (in Kubernetes, you can also leverage secrets to do this). ```shell theme={null} export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T1234567/AAAAAAAA/ZZZZZZ" ``` ### Step 2. Create a subscription group Here is a sample payload for creating the subscription group. You can submit this to `POST /api/subscription-groups`. ```json theme={null} [ { "name": "slack-webhook", "cron": "0 */5 * * * ?", "specs": [ { "type": "slack", "params": { "webhookUrl": "${SLACK_WEBHOOK_URL}", "sendOneMessagePerAnomaly": false, "oneMessagePerAnomalyBurstLimit": 20, "textConfiguration": { "owner": "Acccounting team", "mentionMemberIds": ["U03LORTRY9H"] } } } ] } ] ``` * `webhookUrl`: the slack webhook url * `sendOneMessagePerAnomaly`: if false: anomalies notified at the same time are sent in a single message. If true, one message is sent for each anomaly. * `oneMessagePerAnomalyBurstLimit`: if `sendOneMessagePerAnomaly` is true and there are a lot of anomalies at the same time, sending many messages could be blocked by Slack API quotas. Limit number of messages to send by a single notification task. * `textConfiguration`: * `owner`: free text to add an owner to the alert * `mentionMemberIds`: A list of Slack member ids to mention (tag) in the message. ### Step 3: Use your subscription group with any Alert The subscription group can be tied to any alert as usual. Also note that one subscription group supports multiple specs. Therefore, it can route to the same or different notification channels in the same subscription group. ## Set Up PagerDuty integration To integrate PagerDuty notifications for ThirdEye alert notifications, do this: 1. **Create a PagerDuty Service**: If you don't already have one, create a new service in your PagerDuty account. This service will be the destination for the alerts from your system. 2. **Obtain an Events Integration Key**: [See the PagerDuty documentation for details](https://support.pagerduty.com/docs/services-and-integrations#create-a-generic-events-api-integration). * In your PagerDuty service, add an integration. [See the PagerDuty documentation for details](https://support.pagerduty.com/docs/service-directory). * Choose the **Events API v2** integration type. * Once the integration is added, you will receive an **Integration Key**. This key is unique to the service and is used to authenticate events sent from your system to PagerDuty. 3. **Secure the Integration Key**: * To ensure security, it's advisable not to expose the Integration Key directly in your application code. * In a Kubernetes environment, consider using Kubernetes secrets to store this key. * For other environments, use appropriate methods to secure the key, like environment variables. * Example of setting an environment variable: ```shell theme={null} export PAGERDUTY_INTEGRATION_KEY="your-integration-key-here" ``` ### Create a subscription group with PagerDuty specs 1. **Prepare the payload**: Construct a JSON payload to create a new subscription group that will use the PagerDuty integration. Here is a sample payload: ```json theme={null} [ { "name": "pagerduty-servicename", "cron": "0 */5 * * * ?", "specs": [ { "type": "pagerduty", "params": { "eventsIntegrationKey": "${PAGERDUTY_INTEGRATION_KEY}" } } ] } ] ``` * This cron job example runs every 5 minutes. Adjust as needed. * The `"eventsIntegrationKey"` parameter is where you use the environment variable you created to store your PagerDuty Integration Key. It is strongly discouraged to directly use the Integration Key in the payload, for security reasons. 2. **Send the request**: Use the above payload to make a `POST` request to the `/api/subscription-groups` endpoint of your system. This request will create a new subscription group configured to send alerts to PagerDuty. ### Connect the subscription group to alerts * Once the subscription group is created, you can associate it with any alert in your system. * The alerts tied to this group will trigger notifications to the configured PagerDuty service based on the rules and schedules defined in the subscription group. ## Webhook \[API Support] The webhook notification allows you to input a URL which ThirdEye can use to POST a payload. Here is a sample JSON payload to build a webhook notification. ```json theme={null} [ { "name": "webhook", "cron": "0 */5 * * * ?", "specs": [ { "type": "webhook", "params": { "url": "http://localhost:8080/internal/webhook" } } ] } ] ``` The POSTed payload contains all the anomaly reports relevant for the notification service. Here's an example payload. It contains: * The subscription group itself * Array of anomaly reports * A report object around all the anomalies ```json theme={null} { "subscriptionGroup": { "id": 188059, "name": "webhook", "alertAssociations": [ { "alert": { "id": 136062 } } ], "cron": "0 */5 * * * ?", "specs": [ { "type": "webhook", "params": { "url": "http://localhost:8080/internal/webhook" } } ] }, "anomalyReports": [ { "anomaly": { "id": 188180, "startTime": 1583020800000, "endTime": 1588291200000, "avgCurrentVal": 2.3475211E7, "avgBaselineVal": 6.1570800760419734E7, "score": 0.0, "weight": 0.0, "impactToGlobal": 0.0, "sourceType": "ANOMALY_REPLAY", "created": 1653463092111, "notified": true, "alert": { "id": 188172 }, "metric": { "name": "passenger_count" }, "metadata": { "dataset": { "name": "us_monthly_air_passengers_simplified" }, "metric": { "name": "passenger_count" } } }, "url": "http://localhost:7004/anomalies/188180", "data": { "metric": "passenger_count", "startDateTime": "Mar 01, 2020 00:00", "lift": "-61.87 %", "feedback": "Not Resolved", "anomalyId": "188180", "anomalyURL": "http://localhost:7004/anomalies/", "currentVal": "23,475,211", "baselineVal": "61,570,800.76", "dimensions": [], "swi": "0.00 %", "function": "us_monthly_passengers_simplified-holtwinters", "funcDescription": "Sample description payload for testing", "duration": "1,464 hours", "endTime": "May 01, 2020 00:00", "timezone": "UTC", "anomalyType": "Deviation", "properties": "" } } ], "report": { "startTime": "Sep 01, 2001 00:00", "endTime": "May 01, 2020 00:00", "reportGenerationTimeMillis": 1653464950589, "dashboardHost": "http://localhost:7004", "relatedEvents": [ { "id": 3, "name": "Related Holiday", "type": "HOLIDAY", "startTime": 1546310140005, "endTime": 1546396540005, "targetDimensionMap": {} } ] } } ``` ### Integration with notification endpoint to any other custom service using a webhook Refer to this [github link](https://github.com/rahulbsw/webhook-proxy) to get access to the code. Here is the code snippet: ``` def arrow(cur,base) if ($base < $cur) "⇑" else if ($base > $cur) "⇓" else "⇝" [ for (.anomalyReports) { "markdown" : "---"+"\n "+ "### **"+ .data.function +"** \n "+ "### Metric : **"+ .data.metric+"** \n "+ "### Change : " + .data.lift + " " + arrow(.data.currentVal,.data.baselineVal) +"\n "+ "#### Start Time : " + fallback(.data.startTime,.data.startDateTime) + " " + .data.timezone + " Duration " + .data.duration +"\n "+ "#### Current Value : " + .data.currentVal +"\n "+ "#### Baseline Value : " + .data.baselineVal +"\n "+ "#### dimensions : " + string(.data.dimensions) +"\n "+ "**"+ .data.funcDescription +"** \n "+ "**[Anomaly URL](http:///anomalies/" + .data.anomalyId +")** \n "+ "---"+"\n " } ] ``` ## Alert and notify on anomaly resolution Alert and notify on anomaly resolution aims to create a more complete monitoring experience by informing users not only about problems but also their resolution, leading to better awareness, improved efficiency, and a more proactive approach to metrics health. Currently this feature is supported for Slack and Webhook notifications. ### How to configure alert and notify on anomaly resolution for Slack notifcations? ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/NotificationPeriodEnds.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=930539efb324ad6041f9a2b89ba15086) ### How to configure alert and notify on anomaly resolution for webhook notifcations? Example of a payload ```json theme={null} [{ "name": "webhook test 3", "alerts": [], "alertAssociations": [ { "alert": { "id": 8267190 }, "created": 1712588538303 } ], "cron": "0 */5 * * * ?", "specs": [ { "type": "slack", "params": { "webhookUrl": "https://hooks.slack.com/services/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", "notifyResolvedAnomalies": true } }, { "type": "webhook", "params": { "url": "https://feline-open-robin.ngrok-free.app/thirdeye/srm" } } ], "notifyHistoricalAnomalies": true }] ``` To edit an entity, get the full entity with GET: `GET /api/subscription-groups/{id}/`. To push the update with the PUT method: `PUT /api/subscription-groups`. payload: ```json theme={null} [ { } ] ``` ## Slack notifications with tags and owners Organize slack notifications with tags and quickly see who's responsible for each anomaly with memberid. ### How to configure Slack notifcations for tags and owners? ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/SlackAlertOwner.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=ef34a0fa6ed9e578a114424746bf5764) [Read slack docs for more details](https://api.slack.com/methods/users.identity) # Use Anomaly Filters to Tune Alerts Source: https://docs.startree.ai/thirdeye/how-tos/alert-tuning Anomaly filters enable you to tune your alerts for improved accuracy. Read [concepts - using Anomaly filters as part of post-processor](/thirdeye/operators/post-processor) for background information. These filters can be defined during the alert creation. See [how to create alerts](/thirdeye/getting-started/create-your-first-alert) to learn more about alert creation. The goal of anomaly filters is to reduce false alerts. ## Day of the week filter This filter enables you to define the day of the week that will be used to filter anomalies to reduce false alarms. These days are usually known to be high volume or low volume. ```json theme={null} DaysOfWeek: ["MONDAY", "SATURDAY"], HoursOfDay: [0,1,2,23], DayHoursOfWeek: {"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]} ``` ## Offsets filter This filter enables you to define an offset to ignore anomalies during alert creation/configuration. ```json theme={null} Metric Name: "Business KPI", # customize label Offsets: ["P7D", "P14D"], # parameter of the detector Aggregation: "MEDIAN" # parameter of the detector ``` ## Cold Start filter Usually model predictions are impacted by low data volume in the beginning. This filter enables you to define from which day/time the model should start learning to predict anomalies accurately. ```json theme={null} coldStartPeriod": "P14D", # filter anomalies in the first 14 days of the dataset tableName: "my_dataset" ``` ## Events filter This filter enables you to define specific holiday periods. ```json theme={null} BeforeHolidayMargin: "PT4H", AfterHolidayMargin: "PT4H" ``` ## Threshold filter This filter enables you to apply a threshold on the record volume. ```json theme={null} Min limit: "100", Max limit": "10000000", Metric Name: "controlMetric", # customize label Timestamp: "ts", # time column of the side input Threshold metric: "sideMetric" # column to threshold on in the side input ``` ## Time of the week filter This filter enables you to specify day of the week and the time window during which the anomalies will be ignored. ```json theme={null} Days Of Week": ["MONDAY", "SATURDAY"], Hours Of Day": [0,1,2,23], Day Hours Of Week": {"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]} ``` # Derived Metric Alerts Source: https://docs.startree.ai/thirdeye/how-tos/alert/derived-metric-alert ThirdEye supports monitoring derived metrics. [StarTree templates](/thirdeye/how-tos/alert/use-templates#install-startree-templates) are directly compatible with derived metrics, but a piece of configuration is required for root-cause analysis (RCA) to work. Here is an example of monitoring the 95th percentile of the difference of 2 metrics `met1` and `met2`, with the `startree-threshold-percentile` template: > [!NOTE] > > You must have StarTree templates [installed and up to date](/thirdeye/how-tos/alert/use-templates#install-startree-templates) to use this template. ## Configuration: ```json theme={null} { "name": "derived-metric-95-percentile-threshold", "description": "Monitor 95th percentile of met1-met2 with threshold alert.", "template": { "name": "startree-threshold-percentile" }, "templateProperties": { "dataSource": "yourDatasource", "dataset": "yourDataset", "monitoringGranularity": "P1D", "min": "1000", "max": "2500", "aggregationFunction": "PERCENTILETDIGEST", "aggregationParameter": "95", "aggregationColumn": "met2-met1", "rcaAggregationFunction": "PCT95" }, "cron": "0 0 0 1/1 * ? *" } ``` *derived\_metric\_percentile\_threshold\_monitoring.json* Notes: * `aggregationFunction` calls the Pinot [PERCENTILETDIGEST](https://docs.pinot.apache.org/configuration-reference/functions/percentiletdigest) function * `aggregationParameter` is the second parameter of the function. It sets the percentile to 95. * `aggregationColumn` contains the derived metric. * `rcaAggregationFunction` sets the RCA aggregation function. It binds to `aggregationFunction` in the alert configuration. `PCT95` corresponds to 95th percentile. The set of aggregation functions is limited. For more details, see [Aggregation functions](/thirdeye/rca-aggregation-functions). **RCA will not work without this configuration**. It is necessary because the derived metric is not registered in ThirdEye, so no default aggregation function is available. # How to Use the Alert Creation UI Source: https://docs.startree.ai/thirdeye/how-tos/alert/how-to-use-alert-creation-ui The alert creation UI is where you will create new alerts and there are 2 modes: `Simple` and `Advanced`. * The `Simple` mode will prompt you for the different parts of the alert configuration and build the [configuration](/thirdeye/alert-configuration) JSON. * You can see the current status of the JSON in `Advanced` mode or switch to that mode to paste your own configuration. ## How to access the alert creation UI ![Add Datasource](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddAlert.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=16e1c4a96253832f0c3f91f7d1c80466) 1. Browse to the **Configuration** page. 2. Click **Create**. 3. Click the **Create Alert** menu item. This will take you to the alert creation UI, defaulting to `Advanced` mode. ## Simple Mode You can switch to simple mode in the top right section of the page where there is a switch for the 2 modes. ### The Sections ![Add Datasource](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddAlertSimpleMode.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=19d906f377267608af0c72085ee4153c) #### 1 - Alert Details In this section, you will be prompted to input the `name` of the alert. Ensure this is unique and no other existing alert has the same name. The `description` is optional. The `cron` section is where you can schedule where you can configure how often the pipelines checks for anomalies. The cron is expected to be in [quartz](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) format. #### 2 - Template Property Input This section allows you to select an [Alert Template](/thirdeye/how-tos/alert/use-templates) to use for your configuration. Alert templates are entities in ThirdEye that help make creating new alerts with less copying and pasting. Several come out of the box with a new installation of ThirdEye. Alert templates require you to enter properties specific to your environment which you will be prompted for in the section below `Template type`. #### 3 - Preview Configuration This section allows you to preview the alert configuration by seeing the metric trend and the anomalies that may be detected. You will see a button to preview once all the template properties are input. The preview button looks like the one below: ![Add Datasource](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddAlertChartPreviewButton.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=9de0b2fb1403eb7d7684f4e334ca0811) When data is loaded, it will look like the screenshot below ![Add Datasource](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddAlertChartPreview.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=864470a8ace95267cae0855ca39f513d) #### 4 - Notification Configuration This section allows you to configure which groups will be alerted when there are new anomalies. ## Advanced Mode ### The Sections ![Add Datasource](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddAlertAdvancedMode.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=e0040678014c84841dd4cd3b81231f3b) #### 1 - JSON Editor You can copy and paste JSON into here. #### 2 - Preview Configuration This section allows you to preview the alert configuration by seeing the metric trend and the anomalies that may be detected. You will see a button to preview once all the template properties are input. The preview button looks like the one below: ![Add Datasource](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddAlertChartPreviewButton.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=9de0b2fb1403eb7d7684f4e334ca0811) When data is loaded, it will look like the screenshot below ![Add Datasource](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddAlertChartPreview.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=864470a8ace95267cae0855ca39f513d) #### 3 - Notification Configuration This section allows you to configure which groups will be alerted when there are new anomalies. # Manage Missing Data Source: https://docs.startree.ai/thirdeye/how-tos/alert/manage-missing-data Most of the time(^1), to create buckets of a given granularity, the data fetching step will query with `GROUP BY timeBucket`. Let's consider a granularity of 1 day. What happens if there is no data on a Sunday? There will be no line for this Sunday. There is a great chance this will make things break later. After all, you expect your detection pipeline to know when there is no data. This case can seem non-likely to happen, but as soon as you filter on specific dimensions, the probability of this happening skyrockets. To manage missing data, use a `TimeIndexFiller` node. A usual pipeline will look like this: ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_missing_data_index_filler.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=f1f41c2357cdc3b71426b6bde89bd6f0) Here is the corresponding configuration: ```json theme={null} { "name": "simple-threshold-template", "description": "Sample threshold alert. Runs every hour. Change the template properties to run on your data", "cron": "0 * * * * ? *", "template": { "nodes": [ { "name": "root", "type": "AnomalyDetector", "params": { "type": "THRESHOLD", "component.monitoringGranularity": "${monitoringGranularity}", "component.timestamp": "ts", "component.metric": "met", "component.max": "${max}", "component.min": "${min}", "anomaly.metric": "${aggregateFunction}(${metric})" }, "inputs": [ { "targetProperty": "current", "sourcePlanNode": "missingDataManager", "sourceProperty": "currentData" } ] }, { "name": "missingDataManager", "type": "TimeIndexFiller", "params": { "component.timestamp": "ts" }, "inputs": [ { "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentData" } ], "outputs": [ { "outputName": "currentData" } ] }, { "name": "currentDataFetcher", "type": "DataFetcher", "params": { "component.dataSource": "${dataSource}", "component.query": "SELECT __timeGroup(\"${timeColumn}\", '${timeColumnFormat}', '${monitoringGranularity}') as ts, ${aggregateFunction}(${metric}) as met FROM ${dataset} WHERE __timeFilter(\"${timeColumn}\", '${timeColumnFormat}') GROUP BY ts ORDER BY ts LIMIT 1000" }, "outputs": [ { "outputKey": "pinot", "outputName": "currentData" } ] } ] }, "templateProperties": { "dataSource": "pinotQuickStartAzure", "dataset": "pageviews", "aggregateFunction": "sum", "metric": "views", "monitoringGranularity": "P1D", "timeColumn": "date", "timeColumnFormat": "yyyyMMdd", "max": "850000", "min": "250000" } } ``` *threshold\_with\_missing\_data\_management.json* By default, missing buckets are created with a value of 0. ## Configuration and behavior The `TimeIndexFiller` takes an input and returns it with the time index filled. ```json theme={null} { "name": "missingDataManager", "type": "TimeIndexFiller", "params": { "component.timestamp": "ts" }, "inputs": [ { "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentData" } ], "outputs": [ { "outputName": "currentData" } ] } ``` **Notice there is only one parameter.** Because the `DataFetcher` uses the ThirdEye macros `__timeGroup(...)` and `__timeFilter(...)`, metadata about the granularity and the time predicate is directly given to the `TimeIndexFiller`. ## Manual configuration If the input does not use macros, the `TimeIndexFiller` requires the following parameters: * `component.monitoringGranularity`: the granularity in ISO 8601 format. Eg: `P1D`. * `component.metric`: the name of the metric column. * `component.timestamp`: the name of the time column. * `component.minTimeInference`: the strategy to infer the minimum time. * `component.maxTimeInference`: the strategy to infer the maximum time. * `component.lookback`: Used when time inference uses a lookback time. In ISO 8601 format. The possible strategies are: * `FROM_DATA`: the minimum (resp maximum) time is the minimum (resp maximum) time observed in the input. Does not work well if data is missing at the beginning or at the end. * `FROM_DETECTION_TIME`: the minimum (resp maximum) time is the the minimum (resp maximum) of the analysis timeframe * `FROM_DETECTION_TIME_WITH_LOOKBACK`: same as the previous one, with an offset applied of value `component.lookback`. *** (^1) For classic SQL databases. Timeseries-focused database often introduce group-by-time-bucket capabilities, with empty buckets materialized. Still these bucketings require a range, or use the first and last value as range, which is not correct in our case. # Multiple Inputs Alerts Source: https://docs.startree.ai/thirdeye/how-tos/alert/multiple-inputs-alerts As explained in the [alert configuration documentation](/thirdeye/alert-configuration), you can create detection pipelines that form DAGs with execution dependencies. In this example, we use the [percentage rule detector](/thirdeye/concepts-data-alerts-notifications/concepts-anomaly-detection-algorithms#percentage-change), which requires 2 inputs: `baseline` and `current`. The pipeline looks like this: ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_multi_inputs_percentage_rule.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=6ce6bcd210ce7cfc3ad445f0edd8ba4d) Here is the corresponding configuration: ```json theme={null} { "name": "pageviews", "description": "This is a sample detection", "cron": "0 */1 * * * ?", "template": { "nodes": [ { "name": "root", "type": "AnomalyDetector", "params": { "type": "PERCENTAGE_CHANGE", "component.monitoringGranularity": "${monitoringGranularity}", "component.timestamp": "ts", "component.metric": "met", "component.percentageChange": "${percentageChange}", "component.pattern": "${pattern}", "anomaly.metric": "${metric}" }, "inputs": [ { "targetProperty": "baseline", "sourcePlanNode": "baselineDataFetcher", "sourceProperty": "baselineOutput" }, { "targetProperty": "current", "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentOutput" } ], "outputs": [] }, { "name": "baselineDataFetcher", "type": "DataFetcher", "params": { "component.dataSource": "${dataSource}", "component.query": "SELECT __timeGroup(\"${timeColumn}\", '${timeColumnFormat}', '${monitoringGranularity}') as ts, ${metric} as met FROM ${dataset} WHERE __timeFilter(\"${timeColumn}\", '${timeColumnFormat}', ${lookback}, ${lookback}) GROUP BY ts ORDER BY ts LIMIT 1000" }, "inputs": [], "outputs": [ { "outputKey": "pinot", "outputName": "baselineOutput" } ] }, { "name": "currentDataFetcher", "type": "DataFetcher", "params": { "component.dataSource": "${dataSource}", "component.query": "SELECT __timeGroup(\"${timeColumn}\", '${timeColumnFormat}', '${monitoringGranularity}') as ts, ${metric} as met FROM ${dataset} WHERE __timeFilter(\"${timeColumn}\", '${timeColumnFormat}') GROUP BY ts ORDER BY ts LIMIT 1000" }, "inputs": [], "outputs": [ { "outputKey": "pinot", "outputName": "currentOutput" } ] } ] }, "templateProperties": { "dataSource": "pinotQuickStartLocal", "dataset": "pageviews", "metric": "sum(views)", "monitoringGranularity": "P1D", "timeColumn": "date", "timeColumnFormat": "yyyyMMdd", "percentageChange": "0.2", "offset": "P1W", "pattern": "up_or_down" } } ``` *percentage\_rule\_alert.json* Notice the pipeline has two `DataFetcher` nodes: * `baselineDataFetcher`, which has an output called `baselineOutput` * `currentDataFetcher`, which has an output called `currentOutput` These outputs are then fed to the anomaly detector: * `baselineOutput` goes into the `baseline` property of the detector * `currentOutput` goes into the `current` property of the detector ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "PERCENTAGE_CHANGE", ...}, "inputs": [ { "targetProperty": "baseline", "sourcePlanNode": "baselineDataFetcher", "sourceProperty": "baselineOutput" }, { "targetProperty": "current", "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentOutput" } ] } ``` If `baseline` and `current` properties were not given, the detector would fail. ### About the baseline To ensure the baseline timeframe is correctly computed based on the analysis timeframe,we'll look at the SQL queries: * `currentDataFetcher` uses the macro `where __timeFilter(\"${timeColumn}\", '${timeColumnFormat}')`: the macro automatically filters on the analysis timeframe. * `baselineDataFetcher` uses the same macro, with an offset: `where __timeFilter(\"${timeColumn}\", '${timeColumnFormat}', ${lookback}, ${lookback})`, with `offset=P1W`. * `P1W` means one week. This means the baseline is one week before the analysis timeframe. # Alert Templates Source: https://docs.startree.ai/thirdeye/how-tos/alert/use-templates Creating alerts can take time. To help reduce the time needed, you can make an alert detection logic configurable, and reuse it in multiple alerts with alert templates. ## Install StarTree templates ThirdEye comes packaged with ready-to-use templates. To reinstall these templates, make the following call to the ThirdEye API: ```js theme={null} curl -X POST "/api/alert-templates/load-defaults" -H "Content-Type: application/x-www-form-urlencoded" -d "updateExisting=true" ``` Or use the [Swagger UI](/thirdeye/how-tos/use-the-api): Alert Template → load-defaults. The StarTree templates are actively maintained and updated regularly. To update the templates in your environment, run the same command. > [!NOTE] > > Updates are backward compatible. They will not break your alert configuration nor change the alert behavior, except for bug fixes. If you don't want to update the existing templates and only want to add new templates, use `-d "updateExisting=false"` in the command above. ## Create a new template 1. Click **Create** → **Create Alert Template**. 2. Create your template. The JSON schema is the same as the template field in an alert. See [alert configuration](/thirdeye/alert-configuration#template). 3. Add a `name` and a `description`. 4. Use properties in format `${myProperty}` where you want the detection logic to be configurable. These properties will be set by the alert using the template. ![](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/_create_template.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=aae611270fe7693f121fc37b782d330c) > [!NOTE] > > Most of the time, the template creation process looks like this: > > * Create an alert - experiment, make sure it works fine and validate it can be reused for similar problems. > * Copy the template of the alert inside an alert template. > * Add a `name` and a `description` to the template. Save. > * Reuse the alert ## Use the template See [how to use the alert creation UI](/thirdeye/how-tos/alert/how-to-use-alert-creation-ui). # Connect a Data Source Source: https://docs.startree.ai/thirdeye/how-tos/database A `datasource` is an entity, such as an Apache Pinot instance. Data sources must be registered in ThirdEye to enable ThirdEye to use them. A datasource can have many `datasets` which are like tables. Those datasets can have many `metrics`. Currently ThirdEye supports connecting with Apache Pinot. To connect to other data sources, [reach out to StarTree](https://startree.ai/demo). ## Steps to register a new datasource ![Add Datasource](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddDatasource.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=23f4807baa33afa70654848864ead20b) 1. Visit the Configuration Page 2. Click on Create 3. Click on "Create Datasource" ![Add Datasource Form](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddDatasourceForm.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=49155275ecd07bf02f0851a4a9d21186) 1. Enter the configuration for your datasource 2. Optionally you can choose to auto onboard all the datasets that currently exist in the datasource. Note that this is the only time you can do this 3. Click Next and Finish ## Steps to add a new dataset ![Add Dataset](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddDataset.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=5ccdf8a686aa74eb1ed205bbf669ab01) 1. Visit the Configuration Page 2. Click on Create 3. Click on "Onboard Dataset" ![Add Dataset](https://mintcdn.com/startree/DzRCNOlJNN6ss-9j/img/thirdeye/AddDatasetForm.png?fit=max\&auto=format\&n=DzRCNOlJNN6ss-9j\&q=85\&s=e40250941d2e520145153eefb0b564ad) 1. Enter the name of the dataset you want to add. Make sure its the exact table name 2. Choose the datasource the dataset is from 3. Click Next and Finish # Pinot as a ThirdEye Datasource Source: https://docs.startree.ai/thirdeye/how-tos/database/pinot This datasource plugin is used to connect to Pinot using the [pinot-java-client](https://docs.pinot.apache.org/users/clients/java). It provides multiple ways to connect to Pinot instance such as using: * Zookeeper and controller * Broker and controller * Pinot with authorization (custom headers) ## Zookeeper and controller This option is preferable when Pinot components are accessible by Thirdeye using the endpoints exposed by `/instances/{instanceName}`. ### Configuration ```json theme={null} { "name": "pinotQuickStartLocal", "type": "pinot", "properties": { "zookeeperUrl": "localhost:2123", "clusterName": "QuickStartCluster", "controllerConnectionScheme": "http", "controllerHost": "localhost", "controllerPort": 9000 } } ``` The benefit of using this configuration is that we don't need to worry about the brokers as the list of brokers to query is decided with the help of Zookeeper. This configuration may not work when Pinot is sitting inside a remote cluster and Zookeeper is not exposed or, even if the Zookeeper is exposed, the broker URLs that it provides will mostly be internal and inaccessible from outside. ## Broker and Controller If you experience the problem mentioned above, then this configuration may solve it. This requires the Pinot brokers to be accessible by ThirdEye. ### Configuration ```json theme={null} { "name": "pinotds", "type": "pinot", "properties": { "zookeeperUrl": "localhost:2123", "clusterName": "pinot-quickstart", "controllerConnectionScheme": "http", "controllerHost": "localhost", "controllerPort": 9000, "brokerUrl": "localhost:8099" } } ``` Here, even though we provide `zookeeperUrl` it is just a placeholder and has no impact once we provide `brokerUrl`. One drawback of this configuration is that we are providing a static broker URL, hence all queries will hit one broker unless all broker replicas are sitting behind a load balancer. ## Pinot with authorization (custom headers) We can configure the datasource to pass custom headers along with the requests that are made to Pinot. The obvious use case for this is when authorization is enabled on Pinot, and we are required to pass the `Authorization` header along with the requests. ```json theme={null} { "name": "pinotQuickStartLocal", "type": "pinot", "defaultQueryOptions": {}, "properties": { "zookeeperUrl":"localhost:2123", "clusterName": "QuickStartCluster", "controllerConnectionScheme": "http", "controllerHost": "localhost", "controllerPort": 9000, "oauth": { "enabled": true, "tokenFilePath": "/var/run/secrets/kubernetes.io/serviceaccount/token" } } } ``` Any header can be passed as a key value pair to the `headers`. # Pinot SQL as a ThirdEye Datasource Source: https://docs.startree.ai/thirdeye/how-tos/database/pinot-sql This datasource plugin is used to connect to Pinot using the [pinot-jdbc-client](https://docs.pinot.apache.org/users/clients/jdbc). > [!NOTE] > > This datasource type is still work in progress and may not work for Pinot with ssl/tls and auth, and may also fail with some alert features. ## Configuration ```json theme={null} { "name": "pinotDatasourceSql", "type": "pinot-sql", "properties": { "controllerHost": "localhost", "controllerPort": 9000, "controllerConnectionScheme": "http" } } ``` # Managed Pinot as a ThirdEye Datasource Source: https://docs.startree.ai/thirdeye/how-tos/database/startree-pinot Apache Pinot databases managed by StarTree are automatically connected to StarTree ThirdEye.\ You can find the configuration in Configuration → Datasources. # Manage Holiday Effects with StarTree ETS Source: https://docs.startree.ai/thirdeye/how-tos/manage-holiday-effect Some StarTree ThirdEye models can learn the impact of special events like holidays. Here is an example configuration. ```json theme={null} { "name": "pageviews-ets-with-holiday", "description": "", "template": { "name": "startree-ets" }, "templateProperties": { "dataSource": "pinot", "dataset": "pageviews", "aggregationFunction": "sum", "seasonalityPeriod": "P7D", "lookback": "P90D", "monitoringGranularity": "P1D", "sensitivity": "3", "aggregationColumn": "views", "eventSqlFilter": "'US' member of dimensionMap['countryCode']", "eventTypes": [ "HOLIDAY" ] }, "cron": "0 0 5 ? * * *" } ``` *pageviews\_ets\_holiday.json* The `eventSqlFilter` and `eventTypes` are described [here](/thirdeye/operators/event-fetcher). ### Compatible templates ``` startree-ets startree-ets-percentile ``` For templates that cannot learn the effect of holidays, it is possible to use a filter to ignore anomalies that happen during special events.\ Use the optional parameters: ```json theme={null} { ... "templateProperties": { "eventFilterIgnore": true, "eventFilterSqlFilter": "", "eventFilterTypes": [], "eventFilterBeforeHolidayMargin": "P0D", "eventFilterAfterHolidayMargin": "P0D" } } ``` # Use Pre-Computed Time Columns Source: https://docs.startree.ai/thirdeye/how-tos/optimize/optimize_queries_with_precomputed_timebuckets To begin, it's useful to understand how ThirdEye generates queries. ## Understanding ThirdEye query generation ThirdEye can monitor metrics at any granularity. Consider this alert: ```json theme={null} { "name": "example-alert", "template": { "name": "startree-threshold" }, "templateProperties": { "dataSource": "pinotQuickStartLocal", "dataset": "order_events", "aggregationFunction": "count", "aggregationColumn": "customer_id", "monitoringGranularity": "PT1H", ... # other parameters }, "cron": "0 0 0 1/1 * ? *" } ``` *example\_alert.json* `monitoringGranularity` is `PT1H`. This means ThirdEye will aggregate data into buckets of 1 hour. `monitoringGranularity` can be changed to 15 minutes `PT15M` or daily `P1D`, ThirdEye will generate the corresponding SQL query. A generated query looks like this: ```sql theme={null} SELECT DATETIMECONVERT(, , 1:MILLISECONDS:EPOCH, ) as ts, COUNT(metric), FROM order_events WHERE ... GROUP BY DATETIMECONVERT(timeColum, , 1:MILLISECONDS:EPOCH, ) ORDER BY ts ``` `GROUP BY DATETIMECONVERT(, , 1:MILLISECONDS:EPOCH, )` means: take the time column, convert it to milliseconds and group by buckets of ``. This time transformation can make the query slow, especially when there are a lot of rows in the Pinot table. To avoid this performance hit, you can add a time column that corresponds to the specific\ granularity you want to use and configure ThirdEye to use it. ## Setting pre-computed time columns in dataset configurations Pre-computed time columns are set in the dataset configuration. Your dataset configuration details are found at `https:///configuration/datasets/all`. You can click on a dataset there to edit it. Consider this dataset configuration: ```json theme={null} { "name": "pageviews", "dimensions": [ "country", "browser", ... ], "timeColumn": { "name": "pageviewdate", "interval": 0.001, "format": "SIMPLE_DATE_FORMAT:yyyy-MM-dd", "timezone": "UTC" }, "dataSource": { "name": "pinot" } } ``` *dataset.json* If you know you will often use an hourly monitoringGranularity `PT1H`: 1. add a pre-computed time column in your pinot table that correspond to hourly buckets 2. update the dataset configuration `timeColumns` metadata: ```json theme={null} { "name": "pageviews", "dimensions": [ "country", "browser", ... ], "timeColumn": { "name": "pageviewdate", "interval": 0.001, "format": "SIMPLE_DATE_FORMAT:yyyy-MM-dd", "timezone": "UTC" }, "dataSource": { "name": "pinot" }, #### ADD THIS #### "timeColumns": [ { "name": "hourlyBuckets", # the name of the column "format": "1:HOURS:EPOCH", # the Pinot format of the column "granularity": "PT1H" # the granularity of the buckets - in ISO-8601 format "timezone": "UTC" # the timezone of the buckets } ] } ``` *dataset\_with\_precomputed\_time\_column* With this change, when an alert is in timezone UTC and has granularity hourly `PT1H`, ThirdEye will optimize the query using the `hourlyBuckets` column. For other configurations, it will use the default time column. You can add as many precomputed time columns as you want. For time format see [timeFormat strings](/thirdeye/operators/data-fetcher#timeformat-strings). # How to perform root cause analysis Source: https://docs.startree.ai/thirdeye/how-tos/perform-root-cause-analysis > [!NOTE] > > Before you perform the actions on this page, you must have configured a data source, data set, and created an alert. In this guide we learn how to perform root cause analysis. ## Find anomalies First, find an anomaly to investigate. > [!NOTE] > > If you received an anomaly email you can skip to the [anomaly chart](#anomaly-chart) section of this guide. Click on the tab on the left menu to view all anomalies. You should see the following screen: ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/anomalies.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=42f6a5f4b7272dace676368454da932e) Select an anomaly by clicking on the anomaly in the `Name` column. You should then see the following screen: ![An anomaly](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/anomaly2.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=918f37eb310545b25da362d326e81b78) An anomaly At the top of the screen is a calendar control that we can use to control the date range shown on the screen. ![Date range selection](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/date_select.gif?s=7917eded415d855bda19ba91c1b9577a) Date range selection Below is a summary of the anomaly showing a chart of varied values over time. ![Anomaly Summary](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/anomaly_chart_view.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=9c1f6f0db07ab04ee77e7e779e83f38b) Anomaly Summary Finally we have a heatmap at the bottom of the screen that allows us to drill down into the data based on various properties. ![Heatmap](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/heatmap.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=2344078109f5c4840720e3ef8f7a8d85) Heatmap ## Anomaly chart Let's have a look at how to use the anomaly chart. The animation below shows how to use the various controls: ![Investigating the anomaly chart](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/anomaly-chart2.gif?s=b83d7effdb1c7f9e9736d679d17c2f63) Investigating the anomaly chart You can update the chart to include add or remove anomalies, upper and lower bounds, as well as the values themselves. You can also zoom in to specific parts of the chart to do further investigation, as shown in the animation below: ![Investigating the anomaly chart](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/anomaly-chart-zoom2.gif?s=125c8cf155fff353341a4c017a40b35e) Investigating the anomaly chart ## Heatmap Next let's have a look at how to use the heatmap. The heatmap shows how the data has changed compared to the same period seven days ago. You can then drill down by each property to see the current value and the percentage change, as shown in the animation below. ![Investigating the heatmap](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/heatmap2.gif?s=5a91d577299c5c2076f80cdc74f9d932) Investigating the heatmap In this example, there's been a significant decrease in page views for the Chrome browser. Our next step is to determine what's changed, causing Chrome's page views to decrease. ## Events You could also ingest events like holiday events and code check-ins, for example, to ThirdEye. Let's have a look at how to create an event. To view your events, click on the **Events** tab on your Configuration page. You would then see your list of events. ![Events page](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/events.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=1cd3a33dbe9e2b27aa1d32aeb1f38be9) Investigating related events If you do not have events and want to create an event, click on the **Create** dropdown button and select **Create Event**. For ingesting events in bulk, you could view our API Swagger documentation for available Event API routes to configure event ingestion. ![Create event](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/create_event.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=a5e9b12e80b39f919b2f268d93de7480) # Monitoring Ad Campaign Performance Source: https://docs.startree.ai/thirdeye/how-tos/thirdeye_recipes/ad_campaign_performance_monitoring Ad campaigns are very crucial to the business and success of a product or the company. The difference in the customer impact created by the advertisements can decide whether a product skyrockets or fails in the market. These high stakes makes it critical to measure the performance of an ad campaign and take instant measures to improve the performance and make the campaign successful. One of the important factor here is identifying the issues as early as possible as every aspect is time sensitive and losses are exponential with respect to time. Minimizing the damage by identifying and investigating the issue in real time is where ThirdEye comes into picture. In this recipe, we'll use Startree ThirdEye to monitor ad campaign data and investigate the anomalies generated by ThirdEye for the anomalous behaviors. ## Architecture diagram ![image](https://user-images.githubusercontent.com/98617940/199865359-7d6b77ea-6361-4241-9b4a-ee4272f0d8c0.png) ## Here are the steps to quickly try this out * Data preparation and availability * Data ingestion * Alert creation and notifications * Anomaly reporting * Root-cause analysis: anomaly investigation ### Data Preparation and Availability * **Identify key metrics** to monitor. One of the core metrics for learning about ad campaign is: * Number of clicks * **Identify the dimensions** that are applicable for a given metric: * Country * Exchange * **Decide granularity for your detection**: * Granularity enables you for real-time or near real-time monitoring (hourly/daily/weekly time series). | AdCampaignSize | AdCampaignType | CampaignDeliveryTimestamp | Clicks | Country | Exchange | FailedBids | Impressions | LostBids | | - | - | - | - | - | - | - | - | - | | Small | Display | 1512867600 | 22037.0945 | USA | DoubleClick | 2455.1668 | 6784.6904 | 2973.6278 | | Small | Display | 1512871200 | 22505.3753 | USA | DoubleClick | 2473.1984 | 6839.1905 | 2940.1981 | | Small | Display | 1512874800 | 22758.7071 | USA | DoubleClick | 2485.5029 | 6909.3346 | 3003.7753 | | Small | Display | 1512878400 | 23112.5034 | USA | DoubleClick | 2590.7555 | 7158.9725 | 2966.8914 | | Small | Display | 1512882000 | 23830.7055 | USA | DoubleClick | 2639.0802 | 7235.7574 | 2950.0345 | | Small | Display | 1512885600 | 23979.5515 | USA | DoubleClick | 2666.9261 | 7176.2341 | 3086.7601 | | Small | Display | 1512889200 | 24021.3573 | USA | DoubleClick | 2683.2993 | 7427.7744 | 2969.4343 | | Small | Display | 1512892800 | 24228.0634 | USA | DoubleClick | 2792.0565 | 7568.5963 | 3047.8218 | | Small | Display | 1512896400 | 24914.9754 | USA | DoubleClick | 2896.0142 | 7854.9466 | 3034.1817 | | Small | Display | 1512900000 | 25691.9473 | USA | DoubleClick | 2804.6276 | 7685.0893 | 3032.0785 | ### Data ingestion * Download this [CSV file](https://docs.google.com/spreadsheets/d/1SsO_1f6DXCmuyebpmxNuARpZPg-X9u2FpshgWPq8tZk/edit?usp=sharing). * Ingest the CSV data into Pinot \[using Data Manager] (/docs/use-data-manager/upload-file)) or the Pinot API. * The Pinot schema should look like this: ```json theme={null} { "schemaName": "AdCampaignData", "dimensionFieldSpecs": [ { "name": "AdCampaignSize", "dataType": "STRING" }, { "name": "Country", "dataType": "STRING" }, { "name": "AdCampaignType", "dataType": "STRING" }, { "name": "Exchange", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "Impressions", "dataType": "DOUBLE" }, { "name": "Clicks", "dataType": "DOUBLE" }, { "name": "LostBids", "dataType": "DOUBLE" }, { "name": "FailedBids", "dataType": "DOUBLE" } ], "dateTimeFieldSpecs": [ { "name": "CampaignDeliveryTimestamp", "dataType": "LONG", "format": "1:SECONDS:EPOCH", "granularity": "1:SECONDS" } ] } ``` > [!NOTE] > > The “**dateTimeFieldSpecs**” will be used to set the granularity and understand the seasonality for accurate predictions by anomaly detectors. ### **Alert creation and notifications** * See how to [create an alert](/thirdeye/getting-started/create-your-first-alert)) and use the following alert configurations to create alerts * Subscribe to notifications ([link](/thirdeye/how-tos/add_notification)) ### Create a simple percentage rule-based alert ```json theme={null} { "name": "AdCampaignData-percentage-sample-alert-test", "description": "Monitor number of clicks", "template": { "name": "startree-percentage-rule" }, "templateProperties": { "dataSource": "pinot", "dataset": "AdCampaignData", "aggregationFunction": "sum", "seasonalityPeriod": "P7D", "monitoringGranularity": "P1D", "aggregationColumn": "Clicks", "baselineOffset": "P7D", "percentageChange": "0.01" }, "cron": "0 0 5 ? * * *" } ``` ### Use the following alert configuration for “Startree-ets alert creation” Advanced detection model based on metrics pattern and seasonality ```json theme={null} { "name": "AdCampaignData-seasonal-sample-alert", "description": "Monitor number of clicks", "template": { "name": "startree-ets" }, "templateProperties": { "dataSource": "pinot", "dataset": "AdCampaignData", "aggregationFunction": "sum", "seasonalityPeriod": "P7D", "lookback": "P20D", "monitoringGranularity": "P1D", "sensitivity": "1", "aggregationColumn": "Clicks" }, "cron": "0 0 5 ? * * *" } ``` **Use the following alert configuration for “Startree-ets multidimensional alert creation”** [Exponential smoothing (ETS)](https://docs.aws.amazon.com/forecast/latest/dg/aws-forecast-recipe-ets.html) is a technique for smoothing time series data that uses a window or tapering function. This is an advanced detection model based on metrics pattern and seasonality plus dimension exploration (multiple-timeseries). [Learn more about dimension exploration concepts](/thirdeye/dimension-exploration). ```json theme={null} { "name": "AdCampaignData-seasonal-dx-sample-alert", "description": "Monitor number of clicks across multiple dimensions such as Country and Ad Campaign Size", "template": { "name": "startree-ets-dx" }, "templateProperties": { "dataSource": "pinot", "dataset": "AdCampaignData", "aggregationFunction": "sum", "seasonalityPeriod": "P7D", "lookback": "P20D", "monitoringGranularity": "P1D", "sensitivity": "1", "aggregationColumn": "Clicks", "queryFilters": "${queryFilters}", "enumerationItems": [ { "name": "Overall", "params": { "queryFilters": "" } }, { "name": "DoubleClick-Canada", "params": { "queryFilters": " AND Country='Canada'" } }, { "name": "DoubleClick-US", "params": { "queryFilters": " AND Country='USA'" } } ] }, "cron": "0 0 5 ? * * *" } ``` ### Anomaly reporting Anomalies can be reported in multiple ways using ThirdEye, including: * Email * Slack * Webhook * Using APIs * Viewing the anomalies directly in ThirdEye Learn different ways to [add notifications](/thirdeye/how-tos/add_notification) (for example, email, Slack, and webhooks). You can also [use the ThirdEye API](/thirdeye/how-tos/use-the-api) to report anomalies your custom application, or [access anomalies in the ThirdEye UI](/thirdeye/how-tos/perform-root-cause-analysis). ### Analyze anomalies to find the root cause Follow [this guide](/thirdeye/how-tos/perform-root-cause-analysis) to perform root cause analysis with heatmaps, custom events, and other signals. From this analysis you will see that out of the few anomalies that are detected by ThirdEye, the one detected from March 28, 2018 to March 29, 2018 shows a dip of around 120k clicks fewer than predicted. In the heatmap, you'll see no prominent dimension contributing to the dip. ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/ad-campaign-investigation-0.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=98c18f09b7a4ac31db6fecc83bbed2d3) But if you observe the list of events around that time, you'll see why the dip happened. Around the same time, two events took place: * ProductCatalogChange * API Failures ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/ad-campaign-investigation-1.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=09c6ce2f60716839169b6da57306a6eb) After noting the correlation between the dip and the two events, we can narrow our focus to these two events and validate our findings because we now see it is unlikely that the API failures were related to the ad campaign section, so we will assume there is nothing wrong with the ad campaign strategy. But it's also interesting to see if the change in the catalog was not well received at first by the users, and it eventually worked (as the following trend looks normal). This can be an input to the catalog designers that users experienced some initial friction upon the catalog changes. ## Want to learn more? To learn more, join the [StarTree Slack community](https://join.slack.com/t/startreecommunity/shared_invite/zt-1h7hwy5g3-1xAIxXtaScIS4iKIiLrK~w). StarTree ThirdEye comes with a lot of API calls [link](/thirdeye/how-tos/use-the-api) you can use to build your own custom portals/web wrappers over StarTree ThirdEye. As a user, you have more control over using and customizing StarTree ThirdEye in creating great user experiences. ## Interested in trying out StarTree ThirdEye? Follow this [link](/thirdeye/getting-started/install-thirdeye) to find different deployment options for trying out StarTree ThirdEye. # Monitoring Consumer Product User Experience KPIs Source: https://docs.startree.ai/thirdeye/how-tos/thirdeye_recipes/consumer_product_user_experience In this recipe, we'll use StarTree ThirdEye to monitor the "number of page views across multiple dimensions such as location, device, and browser". Number of page views is a critical metric to measure the success of consumer product user experience. In addition to monitoring the page views, this recipe covers how to perform root cause analysis to identify which dimensions changed and contributed the most to the outlier/deviation in the metric (spike or drop or trend change). **Monitor KPIs** ![Monitor KPIs](https://user-images.githubusercontent.com/98617940/199865678-bbfa5f2d-d058-4d12-b36a-ad95e4ff5591.png) **Detect outliers and investigate** ![Detect outliers and investigate](https://user-images.githubusercontent.com/98617940/199865728-365885c0-021e-43d6-b66c-4b9d30cab191.png) ## Challenges that impact user experience User experience is crucial to the success of every consumer product and business. Improving user experience increases user engagement and revenue for consumer products, including retail, eCommerce, and fintech. For example, increasing user engagement for consumer user-facing products such as LinkedIn and Facebook can result in increased ads revenue. Companies with the highest user engagement scores have generated twice as much shareholder value over the last few years relative to the average score. * Unexpected surges in user traffic to your website could result from an A/B test or a new campaign launch. Knowing the root cause can help you make informed decisions for running A/B tests and campaign performance management. * Unexpected drops in website traffic are important to monitor. For example, a browser upgrade may increased load time for pages or timeout on your website, which can drive users away from your retail website resulting in revenue loss. ## What is the solution? Anomaly detection in user experience To drive user engagement and improved user experience for consumer products, one must constantly monitor the essential consumer product KPIs (key performance indicators) such as page views to detect anomalies across geographies, products/services, and devices (browser/desktop), etc. An anomaly is an unexpected change. Anomaly detection can help by detecting the anomaly, notifying the product analytics and business operations teams, and providing the root cause to make better-informed decisions. ## How StarTree ThirdEye will help improve the user experience? Metrics can be explored and analyzed in any data visualization and reporting platform such as Superset, Tableau, and PowerBI – The visualization tools do not support: * Analyzing multi-dimensional metrics * Smart alerting mechanisms with shifting metrics trends * RCAs (root cause analysis) Here are some benefits to using StarTree ThirdEye: * Determining which devices and geographies were impacted and by how much on a daily basis is a laborious task. StarTree ThirdEye can analyze multi-dimensional metrics (for example, page views can be analyzed by multiple dimensions such as geography, devices, and browsers) and can generate multiple time series for the same metric. * Monitoring the “page views” metric across many geographies and devices (desktop vs mobile) will generate many time series for the “page views” metric across all the permutations of geographies and devices. * Lacking a robust and smart alerting mechanism to identify the most impacted area will result in blind spots and increased costs due to manual investigations. StarTree ThirdEye has a smart alerting feature that can configure a multi-level alerting hierarchy based on the severity of the alerts and the multiple dimensions of a metric with a single alert configuration. It sends notifications to slack, email, and custom notifications platforms (using webhooks). It can also associate with custom events (such as browser upgrades that could have resulted in a drop in page views). * Analyzing and identifying the root cause of an issue across multiple dimensions and related events in real-time is of the most difficult challenges for every organization. StarTree ThirdEye can perform root cause analysis across metrics, dimensions, and the impact of one event on the metrics in an easy-to-consumable graphic. ### Identify patterns of seasonality Page views seasonality by the weekend: Many retail websites have spikes or drops during holidays or weekends. Identifying outliers/anomalies in a data stream is useless if the algorithms cannot account for seasonal fluctuations. Here is a sample visualization of daily page views over a period of time as compared to simple threshold-based alerting. As seen here, StarTree ThirdEye does not flag every dip as an anomaly. It can identify the anomalies while considering the seasonality. ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/dailypageviews.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=6d4f34f495890e603bf9e0758a295656) > [!NOTE] > > Join the StarTree Slack community to ask questions and learn more ([link](https://join.slack.com/t/startreecommunity/shared_invite/zt-1h7hwy5g3-1xAIxXtaScIS4iKIiLrK~w)). StarTree ThirdEye comes with a lot of API calls ([link](/thirdeye/how-tos/use-the-api)) making it possible for you to build your own custom portals/web wrapper over StarTree ThirdEye. As a user, you have more control over using and customizing StarTree ThirdEye in creating great user experiences. ## Interested in trying out StarTree ThirdEye? Follow this [link](/thirdeye/getting-started/install-thirdeye) to find different deployment options for trying out StarTree ThirdEye. ## Architecture diagram ![image](https://user-images.githubusercontent.com/98617940/199865359-7d6b77ea-6361-4241-9b4a-ee4272f0d8c0.png) ## Use StarTree ThirdEye anomaly detection for real-time or near real-time monitoring 1. Data preparation and availability 2. Data ingestion 3. Alert creation and notifications 4. Tuning anomaly detection models (if needed based on use-case and context) 5. Anomaly reporting 6. Root-cause analysis: Anomaly investigation ### Data preparation and availability * **Identify key metrics** to monitor. One of the core metrics for learning about user engagement is: * Number of pageviews * **Identify the dimensions** that are applicable for a given metric: * Country * Browser version * Device * Mobile OS version * **Decide granularity for your detection**: Granularity enables you for real-time or near real-time monitoring (hourly/daily/weekly time series). ### Types of time series data ![](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/timeseriesdata.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=d3172f343283a71e4e4eb1e6b6c8ddb2) | country | version | browser | gender | device | os | views | pageviewdate | | - | - | - | - | - | - | - | - | | US | 0.3 | chrome | M | phone | iOS | 4888 | 2020-02-02 | | US | 0.3 | chrome | M | phone | Android | 5703 | 2020-02-02 | | US | 0.3 | chrome | M | phone | OSX | 1629 | 2020-02-03 | | US | 0.3 | chrome | M | phone | Windows | 3259 | 2020-02-03 | | US | 0.3 | chrome | M | phone | Linux | 814 | 2020-02-03 | | US | 0.3 | chrome | M | desktop | iOS | 2851 | 2020-02-03 | | US | 0.3 | chrome | M | desktop | Android | 3326 | 2020-02-04 | | US | 0.3 | chrome | M | desktop | OSX | 950 | 2020-02-04 | ### Data ingestion * Download this CSV file ([link](https://drive.google.com/file/d/1CFgDpKueUDOCVBpxI_e36mv7YX9X0KEb/view?usp=sharing)). * Ingest this CSV data to Pinot using the Data Manager or the Pinot API. * The Pinot schema should look like this: ```json theme={null} { "schemaName": "CleanPageViewDataDemo", "**dimensionFieldSpecs**": [ { "name": "browser", "dataType": "STRING" }, { "name": "country", "dataType": "STRING" }, { "name": "device", "dataType": "STRING" }, { "name": "gender", "dataType": "STRING" }, { "name": "os", "dataType": "STRING" }, { "name": "version", "dataType": "STRING" } ], "**metricFieldSpecs**": [ { "name": "views", "dataType": "LONG" } ], "**dateTimeFieldSpecs**": [ { "name": "pageviewdate", "dataType": "STRING", "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd", "granularity": "1:MILLISECONDS" } ] } ``` > [!NOTE] > > The “**dateTimeFieldSpecs**” will be used to set the granularity and understand the seasonality for accurate predictions by anomaly detectors. ### **Alert creation and notifications** * Refer to this how-to guide to create an alert ([link](/thirdeye/getting-started/create-your-first-alert)) and use the following alert configurations to create alerts * Subscribe to notifications ([link](/thirdeye/how-tos/add_notification)) **Use the following alert configuration for “Simple threshold based alert creation”** ```json theme={null} { "name": "threshold-alert-views-sample-alert", "description": "Threshold alert on page views", "template": { "name": "startree-threshold" }, "templateProperties": { "dataSource": "pinot", "dataset": "CleanPageViewDataDemo", "aggregationFunction": "max", "aggregationColumn": "views", "monitoringGranularity": "P1D", "max": "15000", "min": "3000" }, "cron": "0 0 0/2 1/1 * ? *" } ``` **Use the following alert configuration for “Startree-ets alert creation” (advanced detection model based on metrics pattern and seasonality)** ```json theme={null} { "name": "Startree-ETS-views-sample-alert", "description": "Uses template startree-ets.", "template": { "name": "startree-ets" }, "templateProperties": { "dataSource": "pinot", "dataset": "CleanPageViewDataDemo", "aggregationFunction": "sum", "aggregationColumn": "views", "seasonalityPeriod": "P30D", "lookback": "P120D", "monitoringGranularity": "P1D", "sensitivity": "7" }, "cron": "0 0 0 1/1 * ? *" } ``` ******Use the following alert configuration for “Startree-ets alert creation” (advanced detection model based on metrics pattern and seasonality + dimension exploration (multiple-timeseries)) (******[******Learn more about the concepts.******](/thirdeye/operators/dimension-exploration)******)****** ```json theme={null} { "name": "pageviewsdemo-ets-dx", "description": "", "template": { "name": "startree-ets-dx" }, "templateProperties": { "dataSource": "pinot", "dataset": "CleanPageViewDataDemo", "aggregationFunction": "sum", "seasonalityPeriod": "P7D", "lookback": "P20D", "monitoringGranularity": "P1D", "sensitivity": "1", "aggregationColumn": "views", "queryFilters": "${queryFilters}", "enumerationItems": [ { "name": "overall", "params": { "queryFilters": "" } }, { "name": "US", "params": { "queryFilters": " AND country = 'US'" } }, { "name": "chrome", "params": { "queryFilters": " AND browser = 'chrome'" } } ] }, "cron": "0 0 5 ? * * *" } ``` ### Tuning anomaly detection models To reduce the number of false alarms, based on the business logic and contexts, StarTree ThirdEye allows you to customize anomaly criteria in the following way (Learn more about alert tuning by clicking this [link](/thirdeye/how-tos/alert-tuning)): Adjust the sensitivity and rules (you can use query filters) at various levels (metric, group, series) * Metric level: In the alert config you have just provided metric and no filters then it will tune the time series under that metrics * Specific dimension value or group of dimensions: In the alert, config specify query filters and this gives the ability to tune the time series at a group of dimensions level for the given metrics * Once anomaly detected feedback can be given on the anomaly or missed anomaly can be reported for adaptive learning and better accuracy * StarTree ThirdEye allows you to configure "Anomaly filters" * Coldstart filter * Weekday filter * Threshold filter * Guardrail metric threshold filter * Holiday filter ```json theme={null} { "name": "pageviews-sample-tune-alert", "description": "test", "template": { "name": "startree-ets" }, "templateProperties": { "dataSource": "pinot", "dataset": "CleanPageViewDataDemo", "aggregationFunction": "sum", "lookback": "P28D", "monitoringGranularity": "P1D", "sensitivity": "8", "aggregationColumn": "views", "queryFilters": "AND browser='chrome' AND os='iOS'", "eventSqlFilter": "'US' member of dimensionMap['countryCode']", "eventTypes": [ "HOLIDAY" ], "daysOfWeek": [], "hoursOfDay": [], "dayHoursOfWeek": {}, "thresholdFilterMin": "-1", "thresholdFilterMax": "-1", "guardrailMetricMin": "-1", "guardrailMetricMax": "-1", "guardrailMetric": "COUNT(*)" }, "cron": "0 0 5 ? * * *" } ``` ### Anomaly reporting Anomalies can be reported in multiple ways using ThirdEye, such as: * Email * Slack * Webhook * Using APIs * Viewing the anomalies directly in ThirdEye Here is the [link to a “how to guide”](/thirdeye/how-tos/add_notification) which describes how the different types of notifications (Email, Slack, Webhook) can be used to report anomalies. You can also pull these anomalies using the ThirdEye API [link to guide](/thirdeye/how-tos/use-the-api) and report those anomalies your custom application. Access anomalies directly in the ThirdEye UI. ([link](/thirdeye/operators/anomaly-detector)) ![](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/access_anomalies.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=19be05648e4579654edd8084a083991f) ### Root cause analysis: Anomaly investigation Here is the how-to guide to perform root cause analysis by investigating anomalies with signals from the heatmap, top contributors, and custom events: ([link](/thirdeye/how-tos/perform-root-cause-analysis)). ![](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/rca_investigate.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=b95dd066d9e10c600425c42be5809771) ![](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/rca_investigate2.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=f7a206fd57145752fce9ad4768729bd3) ![](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/rca_investigate3.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=bcc9e31dbf49297af277375ee6ca6c32) You can save investigations and provide feedback on the anomaly which can be an input to improve model performance and accuracy. ![](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/rca_investigate4.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=99c204a99b9b69718a1daf3a6aac4de6) # Monitoring Store Sales KPI for an eCommerce Retail Store Source: https://docs.startree.ai/thirdeye/how-tos/thirdeye_recipes/eCommerce_store_sales In this recipe, we'll use Startree ThirdEye to monitor the *number of orders across multiple dimensions such as products, sales channel, sales region, and store locations* using StarTree ThirdEye. Number of orders is a critical metric to measure the success of the business performance of eCommerce retail stores. In addition to monitoring the orders, this recipe covers how to perform root-cause analysis to identify which dimensions changed and contributed the most to the outlier/deviation in the metric (spike or drop, or trend change). ![image](https://user-images.githubusercontent.com/98617940/202967845-8a34645f-bc0b-452c-aaf3-a37edae985cc.png) ![image](https://user-images.githubusercontent.com/98617940/202967900-31be04a8-8cac-499b-833e-9b1faa0386c0.png) ## Common challenges faced by eCommerce retail stores in sales The COVID-19 pandemic disrupted e-commerce retail stores globally. Since 2020 there has been a significant surge in online shopping, and eCommerce retail stores are struggling to keep up with the demand. Retailers face challenges such as managing demand vs supply, pricing, and trend prediction. * Unexpected surges in-store sales can be a result of holiday sales or due to a popular campaign run or pricing changes. In ability to spot the right reason in right time can result in lost opportunities to grow the business. * An unexpected drop in-store sales can be a result of competition or user behavioral changes or market conditions or change in pricing etc. In ability to find the root-cause in a timely fashion can result in revenue losses. ## What is the solution? Anomaly detection in store sales As online sales surge, eCommerce retail stores generate a huge amount of data, making it difficult to make decisions. Anomaly detection can help you identify fluctuation in store sales, which lets you mitigate risk and open up avenues for higher sales and performance. To drive better business performance and customer engagement, you must constantly monitor the volume of sales, related dimensions, and detect anomalies across your business. ThirdEye helps detect an anomaly, notify operations and product analytics teams of the root cause, and supports making data-informed business decisions. The following are a few examples of types of anomalies found in eCommerce retail store sales: * **Order sales anomaly (sudden spike in sales):** For eCommerce retail companies, anomaly detection techniques assist teams with monitoring any sudden changes in the number of completed transactions, including an unexpected surge in demand. Spikes in demand occur for a number of reasons, including holiday shopping and sudden changes in weather conditions. By identifying increases in sales quickly, businesses can analyze their inventory levels and make important business decisions, such as ordering additional supplies or identifying alternative products that effectively meet their customers’ needs. * **Delivery anomaly:** distribution provides guidelines to ensure that products seamlessly move through the warehouses and is stocked for the right amount of time. Sometimes the distribution time does not follow the trend and results in anomalies. * **Pricing anomaly:** Anomaly detection solution makes it easier to spot hard-to-find glitches, such as a mispriced item. If revenue dropped lower than expected but conversion rates remained high, then there’s a chance that an item was mispriced, or that a teammate accidentally let a promotion go on for too long. * **Monitoring marketing campaigns:** Retailers and e-commerce businesses can use anomaly detection technology to evaluate the effectiveness of a marketing campaign or new promotions. * **Identifying business opportunities:** The anomaly detection solution assist with identifying new business opportunities, preventing revenue leakage, and better understanding of target customer’s demographics, including age, gender, and geographic location. Developing a deeper understanding of user, and their buying behaviors, is critical to deliver the products, goods, and services that users want and need. ## How StarTree ThirdEye can help improve the store sales of an eCommerce retail store? Business metrics can be explored and analyzed in any Data Visualization, and reporting platform such as Superset, Tableau, and PowerBI. However, the visualization tools do not support: * Analyzing multi-dimensional metrics * Smart alerting mechanisms with shifting metrics trends * RCAs (Root Cause Analysis) **Using StarTree ThirdEye the following benefits can be achieved** * **Dimesion drills:** Determining which store locations and sales regions were impacted and by how much on a daily basis is a laborious task. StarTree ThirdEye can analyze multi-dimensional metrics (Eg: volume of order sales can be analyzed by multiple dimensions such as store locations, sales regions, and warehouses) and can generate multiple time series for the same metric. Eg: To monitor the “volume of order sales” metric across many sales regions and store locations, it will generate many time series’ for the “# of orders” metric across all the permutations of Sales regions and store locations * **Robust smart detection and tuning:** Lacking a robust and smart alerting mechanism to identify the most impacted area will result in blind spots and increased costs due to manual investigations. StarTree ThirdEye has a smart alerting feature that can configure a multi-level alerting hierarchy based on the severity of the alerts and the multiple dimensions of a metric with a single alert configuration. It sends notifications to slack, email, and custom notifications platforms (using webhook). It can also associate with custom events (such as browser upgrades that could have resulted in a drop in page views). * **Identifying the Root Cause Analysis** across multiple dimensions and related events in real-time is of the hardest challenges for every organization. StarTree ThirdEye can perform Root Cause Analysis across metrics, dimensions, and the impact of one event on the metrics in an easy-to-consumable graphic. ## Interested in trying out StarTree ThirdEye? Follow this [link](/thirdeye/getting-started/install-thirdeye) to find different deployment options for trying out StarTree ThirdEye. ## Architecture diagram ![image](https://user-images.githubusercontent.com/98617940/202968993-7d5c85c5-a33c-461a-89a4-f84529172ac0.png) ## Steps * Data preparation and availability * Data ingestion * Alert creation and notifications * Tuning anomaly detection models (if needed based on use-case and context) * Anomaly reporting * Root cause analysis: Anomaly investigation ### Data Preparation and Availability * Identify key metrics to monitor. The core metrics for learning about store sales is: * Number of orders * Unit price * Identify the dimensions that are applicable for a given metric: * SalesRegion, StoreLocation, WarehouseCode, SalesChannel * Decide granularity for your detection: * Granularity enables you for real-time or near real-time monitoring (hourly/daily/weekly time series). **Types of time series data** ![image](https://user-images.githubusercontent.com/98617940/202969262-e6820218-2898-4d38-baef-76ac47251ed9.png) ### Data Ingestion You can download this CSV file [link](https://drive.google.com/file/d/18p3keaMOZJQHL6PkgAVSORhRe8dx2LR9/view?usp=sharing) Ingest this CSV data to Pinot using the Pinot API. The Pinot schema should look like this ```json theme={null} { "schemaName": "USStoreSalesOrderData", "dimensionFieldSpecs": [ { "name": "SalesChannel", "dataType": "STRING" }, { "name": "WarehouseCode", "dataType": "STRING" }, { "name": "SalesRegion", "dataType": "STRING" }, { "name": "CustomerName", "dataType": "STRING" }, { "name": "StoreLocation", "dataType": "STRING" }, { "name": "ProductName", "dataType": "STRING" }, { "name": "OrderQuantity", "dataType": "LONG" } ], "metricFieldSpecs": [ { "name": "DiscountApplied", "dataType": "DOUBLE" }, { "name": "UnitPrice", "dataType": "DOUBLE" }, { "name": "UnitCost", "dataType": "DOUBLE" } ], "dateTimeFieldSpecs": [ { "name": "OrderDate", "dataType": "STRING", "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd", "granularity": "1:MILLISECONDS" }, { "name": "ShipDate", "dataType": "STRING", "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd", "granularity": "1:MILLISECONDS" }, { "name": "DeliveryDate", "dataType": "STRING", "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd", "granularity": "1:MILLISECONDS" } ] } **Note: ** The “dateTimeFieldSpecs” will be used to set the granularity and understand the seasonality for accurate predictions by anomaly detectors. ``` ### Alert creation and notifications * Refer to how to guide to create an alert [link](/thirdeye/getting-started/create-your-first-alert) and use the following alert configurations to create alerts * Subscribe to notifications [link](/thirdeye/how-tos/add_notification) #### Use the following alert configuration for “Simple threshold based alert creation” ```json theme={null} { "name": "threshold-alert-order-sales-sample-alert", "description": "Threshold alert on order sales", "template": { "name": "startree-threshold" }, "templateProperties": { "dataSource": "pinot", "dataset": "USOrderSalesData", "aggregationFunction": "avg", "aggregationColumn": "OrderQuantity", "monitoringGranularity": "P1D", "max": "20", "min": "4" }, "cron": "0 0 0/2 1/1 * ? *" } ``` **Use the following alert configuration for “Startree-ets alert creation” (advanced detection model based on metrics pattern and seasonality)** ```json theme={null} { "name": "USSalesOrderData-anomaly-detection", "description": "Uses template startree-ets.", "template": { "name": "startree-ets" }, "templateProperties": { "dataSource": "pinot", "dataset": "USOrderSalesData", "aggregationFunction": "sum", "aggregationColumn": "OrderQuantity", "seasonalityPeriod": "P7D", "lookback": "P120D", "monitoringGranularity": "P1D", "sensitivity": "3" }, "cron": "0 0 0 1/1 * ? *" } ``` **Use the following alert configuration for “Startree-ets-dx alert creation” (advanced detection model based on metrics pattern and seasonality + dimension exploration (multiple timeseries)) (Learn more about the concepts by clicking this **[**link**](/thirdeye/dimension-exploration)**)** ```json theme={null} { "name": "USStoreSalesOrderData-Seasonal-DX-Demo", "description": "Uses template startree-ets-dx.", "template": { "name": "startree-ets-dx" }, "templateProperties": { "dataSource": "pinot", "dataset": "USOrderSalesData", "aggregationFunction": "sum", "aggregationColumn": "OrderQuantity", "seasonalityPeriod": "P7D", "lookback": "P120D", "monitoringGranularity": "P1D", "sensitivity": "3", "rcaAggregationFunction": "sum", "queryFilters": "${queryFilters}", "enumerationItems": [ { "name": "overall", "params": { "queryFilters": "" } }, { "params": { "queryFilters": " AND SalesRegion = 'Midwest'" } }, { "params": { "queryFilters": " AND SalesRegion = 'West'" } }, { "params": { "queryFilters": " AND SalesRegion = 'West' AND SalesChannel = 'In-Store'" } }, { "params": { "queryFilters": " AND SalesRegion = 'Northeast'" } } ] }, "cron": "0 0 0 1/1 * ? *" } ``` ### Tuning anomaly detection models To reduce the number of false alarms, based on the business logic and contexts, StarTree ThirdEye allows you to customize anomaly criteria in the following way (Learn more about alert tuning by clicking this [link](/thirdeye/how-tos/alert-tuning)): Adjust the sensitivity and rules (you can use query filters) at various levels (metric, group, series): * Metric level: In the alert config you have just provided metric and no filters then it will tune the time series under that metrics * Specific dimension value or group of dimensions: In the alert, config specify query filters and this gives the ability to tune the time series at a group of dimensions level for the given metrics * Once anomaly detected feedback can be given on the anomaly or missed anomaly can be reported for adaptive learning and better accuracy * StarTree ThirdEye allows you to configure "Anomaly filters" * coldstart filter * weekday filter * threshold filter * guardrail metric threshold filter * holiday filter ```json theme={null} { "name": "USStoresSales-sample-tune-alert", "description": "US Store sales sample tune alert", "template": { "name": "startree-ets" }, "templateProperties": { "dataSource": "pinot", "dataset": "USOrderSalesData", "aggregationFunction": "sum", "lookback": "P28D", "monitoringGranularity": "P1D", "sensitivity": "8", "aggregationColumn": "OrderQuantity", "queryFilters": " AND SalesRegion = 'West' AND SalesChannel = 'In-Store'", "eventSqlFilter": "'US' member of dimensionMap['countryCode']", "eventTypes": [ "HOLIDAY" ], "daysOfWeek": [], "hoursOfDay": [], "dayHoursOfWeek": {}, "thresholdFilterMin": "-1", "thresholdFilterMax": "-1", "guardrailMetricMin": "-1", "guardrailMetricMax": "-1", "guardrailMetric": "COUNT(*)" }, "cron": "0 0 5 ? * * *" } ``` ### Anomaly reporting Anomalies can be reported in multiple ways using ThirdEye, such as: * Email * Slack * Webhook * Using APIs * Viewing the anomalies directly in ThirdEye Here is the [link to a “how to guide”](/thirdeye/how-tos/add_notification) which describes how the different types of notifications (Email, Slack, Webhook) can be used to report anomalies. You can also pull these anomalies using the ThirdEye API [link to guide](/thirdeye/how-tos/use-the-api) and report those anomalies your custom application. Access anomalies directly in the ThirdEye UI. [link](/thirdeye/how-tos/perform-root-cause-analysis) ![image](https://user-images.githubusercontent.com/98617940/202970456-2d003cda-b916-4445-be2f-847bf3e70ade.png) ### Root-cause analysis: anomaly investigation Here is the how-to guide to perform root cause analysis by investigating anomalies with signals from the heatmap, top contributors, and custom events: [link](/thirdeye/how-tos/perform-root-cause-analysis). ![image](https://user-images.githubusercontent.com/98617940/202970669-e880e92d-cd23-4ce7-b443-8472cd7da29c.png) ![image](https://user-images.githubusercontent.com/98617940/202970717-f1777359-d120-4c88-81c2-1e2fa6e1bde7.png) ![image](https://user-images.githubusercontent.com/98617940/202970766-d6397598-2f07-4fae-bde9-122d870e975e.png) **You can save investigations and provide feedback on the anomaly which can be an input to improve model performance and accuracy.** ![image](https://user-images.githubusercontent.com/98617940/202970850-94777ef3-2b9c-4796-aba5-454dfba60d0a.png) # Using Role-Based Access Control with ThirdEye Source: https://docs.startree.ai/thirdeye/how-tos/thirdeye_recipes/rbac-recipe In this recipe, we'll create access control policies in Startree ThirdEye. Here we restrict access by region, however access control policies can be used to restrict access of any data dimension to any set of Okta groups or other user attributes. For a more in-depth explanation of how access control and authorization works in ThirdEye, see [Access Control in Thirdeye](/thirdeye/access-control-in-thirdeye). We'll create an alert with regional data in the US and Canada, and restrict access to the alert data to 3 categories of users: * Regional analysts * Assigned to Okta group `regional_analysts_us` or `regional_analysts_ca`. * View alerts for their region. * Manage anomalies and investigations for their region. * Global Viewers * Assigned to Okta group `global_viewers`. * View alerts, anomalies, and investigations for any region. * Admins/makers: * Assigned to Okta group `thirdeye_admin`. * Create alerts for any region. * Manage authorization policies. ## Recipe outline 1. Handle prerequisites and setup 2. Create the namespaces we'll use to group our ThirdEye resources 3. Restrict access to ThirdEye resources with access control policies 4. Create an alert with separate timeseries for US and Canada data using dimension exploration ## Prerequisites 1. [Obtain your StarTree cloud bearer token](/thirdeye/how-tos/use-the-api). 2. Log into the Authorization server's UI. If your ThirdEye is at [https://thirdeye.my-env.my-org.startree.cloud](https://thirdeye.my-env.my-org.startree.cloud), then the authorization server ui is at [https://my-env.my-org.startree.cloud/auth/swagger-ui/index.html](https://my-env.my-org.startree.cloud/auth/swagger-ui/index.html). The alert will use data from the [pageviews demo dataset](https://github.com/startreedata/thirdeye/tree/master/examples/pageviews) and [dimension exploration](/thirdeye/dimension-exploration) to segment the data into different regions. ## Namespaces First, we'll create a set of namespaces to group our ThirdEye alerts, anomalies and other resources. The access control policies will apply to these namespaces instead of individual resources. Namespaces are string values that must match regex `^[a-z0-9-_]+$`. We set the namespace of alerts and its enumeration items in the alert configuration. Anomalies and investigations are automatically assigned to the namespace of the alert or enumeration item it is assigned to. Resources that do not have a configured namespace are placed in the `default` namespace. In this recipe, we'll create 3 namespaces: * `thirdeye_dx_alerts` - This will contain the dimension exploration alerts. * `regional_analysts_us` - This will contain the US dimension and all related investigations and anomalies. * `regional_analysts_ca` - This will contain the CA dimension and all related investigations and anomalies. All other ThirdEye resources like alert templates, datasets, and data sources will go into the `default` namespace. ![ThirdEye resources placed in labeled containers.](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/access_control/namespace_separate_for_recipe.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=8f91281f4e19d069e89d228038e29b25) The individual namespace permissions are broken down in this table: ![ThirdEye Namespace Permissions](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/thirdeye_namespace_permissions.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=297fa74fcbe42f3c18df07287e268ba8) ### Create the namespaces Here are the JSON payloads needed to create the namespaces. To enable each, copy and paste the code directly into the Swagger UI's `create namespace` tool. > [!NOTE] > > If you get a 403 Forbidden error, this means you do not have permission to create namespaces. Reach out to ThirdEye support for help. #### thirdeye\_dx\_alerts ```json theme={null} { "name": "thirdeye_dx_alerts", "description": "ThirdEye dimension exploration alerts.", "enabled": true } ``` #### regional\_analysts\_us ```json theme={null} { "name": "regional_analysts_us", "description": "Anomalies and investigation for US.", "enabled": true } ``` #### regional\_analysts\_ca ```json theme={null} { "name": "regional_analysts_ca", "description": "Anomalies and investigation for Canada.", "enabled": true } ``` ## Policies Next, let's create the policies. Like ThirdEye resources, policies are assigned to a namespace. A policy will only apply to resources in its namespace, except for one special case: Policies in the default namespace are applied to all resources. ### Create the policies To enable each, copy and paste the code directly into the Swagger UI's `create policy` tool. #### Grant read and write access to the admin group This policy allows users in the thirdeye\_admin Okta group to read and write all entities. We assign the policy to the default namespace, so all entities are covered. ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:default:default", "priority": 1, "rule": "subject_user_groups CONTAINS 'thirdeye_admin'", "description": "Allow users in the thirdeye_admin Okta group to read and write all entities." } ``` #### Grant read access to global viewers This policy allows users in the global\_viewers Okta group to read all entities. Same as before, this policy goes in the default namespace to cover all entities. ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:default:default", "priority": 2, "rule": "subject_user_groups CONTAINS 'global_viewers' AND action='read'", "description": "Allow users in the global_viewers Okta group to read all entities." } ``` #### Grant read access for all alert templates This policy allows all users to read the alert templates. Since alert templates are used to render alerts, but cannot be used to view data by themselves, we can use a lenient read policy. This policy will cover the standard ThirdEye alert templates, which are in the default namespace. ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:default:default", "priority": 3, "rule": "resource_srn_entity='thirdeye-alert_template' AND action='read'", "description": "Allow all users to read ThirdEye alert templates." } ``` #### Grant read access for the dimension exploration alerts This policy allows all users to read the ThirdEye alerts in the `thirdeye_dx_alerts` namespace. When we create an alert, we must also assign it to the `thirdeye_dx_alerts` namespace so all users can see it. ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:thirdeye_dx_alerts:default", "priority": 4, "rule": "action='read'", "description": "Allow all users to read all entities in the thirdeye_dx_alerts namespace." } ``` #### Grant read+write access to anomalies and investigations in the regional namespaces. Finally, let's create policies to allow our regional analysts to read and write anomalies and investigations in their namespaces. ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:regional_analysts_us:default", "priority": 1010, "rule": "subject_user_groups CONTAINS 'regional_analysts_us' AND (resource_srn_entity='thirdeye-anomaly' OR resource_srn_entity='thirdeye-rca_investigation')", "description": "Allow users in the regional_analysts_us okta group to read and write anomalies and investigations in the regional_analysts_us namespace." } ``` ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:regional_analysts_ca:default", "priority": 1110, "rule": "subject_user_groups CONTAINS 'regional_analysts_ca' AND (resource_srn_entity='thirdeye-anomaly' OR resource_srn_entity='thirdeye-rca_investigation')" "description": "Allow users in the regional_analysts_ca okta group to read and write anomalies and investigations in the regional_analysts_ca namespace." } ``` #### (Optional) Grant read access to all entities in the regional namespaces We may want to create alerts without dimension exploration, but still have the anomalies and investigations stay in a regional namespace. For this use-case, we create the alert in the regional namespace instead of the `thirdeye-dx-alerts` namespace. We'll then add a policy to allow our regional analysts read access to all entities in their namespace. ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:regional_analysts_ca:default", "priority": 1100, "rule": "subject_user_groups CONTAINS 'regional_analysts_ca' AND action='read'", "description": "Allow ca viewers read access to us resources." } ``` ```json theme={null} { "policyType": "ALLOW", "namespaceSrn": "srn:zone:namespace:regional_analysts_us:default", "priority": 1000, "rule": "subject_user_groups CONTAINS 'regional_analysts_us' AND action='read'", "description": "Allow us viewers read access to us resources." } ``` ## Create the Alert and Enumeration Items Finally, it's time to create the alert. This alert uses the `startree-threshold-dx` template and the `pageviews` demo dataset. The alert has two enumeration items: one for data from the US and one for data from Canada. The alert itself is created in the `thirdeye_dx_alerts` namespace that anyone can read, and the enumeration items are created in their region-specific namespace. Check that the `dataSource` and `dataset` template properties match the names for your instances. Paste this alert body in ThirdEye's `Create Alert` JSON editor: ```json theme={null} { "name": "Views-Threshold-Dx-Recipe-Demo", "auth": { "namespace": "thirdeye_dx_alerts" }, "template": { "name": "startree-threshold-dx" }, "cron": "0 0 5 ? * MON-FRI *", "templateProperties": { "dataSource": "pinot", "dataset": "pageviews", "aggregationFunction": "SUM", "seasonalityPeriod": "P7D", "lookback": "P90D", "monitoringGranularity": "P1D", "sensitivity": "3", "aggregationColumn": "views", "queryFilters": "${queryFilters}", "max": "${max}", "min": "${min}", "enumerationItems": [ { "name": "country='US'", "params": { "queryFilters": "AND country='US'", "max": "900000", "min": "120000" }, "auth": { "namespace": "regional_analysts_us" } }, { "name": "country='CA'", "params": { "queryFilters": "AND country='CA'", "max": "900000", "min": "10000" }, "auth": { "namespace": "regional_analysts_ca" } } ] } } ``` ### Results With the access control policies in place, the alert will look different for the different user groups. #### ThirdEye admins and global viewers This is how the alert will look for the thirdeye admins and global viewers. We can see the timeseries and anomalies for both US and Canada. ![Alert seen by the thirdeye admin.](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/access_control/alert_thirdeye_admin.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=efac695bbd00226aff9e860b84ca57d0) #### Regional analysts for US This is how the alert will look for the US analysts. Only the timeseries and anomalies for the US are visible. ![Alert seen by the regional analyst in US.](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/access_control/alert_regional_analysts_us.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=1cec7669c32be63736c825dbc0d4119e) #### Regional analysts for Canada This is how the alert will look for the Canada analysts. Only the timeseries and anomalies for Canada are visible. ![Alert seen by the regional analyst in Canada.](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/access_control/alert_regional_analysts_ca.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=bf9cad1d2f9f6d1da595ae00828b5eac) # Monitoring Rideshare Booking Metrics for Customer Satisfaction Source: https://docs.startree.ai/thirdeye/how-tos/thirdeye_recipes/rideshare_recipe In this recipe, we'll use Startree ThirdEye to monitor metrics critical to rideshare customer experience, such as cancellations and wait time. In addition to monitoring, we'll also perform a root cause analysis within ThirdEye to identify dimensions that have changed and contributed the most to the outlier/deviation in the metric (spike or drop or trend change). ## Challenges faced by rideshare apps In order to be competitive, rideshare apps must provide fast and timely rides for their customers. When wait times and ride cancellations begin to spike, quick action is critical to restoring customer satisfaction. Wait times can be affected by external factors such as weather, traffic, driver availability, and more. When wait times are too long, customers begin to look for alternatives. This often leads to customers cancelling their current ride in order to try for a closer driver or choosing an alternative option. Cancelled rides add extra stress and costs to a system already struggling to provide timely rides. Quickly and confidently responding to changes in wait time and other key metrics will have a big impact on customer satisfaction and retention. ## How anomaly detection helps In order to respond to deviations in key metrics, we need a system that detects these deviations and notifies us that they have occurred. When the deviation is large enough, the side effects are readily apparent; site outages, angry customers, and damage to the brand. ThirdEye uses statistical models and dimensional analysis to detect smaller deviations and anomalies before they become big problems. Aggregate timeseries can often mask anomalies that occur in a single dimension of the data. For example, a single city might see wait times increase 10x, but this increase may have a negligible effect on the aggregate wait time across all cities. ThirdEye can run anomaly detection across multiple dimensions of data to detect these local anomalies. ThirdEye automatically detects when our key metrics unexpectedly deviate from their normal behavior, even across multiple dimensions. ThirdEye can report these anomalies to Slack, email, or an external webhook. **Anomaly detect for ride cancellations:** ![Anomaly detection for ride cancellations](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/rideshare_recipe/cancellation_anomalies_3.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=64f820f7ff70e41737a471b76ef51e99) ## Interested in trying out StarTree ThirdEye? [Follow this link](/thirdeye/getting-started/install-thirdeye) to find different deployment options for trying out StarTree ThirdEye. ## Architecture diagram ![image](https://user-images.githubusercontent.com/98617940/199865359-7d6b77ea-6361-4241-9b4a-ee4272f0d8c0.png) ## Steps to try StarTree ThirdEye anomaly detection for real-time or near real-time monitoring * Data preparation and availability * Data ingestion * Alert creation and notifications * Tuning anomaly detection models (if needed based on use-case and context) * Anomaly reporting * Root cause analysis/anomaly investigation ### Data preparation and availability * **Identify key metrics** to monitor. \ This recipe uses: * Ride\_Cancellations * Ride\_Request\_WaitTime * Driver\_Request\_API\_Errors * **Identify the dimensions** that are applicable for a given metric. \ This recipe uses: * City * Device\_Type * RideType * **Decide granularity for your detection**: * Granularity enables you for real-time or near real-time monitoring (hourly/daily/weekly time series). The sample data is aggregated into hourly buckets, so this recipe uses hourly granularity. **Types of time series data** ![image](https://user-images.githubusercontent.com/98617940/202969262-e6820218-2898-4d38-baef-76ac47251ed9.png) ### Data ingestion 1. Download this CSV file [link](https://docs.google.com/spreadsheets/d/1wO2iHjuRJqyMVnaLuFNqviVr034JlLycpgQ10APM_c8/edit?usp=sharing). 2. Ingest this CSV data to Pinot using the Pinot API. Here's a preview of the first 5 rows: | timestamp | RideType | City | Device\_Type | OS\_Version | Ride\_Cancellations | Rides\_Completed | Ride\_Request\_WaitTime | Driver\_Request\_API\_Errors | | - | - | - | - | - | - | - | - | - | | 1512867600 | Shared | Los\_Angeles | Mobile | Android | 6241.5174 | 11320.5833 | 2434.6632 | 1772.2415 | | 1512871200 | Shared | Los\_Angeles | Mobile | Android | 6840.4272 | 11429.2984 | 2611.5766 | 1886.1431 | | 1512874800 | Shared | Los\_Angeles | Mobile | Android | 7421.9474 | 11509.1625 | 2859.7099 | 1970.7529 | | 1512878400 | Shared | Los\_Angeles | Mobile | Android | 7780.715 | 11350.5748 | 2986.9979 | 2076.0819 | | 1512882000 | Shared | Los\_Angeles | Mobile | Android | 7996.7762 | 11804.1036 | 2976.7476 | 2238.1518 | The Pinot schema looks like this: ```json theme={null} { "schemaName": "rideshare", "dimensionFieldSpecs": [ { "name": "ver", "dataType": "STRING" }, { "name": "RideType", "dataType": "STRING" }, { "name": "City", "dataType": "STRING" }, { "name": "Device_Type", "dataType": "STRING" }, { "name": "OS_Version", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "Ride_Cancellations", "dataType": "FLOAT" }, { "name": "Rides_Completed", "dataType": "FLOAT" }, { "name": "Ride_Request_WaitTime", "dataType": "FLOAT" }, { "name": "Driver_Request_API_Errors", "dataType": "FLOAT" } ], "dateTimeFieldSpecs": [ { "name": "timestamp", "dataType": "LONG", "format": "1:SECONDS:EPOCH", "granularity": "1:SECONDS" } ] } ``` ### Alert creation and notifications To create our alerts, we need to inspect the timeseries for trends, seasonality, and obvious outliers. ThirdEye's alert preview lets us view the timeseries and the predicted anomalies. To start, we'll preview the data using the simplest alert temlate, `startree-threshold`. Copy and paste the following into ThirdEye's alert json editor: ```json theme={null} { "name": "", "description": "", "cron": "0 0 5 ? * MON-FRI *", "template": { "name": "startree-threshold" }, "templateProperties": { "dataSource": "mypinot", "dataset": "rideshare", "aggregationColumn": "Ride_Request_WaitTime", "aggregationFunction": "SUM", "seasonalityPeriod": "P7D", "lookback": "P90D", "monitoringGranularity": "PT1H", "sensitivity": "3", "max": "100000", "min": "0" } } ``` You should now see this timeseries in the preview: ![Timeseries of alert wait time](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/rideshare_recipe/alert_preview.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=b938899bb5ef01a84413272519f0cf47) From the preview, we see a fairly cyclic pattern each day, with some variation between the days, and a large outlier towards the end of March. The cyclic pattern is called the seasonality. In fact, all our key metrics, wait time, cancellations, and api errors, show a seasonality period of 1 day. At the beginning of the day, the metrics are fairly low, they reach a peak towards the middle of the day, and end at roughly the same value as the start of the day. A simple threshold alert will not work for this data, since the threshold would depend on the time of day. We also want to monitor different dimensions of the data, which would require their own time-dependent thresholds. Instead, we'll use the [startree-ets-dx](/thirdeye/templates/startree-ets-dx) template which is designed to model data with seasonality and trends. * [Guide: How to create an alert.](/thirdeye/getting-started/create-your-first-alert) * [Guide: How to subscribe to notifications.](/thirdeye/how-tos/add_notification) #### Alert Configuration for Wait Time ```json theme={null} { "id": 36736, "name": "Ride_Request_WaitTime", "description": "", "template": { "name": "startree-ets-dx" }, "templateProperties": { "dataSource": "mypinot", "dataset": "rideshare", "timeColumn": "timestamp", "timeColumnFormat": "EPOCH", "aggregationFunction": "SUM", "seasonalityPeriod": "P1D", "lookback": "P28D", "monitoringGranularity": "PT1H", "aggregationColumn": "Ride_Request_WaitTime", "queryFilters": "${queryFilters}", "enumerationItems": [ { "name": "overall", "params": { "queryFilters": "" } }, { "params": { "queryFilters": " AND City = 'Los_Angeles' and RideType = 'Standard'" } }, { "params": { "queryFilters": " AND City = 'Los_Angeles' and RideType = 'Shared'" } }, { "params": { "queryFilters": " AND City = 'Los_Angeles' and RideType = 'Premium'" } }, { "params": { "queryFilters": " AND City = 'New_York' and RideType = 'Standard'" } }, { "params": { "queryFilters": " AND City = 'New_York' and RideType = 'Shared'" } }, { "params": { "queryFilters": " AND City = 'New_York' and RideType = 'Premium'" } }, { "params": { "queryFilters": " AND City = 'San_Francisco' and RideType = 'Standard'" } }, { "params": { "queryFilters": " AND City = 'San_Francisco' and RideType = 'Shared'" } } ], "pattern": "UP", "sensitivity": "3" }, "cron": "0 0 5 ? * MON-FRI *", "lastTimestamp": 1677690000000, "active": true, "created": 1677686514149, "updated": 1677708931249, "owner": { "principal": "no-auth-user" } } ``` #### Alert Configuration for Cancellations ```json theme={null} { "id": 36735, "name": "Ride_Cancellations", "description": "", "template": { "name": "startree-ets-dx" }, "templateProperties": { "dataSource": "mypinot", "dataset": "rideshare", "timeColumn": "timestamp", "timeColumnFormat": "EPOCH", "aggregationFunction": "SUM", "seasonalityPeriod": "P1D", "lookback": "P28D", "monitoringGranularity": "PT1H", "sensitivity": "3", "aggregationColumn": "Ride_Cancellations", "queryFilters": "${queryFilters}", "enumerationItems": [ { "name": "overall", "params": { "queryFilters": "" } }, { "params": { "queryFilters": " AND City = 'Los_Angeles' and RideType = 'Standard'" } }, { "params": { "queryFilters": " AND City = 'Los_Angeles' and RideType = 'Shared'" } }, { "params": { "queryFilters": " AND City = 'Los_Angeles' and RideType = 'Premium'" } }, { "params": { "queryFilters": " AND City = 'New_York' and RideType = 'Standard'" } }, { "params": { "queryFilters": " AND City = 'New_York' and RideType = 'Shared'" } }, { "params": { "queryFilters": " AND City = 'New_York' and RideType = 'Premium'" } }, { "params": { "queryFilters": " AND City = 'San_Francisco' and RideType = 'Standard'" } }, { "params": { "queryFilters": " AND City = 'San_Francisco' and RideType = 'Shared'" } } ], "impactThreshold": "1000", "pattern": "UP" }, "cron": "0 0 5 ? * MON-FRI *", "lastTimestamp": 1677693600000, "active": true, "created": 1677686514097, "updated": 1677695955731, "owner": { "principal": "no-auth-user" } } ``` #### Alert Configuration for Api Errors ```json theme={null} { "name": "Driver_Request_API_Errors", "description": "", "template": { "name": "startree-ets-dx" }, "templateProperties": { "dataSource": "mypinot", "dataset": "rideshare", "timeColumn": "timestamp", "timeColumnFormat": "EPOCH", "aggregationFunction": "SUM", "seasonalityPeriod": "P7D", "lookback": "P28D", "monitoringGranularity": "PT1H", "sensitivity": "3", "aggregationColumn": "Driver_Request_API_Errors", "queryFilters": "${queryFilters}", "enumerationItems": [ { "name": "overall", "params": { "queryFilters": "" } }, { "params": { "queryFilters": " AND City = 'Los_Angeles' and RideType = 'Standard'" } }, { "params": { "queryFilters": " AND City = 'Los_Angeles' and RideType = 'Shared'" } }, { "params": { "queryFilters": " AND City = 'Los_Angeles' and RideType = 'Premium'" } }, { "params": { "queryFilters": " AND City = 'New_York' and RideType = 'Standard'" } }, { "params": { "queryFilters": " AND City = 'New_York' and RideType = 'Shared'" } }, { "params": { "queryFilters": " AND City = 'New_York' and RideType = 'Premium'" } }, { "params": { "queryFilters": " AND City = 'San_Francisco' and RideType = 'Standard'" } }, { "params": { "queryFilters": " AND City = 'San_Francisco' and RideType = 'Shared'" } } ], "pattern": "UP", "impactThreshold": "100" }, "cron": "0 0 5 ? * MON-FRI *" } ``` ### Anomaly reporting Now that the alerts are created, we view the detected anomalies for each alert and it's dimensions. To see anomalies for a single alert, click **Alerts** in the sidebar, then click on the alert you want to inspect. To see anomalies for all alerts, click on **Anomalies** in the sidebar. Below are screenshots from the anomaly detection results. The graphs are zoomed in to show the most interesting results. **Anomalies in wait time** ![Anomalies in wait time](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/rideshare_recipe/wait_time_anomalies.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=2cac6804fd9da97c1d4fe381df56c036) **Anomalies in ride cacellations** ![Anomalies in ride cancellations](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/rideshare_recipe/cancellation_anomalies_2.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=13cd97293a231eb3915c8b3ace96c9f8) **Anomalies in API errors** ![Anomalies in API errors](https://mintcdn.com/startree/ydxOgkfeGyqN5wki/img/thirdeye/rideshare_recipe/api_errors_anomalies.png?fit=max\&auto=format\&n=ydxOgkfeGyqN5wki\&q=85\&s=2f6d89548f40df803ad563b03a2f19eb) ThirdEye has several options for reporting anomalies: * [View Anomalies directly in ThirdEye.](/thirdeye/how-tos/perform-root-cause-analysis) * [Email, Slack, and webhook notifications.](/thirdeye/how-tos/add_notification) * [Programmatically with ThirdEye's API.](/thirdeye/how-tos/use-the-api) ### Root-cause analysis Follow [this guide](/thirdeye/how-tos/perform-root-cause-analysis) to perform root cause analysis with heatmaps, custom events, and other signals. # How to use the ThirdEye API Source: https://docs.startree.ai/thirdeye/how-tos/use-the-api ThirdEye exposes a CRUD API. This makes it easy to use ThirdEye as a headless service from your apps. The easiest way to discover and try the different endpoints is to use Swagger. ## Using Swagger Swagger is exposed by the coordinator at `[coordinator_url]/swagger`. If you use StarTree ThirdEye or the Helm charts, Swagger is also exposed by the frontend app at `[your_thirdeye_url]/swagger`. 1. Go to `[your_thirdeye_url]/swagger` 2. If your instance uses authentication, click the "Authorize" button and provide a [bearer token](#obtaining-a-bearer-token). 3. If your instance use HTTPS, switch HTTP to HTTPS. ![Swagger UI](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_swagger_ui.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=fa9873c7e6d1e9283b2415090266c125) 4. Try the endpoints. ### Obtaining a bearer token If your instance uses authentication, the frontend passes a bearer token when performing requests. In your browser, in the devtools, you can see the bearer token. ![Swagger bearer token](https://mintcdn.com/startree/zMiJ17Ik7Xto2JoM/img/thirdeye/_swagger_bearer_token.png?fit=max\&auto=format\&n=zMiJ17Ik7Xto2JoM\&q=85\&s=c1eed3cae268cffc3047794c82c40687) You can copy and paste the token into the Swagger UI. ### Obtaining a long-lasting token (basic auth token) **StarTree cloud users**: To obtain a basic auth token for StarTree ThirdEye, contact [StarTree Tech Support](https://support.startree.ai/support/home), and then complete **step 3** below (skipping steps 1-2). #### Step 1 - Update ThirdEye configuration in the Helm chart ```bash theme={null} auth: enabled: true basic: enabled: true users: - username: password: ``` #### Step 2 - Use the generated basic auth token in requests To authenticate a request, add the `Authorization` header with the token value as the base64 encoded value of `username:password` prefixed with `Basic`. **Example:**\ `Authorization: Basic YWRtaW46cGFzcw==` where `YWRtaW46cGFzcw==` is the base64 encoded value of `admin:pass`. To encode on macOs: ```bash theme={null} echo -D ':' | base64 ``` #### Step 3 - Use the Basic Auth token in Swagger Copy and paste the token `YWRtaW46cGFzcw==` in the Swagger UI as `Basic YWRtaW46cGFzcw==`. # Overview Source: https://docs.startree.ai/thirdeye/operators Each node in a detection pipeline uses an operator. Operators have a determined goal, inputs and outputs, and a set of parameters to customize their behavior. The operator type is given in the JSON configuration. ```json theme={null} { "name": "my-detection-rule", ... "template": { "nodes": [ { "name": "root", "type": "AnomalyDetector", # THE OPERATOR TYPE "params": { ... }, ... } ... ] } ``` This section describes all the operators. # AnomalyDetector Source: https://docs.startree.ai/thirdeye/operators/anomaly-detector `AnomalyDetector` operators take one or more inputs, and detect anomalies. There are many types of anomaly detectors. The type is given in the node parameters. ```json theme={null} { "name": "root", # name of the operator "type": "AnomalyDetector", # type of operator "params": { "type": "THRESHOLD", # type of anomaly detector, # other parameters ... "component.timestamp": "ts", "component.metric": "metric", ... }, "inputs" : [...], "outputs" : [...] } ``` ## Shared parameters Anomaly detectors share a set of parameters. Detector-type specific parameters are described in the sub-sections. | name | description | default value | | - | - | - | | `component.timestamp` | The name of the time index column. | `timestamp` | | `component.metric` | The name of the metric column to analyse. | `metric` | | `component.monitoringGranularity` | The granularity of the data used for detection. Mandatory. Used to compute the end time of an anomaly if it is the last data point of the input data. | | ## Commonly-used parameters These parameters are used by many detectors. ### `component.pattern` Detect as an anomaly if the metric drop, rise or both directions. `UP`, `DOWN`, `UP_OR_DOWN`. | name | description | | - | - | | `UP` | Detect an anomaly only if the current time series is above the baseline. | | `DOWN` | Detect an anomaly only if the current time series is below the baseline. | | `UP_OR_DOWN` | Detect an anomaly in both directions. | # ABSOLUTE_CHANGE Source: https://docs.startree.ai/thirdeye/operators/anomaly-detector/absolute Compares current time series to a baseline. If the absolute change is above a certain threshold, detect it as an anomaly. ## Inputs `"targetProperty": "current"`: The data on which to perform detection. `"targetProperty": "baseline"`: The data to use as a baseline. The inputs should have the same size. ## Parameters | name | description | default value | | - | - | - | | `component.absoluteChange` | The absolute change threshold. If the absolute change when compared to the baseline is above this threshold, detect an anomaly. | | | `component.pattern` | Detect as an anomaly if the metric drop, rise or both directions. `UP`, `DOWN`, `UP_OR_DOWN`. | `UP_OR_DOWN` | ## Example ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "ABSOLUTE_CHANGE", "component.absoluteChange": "150", "component.offset": "DOWN", ... # shared parameters }, "inputs": [ { # baseline data "targetProperty": "baseline", "sourcePlanNode": "baselineDataFetcher", "sourceProperty": "baselineOutput" }, { # current data "targetProperty": "current", "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentOutput" } ] } ``` # ETS Source: https://docs.startree.ai/thirdeye/operators/anomaly-detector/ets > [!NOTE] > > Enterprise only Compares current value to the value predicted by a model combining a regression model and an ETS forecasting algorithm(^1). The regression model learns the events effect. Eg Christmas, sales. The ETS model learns the level, the trend and the seasonality in the timeseries after the effect of events have been removed. ## Inputs `"targetProperty": "current"`: The data on which to perform detection. It should contain the historical data to use for training. `"targetProperty": "current_events"`: **Optional**. The events data on which to perform detection. It should contain the historical events data to use for training. ## Parameters | name | description | default value | | - | - | - | | `component.intervalsMethod` | Method to compute intervals. In `CONFIDENCE`, `PERCENTAGE`, `ABSOLUTE`. | `CONFIDENCE` | | `component.sensitivity` | Detection sensitivity. For `CONFIDENCE`, value range is in `[-25, 14]`. 0 means z-score of 1. 10 means z score of 3. | `5` | | `component.lookbackPeriod` | Historical period to use for training. In ISO-8601 format. Requires `component.monitoringGranularity`, [see shared parameters](./#shared-parameters). Eg: `P14D`. If `component.lookbackPeriod` is not set, `component.lookback` is used. | | | `component.pattern` | Detect as an anomaly if the metric drop, rise or both directions. `UP`, `DOWN`, `UP_OR_DOWN`. | `UP_OR_DOWN` | | `component.regressors` | For advanced users. Additional list of features to add to the regression model. See [available regressors](#regressors). These additional features may help the model to learn the effect of events. Events features are created automatically. | `[]` ie only use event regressors | | `component.forecastingProtocol` | Process to use to run the detection. Used to make the alert evaluation calls and the first run of the alert faster. `EXACT_REPLAY` performs an exact \[one-step]\[one-step-eval] backtracking, training one model for each point. Can be slow. `FAST_REPLAY` trains 1 model and then predict and updates this model incrementally. | `EXACT_REPLAY` | | `component.seasonalityPeriod` | Seasonality biggest period to learn. In ISO-8601 format. Requires `component.monitoringGranularity`, [see shared parameters](./#shared-parameters). Eg: `P7D` will learn weekly and smaller seasonalities. | | | `component.trendMode` | Trend mode as defined in ETS paper\[^1] . In `NONE`, `ADDITIVE`, `DAMPED`. | `ADDITIVE` | | `component.seasonalMode` | Seasonal mode as defined in ETS paper\[^1] . In `NONE`, `ADDITIVE`, `MULTIPLICATIVE`. | `ADDITIVE` | | `component.errorMode` | Error mode as defined in ETS paper\[^1] . In `ADDITIVE`, `MULTIPLICATIVE`. | `ADDITIVE` | | `component.alpha` | Level smoothing factor. `In [0,1]` | `-1` - Optimized by BOBYQA | | `component.beta` | Trend smoothing factor. `In [0,1]` | `-1` - Optimized by BOBYQA | | `component.gamma` | Seasonal smoothing factor. `In [0,1]` | `-1` - Optimized by BOBYQA | | `component.phi` | Damping factor. `In [0,1]`. Only used if `trendMode` is set to `DAMPED`. | `-1` - Optimized by BOBYQA | | `component.alphaMax` | Maximum value of alpha if alpha is optimized automatically. | 0.5 | | `component.alphaMin` | Minimum value of alpha if alpha is optimized automatically. | 0.001 | | `component.betaMax` | Maximum value of beta if beta is optimized automatically. | 0.5 | | `component.betaMin` | Minimum value of beta if beta is optimized automatically. | 0.001 | | `component.gammaMax` | Maximum value of gamma if gamma is optimized automatically. | 0.5 | | `component.gammaMin` | Minimum value of gamma if gamma is optimized automatically. | 0.001 | | `component.phiMax` | Maximum value of phi if phi is optimized automatically. | 0.98 | | `component.phiMin` | Minimum value of phi if phi is optimized automatically. | 0.75 | ### Regressors | name | description | | - | - | | `EPOCH` | Unix epoch in milliseconds | | `MINUTE_OF_HOUR` | Minute of the hour | | `HOUR_OF_DAY` | Hour of the day | | `DAY_OF_WEEK` | Day of the week, between 0 and 6 | | `IS_WEEKEND` | True if the day is Saturday or Sunday | | `IS_MONDAY` | True if the day is Monday | | `IS_TUESDAY` | True if the day is Tuesday | | `IS_WEDNESDAY` | True if the day is Wednesday | | `IS_THURSDAY` | True if the day is Thursday | | `IS_FRIDAY` | True if the day is Friday | | `IS_SATURDAY` | True if the day is Saturday | | `IS_SUNDAY` | True if the day is Sunday | | `DAY_OF_MONTH` | Day number of the month | | `DAY_OF_YEAR` | Day number of the year | | `WEEK_OF_YEAR` | Week number of the year | | `MONTH_OF_YEAR` | Month number of the year | ## Example ### Recommended defaults ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "ETS", "component.sensitivity": "3", "component.seasonalityPeriod": "P7D", "component.lookbackPeriod": "P28D", "component.alpha": "0.3", "component.beta": "0.1", "component.gamma": "0.05", ... # other ets parameters ... # shared parameters }, "inputs": [ { "targetProperty": "current", "sourcePlanNode": "currentData", "sourceProperty": "currentOutput" }, { "targetProperty": "current_events", "sourcePlanNode": "currentEvents", "sourceProperty": "events" } ], "outputs": [] } ``` (^1): See [https://www.researchgate.net/publication/5179823\_Prediction\_Intervals\_for\_Exponential\_Smoothing\_State\_Space\_Models](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models) [iso-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) [one-step-eval](https://otexts.com/fpp2/forecasting-on-training-and-test-sets.html#one-step-forecasts-on-test-data) # REMOTE_HTTP Source: https://docs.startree.ai/thirdeye/operators/anomaly-detector/http > [!NOTE] > > Enterprise only The Remote HTTP detector allows the anomaly detection to be performed by a remote HTTP service. The user can configure the alert to point to a REST endpoint. This endpoint must be able to accept the ThirdEye detection payload and respond back with a specific response API. Upon successful exchange, the response is shared back with downstream operators thereby completing the detection workflow. ## Inputs `"targetProperty": "current"`: The data on which to perform detection. It should contain the historical data to use for training. ## Parameters | name | description | default value | | - | - | - | | `component.url` | URL of the endpoint that has to be called by ThirdEye | | ## Payloads format ### Request payload Here is a sample request payload that is sent to the remote http service. ```json theme={null} { "startMillis": 1553468646555, "endMillis": 1653468646555, "spec": { "help": "This is the anomaly detector spec object which is sent as is to the remote service", ... }, "dataframe": { "seriesMap": { "timestamp": [1, 2, 3], "current": ["v1", "v2", "v3"] } } } ``` ### Response Payload Here's a sample response payload. The endpoint must return a json that follows this schema for ThirdEye to execute successfully. In this case, the expectation is to receive a dataframe in a format defined below with a predefined set of columns. * current: The value of the metric at different timestamps * timestamp: The timestamps associated with the observed values of the metric * value: baseline/predicted values of the metric * lower\_bound: The allowed lower bound of the metric * upper\_bound: The allowed upper bound of the metric * anomaly: boolean if this is or isn't an anomaly. ```json theme={null} { "dataframe": { "seriesMap": { "timestamp": [1, 2, 3], "current": ["v1", "v2", "v3"], "value": ["baseline1", "baseline2", "baseline3"], "lower_bound": ["lower_bound1", "lower_bound2", "lower_bound3"], "upper_bound": ["upper_bound1", "upper_bound2", "upper_bound3"], "anomaly": ["true", "false", "false"] } } } ``` ## Example ### Recommended defaults ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "REMOTE_HTTP", "component.url": "http://localhost:5000/api/http-detector", ... # shared parameters }, "inputs": [ { # data with historical data for training "targetProperty": "current", "sourcePlanNode": "currentData", "sourceProperty": "currentOutput" } ] } ``` ### Full alert ```json theme={null} { "name": "sample-alert-using-remote-http", "description": "Sample description payload for testing", "cron": "0 0 0 1/1 * ? *", "template": { "nodes": [ { "name": "root", "type": "AnomalyDetector", "params": { "type": "REMOTE_HTTP", "component.url": "http://localhost:5000/api/http-detector" }, "inputs": [ { "targetProperty": "current", "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentOutput" } ], "outputs": [] }, { "name": "currentDataFetcher", "type": "DataFetcher", "params": { "component.dataSource": "${dataSource}", "component.query": "SELECT __timeGroup(\"${timeColumn}\", '${timeColumnFormat}', '${monitoringGranularity}') as ts, ${metric} as met FROM ${dataset} WHERE __timeFilter(\"${timeColumn}\", '${timeColumnFormat}') GROUP BY ts ORDER BY ts LIMIT 1000" }, "inputs": [], "outputs": [ { "outputKey": "pinot", "outputName": "currentOutput" } ] } ] }, "templateProperties": { "dataSource": "pinotQuickStartLocal", "dataset": "pageviews", "metric": "sum(views)", "monitoringGranularity": "P1D", "timeColumn": "date", "timeColumnFormat": "yyyyMMdd" } } ``` # MATRIX_PROFILE Source: https://docs.startree.ai/thirdeye/operators/anomaly-detector/matrix_profile > [!NOTE] > > Enterprise only Gives an anomaly score to each point using a left matrix profile algorithm(^1). To make the matrix profile approach better at point anomaly detection, a proprietary transformation function is applied to the matrix profile. This detector returns an anomaly score, it does not return expected values, upper and lower bounds. This detector is recommended for granularity \<= 1 hour. ## Inputs `"targetProperty": "current"`: The data on which to perform detection. It should contain the historical data to use for training. ## Parameters | name | description | default value | | - | - | - | | `component.sensitivity` | Anomaly score threshold. Eg for `0.8`` every point that has as a score bigger that `0.8\` is flagged as an anomaly. | | | `component.lookbackPeriod` | Historical period to use as a reference. In \[ISO-8601]\[iso-8601] format. Requires `component.monitoringGranularity`, [see shared parameters](./#shared-parameters). Eg: `P14D`. If `component.lookbackPeriod` is not set, `component.lookback` is used. | | | `component.seasonalityPeriod` | Seasonality biggest period. Used to infer the window size of the matrix profile. In ISO-8601 format. Requires `component.monitoringGranularity`, [see shared parameters](./#shared-parameters). Eg: `P7D`. | `null` (inferred from the data) | | `component.distance` | For advanced users. Whether to use `NORMALIZED` or `NON_NORMALIZED` euclidean distance for the matrix profile computation. | `NORMALIZED` | | `component.computeBounds` | Experimental. If true, attempts to compute expected values, upper and lower bounds. | `false` | | `component.scoringMethod` | Experimental. Experimental. Post-processing of the matrix profile anomaly scores. `FIRST_ORDER_DIFFERENCE` is good at detecting spikes. `DIRECT` is good at detecting drifts. | `FIRST_ORDER_DIFFERENCE` | ## Example ### Recommended defaults ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "MATRIX_PROFILE", "component.sensitivity": "2", "component.lookbackPeriod": "P21D", ... # shared parameters }, "inputs": [ { "targetProperty": "current", "sourcePlanNode": "currentData", "sourceProperty": "currentOutput" } ], "outputs": [] } ``` (^1): See [https://www.cs.ucr.edu/\~eamonn/MatrixProfile.html](https://www.cs.ucr.edu/~eamonn/MatrixProfile.html) [iso-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) # MEAN_VARIANCE Source: https://docs.startree.ai/thirdeye/operators/anomaly-detector/mean_variance Detects an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set via `lookbackPeriod`. ## Inputs `"targetProperty": "current"`: The data on which to perform detection. It should contain the historical data to use for training. ## Parameters | name | description | default value | | - | - | - | | `component.sensitivity` | Detection sensitivity. 5 means `n=1` sigma. The smaller, the less anomalies are detected. | `5` | | `component.lookbackPeriod` | Historical period to use to estimate mean and std. In ISO-8601 format. Requires `component.monitoringGranularity`, [see shared parameters](./#shared-parameters). Eg: `P14D`. If `component.lookbackPeriod` is not set, `component.lookback` is used. | | | `component.lookback` | Deprecated. Prefer `component.lookbackPeriod`. Number of data points to use to estimate mean and std. | `52` | | `component.seasonalityPeriod` | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), `PT0S` (no seasonality management). Eg: with `P7D`, a Monday 12 AM value will be estimated from mean and variance of the previous Monday 12 AM values. Requires `component.monitoringGranularity`, [see shared parameters](./#shared-parameters). | `PTOS` | | `component.pattern` | Detect as an anomaly if the metric drop, rise or both directions. `UP`, `DOWN`, `UP_OR_DOWN`. | `UP_OR_DOWN` | ## Example ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "MEAN_VARIANCE", "component.monitoringGranularity": "P1D", "component.lookbackPeriod": "P14D", "component.sensitivity": "5", "component.pattern": "UP", ... # shared parameters }, "inputs": [ { # data with historical data for mean/std estimation "targetProperty": "current", "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentOutput" } ] } ``` # OFFSET_AGGREGATION Source: https://docs.startree.ai/thirdeye/operators/anomaly-detector/offset_aggregation > [!NOTE] > > Enterprise only Compares current value to the value predicted by a naive model. The naive model is an average/median/percentile of historical values. The upper/lower bounds interval can be computed with an absolute or a percentage rule. ## Inputs `"targetProperty": "current"`: The data on which to perform detection. ## Parameters | name | description | default value | | - | - | - | | `component.intervalsMethod` | Method to compute intervals. In `PERCENTAGE`, `ABSOLUTE`. | | | `component.sensitivity` | Detection sensitivity. Eg with `PERCENTAGE`, set 50 for a 50% percentage change threshold. | | | `component.lookbackPeriod` | Historical period to use for training. In ISO-8601 format. Requires `component.monitoringGranularity`, [see shared parameters](./#shared-parameters). Eg: `P14D`. | | | `component.offsets` | A list of offsets in ISO-8601 format to use as baseline. Eg \[`P7D`, `P14D`] will compare the current value to the aggregation of the values of the 2 previous weeks. | `[P7D]` | | `component.aggregation` | The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | `AVERAGE` | `offsets` and `lookbackPeriod` must be compatible. Eg if `offsets` is \[`P7D`, `P14D`] then `lookbackPeriod` should be at least `P14D`. ## Example ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "OFFSET_AGGREGATION", "component.intervalsMethod": "ABSOLUTE", "component.sensitivity": "200", # detect if abs(current-predicted)>200 "component.lookbackPeriod": "P28D", "component.offsets": ["P7D", "P14D", "P21D", "P28D"], "component.aggregation": "MEDIAN" }, "inputs": [ { # current data "targetProperty": "current", "sourcePlanNode": "currentData", "sourceProperty": "currentOutput" } ] } ``` [iso-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) # PERCENTAGE_CHANGE Source: https://docs.startree.ai/thirdeye/operators/anomaly-detector/percentage Compares the current timeseries to a baseline. If the current value is not in `[baseline - percentage * baseline, baseline + percentage * baseline]`, detects an anomaly. ## Inputs `"targetProperty": "current"`: The data on which to perform detection. `"targetProperty": "baseline"`: The data to use as a baseline. The inputs should have the same size. ## Parameters | name | description | default value | | - | - | - | | `component.percentageChange` | The percentage threshold. If the percentage change is above this threshold, detect an anomaly. In ratio. Eg 0.1 means 10 percent. | `Inf` | | `component.pattern` | Detect as an anomaly if the metric drop, rise or both directions. `UP`, `DOWN`, `UP_OR_DOWN`. | `UP_OR_DOWN` | ## Example ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "PERCENTAGE_CHANGE", "component.percentageChange": "0.2", # 20% threshold "component.pattern": "UP", # only detect change up ... # shared parameters }, "inputs": [ { # baseline data "targetProperty": "baseline", "sourcePlanNode": "baselineDataFetcher", "sourceProperty": "baselineOutput" }, { # current data "targetProperty": "current", "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentOutput" } ] } ``` # THRESHOLD Source: https://docs.startree.ai/thirdeye/operators/anomaly-detector/threshold Detects an anomaly if a metric is above a maximum threshold or below a minimum threshold. ## Inputs `"targetProperty": "current"`: The data on which to perform detection. ## Parameters | name | description | default value | | - | - | - | | `component.min` | If the metric goes below this value, detect an anomaly. | `-Inf` | | `component.max` | If the metric goes above this value, detect an anomaly. | `+Inf` | ## Example ```json theme={null} { "name": "root", "type": "AnomalyDetector", "params": { "type": "THRESHOLD", "component.max": "75", "component.min": "10", ... # shared parameters }, "inputs": [ { # current data "targetProperty": "current", "sourcePlanNode": "currentData", "sourceProperty": "currentOutput" } ] } ``` # DataFetcher Source: https://docs.startree.ai/thirdeye/operators/data-fetcher Fetches data from a datasource. ## Inputs None. ## Outputs One output. No naming constraint for `outputKey` and `outputName`. ## Parameters | name | description | default value | | - | - | - | | `component.dataSource` | ThirdEye datasource to use. | | | `component.query` | Query to execute. The query can use [macros](#macros). | | ## Example ```json theme={null} { "name": "currentDataFetcher", "type": "DataFetcher", "params": { "component.dataSource": "pinot", # ThirdEye datasource "component.query": "SELECT __timeGroup(timeColumn, 'EPOCH', 'P1D') as ts, sum(views) as met FROM pageviews WHERE __timeFilter(timeColumn, 'EPOCH') GROUP BY ts ORDER BY ts LIMIT 10000" }, "inputs": [], "outputs": [ { "outputKey": "currentData", # no constraint "outputName": "currentOutput" # no constraint } ] } ``` ## Macros Macros are special functions that are translated by ThirdEye to SQL before the query is sent to the database. Macros have access to detection runtime information, like the start and the end time of the detection, the timezone. | Macro example | description | | - | - | | `__timeFilter(` `timeColumn,` `'timeFormat'` `)` | Replaced by a time range filter on the timeframe `[detectionStart, detectionEnd[` using the specified column name and the specified timeFormat. For example, `__timeFilter(timeColumn, 'EPOCH_MILLIS')` is replaced by `timeColumn >= 1494410242000 AND timeColumn < 1494410242000)` | | `__timeFilter(` `timeColumn,` `'timeFormat,'` `'lookbackFromStartPeriod'` `)` | Replaced by a time range filter on the timeframe `[detectionStart-lookbackFromStartPeriod, detectionEnd[`. For example, for a detection between January 14 and January 15, `__timeFilter(timeColumn, 'yyyyMMdd', 'P7D')` is replaced by `timeColumn >= '20220107' AND timeColumn < '20220115')` | | `__timeFilter(` `timeColumn,` `'timeFormat,'` `'lookbackFromStartPeriod,'` `'lookbackFromEndPeriod'` `)` | Replaced by a time range filter on the timeframe `[detectionStart-lookbackFromStartPeriod, detectionEnd-lookbackFromEndPeriod[`. For example, for a detection between January 14 and January 15, `__timeFilter(timeColumn, 'yyyyMMdd', 'P7D', 'P7D')` is replaced by `timeColumn >= '20220107' AND timeColumn < '20220108')` | | | | | `__timeGroup(` `timeColumn,` `'timeFormat',` `'granularity'` `)` | Replaced by a transformation of the timeColumn in bins of the given granularity, in milliseconds. Respects the [timezone](/thirdeye/alert-configuration#timezone) of the alert. For example, `__timeGroup(timeColumn, 'yyyyMMddhh', 'P1D')` is replaced by ` DATETIMECONVERT(timeColumn, '1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMddhh', '1:MILLISECONDS:EPOCH', '1:DAYS')` | ### AUTO keyword If the `timeColumn` value is `AUTO`, the macro resolves to the primary time column of the table.\ `timeFormat` is obtained internally and can be left blank. For example: * `__timeFilter(AUTO,'')` * `__timeGroup(AUTO,'','P1D')` ### timeFormat strings Available timeformat strings are: | String | description | aliases | | - | - | - | | `EPOCH_MILLIS` | time column is in milliseconds since [unix epoch](https://en.wikipedia.org/wiki/Unix_time) | `1:MILLISECONDS:EPOCH` | | `EPOCH` | time column is in seconds since unix epoch | `1:SECONDS:EPOCH` | | `EPOCH_MINUTES` | time column is in minutes since unix epoch | `1:MINUTES:EPOCH` | | `EPOCH_HOURS` | time column is in hours since unix epoch | `1:HOURS:EPOCH` | | `EPOCH_DAYS` | time column is in days since unix epoch | `1:DAYS:EPOCH` | | any SimpleDateFormat string | See [SimpleDateFormat documentation](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html). Eg `yyyyMMdd`, `yyyy-MM-dd'T'HH:mm:ss.SSSZ` | `1:DAYS:SIMPLE_DATE_FORMAT:[SIMPLE_DATE_FORMAT]` | ### period strings Period strings uses the [ISO-8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations). Examples: * `'P7D'`: 7 days * `'PT1H'`: 1 hour # Dimension Drift Operators Source: https://docs.startree.ai/thirdeye/operators/dimension-drift The ThirdEye drift operators are used to measure population drift across each value of a dimension. Thirdeye has two drift operators: 1. [Static Drift Operator](/thirdeye/operators/dimension-drift/static-drift-operator) 2. [Lookback Drift Operator](/thirdeye/operators/dimension-drift/lookback-drift-operator) The output of the dimension drift operators can be passed into any anomaly detector. #### Sample alert configuration [How it works](https://drive.google.com/file/d/1lyJWop0K6UHkl39JRbIXgWJgYMiUVDRD/view?usp=sharing) (video) ``` { "name": "USStoreSales-drift-lookback-dx", "description": "", "template": { "name": "startree-drift-lookback-threshold-dx" }, "templateProperties": { "dataSource": "mypinot", "dataset": "USStoreSalesOrderData", "aggregationColumn": "*", "aggregationFunction": "COUNT", "seasonalityPeriod": "P7D", "lookback": "P28D", "monitoringGranularity": "P1D", "sensitivity": "3", "bucketSize": "${bucketSize}", "bucketCount": "${bucketCount}", "bucketStart": "${bucketStart}", "encoder": "${encoder}", "max": "${max}", "min": "0", "enumerationItems": [ { "name": "DiscountApplied", "params": { "dimension": "DiscountApplied", "encoder": "LINEAR_BUCKETS", "bucketSize": 0.05, "bucketCount": 10, "bucketStart": 0, "max": 0.5 } }, { "name": "UnitPrice", "params": { "dimension": "UnitPrice", "encoder": "LINEAR_BUCKETS", "bucketSize": 50, "bucketCount": 120, "bucketStart": 0, "max": 0.5 } }, { "name": "UnitCost", "params": { "dimension": "UnitCost", "encoder": "LINEAR_BUCKETS", "bucketSize": 50, "bucketCount": 120, "bucketStart": 0, "max": 0.5 } }, { "name": "OrderQuantity", "params": { "dimension": "OrderQuantity", "encoder": "LINEAR_BUCKETS", "bucketSize": 1, "bucketCount": 10, "bucketStart": 0, "max": 0.7 } }, { "name": "SalesChannel", "params": { "dimension": "SalesChannel", "encoder": "ENUMERATE", "bucketSize": 0, "bucketCount": 0, "bucketStart": 0, "max": 0.5 } }, { "name": "StoreLocation", "params": { "dimension": "StoreLocation", "encoder": "ENUMERATE", "bucketSize": 0, "bucketCount": 0, "bucketStart": 0, "max": 0.9 } }, { "name": "SalesRegion", "params": { "dimension": "SalesRegion", "encoder": "ENUMERATE", "bucketSize": 0, "bucketCount": 0, "bucketStart": 0, "max": 0.5 } }, { "name": "WarehouseCode", "params": { "dimension": "WarehouseCode", "encoder": "ENUMERATE", "bucketSize": 0, "bucketCount": 0, "bucketStart": 0, "max": 0.5 } } ], "dimension": "${dimension}", "dayHoursOfWeek": {} }, "cron": "0 0 5 ? * MON-FRI *", } ``` ### How does this work? #### What it does? * Detects Changes in Data: This operator is like a microscope for your data. It looks for shifts or unusual patterns in specific parts of your data (dimensions). For example, imagine you're tracking website traffic by country – this operator could alert you if there's a sudden spike in visitors from a new location. * Compares to the Past: The "lookback" part means it compares your current data to a historical snapshot. This helps it figure out what's normal and what's an unexpected change. * Gives You a "Drift Score": The operator calculates how different your current data is from the past. A higher score means a potentially significant change. #### How it Works (simplified)? * Choose Your Focus: You tell the operator which dimension to watch (e.g., country of website visitors, product category, customer age group). * Lookback Window: You set how far back in time it should compare (e.g., last week, last month). * Calculates Differences: It compares the distribution of values in your current data against the historical data. Is the mix of countries, product types, or ages roughly the same? * Assigns "Drift Score": Based on the calculated difference, it gives you a score to indicate the level of change or "drift". #### Why This Matters? * Spotting problems early: If a critical part of your data suddenly changes, it could signal a problem (website outage in a certain country, a product becoming unexpectedly popular) or a new opportunity. * Keeping track of trends: It helps you understand if your data is gradually shifting over time, even if the changes aren't dramatic at any one moment. # Lookback Drift Operator Source: https://docs.startree.ai/thirdeye/operators/dimension-drift/lookback-drift-operator Given dimension and population metrics, computes a timeseries of drift metrics that measures the dimensional drift of the current data compared to a lookback baseline. This operator requires a single input timeseries and derives the baseline dataset for each timestamp according to the lookback and seasonality parameters. For example, the baseline dataset for timestamp `2023-01-29` with a 28 day lookback period and 7 day seasonality will contain the all the data points at the timestamps `2023-01-01`, `2023-01-08`, `2023-01-15`, and `2023-01-22`. The dimension values can be encoded in two ways: enumeration or linear buckets. * The enumeration encoder should be used for string values or low-cardinality numeric values. New values encountered in the current data are always mapped the same integer, which corresponds to the *other* category. * The linear buckets encoder should be used for numerical values that have high cardinality. The linear buckets encoder maps each dimension value to a bucket index. All values less than the smallest bucket are mapped to the smallest bucket. All values larger than the largest bucket are mapped to the largest bucket. At each timestamp, the operator computes a normalized EMD of the distribution of dimension values of the current data and compared to the baseline data. The EMD distance function depends on the encoding: * For enumeration, the distance used is always 1 / number of unique values. * For bucketing, the distance used is the normalized distance between the buckets. ## Inputs | targetProperty | description | | - | - | | currentData | A data frame with columns `ts`, `dim`, and `met`. Missing data is ok, and the time index filler is not recommended. Column `ts` is the timestamp. Column `dim` is the dimension to used to measure drift. Column `met` is the metric or population of the dimension value at the timestamp. | ## Outputs | outputName | description | | - | - | | driftScoreData | The resulting data frame of drift metrics. The data frame has columns `ts` and `met`. Column `ts` is the timestamp. Column `met` is the drift score computed at the timestamp. | ## Parameters ### Operator parameters | name | description | | - | - | | monitoringGranularity | The time granularity of the output timeseries. | | seasonalityPeriod | The step-size used when building the baseline dataset | | lookback | The period of time to consider points for the baseline dataset. | | encoder | The encoder parameters json object. | ### Encoder parameters | name | description | | - | - | | type | The type of encoder. One of `ENUMERATION` or `LINEAR_BUCKETS`. | | linearBucket | The linear bucket parameters json object. | ### Linear bucket parameters | name | description | | - | - | | bucketStart | The value corresponding to the first bucket. | | bucketSize | The size of each bucket. | | bucketCount | The number of buckets to use. | ## Example ```json theme={null} { "name": "driftScore", "type": "LookbackBaselineDrift", "params": { "lookback": "P28D", "monitoringGranularity": "P1D", "seasonalityPeriod": "P7D", "encoder": { "type": "LINEAR_BUCKETS", "linearBucket": { "bucketStart": "0", "bucketSize": "1", "bucketCount": "100" } } }, "inputs": [ { "targetProperty": "currentData", "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentData" } ], "outputs": [ { "outputName": "driftScoreData" } ] } ``` # Static Drift Operator Source: https://docs.startree.ai/thirdeye/operators/dimension-drift/static-drift-operator Given dimension and population metrics, compute a timeseries of drift metrics that measures the dimensional drift of the current data compared to a static baseline. This operator requires two inputs: a timeseries dataset of new data and baseline dataset without a time component. The dimension values can be encoded in two ways: enumeration or linear buckets. * The enumeration encoder should be used for string values or low-cardinality numeric values. New values encountered in the current data are always mapped the same integer, which corresponds to the *other* category. * The linear buckets encoder should be used for numerical values that have high cardinality. The linear buckets encoder maps each dimension value to a bucket index. All values less than the smallest bucket are mapped to the smallest bucket. All values larger than the largest bucket are mapped to the largest bucket. At each timestamp, the operator computes a normalized EMD of the distribution of dimension values of the current data compared to the baseline data. The EMD distance function depends on the encoding: * For enumeration, the distance used is always 1 / number of unique values. * For bucketing, the distance used is the normalized distance between the buckets. ## Inputs | targetProperty | description | | - | - | | currentData | A data frame with columns `ts`, `dim`, and `met`. Missing data is ok, and the time index filler is not recommended. Column `ts` is the timestamp. Column `dim` is the dimension to used to measure drift. Column `met` is the metric or population of the dimension value at the timestamp. | | baselineData | A data frame with columns `dim` and `met`. The baseline data should not have a time component. | ## Outputs | outputName | description | | - | - | | driftScoreData | The resulting data frame of drift metrics. The data frame has columns `ts` and `met`. Column `ts` is the timestamp. Column `met` is the drift score computed at the timestamp. | ## Parameters ### Operator parameters | name | description | | - | - | | monitoringGranularity | The time granularity of the output timeseries. | | encoder | The encoder parameters json object. | ### Encoder parameters | name | description | | - | - | | type | The type of encoder. One of `ENUMERATION` or `LINEAR_BUCKETS`. | | linearBucket | The linear bucket parameters json object. | ### Linear bucket parameters | name | description | | - | - | | bucketStart | The value corresponding to the first bucket. | | bucketSize | The size of each bucket. | | bucketCount | The number of buckets to use. | ## Example ```json theme={null} { "name": "driftScore", "type": "StaticBaselineDrift", "params": { "encoder": { "type": "LINEAR_BUCKETS", "linearBucket": { "bucketStart": "0", "bucketSize": "1", "bucketCount": "100" } }, "monitoringGranularity": "P1D" }, "inputs": [ { "targetProperty": "currentData", "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentData" }, { "targetProperty": "baselineData", "sourcePlanNode": "baselineDataFetcher", "sourceProperty": "baselineData" } ], "outputs": [ { "outputName": "driftScoreData" } ] } ``` # Dimension Exploration Source: https://docs.startree.ai/thirdeye/operators/dimension-exploration Dimension exploration lets you create alerts for a specified dimension or combination of dimensions. See [the procedural doc](/thirdeye/dimension-exploration) if some of these terms don't look familiar. Dimension exploration uses the **enumerator**, **fork join** and **combiner** operators to build loops inside the detection DAG. Here's a simple pipeline for threshold detection. ```json theme={null} "nodes": [ { "name": "enumerator", "type": "Enumerator", "params": { "items": [ { "name":"US Only", "params": { "queryFilters": " AND country = 'US'", "max": 300000, "min": 100000 } }, { "name": "Overall", "params": { "queryFilters": "", "max": 900000, "min": 300000 } } ] } }, { "name": "combiner", "type": "Combiner" }, { "name": "root", "type": "ForkJoin", "params": { "dryRun": false, "enumerator": "enumerator", "root": "anomalyDetector", "combiner": "combiner" } }, { "name": "anomalyDetector", "type": "AnomalyDetector", "params": { "component.min": "${min}", "component.metric": "met", "component.monitoringGranularity": "${monitoringGranularity}", "component.max": "${max}", "anomaly.dataset": "${dataset}", "component.timestamp": "ts", "anomaly.metric": "${aggregationColumn}", "anomaly.source": "threshold-template/root", "type": "THRESHOLD" }, "inputs": [ { "targetProperty": "current", "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentData" } ], "outputs": [] }, { "name": "currentDataFetcher", "type": "DataFetcher", "params": { "component.dataSource": "${dataSource}", "component.query": "SELECT __timeGroup(\"${timeColumn}\", '${timeColumnFormat}', '${monitoringGranularity}') as ts, ${aggregationFunction}(\"${aggregationColumn}\") as met FROM ${dataset} WHERE __timeFilter(\"${timeColumn}\", '${timeColumnFormat}') ${queryFilters} GROUP BY ts ORDER BY ts LIMIT ${queryLimit}" }, "inputs": [], "outputs": [ { "outputKey": "currentData", "outputName": "currentData" } ] } ] ``` ## Enumeration item templatization This is a simple default enumerator which lays out all the enumeration items. The same json can be templatized by using thirdeye's ability to replace objects in strings. In this case, the enumerator node lists all the enumerations and can be split into the template and the value. The template can expose the items array using a variable `"${enumerationItems}"`. ```json theme={null} { "name": "enumerator", "type": "Enumerator", "params": { "items": "${enumerationItems}" } } ``` ThirdEye can detect that `enumerationItems` is an object and replaces the entire string with an object and behaves exactly like the initial pipeline json as shown above. ```json theme={null} { "templateProperties": { "enumerationItems": [ { "name":"US Only", "params": { "queryFilters": " AND country = 'US'", "max": 300000, "min": 100000 } }, { "name": "Overall", "params": { "queryFilters": "", "max": 900000, "min": 300000 } } ] } } ``` ## Sample Alert Using these methods, dimension exploration can be templatized and expressed with templateProperties. Here is a sample dimension exploration alert look at pageviews with different combinations of threshold parameters and query filters. ```json theme={null} { "name": "pageviews-dx-country", "description": "Sample description payload for testing", "template": { "name": "startree-threshold-dx" }, "templateProperties": { "dataSource": "pinot", "dataset": "pageviews", "aggregationFunction": "sum", "aggregationColumn": "views", "monitoringGranularity": "P1D", "timeColumn": "date", "timeColumnFormat": "1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd", "max": "${max}", "min": "${min}", "queryFilters": "${queryFilters}", "enumerationItems": [ { "name": "US", "params": { "queryFilters": " AND country = 'US'", "max": 300000, "min": 100000 } }, { "name": "Overall", "params": { "queryFilters": "", "max": 900000, "min": 300000 } }, { "name": "IN", "params": { "queryFilters": " AND country = 'IN'", "max": 900000, "min": 0 } } ] }, "cron": "0 0 * * * ? *" } ``` # EventFetcher Source: https://docs.startree.ai/thirdeye/operators/event-fetcher Fetch events data from ThirdEye internal events database. These events can be fed to detectors downstream. ## Inputs None. ## Outputs One output. No naming constraint for `outputKey` and `outputName`. ## Parameters | name | description | default value | | - | - | - | | `component.startTimeLookback` | Offset to apply to the startTime of the event timeframe. In ISO-8601 format. Eg `P7D`. | No offset ie `eventStartTime = detectionStartTime` | | `component.endTimeLookback` | Offset to apply to the endTime of the event timeframe. In ISO-8601 format. Eg `P7D`. | No offset ie `eventEndTime = detectionEndTime` | | `component.lookaround` | Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Eg `P1D`. | `P1D` ie events are fetched on \[minTime - 1 day, maxTime + 1 day] | | `component.eventTypes` | List of event types to filter by. Eg `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. | `[]` ie no filtering | | `component.sqlFilter` | Sql filter to apply on the events. See [documentation below](#sql-filter). Eg `type='HOLIDAY' and 'US' member of dimensionMap['countryCode']`. `""` empty string or `null` means no sql filtering. | `""` ie no sql filtering | ### SQL Filter SQL filters are applied to a table with the following schema: | name: String | type: String | dimensionMap: Map\[String, List(String)] | | - | - | - | | `Christmas` | `HOLIDAY` | `{"countryCode": ["US", "CA"]}` | | `Christmas` | `HOLIDAY` | `{"countryCode": ["NL", "FR"]}` | | `Product X deployment` | `CUSTOM` | `{"env": ["prod", "dev"], "device": ["android"]}` | | `Product Y deployment` | `CUSTOM` | `{"env": ["prod", "dev"], "device": ["android", "ios"]}` | | `Product X deployment` | `CUSTOM` | `{"env": ["prod"], "language": ["android"]}` | The same event can happen in different timezone, so on a slightly different timeframe.\ For instance, Christmas in France happens around 7 hours before Christmas in the United-States, so it is good practice to filter the `HOLIDAY` events by a limited set of countries. Example: * Take all `HOLIDAY` events happening in the `US` and all `CUSTOM` events created by the business team:\ `'US' member of dimensionMap['countryCode'] OR (type = 'CUSTOM')` The dialect used is Apache Calcite SQL. It is very close to ANSI SQL. [See SQL reference](https://calcite.apache.org/docs/reference.html). Try Calcite SQL [online here](https://calcite-online.system.gavinray.dev). > [!WARNING] > > **Combining filtering** > > The `eventTypes` filter is applied before the `sqlFilter`. ## Example ```json theme={null} { "name": "eventsDataFetcher", "type": "EventFetcher", "params": { "component.startTimeLookback": "P30D", "component.endTimeLookback": "P0D", "component.lookaround": "P1D", "component.eventTypes": ["HOLIDAY", "CUSTOM"], "component.sqlFilter": "'US' member of dimensionMap['countryCode'] OR (type = 'CUSTOM')" }, "outputs": [ { # no constraint "outputKey": "events", # no constraint "outputName": "events" } ] }, ``` [iso-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) [calcite-sql](https://calcite.apache.org/docs/reference.html) # PostProcessor Source: https://docs.startree.ai/thirdeye/operators/post-processor `PostProcessor` operators apply labels on anomalies returned by `AnomalyDetector` operators. Labels can be filters, priorities or any metadata that can help when investigating an anomaly. Filtered anomalies are not notified, and by default they are hidden in the anomalies list and charts. `PostProcessor` can use side inputs and be chained. There are many types of post-processors detectors. The `type` is given in the node parameters. ```json theme={null} { "name": "root", # name of the operator "type": "PostProcessor", # type of operator "params": { "type": "COLD_START", # type of post-processor # other parameters ... "component.ignore": "true", ... }, "inputs" : [ { "sourcePlanNode": "anomalyDetector" }, ... # other optional inputs ], "outputs" : [...] } ``` ## Inputs Anomalies inputs are passed with the `sourcePlanNode` name only: ```json theme={null} "inputs" : [ { "sourcePlanNode": "anomalyDetector" } ] ``` The post-processor automatically finds the detection result and the data that was used. To chain post-processor operators, apply the next post-processor to the previous post-processor. See [example here](https://github.com/startreedata/thirdeye/blob/master/thirdeye-plugins/thirdeye-bootstrap-open-core/src/main/resources/alert-templates/startree-threshold.json). Other side-inputs should be passed with the usual standard: ```json theme={null} "inputs" : [ { "targetProperty": "current", "sourcePlanNode": "controlSideInputData", "sourceProperty": "controlSideData" } ] ``` Refer to the sub-sections for side inputs. ## Shared parameters Post-processors share a set of parameters. Type specific parameters are described in the sub-sections. | name | description | default value | | - | - | - | | `type` | The type of post-processor. | | | `component.ignore` | If true, anomalies that match the rule will be filtered. The default behaviour depends on the post-processor. | | # ANOMALY_MERGER Source: https://docs.startree.ai/thirdeye/operators/post-processor/anomaly_merger Merges anomalies. See [anomaly merge](/thirdeye/concepts-data-alerts-notifications/concepts-anomaly-merger-rules). ## Inputs The info source is the output of the detector. Anomalies of this output are merged. ```json theme={null} { "sourcePlanNode": "anomalyDetector" } ``` ## Parameters | name | description | default value | | - | - | - | | `component.mergeMaxGap` | Maximum gap between 2 anomalies for the anomalies to be merged. In ISO-8601 format. To disable anomalies merging, set this value to `P0D`. | PT2H | | `component.mergeMaxDuration` | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. | P7D | | `component.reNotifyPercentageThreshold` | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | `component.reNotifyAbsoluteThreshold` | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | ## Example ```json theme={null} { "name": "root", "type": "PostProcessor", "params": { "type": "ANOMALY_MERGER", "component.mergeMaxGap": "PT1H", "component.mergeMaxDuration": "P3D" }, "inputs": [ { "sourcePlanNode": "anomalyDetector" } ] } ``` [iso-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) # COLD_START Source: https://docs.startree.ai/thirdeye/operators/post-processor/cold_start Adds a `ColdStart` label to anomalies that occur at the beginning of the dataset. Used to filter anomalies until the detection model has enough historical data. The start of the dataset is obtained by looking at the minimum time value in the Pinot main time column. The cold start period is defined as `[start, start + coldStartPeriod]`. `coldStartPeriod`is configurable. ## Inputs The info source is the output of the detector. Labels are applied to the anomalies of this input. ```json theme={null} { "sourcePlanNode": "anomalyDetector" } ``` ## Parameters | name | description | default value | | - | - | - | | `component.coldStartPeriod` | Cold start period. In ISO-8601 format. Eg `P7D`. If the anomaly is in \[start, start + Period], label the anomaly. | | | `component.tableName` | Table to look at to find the start. Eg `my_dataset`. | | ## Example ```json theme={null} { "name": "root", "type": "PostProcessor", "params": { "type": "COLD_START", "component.ignore": "true", # filter anomaly in notifications an ui bydefault "component.coldStartPeriod": "P14D", # filter anomalies in the first 14 days of the dataset "component.tableName": "my_dataset" }, "inputs": [ { "sourcePlanNode": "anomalyDetector" } ] } ``` [iso-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) # DETECTOR Source: https://docs.startree.ai/thirdeye/operators/post-processor/detector > [!NOTE] > > Enterprise only Some detectors can be used as postProcessors. A label is added to main detector anomalies if they are not flagged as an anomaly by the postProcessor detector. By default, the postProcessor detector is applied to the metric that was used for the detection. This corresponds to an `AND` operation of 2 detectors. A different metric can also be provided by passing an input `current`. The detector will run on this input. ## Compatible detectors * [OFFSET\_AGGREGATION](/thirdeye/operators/anomaly-detector/offset_aggregation) ## Inputs The info source is the output of the detector. Labels are applied to the anomalies of this input. ```json theme={null} { "sourcePlanNode": "anomalyDetector" } ``` *Optional*: A different side data input to run detection on: ```json theme={null} { "targetProperty": "current", "sourcePlanNode": "controlDataFetcher", "sourceProperty": "controlDataOutput" } ``` ## Parameters All parameters of the anomaly detector can be used. Additional parameters: | name | description | default value | | - | - | - | | `component.ignore` | If true, anomalies that are not detected by the postProcessor detector will be filtered. | `false` | | `component.valueName` | Optional. The name of the metric. Used in the label: `"{ValueName} not anomalous according to detector"` | `Value` | ## Examples Without side input ```json theme={null} { "name": "root", "type": "PostProcessor", "params": { "type": "OFFSET_AGGREGATION", "component.ignore": "true", # filter anomaly in notifications an ui by default, "component.valueName": "Business KPI", # customize label "component.offsets": ["P7D", "P14D"], # parameter of the detector ... "component.aggregation": "MEDIAN" # parameter of the detector }, "inputs": [ { "sourcePlanNode": "anomalyDetector" } ] } ``` With side input ```json theme={null} { "name": "root", "type": "PostProcessor", "params": { "type": "OFFSET_AGGREGATION", "component.ignore": "true", "component.valueName": "GuardRail KPI", # customize label, "component.timestamp": "ts", # time column of the side input "component.metric": "sideMetric" # column in the side input to run detector on "component.offsets": ["P7D", "P14D"], # parameter of the detector ... "component.aggregation": "MEDIAN" # parameter of the detector }, "inputs": [ { "sourcePlanNode": "anomalyDetector" }, { "targetProperty": "current", "sourcePlanNode": "controlDataFetcher", "sourceProperty": "controlDataOutput" } ] } ``` # EVENTS Source: https://docs.startree.ai/thirdeye/operators/post-processor/events Adds an `EVENT` label to anomalies that happen during an event. Events are provided by an [eventFetcher](/thirdeye/operators/event-fetcher). Used to filter anomalies that happen on holidays. ## Inputs The info source is the output of the detector. Labels are applied to the anomalies of this input. ```json theme={null} { "sourcePlanNode": "anomalyDetector" } ``` The output of an [eventFetcher](/thirdeye/operators/event-fetcher). It contains the events to use for labelling. ```json theme={null} { "targetProperty": "current_events", "sourcePlanNode": "eventsDataFetcher", "sourceProperty": "events" } ``` ## Parameters | name | description | default value | | - | - | - | | `component.beforeHolidayMargin` | A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Eg if beforeEventMargin is P1D, if event happens on \[Dec 24 0:00, Dec 25 0:00\[, the label will be applied to anomalies happening on \[Dec 23 0:00 and Dec 25 0:00\[ | `P0D` (no margin) | | `component.afterHolidayMargin` | Same as `beforeHolidayMargin` at the end of the event. | `P0D` (no margin) | ## Example ```json theme={null} { "name": "root", "type": "PostProcessor", "params": { "type": "EVENT", "component.ignore": "true", "component.beforeHolidayMargin": "PT4H", "component.afterHolidayMargin": "PT4H" }, "inputs": [ { "sourcePlanNode": "anomalyDetector" }, { "targetProperty": "current_events", "sourcePlanNode": "eventsDataFetcher", "sourceProperty": "events" } ] } ``` [iso-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) # IMPACT Source: https://docs.startree.ai/thirdeye/operators/post-processor/impact > [!NOTE] > > Enterprise only > [!WARNING] > > Experimental Adds an `IMPACT` label to an anomaly if the sum of the differences between predictions and observed values on a window containing the anomaly is less than a given threshold. Intuitively, it is the sum of impacts in the last n points. ## Inputs The info source is the output of the detector. Labels are applied to the anomalies of this input. ```json theme={null} { "sourcePlanNode": "anomalyDetector" } ``` ## Parameters | name | description | default value | | - | - | - | | `component.threshold` | If the sum of anomalies impact is below this threshold, label the anomaly. | `-1` (special value - no threshold) | | `component.window` | Window to compute the impact. In ISO-8601 format. | | | `component.impactComputationMethod` | Method to compute the impact. `STANDARD_AREA` or `ABSOLUTE_AREA`. See below. | `STANDARD_AREA` | | `component.monitoringGranularity` | Granularity of the timeseries in ISO-8601 format. Should be same value as in the detector configuration. | | ### `impactComputationMethod` * `STANDARD_AREA`\ `impact = sum(actual-expected) on the window` * `ABSOLUTE_AREA`\ `impact = sum(abs(actual-expected)) on the window` ## Example ```json theme={null} { "name": "root", "type": "PostProcessor", "params": { "type": "IMPACT", "component.ignore": "true", "component.monitoringGranularity": "PT1H", "component.impactComputationMethod": "STANDARD_AREA", "component.window": "PT12H", # anomaly can be ignored if the sum of the impacts in the last 12 hours is less than 500 "component.threshold": "500" }, "inputs": [ { "sourcePlanNode": "anomalyDetector" } ] } ``` [iso-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) # THRESHOLD Source: https://docs.startree.ai/thirdeye/operators/post-processor/threshold Adds a `THRESHOLD` label to an anomaly if the metric is above a maximum threshold or below a minimum threshold. By default, the threshold is applied to the metric that was used for the detection. A different metric can be provided by passing an input `current`. The threshold rule will run on this metric. ## Inputs The info source is the output of the detector. Labels are applied to the anomalies of this input. ```json theme={null} { "sourcePlanNode": "anomalyDetector" } ``` *Optional*: a different side data input to perform threshold on: ```json theme={null} { "targetProperty": "current", "sourcePlanNode": "controlDataFetcher", "sourceProperty": "controlDataOutput" } ``` ## Parameters | name | description | default value | | - | - | - | | `component.min` | If `metric <= min`, label the anomaly. | `-1` (special value - no minimum) | | `component.max` | If `metric >= max`, label the anomaly. | `-1` (special value - no maximum) | | `component.valueName` | Optional. The name of the metric. Used in the label: `"{ValueName} bigger than {max}"` | `Value` | | `component.timestamp` | Optional. If a side data input is used, name of the time index column. | `timestamp` | | `component.metric` | Optional. If a side data input is used, name of the column to use for thresholding. | `value` | ## Example Without side input: ```json theme={null} { "name": "root", "type": "PostProcessor", "params": { "type": "THRESHOLD", "component.ignore": "true", # filter anomaly in notifications an ui by default "component.min": "100", "component.max": "10000000" }, "inputs": [ { "sourcePlanNode": "anomalyDetector" } ] } ``` With side input: ```json theme={null} { "name": "root", "type": "PostProcessor", "params": { "type": "THRESHOLD", "component.ignore": "true", "component.min": "100", "component.max": "10000000", "component.valueName": "controlMetric", # customize label "component.timestamp": "ts", # time column of the side input "component.metric": "sideMetric" # column to threshold on in the side input }, "inputs": [ { "sourcePlanNode": "anomalyDetector" }, { "targetProperty": "current", "sourcePlanNode": "controlDataFetcher", "sourceProperty": "controlDataOutput" } ] } ``` # TIME_OF_WEEK Source: https://docs.startree.ai/thirdeye/operators/post-processor/time_of_week Adds a `TIME_OF_WEEK` label to anomalies that happen during a specific time of the week. For instance, can be used to filter anomalies that happen on Saturday. ## Inputs The info source is the output of the detector. Labels are applied to the anomalies of this input. ```json theme={null} { "sourcePlanNode": "anomalyDetector" } ``` ## Parameters | name | description | default value | | - | - | - | | `component.daysOfWeek` | A list of days to match. Eg `["MONDAY", "SATURDAY"]`. | `[]` (no match) | | `component.hoursOfDay` | A list of hours to match. Eg `[0,1,2,23]` | `[]` (no match) | | `component.dayHoursOfWeek` | A map of `>` to match. Eg `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | `{}` (no match) | To determine days, the timezone defined in [metadata\$timezone](/thirdeye/alert-configuration#timezone) is used. ## Example ```json theme={null} { "name": "root", "type": "PostProcessor", "params": { "type": "TIME_OF_WEEK", "component.ignore": "true", "component.daysOfWeek": ["MONDAY", "SATURDAY"], "component.hoursOfDay": [0,1,2,23], "component.dayHoursOfWeek": {"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]} }, "inputs": [ { "sourcePlanNode": "anomalyDetector" } ] } ``` # SqlExecution Source: https://docs.startree.ai/thirdeye/operators/sql-operator Applies arbitrary SQL on any data. SQL is applied in memory. ## Inputs Any number of inputs. `targetProperty` is used as the table name when SQL is executed. ## Outputs One output for each query. `outputKey` names are `0`, `1`, `2` etc... corresponding to the number of the query. ## Parameters | name | description | default value | | - | - | - | | `sql.engine` | SQL Engine to use. Either `CALCITE` or `HYPERSQL`. | `HYPERSQL` | | `sql.queries` | A list of sql queries to execute. | | | `jdbc.parameters` | Additional jdbc parameters to pass. In a key-value format. | | ### Available SQL Engines for `engine` #### HYPERSQL HSQLDB SQL. [See SQL reference](http://hsqldb.org/doc/2.0/guide/sqlgeneral-chapt.html).\ Input data is copied to an in-memory HSQLDB, then SQL is applied. #### CALCITE * Apache Calcite SQL. Very close to ANSI SQL. [See SQL reference](https://calcite.apache.org/docs/reference.html). * Try Calcite SQL [online here](https://calcite-online.system.gavinray.dev). * BigQuery functions [listed here](https://calcite.apache.org/docs/reference.html#dialect-specific-operators) are available. In particular, `TIMESTAMP_MILLIS` and `UNIX_MILLIS` are available to easily convert epoch to time and vice-versa. * The implementation is zero-copy, meaning no data is copied from the ThirdEye app to another process. SQL is run on the Java data directly. ## Example ```json theme={null} { "name": "sql-join", "type": "SqlExecution", "params": { "sql.engine": "HYPERSQL", "sql.queries": [ "SELECT ts as timestamp, met as value FROM baseline WHERE ts BETWEEN 20200715 AND 20200720", "SELECT ts as timestamp, met as value FROM current WHERE ts BETWEEN 20200715 AND 20200720", "SELECT ts, avg(met) as met FROM (SELECT ts, met FROM baseline UNION ALL SELECT ts, met FROM current) WHERE ts BETWEEN 20200715 AND 20200720 GROUP BY ts ORDER BY ts" ], "jdbc.parameters": {"key": "value"} # dictionnary format }, "inputs": [ # two inputs { "targetProperty": "baseline", # name of the table in the queries "sourcePlanNode": "baselineDataFetcher", "sourceProperty": "baselineOutput" }, { "targetProperty": "current", # name of the table in the queries "sourcePlanNode": "currentDataFetcher", "sourceProperty": "currentOutput" } ], "outputs": [ { "outputKey": "0", # result of first query "outputName": "baseline" }, { "outputKey": "1", # result of second query "outputName": "current" }, { "outputKey": "2", # result of last query "outputName": "final" } ] } ``` # TimeIndexFiller Source: https://docs.startree.ai/thirdeye/operators/time-index-filler Fills missing time data points. Used to ensure data fetched with SQL is a correct timeseries for downstream nodes. The SQL language is not timeseries oriented. The `GROUP BY timeColumn` clause does not create a point if there is no data for a time bucket. ## Inputs A table with a time index to fill. ## Outputs The table with the time index filled. No naming constraint for `outputKey` and `outputName`. ## Parameters | name | description | default value | | - | - | - | | `component.timestamp` | The name of the time index column. | `timestamp` | | `component.monitoringGranularity` | Granularity of the time index. In ISO-8601 format. Eg `P1D`. | | | `component.minTimeInference` | Inference strategy for the minimum time constraint. | `FROM_DATA` | | `component.maxTimeInference` | Inference strategy for the maximum time constraint. | `FROM_DETECTION_TIME` | | `component.lookback` | Offset to use for a time inference with lookback. In ISO-8601 format. Eg `P7D`. | | | `component.fillNullMethod` | Method to use to fill null values in the metric column. | `FILL_WITH_ZEROES` | > [!NOTE] > > **Automatic configuration with macros.** > > `minTimeInference`, `maxTimeInference` and `lookback` are **not required** if the `__timeFilter` macro was used to get the input data.\ > `monitoringGranularity` is **not required** if the `__timeGroup` macro was used to get the input data. ### Available inference strategies for `minTimeInference` and `maxTimeInference` The `TimeIndexFiller` needs a start and an end to fill the time index. These boundaries can be computed with different strategies. | name | description | | - | - | | `FROM_DATA` | The minimum (resp maximum) time for which to fill the index is the minimum (resp maximum) time observed in the input data. | | `FROM_DETECTION_TIME` | The minimum (resp maximum) time corresponds to the minimum (resp maximum) time used by the detection pipeline run. | | `FROM_DETECTION_TIME_WITH_LOOKBACK` | Like above, use the detection pipeline boundaries, but apply an offset. Eg: minTime - lookback. | ### Available methods for `fillNullMethod` When a missing time point is added, the metric value is unknown. This value can be filled with one the following methods. | name | description | | - | - | | `FILL_WITH_ZEROES` | Replace null values with zeroes. | | `KEEP_NULL` | Keep the null values. | | `INTERPOLATE` | *Available soon.* Interpolate a value based on the preceding and next available values. | Note that most of the detectors are not compatible with null values for the moment. ## Examples ### Without macros ```json theme={null} { "name": "baselineMissingDataManager", "type": "TimeIndexFiller", "params": { "component.monitoringGranularity": "P1D", "component.timestamp": "ts", "component.minTimeInference": "FROM_DETECTION_TIME_WITH_LOOKBACK", "component.maxTimeInference": "FROM_DETECTION_TIME_WITH_LOOKBACK", "component.lookback": "P7D" }, "inputs": [ { "sourcePlanNode": "baselineDataFetcher", "sourceProperty": "baselineOutput" } ], "outputs": [ { "outputKey": "filler", "outputName": "baselineOutput" } ] } ``` ### With macros If `__timeFilter` and `__timeGroup` macros are used in the upstream DataFetcher node, and if the metric is named `metric`: ```json theme={null} { "name": "missingDataManager", "type": "TimeIndexFiller", "params": { "component.timestamp": "ts" }, "inputs": [ { "sourcePlanNode": "baselineDataFetcher", "sourceProperty": "baselineOutput" } ], "outputs": [ { "outputKey": "filler", "outputName": "baselineOutput" } ] } ``` # Aggregation Functions Source: https://docs.startree.ai/thirdeye/rca-aggregation-functions Unlike the detection pipeline, root cause analysis (RCA) algorithms have a limited set of aggregation functions available. If the [metric used for RCA](/thirdeye/alert-configuration) is onboarded in ThirdEye, a default aggregation function is available. If the metric is a derived metric or is not onboarded in ThirdEye, the aggregation function should be set in the alert configuration [aggregationFunction](/thirdeye/alert-configuration#aggregationfunction) field. ## Available aggregation functions | name | description | | - | - | | `SUM` | Sum of the metric | | `AVG` | Average of the metric | | `COUNT` | Count of the lines | | `COUNT_DISTINCT` | Distinct count of the lines | | `MAX` | Maximum of the metric | | `MIN` | Minimum of the metric | | `PCT50` | 50th percentile of the metric | | `PCT90` | 90th percentile of the metric | | `PCT95` | 95th percentile of the metric | | `PCT99` | 99th percentile of the metric | See [source code](https://github.com/startreedata/thirdeye/blob/master/thirdeye-spi/src/main/java/ai/startree/thirdeye/spi/metric/MetricAggFunction.java). ## Datasource compatibility Some datasources may not be compatible with all aggregation functions. #### Compatibility matrix | function | [Pinot](https://github.com/startreedata/thirdeye/blob/f987a621e4fd23a41d13671ca82b26d12f751a7f/thirdeye-plugins/thirdeye-pinot/src/main/java/ai/startree/thirdeye/plugins/datasource/pinot/PinotSqlExpressionBuilder.java#L179) | | - | - | | `SUM` | YES | | `AVG` | YES | | `COUNT` | YES | | `COUNT_DISTINCT` | YES | | `MAX` | YES | | `MIN` | YES | | `PCT50` | YES | | `PCT90` | YES | | `PCT95` | YES | | `PCT99` | YES | # Resources Source: https://docs.startree.ai/thirdeye/resources/presentations * [Mitigating Algorithmic Bias with Real-Time Anomaly Detection in StarTree ThirdEye | Developer Summit](https://youtu.be/trG_vvksYX0) * [ThirdEye 101 Videos](https://www.youtube.com/watch?v=xjw64iQxZn4\&list=PLihIrF0tCXdc7xMW6549RO23CxgvRkZQO) * [Troubleshooting metrics with StarTree ThirdEye | Conf42 IoT 2023)](https://www.youtube.com/watch?v=6OupH-3bEuA) * [Automated Monitoring & Anomaly Detection (M. Mantri, StarTree & L. Graveland, JustEat) RTA Summit 23](https://www.youtube.com/watch?v=1WAcLDj-XZE) * [Meetup: How to Use Anomaly Detection for Real-World Business Problems](https://youtu.be/Fv4MsfePPbc?feature=shared) * [Preventing Fraudulent Transactions in E-Commerce with Anomaly Detection](https://startree.ai/blog/startree-thirdeye-preventing-fraudulent-transactions-in-e-commerce-with-anomaly-detection) * [Why Logistics, Delivery, and Ridesharing Companies Need Anomaly Detection](https://startree.ai/blog/why-logistics-delivery-and-ridesharing-need-anomaly-detection) * [ThirdEye 101 Course](https://startree.ai/blog/thirdeye-101-course-on-startree-s-anomaly-detection-platform) * [Automated Anomaly Detection for Rideshare Booking Metrics](https://startree.ai/blog/the-power-of-automated-anomaly-detection-for-rideshare-booking-metrics) * [Meetup: Automated Monitoring for Optimizing User Experience in Consumer Products](https://youtu.be/h3H1X1LZWl4) * [StarTree ThirdEye Hackathon](https://startree.ai/blog/cracking-the-code-wins-from-startree-thirdeye-anomaly-detection-hackathon-event) * [Troubleshooting real-time business metrics with StarTree ThirdEye](https://devdays.globalai.community/day-2-deepdive-day/troubleshooting-real-time-business-metrics-with-startree-thirdeye/) * [Unlocking actionable insights from anomalous events, RTA Summit 2022](https://www.youtube.com/watch?v=yW40w2rPLFY) * [StarTree ThirdEye overview](https://docs.google.com/presentation/d/1NMR-X-SBrkYGEQDZ2X2dKSQQ04DVfyvyO50FKi-M8K8/edit?usp=sharing) * [Why did we rewrite ThirdEye?](https://docs.google.com/presentation/d/16l_eou9fNTdUtJlZapA-YTMn61dh8Lq6J52IJCWeYbM/edit?usp=sharing) * [Monitoring thousands of business metrics at scale with Thirdeye](https://www.youtube.com/watch?v=awzTcxXacI8) * [Data Quality: Timeseries Anomaly Detection at Scale with Thirdeye](https://medium.com/the-ab-tasty-tech-blog/data-quality-timeseries-anomaly-detection-at-scale-with-thirdeye-468f771154e6) * [Analyzing anomalies with ThirdEye](https://engineering.linkedin.com/blog/2020/analyzing-anomalies-with-thirdeye) * [Building Real-Time Analytics Applications Using Apache Pinot](https://www.youtube.com/watch?v=mOzjVRf0yt4) * [Smart alerts in ThirdEye, LinkedIn’s real-time monitoring platform](https://engineering.linkedin.com/blog/2019/06/smart-alerts-in-thirdeye--linkedins-real-time-monitoring-platfor) * [Introducing ThirdEye: LinkedIn’s Business-Wide Monitoring Platform](https://engineering.linkedin.com/blog/2019/01/introducing-thirdeye--linkedins-business-wide-monitoring-platfor) # startree-absolute-rule Source: https://docs.startree.ai/thirdeye/templates/startree-absolute-rule ## Description Detect an anomaly if the absolute difference between the current value and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `absoluteChange` property. Aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD eventProcessor>"eventProcessor
EVENTS "]--->root thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ABSOLUTE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager root>"root
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | absoluteChange | Difference threshold. If the absolute difference between the metric and the baseline value is above this threshold, an anomaly is detected. | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-absolute-rule-dx Source: https://docs.startree.ai/thirdeye/templates/startree-absolute-rule-dx ## Description Detect an anomaly if the absolute difference between the current value and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `absoluteChange` property. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ABSOLUTE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager childRoot>"childRoot
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | absoluteChange | Difference threshold. If the absolute difference between the metric and the baseline value is above this threshold, an anomaly is detected. | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value that follows this format: dollar sign followed by property name in curly braces. In the example above the `queryFilter` property must be set to the special value using "queryFilters" as the property name. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-absolute-rule-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-absolute-rule-percentile-dx ## Description Detect an anomaly if the absolute difference between the current value and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `absoluteChange` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ABSOLUTE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager childRoot>"childRoot
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | absoluteChange | Difference threshold. If the absolute difference between the metric and the baseline value is above this threshold, an anomaly is detected. | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-absolute-rule-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-absolute-rule-percentile-query-dx ## Description Detect an anomaly if the absolute difference between the current value and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `absoluteChange` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ABSOLUTE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager childRoot>"childRoot
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | absoluteChange | Difference threshold. If the absolute difference between the metric and the baseline value is above this threshold, an anomaly is detected. | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-absolute-rule-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-absolute-rule-query-dx ## Description Detect an anomaly if the absolute difference between the current value and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `absoluteChange` property. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ABSOLUTE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager childRoot>"childRoot
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | absoluteChange | Difference threshold. If the absolute difference between the metric and the baseline value is above this threshold, an anomaly is detected. | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-drift-continuous Source: https://docs.startree.ai/thirdeye/templates/startree-drift-continuous ## Description Detect an anomaly if the distribution drift metric of a continuous variable is above a maximum threshold. ## Flowchart ```mermaid theme={null} graph TD eventProcessor>"eventProcessor
EVENTS "]--->root guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor driftCalculator("driftCalculator ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->driftCalculator currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | P21D | | baselineOffsets | A list of baseline offsets in ISO-8601 format to use when computing the distribution drift. Eg `[P7D, P14D, P21D]` will compute the drift between the current distribution and the distribution 1 week, 2 weeks, and 3 weeks ago. The final score is the median of the drifts. | - | | distance | The distribution drift distance to use. | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | 0 | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-drift-continuous-dx Source: https://docs.startree.ai/thirdeye/templates/startree-drift-continuous-dx ## Description Detect an anomaly if the distribution drift metric of a continuous variable is above a maximum threshold. Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor driftCalculator("driftCalculator ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->driftCalculator currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | P21D | | baselineOffsets | A list of baseline offsets in ISO-8601 format to use when computing the distribution drift. Eg `[P7D, P14D, P21D]` will compute the drift between the current distribution and the distribution 1 week, 2 weeks, and 3 weeks ago. The final score is the median of the drifts. | - | | distance | The distribution drift distance to use. | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | 0 | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-drift-continuous-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-drift-continuous-query-dx ## Description Detect an anomaly if the distribution drift metric of a continuous variable is above a maximum threshold. Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor driftCalculator("driftCalculator ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->driftCalculator currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | P21D | | baselineOffsets | A list of baseline offsets in ISO-8601 format to use when computing the distribution drift. Eg `[P7D, P14D, P21D]` will compute the drift between the current distribution and the distribution 1 week, 2 weeks, and 3 weeks ago. The final score is the median of the drifts. | - | | distance | The distribution drift distance to use. | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | 0 | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-drift-discrete Source: https://docs.startree.ai/thirdeye/templates/startree-drift-discrete ## Description Detect an anomaly if the distribution drift metric of a continuous variable is above a maximum threshold. ## Flowchart ```mermaid theme={null} graph TD eventProcessor>"eventProcessor
EVENTS "]--->root guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor driftCalculator("driftCalculator ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->driftCalculator currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | P21D | | baselineOffsets | A list of baseline offsets in ISO-8601 format to use when computing the distribution drift. Eg `[P7D, P14D, P21D]` will compute the drift between the current distribution and the distribution 1 week, 2 weeks, and 3 weeks ago. The final score is the median of the drifts. | - | | distance | The distribution drift distance to use. | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | 0 | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-drift-discrete-dx Source: https://docs.startree.ai/thirdeye/templates/startree-drift-discrete-dx ## Description Detect an anomaly if the distribution drift metric of a continuous variable is above a maximum threshold. Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor driftCalculator("driftCalculator ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->driftCalculator currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | P21D | | baselineOffsets | A list of baseline offsets in ISO-8601 format to use when computing the distribution drift. Eg `[P7D, P14D, P21D]` will compute the drift between the current distribution and the distribution 1 week, 2 weeks, and 3 weeks ago. The final score is the median of the drifts. | - | | distance | The distribution drift distance to use. | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | 0 | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-drift-discrete-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-drift-discrete-query-dx ## Description Detect an anomaly if the distribution drift metric of a continuous variable is above a maximum threshold. Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor driftCalculator("driftCalculator ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->driftCalculator currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | P21D | | baselineOffsets | A list of baseline offsets in ISO-8601 format to use when computing the distribution drift. Eg `[P7D, P14D, P21D]` will compute the drift between the current distribution and the distribution 1 week, 2 weeks, and 3 weeks ago. The final score is the median of the drifts. | - | | distance | The distribution drift distance to use. | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | 0 | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-ets Source: https://docs.startree.ai/thirdeye/templates/startree-ets ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. Aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-ets-dx Source: https://docs.startree.ai/thirdeye/templates/startree-ets-dx ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-ets-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-ets-percentile ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-ets-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-ets-percentile-dx ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-ets-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-ets-percentile-query-dx ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-ets-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-ets-query-dx # startree-ets-query-dx ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-ets-ratio Source: https://docs.startree.ai/thirdeye/templates/startree-ets-ratio ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. The metric is constructed as a ratio of 2 metrics. Aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The numerator column of the ratio metric. | - | | aggregationFunction | The numerator aggregation function of the ratio metric. | - | | aggregationColumn2 | The denominator column of the ratio metric. | - | | aggregationFunction2 | The denominator aggregation function of the ratio metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | ratioMultiplier | Metric ratio multiplier. For instance, set it to 100 to return a percentage rather than a value between 0 and 1. | 1 | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-ets-ratio-dx Source: https://docs.startree.ai/thirdeye/templates/startree-ets-ratio-dx ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. The metric is constructed as a ratio of 2 metrics. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The numerator column of the ratio metric. | - | | aggregationFunction | The numerator aggregation function of the ratio metric. | - | | aggregationColumn2 | The denominator column of the ratio metric. | - | | aggregationFunction2 | The denominator aggregation function of the ratio metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | ratioMultiplier | Metric ratio multiplier. For instance, set it to 100 to return a percentage rather than a value between 0 and 1. | 1 | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-ets-ratio-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-ets-ratio-percentile ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. The metric is constructed as a ratio of 2 metrics. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The numerator column of the ratio metric. | - | | aggregationFunction | The numerator aggregation function of the ratio metric. | - | | aggregationColumn2 | The denominator column of the ratio metric. | - | | aggregationFunction2 | The denominator aggregation function of the ratio metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | ratioMultiplier | Metric ratio multiplier. For instance, set it to 100 to return a percentage rather than a value between 0 and 1. | 1 | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-ets-ratio-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-ets-ratio-percentile-dx ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. The metric is constructed as a ratio of 2 metrics. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The numerator column of the ratio metric. | - | | aggregationFunction | The numerator aggregation function of the ratio metric. | - | | aggregationColumn2 | The denominator column of the ratio metric. | - | | aggregationFunction2 | The denominator aggregation function of the ratio metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | ratioMultiplier | Metric ratio multiplier. For instance, set it to 100 to return a percentage rather than a value between 0 and 1. | 1 | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-ets-ratio-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-ets-ratio-percentile-query-dx ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. The metric is constructed as a ratio of 2 metrics. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The numerator column of the ratio metric. | - | | aggregationFunction | The numerator aggregation function of the ratio metric. | - | | aggregationColumn2 | The denominator column of the ratio metric. | - | | aggregationFunction2 | The denominator aggregation function of the ratio metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | ratioMultiplier | Metric ratio multiplier. For instance, set it to 100 to return a percentage rather than a value between 0 and 1. | 1 | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-ets-ratio-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-ets-ratio-query-dx ## Description Detect an anomaly if the metric is outside the prediction boundaries of a model combining a linear regression and an ETS forecasting algorithm. The regression model learns the effect of events The ETS model learns the level, trend and seasonality in the timeseries. The metric is constructed as a ratio of 2 metrics. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
ETS "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector eventsForFilterDataFetcher[("eventsForFilterDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The numerator column of the ratio metric. | - | | aggregationFunction | The numerator aggregation function of the ratio metric. | - | | aggregationColumn2 | The denominator column of the ratio metric. | - | | aggregationFunction2 | The denominator aggregation function of the ratio metric. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | ratioMultiplier | Metric ratio multiplier. For instance, set it to 100 to return a percentage rather than a value between 0 and 1. | 1 | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION #### MAIN | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | - | | alpha | ETS level smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | beta | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | gamma | ETS seasonal smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. | -1 | | phi | ETS trend smoothing factor. In `[0,1]`. `-1` means auto-optimized by BOBYQA. Only used if `trendMode` is set to `DAMPED`. | -1 | | yeoJohnsonLambda | Experimental. Yeo Johnson transformation lambda. Can be used if the timeseries has heteroskedasticity. | - | | robustInitialization | Experimental. Whether the model should be robust to anomalies in the historical data at the initialization phase. Requires at least 3 seasonal periods of lookback. | true | | followDst | Only applies to timezones that have DST changes. If True, the seasonal component of the ETS model follows the DST changes of the timezone. If false, the seasonal component follows the physical time. | true | | robustFitting | Experimental. Whether the model should be robust to anomalies in the historical data at the fitting phase. Requires at least 3 seasonal periods of lookback. | true | | robustIntervalsLambda | Experimental. Whether the intervals should be robust to anomalies in the historical data. Between 0 and 1. If 0, intervals are not robust. The closer to 1, the more the confidence intervals depend on recent observations. | 0.1 | | intervalsMethod | Method to compute intervals. In CONFIDENCE, PERCENTAGE, ABSOLUTE. | CONFIDENCE | | errorMode | ETS error mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | seasonalMode | ETS seasonal mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | ADDITIVE | | trendMode | ETS trend mode as defined [here](https://www.researchgate.net/publication/5179823_Prediction_Intervals_for_Exponential_Smoothing_State_Space_Models). | NONE | | regressors | For advanced users. Additional list of features to add to the regression model. These additional features may help the model to learn the effect of events. Events features are created automatically. [Learn more](http://localhost:3000/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/ets#regressors). | \[] | #### Events | name | description | default value | | - | - | - | | eventSqlFilter | Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventLookaround | When fetching events, additional margin to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P1D | | eventTypes | Type of events to fetch. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` or `null` means no filtering. The default value `["__NO_EVENTS"]` means don't fetch events. | \['\_\_NO\_EVENTS'] | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-experimental-mean-variance-with-mask Source: https://docs.startree.ai/thirdeye/templates/startree-experimental-mean-variance-with-mask ## Description Experimental. Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 1 operand: SUM, MAX,etc...Events can be used to mask some timeframes. ## Flowchart ```mermaid theme={null} graph TD guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->root metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor sqlFilterThresholdProcessor>"sqlFilterThresholdProcessor
THRESHOLD "]--->coldStartProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->sqlFilterThresholdProcessor eventMasker("eventMasker ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->eventMasker currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventMasker sqlFilterExecutor("sqlFilterExecutor ")--->sqlFilterThresholdProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### PREPROCESS | name | description | default value | | - | - | - | | eventMaskerSqlFilter | Used to mask periods based on events. Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventMaskerLookaround | Used to mask periods based on events. Offset to apply on startTime and endTime to look around the timeframe when fetching events. In ISO-8601 format. Example: `P1D`. | P2D | | eventMaskerTypes | Used to mask periods based on events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventMaskerBeforeEventMargin | Used to mask periods based on events. A period in ISO-8601 format that corresponds to a period that should be included in the event. Example: if beforeEventMargin is `P1D` and the event happens on `[Dec 24 0:00, Dec 25 0:00[`, then the event will be considered to happen on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventMaskerAfterEventMargin | Used to mask periods based on events. Same as eventMaskerBeforeEventMargin at the end of the event. | P0D | | eventMaskerStrategy | Strategy to apply the mask. MASK\_WHEN\_IN\_EVENT masks periods where an event happens. MASK\_WHEN\_OUT\_OF\_EVENT masks periods where an event does not happen. | MASK\_WHEN\_IN\_EVENT | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The sensitivity of the model. The smaller, the less anomaly are detected. | - | | lowerSensitivity | The sensitivity for the lower bounds. The smaller, the less anomaly are detected. If used, upperSensitivity must be set. | - | | upperSensitivity | The sensitivity for the upper bounds. The smaller, the less anomaly are detected. If used, lowerSensitivity must be set. | - | | upperBoundMultiplier | Factor applied to the upper bound, such that `finalUpperBound = upperBound * upperBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | lowerBoundMultiplier | Factor applied to the lower bound, such that `finalLowerBound = lowerBound * lowerBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Sql Filter | name | description | default value | | - | - | - | | sqlFilterStatement | Sql statement to ignore anomalies based on values returned by the detector. If the statement evaluates to true, the anomaly is ignored. Available columns: `observed` (the current value), `predicted` (the value predicted by the detector), `upper_bound` (the upper bound value predicted by the detector), `lower_bound` and `ts` (the timestamp of the point). For instance: `(lower_bound < 10) or (lower_bound = upper_bound and upper_bound = 0)`. The default statement `false` means no anomaly is filtered. | false | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | | thresholdFilterMetrics | A list of metrics to apply the threshold filter on. For instance, if set to `["current", "upperBound"]`, the filter will apply when both current and upperBound values are outside the filter range. | - | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxDuration | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. To disable anomalies merging, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-experimental-mean-variance-with-mask-dx Source: https://docs.startree.ai/thirdeye/templates/startree-experimental-mean-variance-with-mask-dx ## Description Experimental. Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 1 operand: SUM, MAX,etc...Events can be used to mask some timeframes. Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->childRoot metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor sqlFilterThresholdProcessor>"sqlFilterThresholdProcessor
THRESHOLD "]--->coldStartProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->sqlFilterThresholdProcessor eventMasker("eventMasker ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->eventMasker currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventMasker sqlFilterExecutor("sqlFilterExecutor ")--->sqlFilterThresholdProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### PREPROCESS | name | description | default value | | - | - | - | | eventMaskerSqlFilter | Used to mask periods based on events. Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventMaskerLookaround | Used to mask periods based on events. Offset to apply on startTime and endTime to look around the timeframe when fetching events. In ISO-8601 format. Example: `P1D`. | P2D | | eventMaskerTypes | Used to mask periods based on events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventMaskerBeforeEventMargin | Used to mask periods based on events. A period in ISO-8601 format that corresponds to a period that should be included in the event. Example: if beforeEventMargin is `P1D` and the event happens on `[Dec 24 0:00, Dec 25 0:00[`, then the event will be considered to happen on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventMaskerAfterEventMargin | Used to mask periods based on events. Same as eventMaskerBeforeEventMargin at the end of the event. | P0D | | eventMaskerStrategy | Strategy to apply the mask. MASK\_WHEN\_IN\_EVENT masks periods where an event happens. MASK\_WHEN\_OUT\_OF\_EVENT masks periods where an event does not happen. | MASK\_WHEN\_IN\_EVENT | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The sensitivity of the model. The smaller, the less anomaly are detected. | - | | lowerSensitivity | The sensitivity for the lower bounds. The smaller, the less anomaly are detected. If used, upperSensitivity must be set. | - | | upperSensitivity | The sensitivity for the upper bounds. The smaller, the less anomaly are detected. If used, lowerSensitivity must be set. | - | | upperBoundMultiplier | Factor applied to the upper bound, such that `finalUpperBound = upperBound * upperBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | lowerBoundMultiplier | Factor applied to the lower bound, such that `finalLowerBound = lowerBound * lowerBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Sql Filter | name | description | default value | | - | - | - | | sqlFilterStatement | Sql statement to ignore anomalies based on values returned by the detector. If the statement evaluates to true, the anomaly is ignored. Available columns: `observed` (the current value), `predicted` (the value predicted by the detector), `upper_bound` (the upper bound value predicted by the detector), `lower_bound` and `ts` (the timestamp of the point). For instance: `(lower_bound < 10) or (lower_bound = upper_bound and upper_bound = 0)`. The default statement `false` means no anomaly is filtered. | false | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | | thresholdFilterMetrics | A list of metrics to apply the threshold filter on. For instance, if set to `["current", "upperBound"]`, the filter will apply when both current and upperBound values are outside the filter range. | - | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxDuration | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. To disable anomalies merging, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-experimental-mean-variance-with-mask-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-experimental-mean-variance-with-mask-percentile ## Description Experimental. Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... ## Flowchart ```mermaid theme={null} graph TD guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->root metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor sqlFilterThresholdProcessor>"sqlFilterThresholdProcessor
THRESHOLD "]--->coldStartProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->sqlFilterThresholdProcessor eventMasker("eventMasker ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->eventMasker currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventMasker sqlFilterExecutor("sqlFilterExecutor ")--->sqlFilterThresholdProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### PREPROCESS | name | description | default value | | - | - | - | | eventMaskerSqlFilter | Used to mask periods based on events. Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventMaskerLookaround | Used to mask periods based on events. Offset to apply on startTime and endTime to look around the timeframe when fetching events. In ISO-8601 format. Example: `P1D`. | P2D | | eventMaskerTypes | Used to mask periods based on events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventMaskerBeforeEventMargin | Used to mask periods based on events. A period in ISO-8601 format that corresponds to a period that should be included in the event. Example: if beforeEventMargin is `P1D` and the event happens on `[Dec 24 0:00, Dec 25 0:00[`, then the event will be considered to happen on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventMaskerAfterEventMargin | Used to mask periods based on events. Same as eventMaskerBeforeEventMargin at the end of the event. | P0D | | eventMaskerStrategy | Strategy to apply the mask. MASK\_WHEN\_IN\_EVENT masks periods where an event happens. MASK\_WHEN\_OUT\_OF\_EVENT masks periods where an event does not happen. | MASK\_WHEN\_IN\_EVENT | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The sensitivity of the model. The smaller, the less anomaly are detected. | - | | lowerSensitivity | The sensitivity for the lower bounds. The smaller, the less anomaly are detected. If used, upperSensitivity must be set. | - | | upperSensitivity | The sensitivity for the upper bounds. The smaller, the less anomaly are detected. If used, lowerSensitivity must be set. | - | | upperBoundMultiplier | Factor applied to the upper bound, such that `finalUpperBound = upperBound * upperBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | lowerBoundMultiplier | Factor applied to the lower bound, such that `finalLowerBound = lowerBound * lowerBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Sql Filter | name | description | default value | | - | - | - | | sqlFilterStatement | Sql statement to ignore anomalies based on values returned by the detector. If the statement evaluates to true, the anomaly is ignored. Available columns: `observed` (the current value), `predicted` (the value predicted by the detector), `upper_bound` (the upper bound value predicted by the detector), `lower_bound` and `ts` (the timestamp of the point). For instance: `(lower_bound < 10) or (lower_bound = upper_bound and upper_bound = 0)`. The default statement `false` means no anomaly is filtered. | false | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | | thresholdFilterMetrics | A list of metrics to apply the threshold filter on. For instance, if set to `["current", "upperBound"]`, the filter will apply when both current and upperBound values are outside the filter range. | - | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxDuration | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. To disable anomalies merging, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-experimental-mean-variance-with-mask-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-experimental-mean-variance-with-mask-percentile-dx ## Description Experimental. Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->childRoot metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor sqlFilterThresholdProcessor>"sqlFilterThresholdProcessor
THRESHOLD "]--->coldStartProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->sqlFilterThresholdProcessor eventMasker("eventMasker ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->eventMasker currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventMasker sqlFilterExecutor("sqlFilterExecutor ")--->sqlFilterThresholdProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### PREPROCESS | name | description | default value | | - | - | - | | eventMaskerSqlFilter | Used to mask periods based on events. Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventMaskerLookaround | Used to mask periods based on events. Offset to apply on startTime and endTime to look around the timeframe when fetching events. In ISO-8601 format. Example: `P1D`. | P2D | | eventMaskerTypes | Used to mask periods based on events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventMaskerBeforeEventMargin | Used to mask periods based on events. A period in ISO-8601 format that corresponds to a period that should be included in the event. Example: if beforeEventMargin is `P1D` and the event happens on `[Dec 24 0:00, Dec 25 0:00[`, then the event will be considered to happen on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventMaskerAfterEventMargin | Used to mask periods based on events. Same as eventMaskerBeforeEventMargin at the end of the event. | P0D | | eventMaskerStrategy | Strategy to apply the mask. MASK\_WHEN\_IN\_EVENT masks periods where an event happens. MASK\_WHEN\_OUT\_OF\_EVENT masks periods where an event does not happen. | MASK\_WHEN\_IN\_EVENT | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The sensitivity of the model. The smaller, the less anomaly are detected. | - | | lowerSensitivity | The sensitivity for the lower bounds. The smaller, the less anomaly are detected. If used, upperSensitivity must be set. | - | | upperSensitivity | The sensitivity for the upper bounds. The smaller, the less anomaly are detected. If used, lowerSensitivity must be set. | - | | upperBoundMultiplier | Factor applied to the upper bound, such that `finalUpperBound = upperBound * upperBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | lowerBoundMultiplier | Factor applied to the lower bound, such that `finalLowerBound = lowerBound * lowerBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Sql Filter | name | description | default value | | - | - | - | | sqlFilterStatement | Sql statement to ignore anomalies based on values returned by the detector. If the statement evaluates to true, the anomaly is ignored. Available columns: `observed` (the current value), `predicted` (the value predicted by the detector), `upper_bound` (the upper bound value predicted by the detector), `lower_bound` and `ts` (the timestamp of the point). For instance: `(lower_bound < 10) or (lower_bound = upper_bound and upper_bound = 0)`. The default statement `false` means no anomaly is filtered. | false | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | | thresholdFilterMetrics | A list of metrics to apply the threshold filter on. For instance, if set to `["current", "upperBound"]`, the filter will apply when both current and upperBound values are outside the filter range. | - | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxDuration | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. To disable anomalies merging, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-experimental-mean-variance-with-mask-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-experimental-mean-variance-with-mask-percentile-query-dx ## Description Experimental. Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->childRoot metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor sqlFilterThresholdProcessor>"sqlFilterThresholdProcessor
THRESHOLD "]--->coldStartProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->sqlFilterThresholdProcessor eventMasker("eventMasker ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->eventMasker currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventMasker sqlFilterExecutor("sqlFilterExecutor ")--->sqlFilterThresholdProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### PREPROCESS | name | description | default value | | - | - | - | | eventMaskerSqlFilter | Used to mask periods based on events. Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventMaskerLookaround | Used to mask periods based on events. Offset to apply on startTime and endTime to look around the timeframe when fetching events. In ISO-8601 format. Example: `P1D`. | P2D | | eventMaskerTypes | Used to mask periods based on events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventMaskerBeforeEventMargin | Used to mask periods based on events. A period in ISO-8601 format that corresponds to a period that should be included in the event. Example: if beforeEventMargin is `P1D` and the event happens on `[Dec 24 0:00, Dec 25 0:00[`, then the event will be considered to happen on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventMaskerAfterEventMargin | Used to mask periods based on events. Same as eventMaskerBeforeEventMargin at the end of the event. | P0D | | eventMaskerStrategy | Strategy to apply the mask. MASK\_WHEN\_IN\_EVENT masks periods where an event happens. MASK\_WHEN\_OUT\_OF\_EVENT masks periods where an event does not happen. | MASK\_WHEN\_IN\_EVENT | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The sensitivity of the model. The smaller, the less anomaly are detected. | - | | lowerSensitivity | The sensitivity for the lower bounds. The smaller, the less anomaly are detected. If used, upperSensitivity must be set. | - | | upperSensitivity | The sensitivity for the upper bounds. The smaller, the less anomaly are detected. If used, lowerSensitivity must be set. | - | | upperBoundMultiplier | Factor applied to the upper bound, such that `finalUpperBound = upperBound * upperBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | lowerBoundMultiplier | Factor applied to the lower bound, such that `finalLowerBound = lowerBound * lowerBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Sql Filter | name | description | default value | | - | - | - | | sqlFilterStatement | Sql statement to ignore anomalies based on values returned by the detector. If the statement evaluates to true, the anomaly is ignored. Available columns: `observed` (the current value), `predicted` (the value predicted by the detector), `upper_bound` (the upper bound value predicted by the detector), `lower_bound` and `ts` (the timestamp of the point). For instance: `(lower_bound < 10) or (lower_bound = upper_bound and upper_bound = 0)`. The default statement `false` means no anomaly is filtered. | false | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | | thresholdFilterMetrics | A list of metrics to apply the threshold filter on. For instance, if set to `["current", "upperBound"]`, the filter will apply when both current and upperBound values are outside the filter range. | - | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxDuration | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. To disable anomalies merging, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-experimental-mean-variance-with-mask-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-experimental-mean-variance-with-mask-query-dx ## Description Experimental. Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 1 operand: SUM, MAX,etc...Events can be used to mask some timeframes. Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->childRoot metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor sqlFilterThresholdProcessor>"sqlFilterThresholdProcessor
THRESHOLD "]--->coldStartProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->sqlFilterThresholdProcessor eventMasker("eventMasker ")--->anomalyDetector missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->eventMasker currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventMasker sqlFilterExecutor("sqlFilterExecutor ")--->sqlFilterThresholdProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### PREPROCESS | name | description | default value | | - | - | - | | eventMaskerSqlFilter | Used to mask periods based on events. Sql filter to apply when fetching events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventMaskerLookaround | Used to mask periods based on events. Offset to apply on startTime and endTime to look around the timeframe when fetching events. In ISO-8601 format. Example: `P1D`. | P2D | | eventMaskerTypes | Used to mask periods based on events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventMaskerBeforeEventMargin | Used to mask periods based on events. A period in ISO-8601 format that corresponds to a period that should be included in the event. Example: if beforeEventMargin is `P1D` and the event happens on `[Dec 24 0:00, Dec 25 0:00[`, then the event will be considered to happen on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventMaskerAfterEventMargin | Used to mask periods based on events. Same as eventMaskerBeforeEventMargin at the end of the event. | P0D | | eventMaskerStrategy | Strategy to apply the mask. MASK\_WHEN\_IN\_EVENT masks periods where an event happens. MASK\_WHEN\_OUT\_OF\_EVENT masks periods where an event does not happen. | MASK\_WHEN\_IN\_EVENT | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The sensitivity of the model. The smaller, the less anomaly are detected. | - | | lowerSensitivity | The sensitivity for the lower bounds. The smaller, the less anomaly are detected. If used, upperSensitivity must be set. | - | | upperSensitivity | The sensitivity for the upper bounds. The smaller, the less anomaly are detected. If used, lowerSensitivity must be set. | - | | upperBoundMultiplier | Factor applied to the upper bound, such that `finalUpperBound = upperBound * upperBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | lowerBoundMultiplier | Factor applied to the lower bound, such that `finalLowerBound = lowerBound * lowerBoundMultiplier`. Can help to stabilize bounds. If not set, no factor is applied. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Sql Filter | name | description | default value | | - | - | - | | sqlFilterStatement | Sql statement to ignore anomalies based on values returned by the detector. If the statement evaluates to true, the anomaly is ignored. Available columns: `observed` (the current value), `predicted` (the value predicted by the detector), `upper_bound` (the upper bound value predicted by the detector), `lower_bound` and `ts` (the timestamp of the point). For instance: `(lower_bound < 10) or (lower_bound = upper_bound and upper_bound = 0)`. The default statement `false` means no anomaly is filtered. | false | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | | thresholdFilterMetrics | A list of metrics to apply the threshold filter on. For instance, if set to `["current", "upperBound"]`, the filter will apply when both current and upperBound values are outside the filter range. | - | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxDuration | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. To disable anomalies merging, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-matrix-profile Source: https://docs.startree.ai/thirdeye/templates/startree-matrix-profile ## Description Detect an anomaly if the anomaly score returned by a [matrix profile](https://www.cs.ucr.edu/~eamonn/MatrixProfile.html) algorithm is above the threshold. Aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MATRIX_PROFILE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | | | matrixProfileDistance | For advanced users. Type of distance to use for the matrix profile computation. `NORMALIZED` or `NON_NORMALIZED`. | NORMALIZED | | sensitivity | Anomaly score threshold. If the anomaly score returned by the algorithm is bigger than this threshold, return an anomaly. The smaller, the more anomalies are detected. | 1 | | computeBounds | For advanced users. If false, the algorithm returns the anomaly scores instead of the predicted upper/lower bounds. Helpful for alert finetuning. | true | | scoringMethod | Experimental. Post-processing of the matrix profile anomaly scores. `FIRST_ORDER_DIFFERENCE` is good at detecting spikes. `DIRECT` is good at detecting drifts. | FIRST\_ORDER\_DIFFERENCE | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-matrix-profile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-matrix-profile-dx ## Description Detect an anomaly if the anomaly score returned by a [matrix profile](https://www.cs.ucr.edu/~eamonn/MatrixProfile.html) algorithm is above the threshold. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MATRIX_PROFILE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | | | matrixProfileDistance | For advanced users. Type of distance to use for the matrix profile computation. `NORMALIZED` or `NON_NORMALIZED`. | NORMALIZED | | sensitivity | Anomaly score threshold. If the anomaly score returned by the algorithm is bigger than this threshold, return an anomaly. The smaller, the more anomalies are detected. | 1 | | computeBounds | For advanced users. If false, the algorithm returns the anomaly scores instead of the predicted upper/lower bounds. Helpful for alert finetuning. | true | | scoringMethod | Experimental. Post-processing of the matrix profile anomaly scores. `FIRST_ORDER_DIFFERENCE` is good at detecting spikes. `DIRECT` is good at detecting drifts. | FIRST\_ORDER\_DIFFERENCE | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-matrix-profile-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-matrix-profile-percentile ## Description Detect an anomaly if the anomaly score returned by a [matrix profile](https://www.cs.ucr.edu/~eamonn/MatrixProfile.html) algorithm is above the threshold. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MATRIX_PROFILE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | | | matrixProfileDistance | For advanced users. Type of distance to use for the matrix profile computation. `NORMALIZED` or `NON_NORMALIZED`. | NORMALIZED | | sensitivity | Anomaly score threshold. If the anomaly score returned by the algorithm is bigger than this threshold, return an anomaly. The smaller, the more anomalies are detected. | 1 | | computeBounds | For advanced users. If false, the algorithm returns the anomaly scores instead of the predicted upper/lower bounds. Helpful for alert finetuning. | true | | scoringMethod | Experimental. Post-processing of the matrix profile anomaly scores. `FIRST_ORDER_DIFFERENCE` is good at detecting spikes. `DIRECT` is good at detecting drifts. | FIRST\_ORDER\_DIFFERENCE | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-matrix-profile-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-matrix-profile-percentile-dx ## Description Detect an anomaly if the anomaly score returned by a [matrix profile](https://www.cs.ucr.edu/~eamonn/MatrixProfile.html) algorithm is above the threshold. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MATRIX_PROFILE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | | | matrixProfileDistance | For advanced users. Type of distance to use for the matrix profile computation. `NORMALIZED` or `NON_NORMALIZED`. | NORMALIZED | | sensitivity | Anomaly score threshold. If the anomaly score returned by the algorithm is bigger than this threshold, return an anomaly. The smaller, the more anomalies are detected. | 1 | | computeBounds | For advanced users. If false, the algorithm returns the anomaly scores instead of the predicted upper/lower bounds. Helpful for alert finetuning. | true | | scoringMethod | Experimental. Post-processing of the matrix profile anomaly scores. `FIRST_ORDER_DIFFERENCE` is good at detecting spikes. `DIRECT` is good at detecting drifts. | FIRST\_ORDER\_DIFFERENCE | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-matrix-profile-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-matrix-profile-percentile-query-dx ## Description Detect an anomaly if the anomaly score returned by a [matrix profile](https://www.cs.ucr.edu/~eamonn/MatrixProfile.html) algorithm is above the threshold. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MATRIX_PROFILE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | | | matrixProfileDistance | For advanced users. Type of distance to use for the matrix profile computation. `NORMALIZED` or `NON_NORMALIZED`. | NORMALIZED | | sensitivity | Anomaly score threshold. If the anomaly score returned by the algorithm is bigger than this threshold, return an anomaly. The smaller, the more anomalies are detected. | 1 | | computeBounds | For advanced users. If false, the algorithm returns the anomaly scores instead of the predicted upper/lower bounds. Helpful for alert finetuning. | true | | scoringMethod | Experimental. Post-processing of the matrix profile anomaly scores. `FIRST_ORDER_DIFFERENCE` is good at detecting spikes. `DIRECT` is good at detecting drifts. | FIRST\_ORDER\_DIFFERENCE | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-matrix-profile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-matrix-profile-query-dx ## Description Detect an anomaly if the anomaly score returned by a [matrix profile](https://www.cs.ucr.edu/~eamonn/MatrixProfile.html) algorithm is above the threshold. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MATRIX_PROFILE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | | | matrixProfileDistance | For advanced users. Type of distance to use for the matrix profile computation. `NORMALIZED` or `NON_NORMALIZED`. | NORMALIZED | | sensitivity | Anomaly score threshold. If the anomaly score returned by the algorithm is bigger than this threshold, return an anomaly. The smaller, the more anomalies are detected. | 1 | | computeBounds | For advanced users. If false, the algorithm returns the anomaly scores instead of the predicted upper/lower bounds. Helpful for alert finetuning. | true | | scoringMethod | Experimental. Post-processing of the matrix profile anomaly scores. `FIRST_ORDER_DIFFERENCE` is good at detecting spikes. `DIRECT` is good at detecting drifts. | FIRST\_ORDER\_DIFFERENCE | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-mean-variance Source: https://docs.startree.ai/thirdeye/templates/startree-mean-variance ## Description Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD eventProcessor>"eventProcessor
EVENTS "]--->root guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-mean-variance-dx Source: https://docs.startree.ai/thirdeye/templates/startree-mean-variance-dx ## Description Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-mean-variance-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-mean-variance-percentile ## Description Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... ## Flowchart ```mermaid theme={null} graph TD eventProcessor>"eventProcessor
EVENTS "]--->root guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-mean-variance-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-mean-variance-percentile-dx ## Description Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-mean-variance-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-mean-variance-percentile-query-dx ## Description Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-mean-variance-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-mean-variance-query-dx ## Description Detect an anomaly if the metric is not in `mean ± n*std`. `mean` and `std` (standard deviation) are estimated with historical data. The amount of historical data to use is set with the `lookback` property. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MEAN_VARIANCE "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | sensitivity | The model will detect fewer anomalies with lower sensitivity and more with higher sensitivity. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Seasonality to consider when computing mean and variance. Possible values are `P7D` (weekly and smaller periods), `P1D` (daily and smaller periods), PT0S (no seasonality). Eg: with P7D, a Monday 12 AM value will be estimated from the mean and variance of the previous Monday 12 AM values. | PT0S | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-mstl Source: https://docs.startree.ai/thirdeye/templates/startree-mstl ## Description Detect an anomaly if the metric is outside the predicted boundaries of a MSTL decomposition model. Aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MSTL "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriods | List of periods. In ISO-8601 format. For instance `["P7D", "P1D"]`. If null, periods are inferred. | - | | sensitivity | Multiplier of the estimated acceptable range, based on the standard deviation. The bigger, the fewer anomalies. 3 corresponds to the 3-sigma rule. | - | | lambda | Box-Cox transformation parameter. If `null` no box-cox transformation is applied. | - | | stlInnerIterations | For advanced users. Number of inner iterations in STL decomposition. | - | | stlRobustnessIterations | For advanced users. Number of robustness iterations in STL decomposition. | - | | confidenceRobustnessIterations | For advanced users. The bigger, the less confidence intervals are impacted by anomalies. | - | | seasonalStrengthThreshold | For advanced users. Seasonal strength threshold when `TS_OUTLIERS` decomposer is used. [Learn more](https://robjhyndman.com/hyndsight/tsoutliers/). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-mstl-dx Source: https://docs.startree.ai/thirdeye/templates/startree-mstl-dx ## Description Detect an anomaly if the metric is outside the predicted boundaries of a MSTL decomposition model. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MSTL "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriods | List of periods. In ISO-8601 format. For instance `["P7D", "P1D"]`. If null, periods are inferred. | - | | sensitivity | Multiplier of the estimated acceptable range, based on the standard deviation. The bigger, the fewer anomalies. 3 corresponds to the 3-sigma rule. | - | | lambda | Box-Cox transformation parameter. If `null` no box-cox transformation is applied. | - | | stlInnerIterations | For advanced users. Number of inner iterations in STL decomposition. | - | | stlRobustnessIterations | For advanced users. Number of robustness iterations in STL decomposition. | - | | confidenceRobustnessIterations | For advanced users. The bigger, the less confidence intervals are impacted by anomalies. | - | | seasonalStrengthThreshold | For advanced users. Seasonal strength threshold when `TS_OUTLIERS` decomposer is used. [Learn more](https://robjhyndman.com/hyndsight/tsoutliers/). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-mstl-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-mstl-percentile ## Description Detect an anomaly if the metric is outside the predicted boundaries of a MSTL decomposition model. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MSTL "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriods | List of periods. In ISO-8601 format. For instance `["P7D", "P1D"]`. If null, periods are inferred. | - | | sensitivity | Multiplier of the estimated acceptable range, based on the standard deviation. The bigger, the fewer anomalies. 3 corresponds to the 3-sigma rule. | - | | lambda | Box-Cox transformation parameter. If `null` no box-cox transformation is applied. | - | | stlInnerIterations | For advanced users. Number of inner iterations in STL decomposition. | - | | stlRobustnessIterations | For advanced users. Number of robustness iterations in STL decomposition. | - | | confidenceRobustnessIterations | For advanced users. The bigger, the less confidence intervals are impacted by anomalies. | - | | seasonalStrengthThreshold | For advanced users. Seasonal strength threshold when `TS_OUTLIERS` decomposer is used. [Learn more](https://robjhyndman.com/hyndsight/tsoutliers/). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-mstl-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-mstl-percentile-dx ## Description Detect an anomaly if the metric is outside the predicted boundaries of a MSTL decomposition model. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MSTL "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriods | List of periods. In ISO-8601 format. For instance `["P7D", "P1D"]`. If null, periods are inferred. | - | | sensitivity | Multiplier of the estimated acceptable range, based on the standard deviation. The bigger, the fewer anomalies. 3 corresponds to the 3-sigma rule. | - | | lambda | Box-Cox transformation parameter. If `null` no box-cox transformation is applied. | - | | stlInnerIterations | For advanced users. Number of inner iterations in STL decomposition. | - | | stlRobustnessIterations | For advanced users. Number of robustness iterations in STL decomposition. | - | | confidenceRobustnessIterations | For advanced users. The bigger, the less confidence intervals are impacted by anomalies. | - | | seasonalStrengthThreshold | For advanced users. Seasonal strength threshold when `TS_OUTLIERS` decomposer is used. [Learn more](https://robjhyndman.com/hyndsight/tsoutliers/). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-mstl-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-mstl-percentile-query-dx ## Description Detect an anomaly if the metric is outside the predicted boundaries of a MSTL decomposition model. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MSTL "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriods | List of periods. In ISO-8601 format. For instance `["P7D", "P1D"]`. If null, periods are inferred. | - | | sensitivity | Multiplier of the estimated acceptable range, based on the standard deviation. The bigger, the fewer anomalies. 3 corresponds to the 3-sigma rule. | - | | lambda | Box-Cox transformation parameter. If `null` no box-cox transformation is applied. | - | | stlInnerIterations | For advanced users. Number of inner iterations in STL decomposition. | - | | stlRobustnessIterations | For advanced users. Number of robustness iterations in STL decomposition. | - | | confidenceRobustnessIterations | For advanced users. The bigger, the less confidence intervals are impacted by anomalies. | - | | seasonalStrengthThreshold | For advanced users. Seasonal strength threshold when `TS_OUTLIERS` decomposer is used. [Learn more](https://robjhyndman.com/hyndsight/tsoutliers/). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-mstl-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-mstl-query-dx ## Description Detect an anomaly if the metric is outside the predicted boundaries of a MSTL decomposition model. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
MSTL "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | seasonalityPeriods | List of periods. In ISO-8601 format. For instance `["P7D", "P1D"]`. If null, periods are inferred. | - | | sensitivity | Multiplier of the estimated acceptable range, based on the standard deviation. The bigger, the fewer anomalies. 3 corresponds to the 3-sigma rule. | - | | lambda | Box-Cox transformation parameter. If `null` no box-cox transformation is applied. | - | | stlInnerIterations | For advanced users. Number of inner iterations in STL decomposition. | - | | stlRobustnessIterations | For advanced users. Number of robustness iterations in STL decomposition. | - | | confidenceRobustnessIterations | For advanced users. The bigger, the less confidence intervals are impacted by anomalies. | - | | seasonalStrengthThreshold | For advanced users. Seasonal strength threshold when `TS_OUTLIERS` decomposer is used. [Learn more](https://robjhyndman.com/hyndsight/tsoutliers/). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-offset-aggregation Source: https://docs.startree.ai/thirdeye/templates/startree-offset-aggregation ## Description Detect an anomaly if the metric is too different from a naive baseline. The naive baseline is a [naive model](https://otexts.com/fpp2/simple-methods.html) configured with `modelOffsets` and `modelAggregation`. For instance the default baseline is the value of last week (`modelOffsets=[P7D]`, `modelAggregation=MEAN`). The difference can be PERCENTAGE or ABSOLUTE, it is set with the `intervalsMethod` property. The threshold is set with `sensitivity`. SQL aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
OFFSET_AGGREGATION "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | sensitivity | Detection sensitivity. For instance with `intervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `intervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | modelOffsets | A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | intervalsMethod | Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | PERCENTAGE | | modelAggregation | The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | mean | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-offset-aggregation-dx Source: https://docs.startree.ai/thirdeye/templates/startree-offset-aggregation-dx ## Description Detect an anomaly if the metric is too different from a naive baseline. The naive baseline is a [naive model](https://otexts.com/fpp2/simple-methods.html) configured with `modelOffsets` and `modelAggregation`. For instance the default baseline is the value of last week (`modelOffsets=[P7D]`, `modelAggregation=MEAN`). The difference can be PERCENTAGE or ABSOLUTE, it is set with the `intervalsMethod` property. The threshold is set with `sensitivity`. SQL aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
OFFSET_AGGREGATION "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | sensitivity | Detection sensitivity. For instance with `intervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `intervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | modelOffsets | A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | intervalsMethod | Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | PERCENTAGE | | modelAggregation | The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | mean | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-offset-aggregation-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-offset-aggregation-percentile ## Description Detect an anomaly if the metric is too different from a naive baseline. The naive baseline is a [naive model](https://otexts.com/fpp2/simple-methods.html) configured with `modelOffsets` and `modelAggregation`. For instance the default baseline is the value of last week (`modelOffsets=[P7D]`, `modelAggregation=MEAN`). The difference can be PERCENTAGE or ABSOLUTE, it is set with the `intervalsMethod` property. The threshold is set with `sensitivity`. SQL aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
OFFSET_AGGREGATION "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | sensitivity | Detection sensitivity. For instance with `intervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `intervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | modelOffsets | A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | intervalsMethod | Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | PERCENTAGE | | modelAggregation | The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | mean | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-offset-aggregation-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-offset-aggregation-percentile-dx ## Description Detect an anomaly if the metric is too different from a naive baseline. The naive baseline is a [naive model](https://otexts.com/fpp2/simple-methods.html) configured with `modelOffsets` and `modelAggregation`. For instance the default baseline is the value of last week (`modelOffsets=[P7D]`, `modelAggregation=MEAN`). The difference can be PERCENTAGE or ABSOLUTE, it is set with the `intervalsMethod` property. The threshold is set with `sensitivity`. SQL aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
OFFSET_AGGREGATION "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | sensitivity | Detection sensitivity. For instance with `intervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `intervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | modelOffsets | A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | intervalsMethod | Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | PERCENTAGE | | modelAggregation | The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | mean | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-offset-aggregation-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-offset-aggregation-percentile-query-dx ## Description Detect an anomaly if the metric is too different from a naive baseline. The naive baseline is a [naive model](https://otexts.com/fpp2/simple-methods.html) configured with `modelOffsets` and `modelAggregation`. For instance the default baseline is the value of last week (`modelOffsets=[P7D]`, `modelAggregation=MEAN`). The difference can be PERCENTAGE or ABSOLUTE, it is set with the `intervalsMethod` property. The threshold is set with `sensitivity`. SQL aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
OFFSET_AGGREGATION "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | sensitivity | Detection sensitivity. For instance with `intervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `intervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | modelOffsets | A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | intervalsMethod | Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | PERCENTAGE | | modelAggregation | The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | mean | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-offset-aggregation-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-offset-aggregation-query-dx ## Description Detect an anomaly if the metric is too different from a naive baseline. The naive baseline is a [naive model](https://otexts.com/fpp2/simple-methods.html) configured with `modelOffsets` and `modelAggregation`. For instance the default baseline is the value of last week (`modelOffsets=[P7D]`, `modelAggregation=MEAN`). The difference can be PERCENTAGE or ABSOLUTE, it is set with the `intervalsMethod` property. The threshold is set with `sensitivity`. SQL aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
OFFSET_AGGREGATION "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | metricMinimumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be smaller than the given value. For instance, set it to 0 if your metric cannot have a negative value. | - | | metricMaximumValue | If set, the predicted value of the detector and the lower/upper bounds cannot be bigger than the given value. For instance, set it to 100 if your metric cannot be bigger than 100. | - | | sensitivity | Detection sensitivity. For instance with `intervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `intervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | modelOffsets | A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | intervalsMethod | Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | PERCENTAGE | | modelAggregation | The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | mean | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-percentage-rule Source: https://docs.startree.ai/thirdeye/templates/startree-percentage-rule ## Description Detect an anomaly if the percentage difference between the metric and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `percentageChange` property. Aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD eventProcessor>"eventProcessor
EVENTS "]--->root thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
PERCENTAGE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager root>"root
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | percentageChange | Percentage threshold. If the percentage change is above this threshold, detect an anomaly. In ratio. Eg 0.1 means 10 percent. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/percentage). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-percentage-rule-dx Source: https://docs.startree.ai/thirdeye/templates/startree-percentage-rule-dx ## Description Detect an anomaly if the percentage difference between the metric and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `percentageChange` property. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
PERCENTAGE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager childRoot>"childRoot
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | percentageChange | Percentage threshold. If the percentage change is above this threshold, detect an anomaly. In ratio. Eg 0.1 means 10 percent. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/percentage). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-percentage-rule-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-percentage-rule-percentile ## Description Detect an anomaly if the percentage difference between the metric and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `percentageChange` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... ## Flowchart ```mermaid theme={null} graph TD eventProcessor>"eventProcessor
EVENTS "]--->root thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
PERCENTAGE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager root>"root
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | percentageChange | Percentage threshold. If the percentage change is above this threshold, detect an anomaly. In ratio. Eg 0.1 means 10 percent. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/percentage). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-percentage-rule-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-percentage-rule-percentile-dx ## Description Detect an anomaly if the percentage difference between the metric and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `percentageChange` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
PERCENTAGE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager childRoot>"childRoot
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | percentageChange | Percentage threshold. If the percentage change is above this threshold, detect an anomaly. In ratio. Eg 0.1 means 10 percent. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/percentage). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-percentage-rule-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-percentage-rule-percentile-query-dx ## Description Detect an anomaly if the percentage difference between the metric and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `percentageChange` property. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
PERCENTAGE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager childRoot>"childRoot
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | percentageChange | Percentage threshold. If the percentage change is above this threshold, detect an anomaly. In ratio. Eg 0.1 means 10 percent. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/percentage). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-percentage-rule-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-percentage-rule-query-dx ## Description Detect an anomaly if the percentage difference between the metric and the baseline is greater than the threshold. The baseline is a previous value, set with the `baselineOffset` property. The threshold is set with the `percentageChange` property. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot thresholdProcessor>"thresholdProcessor
THRESHOLD "]--->eventProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->thresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
PERCENTAGE_CHANGE "]]--->coldStartProcessor baselineMissingDataManager["baselineMissingDataManager
TimeIndexFiller "]--->anomalyDetector baselineDataFetcher[("baselineDataFetcher
DataFetcher ")]--->baselineMissingDataManager childRoot>"childRoot
ANOMALY_MERGER "] currentMissingDataManager["currentMissingDataManager
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->currentMissingDataManager eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | baselineOffset | Offset in ISO-8601 format. Example: `P7D` means "use the value of the last week as a baseline". | - | | percentageChange | Percentage threshold. If the percentage change is above this threshold, detect an anomaly. In ratio. Eg 0.1 means 10 percent. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/anomaly-detector/percentage). | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-remote-http Source: https://docs.startree.ai/thirdeye/templates/startree-remote-http ## Description Remote HTTP Detector Template ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
REMOTE_HTTP "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | P7D | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### OTHER | name | description | default value | | - | - | - | | url | - | - | | eventSqlFilter | - | | | eventTypes | - | \['\_\_NO\_EVENTS'] | | eventLookaround | - | P1D | # startree-remote-http-dx Source: https://docs.startree.ai/thirdeye/templates/startree-remote-http-dx ## Description Remote HTTP Detector Template Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
REMOTE_HTTP "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | P7D | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | ### OTHER | name | description | default value | | - | - | - | | url | - | - | | eventSqlFilter | - | | | eventTypes | - | \['\_\_NO\_EVENTS'] | | eventLookaround | - | P1D | # startree-remote-http-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-remote-http-percentile ## Description Remote HTTP Detector Template ## Flowchart ```mermaid theme={null} graph TD impactProcessor>"impactProcessor
IMPACT "]--->root eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
REMOTE_HTTP "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | P7D | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### OTHER | name | description | default value | | - | - | - | | url | - | - | | eventSqlFilter | - | | | eventTypes | - | \['\_\_NO\_EVENTS'] | | eventLookaround | - | P1D | # startree-remote-http-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-remote-http-percentile-dx ## Description Remote HTTP Detector Template Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
REMOTE_HTTP "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | P7D | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | ### OTHER | name | description | default value | | - | - | - | | url | - | - | | eventSqlFilter | - | | | eventTypes | - | \['\_\_NO\_EVENTS'] | | eventLookaround | - | P1D | # startree-remote-http-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-remote-http-percentile-query-dx ## Description Remote HTTP Detector Template Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
REMOTE_HTTP "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | P7D | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | ### OTHER | name | description | default value | | - | - | - | | url | - | - | | eventSqlFilter | - | | | eventTypes | - | \['\_\_NO\_EVENTS'] | | eventLookaround | - | P1D | # startree-remote-http-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-remote-http-query-dx ## Description Remote HTTP Detector Template Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] impactProcessor>"impactProcessor
IMPACT "]--->childRoot eventProcessor>"eventProcessor
EVENTS "]--->impactProcessor offsetAggregationProcessor>"offsetAggregationProcessor
OFFSET_AGGREGATION "]--->eventProcessor guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->offsetAggregationProcessor metricThresholdProcessor>"metricThresholdProcessor
THRESHOLD "]--->guardrailThresholdProcessor timeOfWeekProcessor>"timeOfWeekProcessor
TIME_OF_WEEK "]--->metricThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->timeOfWeekProcessor anomalyDetector[["anomalyDetector
REMOTE_HTTP "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataWithHistoryFetcher[("currentDataWithHistoryFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->anomalyDetector childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | lookback | Historical time period to use to train the model. In ISO-8601 format. Example: `P21D`. | - | | pattern | Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | seasonalityPeriod | Biggest seasonality period to learn. In ISO-8601 format. Example: `P7D`. | P7D | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Threshold | name | description | default value | | - | - | - | | thresholdFilterMin | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold minimum. If `-1`, no minimum threshold is applied. | -1 | | thresholdFilterMax | Used to ignore anomalies that don't meet the thresholdFilter min and max. Example: set `thresholdFilterMin = 10` to ignore anomalies when the metric is smaller than 10. Can help ignore anomalies happening in low data regimes. Filter threshold maximum. If `-1`, no maximum threshold is applied. | -1 | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Simple baseline | name | description | default value | | - | - | - | | offsetBaselineFilterPattern | Used to ignore anomalies that are not detected as anomalies by a simple model. Whether to detect an anomaly if it's a drop, a spike or any of the two. | UP\_OR\_DOWN | | offsetBaselineFilterSensitivity | Used to ignore anomalies that are not detected as anomalies by a simple model. Detection sensitivity. For instance with `offsetBaselineFilterIntervalsMethod=PERCENTAGE`, set 50 for a 50% percentage change threshold. With `offsetBaselineFilterIntervalsMethod=ABSOLUTE`, set 200 for a 200 absolute difference threshold between the metric and the baseline. | -1 | | offsetBaselineFilterIntervalsMethod | Used to ignore anomalies that are not detected as anomalies by a simple model. Method to compute intervals. `PERCENTAGE` or `ABSOLUTE`. | ABSOLUTE | | offsetBaselineFilterModelOffsets | Used to ignore anomalies that are not detected as anomalies by a simple model. A list of offsets in ISO-8601 format to use as baseline. Eg `[P7D, P14D]` will compare the current value to the aggregation of the values of the 2 previous weeks. | \['P7D'] | | offsetBaselineFilterModelAggregation | Used to ignore anomalies that are not detected as anomalies by a simple model. The aggregation function to use to combine historical values. In `MEDIAN`, `AVERAGE`, `MIN`, `MAX` and any of `PCTXXXXX` eg `PCT05` (5th percentile), `PCT95`, `PCT999` (99.9th percentile). | MEDIAN | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | #### Impact | name | description | default value | | - | - | - | | impactThreshold | Used to ignore anomalies that don't meet the impact threshold. Impact filter threshold. | -1 | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | ### RCA | name | description | default value | | - | - | - | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | ### OTHER | name | description | default value | | - | - | - | | url | - | - | | eventSqlFilter | - | | | eventTypes | - | \['\_\_NO\_EVENTS'] | | eventLookaround | - | P1D | # startree-threshold Source: https://docs.startree.ai/thirdeye/templates/startree-threshold ## Description Detect an anomaly if the metric is above a maximum threshold or below a minimum threshold. Aggregation function with 1 operand: SUM, MAX,etc... ## Flowchart ```mermaid theme={null} graph TD eventProcessor>"eventProcessor
EVENTS "]--->root guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-threshold-dx Source: https://docs.startree.ai/thirdeye/templates/startree-threshold-dx ## Description Detect an anomaly if the metric is above a maximum threshold or below a minimum threshold. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-threshold-percentile Source: https://docs.startree.ai/thirdeye/templates/startree-threshold-percentile ## Description Detect an anomaly if the metric is above a maximum threshold or below a minimum threshold. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc. ## Flowchart ```mermaid theme={null} graph TD eventProcessor>"eventProcessor
EVENTS "]--->root guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller root>"root
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor root ---> END((END)) ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | # startree-threshold-percentile-dx Source: https://docs.startree.ai/thirdeye/templates/startree-threshold-percentile-dx ## Description Detect an anomaly if the metric is above a maximum threshold or below a minimum threshold. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumerationItems` property to configure the different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumerationItems | Array of enumerations. The detection pipeline will run for each enumeration. The format is the following: `json[

{

"name": "US country",

"description": "slice for US only",

"params": {

"queryFilters": " AND country='US'"

}

}

,... # other enumerations

,]

` To make a property configurable for each enumeration, ensure it is set to the special value: `[DOLLAR]{myProperty}` - replace \[DOLLAR] by the dollar character). In the example above the `queryFilter` property must be set to `[DOLLAR]{queryFilters}`. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-threshold-percentile-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-threshold-percentile-query-dx ## Description Detect an anomaly if the metric is above a maximum threshold or below a minimum threshold. Aggregation function with 2 operands: PERCENTILETDIGEST, DISTINCTCOUNTHLL,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | | aggregationParameter | The second argument of the aggregationFunction. Example: for `PERCENTILETDIGEST`: `95`. | - | ### DETECTION | name | description | default value | | - | - | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # startree-threshold-query-dx Source: https://docs.startree.ai/thirdeye/templates/startree-threshold-query-dx ## Description Detect an anomaly if the metric is above a maximum threshold or below a minimum threshold. Aggregation function with 1 operand: SUM, MAX,etc... Use the `enumeratorQuery` property to feed in a query that outputs different dimensions to explore. ## Flowchart ```mermaid theme={null} graph TD subgraph dimension-exploration [Dimension exploration. For each enumeration:
] eventProcessor>"eventProcessor
EVENTS "]--->childRoot guardrailThresholdProcessor>"guardrailThresholdProcessor
THRESHOLD "]--->eventProcessor metricThresholdProcessor>"metricThresholdProcessor
TIME_OF_WEEK "]--->guardrailThresholdProcessor coldStartProcessor>"coldStartProcessor
COLD_START "]--->metricThresholdProcessor anomalyDetector[["anomalyDetector
THRESHOLD "]]--->coldStartProcessor missingDataPointsFiller["missingDataPointsFiller
TimeIndexFiller "]--->anomalyDetector currentDataFetcher[("currentDataFetcher
DataFetcher ")]--->missingDataPointsFiller childRoot>"childRoot
ANOMALY_MERGER "] eventsDataFetcher[("eventsDataFetcher
EventFetcher ")]--->eventProcessor childRoot ---> END((END)) end ``` ## Parameters ### DATA | name | description | default value | | - | - | - | | aggregationColumn | The column to aggregate. Can be a derived metric. | - | | aggregationFunction | The aggregation function to apply on the aggregationColumn. Example: `AVG`. | - | | dataSource | The Pinot datasource to use. | - | | dataset | The dataset to query. | - | | monitoringGranularity | The period of aggregation of the timeseries. In ISO-8601 format. Example: `PT1H`. | - | | timezone | Timezone used to group by time. In [TZ-identifier](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) format. For instance, `UTC` or `US/Pacific`. | UTC | | timeColumn | TimeColumn used to group by time. If set to AUTO (the default value), the Pinot primary time column is used. | AUTO | | timeColumnFormat | Required if timeColumn is not AUTO. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/data-fetcher#timeformat-strings). | | | completenessDelay | The time for your data to be considered complete and ready for anomaly detection. In ISO-8601 format. Example: `PT2H`. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#completenessdelay). | P0D | | queryFilters | Filters to apply when fetching data. Prefix with `AND`. Example: `AND country='US'` | \$ | | queryLimit | Maximum number of timeseries point to fetch. | 100000001 | ### DETECTION | name | description | default value | | - | - | - | | max | If the current value is above this threshold, detect an anomaly. | - | | min | If the current value is below this threshold, detect an anomaly. | - | ### FILTER #### Time of week | name | description | default value | | - | - | - | | daysOfWeek | Used to ignore anomalies that happen at specific time periods. A list of days. Anomalies happening on these days are ignored if timeOfWeekIgnore is true. Example: `["MONDAY", "SUNDAY"]`. | \[] | | hoursOfDay | Used to ignore anomalies that happen at specific time periods. A list of hours. Anomalies happening on these hours are ignored. Example: `[0,1,2,23]` | \[] | | dayHoursOfWeek | Used to ignore anomalies that happen at specific time periods. A mapping of `{DAY: [hours]}`. Anomalies happening on these timeframes are ignored if timeOfWeekIgnore is true. Example: `{"FRIDAY": [22, 23], "SATURDAY": [0, 1, 2]}` | | #### Guardrail metric | name | description | default value | | - | - | - | | guardrailMetricMin | Used to ignore anomalies that don't meet the guardrail threshold. Minimum threshold of the guardrail metric. If `-1`, no minimum threshold is applied. | -1 | | guardrailMetricMax | Used to ignore anomalies that don't meet the guardrail threshold. Maximum threshold of guardrailMetric. If `-1`, no maximum threshold is applied. | -1 | | guardrailMetric | Used to ignore anomalies that don't meet the guardrail threshold. Metric to use as a threshold guardrail. Example: `COUNT(*)` and set `guardrailMetricMin = 100` to ignore anomalies detected when there is less than 100 observations in the period. | COUNT(\*) | #### Special events | name | description | default value | | - | - | - | | eventFilterSqlFilter | Used to ignore anomalies that happen during events. Sql filter to apply on the events. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/reference/operators/event-fetcher#sql-filter) | | | eventFilterLookaround | Used to ignore anomalies that happen during events. Offset to apply on startTime and endTime to look around the timeframe. In ISO-8601 format. Example: `P1D`. | P2D | | eventFilterTypes | Used to ignore anomalies that happen during events. List of event types to fetch by. Example: `["HOLIDAY", "DEPLOYMENT"]`. `[]` fetches all events. Use `["__NO_EVENTS"]` to disable. | \['\_\_NO\_EVENTS'] | | eventFilterBeforeEventMargin | Used to ignore anomalies that happen during events. A period in ISO-8601 format that corresponds to a period that is also impacted by the event. Example: if beforeEventMargin is `P1D`, if event happens on `[Dec 24 0:00, Dec 25 0:00[`, the label will be applied to anomalies happening on `[Dec 23 0:00 and Dec 25 0:00[` | P0D | | eventFilterAfterEventMargin | Used to ignore anomalies that happen during events. Same as eventFilterBeforeEventMargin at the end of the event. | P0D | ### POSTPROCESS #### Data mutability | name | description | default value | | - | - | - | | mutabilityPeriod | Use if your data is mutable. ThirdEye will maintain the detection results up to date on the mutable period. For instance, if your last 10 days of data is mutable, set `P10D`. At each cron detection job, the detection results for the last 10 days will be updated. | P0D | | reNotifyPercentageThreshold | For detection replay when data is mutable. If the percentage difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyAbsoluteThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | | reNotifyAbsoluteThreshold | For detection replay when data is mutable. If the absolute difference between an existing anomaly and a new anomaly on the same time frame is above this threshold, renotify. Combined with `reNotifyPercentageThreshold`. Both thresholds must pass to be re-notified. If zero, always renotify. If null or negative, never re-notifies. | -1 | #### Anomaly merger | name | description | default value | | - | - | - | | mergeMaxGap | Maximum gap between 2 anomalies for anomalies to be merged. In ISO-8601 format. Example: `PT2H`. The default behavior is to merge consecutive anomalies only. To disable anomaly merging entirely, set this value to `P0D`. | | | mergeMaxDuration | Maximum duration of an anomaly merger. At merge time, if an anomaly merger would get bigger than this limit, the anomalies are not merged. In ISO-8601 format. Example: `P7D`. | | ### RCA | name | description | default value | | - | - | - | | rcaAggregationFunction | The aggregation function to use for RCA. If the detection metric name is known to ThirdEye, this parameter is optional. | | | rcaIncludedDimensions | List of the dimensions (columns in the dataset) to use in RCA drill-downs. If not set or empty, all dimensions of the table are used. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#dimensions). | \[] | | rcaExcludedDimensions | List of dimensions (columns in the dataset) to ignore in RCA drill-downs. If not set or empty, all dimensions of the table are used. rcaExcludedDimensions and rcaIncludedDimensions cannot be used at the same time. | \[] | | rcaEventTypes | A list of type to filter on for RCA. Only events that match such types will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#types). | \[] | | rcaEventSqlFilter | A Sql filter for RCA events. Only events that match the filter will be shown in the RCA related events tab. [Learn more](https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/alert-configuration#sqlfilter). | | ### DIMENSION\_EXPLORATION | name | description | default value | | - | - | - | | enumeratorQuery | This is a SQL query that will run on the data source and build enumeration items from that queryExample: "SELECT DISTINCT country, device from pageviews LIMIT 100". In this case, the enumerator will generate one enumeration item for each country/device combination. | - | | enumerationItemIdKeys | List of keys to use to identify the enumeration. The format is the following: `json[

"queryFilters"

]

` The keys must be present in the `params` object of each enumeration. The keys will be used to generate the dimension exploration id. The id will be used to identify the enumeration in the detection pipeline. | \['queryFilters'] | # Frequently Asked Questions Source: https://docs.startree.ai/thirdeye/troubleshooting/faq > **My datasource historical data can be updated. What is the impact for ThirdEye?** > > If your historical data can be updated, ThirdEye will run detection pipelines on data that can change later. > > This has 2 impacts: > > * any historical anomalies created by a detection pipeline run can be invalid after a data update. > * true anomalies caused by a data update can be missed. > > To mitigate this, you should think in terms of [watermark](https://www.oreilly.com/library/view/streaming-systems/9781491983867/ch03.html): > > * what is the expected time for my dataset to be relevant for alerts? > * what is the expected time for my dataset to be immutable? > > **Example:** your data is supposed to arrive in 3 hours, but the data can frequently change within the last 5 days. You could set up: > > 1. one alert pipeline that runs with a delay of 3 hours.\\ > > This alert will be useful to quickly catch problems. It may have false positives. > 2. one pipeline that runs with a delay of 5 days.\\ > > This alert will not be impacted by the data updates. An anomaly raised by this alert is important. > > If data updates don't happen often, you can also reset the alert and replay it on a timeframe. This can be done with the API. > **When I create an alert, do I need to wait for the first detection job run to see anomalies?** > > No need. When an alert is created, ThirdEye will replay the alert on past data and create anomalies. > > Depending on the historical data volume and alert monitoring configuration, this could take between 1 second and 30 minutes. > **What is the timezone used by ThirdEye?** > > * The UI uses the timezone of your browser. > * The CRON in subscription and alert configurations is in UTC. > * The notifications messages timezone is configurable. If you use ThirdEye as SaaS, reach out to the StarTree team to change the timezone. The alert timezone can be configured in [metadata\$timezone](/thirdeye/alert-configuration#timezone) > **What is the alert lastTimestamp? When a detection cron job is run, what is the detection timeframe?** > > `lastTimestamp` is the detection watermark. It corresponds to the last detection upper bound (excluded) of the alert. > > When an alert runs for the first time, `lastTimestamp` is 0. > > When a detection cron job runs, the detection timeframe is computed as follows: > > * The base timeframe is `[lastTimestamp, cronScheduleTime]`.\ > `cronScheduleTime` is the exact cron time. For instance, if you schedule the cron at 3am, cronScheduleTime = 3:00, even if the processing runtime is slightly different, like 3:01. > * The timeframe is corrected with [completenessDelay](/thirdeye/alert-configuration#completenessdelay): `[lastTimestamp, cronScheduleTime-completenessDelay]` > * The timeframe is corrected with [monitoringGranularity](/thirdeye/alert-configuration#granularity): `[lastTimestamp, floorByGranularity(cronScheduleTime-completenessDelay)]` > The goal is to avoid reading buckets that are incomplete. > > `completenessDelay` and `granularity` are optional but highly recommended. ## Troubleshooting Resources You can [ask on the StartTree Community Slack](https://inviter.co/startree-community), or [create issues on GitHub](https://github.com/startreedata/thirdeye/issues). # ThirdEye Observability and Monitoring Source: https://docs.startree.ai/thirdeye/troubleshooting/thirdeye-observability-and-monitoring To check the health of ThirdEye, set up the following metrics to monitor: | Category | Metrics to be monitored | | - | - | | Environment | Environment availability | | Scheduler | Task stats | | Anomalies & alerts | Total number of anomalies created, number of anomalies created - grouped by metrics/datasets, number of notifications sent, number active alerts | | Data | Persistent data and storage, queries per second (using Pinot broker) (executed from alerts), connectivity with Pinot | | Availability | TP95, API availability and latency, Swagger API is up and working, TE-Auth server connectivity check | | Usage | DAU and MAU, API calls | # Scaling Workers Source: https://docs.startree.ai/thirdeye/worker-scaling ThirdEye is comprised of three components: 1. Coordinator 2. Scheduler 3. Worker The coordinator receives REST API requests, can perform CRUD operations on ThirdEye entities, simulate detection pipeline, and run interactive root cause analysis (RCA). The scheduler runs cron tasks and creates task objects. The worker is responsible for doing the heavy lifting of running different kinds of tasks. As we introduce more and more alerts, the task count grows proportionally. ThirdEye provides the flexibility to scale the workers horizontally as well as vertically to support the growing tasks. ## Vertical Scaling The ThirdEye worker has an internal thread pool which it uses to run the tasks in parallel. We can control the parallelism with the help of `maxParallelTasks` configuration. ### When to do this? This should be configured based on the frequent load expected on the worker, for example if there are 10 alerts which have detection scheduled for every minute then it makes sense to have 10 tasks run in parallel. ### Configuration For helm installation: ```yaml theme={null} ... worker: config: maxParallelTasks: 10 # default is 5 ``` *values.yaml* For bare metal installation: ```yaml theme={null} ... taskDriver: maxParallelTasks: 10 ``` *server.yaml* ## Horizontal Scaling ThirdEye workers are scaled by replicating them. ### When to do this? This is a more flexible scaling as it allows us to scale up/down the number of workers based on sudden temporary changes in task rates, for example, if there are 5 alerts that run detection after every minute, but we also have another 10 alerts which just trigger once a day at 2 am, in this case it makes sense to spin up another worker before 2 am and take it down once the tasks are served. > [!NOTE] > > HorizontalPodAutoscaler (HPA) is not yet supported in official ThirdEye Helm charts as the trigger to scale can vary based on the use case. ### Configuration For Helm installation, provide the number of replicas needed. ```yaml theme={null} ... worker: replicas: 2 ... config: randomWorkerIdEnabled: true ``` *values.yaml* For bare metal installation, run multiple workers on different ports. ```yaml theme={null} ... taskDriver: enabled: true randomWorkerIdEnabled: true ``` *server.yaml* ## FAQs > **What is the max number of parallel tasks can be run in case of vertical scaling?** > > Each thread will require cpu time and heap memory so long-running and high memory demanding tasks will prefer less parallel threads while quick running and low memory demanding tasks can afford to have more parallelism. The default is 5. > **What is the max number of workers one can set in case of horizontal scaling?** > > There is no limit as such from the ThirdEye side, but basically it will be (resources available)/(resources required per worker). > **How about consistency and reliability? In which case (vertical or horizontal scaling) does ThirdEye perform well?** > > This is a scenario-based behavior as to which scaling suits best. Neither of the two methods will put ThirdEye in inconsistent state. As for reliability, as long as enough resources are available for a given setup, the system won't be affected by scaling.