@import url('https://fonts.googleapis.com/css2?family=Inter:ital,opsz,wght@0,14..32,100..900;1,14..32,100..900&family=Montserrat:wght@600&family=Poppins:ital,wght@0,100;0,200;0,300;0,400;0,500;0,600;0,700;0,800;0,900;1,100;1,200;1,300;1,400;1,500;1,600;1,700;1,800;1,900&display=swap');
@import url('https://fonts.googleapis.com/css2?family=Fira+Code:wght@300..700&family=Inter:ital,opsz,wght@0,14..32,100..900;1,14..32,100..900&family=Montserrat:wght@600&family=Poppins:ital,wght@0,100;0,200;0,300;0,400;0,500;0,600;0,700;0,800;0,900;1,100;1,200;1,300;1,400;1,500;1,600;1,700;1,800;1,900&display=swap');
.font-inter  h4{
  font-family: 'Inter';
}

.font-fira-code {
  font-family: 'Fira Code';
}

.card-icon a .fill-gray-800, .card-icon-dark a .fill-gray-800{
  height: 48px;
  width: 48px;
}
.card-icon a div h2, .card-icon-dark a div h2{
  color: #002654;
  font-size: 1.25rem;
  font-weight: 500;
  line-height: 130%;
  letter-spacing: 0%;
  font-family: 'Inter';
}
.card-icon a div .top-5, .card-icon-dark a div .top-5{
  display: none;
}
.card-icon-dark .card{
  background-color: #0C2E5C;
}

.card-icon a, .card-icon-dark a{
  height: 100%;
  width: 100%;
  margin-top: 0px;
  margin-bottom: 0px;
}

.gap-cards{
  gap: 1rem;
}


Step 1: In the Data Portal, click Tables and then click Create Table.

​Step 2: Select BigQuery as the Data Source.

​Step 3: Create a New Connection.

Property Descriptions

Step 4: Configure Connection Parameters

Step 5: Test the Connection and Configure Data Ingestion

Step 6: Preview the Data

BigQuery

Connect to Google BigQuery

StarTree Docs

What is Real-Time Analytics?

Overview

Ingest Data

Connect to Any Data Source

Learn how to configure advanced settings for table creation in the Data Portal, including ingestion behavior and time partitioning for Apache Pinot tables.

Additional Configuration

Create Table

Learn about off-heap upserts and how to use them in StarTree

Off-Heap Upserts

Learn about off-heap dedup and how to use them in StarTree

Off-Heap Dedup

Off-Heap Deduplication

Hybrid Tables

Tableau

Connect Tableau to StarTree Cloud

Learn how to connect Superset to StarTree Cloud.

Superset

Connect Superset to StarTree Cloud

Security Manager

Managing Access

API Token Management

Custom Policy Configuration

Actions

Using RBAC API

Using the StarTree Role-Based Access Control (RBAC) API

0.10.1 Release Version 0.10.1: February 2025

Definitions for real-time user-facing analytics, StarTree products, and platforms

Glossary

Cluster Health

StarTree Cloud Cluster Health Dashboard

Getting Started with ThirdEye

ThirdEye Architecture

ThirdEye Entities

ThirdEye Worker Management

DataFetcher

Dimension Exploration

EventFetcher

SqlExecution

TimeIndexFiller

Optimize Performance

Use Pre-Computed Time Columns

Add a Notification System

Subscription Groups and Notification Systems

Alert Tuning

Use Anomaly Filters to Tune Alerts

Root Cause Analysis

How to perform root cause analysis

Manage Holiday Effects

Manage Holiday Effects with StarTree ETS

Using ThirdEye API

How to use the ThirdEye API

Access Control in ThirdEye

Frequently Asked Questions

Observability & Monitoring

ThirdEye Observability and Monitoring

Alert Configuration

Alert Configuration and Execution

Scaling Workers

Dimensions Recommender

ThirdEye Dimensions Recommender

Aggregation Functions

Resources

Download Recipes

Clickstream Analytics Dashboard

Clickstream Analytics Dashboard with StarTree Cloud Free Tier and Streamlit

Learn about StarTree Cloud APIs for managing tables and querying data at scale.

Introduction

Run SQL queries on StarTree Cloud's real-time analytics engine.

Query Data

Simulates a preview of the table given the table config and schema

preview

Lists all tables in cluster

Creates a segment using given file and pushes it to Pinot. 
 All steps happen on the controller. This API is NOT meant for production environments/large input files. 
 Example usage (query params need encoding):
```
curl -X POST -F file=@data.json -H "Content-Type: multipart/form-data" "http://localhost:9000/ingestFromFile?tableNameWithType=foo_OFFLINE&
batchConfigMapStr={
  "inputFormat":"csv",
  "recordReader.prop.delimiter":"|"
}" 
```

Ingest a file

Creates a segment using file at the given URI and pushes it to Pinot. 
 All steps happen on the controller. This API is NOT meant for production environments/large input files. 
Example usage (query params need encoding):
```
curl -X POST "http://localhost:9000/ingestFromURI?tableNameWithType=foo_OFFLINE
&batchConfigMapStr={
  "inputFormat":"json",
  "input.fs.className":"org.apache.pinot.plugin.filesystem.S3PinotFS",
  "input.fs.prop.region":"us-central",
  "input.fs.prop.accessKey":"foo",
  "input.fs.prop.secretKey":"bar"
}
&sourceURIStr=s3://test.bucket/path/to/json/data/data.json"
```

Ingest from the given URI

Get the instance partitions

Create/update the instance partitions

Remove the instance partitions

Replace an instance in the instance partitions

Assign server instances to a table

Pause the consumption of a realtime table

Pause consumption of a realtime table

Resume the consumption for a realtime table. ConsumeFrom parameter indicates from which offsets consumption should resume. Recommended value is 'lastConsumed', which indicates consumption should continue based on the offsets in segment ZK metadata, and in case the offsets are already gone, the first available offsets are picked to minimize the data loss.

Resume consumption of a realtime table

Get status for a submitted force commit operation

Gets the segments that are in error state and segments with COMMITTING status in ZK metadata

Returns state of pauseless table

Return pause status of a realtime table along with list of consuming segments.

Return pause status of a realtime table

Force commit the current segments in consuming state and restart consumption. This should be used after schema/table config changes. Please note that this is an asynchronous operation, and 200 response does not mean it has actually been done already.If specific partitions or consuming segments are provided, only those partitions or consuming segments will be force committed.

Force commit the current consuming segments

Gets the status of consumers from all servers.Note that the partitionToOffsetMap has been deprecated and will be removed in the next release. The info is now embedded within each partition's state as currentOffsetsMap.

Returns state of consuming segments

List table instances

List tables to live brokers mappings based on EV

List tables to live brokers mappings

List live brokers of the given table based on EV

List the brokers serving a table

Recommend config

Lists the table configs

Updates table config for a table

Deletes a table

This API returns the table config that matches the one you get from 'GET /tables/{tableName}'. This allows us to validate table config before apply.

Validate table config for a table

Rebalances a table (reassign instances and segments for a table)

Cancel all rebalance jobs for the given table, and noop if no rebalance is running

Gets detailed stats of a rebalance operation

Provides status of the table including ingestion status

table status

Get current table state

Enable/disable a table

Provides metadata info/stats about the table.

table stats

Adds a table

Get the aggregate validDocIds metadata of all segments for a table

Get the aggregate metadata of all segments for a table

Get the aggregate index details of all segments for a table

Get list of controller jobs for this table

Set hybrid table query time boundary based on offline segments' metadata

Delete hybrid table query time boundary

Rebuild broker resource for table

Validate the TableConfigs

Lists all TableConfigs in cluster

Add the TableConfigs using the tableConfigsStr json

Get the TableConfigs for a given raw tableName

Update the TableConfigs provided by the tableConfigsStr json

Delete the TableConfigs

Get table size details. Table size is the size of untarred segments including replication

Read table sizes

Get segment names to segment status map

Get table ideal state

Get table external view

Get the list of servers to restart in sequence

Get cached cluster health details for all pinot entities

Enable / disable the periodic cluster health check task. Note that this setting isn't persisted across controller restarts and /cluster/configs should be used to disable it permanently

Get all available cluster health checks and their details

Get cluster properties if deployed in AZ aware mode

Browse resources in a connection. Eg list directories/files in a s3 bucket, list tables in a database, list topics in a Kafka cluster etc.

browse

Update consistentPush state

Add metadata location of a new dedup snapshot

Delete dedup snapshot and its metadata

Get dedup snapshot metadata by the snapshot name

Add metadata of a new dedup snapshot

Get all dedup snapshot names for the table

Get latest dedup snapshot names for all partitions

Get perf advisor recommendations given a query

Update rate limiter configs

Mark the start of data ingestion to upload multiple segments

Get Segment to rows factor from customMap of progress metadata

Delete the minion task metadata

Mark the end of data ingestion to upload multiple segments

Restream data from a stream to a new table. Then replace the table

Gets detailed stats of a restream operation

DEBUG ONLY: Process the next stage of the restream workflow.

Cancel a restream operation

Apply tuner(s) to a table

Apply specific tuner to a table

Get progress and estimated completion time for the task

Add metadata of a new upsert snapshot

Delete upsert snapshot metadata

Get all upsert snapshot names for the table

Get latest upsert snapshot names for all partitions

Add metadata location of a new upsert snapshot

Delete upsert snapshot and its metadata

Get upsert snapshot metadata by the snapshot name

Get debug information for table.

Get debug information for segment.

Get the configuration for Groovy static analysis

Get the configuration for Groovy Static analysis

Update Groovy static analysis configuration

Get cluster Info

List cluster configurations

Update cluster configuration

Delete cluster configuration

Get the default configuration for Groovy static analysis

Get the default configuration for Groovy Static analysis

List all uses in cluster

Add a user

Get an user in cluster

Update user config for a user

Delete a user

Get application qps quota

Update application quota

Get all application qps quotas

List tenants to brokers mappings

List tables to brokers mappings

Enable/disable the query rate limiting for a broker instance

Property	Required	Description
`sql.bigquery.projectId`	Yes	The GCP project ID. Find your project ID in the Google Cloud console.
`sql.bigquery.jsonKey`	Yes	The string-encoded Google service account key. Ensure the correct JSON key format. See Google’s documentation for creating a service account key.

Property	Required	Description
`sql.queryTemplate`	Yes	The SQL query template for fetching data from BigQuery.
`sql.timeColumnName`	Yes	The time column used for filtering data while ingesting.
`sql.timeColumnFormat`	Yes	The datetime format of the time column used for filtering.
`sql.startTime`	Yes	The start time for data extraction. This value is dynamically used in the `$START` placeholder of `sql.queryTemplate`. It should match the specified time column format.
`sql.endTime`	Yes	The end time for data extraction. This value is dynamically used in the `$END` placeholder of `sql.queryTemplate`. It should match the specified time column format.

Next Step

Get Started

Ingestion

Query Data

Manage Data

Visualize Data

Manage Security

Release Notes

Reference

Connect to Google BigQuery

Step 1: In the Data Portal, click Tables and then click Create Table.

Step 2: Select BigQuery as the Data Source.

Step 3: Create a New Connection.

Step 4: Configure Connection Parameters

Property Descriptions

Step 5: Test the Connection and Configure Data Ingestion

Property Descriptions

Step 6: Preview the Data

Get Started

Ingestion

Query Data

Manage Data

Visualize Data

Manage Security

Release Notes

Reference

​Step 1: In the Data Portal, click Tables and then click Create Table.

​​Step 2: Select BigQuery as the Data Source.

​​Step 3: Create a New Connection.

​Step 4: Configure Connection Parameters

​Property Descriptions

​Step 5: Test the Connection and Configure Data Ingestion

​Property Descriptions

​Step 6: Preview the Data

Next Step

Step 1: In the Data Portal, click Tables and then click Create Table.

Step 2: Select BigQuery as the Data Source.

Step 3: Create a New Connection.

Step 4: Configure Connection Parameters

Property Descriptions

Step 5: Test the Connection and Configure Data Ingestion

Property Descriptions

Step 6: Preview the Data