Ingest a file
Creates a segment using given file and pushes it to Pinot. All steps happen on the controller. This API is NOT meant for production environments/large input files. Example usage (query params need encoding):
curl -X POST -F file=@data.json -H "Content-Type: multipart/form-data" "http://localhost:9000/ingestFromFile?tableNameWithType=foo_OFFLINE&
batchConfigMapStr={
"inputFormat":"csv",
"recordReader.prop.delimiter":"|"
}"
POST
/
ingestFromFile
Copy
Ask AI
curl --request POST \
--url https://dev.startree.ai/ingestFromFile \
--header 'Authorization: <api-key>' \
--header 'Content-Type: multipart/form-data' \
--form 'contentDisposition={
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
}' \
--form 'entity={}' \
--form 'headers={}' \
--form 'mediaType={
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
}' \
--form 'messageBodyWorkers={}' \
--form 'parent={
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"bodyParts": [
{
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"parameterizedHeaders": {}
}
],
"parameterizedHeaders": {}
}' \
--form 'providers={}' \
--form 'bodyParts=[
{
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"parameterizedHeaders": {}
}
]' \
--form 'fields={}' \
--form 'parameterizedHeaders={}'Assistant
Responses are generated using AI and may contain mistakes.
Authorizations
The format of the key is "Basic <token>" or "Bearer <token>"
Query Parameters
Name of the table to upload the file to
Batch config Map as json string. Must pass inputFormat, and optionally record reader properties. e.g. {"inputFormat":"json"}
Body
multipart/form-data
Was this page helpful?
Copy
Ask AI
curl --request POST \
--url https://dev.startree.ai/ingestFromFile \
--header 'Authorization: <api-key>' \
--header 'Content-Type: multipart/form-data' \
--form 'contentDisposition={
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
}' \
--form 'entity={}' \
--form 'headers={}' \
--form 'mediaType={
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
}' \
--form 'messageBodyWorkers={}' \
--form 'parent={
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"bodyParts": [
{
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"parameterizedHeaders": {}
}
],
"parameterizedHeaders": {}
}' \
--form 'providers={}' \
--form 'bodyParts=[
{
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"parameterizedHeaders": {}
}
]' \
--form 'fields={}' \
--form 'parameterizedHeaders={}'API Documentation
Broker APIs
Table
- POSTpreview
- GETLists all tables in cluster
- DELDelete tables partitions
- POSTIngest a file
- POSTIngest from the given URI
- GETGet the instance partitions
- PUTCreate/update the instance partitions
- DELRemove the instance partitions
- POSTReplace an instance in the instance partitions
- POSTAssign server instances to a table
- POSTPause consumption of a realtime table
- POSTResume consumption of a realtime table
- GETGet status for a submitted force commit operation
- GETReturns state of pauseless table
- GETReturn pause status of a realtime table
- POSTForce commit the current consuming segments
- GETReturns state of consuming segments
- GETList table instances
- GETList tables to live brokers mappings
- GETList the brokers serving a table
- PUTRecommend config
- GETLists the table configs
- PUTUpdates table config for a table
- DELDeletes a table
- POSTValidate table config for a table
- POSTRebalances a table (reassign instances and segments for a table)
- DELCancel all rebalance jobs for the given table, and noop if no rebalance is running
- GETGets detailed stats of a rebalance operation
- GETtable status
- GETGet current table state
- PUTEnable/disable a table
- GETtable stats
- GETLists all tables in cluster
- POSTAdds a table
- GETGet the aggregate validDocIds metadata of all segments for a table
- GETGet the aggregate metadata of all segments for a table
- GETGet the aggregate index details of all segments for a table
- GETGet list of controller jobs for this table
- POSTSet hybrid table query time boundary based on offline segments' metadata
- DELDelete hybrid table query time boundary
- POSTRebuild broker resource for table
- POSTValidate the TableConfigs
- GETLists all TableConfigs in cluster
- POSTAdd the TableConfigs using the tableConfigsStr json
- GETGet the TableConfigs for a given raw tableName
- PUTUpdate the TableConfigs provided by the tableConfigsStr json
- DELDelete the TableConfigs
- GETRead table sizes
- GETGet segment names to segment status map
- GETGet table ideal state
- GETGet table external view
ClusterHealth
- GETGet cached cluster health details for all pinot entities
- PUTEnable / disable the periodic cluster health check task. Note that this setting isn't persisted across controller restarts and /cluster/configs should be used to disable it permanently
- GETGet all available cluster health checks and their details
- GETGet cluster properties if deployed in AZ aware mode
Connection
ConsistentPush
DedupSnapshot
RateLimiter
AtomicIngestion
Restream
UpsertSnapshot
- POSTAdd metadata of a new upsert snapshot
- DELDelete upsert snapshot metadata
- GETGet all upsert snapshot names for the table
- GETGet latest upsert snapshot names for all partitions
- POSTAdd metadata location of a new upsert snapshot
- DELDelete upsert snapshot and its metadata
- GETGet upsert snapshot metadata by the snapshot name
Cluster
- GETGet debug information for table.
- GETGet debug information for segment.
- GETGet the configuration for Groovy Static analysis
- POSTUpdate Groovy static analysis configuration
- GETGet cluster Info
- GETList cluster configurations
- POSTUpdate cluster configuration
- DELDelete cluster configuration
- GETGet the default configuration for Groovy Static analysis
User
Application
Broker
- GETList tenants to brokers mappings
- GETList tables to brokers mappings
- GETList tenants to brokers mappings
- GETList tables to brokers mappings
- POSTEnable/disable the query rate limiting for a broker instance
- GETList tenants and tables to brokers mappings
- GETList brokers for a given tenant
- GETList brokers for a given tenant
- GETList brokers for a given table
- GETList brokers for a given table
- GETList tenants and tables to brokers mappings
AppConfigs
Health
Logger
PeriodicTask
Database
Instance
- PUTEnable/disable an instance
- POSTEnable/disable/drop an instance
- POSTCheck if it's safe to update the tags of the given instances. If not list all the reasons.
- GETList all instances
- POSTCreate a new instance
- GETList all live instances
- POSTUpdate the tables served by the specified broker instance in the broker resource
- GETGet instance information
- PUTUpdate the specified instance
- DELDrop an instance
- PUTUpdate the tags of the specified instance
- GETCheck if it's safe to drop the given instances. If not list all the reasons why its not safe.
Query
Schema
Segment
- GETGet the metadata for a segment
- GETList all segments. An optional 'excludeReplacedSegments' parameter is used to get the list of segments which has not yet been replaced (determined by segment lineage entries) and can be queried from the table. The value is false by default.
- DELDelete the list of segments provided in the queryParam else all segments
- DELDelete selected segments. An optional 'excludeReplacedSegments' parameter is used to get the list of segments which has not yet been replaced (determined by segment lineage entries) and can be queried from the table. The value is false by default.
- GETGet a map from segment to CRC of the segment (only apply to OFFLINE table)
- GETGet status for a submitted reload operation
- GETGet the server metadata for all table segments
- GETGet the zookeeper metadata for all table segments
- GETGet storage tiers for the given segment
- GETGet selected segments timestamps
- POSTUpdate the start and end time of the segments based on latest schema
- GETGet storage tier for all segments in the given table
- GETGets the metadata of reload segments check from servers hosting the table
- GETGets a list of segments that are stale from servers hosting the table
- GETDownload a segment
- DELDelete a segment
- POSTReload a segment
- POSTDelete the segments in the JSON array payload
- POSTReload all segments
- GETList segment lineage
- GETGet a map from server to segments hosted by the server
- POSTResets a segment by first disabling it, waiting for external view to stabilize, and finally enabling it again
- POSTResets all segments (when errorSegmentsOnly = false) or segments with Error state (when errorSegmentsOnly = true) of the table, by first disabling them, waiting for external view to stabilize, and finally enabling them
- POSTUpload a segment
- POSTUpload a segment
- POSTReingest a realtime segment
- POSTUpload a batch of segments
- POSTStart to replace segments
- POSTEnd to replace segments
- POSTRevert segments replacement
Task
- GETFetch cron scheduler job keys
- GETGet the task config (a list of child task configs) for the given task
- GETGet task metadata for the given task type and table
- DELDelete task metadata for the given task type and table
- GETList all task types
- GETCount of all tasks for the given task type
- PUTClean up finished tasks (COMPLETED, FAILED) for the given task type
- PUTStop all running/pending tasks (as well as the task queue) for the given task type
- PUTResume all stopped tasks (as well as the task queue) for the given task type
- DELDelete all tasks (as well as the task queue) for the given task type
- GETFetch task generation information for the recent runs of the given task for the given table
- GETFetch cron scheduler information
- GETFetch job details for table tasks
- POSTExecute a task on minion
- GETList all tasks for the given task type
- GETGet the states of all the sub tasks for the given task
- GETGet the configs of specified sub tasks for the given task
- GETGet progress of specified sub tasks for the given task tracked by minion worker in memory
- GETList all tasks for the given task type
- GETFetch information for all the tasks for the given task type
- GETFetch information for all the tasks for the given task type and table
- GETFetch information for the given task name
- GETGet progress of all subtasks with specified state tracked by minion worker in memory
- DELDelete a single task given its task name
- GETFetch count of sub-tasks for each of the tasks for the given task type
- GETGet the task state for the given task
- POSTSchedule tasks and return a map from task type to task name scheduled. If task type is missing, schedules all tasks. If table name is missing, schedules tasks for all tables in the database. If database is missing in headers, uses default.
- GETGet a map from task to task state for the given task type
- GETGet the state (task queue state) for the given task type
- GETGet the task runtime config for the given task
Tenant
- GETGet the instance partitions of a tenant
- PUTUpdate an instance partition for a server type in a tenant
- GETGets detailed stats of a tenant rebalance operation
- GETList all tenants
- PUTUpdate a tenant
- POSTCreate a tenant
- GETList instance for a tenant
- POSTenable/disable a tenant
- DELDelete a tenant
- GETList tables on a server or broker tenant
- GETGet tenant information
- POSTChange tenant state
- POSTRebalances all the tables that are part of the tenant
Zookeeper
Ingest a file
Creates a segment using given file and pushes it to Pinot. All steps happen on the controller. This API is NOT meant for production environments/large input files. Example usage (query params need encoding):
curl -X POST -F file=@data.json -H "Content-Type: multipart/form-data" "http://localhost:9000/ingestFromFile?tableNameWithType=foo_OFFLINE&
batchConfigMapStr={
"inputFormat":"csv",
"recordReader.prop.delimiter":"|"
}"
POST
/
ingestFromFile
Copy
Ask AI
curl --request POST \
--url https://dev.startree.ai/ingestFromFile \
--header 'Authorization: <api-key>' \
--header 'Content-Type: multipart/form-data' \
--form 'contentDisposition={
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
}' \
--form 'entity={}' \
--form 'headers={}' \
--form 'mediaType={
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
}' \
--form 'messageBodyWorkers={}' \
--form 'parent={
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"bodyParts": [
{
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"parameterizedHeaders": {}
}
],
"parameterizedHeaders": {}
}' \
--form 'providers={}' \
--form 'bodyParts=[
{
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"parameterizedHeaders": {}
}
]' \
--form 'fields={}' \
--form 'parameterizedHeaders={}'Assistant
Responses are generated using AI and may contain mistakes.
Authorizations
The format of the key is "Basic <token>" or "Bearer <token>"
Query Parameters
Name of the table to upload the file to
Batch config Map as json string. Must pass inputFormat, and optionally record reader properties. e.g. {"inputFormat":"json"}
Body
multipart/form-data
Was this page helpful?
Copy
Ask AI
curl --request POST \
--url https://dev.startree.ai/ingestFromFile \
--header 'Authorization: <api-key>' \
--header 'Content-Type: multipart/form-data' \
--form 'contentDisposition={
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
}' \
--form 'entity={}' \
--form 'headers={}' \
--form 'mediaType={
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
}' \
--form 'messageBodyWorkers={}' \
--form 'parent={
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"bodyParts": [
{
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"parameterizedHeaders": {}
}
],
"parameterizedHeaders": {}
}' \
--form 'providers={}' \
--form 'bodyParts=[
{
"contentDisposition": {
"type": "<string>",
"parameters": {},
"fileName": "<string>",
"creationDate": "2023-11-07T05:31:56Z",
"modificationDate": "2023-11-07T05:31:56Z",
"readDate": "2023-11-07T05:31:56Z",
"size": 123
},
"entity": {},
"headers": {},
"mediaType": {
"type": "<string>",
"subtype": "<string>",
"parameters": {},
"wildcardType": true,
"wildcardSubtype": true
},
"messageBodyWorkers": {},
"parent": {},
"providers": {},
"parameterizedHeaders": {}
}
]' \
--form 'fields={}' \
--form 'parameterizedHeaders={}'