Overview
Pinot runs a lot of work in the background on your tables — ingesting new files, purging rows, refreshing segments, snapshotting upserts, syncing external catalogs, and more — using its minion task framework. Historically, checking on one of these jobs meant calling the controller’s task REST API directly. Task Observability brings that same information into the Data Portal as a readable, real-time view: a list of every task plan (a tracked run of one task type against one table, potentially spanning several batches) with drill-down into each batch and its subtasks, without leaving the UI.A task plan is the same concept described in Minion Task Orchestration — this page covers how that data shows up in the Data Portal. Not every task type you’ll see here supports full orchestration (automatic batch chaining); task types that don’t are still tracked for progress and history, they just don’t automatically chain their next batch for you.
Requirements
Task Observability is built on the same task-plan machinery as Minion Task Orchestration, so the two share a cluster-level dependency:| Setting | What it actually controls |
|---|---|
controller.startree.task.manager.enableTaskOrchestration (cluster config) | The master switch for all task-plan tracking, orchestrated or not. On by default since StarTree 0.16.0 — if a cluster predates that or has it explicitly turned off, no task plans exist anywhere and Task Observability has nothing to show. |
enableTaskOrchestration (per task type, in the table/task config) | Only controls whether a plan automatically chains its next batch. It is not required for a scheduled task to show up here — a cron-triggered run of a StarTree task is still tracked as a plan (status, progress, batch history) without this flag, it just won’t self-submit its next batch. Ad hoc (manual) runs are the exception: they still need this flag set to produce a trackable plan at all. |
In short: cluster-level orchestration has to be on for anything to appear in Task Observability, but you don’t need to opt every scheduled task into full orchestration just to watch it — cron-driven runs are tracked automatically. Automatic batch chaining stays opt-in per task type, and today only supported task types benefit from it.
Where to find it
Task Observability has two entry points:- Tasks — a top-level nav item (next to Tables and Spaces) showing task plans across every table you have access to.
- Tasks tab — on a table’s details page, showing the complete task history for that one table.
The global Tasks list is a live/recent view, not a full historical log: it shows every plan that’s currently active, plus any completed plan your browser has observed recently. If you need a complete, dependable history for a specific table, use that table’s Tasks tab instead — it’s backed by a per-table query rather than what your browser happened to see.
Reading the task list
Both the global list and the per-table tab show one row per task plan, refreshing automatically (roughly every 20 seconds) with a manual refresh option:| Column | What it shows |
|---|---|
| Task Plan ID | Opens the plan’s detail page. |
| Table | The table the plan runs against (hidden on the per-table tab, since it’s implied). |
| Task Type | Which minion task this plan is running, e.g. File Ingestion, Segment Purge, Segment Refresh. |
| Status | Active, Completed, Cancelled, Aborting, Failed, or Unknown. |
| Progress | A completion bar with a tooltip showing percent complete and units processed. |
| Created At / Completed At | When the plan started and (if finished) ended. |
| Trigger | Manual (an ad hoc run) or Automatic (a scheduled/cron run). |
| Note | A human-readable status message, when one is available. |
Task plan detail
Opening a plan shows its full detail page:- Refresh and Copy link (a shareable URL to this exact plan).
- Schedule Now — appears only when the plan has pending work but isn’t otherwise going to run again on its own (no active schedule or orchestration in progress). It nudges the task type to pick up the remaining work immediately.
- Abort — enabled only while the plan is Active. You’re asked to type “abort” to confirm. Aborting stops the plan from generating any further batches; a batch that’s already running is left to finish rather than being killed outright.
- Tasks — every batch submitted so far for this plan, each with its own state, progress, and submission time.
- Logs — a progress view for the plan’s execution, with a time-range selector.
- Properties — the plan’s configuration, as a flat, copyable key/value list.
- Custom — task-type-specific statistics reported by the running task, also copyable.
Drilling into an individual run
From a plan’s Tasks tab, opening one submitted batch shows Pinot’s actual per-task subtask table: each subtask’s ID, state (waiting, running, completed, error, dropped, timed out, aborted), start/finish time, and duration. Opening a subtask goes one level deeper, into its progress log:- A search box and multi-select filters for log level and stage (for example, narrowing down to just the segment-upload stage).
- A running count of how many lines are shown out of the total, newest first.
- A metadata panel with the subtask’s configuration in plain terms — task type, table, sequence ID, size settings, and the input files it’s processing.
Retention of task execution details
Once a task finishes, Pinot keeps its detailed execution metadata — subtask states, progress, and configuration — for a limited window before cleaning it up (by default, on the order of a day for successful runs and a few days for failed ones; your cluster’s operator can tune this). Once that window passes, drilling into an old batch shows a clear Task Execution Data No Longer Available message instead of an empty or broken page. This is a separate thing from the global Tasks list’s own “recently seen” behavior described above — one is how long Pinot keeps a task’s execution details, the other is how long your browser remembers a completed plan is worth showing in the all-tables view. If you need to look back at a plan that finished a while ago, the per-table Tasks tab is the more reliable place to start, but the underlying execution detail can still have aged out on the Pinot side either way.Permissions
Viewing and acting on tasks from Data Portal uses the same fine-grained authorization as the rest of Data Portal. Depending on what you’re doing, your account needs the underlying Pinot actions — running a task now requiresExecuteTask, viewing task status/debug info requires GetTask/DebugTask, and changing a table’s schedule requires table-config read/write actions. See RBAC Actions for the full list. If a control is missing or disabled for you, ask your StarTree Cloud admin to confirm your permissions.
Related
- Minion Task Orchestration — the task plan concept and REST API this feature is built on.
- File Ingestion Task, Purge Task, Segment Backfill Task, Segment Import Task, Alter Table Task — task types you can monitor here.
- External Tables: Observability — status, checkpoint, and schedule-now APIs for External Table sync runs specifically, also available from a table’s details page in Data Portal.