Skip to main content

​
Overview

Pinot runs a lot of work in the background on your tables — ingesting new files, purging rows, refreshing segments, snapshotting upserts, syncing external catalogs, and more — using its minion task framework. Historically, checking on one of these jobs meant calling the controller’s task REST API directly. Task Observability brings that same information into the Data Portal as a readable, real-time view: a list of every task plan (a tracked run of one task type against one table, potentially spanning several batches) with drill-down into each batch and its subtasks, without leaving the UI.
A task plan is the same concept described in Minion Task Orchestration — this page covers how that data shows up in the Data Portal. Not every task type you’ll see here supports full orchestration (automatic batch chaining); task types that don’t are still tracked for progress and history, they just don’t automatically chain their next batch for you.

​
Requirements

Task Observability is built on the same task-plan machinery as Minion Task Orchestration, so the two share a cluster-level dependency:
In short: cluster-level orchestration has to be on for anything to appear in Task Observability, but you don’t need to opt every scheduled task into full orchestration just to watch it — cron-driven runs are tracked automatically. Automatic batch chaining stays opt-in per task type, and today only supported task types benefit from it.

​
Where to find it

Task Observability has two entry points:
  • Tasks — a top-level nav item in the Monitoring group, showing task plans across every table you have access to.
  • Monitoring tab — on a table’s details page, showing the task history for that one table.
If you don’t see Tasks in the navigation, the feature isn’t turned on for your environment yet — contact your StarTree account team.
The global Tasks list is a live/recent view, not a full historical log: it shows every plan that’s currently active, plus completed plans this browser has previously seen (remembered per workspace in browser local storage, and shown for as long as the controller still has the plan — Pinot keeps roughly the newest 10 completed plans per table and task type). If you need a dependable history for a specific table, use that table’s Monitoring tab instead — it’s backed by a per-table query rather than what your browser happened to see.

​
Reading the task list

Both the global list and the per-table tab show one row per task plan, refreshing automatically (roughly every 20 seconds) with a manual refresh option: Additional columns (input unit, input units to process, in progress, and processed) are hidden by default but available from the column picker. Every column except the progress bar can be filtered independently. The global list also has a Create task plan action — a short wizard for starting a new orchestrated run (currently File Ingestion and Segment Purge) by picking the task type, the table, and the task configuration.

​
Task plan detail

Opening a plan shows its full detail page, which refreshes on its own more aggressively than the lists (roughly every 10 seconds while the plan is running):
  • Refresh and Copy link (a shareable URL to this exact plan).
  • Schedule Now — appears only for a non-orchestrated plan that still has work left and nothing currently in flight. It nudges the task type to pick up the remaining work immediately.
  • Abort — enabled only while the plan is Active. You’re asked to type “abort” to confirm. Aborting stops the plan from generating any further batches; a batch that’s already running is left to finish rather than being killed outright.
An overview strip at the top shows the plan’s progress donut alongside its key execution charts. Below it, the page is organized into tabs:
  • Logs (the default tab) — the plan’s execution charts, with a time-range selector. For a finished plan, the window freezes to the run’s start and end.
  • Tasks — every batch submitted so far for this plan, each with its own status, progress, sequence, and submission time (newest first).
  • Properties — the plan’s configuration, as a flat, copyable key/value list.
  • Custom — task-type-specific statistics reported by the running task, also copyable.
A collapsible sidebar shows the plan’s metadata (ID, table, task type, status, subtask counts, timestamps) with a one-click Copy of the full plan as JSON.

​
Drilling into an individual run

From a plan’s Tasks tab, opening one submitted batch shows Pinot’s actual per-task subtask table: each subtask’s ID, state (waiting, running, completed, error, dropped, timed out, aborted), start/finish time, and duration. Opening a subtask goes one level deeper, into its progress log:
  • A switch between Live Progress (what the minion is reporting right now) and Log History (the recorded log), with an explanation when live progress is no longer available.
  • A search box, multi-select filters for log level and label, and a show-timestamps toggle.
  • A running count of how many lines are shown out of the total, newest first.
  • A metadata panel with the subtask’s configuration in plain terms — task type, table, sequence ID, size settings, and the input files it’s processing.

​
Retention of task execution details

Once a task finishes, Pinot keeps its detailed execution metadata — subtask states, progress, and configuration — for a limited window before cleaning it up (on the order of a day by default; your cluster’s operator can tune this). Once that window passes, drilling into an old batch shows a clear Task Execution Data No Longer Available message instead of an empty or broken page. The execution charts have their own, longer retention: after about two weeks a plan’s charts show a Task Metrics No Longer Available notice. These are separate from the global Tasks list’s own “recently seen” behavior described above — one is how long Pinot keeps a task’s execution details, the other is how long your browser remembers a completed plan is worth showing in the all-tables view. If you need to look back at a plan that finished a while ago, the per-table Monitoring tab is the more reliable place to start, but the underlying execution detail can still have aged out on the Pinot side either way.

​
Permissions

Viewing and acting on tasks from Data Portal uses the same fine-grained authorization as the rest of Data Portal. Viewing task plans requires the GetTaskPlan action on the plan’s table — the global list silently omits plans for tables you can’t see — and aborting a plan requires AbortTaskPlan. Triggering a run (Schedule Now, Create task plan) goes through Pinot’s task APIs and their standard task actions, and the per-subtask debug and log views use Pinot’s task status/debug actions. See RBAC Actions for the full list. If a control is missing or disabled for you, ask your StarTree Cloud admin to confirm your permissions.