Observability

Once an External Table is created, the controller’s External Table watcher ingests data on a schedule. These read-only endpoints let you check each run’s status and see how far the queryable data lags behind the upstream table. There is also an endpoint to trigger a run immediately instead of waiting for the next scheduled tick.

The observability endpoints require executor=controller in the table’s ExternalTableSyncTask config (the default for tables onboarded via the Data Portal or the preview API).All paths are relative to your data-plane base URL — set export BASE_URL=https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot (the StarTree Cloud data-plane proxy). Add an Authorization header if your controller requires auth.

Run status is not the same as data freshness. A fileOnboardingRun.status of COMPLETED only means the last sync run finished — the table can still be behind the source, because the upstream table may have committed more data since. To judge whether your data is current, use the lag object (caughtUp, percentDataIngested, snapshotsBehind), not the run status.

Common tasks

A map from what you want to know to the call that answers it and the field to read. Each row links to the full reference below.

I want to…	Call	Look at
Check the last sync succeeded	`GET …/externalTable/status`	`fileOnboardingRun.status`
Diagnose a stuck or failed sync	`GET …/externalTable/status`	`fileOnboardingRun.failurePhase` + `errorMessage`, then Troubleshooting
Check my table is caught up with the source	`GET …/externalTable/status?includeLag=true`	`lag.caughtUp` / `lag.percentDataIngested`
Know how far behind my data is, as a duration	`GET …/externalTable/status?includeLag=true`	`lag.upstream.committedAtMs` − `lag.synced.committedAtMs`
Trigger a sync now instead of waiting	`POST /tasks/schedule`	Trigger a sync run

Prefer clicking to curl? A table’s details page in the Data Portal shows the same run status and sync controls — see Trigger a sync run.

API Endpoints Quick Reference

Method	Endpoint	Purpose
`GET`	`/tables/{tableNameWithType}/externalTable/status`	Status of the last (or current) sync run. Add `includeLag=true` for ingestion lag.
`POST`	`/tasks/schedule`	Optionally trigger a sync run now instead of waiting for the schedule.
`POST`	`/tables/{tableNameWithType}/iceberg/isSnapshotReady`	Check whether a specific Iceberg snapshot’s deletion vectors have converged on all servers.

Run status

GET /tables/{tableNameWithType}/externalTable/status Use this to answer “did the last sync run work?” and — with includeLag=true — “is my data current?” Returns the External Table sync watcher’s last run status for the table, under a fileOnboardingRun object. With includeLag=true, the response additionally reports how far the table’s queryable data is behind the upstream Iceberg table — see Ingestion lag.

Query parameter	Default	Description
`includeLag`	`false`	Also return ingestion lag as a nested `lag` object (available starting in StarTree release 0.16.0). Adds one catalog read; the default call stays cheap (a single ZooKeeper read, no catalog call).
`pendingExact`	`false`	Only with `includeLag=true`: compute pending files and bytes from an exact manifest diff instead of snapshot summary totals. See Exact pending mode.

Top-level fields:

Field	Type	Description
`tableNameWithType`	string	Table name with the `_OFFLINE` suffix.
`fileOnboardingRun`	object	The last (or current) sync run — see below. Always present.
`lag`	object \| null	Only with `includeLag=true` (the key is absent otherwise): the ingestion lag object, or `null` when lag does not apply to the table or could not be computed.
`lagMessage`	string \| null	Only with `includeLag=true` (the key is absent otherwise): explains why `lag` is `null`; `null` when lag was computed.

fileOnboardingRun fields:

Field	Type	Description
`status`	string	`IDLE` \| `RUNNING` \| `COMPLETED` \| `FAILED`.
`startTime`	string \| null	Run start, ISO-8601 UTC (e.g. `2026-02-09T18:53:20Z`); `null` if never started.
`startTimeMs`	long	Run start as raw epoch milliseconds; `0` if never started.
`endTime`	string \| null	Run end, ISO-8601 UTC; `null` while `RUNNING` or if never finished.
`endTimeMs`	long	Run end as raw epoch milliseconds; `0` while `RUNNING` or if never finished.
`filesDiscovered`	int	Source files found in this run.
`segmentsUploaded`	int	Segments successfully created and uploaded.
`errorMessage`	string \| null	Populated only when `status=FAILED`.
`failurePhase`	string \| null	When failed, where it failed: `FILE_LISTING` (credentials/path), `SEGMENT_GENERATION` (type mismatch), `SEGMENT_COMPRESSION` (server resources), `SEGMENT_UPLOAD` (deep-store permissions), `SNAPSHOT_FINALIZE`, `IS_EV_CONVERGENCE`, `SNAPSHOT_READINESS_POLL`, `CHECKPOINT_SAVE` (ZooKeeper), or `SCHEMA_UPDATE`. `null` on success. See Troubleshooting.
`checkpointValue`	string \| null	Snapshot ID / watermark after a successful run.
`requestId`	string \| null	Identifier of the run, for log correlation.

`status`	Meaning
`IDLE`	No run yet, or sitting between scheduled runs.
`RUNNING`	A run is in progress.
`COMPLETED`	Last run succeeded — check `checkpointValue`.
`FAILED`	Last run failed — check `errorMessage` and `failurePhase`. See Troubleshooting → Sync not progressing for a diagnosis guide per phase.

curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/status"

{
  "tableNameWithType": "nyc_taxi_trips_OFFLINE",
  "fileOnboardingRun": {
    "status": "COMPLETED",
    "startTime": "2026-02-09T18:53:20Z",
    "startTimeMs": 1770663200000,
    "endTime": "2026-02-09T18:54:20Z",
    "endTimeMs": 1770663260000,
    "filesDiscovered": 15,
    "segmentsUploaded": 15,
    "errorMessage": null,
    "failurePhase": null,
    "checkpointValue": "1234567890123456789",
    "requestId": "a1b2c3d4"
  }
}

Status codes:

Code	When
`200` + `lag: null` + `lagMessage`	OK, but lag does not apply — the table is not `catalogType=iceberg-rest` with `executor=controller`, or has no sync task config.
`200` + `lag` with some `null` fields + `lag.message`	OK — a transient catalog/ZooKeeper issue degraded individual lag fields. Lag never returns `500`.
`404`	Table not found.
`500`	Internal error (does not apply to lag computation, which degrades to `null` fields instead).

Ingestion lag

GET /tables/{tableNameWithType}/externalTable/status?includeLag=true Use this when you need to know whether the table is current with the source — and, if not, by how much.

Ingestion lag reporting is available starting in StarTree release 0.16.0.

Reports how far the table’s queryable data is behind the upstream Iceberg table. Lag is expressed as a snapshot delta rather than a time lag: the last fully synced snapshot versus the upstream head snapshot, plus the files and bytes still pending. Lag applies to catalogType=iceberg-rest tables synced with executor=controller. For any other table the endpoint still returns 200 with lag: null and a lagMessage explaining why; the status response is never failed because of lag. Snapshot lag is undefined for raw S3 listings, for example.

curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/status?includeLag=true"

{
  "tableNameWithType": "nyc_taxi_trips_OFFLINE",
  "fileOnboardingRun": {
    "status": "COMPLETED",
    "startTime": "2026-02-09T18:53:20Z",
    "startTimeMs": 1770663200000,
    "endTime": "2026-02-09T18:54:20Z",
    "endTimeMs": 1770663260000,
    "filesDiscovered": 15,
    "segmentsUploaded": 15,
    "errorMessage": null,
    "failurePhase": null,
    "checkpointValue": "1234567890123456789",
    "requestId": "a1b2c3d4"
  },
  "lag": {
    "tableNameWithType": "nyc_taxi_trips_OFFLINE",
    "synced": {
      "snapshotId": 1234567890123456789,
      "sequenceNumber": 41,
      "committedAtMs": 1770663000000,
      "committedAt": "2026-02-09T18:50:00Z",
      "totalFiles": 120,
      "totalSize": "5.00 GB",
      "totalRecords": 48000000
    },
    "upstream": {
      "snapshotId": 2345678901234567890,
      "sequenceNumber": 44,
      "committedAtMs": 1770663480000,
      "committedAt": "2026-02-09T18:58:00Z",
      "totalFiles": 192,
      "totalSize": "8.00 GB",
      "totalRecords": 76800000
    },
    "snapshotsBehind": 3,
    "caughtUp": false,
    "filesPending": 72,
    "bytesPending": 3221225472,
    "pendingExact": false,
    "percentDataIngested": 62.5,
    "controllerTimeMs": 1770663300000,
    "controllerTime": "2026-02-09T18:55:00Z",
    "message": null
  },
  "lagMessage": null
}

lag object fields:

Field	Type	Description
`tableNameWithType`	string	Echoes the table name with the `_OFFLINE` suffix.
`synced`	object \| null	The last fully synced snapshot: the newest snapshot whose data is actually queryable. It is recorded only after the ingested segments have converged on all servers. `null` when the table has never synced or when the synced state could not be read; `message` distinguishes the two cases.
`upstream`	object \| null	The upstream Iceberg head snapshot, read from the catalog. `null` when the catalog is unreachable or the upstream table has no snapshots.
`snapshotsBehind`	int \| null	How many upstream snapshots sit between the synced snapshot and the head. Best-effort: exact for the linear histories ingestion produces, but branches, tags, WAP snapshots, or retention can skew it. `null` on Iceberg v1 metadata.
`caughtUp`	boolean \| null	`true` when the synced snapshot is the upstream head. A table that has never synced reports `false`, not `null`; `null` appears only when the synced state or the head could not be read.
`filesPending`	long \| null	Data files not yet queryable. A net delta of snapshot summary totals by default, or an exact manifest diff with `pendingExact=true`. `null` when it cannot be computed honestly.
`bytesPending`	long \| null	Bytes not yet queryable, as a raw number, computed the same way as `filesPending`.
`pendingExact`	boolean	Whether the pending numbers came from the exact manifest diff (`true`) or from summary totals (`false`).
`percentDataIngested`	double \| null	Percent of upstream bytes already queryable. `100` when caught up, `0` when never synced, `null` when it cannot be computed.
`controllerTimeMs`	long	Controller clock at computation time, raw epoch milliseconds.
`controllerTime`	string	Controller clock at computation time, ISO-8601 UTC (display companion of `controllerTimeMs`).
`message`	string \| null	Explains any degraded or missing fields, for example summary totals being unavailable or the exact-diff cap being hit. `null` when everything was computed cleanly.

Snapshot fields (synced and upstream):

Field	Type	Description
`snapshotId`	long \| null	Iceberg snapshot ID.
`sequenceNumber`	long \| null	Iceberg sequence number.
`committedAtMs`	long \| null	The upstream Iceberg commit time as raw epoch milliseconds. This is always catalog commit time, never Pinot ingest time.
`committedAt`	string \| null	The commit time as ISO-8601 UTC, truncated to seconds (display companion of `committedAtMs`).
`totalFiles`	long \| null	Total data files at this snapshot, from the snapshot summary. `null` when the snapshot has expired from table metadata or the summary keys are absent.
`totalSize`	string \| null	Total data size at this snapshot, human-readable (e.g. `"5.00 GB"`), from the snapshot summary. Same nullability as `totalFiles`.
`totalRecords`	long \| null	Total records at this snapshot, from the snapshot summary. Same nullability as `totalFiles`.

There is deliberately no lagMs field. To express lag as a duration, subtract lag.synced.committedAtMs from lag.upstream.committedAtMs (raw epoch milliseconds; the ISO-8601 committedAt strings are seconds-precision display companions). Both timestamps come from the same upstream catalog clock, so the difference is not skewed by clock drift between the catalog and the Pinot controller.

Transient catalog or ZooKeeper failures never turn into a 500. The affected lag fields are returned as null with an explanatory message, and a synced state that exists but cannot be read is reported as unavailable rather than as a confident “nothing ingested”.

Exact pending mode

By default, filesPending and bytesPending are derived from the snapshot summary totals of the two snapshots: a single metadata read, cheap at any table size. Because it is a net delta, compaction rewriting files can make the numbers approximate. With pendingExact=true, the controller instead diffs the live data-file sets of the head and synced snapshots. The result is exact, but the diff enumerates files and is therefore capped at iceberg.snapshotProcessing.lagMaxDiffFiles files per snapshot enumeration (default 100,000) — a per-table key in the ExternalTableSyncTask config. Past the cap, the response degrades back to summary estimates and says so in lag.message, with pendingExact: false.

Integration notes

Pending fields can be null even when caughtUp=false. A compaction, overwrite, or delete between snapshots makes the cheap summary estimate unreliable, so filesPending, bytesPending, and percentDataIngested may come back null. Fall back to snapshotsBehind, or retry with pendingExact=true; lag.message explains.
Two different message fields. Top-level lagMessage means lag itself is null (lag does not apply to the table); without includeLag=true neither key appears at all. lag.message means lag is present but an individual field is degraded.
Display strings vs numbers. startTime, endTime, committedAt, controllerTime, and totalSize are display strings. Every timestamp string has a raw epoch-milliseconds companion (startTimeMs, endTimeMs, committedAtMs, controllerTimeMs) — use those for arithmetic. filesPending, bytesPending, percentDataIngested, totalFiles, totalRecords, and snapshotsBehind remain numbers.

Metrics and alerting

The status and lag endpoints answer on-demand questions. For dashboards and continuous monitoring, the controller also emits per-sync metrics (available wherever you scrape StarTree Pinot controller metrics). Per-phase sync timers (milliseconds) — how long each stage of a run takes; useful for latency dashboards and for finding where a slow sync spends its time:

Metric	Stage
`EXTERNAL_TABLE_SYNC_FILE_LISTING_TIME_MS`	Listing source files from the catalog
`EXTERNAL_TABLE_SYNC_SEGMENT_GENERATION_TIME_MS`	Converting Parquet to segments
`EXTERNAL_TABLE_SYNC_SEGMENT_COMPRESSION_TIME_MS`	Compressing segments
`EXTERNAL_TABLE_SYNC_SEGMENT_UPLOAD_TIME_MS`	Uploading segments to deep store
`EXTERNAL_TABLE_SYNC_SNAPSHOT_FINALIZE_TIME_MS`	Finalizing the snapshot
`EXTERNAL_TABLE_SYNC_CHECKPOINT_SAVE_TIME_MS`	Persisting the checkpoint / watermark
`EXTERNAL_TABLE_SYNC_OVERALL_TIME_MS`	End-to-end run duration

Deletion-vector convergence — only when deletion vectors are enabled; covers the server-readiness handshake:

Metric	Meaning
`EXTERNAL_TABLE_SYNC_READINESS_POLL_SUCCESS` / `_TIMEOUT` / `_FAST_FAIL` / `_SKIPPED` / `_ATTEMPTS`	Outcomes of polling servers for snapshot readiness
`EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIMEOUT`	A snapshot’s ideal-state / external-view convergence timed out
`EXTERNAL_TABLE_SYNC_SNAPSHOT_READINESS_POLL_TIME_MS` / `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIME_MS`	Time spent polling for readiness and waiting for convergence

There is no metric for ingestion lag or per-run success/failure today — monitor those through the status API:

Sync failed — poll …/externalTable/status and alert when fileOnboardingRun.status is FAILED (triage with failurePhase).
Data falling behind — poll …?includeLag=true and alert when lag.caughtUp is false, or lag.percentDataIngested stays below your threshold for longer than one sync interval.
Snapshot silently skipped — a run that reports COMPLETED with segmentsUploaded: 0 while lag.caughtUp is false. See Sync says COMPLETED but no data lands.

Trigger a sync run (optional)

POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=<tableNameWithType>

You normally do not need this. After a table is created, the controller’s watcher runs the first sync automatically and then re-syncs on the table’s schedule. Use this endpoint only to start a run immediately instead of waiting for the next tick.

Query parameter	Required	Description
`taskType`	Yes	Always `ExternalTableSyncTask`.
`tableName`	Yes	Table name with the `_OFFLINE` suffix, e.g. `nyc_taxi_trips_OFFLINE`.

curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE"

{ "ExternalTableSyncTask": "Task_ExternalTableSyncTask_3f2a1c9e-4b7d-4a1f-9c3e-2d8f5a6b7c9d_1707500000000" }

A table’s details page in the Data Portal also exposes Schedule Now, Pause Sync, and Resume Sync for the sync task, so you don’t need to call these endpoints directly. Pausing removes the table’s sync schedule (already-ingested data is untouched); resuming restores it. See Task Observability for the broader task-monitoring view these controls sit alongside.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Observability

Common tasks

API Endpoints Quick Reference

Run status

Ingestion lag

Exact pending mode

Integration notes

Metrics and alerting

Trigger a sync run (optional)

​Common tasks

​API Endpoints Quick Reference

​Run status

​Ingestion lag

​Exact pending mode

​Integration notes

​Metrics and alerting

​Trigger a sync run (optional)

Common tasks

API Endpoints Quick Reference

Run status

Ingestion lag

Exact pending mode

Integration notes

Metrics and alerting

Trigger a sync run (optional)