> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> REST endpoints to check an External Table sync run's status and ingestion lag against the upstream table, or trigger a run on demand.

Once an External Table is created, the controller's External Table watcher ingests data on a schedule. These read-only endpoints let you check each run's status and see how far the queryable data lags behind the upstream table. There is also an endpoint to trigger a run immediately instead of waiting for the next scheduled tick.

<Note>
  The observability endpoints require `executor=controller` in the table's `ExternalTableSyncTask` config (the default for tables onboarded via the Data Portal or the preview API).

  All paths are relative to your data-plane base URL — set `export BASE_URL=https://dp.<data-plane-id>.cp.<region>.startree.cloud/api/pinot` (the StarTree Cloud data-plane proxy). Add an `Authorization` header if your controller requires auth.
</Note>

<Info>
  **Run status is not the same as data freshness.** A `fileOnboardingRun.status` of `COMPLETED` only means the last sync run finished — the table can still be behind the source, because the upstream table may have committed more data since. To judge whether your data is current, use the [`lag`](#ingestion-lag) object (`caughtUp`, `percentDataIngested`, `snapshotsBehind`), not the run status.
</Info>

***

## Common tasks

A map from what you want to know to the call that answers it and the field to read. Each row links to the full reference below.

| I want to…                                    | Call                                         | Look at                                                                                                                          |
| --------------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Check the last sync succeeded                 | `GET …/externalTable/status`                 | [`fileOnboardingRun.status`](#run-status)                                                                                        |
| Diagnose a stuck or failed sync               | `GET …/externalTable/status`                 | [`fileOnboardingRun.failurePhase`](#run-status) + `errorMessage`, then [Troubleshooting](./troubleshooting#sync-not-progressing) |
| Check my table is caught up with the source   | `GET …/externalTable/status?includeLag=true` | [`lag.caughtUp`](#ingestion-lag) / `lag.percentDataIngested`                                                                     |
| Know how far behind my data is, as a duration | `GET …/externalTable/status?includeLag=true` | `lag.upstream.committedAtMs` − `lag.synced.committedAtMs`                                                                        |
| Trigger a sync now instead of waiting         | `POST /tasks/schedule`                       | [Trigger a sync run](#trigger-a-sync-run-optional)                                                                               |

<Tip>
  Prefer clicking to curl? A table's details page in the Data Portal shows the same run status and sync controls — see [Trigger a sync run](#trigger-a-sync-run-optional).
</Tip>

***

## API Endpoints Quick Reference

| Method | Endpoint                                              | Purpose                                                                                                           |
| ------ | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `GET`  | `/tables/{tableNameWithType}/externalTable/status`    | Status of the last (or current) sync run. Add `includeLag=true` for [ingestion lag](#ingestion-lag).              |
| `POST` | `/tasks/schedule`                                     | Optionally trigger a sync run now instead of waiting for the schedule.                                            |
| `POST` | `/tables/{tableNameWithType}/iceberg/isSnapshotReady` | Check whether a specific Iceberg snapshot's [deletion vectors](./deletion-vectors) have converged on all servers. |

***

## Run status

`GET /tables/{tableNameWithType}/externalTable/status`

*Use this to answer "did the last sync run work?" and — with `includeLag=true` — "is my data current?"*

Returns the External Table sync watcher's last run status for the table, under a `fileOnboardingRun` object. With `includeLag=true`, the response additionally reports how far the table's queryable data is behind the upstream Iceberg table — see [Ingestion lag](#ingestion-lag).

| Query parameter | Default | Description                                                                                                                                                                                             |
| --------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `includeLag`    | `false` | Also return ingestion lag as a nested `lag` object (available starting in **StarTree release 0.16.0**). Adds one catalog read; the default call stays cheap (a single ZooKeeper read, no catalog call). |
| `pendingExact`  | `false` | Only with `includeLag=true`: compute pending files and bytes from an exact manifest diff instead of snapshot summary totals. See [Exact pending mode](#exact-pending-mode).                             |

**Top-level fields:**

| Field               | Type           | Description                                                                                                                                                                     |
| ------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tableNameWithType` | string         | Table name with the `_OFFLINE` suffix.                                                                                                                                          |
| `fileOnboardingRun` | object         | The last (or current) sync run — see below. Always present.                                                                                                                     |
| `lag`               | object \| null | Only with `includeLag=true` (the key is absent otherwise): the [ingestion lag object](#ingestion-lag), or `null` when lag does not apply to the table or could not be computed. |
| `lagMessage`        | string \| null | Only with `includeLag=true` (the key is absent otherwise): explains why `lag` is `null`; `null` when lag was computed.                                                          |

**`fileOnboardingRun` fields:**

| Field              | Type           | Description                                                                                                                                                                                                                                                                                                                                                                                           |
| ------------------ | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status`           | string         | `IDLE` \| `RUNNING` \| `COMPLETED` \| `FAILED`.                                                                                                                                                                                                                                                                                                                                                       |
| `startTime`        | string \| null | Run start, ISO-8601 UTC (e.g. `2026-02-09T18:53:20Z`); `null` if never started.                                                                                                                                                                                                                                                                                                                       |
| `startTimeMs`      | long           | Run start as raw epoch milliseconds; `0` if never started.                                                                                                                                                                                                                                                                                                                                            |
| `endTime`          | string \| null | Run end, ISO-8601 UTC; `null` while `RUNNING` or if never finished.                                                                                                                                                                                                                                                                                                                                   |
| `endTimeMs`        | long           | Run end as raw epoch milliseconds; `0` while `RUNNING` or if never finished.                                                                                                                                                                                                                                                                                                                          |
| `filesDiscovered`  | int            | Source files found in this run.                                                                                                                                                                                                                                                                                                                                                                       |
| `segmentsUploaded` | int            | Segments successfully created and uploaded.                                                                                                                                                                                                                                                                                                                                                           |
| `errorMessage`     | string \| null | Populated only when `status=FAILED`.                                                                                                                                                                                                                                                                                                                                                                  |
| `failurePhase`     | string \| null | When failed, where it failed: `FILE_LISTING` (credentials/path), `SEGMENT_GENERATION` (type mismatch), `SEGMENT_COMPRESSION` (server resources), `SEGMENT_UPLOAD` (deep-store permissions), `SNAPSHOT_FINALIZE`, `IS_EV_CONVERGENCE`, `SNAPSHOT_READINESS_POLL`, `CHECKPOINT_SAVE` (ZooKeeper), or `SCHEMA_UPDATE`. `null` on success. See [Troubleshooting](./troubleshooting#sync-not-progressing). |
| `checkpointValue`  | string \| null | Snapshot ID / watermark after a successful run.                                                                                                                                                                                                                                                                                                                                                       |
| `requestId`        | string \| null | Identifier of the run, for log correlation.                                                                                                                                                                                                                                                                                                                                                           |

| `status`    | Meaning                                                                                                                                                                          |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `IDLE`      | No run yet, or sitting between scheduled runs.                                                                                                                                   |
| `RUNNING`   | A run is in progress.                                                                                                                                                            |
| `COMPLETED` | Last run succeeded — check `checkpointValue`.                                                                                                                                    |
| `FAILED`    | Last run failed — check `errorMessage` and `failurePhase`. See [Troubleshooting → Sync not progressing](./troubleshooting#sync-not-progressing) for a diagnosis guide per phase. |

```bash theme={null}
curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/status"
```

```json theme={null}
{
  "tableNameWithType": "nyc_taxi_trips_OFFLINE",
  "fileOnboardingRun": {
    "status": "COMPLETED",
    "startTime": "2026-02-09T18:53:20Z",
    "startTimeMs": 1770663200000,
    "endTime": "2026-02-09T18:54:20Z",
    "endTimeMs": 1770663260000,
    "filesDiscovered": 15,
    "segmentsUploaded": 15,
    "errorMessage": null,
    "failurePhase": null,
    "checkpointValue": "1234567890123456789",
    "requestId": "a1b2c3d4"
  }
}
```

**Status codes:**

| Code                                                  | When                                                                                                                             |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `200` + `lag: null` + `lagMessage`                    | OK, but lag does not apply — the table is not `catalogType=iceberg-rest` with `executor=controller`, or has no sync task config. |
| `200` + `lag` with some `null` fields + `lag.message` | OK — a transient catalog/ZooKeeper issue degraded individual lag fields. Lag never returns `500`.                                |
| `404`                                                 | Table not found.                                                                                                                 |
| `500`                                                 | Internal error (does not apply to lag computation, which degrades to `null` fields instead).                                     |

### Ingestion lag

`GET /tables/{tableNameWithType}/externalTable/status?includeLag=true`

*Use this when you need to know whether the table is current with the source — and, if not, by how much.*

<Note>
  Ingestion lag reporting is available starting in **StarTree release 0.16.0**.
</Note>

Reports how far the table's queryable data is behind the upstream Iceberg table. Lag is expressed as a snapshot delta rather than a time lag: the last fully synced snapshot versus the upstream head snapshot, plus the files and bytes still pending.

Lag applies to `catalogType=iceberg-rest` tables synced with `executor=controller`. For any other table the endpoint still returns `200` with `lag: null` and a `lagMessage` explaining why; the status response is never failed because of lag. Snapshot lag is undefined for raw S3 listings, for example.

```bash theme={null}
curl "$BASE_URL/tables/nyc_taxi_trips_OFFLINE/externalTable/status?includeLag=true"
```

```json theme={null}
{
  "tableNameWithType": "nyc_taxi_trips_OFFLINE",
  "fileOnboardingRun": {
    "status": "COMPLETED",
    "startTime": "2026-02-09T18:53:20Z",
    "startTimeMs": 1770663200000,
    "endTime": "2026-02-09T18:54:20Z",
    "endTimeMs": 1770663260000,
    "filesDiscovered": 15,
    "segmentsUploaded": 15,
    "errorMessage": null,
    "failurePhase": null,
    "checkpointValue": "1234567890123456789",
    "requestId": "a1b2c3d4"
  },
  "lag": {
    "tableNameWithType": "nyc_taxi_trips_OFFLINE",
    "synced": {
      "snapshotId": 1234567890123456789,
      "sequenceNumber": 41,
      "committedAtMs": 1770663000000,
      "committedAt": "2026-02-09T18:50:00Z",
      "totalFiles": 120,
      "totalSize": "5.00 GB",
      "totalRecords": 48000000
    },
    "upstream": {
      "snapshotId": 2345678901234567890,
      "sequenceNumber": 44,
      "committedAtMs": 1770663480000,
      "committedAt": "2026-02-09T18:58:00Z",
      "totalFiles": 192,
      "totalSize": "8.00 GB",
      "totalRecords": 76800000
    },
    "snapshotsBehind": 3,
    "caughtUp": false,
    "filesPending": 72,
    "bytesPending": 3221225472,
    "pendingExact": false,
    "percentDataIngested": 62.5,
    "controllerTimeMs": 1770663300000,
    "controllerTime": "2026-02-09T18:55:00Z",
    "message": null
  },
  "lagMessage": null
}
```

**`lag` object fields:**

| Field                 | Type            | Description                                                                                                                                                                                                                                                                                    |
| --------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tableNameWithType`   | string          | Echoes the table name with the `_OFFLINE` suffix.                                                                                                                                                                                                                                              |
| `synced`              | object \| null  | The last fully synced snapshot: the newest snapshot whose data is actually queryable. It is recorded only after the ingested segments have converged on all servers. `null` when the table has never synced or when the synced state could not be read; `message` distinguishes the two cases. |
| `upstream`            | object \| null  | The upstream Iceberg head snapshot, read from the catalog. `null` when the catalog is unreachable or the upstream table has no snapshots.                                                                                                                                                      |
| `snapshotsBehind`     | int \| null     | How many upstream snapshots sit between the synced snapshot and the head. Best-effort: exact for the linear histories ingestion produces, but branches, tags, WAP snapshots, or retention can skew it. `null` on Iceberg v1 metadata.                                                          |
| `caughtUp`            | boolean \| null | `true` when the synced snapshot is the upstream head. A table that has never synced reports `false`, not `null`; `null` appears only when the synced state or the head could not be read.                                                                                                      |
| `filesPending`        | long \| null    | Data files not yet queryable. A net delta of snapshot summary totals by default, or an exact manifest diff with `pendingExact=true`. `null` when it cannot be computed honestly.                                                                                                               |
| `bytesPending`        | long \| null    | Bytes not yet queryable, as a raw number, computed the same way as `filesPending`.                                                                                                                                                                                                             |
| `pendingExact`        | boolean         | Whether the pending numbers came from the exact manifest diff (`true`) or from summary totals (`false`).                                                                                                                                                                                       |
| `percentDataIngested` | double \| null  | Percent of upstream bytes already queryable. `100` when caught up, `0` when never synced, `null` when it cannot be computed.                                                                                                                                                                   |
| `controllerTimeMs`    | long            | Controller clock at computation time, raw epoch milliseconds.                                                                                                                                                                                                                                  |
| `controllerTime`      | string          | Controller clock at computation time, ISO-8601 UTC (display companion of `controllerTimeMs`).                                                                                                                                                                                                  |
| `message`             | string \| null  | Explains any degraded or missing fields, for example summary totals being unavailable or the exact-diff cap being hit. `null` when everything was computed cleanly.                                                                                                                            |

**Snapshot fields (`synced` and `upstream`):**

| Field            | Type           | Description                                                                                                                                            |
| ---------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `snapshotId`     | long \| null   | Iceberg snapshot ID.                                                                                                                                   |
| `sequenceNumber` | long \| null   | Iceberg sequence number.                                                                                                                               |
| `committedAtMs`  | long \| null   | The upstream Iceberg commit time as raw epoch milliseconds. This is always catalog commit time, never Pinot ingest time.                               |
| `committedAt`    | string \| null | The commit time as ISO-8601 UTC, truncated to seconds (display companion of `committedAtMs`).                                                          |
| `totalFiles`     | long \| null   | Total data files at this snapshot, from the snapshot summary. `null` when the snapshot has expired from table metadata or the summary keys are absent. |
| `totalSize`      | string \| null | Total data size at this snapshot, human-readable (e.g. `"5.00 GB"`), from the snapshot summary. Same nullability as `totalFiles`.                      |
| `totalRecords`   | long \| null   | Total records at this snapshot, from the snapshot summary. Same nullability as `totalFiles`.                                                           |

<Note>
  There is deliberately no `lagMs` field. To express lag as a duration, subtract `lag.synced.committedAtMs` from `lag.upstream.committedAtMs` (raw epoch milliseconds; the ISO-8601 `committedAt` strings are seconds-precision display companions). Both timestamps come from the same upstream catalog clock, so the difference is not skewed by clock drift between the catalog and the Pinot controller.
</Note>

Transient catalog or ZooKeeper failures never turn into a `500`. The affected lag fields are returned as `null` with an explanatory `message`, and a synced state that exists but cannot be read is reported as unavailable rather than as a confident "nothing ingested".

#### Exact pending mode

By default, `filesPending` and `bytesPending` are derived from the snapshot summary totals of the two snapshots: a single metadata read, cheap at any table size. Because it is a net delta, compaction rewriting files can make the numbers approximate.

With `pendingExact=true`, the controller instead diffs the live data-file sets of the head and synced snapshots. The result is exact, but the diff enumerates files and is therefore capped at `iceberg.snapshotProcessing.lagMaxDiffFiles` files per snapshot enumeration (default 100,000) — a per-table key in the `ExternalTableSyncTask` config. Past the cap, the response degrades back to summary estimates and says so in `lag.message`, with `pendingExact: false`.

### Integration notes

* **Pending fields can be `null` even when `caughtUp=false`.** A compaction, overwrite, or delete between snapshots makes the cheap summary estimate unreliable, so `filesPending`, `bytesPending`, and `percentDataIngested` may come back `null`. Fall back to `snapshotsBehind`, or retry with `pendingExact=true`; `lag.message` explains.
* **Two different message fields.** Top-level `lagMessage` means `lag` itself is `null` (lag does not apply to the table); without `includeLag=true` neither key appears at all. `lag.message` means `lag` is present but an individual field is degraded.
* **Display strings vs numbers.** `startTime`, `endTime`, `committedAt`, `controllerTime`, and `totalSize` are display strings. Every timestamp string has a raw epoch-milliseconds companion (`startTimeMs`, `endTimeMs`, `committedAtMs`, `controllerTimeMs`) — use those for arithmetic. `filesPending`, `bytesPending`, `percentDataIngested`, `totalFiles`, `totalRecords`, and `snapshotsBehind` remain numbers.

***

## Metrics and alerting

The status and lag endpoints answer on-demand questions. For dashboards and continuous monitoring, the controller also emits per-sync metrics (available wherever you scrape StarTree Pinot controller metrics).

**Per-phase sync timers** (milliseconds) — how long each stage of a run takes; useful for latency dashboards and for finding where a slow sync spends its time:

| Metric                                            | Stage                                 |
| ------------------------------------------------- | ------------------------------------- |
| `EXTERNAL_TABLE_SYNC_FILE_LISTING_TIME_MS`        | Listing source files from the catalog |
| `EXTERNAL_TABLE_SYNC_SEGMENT_GENERATION_TIME_MS`  | Converting Parquet to segments        |
| `EXTERNAL_TABLE_SYNC_SEGMENT_COMPRESSION_TIME_MS` | Compressing segments                  |
| `EXTERNAL_TABLE_SYNC_SEGMENT_UPLOAD_TIME_MS`      | Uploading segments to deep store      |
| `EXTERNAL_TABLE_SYNC_SNAPSHOT_FINALIZE_TIME_MS`   | Finalizing the snapshot               |
| `EXTERNAL_TABLE_SYNC_CHECKPOINT_SAVE_TIME_MS`     | Persisting the checkpoint / watermark |
| `EXTERNAL_TABLE_SYNC_OVERALL_TIME_MS`             | End-to-end run duration               |

**Deletion-vector convergence** — only when [deletion vectors](./deletion-vectors) are enabled; covers the server-readiness handshake:

| Metric                                                                                                  | Meaning                                                        |
| ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| `EXTERNAL_TABLE_SYNC_READINESS_POLL_SUCCESS` / `_TIMEOUT` / `_FAST_FAIL` / `_SKIPPED` / `_ATTEMPTS`     | Outcomes of polling servers for snapshot readiness             |
| `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIMEOUT`                                                         | A snapshot's ideal-state / external-view convergence timed out |
| `EXTERNAL_TABLE_SYNC_SNAPSHOT_READINESS_POLL_TIME_MS` / `EXTERNAL_TABLE_SYNC_IS_EV_CONVERGENCE_TIME_MS` | Time spent polling for readiness and waiting for convergence   |

<Warning>
  There is **no metric for ingestion lag or per-run success/failure** today — monitor those through the status API:

  * **Sync failed** — poll `…/externalTable/status` and alert when `fileOnboardingRun.status` is `FAILED` (triage with `failurePhase`).
  * **Data falling behind** — poll `…?includeLag=true` and alert when `lag.caughtUp` is `false`, or `lag.percentDataIngested` stays below your threshold for longer than one sync interval.
  * **Snapshot silently skipped** — a run that reports `COMPLETED` with `segmentsUploaded: 0` while `lag.caughtUp` is `false`. See [Sync says COMPLETED but no data lands](./troubleshooting#sync-says-completed-but-no-data-lands).
</Warning>

***

## Trigger a sync run (optional)

`POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=<tableNameWithType>`

<Note>
  You normally do **not** need this. After a table is created, the controller's watcher runs the first sync automatically and then re-syncs on the table's schedule. Use this endpoint only to start a run immediately instead of waiting for the next tick.
</Note>

| Query parameter | Required | Description                                                               |
| --------------- | -------- | ------------------------------------------------------------------------- |
| `taskType`      | Yes      | Always `ExternalTableSyncTask`.                                           |
| `tableName`     | Yes      | Table name **with the `_OFFLINE` suffix**, e.g. `nyc_taxi_trips_OFFLINE`. |

```bash theme={null}
curl -X POST "$BASE_URL/tasks/schedule?taskType=ExternalTableSyncTask&tableName=nyc_taxi_trips_OFFLINE"
```

```json theme={null}
{ "ExternalTableSyncTask": "Task_ExternalTableSyncTask_3f2a1c9e-4b7d-4a1f-9c3e-2d8f5a6b7c9d_1707500000000" }
```

<Tip>
  A table's details page in the Data Portal also exposes **Schedule Now**, **Pause Sync**, and **Resume Sync** for the sync task, so you don't need to call these endpoints directly. Pausing removes the table's sync schedule (already-ingested data is untouched); resuming restores it. See [Task Observability](/corecapabilities/observability/task-observability) for the broader task-monitoring view these controls sit alongside.
</Tip>