> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Symptom-based fixes for common External Table onboarding, schema, query, and sync errors, indexed by the exact error string to search for.

<Warning>
  This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
</Warning>

Common issues when onboarding and querying External Tables, grouped by symptom. Each entry lists the likely cause and the fix. If an issue isn't covered here, reach out to StarTree support.

<Note>
  **Quick lookup — find your error:**

  | Error message or symptom                                        | Jump to                                                                                                               |
  | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
  | AWS region error / `s3://` in bucket field                      | [Onboarding fails with a region error](#onboarding-fails-with-a-region-error)                                         |
  | `Access Denied` / HTTP 403                                      | [Access Denied when creating the table](#access-denied-http-403-when-creating-the-table)                              |
  | Preview succeeds but no data / wrong files                      | [Preview can't read the files](#preview-cant-read-the-files--wrong-path)                                              |
  | `NumberFormatException: For input string: "Binary{...}"`        | [Segment generation fails with NumberFormatException](#segment-generation-fails-with-numberformatexception)           |
  | `Cannot create inverted index on column ... without dictionary` | [Cannot create inverted index without dictionary](#cannot-create-inverted-index-on-column--without-dictionary)        |
  | `Failed to create FieldIndexConfigs`                            | [Failed to create FieldIndexConfigs](#failed-to-create-fieldindexconfigs)                                             |
  | `servers not responded` / query timeout                         | [Query times out or returns servers not responded](#query-times-out-or-returns-servers-not-responded)                 |
  | First query slow, later queries fast                            | [First query on a column is slow](#first-query-on-a-column-is-slow-later-queries-are-fast)                            |
  | `Native memory allocation (mmap) failed` / pod OOM restarts     | [Server OOM / pod restarts](#server-oom--pod-restarts-under-query-load)                                               |
  | `Could not acquire distributed lock for orchestrated task type` | [Distributed lock error](#could-not-acquire-distributed-lock-for-orchestrated-task-type--externaltablesynctask)       |
  | Sync fails on one bad file                                      | [Sync run fails because of one unreadable file](#a-sync-run-fails-because-of-one-unreadable-file)                     |
  | `FileNotFoundException` / missing file after source compaction  | [Segments fail to load or serve after upstream compaction](#segments-fail-to-load-or-serve-after-upstream-compaction) |
  | Table created, `status` stays `IDLE`                            | [Table created but no sync ever starts](#table-created-but-no-sync-ever-starts)                                       |
  | `FAILED` status — what does `failurePhase` mean?                | [Sync not progressing](#sync-not-progressing)                                                                         |
</Note>

***

## Onboarding

### Onboarding fails with a region error

**Symptom:** Validation or preview fails with an AWS region error, often after entering the bucket path.

**Cause:** The AWS region isn't set, or the bucket field contains an `s3://...` URL or a trailing slash.

**Fix:**

* Enter the **bucket name only** in the bucket field — not `s3://bucket/...`.
* Remove any trailing `/` from the prefix.
* Set the AWS **region in the connection config** (`catalog.s3.region` / the tier `region`). When the region is in config, the `AWS_REGION` environment variable is only a fallback and isn't required.

### `Access Denied` (HTTP 403) when creating the table

**Symptom:**

```
software.amazon.awssdk.services.s3.model.AccessDeniedException: Access Denied
(Service: S3, Status Code: 403, ...)
```

**Cause:** The cluster cannot read the source bucket. Credentials valid elsewhere are not necessarily the ones the cluster uses.

**Fix:**

1. Confirm whatever the table uses for access — an assumed IAM role (`roleArn`/`externalId`), the cluster's node IAM role, or static access keys — has `s3:GetObject` and `s3:ListBucket` on the source bucket and prefix.
2. Verify from inside the cluster (e.g. a debug pod):
   ```bash theme={null}
   aws s3 ls s3://<bucket>/<prefix>/
   ```
3. If listing fails there, fix the bucket policy / role before retrying onboarding.

### Preview can't read the files / wrong path

**Symptom:** Validation succeeds but preview errors, or it looks for a `default` table.

**Cause:** The prefix points at the wrong level — for raw Parquet it should point at the folder that directly contains the `.parquet` files (or the table root for Iceberg).

**Fix:** Adjust the prefix to the correct level. Use `aws s3 ls` to confirm the path you give actually lists Parquet files (or an Iceberg `metadata/` + `data/` layout).

### Very large source (hundreds of thousands of files)

**Symptom:** Onboarding a path with hundreds of thousands or millions of files is slow or never produces segments.

**Cause:** Every file becomes work for the catalog scan and segment generation.

**Fix:** For an initial proof-of-concept, point at a smaller sub-prefix (for example, a single month's partition). Scale the cluster for the full dataset. There is a per-table segment threshold; work with StarTree support for very large tables.

***

## Schema & table creation

### Segment generation fails with `NumberFormatException`

**Symptom:**

```
java.lang.NumberFormatException: For input string: "Binary{2 reused bytes [48 50]}"
```

**Cause:** A column's data type was changed away from what preview inferred — for example a binary/string column was set to `INT`/`LONG`. The Parquet data no longer matches the declared Pinot type.

**Fix:** Use the data types the preview step produced. Don't override inferred types in the schema. See [Data Type Mapping](./data-type-mapping).

### `Cannot create inverted index on column ... without dictionary`

**Symptom:** Table creation is rejected for an inverted (or FST/IFST) index on a RAW column.

**Cause:** Since release 2.164.0, dictionary-backed indexes require an explicit `dictionary` block — they no longer build an implicit dictionary.

**Fix:** Keep the forward index `RAW` and add a `dictionary` block alongside the index:

```json theme={null}
"indexes": {
  "forward":    { "encodingType": "RAW" },
  "dictionary": {},
  "inverted":   { "disabled": false }
}
```

### `Failed to create FieldIndexConfigs`

**Symptom:** Table creation fails with this generic message.

**Cause:** A malformed or conflicting index configuration — often a hand-edited combination of `forward`, `dictionary`, and an index entry.

**Fix:** Start from the table config the preview step generates and add only supported indexes. Don't mix incompatible options on one column.

### Inverted index on a multi-value column fails to build

**Symptom:** Segment build fails on a multi-value column that has an inverted index (e.g. `Cannot create inverted index for raw index column`, or a "raw inverted index not supported for multi-value columns" message).

**Cause:** The inverted index needs a dictionary, and a raw (no-dictionary) inverted index isn't supported — multi-value columns are especially likely to hit this.

**Fix:** Add a `dictionary` block to the column (as above). If the build still fails specifically on a multi-value column, remove the inverted index from it and reach out to support.

***

## Queries

### Query times out or returns `servers not responded`

**Symptom:**

```
427: N servers [...] not responded
```

or a group-by/aggregation that never returns within the timeout.

**Causes & fixes — check in order:**

1. **Missing or conflicting index config (most common).** Aggregations and filters scanning remote data without the right index are slow. Add a [supported index](./indexes) for your filter/group-by columns, and remove conflicting or leftover index configs.
2. **Caching not enabled.** Turn on the page cache and preload so index data is local. See [Best Practices and Configs](./best-practices-and-configs) (`enable.prefetch.page.cache`, `preload.enable`, `preload.index.keys.override`).
3. **Group-by on a derived/computed column** or a `$segmentName` filter — these defeat pruning. Group by a real column and drop debugging filters.
4. **Under-provisioned servers.** A single small server against a large dataset will be CPU-bound. Scale out.
5. **Raise the query timeout** while debugging: `SET "timeoutMs" = '60000';`.

### First query on a column is slow, later queries are fast

**Symptom:** Cold query is slow; the same query is fast afterward.

**Cause:** The first read populates the cache from object storage.

**Fix:** Expected behavior. To pay this cost at load time instead of on the first query, enable pre-warm (`pinot.parquet.prewarm.enabled`) and `preload.enable`. See [Data and Index Caching](./data-and-index-caching).

### Tuning a large scan

For wide scans over big datasets, these query options help (see [Best Practices and Configs](./best-practices-and-configs)):

```sql theme={null}
SET "enable.prefetch.page.cache" = 'true';
SET "prefetch.projection.queue.size" = '10';
SET "readAhead.enable" = 'true';
```

### Server OOM / pod restarts under query load

**Symptom:** A server runs out of native memory or is OOM-killed, often during a large scan (`Native memory allocation (mmap) failed`, or repeated pod restarts).

**Causes & fixes:**

1. **Too many memory maps.** Wide tables create one mmap per column index; this can exhaust the OS `max_map_count`. Enable index consolidation (`preload.enable.index.consolidation=true`) to pack a segment's indexes into one file.
2. **Cache / prefetch over-allocation.** Cap the in-memory caches and prefetch buffer (`pinot.parquet.page.cache.memory.*`, `...prefetch.size.mb`) relative to server heap.
3. **A heavy query.** Enable [query OOM protection](./best-practices-and-configs#query-oom-protection-large-scans) so one query is killed instead of the server.

***

## Sync & operations

### `Could not acquire distributed lock for orchestrated task type ... ExternalTableSyncTask`

**Symptom:**

```
Could not acquire distributed lock for orchestrated task type ExternalTableSyncTask
on table <name>_OFFLINE. Another controller may be generating tasks.
```

**Cause:** Another controller is already running a sync for the table.

**Fix:** Benign — retry later. The run in progress continues normally.

### A sync run fails because of one unreadable file

**Symptom:** A sync run fails, and the failure traces back to a single problematic Parquet file.

**Cause:** By default a run fails if any file can't be read, so the whole snapshot is rejected.

**Fix:** Set `continueOnFileError=true` in the `ExternalTableSyncTask` config to skip unreadable files and continue. The run still ends as `status=COMPLETED`; compare `filesDiscovered` vs `segmentsUploaded` in the [status endpoint](./observability#run-status) to spot skipped files, then investigate them separately.

<Note>
  `continueOnFileError` only affects what a **sync run** tolerates — it skips a file that can't be read *while segments are being generated*. It does **not** protect a segment that has already been ingested: if that segment's backing Parquet file later disappears (for example, [compacted away upstream](#segments-fail-to-load-or-serve-after-upstream-compaction)), the server fails to load or serve it regardless of this setting.

  Don't confuse it with `continueOnError` (an OSS `ingestionConfig` field): `continueOnError` skips **individual bad rows** while reading a file or stream, whereas `continueOnFileError` skips an **entire unreadable file**. They are separate configs set in different places.
</Note>

### Segments fail to load or serve after upstream compaction

**Symptom:** Queries fail, or a segment reload stalls (`status=IN_PROGRESS` with a rising `estimatedTimeRemainingInMinutes`), and server logs show a missing Parquet file:

```
java.io.FileNotFoundException: No such file or directory:
s3a://<bucket>/<prefix>/data/partition_date=.../<file>.parquet
```

**Cause:** The table uses a **raw S3/GCS Data Lake source** (`catalogType=s3` or `gcs-interop`), which pins each ingested segment to a physical Parquet file path. When the upstream source **compacts** — rewrites many small files into fewer larger ones and deletes the originals — the paths those already-ingested segments point at no longer exist, so the server can't load or serve them. `continueOnFileError` does not help here: it only skips files *during a sync run*, not after a segment is already in the table.

**Fix:**

* **Use an Iceberg REST catalog for any source that compacts or rewrites files.** A REST catalog (`catalogType=iceberg-rest` — AWS Glue, S3 Tables, Unity Catalog, or generic Nessie/REST) tracks the table's live file set through snapshot metadata, so each sync reconciles to the current post-compaction files instead of leaving segments pointing at deleted paths. Raw S3/GCS is best for **append-only** sources whose files are never rewritten. See [Connect a Catalog](./getting-started/connect-a-catalog).
* **On an existing raw source**, the affected segments must be rebuilt against the new files (re-onboard the table, or reload once the files are stable) — there is no config that repoints a segment at a compacted-away file. Setting `continueOnError=true` only lets ongoing ingestion skip rows it can't read; it accepts inconsistent results and does not restore a segment whose file is gone.

### Table created, but no sync ever starts

**Symptom:** The table exists, but `status` stays `IDLE` and the observability endpoints return nothing useful.

**Cause:** The table isn't on the controller-watcher path — usually `executor=controller` is missing from the `ExternalTableSyncTask` config, or the feature isn't enabled on the cluster.

**Fix:** Confirm `executor: controller` is in the task config (the preview/onboarding flow sets it), and that the External Table feature is enabled on the cluster. Then trigger a run to start immediately: `POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=<name>_OFFLINE`.

### Sync not progressing

Use the [status endpoint](./observability#run-status) to diagnose:

* **`IDLE` and never advancing** — check `executor=controller`, a valid `schedule` cron, and that the feature is enabled (see above).
* **`RUNNING` for a long time** — large source or under-provisioned servers; reduce the prefix or scale out.
* **`FAILED`** — read `failurePhase`:
  * `FILE_LISTING` → credentials/path issue
  * `SEGMENT_GENERATION` → data-type mismatch (see `NumberFormatException` above; escalate if the config is valid)
  * `SEGMENT_COMPRESSION` → server resources
  * `SEGMENT_UPLOAD` → deep-store permissions
  * `CHECKPOINT_SAVE` → controller/ZooKeeper issue
  * `SNAPSHOT_FINALIZE`, `IS_EV_CONVERGENCE`, `SNAPSHOT_READINESS_POLL`, `SCHEMA_UPDATE` → later stages of the sync pipeline (snapshot commit, Helix ideal-state/external-view convergence, and schema-evolution update); escalate to StarTree support with the `requestId` if you hit one of these repeatedly

### Sync says `COMPLETED` but no data lands

**Symptom:** `fileOnboardingRun.status` is `COMPLETED`, yet the table is empty or missing recent data, and `segmentsUploaded` is `0`.

**Cause:** A snapshot whose segment generation keeps failing (for example a persistent `SEGMENT_GENERATION` error on every file) is retried up to `maxRetries` (default 3) consecutive runs, then marked terminally failed and abandoned. Once it is abandoned, the next run finds no *newer* snapshot to process and reports `COMPLETED` with `filesDiscovered: 0` / `segmentsUploaded: 0`. Because the run-status record keeps only the latest run, the earlier `FAILED` runs are overwritten — so the table looks healthy while that snapshot's data was never ingested.

**Diagnose:**

* Compare `filesDiscovered` vs `segmentsUploaded` in the [status endpoint](./observability#run-status). A `COMPLETED` run with `0` uploads that is not caught up is the tell.
* Add `?includeLag=true`: if `lag.caughtUp` is `false` (or `lag.percentDataIngested` is well under `100`) while `status` is `COMPLETED`, a snapshot was skipped rather than ingested.
* Find the underlying error from the runs that actually failed — most often a `SEGMENT_GENERATION` failure (a data-type or Parquet-read issue). See [Segment generation fails](#segment-generation-fails-with-numberformatexception).

**Fix:** Resolve the underlying per-file failure (correct the schema/type mismatch, or set `continueOnFileError=true` to skip a genuinely bad file and continue), then trigger a fresh run: `POST /tasks/schedule?taskType=ExternalTableSyncTask&tableName=<name>_OFFLINE`. If `SEGMENT_GENERATION` keeps failing on a valid config, escalate with the `requestId`.

### Checking sync health

To see why ingestion isn't progressing, use the [observability endpoints](./observability): run status (and `failurePhase` on failure), the ingestion checkpoint, and the source file count.

***

## When to escalate to engineering

Most issues are self-serviceable — **auth/region/path, index and dictionary config, cron, and cache/OOM tuning** are all covered above. Escalate to engineering when:

* `SEGMENT_GENERATION` or `CHECKPOINT_SAVE` keeps failing with a valid config.
* Servers still OOM after capping caches and enabling consolidation + OOM protection.
* `status=COMPLETED` but query results are wrong or missing.
* A specific Parquet type fails to read (e.g. a `getBytes`/decimal error), or you hit a very-large-table segment threshold.