> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Advanced Sizing & Operational Guidance

> Capacity guardrails, deployment recommendations, and additional troubleshooting fixes for External Tables, beyond the core Best Practices and Troubleshooting references.

<Warning>
  This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
</Warning>

Supplemental guidance for running External Tables at scale — capacity guardrails, deployment recommendations, and a couple of additional error fixes not covered in [Best Practices and Configs](./best-practices-and-configs) or [Troubleshooting](./troubleshooting).

***

## Why remote Parquet queries are fast

Most engines that query Iceberg/Parquet directly (e.g. Trino, Athena) read at the **row-group / column-chunk** level. Pinot reads at the **Parquet page** level — a finer granularity — and uses its own indexes (inverted, range) to skip straight to the pages that matter instead of scanning a whole column chunk. This page-level skip is what makes an indexed External Table competitive with a native Pinot segment despite the data living remotely.

External Tables also federate cleanly with regular Pinot tables. A common pattern keeps recent data (e.g. the last 24h) in a REALTIME table fed from Kafka, while the same events land in Iceberg for long-term retention — a single query can span both, giving one query surface across the full time range without copying the archive into Pinot.

***

## Capacity guardrails

These aren't hard limits, but the confident, well-tested operating range today:

| Guardrail            | Guidance                                                                                                                                                                                                                   |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Files per table      | Up to **\~350,000** Parquet files. Each source file maps 1:1 to a pseudo-segment today, so file count directly drives segment count.                                                                                       |
| Segments per cluster | Stay well inside **\~500,000** total — treat this and the per-table file guardrail as overlapping budgets, not independent ones.                                                                                           |
| Segments per server  | Rule of thumb: **don't exceed \~10,000 segments per server**.                                                                                                                                                              |
| File size            | **\< 100 MB** is too small — too many files for the benefit; ask the source to compact. **128 MB – 512 MB** is the well-tested sweet spot. Up to **\~2 GB** generally works; beyond that (up to \~4 GB) is less exercised. |

**Millions of files isn't viable today.** If a source has millions of small files, either compact upstream to raise average file size and cut file count below the guardrail, or onboard a subset first (e.g. the most recent partitions) and expand later rather than pointing at the full dataset on day one.

**Full-refresh Iceberg tables** (where a snapshot replaces all existing files rather than appending) are supported, but a full replace can spike the segment count for that cycle. Before onboarding one, check how many files the refreshed snapshot contains, whether old segments need cleanup after each refresh, and whether the file count stays inside the guardrails above.

***

## Deployment

* **Use a dedicated tenant** for External Table workloads rather than an existing production tenant — it isolates ingestion and query resource usage from other traffic on the cluster.
* **Default replication factor is 1**, chosen to maximize ingestion throughput; pseudo-segments are small (single-digit KB to a few MB before indexes are attached), so the cost of raising it later — e.g. to 2 once onboarding is stable — is low.
* Both **SSE and MSE** query execution modes are supported against External Tables.
* For **AWS Glue** sources specifically, the cluster's IAM role/credentials also need Glue API access, in addition to the S3 permissions in [Troubleshooting → Access Denied](./troubleshooting#access-denied-http-403-when-creating-the-table): `glue:GetTable`, `glue:GetTables`, `glue:GetCatalog`, `glue:GetDatabase`, `glue:GetDatabases`. Scope these to the specific Glue databases/tables the cluster needs, and prefer an assumed IAM role over static access keys.

***

## Ingestion mechanics

Each sync run has two phases, which explains a couple of behaviors that are otherwise easy to mistake for a bug:

1. **Segment generation (controller).** Source files are processed in batches (`taskMaxNumFiles`, default `100`). The controller reads Parquet footer statistics (row count, min/max) and creates pseudo-segments — metadata and index pointers, not materialized data.
2. **Download and index (servers).** Servers download the segment artifacts and build indexes. **Queries can return partial results while this phase is in flight** — for example, a response noting some segments are unavailable — until all servers finish loading.

**Sort or cluster upstream data** on the column most often used in equality filters (in the Iceberg/Parquet writer, before StarTree ever sees it) — this reduces random scans at read time, on top of whatever indexes are added on the Pinot side.

**Disable the sync schedule after the initial bootstrap** for a table that only watches an S3 path for a one-time or occasional load (`enabled=false` on `ExternalTableSyncTask`). Leaving it on means every scheduled tick re-scans and bootstraps newly-discovered files, so the segment count keeps growing even when there's no new data to justify it. For catalog-backed incremental sources, rely on checkpointing (see [Observability](./observability#checkpoint)) instead of re-bootstrapping.

***

## Null handling — why it matters

`nullHandlingEnabled` defaults to `false`, and the onboarding/preview flow never sets it automatically. Without it, Pinot doesn't build null bitmap vectors during segment generation, so columns containing nulls in the source Parquet aren't tracked — `IS NULL`, `IS NOT NULL`, and aggregations that depend on the distinction (`COUNT(col)` vs. `COUNT(*)`) will be wrong on those columns.

Turn it on explicitly if the source has nullable columns you query on:

```json theme={null}
"tableIndexConfig": {
  "nullHandlingEnabled": true
}
```

In the Data Portal, this is a toggle on the **last step** of table creation — it's off by default and must be turned on manually.

***

## Additional troubleshooting

### ZooKeeper packet-size overflow as segment count grows

**Symptom:**

```
Caused by: org.apache.zookeeper.KeeperException$...
Packet len ... is out of range
```

**Cause:** The serialized ZNode record for the table has grown past ZooKeeper's default 1 MB packet limit — this shows up as segment count increases.

**Fix:**

1. Cluster config: `"zk.serializer.znrecord.write.size.limit.bytes": "62914560"` (60 MB — raise further if the error persists as more segments are added).
2. JVM flag on **both** Controller and Server: `-Djute.maxbuffer=62914560`.
3. Restart Controller and Server.

### 403 on segment upload with 1970-era timestamps

**Symptom:**

```
Got error status code: 403 (Forbidden) with reason:
"Invalid segment start/end time: 1970-01-21T11:51:21.667Z / ... for segment: ..."
```

**Cause:** Iceberg-generated Parquet files store timestamps in milliseconds; Pinot's segment time-value sanity check misreads them as seconds, producing 1970-era dates and rejecting the segment.

**Fix:** Add to the table's `ingestionConfig` (at creation, or via a config update followed by re-triggering ingestion):

```json theme={null}
"ingestionConfig": {
  "segmentTimeValueCheck": false
}
```