Skip to main content
This feature requires StarTree release 0.15.0 or later, and must be enabled on demand — contact StarTree support to activate it.
Supplemental guidance for running External Tables at scale — capacity guardrails, deployment recommendations, and a couple of additional error fixes not covered in Best Practices and Configs or Troubleshooting.

​
Why remote Parquet queries are fast

Most engines that query Iceberg/Parquet directly (e.g. Trino, Athena) read at the row-group / column-chunk level. Pinot reads at the Parquet page level — a finer granularity — and uses its own indexes (inverted, range) to skip straight to the pages that matter instead of scanning a whole column chunk. This page-level skip is what makes an indexed External Table competitive with a native Pinot segment despite the data living remotely. External Tables also federate cleanly with regular Pinot tables. A common pattern keeps recent data (e.g. the last 24h) in a REALTIME table fed from Kafka, while the same events land in Iceberg for long-term retention — a single query can span both, giving one query surface across the full time range without copying the archive into Pinot.

​
Capacity guardrails

These aren’t hard limits, but the confident, well-tested operating range today:
GuardrailGuidance
Files per tableUp to ~350,000 Parquet files. Each source file maps 1:1 to a pseudo-segment today, so file count directly drives segment count.
Segments per clusterStay well inside ~500,000 total — treat this and the per-table file guardrail as overlapping budgets, not independent ones.
Segments per serverRule of thumb: don’t exceed ~10,000 segments per server.
File size< 100 MB is too small — too many files for the benefit; ask the source to compact. 128 MB – 512 MB is the well-tested sweet spot. Up to ~2 GB generally works; beyond that (up to ~4 GB) is less exercised.
Millions of files isn’t viable today. If a source has millions of small files, either compact upstream to raise average file size and cut file count below the guardrail, or onboard a subset first (e.g. the most recent partitions) and expand later rather than pointing at the full dataset on day one. Full-refresh Iceberg tables (where a snapshot replaces all existing files rather than appending) are supported, but a full replace can spike the segment count for that cycle. Before onboarding one, check how many files the refreshed snapshot contains, whether old segments need cleanup after each refresh, and whether the file count stays inside the guardrails above.

​
Deployment

  • Use a dedicated tenant for External Table workloads rather than an existing production tenant — it isolates ingestion and query resource usage from other traffic on the cluster.
  • Default replication factor is 1, chosen to maximize ingestion throughput; pseudo-segments are small (single-digit KB to a few MB before indexes are attached), so the cost of raising it later — e.g. to 2 once onboarding is stable — is low.
  • Both SSE and MSE query execution modes are supported against External Tables.
  • For AWS Glue sources specifically, the cluster’s IAM role/credentials also need Glue API access, in addition to the S3 permissions in Troubleshooting → Access Denied: glue:GetTable, glue:GetTables, glue:GetCatalog, glue:GetDatabase, glue:GetDatabases. Scope these to the specific Glue databases/tables the cluster needs, and prefer an assumed IAM role over static access keys.

​
Ingestion mechanics

Each sync run has two phases, which explains a couple of behaviors that are otherwise easy to mistake for a bug:
  1. Segment generation (controller). Source files are processed in batches (taskMaxNumFiles, default 100). The controller reads Parquet footer statistics (row count, min/max) and creates pseudo-segments — metadata and index pointers, not materialized data.
  2. Download and index (servers). Servers download the segment artifacts and build indexes. Queries can return partial results while this phase is in flight — for example, a response noting some segments are unavailable — until all servers finish loading.
Sort or cluster upstream data on the column most often used in equality filters (in the Iceberg/Parquet writer, before StarTree ever sees it) — this reduces random scans at read time, on top of whatever indexes are added on the Pinot side. Disable the sync schedule after the initial bootstrap for a table that only watches an S3 path for a one-time or occasional load (enabled=false on ExternalTableSyncTask). Leaving it on means every scheduled tick re-scans and bootstraps newly-discovered files, so the segment count keeps growing even when there’s no new data to justify it. For catalog-backed incremental sources, rely on checkpointing (see Observability) instead of re-bootstrapping.

​
Null handling — why it matters

nullHandlingEnabled defaults to false, and the onboarding/preview flow never sets it automatically. Without it, Pinot doesn’t build null bitmap vectors during segment generation, so columns containing nulls in the source Parquet aren’t tracked — IS NULL, IS NOT NULL, and aggregations that depend on the distinction (COUNT(col) vs. COUNT(*)) will be wrong on those columns. Turn it on explicitly if the source has nullable columns you query on: 
"tableIndexConfig": {
  "nullHandlingEnabled": true
}
In the Data Portal, this is a toggle on the last step of table creation — it’s off by default and must be turned on manually.

​
Additional troubleshooting

​
ZooKeeper packet-size overflow as segment count grows

Symptom: 
Caused by: org.apache.zookeeper.KeeperException$...
Packet len ... is out of range
Cause: The serialized ZNode record for the table has grown past ZooKeeper’s default 1 MB packet limit — this shows up as segment count increases. Fix:
  1. Cluster config: "zk.serializer.znrecord.write.size.limit.bytes": "62914560" (60 MB — raise further if the error persists as more segments are added).
  2. JVM flag on both Controller and Server: -Djute.maxbuffer=62914560.
  3. Restart Controller and Server.

​
403 on segment upload with 1970-era timestamps

Symptom: 
Got error status code: 403 (Forbidden) with reason:
"Invalid segment start/end time: 1970-01-21T11:51:21.667Z / ... for segment: ..."
Cause: Iceberg-generated Parquet files store timestamps in milliseconds; Pinot’s segment time-value sanity check misreads them as seconds, producing 1970-era dates and rejecting the segment. Fix: Add to the table’s ingestionConfig (at creation, or via a config update followed by re-triggering ingestion): 
"ingestionConfig": {
  "segmentTimeValueCheck": false
}