> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Data Type Mapping
> How Parquet and Iceberg source types map to Pinot types in an External Table, including time-column detection, granularity, and known limitations.
This feature is available starting in **StarTree release 0.15.0**. It must be enabled on demand — contact your StarTree representative to have it activated for your environment.
When you onboard an External Table, StarTree infers a Pinot schema from the source metadata. The mapping depends on the source:
* **Raw S3 Parquet** (S3 Data Lake, `catalogType=s3`) — inferred from the Parquet file's physical and logical types.
* **Iceberg** (AWS Glue / S3 Tables, `catalogType=iceberg-rest`) — inferred from the Iceberg table schema.
Numeric columns (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `BIG_DECIMAL`) are inferred as **metrics**; everything else (`BOOLEAN`, `STRING`, `JSON`, `BYTES`, time columns) as **dimensions**. You can adjust this in the schema before creating the table.
***
## Raw S3 Parquet → Pinot
For the **S3 Data Lake** source. Types come from the Parquet file metadata.
## Primitive Types
| Parquet Type | Pinot Type | Notes |
| ---------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `BOOLEAN` | `BOOLEAN` | Optional and required variants both supported. |
| `INT32` | `INT` | |
| `INT64` | `LONG` | |
| `FLOAT` | `FLOAT` | |
| `DOUBLE` | `DOUBLE` | |
| `BINARY` | `STRING` | Raw binary data is readable as a string. |
| `FIXED_LEN_BYTE_ARRAY` | `STRING` | Fixed-width byte arrays are stored as strings. |
| `INT96` | `TIMESTAMP` | Deprecated Parquet type. Converted to milliseconds since epoch; no column statistics available. Avoid using as a time column. |
***
## Logical Types
### Integer Subtypes
| Parquet Logical Type | Pinot Type | Notes |
| -------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `INT(8, signed)` | `INT` | Range: -128 to 127 |
| `INT(16, signed)` | `INT` | Range: -32,768 to 32,767 |
| `INT(32, signed)` | `INT` | |
| `INT(64, signed)` | `LONG` | |
| `INT(8, unsigned)` | `INT` | Range: 0 to 255 |
| `INT(16, unsigned)` | `INT` | Range: 0 to 65,535 |
| `INT(32, unsigned)` | `LONG` | **Known limitation:** values are stored as signed. Values above `2,147,483,647` may be corrupted. See [Unsigned Integer Limitation](#unsigned-integer-types). |
| `INT(64, unsigned)` | `LONG` | **Known limitation:** same signed-storage issue. Values above `Long.MAX_VALUE` are not preserved accurately. |
### String and Binary Types
| Parquet Logical Type | Pinot Type | Notes |
| ----------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `STRING` | `STRING` | UTF-8 encoded strings. |
| `ENUM` | `STRING` | Enum values stored as strings. |
| `JSON` | `JSON` | Parsed into Pinot's native `JSON` type, not preserved as a plain string. |
| `BSON` | `STRING` | Binary data is decoded as a UTF-8 string, not preserved as a distinct binary type. Non-UTF8-safe byte sequences may not round-trip cleanly — verify BSON columns after ingestion. |
| `UUID` | `STRING` | **Known limitation:** the raw 16-byte binary is decoded with a plain UTF-8 string conversion rather than a proper UUID formatter, which can produce garbled values or serialization errors. Verify UUID columns render correctly after ingestion. |
| `INTERVAL` | `STRING` | |
| `GEOMETRY`, `GEOGRAPHY` | `BYTES` | Stored as raw bytes; no geospatial functions or H3 index are available on external tables. |
### Temporal Types
| Parquet Logical Type | Pinot Type | Notes |
| -------------------------- | ----------- | ------------------------------------------------------------------------------------------------------ |
| `DATE` | `LONG` | Stored as epoch days. |
| `TIME(MILLIS)` | `LONG` | Milliseconds since midnight; range 0 to 86,399,999. |
| `TIME(MICROS)` | `LONG` | Microseconds since midnight; range 0 to 86,399,999,999. |
| `TIME(NANOS)` | `LONG` | Nanoseconds since midnight. |
| `TIMESTAMP(MILLIS, UTC)` | `TIMESTAMP` | |
| `TIMESTAMP(MILLIS, local)` | `TIMESTAMP` | |
| `TIMESTAMP(MICROS, UTC)` | `TIMESTAMP` | Native microsecond granularity is preserved (`EPOCH\|MICROSECONDS`) — see [Granularity](#granularity). |
| `TIMESTAMP(MICROS, local)` | `TIMESTAMP` | Native microsecond granularity is preserved — see [Granularity](#granularity). |
| `TIMESTAMP(NANOS, UTC)` | `TIMESTAMP` | Native nanosecond granularity is preserved (`EPOCH\|NANOSECONDS`) — see [Granularity](#granularity). |
| `TIMESTAMP(NANOS, local)` | `TIMESTAMP` | Native nanosecond granularity is preserved — see [Granularity](#granularity). |
### Decimal Types
All Parquet `DECIMAL` variants map to Pinot `BIG_DECIMAL`, preserving scale and precision.
| Parquet Physical + Logical Type | Pinot Type | Example |
| ---------------------------------------- | ------------- | ----------------- |
| `INT32` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(9, 2)` |
| `INT64` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(18, 4)` |
| `FIXED_LEN_BYTE_ARRAY` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(28, 6)` |
| `BINARY` + `DECIMAL(p, s)` | `BIG_DECIMAL` | `DECIMAL(38, 10)` |
***
## Complex Types
| Source type | Pinot type | Notes |
| ------------------------------- | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `LIST` / `ARRAY` of a primitive | Multi-value column of the element type | Element types: `INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`, `BOOLEAN`. A list of timestamps is stored as multi-value `LONG`. |
| `LIST` of a complex element | `JSON` | Lists of structs / maps / nested lists are serialized to a JSON string. |
| `MAP` | `MAP` | Native map for primitive values (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`). |
| `MAP` (other shapes) | `JSON` | Non-string keys or complex values are serialized to JSON. |
| `STRUCT` / nested group | `JSON` | Serialized to a JSON string; query with JSON functions. |
To use `JSON_MATCH` or `JSON_EXTRACT_SCALAR` on complex columns, you must enable a JSON index on those columns. Without it, queries will fail with a `Cannot apply JSON_MATCH on column without json index` error. Configure the index in the table's index settings.
Multi-value columns are capped at **1,000 entries per row** — the segment metadata is written with a fixed upper bound, and rows whose source array is longer are silently truncated at query time. If your lists can exceed 1,000 elements, keep the column as `JSON` instead.
***
## Iceberg → Pinot
For the **AWS Glue** and **AWS S3 Tables** sources. Types come from the Iceberg table schema.
| Iceberg type | Pinot type | Notes | |
| -------------------------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------- | -------------- |
| `boolean` | `BOOLEAN` | | |
| `int` | `INT` | | |
| `long` | `LONG` | | |
| `float` | `FLOAT` | | |
| `double` | `DOUBLE` | | |
| `decimal(p, s)` | `BIG_DECIMAL` | Precision and scale preserved. | |
| `string` | `STRING` | | |
| `uuid` | `STRING` | | |
| `binary`, `fixed` | `STRING` | | |
| `date` | `LONG` | \`EPOCH | DAYS\`. |
| `time` | `LONG` | Stored as the raw long; not a time-column candidate. | |
| `timestamp`, `timestamptz` | `TIMESTAMP` | Granularity inferred from the type — see note below. | |
| `timestamp_ns`, `timestamptz_ns` | `TIMESTAMP` | \`EPOCH | NANOSECONDS\`. |
| `struct` | `JSON` | Serialized to a JSON string. | |
| `list` | Multi-value column of the element type | `INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`, `BOOLEAN`; `list` becomes multi-value `LONG`. | |
| `list` | `JSON` | | |
| `map` | `MAP` | | |
| `map` (other shapes) | `JSON` | | |
| (unrecognized) | `STRING` | Falls back to `STRING`. | |
**Timestamp granularity is inferred from the Iceberg type** (current releases):
* `timestamp_ns` / `timestamptz_ns` → `EPOCH|NANOSECONDS`
* `timestamp(p)` with an explicit precision → milliseconds for `p ≤ 3`, microseconds for `p ≤ 6`, nanoseconds above that
* plain `timestamp` / `timestamptz` → **`EPOCH|MICROSECONDS`**, matching Iceberg's on-disk microsecond storage
Older 0.16.0 builds inferred plain `timestamp` as `EPOCH|MILLISECONDS`, which mislabeled microsecond data (values \~1000× off). Tables onboarded on those builds keep their inferred format — set the time column's granularity explicitly if it doesn't match the data. See [Time column](#time-column).
***
## Time column
A Pinot table can designate one column as its **primary time column** — a `DateTimeFieldSpec` that records the event time of each row. See the Apache Pinot [DateTimeFieldSpec reference](https://docs.pinot.apache.org/configuration-reference/schema#datetimefieldspec) for the general schema requirement and format syntax.
### How the time column is used in an External Table
* **Segment pruning** — a query with a time filter skips segments whose time range can't match, so fewer remote files are read.
* **Retention** — if a retention period is configured on the table, segments older than it are dropped.
The time column is **optional**: without one, the table is still created and fully queryable, but loses time-based segment pruning and retention.
### How the time column is detected
During onboarding, after the schema is inferred, StarTree selects a time column automatically — choosing, in order:
1. The first non-nullable millisecond-granularity datetime column.
2. Otherwise, the first millisecond-granularity datetime column (nullable allowed).
3. Otherwise, the first datetime column of any unit.
Only columns that map to a Pinot `DateTimeFieldSpec` are candidates — `TIMESTAMP`, `DATE`, and (raw Parquet) `INT96`. A plain integer column holding an epoch value is **not** auto-detected; set it as the time column manually if needed. If the source has no timestamp/date column, no time column is set; you can assign one in the schema before creating the table.
### Granularity
For **raw Parquet**, the inferred `DateTimeFieldSpec` preserves the source unit:
| Source timestamp | Inferred format | Granularity |
| ------------------- | --------------------- | ---------------- |
| `INT96` (legacy) | `EPOCH\|MILLISECONDS` | `1:MILLISECONDS` |
| `TIMESTAMP(millis)` | `EPOCH\|MILLISECONDS` | `1:MILLISECONDS` |
| `TIMESTAMP(micros)` | `EPOCH\|MICROSECONDS` | `1:MICROSECONDS` |
| `TIMESTAMP(nanos)` | `EPOCH\|NANOSECONDS` | `1:NANOSECONDS` |
| `DATE` | `EPOCH\|DAYS` | `1:DAYS` |
For **Iceberg**, the granularity follows the type: `timestamp_ns` → nanoseconds, `timestamp(p)` → from the precision, and plain `timestamp` / `timestamptz` → **microseconds** (older 0.16.0 builds inferred milliseconds — see the note in the Iceberg → Pinot section above).
Values are stored in the source unit — there is no implicit downscaling — so account for the unit in time-range queries. If the inferred granularity doesn't match what the underlying Parquet actually stores (for example, a table onboarded on an older build), set the time column's granularity explicitly.
***
## Known Limitations
### Unsigned Integer Types
Parquet `UINT32` and `UINT64` (unsigned 32-bit and 64-bit integers) are stored in Pinot as a signed `LONG`. This means:
* Values that fit within the signed range are accurate.
* Values above `2,147,483,647` (for UINT32) are stored as negative numbers or wrap around, silently corrupting approximately half the data.
If your dataset contains unsigned integers larger than `INT32_MAX`, treat the Pinot `LONG` values with caution or apply a transformation at ingestion time.
### Null Handling in Complex Types
`MAP` columns that contain null values in keys or values may not propagate null bitmaps correctly to Pinot segments. This can result in nulls being replaced by type defaults (e.g., `0` for integers, empty string for strings). Verify null behavior after ingestion.
Current onboarding flows set `nullHandlingEnabled: true` on the generated table config automatically, and when the source schema has nullable fields, the inferred Pinot schema marks them nullable and enables column-based null handling (`enableColumnBasedNullHandling`). Tables created on older builds may have null handling off — see [Advanced Guidance](/corecapabilities/external-table/operational-guidance).
### INT96 (Deprecated)
`INT96` has no column statistics in Parquet, which affects min/max pruning during segment generation. It should not be used as a `timeColumnName`.