Data Type Mapping

This feature is available starting in StarTree release 0.15.0. It must be enabled on demand — contact your StarTree representative to have it activated for your environment.

When you onboard an External Table, StarTree infers a Pinot schema from the source metadata. The mapping depends on the source:

Raw S3 Parquet (S3 Data Lake, catalogType=s3) — inferred from the Parquet file’s physical and logical types.
Iceberg (AWS Glue / S3 Tables, catalogType=iceberg-rest) — inferred from the Iceberg table schema.

Numeric columns (INT, LONG, FLOAT, DOUBLE, BIG_DECIMAL) are inferred as metrics; everything else (BOOLEAN, STRING, JSON, BYTES, time columns) as dimensions. You can adjust this in the schema before creating the table.

Raw S3 Parquet → Pinot

For the S3 Data Lake source. Types come from the Parquet file metadata.

Primitive Types

Parquet Type	Pinot Type	Notes
`BOOLEAN`	`BOOLEAN`	Optional and required variants both supported.
`INT32`	`INT`
`INT64`	`LONG`
`FLOAT`	`FLOAT`
`DOUBLE`	`DOUBLE`
`BINARY`	`STRING`	Raw binary data is readable as a string.
`FIXED_LEN_BYTE_ARRAY`	`STRING`	Fixed-width byte arrays are stored as strings.
`INT96`	`TIMESTAMP`	Deprecated Parquet type. Converted to milliseconds since epoch; no column statistics available. Avoid using as a time column.

Logical Types

Integer Subtypes

Parquet Logical Type	Pinot Type	Notes
`INT(8, signed)`	`INT`	Range: -128 to 127
`INT(16, signed)`	`INT`	Range: -32,768 to 32,767
`INT(32, signed)`	`INT`
`INT(64, signed)`	`LONG`
`INT(8, unsigned)`	`INT`	Range: 0 to 255
`INT(16, unsigned)`	`INT`	Range: 0 to 65,535
`INT(32, unsigned)`	`LONG`	Known limitation: values are stored as signed. Values above `2,147,483,647` may be corrupted. See Unsigned Integer Limitation.
`INT(64, unsigned)`	`LONG`	Known limitation: same signed-storage issue. Values above `Long.MAX_VALUE` are not preserved accurately.

String and Binary Types

Parquet Logical Type	Pinot Type	Notes
`STRING`	`STRING`	UTF-8 encoded strings.
`ENUM`	`STRING`	Enum values stored as strings.
`JSON`	`JSON`	Parsed into Pinot’s native `JSON` type, not preserved as a plain string.
`BSON`	`STRING`	Binary data is decoded as a UTF-8 string, not preserved as a distinct binary type. Non-UTF8-safe byte sequences may not round-trip cleanly — verify BSON columns after ingestion.
`UUID`	`STRING`	Known limitation: the raw 16-byte binary is decoded with a plain UTF-8 string conversion rather than a proper UUID formatter, which can produce garbled values or serialization errors. Verify UUID columns render correctly after ingestion.
`INTERVAL`	`STRING`
`GEOMETRY`, `GEOGRAPHY`	`BYTES`	Stored as raw bytes; no geospatial functions or H3 index are available on external tables.

Temporal Types

Parquet Logical Type	Pinot Type	Notes
`DATE`	`LONG`	Stored as epoch days.
`TIME(MILLIS)`	`LONG`	Milliseconds since midnight; range 0 to 86,399,999.
`TIME(MICROS)`	`LONG`	Microseconds since midnight; range 0 to 86,399,999,999.
`TIME(NANOS)`	`LONG`	Nanoseconds since midnight.
`TIMESTAMP(MILLIS, UTC)`	`TIMESTAMP`
`TIMESTAMP(MILLIS, local)`	`TIMESTAMP`
`TIMESTAMP(MICROS, UTC)`	`TIMESTAMP`	Native microsecond granularity is preserved (`EPOCH\|MICROSECONDS`) — see Granularity.
`TIMESTAMP(MICROS, local)`	`TIMESTAMP`	Native microsecond granularity is preserved — see Granularity.
`TIMESTAMP(NANOS, UTC)`	`TIMESTAMP`	Native nanosecond granularity is preserved (`EPOCH\|NANOSECONDS`) — see Granularity.
`TIMESTAMP(NANOS, local)`	`TIMESTAMP`	Native nanosecond granularity is preserved — see Granularity.

Decimal Types

All Parquet DECIMAL variants map to Pinot BIG_DECIMAL, preserving scale and precision.

Parquet Physical + Logical Type	Pinot Type	Example
`INT32` + `DECIMAL(p, s)`	`BIG_DECIMAL`	`DECIMAL(9, 2)`
`INT64` + `DECIMAL(p, s)`	`BIG_DECIMAL`	`DECIMAL(18, 4)`
`FIXED_LEN_BYTE_ARRAY` + `DECIMAL(p, s)`	`BIG_DECIMAL`	`DECIMAL(28, 6)`
`BINARY` + `DECIMAL(p, s)`	`BIG_DECIMAL`	`DECIMAL(38, 10)`

Complex Types

Source type	Pinot type	Notes
`LIST` / `ARRAY` of a primitive	Multi-value column of the element type	Element types: `INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`, `BOOLEAN`. A list of timestamps is stored as multi-value `LONG`.
`LIST` of a complex element	`JSON`	Lists of structs / maps / nested lists are serialized to a JSON string.
`MAP<STRING, primitive>`	`MAP`	Native map for primitive values (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`).
`MAP` (other shapes)	`JSON`	Non-string keys or complex values are serialized to JSON.
`STRUCT` / nested group	`JSON`	Serialized to a JSON string; query with JSON functions.

To use JSON_MATCH or JSON_EXTRACT_SCALAR on complex columns, you must enable a JSON index on those columns. Without it, queries will fail with a Cannot apply JSON_MATCH on column without json index error. Configure the index in the table’s index settings.

Multi-value columns are capped at 1,000 entries per row — the segment metadata is written with a fixed upper bound, and rows whose source array is longer are silently truncated at query time. If your lists can exceed 1,000 elements, keep the column as JSON instead.

Iceberg → Pinot

For the AWS Glue and AWS S3 Tables sources. Types come from the Iceberg table schema.

Iceberg type	Pinot type	Notes
`boolean`	`BOOLEAN`
`int`	`INT`
`long`	`LONG`
`float`	`FLOAT`
`double`	`DOUBLE`
`decimal(p, s)`	`BIG_DECIMAL`	Precision and scale preserved.
`string`	`STRING`
`uuid`	`STRING`
`binary`, `fixed`	`STRING`
`date`	`LONG`	`EPOCH	DAYS`.
`time`	`LONG`	Stored as the raw long; not a time-column candidate.
`timestamp`, `timestamptz`	`TIMESTAMP`	Granularity inferred from the type — see note below.
`timestamp_ns`, `timestamptz_ns`	`TIMESTAMP`	`EPOCH	NANOSECONDS`.
`struct`	`JSON`	Serialized to a JSON string.
`list<primitive>`	Multi-value column of the element type	`INT`, `LONG`, `FLOAT`, `DOUBLE`, `STRING`, `BOOLEAN`; `list<timestamp>` becomes multi-value `LONG`.
`list<complex>`	`JSON`
`map<string, primitive>`	`MAP`
`map` (other shapes)	`JSON`
(unrecognized)	`STRING`	Falls back to `STRING`.

Timestamp granularity is inferred from the Iceberg type (current releases):

timestamp_ns / timestamptz_ns → EPOCH|NANOSECONDS
timestamp(p) with an explicit precision → milliseconds for p ≤ 3, microseconds for p ≤ 6, nanoseconds above that
plain timestamp / timestamptz → EPOCH|MICROSECONDS, matching Iceberg’s on-disk microsecond storage

Older 0.16.0 builds inferred plain timestamp as EPOCH|MILLISECONDS, which mislabeled microsecond data (values ~1000× off). Tables onboarded on those builds keep their inferred format — set the time column’s granularity explicitly if it doesn’t match the data. See Time column.

Time column

A Pinot table can designate one column as its primary time column — a DateTimeFieldSpec that records the event time of each row. See the Apache Pinot DateTimeFieldSpec reference for the general schema requirement and format syntax.

How the time column is used in an External Table

Segment pruning — a query with a time filter skips segments whose time range can’t match, so fewer remote files are read.
Retention — if a retention period is configured on the table, segments older than it are dropped.

The time column is optional: without one, the table is still created and fully queryable, but loses time-based segment pruning and retention.

How the time column is detected

During onboarding, after the schema is inferred, StarTree selects a time column automatically — choosing, in order:

The first non-nullable millisecond-granularity datetime column.
Otherwise, the first millisecond-granularity datetime column (nullable allowed).
Otherwise, the first datetime column of any unit.

Only columns that map to a Pinot DateTimeFieldSpec are candidates — TIMESTAMP, DATE, and (raw Parquet) INT96. A plain integer column holding an epoch value is not auto-detected; set it as the time column manually if needed. If the source has no timestamp/date column, no time column is set; you can assign one in the schema before creating the table.

Granularity

For raw Parquet, the inferred DateTimeFieldSpec preserves the source unit:

Source timestamp	Inferred format	Granularity
`INT96` (legacy)	`EPOCH\|MILLISECONDS`	`1:MILLISECONDS`
`TIMESTAMP(millis)`	`EPOCH\|MILLISECONDS`	`1:MILLISECONDS`
`TIMESTAMP(micros)`	`EPOCH\|MICROSECONDS`	`1:MICROSECONDS`
`TIMESTAMP(nanos)`	`EPOCH\|NANOSECONDS`	`1:NANOSECONDS`
`DATE`	`EPOCH\|DAYS`	`1:DAYS`

For Iceberg, the granularity follows the type: timestamp_ns → nanoseconds, timestamp(p) → from the precision, and plain timestamp / timestamptz → microseconds (older 0.16.0 builds inferred milliseconds — see the note in the Iceberg → Pinot section above).

Values are stored in the source unit — there is no implicit downscaling — so account for the unit in time-range queries. If the inferred granularity doesn’t match what the underlying Parquet actually stores (for example, a table onboarded on an older build), set the time column’s granularity explicitly.

Known Limitations

Unsigned Integer Types

Parquet UINT32 and UINT64 (unsigned 32-bit and 64-bit integers) are stored in Pinot as a signed LONG. This means:

Values that fit within the signed range are accurate.
Values above 2,147,483,647 (for UINT32) are stored as negative numbers or wrap around, silently corrupting approximately half the data.

If your dataset contains unsigned integers larger than INT32_MAX, treat the Pinot LONG values with caution or apply a transformation at ingestion time.

Null Handling in Complex Types

MAP columns that contain null values in keys or values may not propagate null bitmaps correctly to Pinot segments. This can result in nulls being replaced by type defaults (e.g., 0 for integers, empty string for strings). Verify null behavior after ingestion. Current onboarding flows set nullHandlingEnabled: true on the generated table config automatically, and when the source schema has nullable fields, the inferred Pinot schema marks them nullable and enables column-based null handling (enableColumnBasedNullHandling). Tables created on older builds may have null handling off — see Advanced Guidance.

INT96 (Deprecated)

INT96 has no column statistics in Parquet, which affects min/max pruning during segment generation. It should not be used as a timeColumnName.

Get Started

Ingestion

External Tables

Query Data

Manage Data

Observability

Visualize Data

Cluster Operations

Manage Security

Release Notes

Reference

Raw S3 Parquet → Pinot

Primitive Types

Logical Types

Integer Subtypes

String and Binary Types

Temporal Types

Decimal Types

Complex Types

Iceberg → Pinot

Time column

How the time column is used in an External Table

How the time column is detected

Granularity

Known Limitations

Unsigned Integer Types

Null Handling in Complex Types

INT96 (Deprecated)

​Raw S3 Parquet → Pinot

​Primitive Types

​Logical Types

​Integer Subtypes

​String and Binary Types

​Temporal Types

​Decimal Types

​Complex Types

​Iceberg → Pinot

​Time column

​How the time column is used in an External Table

​How the time column is detected

​Granularity

​Known Limitations

​Unsigned Integer Types

​Null Handling in Complex Types

​INT96 (Deprecated)

Raw S3 Parquet → Pinot

Primitive Types

Logical Types

Integer Subtypes

String and Binary Types

Temporal Types

Decimal Types

Complex Types

Iceberg → Pinot

Time column

How the time column is used in an External Table

How the time column is detected

Granularity

Known Limitations

Unsigned Integer Types

Null Handling in Complex Types

INT96 (Deprecated)