Forward Block Index

​

Overview and Purpose

• Remote segment reads from cloud object storage.
• Large raw forward indexes with compressed chunks.
• Queries with selective filters followed by projection or expression evaluation.
• Columns where reading exactly the needed byte ranges is cheaper than scanning broad forward-index regions.

​

Availability

​

How the Index Works

1. Another predicate, index, or scan produces candidate document IDs.
2. StarTree Cloud groups those document IDs by forward block.
3. The Forward Block index identifies the byte ranges needed for those blocks.
4. The server can prefetch those byte ranges before decoding values.

​

Configuration

{
  "fieldConfigList": [
    {
      "name": "payload",
      "encodingType": "RAW",
      "indexes": {
        "forward": {
          "compressionCodec": "ZSTD"
        },
        "forwardBlock": {
          "blockSize": 1024
        }
      }
    }
  ]
}

​

Configuration Parameters

​

Query Examples

SELECT userId, payload
FROM clickstream
WHERE tenantId = 'acme'
  AND eventTimeMillis >= 1764556800000
LIMIT 100;

SELECT count(*)
FROM clickstream
WHERE tenantId = 'acme'
  AND regexp_like(payload, 'checkout')
  AND eventTimeMillis >= 1764556800000;

​

Tuning Guidance

• Use a smaller blockSize when filters are very selective and matching rows are scattered.
• Use a larger blockSize when scans usually touch adjacent rows and prefetch overhead matters more than precision.
• Start with 1024 when you do not have enough benchmark data, then compare latency and remote bytes read.

​

Performance Considerations

1. Best with remote storage: The benefit is largest when forward-index reads come from cloud object storage.
2. Complements other filters: Pair it with selective filters, inverted indexes, range indexes, Min/Max indexes, or sorted columns.
3. Adds metadata: Smaller blocks create more metadata. Measure index size when reducing blockSize.
4. Reload existing segments: After enabling the index, reload existing segments so StarTree Cloud builds the forward block metadata.