Text Index (Native)

​

Overview and Purpose

A native text index is an experimental indexing solution, built from the ground up specifically for Apache Pinot, that accelerates text search operations without relying on external libraries like Lucene. It provides optimized performance for the most common text search patterns while reducing storage requirements.

In StarTree Cloud (powered by Apache Pinot), native text indexes are designed to address the specific text search needs of OLAP systems rather than implementing a full text search DSL like traditional search engines. These indexes are particularly valuable for:

• Prefix wildcard queries (like “pino*”)
• Suffix wildcard queries (like “*inot”)
• Term queries (like “pinot”)
• Real-time text search on streaming data
• Applications requiring lower storage overhead for text search capabilities

​

How the Index Works

​

Core Concepts

Traditional text indexing in Pinot uses Lucene as a sidecar to the main Pinot segments. While effective, this approach limits optimization possibilities for Pinot-specific use cases.

The native text index in StarTree Cloud:

1. Custom Indexing Engine: Built from the ground up specifically for Pinot’s access patterns and workloads.
2. Integration with Inverted Indexes: Leverages Pinot’s powerful inverted index capabilities rather than relying on external libraries.
3. Optimized for Common Patterns: Specialized for the most frequent text search patterns in SQL-based systems rather than supporting a full text search DSL.
4. Real-time Capability: Supports concurrent indexing and searching, enabling true real-time text search on streaming data.

​

Key Benefits

1. Performance: Runs 80-120% faster than Lucene-based indexes for common text search patterns.
2. Storage Efficiency: Requires approximately 40% less disk space compared to Lucene-based indexes.
3. Real-time Text Search: Unlike traditional text indexes that require sealing before searching, native text indexes support concurrent indexing and searching.

​

Configuration

​

Enabling Native Text Index

To enable a native text index on a column in your StarTree Cloud table, add the following configuration to your table definition:

{
  "fieldConfigList": [
    {
      "name": "description",
      "encodingType": "RAW",
      "indexTypes": ["TEXT"],
      "properties": {
        "fstType": "native"
      }
    }
  ]
}

​

Important Configuration Considerations

1. Experimental Status: The native text index is currently experimental and recommended only for testing environments.
2. Column Encoding: Like the standard text index, native text indexes work with RAW encoded columns (not dictionary-encoded).
3. Real-time Tables: Native text indexes are particularly beneficial for real-time tables where concurrent indexing and searching is required.

​

Performance Considerations

1. Experimental Status: As an experimental feature, performance characteristics may change in future releases.
2. Use Case Fit: Native text indexes excel at common SQL text search patterns (prefix, suffix, term) but may not support all features of full-text search engines.
3. Real-time Performance: For real-time tables with text search requirements, native text indexes offer true real-time capabilities without the near-real-time limitation of traditional approaches.
4. Storage Benefits: The 40% reduction in index size can be significant for large text columns or tables with many text-indexed columns.
5. Development Trajectory: As this feature matures, it’s expected to eventually replace the Lucene-based approach as the recommended text index for StarTree Cloud.