Set up Tiered Storage with Cloud Object Store

​

Set the segment age boundary

​

Set up a tenant

• (Recommended) Set up a server as a dedicated tenant
• (Default) Use the same server as your table as your tenant. We do not recommend this option for production environments.

​

Set up a dedicated tenant server (Recommended)

1. Add servers. You’ll need at least as many servers as in your table replication setting in the Pinot table configuration.
2. Tag these as RemoteTenant_OFFLINE (or whatever you prefer) using the /instances/{instanceName}/updateTags API in the Swagger UI on the host where Pinot is running. This link is only accessible when Pinot is running.

​

Use same tenant as your table (Default)

​

Add cluster configs and restart servers

• Add the configs as part of the server instance configs or
• From the Swagger UI on the host where Pinot is running, use the POST /cluster/configs API to add the following cluster-level tier storage configurations.

{
  "pinot.server.instance.tier.enabled": "true",
  "pinot.server.instance.segment.cache.directory": "/home/pinot/data/tieredStorage/segmentCache/",
  "pinot.server.instance.buffer.reference.manager.ondemand.total.size": "24G",
  "pinot.server.instance.buffer.reference.manager.ondemand.reserved.size": "4G"
}

​

Update the Pinot table configuration

1. If not already set, set in the Pinot table configuration.
2. Add tierConfigs to your Pinot table configuration (example below). The segmentAge will be calculated using the primary timeColumnName as set in Pinot table configuration.

"tierConfigs": [
      {
        "name": "remoteTier",
        "segmentSelectorType": "time",
        "segmentAge": "7d",
        "storageType": "pinot_server",
        "serverTag": "RemoteTenant_OFFLINE",
        "tierBackend": "s3",
        "tierBackendProperties": {
             "pathPrefix": "<pathPrefix>",
             "region": "<region>",
             "bucket": "<bucket>",
             ...
        }
      }
    ],

​

Tiered storage with Google Cloud Storage (GCS)

1. To use GCS’s interoperable APIs with a S3 client, you must generate HMAC keys for the GCP bucket. Each HMAC key consists of an access ID and a secret key and can be associated with any service account.

$ gsutil hmac create <SERVICE_ACCOUNT_EMAIL>
AccessId: <HMAC access id>
SecretKey: <HMAC secret key>

2. For testing purposes, you can store the HMAC key pair in tierBackendProperties, as shown in the following example:

"tierBackendProperties": {
  "bucket": "<gcs bucket name>",
  "endpoint": "https://storage.googleapis.com",
  "region": "auto",
  "accesskey": "<HMAC access id>",
  "secretkey": "<HMAC secret key>",
  ...
}

1. Enable the GCP secret manager API, and then add the key pair as secrets (one for the access ID and the other for the secret key).
2. Ensure that the service account used to create the cluster has access to read the added secrets.
3. Set the following properties in tierBackendProperties:
   Copy

   Ask AI

   "tierBackendProperties": {
     "bucket": "<gcs bucket name>",
     "endpoint": "https://storage.googleapis.com",
     "accesskey": "<secret namae for HMAC AccessId>",
     "secretkey": "<secret namae for HMAC SccessKey>",
     "region": "auto",
     "keyType": "SECRET",
     "secretmanagertype": "GCS",
     "gcpprojectid": "<projectId>",
     "gcpkeypath": "<path to key.json on the server>",
     ...
     }
   

   The gcpprojectid and gcpkeypath are required to be able to access the GCP secret manager. The gcpkeypath should be set to the absolute path where the gcp key is mounted on the server (typically /home/pinot/gcp/credentials.json).

​

Upload new segments and migrate existing segments

​

Validate and query

1. From the Swagger UI on the host where Pinot is running use the segments/{tableName}/tiers API to check segments’ current tier
2. Segments are not present on the pinot-server’s local disk (in dataDir and indexDir)
3. Segments are present in the remote tier backend in uncompressed form

​

Fine tunings for query speed

​

Preloaded index

1. To enable the buffer preloading, use cluster config apis to add the following option to your cluster configuration:

{
   "pinot.server.instance.buffer.reference.manager.preload.enable": "true",
   "pinot.server.instance.buffer.reference.manager.preload.total.size": "100G",
   "pinot.server.instance.buffer.reference.manager.preload.dir": "/home/pinot/data/tieredStorage/preload/",
   "pinot.server.instance.buffer.reference.manager.preload.load.existing.buffers": "true",
   "pinot.server.instance.buffer.reference.manager.preload.index.keys.override": "<tableNameWithType1>,col1.bloom_filter,col2.sparse;<tableNameWithType2>,startree_index_0.tree"
}

2. Once enabled, set global buffer preloading using the following cluster configuration:

{
   "pinot.server.instance.buffer.reference.manager.preload.index.keys": "*.bloom_filter,col2.sparse,startree_index_0.tree"
}

3. To override the global cluster level configuration, specify the configuration option on the table level:

• To specify the override in the cluster configuration, set the pinot.server.instance.buffer.reference.manager.preload.index.keys.override cluster configuration option as follows:

{
   "pinot.server.instance.buffer.reference.manager.preload.index.keys.override": "<tableNameWithType1>,*.bloom_filter,col2.sparse;<tableNameWithType2>,startree_index_0.tree"
}

• To specify the override in the table configuration, set the preload.index.keys.override property inside tierBackendProperties as follows:

"tierConfigs": [
      {
        "name": "remoteTier",
         ...
        "tierBackendProperties": {
             "region": "<region>",
             "bucket": "<bucket>",
             "preload.index.keys.override": "*.bloom_filter,col2.sparse,col3.inverted_index"
             ...
        }
      }
    ],

4. Both clusterConfig and tableConfig changes for preloading index as said above can be applied by reloading segments. Btw, restarting servers also works if wanted.

​

Automatic caching (Experimental)

{
   "pinot.server.instance.buffer.reference.manager.mmap.enable": "true",
   "pinot.server.instance.buffer.reference.manager.mmap.total.size": "100G",
   "pinot.server.instance.buffer.reference.manager.mmap.dir": "/home/pinot/data/tieredStorage/mmap/",
   "pinot.server.instance.buffer.reference.manager.mmap.load.existing.buffers": "true"
}

​

On demand access

{
  "pinot.server.instance.buffer.reference.manager.ondemand.total.size": "24G",
  "pinot.server.instance.buffer.reference.manager.ondemand.reserved.size": "4G"
}

​

Access methods

SET "readAhead.enable" = true;
SET "prefetch.column.index.list.processing.phase"='col1.dictionary,col2.dictionary';
SET "readAhead.column.index.list.processing.phase"='col3.dictionary.initialBytes=1K;maxBytes=1M;numTriesBeforeDouble=10,teamID.dictionary.downloadAll=true';
SELECT ....

1. For columns in non-predicate clauses (such as projection, groupBy, and orderBy): if they are dictionary encoded, their dictionaries are prefetched; but in any case, their forward index data is accessed via the readAhead method, because only part of the data is needed to evaluate the clauses.
2. For columns in predicate clauses (such as a where clause): if they have no index to be used to evaluate the predicates, their forward index data is prefetched as a whole, because the whole forward index data is scanned to evaluate the predicates anyway; otherwise, their index data (like an inverted index, range index or json index) is accessed via the readAhead method, because only part of them is needed to evaluate the predicates.

​

Star-tree index

"tierConfigs": [
      {
        "name": "remoteTier",
         ...
        "tierBackend": "s3",
        "tierBackendProperties": {
             "region": "<region>",
             "bucket": "<bucket>",
             "enable.startree.index": "true"
             ...
        }
      }
    ],

{
   "pinot.server.instance.buffer.reference.manager.preload.index.keys.override": "<tableNameWithType1>,startree_index_0.tree,startree_index_1.tree"
}

​

Tier overwrites

​

S3 client configs

​

Configs for ease of operations

​

Server restart optimizations

​

Minion Tasks

​

Note

    "task": {
      "taskTypeConfigsMap": {
        "SegmentRefreshTask": {
          "directUploadToTier": "true"
        }
      }
    }

    "task": {
      "taskTypeConfigsMap": {
        "StarTreeAlterTableTask": {
          "directUploadToTier": "true"
        }
      }
    }

curl -X POST "http://<CONTROLLER_URL>/tasks/schedule?taskType=SegmentRefreshTask&tableName=<TABLE_NAME>" -H "accept: application/json"

​

Stale segment directory cleanup

{
  "controller.tieredStorage.segment.cleanup.frequencyPeriod": "1h"
}

​

Monitoring metrics

• Segments get uploaded to the tier backend, such as data volume and operation duration, rates or failures
• Queries fetch data from the remote segments, such as data volume and query duration breakdown over query operators, query rates or failures
• Segment data is kept on servers temporarily, such as data volume from segments temporarily held in memory or on disk