Fetches data from a datasource.

​
Inputs

None.

​
Outputs

One output. No naming constraint for outputKey and outputName.

​
Parameters

namedescriptiondefault value
component.dataSourceThirdEye datasource to use.
component.queryQuery to execute. The query can use macros.

​
Example

{
  "name": "currentDataFetcher",
  "type": "DataFetcher",
  "params": {
    "component.dataSource": "pinot",     # ThirdEye datasource
    "component.query": "SELECT __timeGroup(timeColumn, 'EPOCH', 'P1D') as ts, sum(views) as met FROM pageviews WHERE __timeFilter(timeColumn, 'EPOCH') GROUP BY ts ORDER BY ts LIMIT 10000"
  },
  "inputs": [],
  "outputs": [
    {
      "outputKey": "currentData",        # no constraint
      "outputName": "currentOutput"      # no constraint
    }
  ]
}

​
Macros

Macros are special functions that are translated by ThirdEye to SQL before the query is sent to the database. Macros have access to detection runtime information, like the start and the end time of the detection, the timezone.
Macro exampledescription
__timeFilter( timeColumn, 'timeFormat' )Replaced by a time range filter on the timeframe [detectionStart, detectionEnd[ using the specified column name and the specified timeFormat. For example, __timeFilter(timeColumn, 'EPOCH_MILLIS') is replaced by timeColumn >= 1494410242000 AND timeColumn < 1494410242000)
__timeFilter( timeColumn, 'timeFormat,' 'lookbackFromStartPeriod' )Replaced by a time range filter on the timeframe [detectionStart-lookbackFromStartPeriod, detectionEnd[. For example, for a detection between January 14 and January 15, __timeFilter(timeColumn, 'yyyyMMdd', 'P7D') is replaced by timeColumn >= '20220107' AND timeColumn < '20220115')
__timeFilter( timeColumn, 'timeFormat,' 'lookbackFromStartPeriod,' 'lookbackFromEndPeriod' )Replaced by a time range filter on the timeframe [detectionStart-lookbackFromStartPeriod, detectionEnd-lookbackFromEndPeriod[. For example, for a detection between January 14 and January 15, __timeFilter(timeColumn, 'yyyyMMdd', 'P7D', 'P7D') is replaced by timeColumn >= '20220107' AND timeColumn < '20220108')
__timeGroup( timeColumn, 'timeFormat', 'granularity' )Replaced by a transformation of the timeColumn in bins of the given granularity, in milliseconds. Respects the timezone of the alert. For example, __timeGroup(timeColumn, 'yyyyMMddhh', 'P1D') is replaced by DATETIMECONVERT(timeColumn, '1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMddhh', '1:MILLISECONDS:EPOCH', '1:DAYS')

​
AUTO keyword

If the timeColumn value is AUTO, the macro resolves to the primary time column of the table.
timeFormat is obtained internally and can be left blank. For example:
  • __timeFilter(AUTO,'')
  • __timeGroup(AUTO,'','P1D')

​
timeFormat strings

Available timeformat strings are:
Stringdescriptionaliases
EPOCH_MILLIStime column is in milliseconds since unix epoch1:MILLISECONDS:EPOCH
EPOCHtime column is in seconds since unix epoch1:SECONDS:EPOCH
EPOCH_MINUTEStime column is in minutes since unix epoch1:MINUTES:EPOCH
EPOCH_HOURStime column is in hours since unix epoch1:HOURS:EPOCH
EPOCH_DAYStime column is in days since unix epoch1:DAYS:EPOCH
any SimpleDateFormat stringSee SimpleDateFormat documentation. Eg yyyyMMdd, yyyy-MM-dd'T'HH:mm:ss.SSSZ1:DAYS:SIMPLE_DATE_FORMAT:[SIMPLE_DATE_FORMAT]

​
period strings

Period strings uses the ISO-8601 format. Examples:
  • 'P7D': 7 days
  • 'PT1H': 1 hour