The more complicated an alert condition becomes, the harder it is to understand and debug (^1) The above is theory that applies well to simple metric monitoring, like CPU usage in computers, but it can be harder to apply for operations and business monitoring, the core use cases of ThirdEye. Business monitoring is full of complex patterns, special events and seasonalities (^2) When designing alerts, you will have to find your place on the quadrant below.
- Do not try to monitor everything. Think about what is important and what is actionable.
- Start simple and iterate(^3) to finetune your alert.
- Pareto principle applies. Root for the 80% before going for the very complex 20%. (^4)
Detection algorithms
Click on a detector name to go to its template documentation.
Threshold
Simplest method. Detect an anomaly if a metric is above a maximum threshold or below a minimum threshold. Good for signals that are mostly flat, or should not go in certain range.\
Mean Variance
Estimate the mean. Estimate the standard deviation, and consider the standard deviation is caused by noise only. If the value is abovemean + n*std or below mean + n*std, detects an anomaly.
Good if your signal is flat with a lot of noise.
Percentage change
Compare current timeseries to a baseline. If the percentage change is above a certain threshold, detects an anomaly. A simple way of managing seasonalities. You define the baseline yourself. A common usage is to compare your current value to the value of last weeks. For instance, compare Thursday 8, 8pm to Thursday 1, 8pm. This way, you manage hourly and weekly seasonalities. Most of the time, percentage rule is easy to understand, but because percentage rules use division, it can be sensitive with values close to 0. It can also be sensitive to noise: if you define a percentage change limit of 20%, and the noise is such that it is common to do -+10% around the mean, you will have false positives when you have -10% in the baseline followed by +10% in the observed value.
Pros: easiest way to manage seasonality. Easy to understand.Cons: Sensitive to noise(^5) Sensitive to big trends(^6)
Absolute change
Compares current time series to a baseline. If the absolute change is above a certain threshold, detect it as an anomaly. An alternative to percentage rule, that works better for noisy and small values.
Exponential time smoothing (ETS)
Exponential time smoothing (simple, double, Holt-Winters) methods(^7) are a set of statistical forecasting algorithm. Timeseries are extracted as a sum of trend, seasonality and noise(^8) The algorithm estimates these components. This algorithm performs well for daily data and hourly data. Holt-Winters method is faster than most other ML methods while keeping a very good short-term forecasting performance. The model detects an anomaly when the observed value is too far from the predicted value. The sensitivity can be finetuned.
Pros: manages seasonality, trend and noise.Cons: as a model that heavily relies on historical data, it is sensitive to concept drift, false trends(^9), special events, bad data, etc.
MSTL
The MSTL method(^10) decomposes a timeseries in trend, multiple seasonal patterns and residuals.Residuals can be seen as containing the noise and the outliers. If a residual goes outside the
expected residual range, it is considered an anomaly. The complete algorithm is an adaptation of the
R tsoutliers (^11) algorithm for online detection.\
Cons: Slower than ETS. Control on how the model behaves is limited.
Matrix Profile
The matrix profile method(^12) is a “direct” anomaly detection algorithm. It returns an anomaly score, it does not return upper and lower bounds. Points are detected as an anomaly if they exceed a given threshold. The matrix profile method only works well with seasonal patterns. It works well from minutely do daily data. The idea of the algorithm is to compare snippets of the timeseries between each other. If one snippet does not look like any other snippet, it has a high anomaly score.
Remote HTTP
The Remote HTTP detector allows the anomaly detection to be performed by a remote HTTP service. The user can configure the alert to point to a REST endpoint. This endpoint must be able to accept the thirdeye detection payload and respond back with a specific response API. With this detector, you can use your favorite models in any language like R or Python.Choosing the right algorithm
To pick the correct algorithm, you can follow the decision chart below. Make sure to look at your timeseries first. To display a timeseries in ThirdEye quickly, pick the threshold template. See How to create an alert. You can change the template easily, ThirdEye has an interactive feedback loop that makes it easy to try different configurations.Auomated Anomaly Detection Algorithm Recommendation and tuning (Based on data pattern during alert creation)
