Anomaly detection with autoencoders

This log is a more in depth writeup of the same project I have, with self-contained Python examples to display the concepts I discuss.

Part 1: The core idea

When you are working with a large amount of sensor data, it stands to reason that manual flagging of anomalies is a huge issue of interest. Identifying an anomaly in the first place means identifying “normal” behavior for each type of sensor reading. Thus, we need something that can adapt this sense of normality per type of reading but also in a quick and efficient way so action can be taken before it’s too late.

An autoencoder is two networks chained together. The encoder compresses the input into a low-dimensional intermediate output (AKA the bottleneck). The decoder expands that code back to the original dimension. The network is trained to minimize the difference between input and reconstruction, ideally preserving the most valuable structure and patterns in the process.

Anomalies are then identified as those that don’t match the training distribution, so they either get lost in compression and show up later as reconstruction error (Chandola et al., 2009).

The simplest autoencoder is linear, such as PCA (Bishop, 2006). All you do is compress to one component, reconstruct, and then measure error:

Most normal samples here sit below the 95th percentile threshold. The anomaly (the red X at x=200, a point where every feature is 2–4 standard deviations from the mean) is well above this threshold. In fact, its reconstruction error is roughly 10x the median.

This is the idea in action: the linear autoencoder has learned the covariance structure of normal data, and anything that violates that structure cannot be easily reconstructed from what has been learned.

An encoder $f_\phi$ maps input $x \in \mathbb{R}^d$ to latent $z \in \mathbb{R}^p$ where $p \ll d$:

A decoder $g_\theta$ maps $z$ back to $\hat{x} \in \mathbb{R}^d$:

The bottleneck dimension $p$ controls the trade-off.

Set $p$ too small (e.g. 1 for a 100-dimensional input) and the autoencoder cannot capture enough structure. This means even points that truly are normal don’t show up as such.

Set $p$ too large and anomalies also reconstruct well, which goes against the whole point detection.

In practice, $p$ is often chosen by plotting reconstruction error on validation data and using the elbow heuristic (where the error stops dropping sharply).

We ultimately minimize mean squared error (MSE) between the input and reconstruction:

Part 2: Why not an LSTM?

The linear autoencoder treats each reading as independent. However, as these sensor readings are ultimately time-series data, this condition is far from ideal for our use case. We need to be able to reference “context” to determine whether something deviates from normal behavior.

This brings us to the potential application of the LSTM (Long Short Term Memory).

A brief on RNNs

LSTMs handle vanishing gradients through a gated cell state (Hochreiter & Schmidhuber, 1997):

Each gate controls what the LSTM remembers or forgets. The forget gate $f_t$ decides how much of the previous cell state to carry forward. The input gate $i_t$ decides how much new information to write in. The candidate cell state $\tilde{c}_t$ is the actual new information computed from the current input and previous hidden state. Together, $i_t$ and $\tilde{c}_t$ update the cell, and the output gate $o_t$ controls what gets exposed as the hidden state $h_t$.

LSTMs process sequentially, which gives us the context we desire but also means each timestep depends on the previous hidden state. For our project this was the main problem with the LSTM approach. We had a large amount of sensors, all recording data in parallel, running across countless operational trucks simultaneously. Training an LSTM on that volume meant days of sequential GPU time. Thus, we needed something that could ingest entire windows in parallel so we could batch data and make better use of our GPU resources.

Step 3: The transformer

The transformer replaces recurrence with attention. Instead of compressing the entire past into a single hidden state, it looks at every position in the window simultaneously and computes which positions matter most for reconstructing each point.

The transformer (Vaswani et al., 2017) eliminates recurrence. Scaled dot-product attention processes the entire sequence in parallel:

$Q$, $K$, $V$ are linear projections of the input into query, key, and value spaces. $QK^T$ measures pairwise similarity; $\sqrt{d_k}$ prevents softmax saturation.

Multi-head attention runs this in parallel $h$ times:

where:

In practice, this means the model can learn that a temperature spike at time $t$ is relevant to interpreting a pressure reading at some time $t+50$ without waiting for 50 sequential steps to propagate that information through a hidden state. This direct path is critical for anomaly detection because the signal of interest can affect sensors at arbitrary times and may require quick action.

For time series, this gives us a direct attention path between any two positions (Lim & Zohren, 2021), unlike LSTMs which propagate sequentially through a hidden state.

Positional encoding injects order information since the transformer has no built-in notion of sequence position (important component of our problem) (Vaswani et al., 2017):

Step 4: Anomaly detection pipeline

Ultimately, the full pipeline worked like this:

Raw sensor readings were grouped into non-overlapping windows of 128 timesteps across all channels. Each window was normalized per-channel using rolling statistics from the previous hour. The transformer encoder then compressed each window into a latent representation, and the decoder reconstructed the window from that bottleneck.

Reconstruction error per timestep was smoothed with a 3-step moving average, then compared against a per-channel threshold set at the 95th percentile of errors on a labeled normal-only validation set. Any block of 5 or so timesteps above the threshold triggered an alert.

For a sliding window $X = [x_1, \dots, x_T]$ with $x_t \in \mathbb{R}^d$:

Flag an anomaly at time $t$ if $e_t > \tau$, where $\tau$ is the 95th percentile of reconstruction errors on normal validation data.

That’s the entire autoencoder pipeline! Check out my project page if you’d like to see a more conversational, higher level explanation and learn more about my approach.