How Do You Measure Forecast Accuracy? Understanding MAPE, Bias and Tracking Signals

## The short answer

Forecast accuracy is measured by comparing forecasts to actuals over time, and you need at least two metrics to do it properly: an *error* metric that captures how far off you are (such as MAPE, the mean absolute percentage error) and a *bias* metric that captures whether you consistently over- or under-forecast. A forecast can have low error but persistent bias, or it can be unbiased on average yet wildly inaccurate in any given period. Measuring only one number hides one of these problems, so always track both.

## Why a single number isn't enough

Imagine two forecasters. One is off by 10% every month, always too high. The other is off by 10% too, but randomly high or low. By a simple error metric they look identical. Yet they are completely different problems: the first has a systematic flaw you can correct by adjusting an assumption; the second has noise you may not be able to reduce. Distinguishing between them is the whole point of measuring accuracy with more than one lens.

## The core metrics

### MAPE - mean absolute percentage error

MAPE averages the absolute percentage difference between forecast and actual across periods. It's popular because it's intuitive and unit-free: "our forecasts are off by about 8% on average" is easy to communicate. Its weaknesses are worth knowing:

- It breaks down when actuals are near zero, because dividing by a tiny number inflates the error.

- It penalises over-forecasting and under-forecasting asymmetrically.

- It can be dominated by a few low-volume items with large percentage swings.

### MAE and RMSE - absolute and squared errors

Mean absolute error (MAE) measures average error in the original units, which is useful when percentage terms mislead. Root mean squared error (RMSE) squares errors before averaging, so it punishes large misses more heavily - handy when occasional big errors are far more costly than many small ones.

### Bias - the direction of error

Bias measures whether errors net out to zero or lean one way. A simple approach is the mean error (forecast minus actual, *without* taking the absolute value). If it's consistently positive, you over-forecast; consistently negative, you under-forecast. Bias is the most actionable metric because systematic bias usually traces to a fixable assumption - an optimistic conversion rate, an outdated seasonality factor.

### Tracking signal

A tracking signal accumulates bias over time relative to typical error size. When it drifts beyond a threshold, it warns you that the forecast has gone systematically off - a useful automated trigger to re-examine the model rather than waiting for a quarterly review.

## Which metric to use when

- **Communicating to non-specialists?** MAPE, because percentages are intuitive.

- **Items with volumes near zero?** Avoid MAPE; use MAE or a scaled error metric.

- **Big misses much costlier than small ones?** RMSE, which weights them heavily.

- **Suspect systematic over- or under-forecasting?** Mean error and a tracking signal.

In practice you report a small dashboard - an error metric plus bias - rather than crowning a single champion metric.

## Set a baseline before you celebrate

An 8% MAPE means nothing in isolation. Always compare against a naive baseline: "next period equals this period", or "same as last year". If your sophisticated model can't beat a naive forecast, the sophistication is adding cost without value. The right question is never "is our accuracy good?" but "is it better than the simplest alternative, by enough to justify the effort?"

## Measure at the right level

Accuracy aggregates deceptively. Errors at the line-item level often cancel out when summed, so a total forecast can look accurate while every component is wrong - the misses simply offset each other. Decide which level matters for your decisions and measure there. If you allocate inventory per product, per-product accuracy is what counts, not the flattering total.

## Build accuracy tracking into the process

Measuring accuracy once is an audit; measuring it continuously is a feedback loop. The most valuable practice is logging each forecast and comparing it to actuals as they land, so error and bias trends are visible over time. That history tells you whether the model is improving, which drivers are consistently mis-estimated, and when a structural break has occurred. Maintaining this loop systematically across many forecasts and units is exactly where enterprise tooling helps, and an area where neart.ai builds enterprise-grade products - turning accuracy measurement from a manual chore into an automatic part of the forecasting cycle.

## Don't forget scenario forecasts

Scenario forecasts need a different accuracy lens. You can't fairly grade a downside scenario against actuals if the downside didn't occur. Instead, judge scenario forecasts on calibration: when reality lands, was it within the range your scenarios spanned? If actuals repeatedly fall outside your whole scenario set, your range is too narrow - a more important finding than any single point error.

## Practical takeaway

Never judge a forecast by one number. Pair an error metric (MAPE, MAE or RMSE depending on your data) with a bias measure and a tracking signal, always compare against a naive baseline, and measure at the decision-relevant level. Log every forecast against actuals so trends in error and bias surface early. Accuracy is a feedback loop, not a scorecard - and the loop is what makes the next forecast better.