Beyond Accuracy: Choosing the Right Evaluation Metric for Your Time Series Forecast

8th April 2022

In forecasting, “accuracy” is not a single, simple concept. A model that looks great on paper can fail spectacularly in practice if measured against the wrong benchmark. The evaluation metric you choose is more than a final score; it’s the compass that guides your modeling process and defines what “good” means for your specific business problem. Understanding these metrics helps us make better decisions, invest time wisely, and produce forecasts that users can trust without needing manual adjustments.

Why One Metric Doesn’t Fit All

Imagine you are forecasting daily product sales in two scenarios:

• Scenario A: Your forecast is off by 10 units every single day.
• Scenario B: Your forecast is perfect for 29 days but is off by 300 units on one day.

 

Which forecast is better? The answer depends entirely on your business costs. For a low-cost item like coffee, Scenario A’s consistent small errors might be acceptable. But for a high-cost item like a laptop, Scenario B’s single massive error could create a huge, expensive inventory problem. The metric you choose formalizes this business logic, translating your operational priorities into a mathematical objective.

The Main Families of Forecasting Metrics

Forecasting metrics generally fall into three categories based on how they handle the scale of your data. Each has its own “personality,” including its sensitivity to change, stability, and interpretability.

1. Scale-Dependent Errors

These metrics express error in the same units as the data (e.g., dollars, units sold). They are easy to interpret but cannot be used to compare forecasts across datasets with different scales.

• Mean Absolute Error (MAE): The average of the absolute differences between forecasted and actual values, MAE tells you, on average, how far off your predictions are.

• Best For: Cases where the cost of an error increases linearly (e.g., being off by 20 units is exactly twice as bad as being off by 10).
• Limitations: MAE doesn’t heavily penalize large, occasional errors and has lower sensitivity to outliers.
• Root Mean Squared Error (RMSE): The square root of the average of squared errors, RMSE penalizes larger errors much more heavily than smaller ones.

• Best For: Situations where large errors are disproportionately costly, like forecasting electricity demand where a major under-forecast could cause a blackout.
• Limitations: RMSE is very sensitive to outliers. A single “bad day” in noisy data can make a good model look terrible, making the metric less stable.

 

2. Percentage Errors

These scale-independent metrics express errors as a percentage, which is useful for comparing forecast performance across different time series.

• Mean Absolute Percentage Error (MAPE): Calculates the average absolute error as a percentage of the actual value.

• Best For: Presenting forecast accuracy to business stakeholders in an intuitive format.
• Limitations: MAPE is undefined when the actual value is zero, making it unusable for data with intermittent demand. It is also biased, as it penalizes forecasts that are higher than the actuals more heavily than those that are lower.

• Accuracy Scaled by the Mean: An alternative that scales the MAE by the mean of the actual values, avoiding MAPE’s division-by-zero issue.

• Best For: Creating a percentage-based score that is more stable than MAPE when dealing with low-volume or zero-value data.
• Limitations: This metric can become negative if the absolute error is very large, and its interpretability can be tricky.

 

3. Scale-Free Relative Errors

This class of metrics compares a forecast’s error to the error of a simple benchmark, like a naive seasonal model.

• Mean Absolute Scaled Error (MASE): Measures the forecast’s MAE relative to the MAE of a naive forecast.

• A MASE value < 1 means your model is better than the benchmark.
• A MASE value > 1 means your model is worse.
• Best For: Almost any time series task, especially with data that has strong seasonality or intermittency. It is robust and avoids MAPE’s drawbacks.
• Limitations: It can be more complex to calculate and may be less familiar to business stakeholders.

 

The Aggregation Trap: Why High-Level Accuracy Can Be Deceiving

A critical, often overlooked aspect of evaluation is the level of aggregation. The same predictions can yield vastly different accuracy scores depending on how you summarize them.

• Global Accuracy: Calculated by summing all predictions and actuals before computing the error on the totals. This method is highly misleading because positive and negative errors can cancel each other out, hiding poor granular performance.
• Standard Accuracy (Row-wise): The standard approach, where the absolute error is calculated for each data point before being averaged. This is a more honest measure because errors cannot cancel out.
• Average of Accuracies: Involves calculating a metric like MAPE for each individual time series (e.g., each store) and then averaging the scores. This is the most detailed method but is also the most sensitive to errors on series with small values.

 

High-level metrics can mask underlying problems. For example, inventory item forecast in retail use case might have 97% daily accuracy when viewed in aggregate, but only 77% accuracy at the individual store level. This lower granular accuracy means the store-specific predictions are unreliable for critical tasks like inventory planning, which can lead to stockouts or overstocking. As a result, managers cannot trust the forecast and are forced to make frequent manual adjustments to correct inventory orders.

Practical Considerations

Before calculating any metric, the data must be handled correctly.

• Handling Sparse Data: Real-world data is often sparse. If your model predicts values for every hour but you only have actuals for hours when a sale occurred, you must first fill in the missing timestamps in your actual data with zeros to evaluate properly. You should not evaluate accuracy only for the times something happened.
• Defining the Test Period: Accuracy must be computed on a “test” period – data the model was not trained on. For example, train a model on data from January to June, then test its forecasts for July and August against the known actuals.
• Filtering and Alignment: Ensure your prediction and actuals datasets cover the same locations and time periods before calculating accuracy.

 

Python Simulation: Metrics in Action

Let’s simulate two scenarios to see how different aggregation methods tell different stories.

Step 1: Setup and Data Creation

We will test two cases: symmetric errors (model is consistently off by 10%) and a scenario with an outlier in the actuals.

import numpy as np

# Scenario 1: Symmetric Errors 

y_true_symmetric = np.array([100, 100, 100, 100, 100, 100, 100, 100, 100, 100]) 

y_pred_symmetric = np.array([110, 90, 110, 90, 110, 90, 110, 90, 110, 90]) 

# Scenario 2: One Massive Outlier in Actuals 

y_true_outlier = np.array([1, 100, 100, 100, 100, 100, 100, 100, 100, 100]) 

y_pred_outlier = np.array([110, 90, 110, 90, 110, 90, 110, 90, 110, 90]) 

Step 2: Define the Accuracy Functions

We’ll define our three accuracy calculation methods.

# Definition 1: Global Accuracy 

# Sums everything first. Can be misleading as errors cancel out. 

def global_accuracy(y_true, y_pred):

    return 100 – 100 * np.abs(y_true.sum() – y_pred.sum()) / np.sum(y_true) 

# Definition 2: Standard Accuracy (Row-wise)

# Calculates absolute error for each point before summing. More robust.

def standard_accuracy(y_true, y_pred):

    return 100 – 100 * np.sum(np.abs(y_true – y_pred)) / np.sum(y_true) 

# Definition 3: Average of Row-wise Accuracies [cite: 105]

# Averages the accuracy of each point. Highly sensitive to small actuals. [cite: 106]

def average_of_accuracies(y_true, y_pred):

    # To prevent division by zero, replace any true zeros with 1.     y_true_safe = np.where(y_true == 0, 1, y_true) 

    accuracies = 100 – 100 * np.abs(y_true – y_pred) / y_true_safe 

    return np.mean(accuracies) 

Step 3: Analyze Results

• Analysis of Scenario 1: Symmetric Errors
  The model is off by 10% on every data point, but the errors are balanced.

— Scenario 1: Symmetric Errors —

Global Accuracy: 100.0%

Standard Accuracy: 90.0%

Average of Accuracies: 90.0%

• Insight: The Global Accuracy is a dangerously misleading 100% because the over- and under-predictions cancel each other out. The other two metrics correctly report 90% accuracy, reflecting the consistent 10% error. This is a classic example of the aggregation trap.
• Analysis of Scenario 2: Outlier in Actuals
  Here, one data point has an unusually small “actual” value.

— Scenario 2: Outlier in Actuals —

Global Accuracy: 89.0%

Standard Accuracy: 77.9%

Average of Accuracies: -999.0%

• Insight: The Average of Accuracies tanks to -999.0%. This is because the first data point (actual=1, prediction=110) has an enormous percentage error that dominates the average. The Standard Accuracy is more stable, providing a reasonable score of 77.9%. This shows the “Average of Accuracies” method is highly unstable and should not be used with low or zero actual values.

 

How to Choose Your Metric

This simulation shows that no single metric tells the whole story. Use this checklist to select the right metric for your needs:

• What is the business cost of errors? Are large errors a disaster (use RMSE) or are they proportional to their size (use MAE)?
• Do I need to compare across different scales? If yes, avoid MAE/RMSE and use a scaled or relative metric like MASE.
• Does my data contain many zeros? If yes, avoid MAPE at all costs. MASE or an accuracy scaled by the mean are better choices.
• Who is my audience? Interpretability is key. MAE is often clear for stakeholders, but be ready to explain why “global accuracy” might be misleading.
• What is the true goal? Is it to get the grand total right or to minimize manual adjustments at a granular level? The latter is often the more meaningful goal.

 

Author Bio :

Madhura Raut is a Senior Data Scientist at Workday, where she leads the design of large-scale machine learning systems for labor demand forecasting. Her work integrates time series modeling, AI, and reinforcement learning to enable real-time workforce optimization. With six years of experience spanning enterprise machine learning and applied research, Madhura is passionate about building intelligent, production-grade systems that address complex operational challenges. Besides her professional work, she actively contributes to the broader AI and data science community and is mentor to many aspiring women in stem.