Forecast Evaluations

You can select forecasting methods to generate as many forecasts for each product. Each forecasting method might create a slightly different projection. When thousands of products are forecast, a subjective decision is impractical regarding which forecast to use in the plans for each product.

The system automatically evaluates performance for each forecasting method that you select and for each product that you forecast. You can select between two performance criteria: MAD and POA. MAD is a measure of forecast error. POA is a measure of forecast bias. Both of these performance evaluation techniques require actual sales history data for a period specified by you. The period of recent history used for evaluation is called a holdout period or period of best fit.

To measure the performance of a forecasting method, the system:

• Uses the forecast formulas to simulate a forecast for the historical holdout period.
• Makes a comparison between the actual sales data and the simulated forecast for the holdout period.

When you select multiple forecast methods, this same process occurs for each method. Multiple forecasts are calculated for the holdout period and compared to the known sales history for that same period. The forecasting method that produces the best match (best fit) between the forecast and the actual sales during the holdout period is recommended for use in the plans. This recommendation is specific to each product and might change each time that you generate a forecast.

Mean Absolute Deviation

Mean Absolute Deviation (MAD) is the mean (or average) of the absolute values (or magnitude) of the deviations (or errors) between actual and forecast data. MAD is a measure of the average magnitude of errors to expect, given a forecasting method and data history. Because absolute values are used in the calculation, positive errors do not cancel out negative errors. When comparing several forecasting methods, the one with the smallest MAD is the most reliable for that product for that holdout period. When the forecast is unbiased and errors are normally distributed, a simple mathematical relationship exists between MAD and two other common measures of distribution, which are standard deviation and Mean Squared Error. For example:

MAD = (Σ | (Actual) – (Forecast)|)n

Standard Deviation, (σ) ≅ 1.25 MAD

Mean Squared Error ≅ –σ2

Percent of Accuracy

Percent of Accuracy (POA) is a measure of forecast bias. When forecasts are consistently too high, inventories accumulate and inventory costs rise. When forecasts are consistently too low, inventories are consumed and customer service declines. A forecast that is 10 units too low, then 8 units too high, then 2 units too high is an unbiased forecast. The positive error of 10 is canceled by negative errors of 8 and 2.

(Error) = (Actual) – (Forecast)

When a product can be stored in inventory, and when the forecast is unbiased, a small amount of safety stock can be used to buffer the errors. In this situation, eliminating forecast errors is not as important as generating unbiased forecasts. However, in service industries, the previous situation is viewed as three errors. The service is understaffed in the first period, and then overstaffed for the next two periods. In services, the magnitude of forecast errors is usually more important than is forecast bias.

POA = [(ΣForecast sales during holdout period) / (ΣActual sales during holdout period)] × 100 percent

The summation over the holdout period enables positive errors to cancel negative errors. When the total of forecast sales exceeds the total of actual sales, the ratio is greater than 100 percent. Of course, the forecast cannot be more than 100 percent accurate. When a forecast is unbiased, the POA ratio is 100 percent. A 95 percent accuracy rate is more desirable than a 110 percent accurate rate. The POA criterion selects the forecasting method that has a POA ratio that is closest to 100 percent.