Before a hotel trusts a forecast to steer pricing, it needs evidence that the forecast works. In a new article in its The Signal series, FLYR Hospitality's Head of Data Science, James Butler, walks through backtesting, the technique FLYR uses to validate its demand forecasting models.

What backtesting is

Backtesting means running a predictive model over historical data and comparing its predictions with what actually happened. The alternative, forward testing, requires forecasting a future date and waiting for it to arrive. Butler points out how slow that is: validating a 90-day lead-time forecast means waiting 90 days for one data point, and covering a full year of conditions would take a year. Continuous model updates would also make earlier test data obsolete. Because backtesting uses periods that have already happened, results can be scored immediately, and many test points can be generated by choosing different past dates as the forecast's starting point. The same approach is common in quantitative trading, weather prediction, epidemiology and demand forecasting.

Why one accuracy number is not enough

Butler notes that two forecasts can share the same average error yet behave very differently: one may be consistently slightly off, another usually right but occasionally far off. A forecast can also carry a directional bias, consistently over- or under-stating demand, which can lead to misreading demand strength and setting suboptimal prices. Backtesting surfaces these differences and lets teams compare a new model against the one already in production on the same history.

FLYR's approach

FLYR's production backtest runner evaluates its current forecasting model against a rolling window of historical outcomes, split by how far ahead each forecast was made. A forecast made 30 days before a stay date and one made 90 days out face different problems, so FLYR assesses them separately instead of blending them into a single figure. The team tracks MAPE and symmetric MAPE (sMAPE), the latter being more robust in low-demand periods, alongside mean bias error to check whether the model leans consistently high or low.

Originally published by FLYR: FLYR Explains Backtesting and Why It Matters for Hotel Demand Forecasts