Why backtesting is not enough
A practical explanation of chronological backtests, locked tests, and why a forecasting model still needs forward evidence.
History is useful—and dangerously persuasive
Historical data makes it possible to develop and compare forecasting ideas quickly. It also creates many opportunities to choose features, dates, parameters, or evaluation rules that happen to fit the past. The more decisions made after seeing results, the less independent those results become.
Chronology matters
A time-series experiment should preserve the order in which information became available. Training on later market conditions and testing on earlier ones leaks knowledge backwards. BTC Horizon therefore separates chronological training, validation, and locked test periods instead of randomly shuffling observations.
The locked period is evaluated only after the candidate and its promotion rules are frozen. A candidate must be compared with simple controls under the same dates and scoring rules; complexity does not earn credit by itself.
A locked test is still not the live future
Even a disciplined historical test uses a period that already exists. Data revisions, implementation mistakes, publication timing, market regime changes, and operational failures may only become visible when the system runs forward in real time.
Forward validation means producing a forecast using only information available then, preserving it unchanged, and resolving it after the horizon matures. The sample accumulates slowly, but that inconvenience is part of the evidence rather than a problem to optimise away.
What counts as progress
A few successful forecasts do not prove predictive skill. BTC Horizon looks for a sufficiently large resolved sample, calibration, performance against frozen simple alternatives, stability across market conditions, and reliable operation. A formula version changes because declared evidence or methodology warrants it—not merely because time passed.