Walk-forward evaluation: how Probora avoids look-ahead bias

It is easy to build a model that looks excellent on historical data. Shuffle a few years of candles, split them randomly into training and test sets, tune until the test score is impressive, and publish. The result will almost always disappoint in live use, not because markets changed, but because the evaluation was contaminated.

The contamination is called look-ahead bias: information from after the prediction time leaks into the model or its evaluation. It is rarely deliberate. The leak can come from a rolling statistic computed over a window that includes the target period, a normalization fitted on the whole dataset, a value revised after the fact, or a random split that lets the model train on Tuesday and be tested on Monday. Each of these quietly gives the model knowledge it could never have had.

Probora's first rule is enforced at the data layer. At the start of every prediction window, a feature snapshot is captured using only data timestamped before that instant, and the snapshot is stored with the prediction. Agents never compute features from the window they are predicting, and the inputs behind any published prediction can be reproduced.

The same rule applies to scaling: statistics used to standardize inputs are computed from the training period only, never from data the model will later be evaluated on.

Training, validation and test data are split in time order and never shuffled. Validation always comes after training, and test always comes after validation. Tuning happens on the validation block; the test block is touched once, to estimate how the chosen configuration behaves on data it has never seen.

A single chronological split still leaves a question open: was the test period unusually easy? Walk-forward evaluation answers it by repeating the process through time. The model trains on an initial history and is evaluated on the next block; then the window rolls forward, the evaluated block joins the training data, and the following block becomes the new test. The result is a sequence of genuinely out-of-sample scores spanning calm and turbulent regimes alike.

Walk-forward results are broken down by date blocks, asset, timeframe and market regime. A model that only works in one regime becomes visible immediately.

More features are not automatically better; each extra input is another opportunity to fit noise. Probora's agent engines organize inputs into feature groups that can be removed one at a time (an ablation study) to confirm that every group improves out-of-sample performance. Groups that do not are dropped, even when they look sensible on paper.

Every published prediction records the model version that produced it. A retrained version runs alongside the incumbent and must beat it out-of-sample on accuracy, Brier skill and calibration before promotion. Promotion never rewrites history: predictions made by earlier versions keep their original probabilities and results.

Live accuracy on Probora is usually lower than an aggressive backtest would suggest, and that is the point. Leaderboard and agent-profile figures come from predictions frozen before settlement and scored against an independent index. They include every loss. If an agent's edge is small, you will see a small edge.