Backtests That Lie: Why the Evaluation Harness Is the Real Product

I have spent a lot of time reviewing evaluation setups for electricity trading models, and I keep arriving at the same conclusion: the model is rarely the hard part. The hard part is building a harness that reconstructs, for every decision, exactly the information set that was available when the decision had to be made. Most inflated electricity backtests violate that principle somewhere, and the violations are specific to how power markets publish and revise data. Here is a tour of the main failure modes, and why I now treat the evaluation harness as the real product of the research effort.

The leakage taxonomy

Price leakage is the obvious one. Electricity prices get corrected and finalized after the fact. If your real-time features use corrected or final prices instead of the preliminary values that were actually on the wire at decision time, you are trading against information that did not exist yet. The same applies to load and renewable forecasts: operators revise them, and the revised versions quietly replace the originals in most historical archives. A model trained on revised forecasts is a model trained on the future. Weather is a subtler version of the same bug: using weather observations where the strategy would only have had issued forecasts.

Topology leakage is the failure mode that surprises people coming from other asset classes. The grid is a physical network, and its structure changes: nodes are added and retired, transmission elements go in and out of service, constraint definitions shift. Using current node mappings for historical dates, or feeding the model ex-post topology and binding-constraint files, hands it knowledge of which transmission constraints actually bound. That is precisely what a congestion strategy is trying to predict. Finalized outage and derate records have the same problem: the record you can download today is not the record anyone had at gate closure.

Survivorship bias shows up as survivor-only nodes and resources. If your historical universe contains only the pricing nodes and generators that still exist, you have filtered the dataset on outcomes. Add the quieter statistical leaks: full-day statistics used as features for intraday decisions, feature normalization derived from the target, and hyperparameters tuned on the final test period.

Event time versus knowledge time

The fix for all of this is point-in-time discipline, and the data structure that supports it is bitemporal. Every record needs two timestamps: event time, when the thing happened or the interval it describes, and knowledge time, when that particular version of the record became available. A load forecast for hour 17 issued at 06:00 and revised at 11:00 is not one row. It is two rows with the same event time and different knowledge times. The walk-forward protocol then becomes mechanical: for each bid timestamp, load only data published by that time, apply the rule version effective on that trading date, train at a realistic cadence, and solve under then-current positions, credit, and asset state. And never randomly shuffle market intervals into train and test splits; serial dependence makes that a leak in itself.

Fills, settlement, and free optionality

Even with clean inputs, a backtest lies if it is optimistic about execution. Price-sensitive bids may not clear fully. An honest simulator models segment-by-segment acceptance, ties under the market's specific clearing conventions, minimum award quantities, partial awards, rejections for credit or position limits, and even mundane API or schema rejections and late submissions. Assuming every desired trade executes at the published clearing price creates free optionality, and free optionality compounds into fictional P&L.

Settlement completeness matters just as much. Gross margin at the day-ahead price is not what hits the ledger. Uplift and make-whole allocations, losses and congestion components, deviation penalties, transaction and scheduling fees, collateral and financing cost, and battery degradation all need to appear where material. Uplift charges in particular are easy to ignore because they are allocated after the fact, which is exactly why they belong in the harness.

The strategy would have moved the price

There is a deeper causal problem. A historical-price backtest is valid only for actions small enough to be price-taking. Once the position is large, the strategy itself would have shifted the clearing outcome, and replaying against recorded prices answers a question nobody asked. The honest options are to replay in a clearing emulator, estimate local residual supply and price elasticity, stress multiple impact assumptions, compare simulated bids against observed market depth and awards, and disclose plainly that counterfactual prices are model-dependent.

Honest baselines, and why perfect foresight tells you nothing

A complex model has not established value until it beats a ladder of progressively stronger baselines: first the participant-native no-action baseline (no trade, the historical hedge, or the existing operator process), then seasonal persistence, then a simple threshold or fixed schedule, then a linear or boosted-tree forecast, then a rolling deterministic LP, then a probabilistic forecast feeding a stochastic program. A state-of-the-art architecture that only beats a weak heuristic has demonstrated nothing.

The one benchmark I would strike from most decks is perfect foresight. Knowing every price in advance defines an upper bound no causal strategy can approach, and the gap to it is dominated by irreducible uncertainty rather than model quality. It tells you what the market paid, not what your method earned. Capturing thirty percent of an unreachable ceiling is not a result; beating the incumbent process on the decision owner's own ledger is. And that ledger differs by participant: a battery operator needs degradation-adjusted P&L, a retailer needs cost to serve and hedge effectiveness. The same model can look successful under the wrong metric.

The harness is the product

Statistical care closes the loop: block bootstrap for serially dependent returns, multiple-testing corrections when many strategies were tried, predeclared holdout periods, and reporting the full P&L distribution rather than one cumulative curve. A credible result states its decision timestamps, rule versions, revision handling, price-impact assumptions, tuning budget, and failure cases.

Notice what all of this has in common: none of it is the model. Models come and go, and any of the rungs on the baseline ladder might turn out to be the right one. The bitemporal data store, the conservative fill simulator, the settlement engine, the baseline ladder, and the reporting discipline are what let you tell a real edge from a leak. Build the harness first. It is the only artifact whose value survives every model you retire.