The Forecasting Frontier in Power Markets Is Probabilistic, and Joint

I have been reading through the electricity price forecasting literature with a statistician's eye, and one theme keeps surfacing: the interesting work has stopped chasing point forecasts. The frontier produces calibrated distributions, or coherent scenarios across time, location, products, and market stages. This is not a stylistic preference. It follows directly from what the downstream decisions need.

Why a point forecast is not enough

A point forecast answers one question: what is the expected price at this hour? Most decisions in power markets do not hinge on that number. They hinge on the shape of the distribution around it.

Consider a few examples from the decision side. A battery operator needs a path of future prices, not independent hourly estimates, because state of charge couples decisions across time: if you discharge now, you cannot discharge later. A load-serving entity (an LSE, the firm that buys wholesale power to serve retail customers) needs the joint tail of load and price, because its worst days are the ones where both spike together. A renewable bidder needs to know how production and price co-move, and specifically that low renewable output can coincide with high prices. None of these questions can be answered by a single number per hour.

Electricity prices make this worse than in most domains. The series are heteroskedastic, heavy-tailed, seasonal, bounded by market rules (caps and floors, negative prices), and structurally nonstationary. The mean is often the least interesting part of the distribution. The few intervals with spikes or scarcity conditions are where most of the optionality value lives.

Calibration, quantiles, and pinball loss

Since the output should be a distribution, the tooling changes. Three ideas do most of the work.

Calibration means the forecast probabilities match observed frequencies. If a model says there is a 10 percent chance the price exceeds some level, then across many such statements the price should exceed that level about 10 percent of the time. An uncalibrated model that is overconfident in the tails will systematically misprice risk, no matter how good its central estimate looks. Interval coverage tests and conformal prediction (a distribution-free method for producing intervals with guaranteed empirical coverage) are the standard checks.

Quantiles are the practical currency. Instead of one prediction, you predict the 5th, 50th, 95th percentiles, and so on. The 95th percentile of price is directly useful for someone deciding how much exposure to carry into an evening peak.

Pinball loss (also called quantile loss) is how you train and score quantile forecasts. For a target quantile level tau, the loss penalizes under-predictions with weight tau and over-predictions with weight one minus tau. A model minimizing pinball loss at tau equal to 0.95 is pushed to produce a value that the realized price stays below about 95 percent of the time. Aggregating pinball loss across many quantile levels approximates the continuous ranked probability score (CRPS), which scores the entire predictive distribution against the realized outcome. CRPS, energy scores, and coverage tests are the statistical scoreboard for probabilistic forecasts, the way RMSE is for point forecasts.

The important asymmetry: a model with slightly worse RMSE can be more valuable if it captures the few intervals that determine optionality value. Optimizing symmetric squared error on a heavy-tailed, asymmetric decision problem is optimizing the wrong thing.

Joint structure: where independent forecasts fall apart

Even a set of perfectly calibrated marginal distributions is not enough, because decisions consume joint structure. If you forecast each hour and each location independently and then sample from those marginals, you generate impossible scenarios: every node in the network spikes independently of its neighbors, reserve prices detach from energy scarcity, or day-ahead and real-time forecast errors carry inconsistent signs. An optimizer fed these scenarios will hedge against events that cannot happen and ignore correlated events that can.

A useful scenario set has to preserve temporal autocorrelation, cross-node correlation, price-load-renewable dependence, joint scarcity regimes, market caps and floors including negative-price behavior, and the dependence between day-ahead and real-time markets. Tools for this include copulas (functions that couple marginal distributions into a joint one), scenario generators built on stochastic differential equations with jump processes, and newer generative approaches. One example of the latter is the Cornell WIAE-GPF work, which learns weak innovation representations and generates probabilistic paths while keeping the innovation structure interpretable.

A design pattern, and what to score

The pattern I find most convincing separates concerns rather than asking one model to do everything: deterministic fundamentals and structural features first, then normal-regime residual behavior, then spike and negative-price occurrence, then spike magnitude, then dependence across dimensions, and finally calibration and scenario reduction. An NREL report on stochastic price scenario generation follows this shape: ARIMA captures mean reversion while a stochastic differential equation with jump structure generates the spikes, followed by scenario reduction so the optimizer has a tractable set.

Scoring should be two-sided. Statistically: CRPS, energy score, interval coverage, and specifically tail calibration and scarcity recall, since averages hide exactly the intervals that matter. Decision-side: downstream profit or regret, CVaR and drawdown, and stability across regimes and market-rule versions. Validation needs the same discipline: walk-forward evaluation, no random train/test splits, and performance broken out by season, hour, and price regime. A common failure mode is clipping spikes in preprocessing and then claiming improved accuracy, which is precisely deleting the events the distribution exists to describe.

The takeaway

Nobody can predict prices, and the probabilistic framing is honest about that. The goal is a calibrated, coherent description of what might happen, with the tails and the dependence structure preserved, because that description is the actual input to every downstream optimization. Shaving average RMSE on the median hour is a solved-looking problem that mostly does not matter. Getting the joint distribution right in the hours that do is where the work is.