The Operator's Core Engine Is Optimization, Not Machine Learning

I have been reading into US wholesale electricity markets recently, coming at them from a machine learning and statistics background. The first thing that reoriented my thinking is simple to state: the system operator's core pricing engine is not a machine learning model. It is a large constrained optimization problem, solved on a schedule, under explicit physical and regulatory rules. Everything a market participant sees, including prices, follows from that.

What the operator actually runs

In the US organized markets, an independent system operator (ISO) or regional transmission organization (RTO) clears the market with two related programs.

The day-ahead engine is security-constrained unit commitment, or SCUC. It decides which generators turn on and how much they produce over the next day, minimizing offered costs (energy, no-load, and startup costs) subject to energy balance, minimum up and down times, ramp limits, reserve requirements, transmission line limits, and contingency constraints. Because on/off decisions are binary variables, this is a mixed-integer linear program (MILP), which is combinatorial and generally hard.

The real-time engine is security-constrained economic dispatch, or SCED. With commitment mostly fixed, it re-solves output levels every few minutes. With piecewise-linear offers, the core is usually a linear program.

This is not a stylized textbook account. CAISO's nodal-pricing documentation describes a day-ahead MILP that iterates with AC power-flow calculations to update loss and transmission sensitivities. PJM describes least-cost security-constrained commitment and dispatch in the day-ahead market and five-minute SCED with co-optimization in real time. ERCOT describes real-time SCED as least-cost dispatch subject to system and transmission constraints, with price mitigation integrated into the process. The implementations differ across markets and are more complicated than the canonical formulations, but the central fact is stable: dispatch and pricing logic is grounded in explicit constraints and optimization.

Prices are shadow prices

The prices participants trade against, locational marginal prices (LMPs), are dual variables: the shadow prices of the energy balance constraints, after the market's pricing rules are applied. A shadow price is the marginal change in the objective from relaxing a constraint by one unit. When a transmission line hits its limit, its constraint acquires a nonzero shadow price and LMPs split across locations. That is congestion.

This means an LMP is not a free-floating market signal to model like an equity price. It is a computed output of a constrained physical dispatch. Two consequences from the source material stuck with me. First, integer commitment makes costs nonconvex, so energy LMPs alone may not recover startup and no-load costs; side payments called uplift fill some of the gap, and a price model that ignores this is missing part of the settlement. Second, treating published prices as independent labels, rather than as outputs of an optimization, is listed as a failure mode for good reason: the data-generating process has structure, and regime changes in that structure (which constraints bind, which unit is marginal) drive the interesting price behavior.

The tempting wrong move: rebuild the optimizer

If prices come out of an optimizer, the obvious idea is to build your own copy of the optimizer and run it forward. I think of this as structural replication. For next-day horizons, it is dominated by a statistical approach, for reasons that are structural rather than a matter of effort.

Consider what exact emulation requires: the full network model and monitored constraints, every unit's characteristics and submitted offers, outages and derates, the operator's own load and renewable forecasts, reserve requirements, commitment carryover, operator adjustments, and the exact market software version. Almost none of this is fully public. Offers are confidential. Detailed network models are restricted as critical energy infrastructure information (CEII), a regulatory category that limits access to grid data. Even with the inputs, SCUC is a combinatorial MILP that operators solve with commercial solvers, decomposition, constraint screening, and warm starts under operational deadlines. And the published outcome still reflects out-of-market operator actions and exceptional adjustments that no replica will reproduce.

So a would-be replicator faces missing confidential inputs, a restricted network model, a hard combinatorial problem, and a target contaminated by discretionary actions. Any one of these is a serious obstacle. Together they mean your replica solves a different problem than the operator does, and errors in inputs propagate through a nonconvex program in ways that are hard to bound.

The better framing: forecast the optimizer's outputs

The alternative is to treat the participant's job as statistical forecasting of an optimization's outputs. You are not the operator. You are predicting what the operator's engine will emit, and the structure of that engine tells you what to model.

The source material suggests where the leverage is. Forecast active constraints: which transmission elements will bind, which units will be marginal, what the outage and congestion regime looks like. Binding constraints and marginal units are the causes; LMP levels are symptoms. Use dual information where it is published, since shadow prices and sensitivity factors make interpretable features and support attribution. Version the rules, because offer caps, shortage pricing, mitigation, and commitment logic change the output distribution, and a model fit across a rule change is fit to two different processes. Evaluate on the things that matter: binding-constraint precision and recall, congestion component error, and behavior during scarcity intervals, not just average price error.

There is still a real role for optimization on the participant side. If you need to answer counterfactual questions about your own bids (what happens if I offer differently), historical-price models cannot answer them, and some emulation of the clearing process is necessary where your own price impact matters. And hard feasibility should stay outside the learned model: let ML rank constraints or approximate values, and let an optimizer or a validated rules engine approve actions.

Where ML genuinely fits

None of this is anti-ML. The operators themselves have credible uses for learning: predicting which security constraints will be active, warm-starting branch and bound, predicting commitment patterns, surrogate feasibility checks, anomaly detection. What is not credible is replacing the production clearing engine with a black-box policy, because reliability, auditability, and exact rule compliance dominate small average speed gains.

The same logic applies to participants. The honest framing of AI for energy trading, as I currently understand it, is not "predict prices" and not "rebuild SCUC." It is: learn the statistical regularities in the outputs of a security-constrained optimization you cannot see inside, respect the structure that optimization imposes on the data, and keep an actual optimizer in the loop wherever feasibility is non-negotiable.