A Reference Architecture for Production Market-Bidding Systems

A production bidding system for electricity markets must be economically useful, physically feasible, tariff-valid, auditable, resilient to bad data, and safe under model failure. The correct optimization target is controlled decision quality, not forecast accuracy in isolation. This article describes a reference architecture as a pipeline of layers, each with a narrow contract, and the failure modes each layer exists to prevent.

Pipeline overview

Data flows from external and internal sources (ISO files and APIs, weather ensembles, fuel prices, outages, asset telemetry, market rules) through ingestion into bitemporal storage, then into a market-state and feature layer. From there, probabilistic forecast services and a clearing emulator feed a scenario engine, which supplies a rolling decision engine. Every proposed action passes through independent risk and compliance controls before a submission and operations layer sends it to the market operator. Awards, telemetry, settlements, attribution, monitoring, and replay close the loop.

One platform should not force every participant into one optimizer. A physical generator, a storage operator, a load-serving entity (a retailer that must buy power to serve customer demand), and a virtual trader have different state and objectives, so the architecture hosts specialized decision engines behind shared data, scenario, control, and market-adapter contracts. The market adapter is deliberately separate from the participant engine: a battery engine is reusable across CAISO, ERCOT, and PJM only if each adapter correctly encodes that market's products, sign conventions, gates, state-of-charge rules, mitigation, credit, and settlement.

Layer 1: Bitemporal ingestion

The data layer stores every fact on two time axes:

• Event time: when the underlying condition applied in the real world.
• Knowledge time: when the system actually learned it.

Electricity market data is heavily revised. Load forecasts are reissued, prices are corrected, outage reports arrive late. Without knowledge time, backtests silently use revised load forecasts, corrected prices, or outage information that only arrived after the submission deadline (the gate closure), producing strategies that cannot be reproduced live. Bitemporal storage, with lineage, revision history, and quality flags, is the foundation of everything downstream.

Core domains include the market calendar and gates, day-ahead and real-time prices and awards, bids and their acknowledgements, forecasts with issue times, weather, fuel, outages and topology, asset telemetry, contracts, credit and settlements, and versioned tariff and rule documents. Every feed needs freshness, completeness, range, schema, duplicate, monotonicity, and cross-source checks, with quality flags flowing into model uncertainty and operational decisions rather than being hidden in logs.

Layer 2: Market state and features

Above raw storage sits a market-state and feature layer: fundamentals, grid topology, asset state, current positions, and the applicable rule version. Timestamp-correct feature construction is one of the things worth building in-house, because a feature computed from data the system did not yet know at decision time invalidates both backtests and live behavior.

Layer 3: Probabilistic forecasts

Forecast services expose distributions, not only means. A forecast contract includes the target and market interval, issue timestamp and valid horizon, quantiles or scenario identifiers, model and data versions, calibration diagnostics, missing-data and fallback state, regime probabilities, and confidence flags. Ensembles diversify model error; a practical stack combines a structural model, boosted trees, and a spatiotemporal deep model with online calibration. Targets should match the participant: joint load-price tails for retailers, deliverable flexibility for aggregators, constraint-binding probabilities for congestion traders.

Layer 4: Scenario generation and reduction

The scenario engine converts forecast distributions into coherent paths suitable for optimization: temporal and spatial dependence, coherence between day-ahead and real-time markets and between energy and reserves, renewable-load-weather coupling, explicit scarcity and negative-price regimes, probability weights, and reproducible random seeds.

Because optimizers cannot handle unlimited scenarios, the set must be reduced with tail preservation. K-means clustering is a common baseline, but ordinary Euclidean clustering may discard exactly the rare economic tails (price spikes, scarcity events) where money is made or lost. Alternatives include k-medoids, Wasserstein clustering, importance sampling, and optimization-aware reduction.

Layer 5: Rolling decision engine

The default engine is a rolling-horizon LP or MILP (linear or mixed-integer program re-solved as new information arrives), or a stochastic portfolio model, incorporating all physical, contractual, customer, credit, and market constraints; an objective of expected profit or participant utility plus CVaR (conditional value at risk, the expected loss in the worst tail of outcomes) or a risk budget; transaction and degradation costs; terminal values or inventory targets; scenario recourse; and warm starts with deterministic solve limits.

Optional modules extend it: distributionally robust optimization when the forecast distribution itself may be wrong, a bilevel or price-impact model when bids materially move prices, bounded reinforcement learning for regime selection or residual actions, and human strategy constraints such as approved price bands and exposure caps.

The output contract matters as much as the objective. The optimizer should emit more than a bid: the action object and alternatives, an objective decomposition, the expected P&L distribution, risk measures, binding constraints and dual values, scenario sensitivities, a fallback recommendation, and an explanation suitable for operations and compliance.

Layer 6: Independent risk and compliance controls

The central design principle: never let the same code that proposes the trade be the only code that validates it. A separate control layer, with its own implementation, checks four families of constraints:

• Physical and service delivery: power, energy, ramp, and state-of-charge feasibility; reserve deliverability; telemetry validity; must-perform obligations.
• Market schema: valid nodes, products, and hours; monotone bid curves and segment limits; price caps and floors; gate and revision status.
• Financial risk: gross and net exposure, concentration by node, hour, and constraint, CVaR and drawdown limits, collateral and credit use, and stale-model limits.
• Compliance: approved strategies and instruments, mitigation and reference-level boundaries, must-offer obligations, prohibited-pattern checks, and escalation for unusual bids or overrides.

Layer 7: Submission and operations

Submission is a safety-critical integration service, not a fire-and-forget API call. It requires idempotent request identifiers, deterministic serialization and unit conversion, retry rules that cannot duplicate positions, clock synchronization with gate buffers, degradation modes for operator API failures, manual takeover, and exact replay of what was sent and received.

A key failure mode: a successful HTTP response is not an accepted market bid. The transport can succeed while the operator rejects the bid semantically. Production systems parse operator-level acknowledgements and validate them, rather than trusting transport status.

Service-level objectives are defined around gate closure, not generic uptime: latest safe start time, maximum data age, forecast and optimization deadlines, acknowledgement deadlines, and the maximum tolerated period of unknown position state. Where an unknown state could create duplicate or invalid exposure, the system fails closed.

Layer 8: Settlement-grade attribution and continuous backtesting

The final layer reconciles awards, telemetry, and settlements against expectations: expected versus realized P&L, slippage and settlement adjustments, constraint and opportunity-cost attribution, and mitigation or operator intervention. Monitoring also covers data and models (feed delay and revision rates, feature drift, forecast calibration, scenario coverage, solver infeasibility and solve-time tails) and safety (fallback frequency, validator disagreements, human overrides, rejected submissions, and policy actions outside historical support).

Backtesting is continuous, not a one-time gate. Deployment progresses through research replay on frozen datasets with exact knowledge-time reconstruction, paper trading on live inputs, shadow mode compared against the existing process, limited capital, and graduated expansion behind predefined statistical, operational, and compliance gates. A continuous champion-challenger setup keeps a validated baseline available, and every model carries governance metadata: owner, approved use, validation report, monitoring thresholds, retraining and rollback policy, and retirement criteria.

Failure modes this architecture prevents

• Lookahead leakage: backtests using data (revisions, corrections, late outage reports) that arrived after gate closure. Prevented by bitemporal storage.
• Discarded tails: scenario reduction that averages away scarcity and negative-price regimes. Prevented by tail-preserving reduction methods.
• Self-validating trades: the proposing optimizer being the only validator of feasibility, schema, risk, and compliance. Prevented by independent controls.
• Phantom acceptance: treating an HTTP 200 as an accepted bid. Prevented by parsing operator acknowledgements.
• Duplicate exposure: retries that resubmit positions. Prevented by idempotent identifiers and fail-closed behavior on unknown state.
• Silent model decay: uncaught drift, miscalibration, or actions outside historical support. Prevented by continuous monitoring and champion-challenger comparison.

Technology choice matters less than temporal correctness, rule fidelity, and independent controls.

Glossary

• Gate closure: the deadline after which the market operator no longer accepts or revises bids for an interval.
• Bitemporal storage: recording both when a condition applied (event time) and when the system learned it (knowledge time).
• Day-ahead / real-time markets: the forward auction cleared the day before delivery, and the balancing market cleared near delivery.
• LP / MILP: linear and mixed-integer linear programming, the standard optimization formulations for scheduling and bidding.
• CVaR: conditional value at risk, the expected loss conditional on being in the worst tail of the outcome distribution.
• Distributionally robust optimization (DRO): optimizing against the worst case over a set of plausible probability distributions.
• Mitigation: market-power rules that can replace a bid with a reference level if it deviates too far from estimated cost.
• Scenario reduction: shrinking a large set of sampled future paths to a tractable subset while preserving its statistical (especially tail) properties.
• Champion-challenger: running a validated baseline model alongside candidates, promoting a challenger only after it passes predefined gates.
• Settlement: the operator's after-the-fact financial reconciliation of awards, delivery, and charges.