2026 Midterm
Forecast Engine

What is actually on the 2026 ballot

Three simultaneous forecasts over one shared national environment. They differ in polling density, correlation structure, and how much of the answer the priors have to carry.

The one structural fact that shapes the whole House model

After the 2025–26 mid‑decade redistricting round, the median district is Virginia’s 1st, which Trump carried by 4.9 points — roughly 3.4 points to the right of the nation. Democrats can win the national House vote outright and still lose the chamber. Your seats‑votes curve has to encode that bias explicitly; a model that treats national margin as if it maps symmetrically onto seats will be systematically wrong in one direction.

The simulation matrix is the product

Persist every draw, not the summary statistics. This is the single decision that separates a forecast you can drill into from one you can only look at.

Each run produces one array of shape n_sims × n_races holding a margin and a winner per cell. Every headline probability, every seat histogram, every tipping‑point statistic, and every conditional question is then a reduction over that array rather than a separate model:

# topline
P(House) = mean(seats[:, HOUSE].sum(axis=1) >= 218)

# conditional — the feature people actually come back for
mask     = win[:, "GA-SEN"] == DEM
P(Senate | GA-SEN goes D) = mean(seats[mask, SENATE].sum(1) >= 51)

# tipping point — sort each sim's races by margin, find the seat that crosses 218
# correlation view — corr(margin[:, i], margin[:, j])

Homegrown forecasters routinely store only per‑race win probabilities, then discover they can never answer a conditional question without re‑running the model. The storage cost of avoiding that is trivial:

One Parquet file per run date, partitioned by chamber. Precompute the L1/L2 aggregates into static JSON at run time; only conditional queries ever touch the matrix.

Engine pipeline

A batch pipeline, not a service. It runs once a day, writes immutable run artifacts, and the API only ever reads them. That makes every historical forecast reproducible and every regression bisectable.

The race registry is a first‑class input, not a constant

Districts, candidates, incumbency and map versions all change during the cycle — Texas’s map is still on appeal at the Supreme Court as of this writing. Keep districts_2026.csv and races.yaml versioned inputs with an effective date, so a late map change is a data update and a re‑run, not a code change. Never hardcode a district list.

Key every race on (cycle, map_version, district), never on the district string alone. Texas’s 35th district under the 2021 map and under the 2025 map are different electorates wearing the same name. A join on the bare string will silently blend them, and the resulting prior will look entirely reasonable.

The naive baseline, specified

This is what Phase 0 and 1 build. It is wrong in known, enumerated ways — and it produces a plausible, end‑to‑end forecast that the dashboard can be built against immediately.

### 1. Candidate roster  ────────────────────────────────
FEC bulk cn26 → party, office, district, incumbency (CAND_ICI)
# authoritative for federal races. Governors are not federal.

### 2. Resolve each poll to a matchup  ──────────────────
answers name PEOPLE, not parties: {"Casey Askar": 50.0}
  → surname match inside this race's roster, fuzzy at 0.85
  → drop matchups >45d behind the race's newest poll
    # TX-SEN polled Talarico-v-Paxton AND Talarico-v-Cornyn;
    # those are different races. Averaging them describes neither.

### 3. Weighted matchup average  ────────────────────────
w_p   = exp(-days_old / 30) · 0.35^partisan · 0.35^internal
eff_n = (Σw)² / Σw²                        # NOT the raw poll count
p_i   = Σ(w_p · margin_p) / Σ(w_p)

### 4. Prior — the fallback, not the driver  ────────────
lean_i   = pres24_margin_i - pres24_national
prior_i  = lean_i + generic_ballot + inc_i   # inc from FEC

### 5. Blend on effective n  ────────────────────────────
w_i   = eff_n / (eff_n + 3)
mu_i  = w_i·p_i + (1-w_i)·prior_i

### 6. Sigma conditional on PRIOR QUALITY  ──────────────
sigma_i = (1-w_i)·sigma_prior_only[chamber] + w_i·2.5
sigma_prior_only = {house 4.0, senate 6.5, governor 7.0}
    # a presidential-lean prior for a Senate race carries far
    # more error than a candidate-poll-informed one

### 7. Simulate, S = 50,000  ────────────────────────────
margin[s,i] = mu_i + 3.0·z[s] + 1.5·z[region,s] + sigma_i·t[s,i]

What v0 knowingly gets wrong

• No house effects. A pollster that leans three points red is counted at face value. Biggest single accuracy loss.
• No pollster quality weighting. A partisan IVR poll counts the same as a gold‑standard live‑caller poll.
• No fundamentals. Presidential approval and the midterm penalty — the most reliable signals in a midterm — are absent, so sparse‑poll races float on priors alone.
• Elasticity fixed at 1.0. Real districts vary in how much they swing with the nation; college‑educated suburbs move more than ancestral‑partisan rural seats.
• Crude correlation. A three‑level hierarchy, not a covariance matrix built from demographic similarity. Regions are a poor proxy for “races that miss together.”
• Sign‑of‑margin decides the winner, which is simply false in Louisiana, in California and Washington top‑two generals that can be D vs D, and under ranked‑choice voting in Maine and Alaska — all of which have races in 2026.
• No candidate quality, fundraising, scandal, or retirement effects.
• Undecideds are allocated proportionally by silent default, which is what working in margins rather than vote shares amounts to. Fine in a clean two‑way race, wrong wherever undecideds run high or a third party is real. Make the rule explicit and configurable rather than implicit in the arithmetic.

Each of these is a named upgrade with a phase attached below. Shipping them in that order is deliberate: the accuracy gains are roughly monotonic down the list.

From baseline to a real model

Sources, and the hole where FiveThirtyEight used to be

Licensing is a merge requirement, not a footnote

Every adapter records the terms it operates under, checked before it is written rather than after it is running. Wikipedia is CC BY‑SA and carries attribution obligations that propagate into anything public built on it; aggregator terms vary and some prohibit reuse outright. This costs an hour per source now and can invalidate a published product later.

Normalized poll schema

poll_id  source  pollster  sponsor  race_id  start_date  end_date
population {LV|RV|A}  mode {live|IVR|online|mixed}  sample_n  moe
dem_pct  gop_pct  other_pct  undecided_pct  margin
partisan_sponsor {none|D|R}  url  fetched_at  superseded_by

Everything else — weights, house‑effect adjustments, decay — is computed downstream and never written back onto the poll row. Ingestion stays dumb and auditable.

Drill‑down structure

The information design goal: a visitor should be able to get a number in two seconds, understand where it came from in thirty, and interrogate it for an hour.

L1 — Overview

L2 — Chamber

L3 — Single race

Cross‑cutting: scenario explorer and diagnostics

• Scenario explorer. Pin outcomes for any subset of races; every other probability recomputes by filtering the simulation matrix. Effectively free once the matrix is persisted, and the single most‑used feature on forecasts that have it.
• Correlation view. Which races move together, read straight off the matrix — makes the model’s error structure legible instead of a black box.
• Model diagnostics page. Calibration curve, house‑effect table, backtest results, methodology writeup. If this is ever public, this page is what makes it defensible.

API contract

GET  /runs                          → available run dates
GET  /runs/{date}/summary           → topline probs, seat medians + intervals
GET  /runs/{date}/races             → per-race table (L2)
GET  /runs/{date}/races/{race_id}   → detail: history, polls, decomposition (L3)
GET  /runs/{date}/trend             → daily topline history for the fan charts
GET  /runs/{date}/matrix.parquet    → raw draws, for offline analysis

POST /runs/{date}/conditional
     {"given": {"GA-SEN": "D", "NC-SEN": "R"}, "ask": ["senate_control"]}
     → boolean-mask the matrix, reduce, return

The first five are files on disk, generated at run time and cacheable indefinitely — a run is immutable once written. Only /conditional needs a live process, and it is a masked mean over a memory‑mapped array: single‑digit milliseconds.

Backtesting and calibration

The σ values in the v0 spec are guesses. The only honest way to set them is to replay past cycles with data frozen at each date and check whether the resulting intervals were the right width.

Replay 2018 and 2022 — both midterms, both with a president of a known party, one a wave and one emphatically not. Freeze the poll corpus at each historical date, run the full pipeline, and score:

The baseline to beat is deliberately unglamorous: uniform national swing applied to the previous result in each seat. It is surprisingly hard to beat, and a model that cannot beat it is adding complexity rather than information. Nothing ships past Phase 2 until it does.

Watch specifically for over‑confidence at the topline: if coverage looks fine per race but the seat‑count distribution is too narrow, σ_nat is too small. That failure mode is exactly what produced the famous forecast misses, and it is invisible unless you score the aggregate rather than the races.

Score the competitive races, not all 435

A Brier score computed over every House seat is dominated by the roughly 370 districts that any model calls correctly at 99%. The engine and the baseline both score near‑perfect there, and the aggregate cannot tell them apart — a model could be badly wrong in every genuinely contested race and still post a fine overall number. Gate on the subset where the baseline gives between 5% and 95%. Report the full score too, but do not decide anything with it.

Two cycles cannot estimate a variance

This is the hardest limit on the whole plan and it deserves stating plainly. σ_nat is the parameter that most determines the topline, and replaying 2018 and 2022 supplies exactly two draws of the national error to fit it against. Fitting a variance to two observations is not estimation. So: take σ_nat from published polling‑error research as an informative prior, use the backtest to check that it is not too narrow, and never to shrink it. Add 2010 and 2014 for two more midterm draws if the data assembly is affordable. Where the backtest and the prior disagree, widen.

Phase plan

The effort assumption, stated so it can be wrong

The sequence above assumes one engineer at close to full time, already fluent in Python numerics and React, with no other commitments. At half time, or split across other work, it does not fit — and the honest response is to cut scope against the ladder below rather than triage in the last fortnight. The phase and workstream dates this assumption was originally attached to have been removed: they forecast one particular pace, and the first three workstreams beat them by enough that the calendar had become misleading rather than useful. What still binds is the ordering, the gates, and the freeze.

Cut ladder — drop in this order

If the gate has not passed by the freeze

Decide the rule now, in August, not under pressure in late October. If the backtest gate has not passed by 20 October, the forecast is either labelled experimental with prominent uncertainty, or it is not published at all. Both are defensible. Publishing a forecast that failed its own quality gate, quietly, is not.

Repository shape

Polars over pandas for the poll and matrix work — the group‑bys and the Parquet round‑trips are the hot path, and it matters more than it usually would at this data size because you will re‑run the backtest harness hundreds of times.

What I need from you

1. Who is building this, and at what capacity? Every date in the phase plan rests on one full‑time engineer fluent in the stack. This is the answer that most changes the plan, and it is the one I do not have.
2. Public or internal? A published forecast needs the methodology page, the calibration disclosures, and a much higher bar on house‑effect defensibility. An internal analysis tool does not.
3. Any budget for paid data? Free‑only is entirely viable but slower to build and gappier in under‑polled House races. A paid feed mostly buys back ingestion time, which is the scarcest thing on this schedule.
4. Governors in v1, or defer? 36 races, weakest correlation structure, no control stake. It is the cheapest thing to cut if the calendar tightens, and cutting it early is better than cutting it in October.
5. Election night: build, buy, or skip? Needs deciding by 1 October, because building it means starting in P3.
6. How many simulations? 50,000 is the assumption above. 10,000 is enough for toplines and cuts run time by 5×; conditional queries on rare scenarios get noisy below about 40,000.

What shipped, and what it corrected

The pipeline runs end to end on real data. Five of the plan’s assumptions survived contact; five did not, and four of the failures were only visible once real polls were flowing.

Candidate-level, not generic ballot

The forecast is now driven by actual candidate matchup polling, with the generic ballot demoted to a fallback prior for the ~465 races that have none. Candidates come from the FEC bulk candidate master, which is authoritative for federal races. The effect was not marginal:

Five findings the plan did not anticipate

Two plan assumptions that were simply wrong

• “The 538 archive is gone” was itself wrong — corrected. The repo is live (last pushed Feb 2025); the 404s were bad paths, not a deleted repo. pollster-ratings/raw_polls.csv holds 7,963 congressional general‑election polls with actual results attached. That single file unlocks house effects, our own pollster ratings, the empirical variance curve, and the backtest spine. Recording the correction because the wrong version briefly shaped the plan.
• The generic ballot feed is 53 days stale at source. VoteHub’s newest generic‑ballot poll is 2026‑06‑30 while its approval polls run to 2026‑08‑18. Verified not to be an adapter bug. The staleness alarm from Stress Test #05 fired on day one, which is the only reason it was caught.

Road to parity

Calibration before coverage. The model’s biggest defect was not missing signal — it was overconfidence in the signal it already had. All seven workstreams are now closed, and the ordering paid for itself twice: the backtest’s own headline finding turned out to be an artefact of its stand‑in prior, and one workstream’s scoped mechanism was rejected on measurement rather than argued about.

Then P3, the drill‑down dashboard, and the freeze on 20 October, both unchanged.

Where the first draft broke

Fourteen failure modes, ordered by damage. Three of them changed the plan structurally rather than cosmetically, and those are worth reading in full.

The other eleven

Where this goes wrong

• Ingestion, not mathematics, is the schedule risk. The model in this document is a week of work. Getting clean, deduplicated, correctly‑attributed polls for 506 races every day, from sources that did not exist to serve you, is the rest of it.
• Redistricting is still moving. Texas’s 2025 map is before the Supreme Court on appeal. Keep the map a versioned, dated input so a late ruling triggers a data change and a re‑run — never a code change under deadline.
• Most House districts have no polls at all. The forecast for roughly 370 seats is a prior wearing a probability. Say so in the interface: show the poll count on every race, and let the decomposition panel make the ratio of prior to data visible rather than flattering.
• Correlated error is where forecasts actually fail. Under‑specifying σ_nat produces a topline that looks confident and precise and is neither. Tune it on backtests. Never on intuition.
• Consensus‑chasing. If v0 disagrees with published forecasts, the overwhelmingly likely cause is a bug. But once the backtest gate is passed, resist tuning toward other people’s numbers — at that point you are fitting to their models, not to elections.