Document the stabilization pass

README: window unknown states, what configuration changes do to the learned
model, inputs_changed tolerances, coefficient confidence, model_changes, and
Model Error vs Forecast Error. docs/MODEL.md: configuration-change table,
confidence semantics, both validation modes. CHANGELOG gains an Unreleased
section (version unchanged); AUDIT and ROADMAP record findings, evidence and
remaining risks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

AUDIT.md

@@ -320,3 +320,80 @@ weather forecasts. Learners of removed zones keep only the base model.
 **Verification.** `sh scripts/check.sh` passes in both environments (lint, 125
 offline tests, 26 HA Core tests, compilation, package/translation/changelog checks).
 The offline tests also pass on Python 3.9.6.
+
+## Stabilization after 0.4.0 — 2026-10-27
+
+Scope: correctness of 0.4.0, no new features, version unchanged. Repository code
+and tests only (HA Core 2025.12.0 and 2026.2.3 harness); no HAOS, device or
+deployment testing. Still no actuator code; the static and runtime service-call
+guards pass.
+
+Confirmed issues and fixes:
+
+1. **Unknown window states read as closed** (2bddc74). Reproduced: an open window
+   whose sensor became unavailable ended the ventilation event, started recovery
+   at full weight and let the forecast assume a closed window. Windows are now
+   tri-state with a remembered last known state: last seen open → possibly open
+   (base and extended layers do not learn, the forecast assumes open); a close
+   while unknown is logged as unconfirmed with its longest possible duration;
+   recovery is learned only after a confirmed close. Seven transition tests
+   (open/closed/unknown in every order, restart while unknown, polarity).
+2. **Learned coefficients reused after incompatible changes** (099c63c, 8eef04c).
+   By construction: changing a zone's sensors or weight ratio, a heat source, a
+   window or the recovery time kept every coefficient. New
+   `compatibility.py` fingerprints each component's configuration; on restore
+   only changed components are reset (a measurement change recalibrates the zone:
+   coefficients kept as a starting point, uncertainty, counts and accuracy
+   statistics reset). Comfort weight, target offset and names never invalidate.
+   Heat-source baseline samples are dropped when the effective power's definition
+   changes. 0.4.0 data has no fingerprints, is assumed compatible, and is
+   fingerprinted at the first start. Resets are logged in `model_changes`.
+3. **Plans published from changed inputs** (d73e32a). The pre/post-optimization
+   check covered only temperatures, relay and thermostat, and an overtaken update
+   could still publish. The input snapshot now covers zone and outdoor
+   temperatures, electrical power, wind, windows, relay and thermostat, with
+   tolerances so sensor noise does not discard every plan; a run counter keeps an
+   older run from overwriting a newer result; shutdown is checked after saving
+   and after the worker; a per-entry storage lock serializes save, restore and
+   delete (also fixing a delete-during-save race that could recreate the file).
+4. **Model accuracy reported as forecast accuracy** (26c415f). The Forecast Error
+   sensors replayed the model with the actual future relay, electrical heat and
+   windows. They are now **Model Error** (unique IDs unchanged) with RMSE,
+   effective samples and a quality label, and their state stays unknown below 5
+   effective samples. A new **Forecast Error** sensor scores published forecasts
+   with the plan and assumptions of the moment, only when the relay followed the
+   plan (≥ 12 of every 15 minutes in the planned state); otherwise the forecast is
+   counted as counterfactual, unverifiable or missing, never as error.
+
+Also reviewed:
+
+- **Confidence** (a44163a). Extended and ventilation coefficients already used
+  posterior variance. Base coefficients now report it too; a test shows two weeks
+  of non-exciting data (relay off, steady difference) keep `heat` at 0 and `loss`
+  and `bias` below 0.8, while four varied days bring all three above 0.8.
+  Readiness still needs sample counts; it is a gate, not a confidence.
+- **Lifecycle** (4911b32). Three reloads leave one relay listener and one polling
+  coordinator; after unload nothing is recorded or published. The test was
+  checked to fail with the unsubscribe removed.
+- **Sensor weights UI** (9bb64e7, user report). Per-sensor weights existed but
+  were an unlabelled second step; they are now numbered fields with the sensors
+  listed, in English and Romanian.
+
+Remaining risks:
+
+- In observation mode the relay follows the MPC plan mainly when Generic
+  Thermostat happens to agree, so Forecast Error may stay `insufficient` for a
+  long time, and the scored subset is biased towards those periods.
+- `effective_samples` corrects for overlapping horizons and shared zones only; it
+  is not an autocorrelation estimate. Quality thresholds (5, 30) are judgement.
+- Configuration changes made before this update cannot be detected: 0.4.0 models
+  are fingerprinted as they are. A sensor replaced under the same entity ID, or a
+  weather provider swap with a systematic offset, does not change any fingerprint.
+- Input-change tolerances are fixed constants, not options.
+- Published forecasts awaiting their horizon are lost on restart.
+- None of this has run on HAOS or with real devices.
+
+**Verification.** In both environments (HA 2025.12.0 and 2026.2.3, Python 3.13):
+ruff, 159 offline tests, 28 HA Core tests; the offline tests also pass on Python
+3.9.6. `sh scripts/check.sh` additionally validates packaging, translations and
+the changelog.

CHANGELOG.md

@@ -2,6 +2,81 @@
 
 Engineering findings and test evidence behind each change are in [AUDIT.md](AUDIT.md).
 
+## Unreleased — stability and correctness
+
+Fixes four correctness problems found in 0.4.0 and separates model accuracy from
+forecast accuracy. No new product features; the version stays 0.4.0 until
+released. Still **observation-only**. Tested in the Home Assistant Core harness
+with HA 2025.12.0 and 2026.2.3; not yet on HAOS.
+
+### Upgrade notes
+
+- **Forecast Error 15/30/60/120 min are renamed Model Error.** Unique IDs and,
+  on existing installs, entity IDs are unchanged, so history continues. Their
+  state now stays *unknown* until there are at least 5 roughly independent spans
+  (`quality` is `insufficient`); the numbers stay in the attributes meanwhile.
+- **A new Heating MPC Forecast Error sensor** scores forecasts as they were
+  published. It stays unknown until the relay has followed enough MPC plans; while
+  Generic Thermostat disagrees with the plan, most forecasts are counted as
+  `counterfactual` instead of being scored.
+- On the first start, the saved model gets configuration fingerprints. Models
+  learned under 0.4.0 are assumed to match the configuration they are loaded with;
+  later changes are checked against these fingerprints.
+- Zone forecast attribute `coefficients.base` now gives `value` and `confidence`
+  per coefficient instead of a bare value.
+- Downgrading to 0.4.0 keeps the learned models but drops the Model Error
+  statistics (0.4.0 rejects their new format).
+
+### Fixed
+
+- **Unknown window states no longer pass as closed.** An unavailable window
+  sensor used to end an open-window event, start recovery and let the forecast
+  assume a closed window. Now the last known state is kept: a window last seen
+  open is treated as possibly open (no base learning, forecast assumes open), an
+  event closed while the sensor was unknown is marked unconfirmed with a bounded
+  duration, and recovery is learned only after a confirmed close.
+- **Learned parameters are no longer reused after incompatible changes.** Each
+  learned component has a fingerprint of the configuration it was learned under
+  (see *Persisted data*). Changing a zone's sensors or the ratio of their weights
+  recalibrates that zone; changing a heat source, neighbour, window, orientation
+  or recovery time resets only the affected coefficients; comfort weight, target
+  offset and names never invalidate anything. Every recalibration is listed in the
+  Status attribute `model_changes`.
+- **Plans are no longer published from inputs that changed meanwhile.** Before
+  and after the plan is computed, the integration compares every input it used
+  (zone and outdoor temperatures beyond 0.25 °C, power beyond 50 W, wind, window
+  states, relay and thermostat) and discards the plan if one changed meaningfully
+  (`inputs_changed`, with `changed_inputs`). An older run finishing late never
+  overwrites a newer result, nothing is published after unload, and saving and
+  deleting the model can no longer overlap.
+
+### Changed
+
+- **Model accuracy and forecast accuracy are reported separately.** Model Error
+  replays the model with the inputs that actually happened (relay, electrical
+  heat, windows); Forecast Error uses only what was known when the forecast was
+  published, and scores it only when the relay followed the forecast's plan
+  (in the planned state for at least 12 minutes of every 15-minute block). Both report MAE, RMSE, bias, the
+  no-change baseline, the improvement over it, sample counts, effective
+  (roughly independent) sample counts and a quality label.
+- Base-model coefficients (`loss`, `heat`, `bias`) report a confidence from their
+  posterior variance, like the other layers: it reflects whether the data varied
+  their inputs, not how many samples there are.
+- The zone form's sensor weights are numbered fields with the sensors listed
+  above them (English and Romanian), and the zone form explains the two kinds of
+  weight.
+
+### Persisted data
+
+- `schema_version` stays 2. New optional keys: `fingerprint_version`,
+  `fingerprints`, `model_changes` (last 20), `operational_evaluation` (Forecast
+  Error statistics, 7 days) and per-source baseline fingerprints in
+  `heat_sources`. Retired zone models keep their fingerprint.
+- Model Error statistics gain RMSE columns; 0.3 and 0.4.0 statistics load (their
+  RMSE stays empty until new forecasts are scored).
+- Pending published forecasts are kept in memory only; a restart drops those not
+  yet due.
+
 ## 0.4.0 — 2026-10-27
 
 Thermal zones, electrical heat sources, windows with ventilation and wind, and

README.md

@@ -26,14 +26,24 @@ heat source** and **Add window** create items that can each be edited or deleted
 later. **Configure** holds everything else: thermostat, relay, weather sources,
 report-age limits, MPC settings and model options.
 
-**Zones** are rooms or areas. Each has a name, one or more temperature sensors with
-sensor weights (how much each counts in the zone temperature), a comfort weight
-(how much the zone matters to the MPC), an optional target offset relative to the
-thermostat's target, and optional adjacent zones it exchanges heat with. Sensor and
-comfort weights never change the learned thermal coefficients. A sensor belongs to
-one zone only. Upgrading from 0.3 turns each room sensor into a one-sensor zone
-with the same comfort weight; to group two sensors of one room, edit one zone to
-include both and delete the other.
+**Zones** are rooms or areas. Each has a name, one or more temperature sensors, a
+comfort weight (how much the zone matters to the MPC), an optional target offset
+relative to the thermostat's target, and optional adjacent zones it exchanges heat
+with. A zone with several sensors gets a second step, **Sensor weights**: one
+numbered field per sensor (listed above the fields), setting how much each counts
+in the zone temperature. A sensor belongs to one zone only. Upgrading from 0.3
+turns each room sensor into a one-sensor zone with the same comfort weight; to
+group two sensors of one room, edit one zone to include both and delete the other.
+
+**What a change does to the learned model.** The zone temperature is defined by
+its sensors and the ratio of their weights; changing either recalibrates that
+zone (its coefficients are kept as a starting point, but their confidence, sample
+counts and accuracy statistics start over, so *Ready* has to be earned again).
+Scaling all of a zone's sensor weights by the same factor changes nothing. Adding,
+editing or removing a heat source, a neighbour, a window, a window's orientation
+or the recovery time resets only the coefficients that depend on it. Comfort
+weight, target offset and names never touch the model. Each recalibration is
+listed in the Status attribute `model_changes`.
 
 **Heat sources** are equipment whose electricity ends up as heat in a zone, such as
 a UPS feeding computers. Settings: an active-power sensor (W or kW; VA and % are
@@ -48,7 +58,11 @@ modeled; the persistent load is already part of the zone's learned behavior.
 reports *on* when closed (inverted), the direction the outside of the window faces
 (degrees clockwise from north; default 345°, editable) and an opening factor
 (1.0 normal, lower for tilted, higher for large). Unknown or unavailable states are
-treated as unknown, never as closed.
+treated as unknown, never as closed: a window last seen open counts as possibly
+open (the forecast assumes it open, the zone's base model does not learn), and an
+opening that ended while the sensor was unknown is logged as unconfirmed, with its
+longest possible duration. Recovery after closing is learned only from a confirmed
+close.
 
 Model options in **Configure**: how fast extra electrical heat is assumed to fade in
 forecasts (half-life, default 30 minutes), how long an open window is assumed to
@@ -107,13 +121,16 @@ Entities (entity IDs may vary):
 - **Heating MPC Last Sample** — what the last poll did for learning (for example
   *Learned*, *Relay switched*, *Waiting for radiators to settle*); its history shows
   when learning happens.
-- **Heating MPC Forecast Error 15/30/60/120 min** — how far the thermal model's
-  forecasts were from what actually happened (see *Forecast accuracy*).
+- **Heating MPC Model Error 15/30/60/120 min** — how well the thermal model
+  explains what happened, replayed with the inputs that actually occurred.
+- **Heating MPC Forecast Error** — how far published forecasts were from what
+  happened, using only what was known when they were made (see *Forecast
+  accuracy*).
 
 ## Known limitations / safety
 
 - **Do not** connect the proposed binary sensor directly to the heating switch or build an automation from it. A production controller would need watchdog, manual override, safety handling, and an exclusive relay owner with Generic Thermostat fallback.
-- Weather variables other than temperature, wind and gusts are **captured but not fitted**. Wind and gusts are used only for open windows. Keep optional model terms only when **Forecast Error** shows they help (`improvement_vs_base`).
+- Weather variables other than temperature, wind and gusts are **captured but not fitted**. Wind and gusts are used only for open windows. Keep optional model terms only when **Model Error** shows they help (`improvement_vs_base`).
 - One temperature per zone (no separate wall mass), configured response times, and single-scenario forecasts of electrical heat and windows. See [docs/MODEL.md](docs/MODEL.md#limitations).
 - All rooms share one heating actuator. The model does not command individual radiators.
 - No weather forecast: assumes outdoor temperature remains constant during simulated horizon.
@@ -142,7 +159,10 @@ with it OFF, which can take a few days depending on how often the boiler runs.
 The **Heating MPC Status** entity's state is one of `waiting_for_sensors`,
 `no_zones` (add a zone), `learning`, `paused` (thermostat not in `heat`), `waiting_for_target`, `ready`,
 `inputs_changed` or `prediction_error`. Proposed heat is available only while it
-is `ready`. Useful attributes:
+is `ready`. `inputs_changed` means an input moved meaningfully while the plan was
+being computed (beyond 0.25 °C, 50 W, 1 m/s wind, 2 m/s gust or 20° direction, or
+any change of a window, the relay or the thermostat); `changed_inputs` lists them,
+the plan is discarded and the next poll computes a fresh one. Useful attributes:
 
 - `input_quality` — per zone (with each of its sensors) and per weather source
   (grouped by metric): `status` (`valid`, `missing`, `unknown`, `unavailable`,
@@ -156,8 +176,10 @@ is `ready`. Useful attributes:
 - `heat_sources` — per source: measured and effective power (after removing
   included sources), baseline, variable power above it, thermal response, whether it
   has settled.
-- `windows` — per window: state, minutes open, wind direction relative to the
-  window's facing and the resulting exposure (1 = wind straight at it).
+- `windows` — per window: state (`open`, `closed` or `unknown`), the last known
+  state and for how long the sensor has been unknown, minutes open, wind direction
+  relative to the window's facing and the resulting exposure (1 = wind straight at
+  it).
   `ventilation` — per zone: current opening, recovery, temperature before opening,
   its contribution in °C/h and the ventilation coefficients. `ventilation_events` —
   the last five openings with temperatures before, lowest, at closing and 30
@@ -170,14 +192,20 @@ is `ready`. Useful attributes:
   `ventilation` (only window-affected zones; given to the ventilation model),
   `partly_rejected` (an outlier in some zone), `interval_out_of_range`,
   `relay_changed`, `outside_source_changed`, `lag_uncertain` or `waiting_for_inputs`.
-- `models` — per-zone base coefficients, accepted sample counts and the in-sample
-  5-minute error (`mae_5m`, **not** a forecast accuracy), `model_in_use` and the
-  extended layer's coefficients with confidence.
+- `models` — per-zone base coefficients with their `confidence`, accepted sample
+  counts and the in-sample 5-minute error (`mae_5m`, **not** a forecast accuracy),
+  `model_in_use` and the extended layer's coefficients with confidence. Confidence
+  (0 to 1) is the share of a coefficient's information that comes from data rather
+  than the starting assumption. It grows only while the data varies that
+  coefficient's input: weeks with the relay always off leave `heat` at 0, however
+  many samples there are.
+- `model_changes` — the last recalibrations: when, which zone, which coefficients
+  were reset and why.
 - `retired_zone_models` — removed zones whose learned models are kept.
 
-`input_quality`, `weather_comparison`, `heat_sources`, `windows`, `ventilation`
-and `ventilation_events` change every poll, so they are not written to the
-Recorder; the other attributes are.
+`input_quality`, `weather_comparison`, `heat_sources`, `windows`, `ventilation`,
+`ventilation_events` and `model_changes` are not written to the Recorder; the other
+attributes are.
 
 ### Choosing freshness limits
 
@@ -216,22 +244,35 @@ age limit.
 
 ## Forecast accuracy
 
-The four **Forecast Error** sensors measure the thermal model against reality,
-continuously and out of sample. At every poll the integration records each zone's
-reading, the outdoor temperature, the radiator-lag estimate and the model's
-parameters. 15, 30, 60 and 120 minutes later it replays the zones from the earlier
-reading using the parameters it had **then**, the outdoor temperature held at its
-starting value, and the radiator heat that actually followed from the relay's real
-ON/OFF history. The result is compared with the current reading.
+Two different questions, two kinds of sensor:
 
-This tests the thermal model, not the MPC plan, because the plan is never
-executed. All zones are replayed together, with the electrical heat and window
-openings that actually happened, and the base model is scored on the same spans. Each sensor's state is the mean absolute error (°C) over the last 7 days,
-across all zones. Attributes, all in °C except counts:
+- **Model Error** (four sensors, 15/30/60/120 min): *does the thermal model explain
+  what happened?* Replayed with the relay, electrical heat and windows that
+  actually followed. Useful for judging the model and its options. It is **not**
+  a forecast accuracy: a real forecast does not know when the relay will switch or
+  a window will open.
+- **Forecast Error** (one sensor): *how good were the forecasts as published?*
+  Uses only what was known when each forecast was made.
 
-- `samples` — number of scored forecasts. Consecutive forecasts overlap, so they
-  are not independent; treat small counts as indicative only.
-- `bias` — mean error: negative means the model predicts too cold.
+### Model Error
+
+At every poll the integration records each zone's reading, the outdoor
+temperature, the radiator-lag estimate and the model's parameters. 15, 30, 60 and
+120 minutes later it replays all zones together from that reading using the
+parameters it had **then**, the outdoor temperature held at its starting value,
+and the relay, electrical heat and window history that actually followed. The base
+model is scored on the same spans. Each sensor's state is the mean absolute error
+(°C) over the last 7 days, across all zones. Attributes, all in °C except counts:
+
+- `samples` — number of scored forecasts; `effective_samples` — roughly
+  independent ones. Forecasts start every 5 minutes, so 60-minute forecasts
+  overlap twelvefold, and zones share one boiler, so `effective_samples` is
+  `samples × 5 / horizon / zones`.
+- `quality` — `insufficient` below 5 effective samples (the state then stays
+  unknown; the numbers are still in the attributes), `indicative` below 30, `ok`
+  from 30.
+- `mae`, `rmse`, `bias` — mean absolute, root-mean-square and mean error
+  (negative bias: the model predicts too cold).
 - `persistence_mae` — error of simply assuming no change; the baseline to beat.
 - `skill` — `1 - mae / persistence_mae`: 1 is perfect, 0 is no better than
   assuming no change, negative is worse. Empty while the temperature did not change.
@@ -240,13 +281,35 @@ across all zones. Attributes, all in °C except counts:
   make it worse: switch that option off).
 - `zones` — the same numbers per zone (not written to the Recorder).
 
-Forecasts are scored only where the inputs were known: polls 4.5–5.5 minutes
-apart, a usable reading at the start and the end, and a well-known radiator-lag
-estimate throughout. A missing relay, outdoor temperature or all zones restarts the
-replay window, so scores resume 15–120 minutes later. Statistics are kept with the
-model and survive restarts; deleting the model resets them.
+Spans are scored only where the inputs were known: polls 4.5–5.5 minutes apart, a
+usable reading at the start and the end, and a well-known radiator-lag estimate
+throughout. A missing relay, outdoor temperature or all zones restarts the replay
+window, so scores resume 15–120 minutes later. Statistics are kept with the model
+and survive restarts; recalibrating a zone or deleting the model resets them.
 
-What it does not cover: outdoor temperature changes during the horizon (held
+### Forecast Error
+
+Each time a plan is published, its forecast is recorded as it was: start
+temperatures, the trajectory, the plan, the model revision and the assumptions
+(outdoor temperature and wind held, electrical heat fading, open windows closing).
+When the forecast's horizon comes, it is scored against the reading, but only if
+the relay actually followed the plan: in every 15-minute block up to that horizon
+the relay must have been in the planned state for at least 12 of the 15 minutes.
+Otherwise the forecast is not an error measurement, and is counted instead:
+
+- `counterfactual` — the relay did something else: Generic Thermostat, which
+  controls it, disagreed with the plan. Expected in observation mode whenever the
+  two disagree.
+- `unverifiable` — the relay history is incomplete for that span.
+- `missing` — no poll near the due time, or no reading for a zone.
+
+The state is the 60-minute MAE once its quality is at least `indicative`; the
+attribute `horizons` (not written to the Recorder) gives MAE, RMSE, bias, the
+no-change error, `improvement_vs_persistence`, sample counts, quality, the counts
+above and the per-zone figures for each horizon. Statistics cover 7 days and
+survive restarts; forecasts not yet due when HA restarts are dropped.
+
+What neither covers: outdoor temperature changes during the horizon (held
 constant, as in the MPC), and periods when the model was still learning (included,
 so early numbers are pessimistic). The roadmap's working target is about
 0.3–0.4 °C at 60 minutes with clearly positive skill, before any control is
@@ -287,7 +350,8 @@ The brand icons in `custom_components/heating_mpc/brand/` are drawn by
 `python scripts/make_brand_icons.py` (uses Pillow from the test requirements);
 the package check verifies their sizes and transparency.
 
-HA tests cover setup, config/options forms, migration, reload and entity cleanup,
+HA tests cover setup, config/options forms, migration, reload (a single relay
+listener and polling coordinator after repeated reloads) and entity cleanup,
 scheduled sampling and poll cancellation, the relay listener, stored model
 restoration, retention and deletion, missing inputs, manual off/presets, optimizer
 failure and Recorder exclusions. A fixture rejects all HA service calls throughout

ROADMAP.md

@@ -85,6 +85,16 @@ Release 0.2.2 adds that event-based relay history (read-only listener), keeps
 models of rooms removed in Options, shows each sensor's longest report gap, stops
 recording volatile attributes, and tests against HA 2025.12.0 and 2026.2.3.
 
+### Stabilization after 0.4.0 — 2026-10-27
+
+See AUDIT.md. Four confirmed problems fixed, each with regression tests: unknown
+window states read as closed; learned coefficients reused after incompatible
+configuration changes (now per-component fingerprints with selective
+recalibration, logged in `model_changes`); plans published from inputs that
+changed during optimization or by an overtaken run; and model accuracy reported
+as if it were forecast accuracy. Reload/unload tests now cover repeated reloads
+(one relay listener, one polling coordinator). Not yet released or tested on HAOS.
+
 ## Milestone 1 — Explainable forecasts and dashboards (target v0.3.0)
 
 **Objective:** Users can verify what the model predicts and whether it is actually accurate.
@@ -92,9 +102,9 @@ recording volatile attributes, and tests against HA 2025.12.0 and 2026.2.3.
 - [ ] Expose trajectories for each room at 5–15-minute resolution for 2–3 hours, with forecast generation timestamp, horizon and input provenance. *(0.3.0: 15-minute trajectory over the horizon; generation timestamp and provenance still to add.)*
 - [x] Expose predicted +15/+30/+60/+120 minute temperatures and full proposed ON/OFF schedule. *(0.3.0: `forecast_15min` and `plan_15min` on each room forecast entity.)*
 - [ ] Provide an HA dashboard example using native cards or a clearly documented optional chart card (do not bundle a heavyweight frontend unnecessarily).
-- [ ] Make clear that the **MPC plan forecast is counterfactual** while Generic Thermostat is controlling the relay; add a separate actual-policy forecast/evaluation pathway. *(0.3.0: plan forecasts labelled counterfactual; evaluation uses the actual relay history. An actual-policy forecast, simulating Generic Thermostat, is still open.)*
-- [ ] Store a bounded history of predictions with snapshots of inputs, proposed plan and eventually observed outcome.
-- [x] Implement out-of-sample/walk-forward MAE and bias at 15, 30, 60 and 120 minutes; include sample counts and confidence/quality labels. *(0.3.0: Forecast Error sensors; replay with parameters frozen at the forecast start and the actual relay history; `skill` against persistence. No confidence intervals yet.)*
+- [ ] Make clear that the **MPC plan forecast is counterfactual** while Generic Thermostat is controlling the relay; add a separate actual-policy forecast/evaluation pathway. *(0.3.0: plan forecasts labelled counterfactual; evaluation uses the actual relay history. Stabilization: published forecasts are scored only when the relay followed their plan and otherwise counted as `counterfactual`. An actual-policy forecast, simulating Generic Thermostat, is still open.)*
+- [ ] Store a bounded history of predictions with snapshots of inputs, proposed plan and eventually observed outcome. *(Stabilization: each published forecast is kept with its start readings, plan, model revision and assumptions until scored, in memory only; the scored statistics persist. No browsable history yet.)*
+- [x] Implement out-of-sample/walk-forward MAE and bias at 15, 30, 60 and 120 minutes; include sample counts and confidence/quality labels. *(0.3.0: replay with parameters frozen at the forecast start and the actual relay history; `skill` against persistence. Stabilization: these are now the Model Error sensors, with RMSE, effective sample counts and quality labels; a separate Forecast Error sensor scores published forecasts. No confidence intervals yet.)*
 - [ ] Compare against simple benchmarks such as current-temperature persistence and a fixed thermal baseline. *(0.3.0: persistence done; fixed baseline open.)*
 - [ ] Add inspectable explanations: expected heat loss, expected heating response, target, room weights and reason for the next proposed action.
 
@@ -175,8 +185,11 @@ Items 1–3 of the original backlog are done (see AUDIT.md and CHANGELOG.md 0.2.
    control, decide a minimum `weight_coverage` and the outdoor-temperature outage
    policy, which still pauses.
 3. Configure zones, the UPS heat source and the windows on HAOS; after a week of
-   heating review `improvement_vs_base`, per-zone bias and which coefficients are
-   identified. Switch off optional terms that do not help.
+   heating review Model Error `improvement_vs_base` and `quality`, per-zone bias,
+   coefficient `confidence` (base layer included), `model_changes` after any
+   reconfiguration, and the Forecast Error counts (how many forecasts were scored
+   vs counterfactual). Switch off optional terms that do not help. Also try an
+   unavailable window sensor and a zone edit during HAOS acceptance.
 4. Milestone 1 (rest): a dashboard example; forecast timestamps/provenance;
    explanations for the proposed action; an actual-policy forecast that simulates
    Generic Thermostat.

docs/MODEL.md

@@ -93,9 +93,16 @@ measure the input side or lower the heat fraction accordingly.
 
 Present when a zone has an enabled window. Learned separately, from the part of
 the measured change the closed-window layers cannot explain, only while a window is
-open, its state is unknown, or the zone is recovering. During those periods the
-base and extended layers do **not** learn, so window events cannot inflate `loss`
-or shift `bias`.
+open or the zone is recovering, and only while every window state in the zone is
+known. While a window is open, possibly open (its sensor is unknown and it was last
+seen open) or the zone is recovering, the base and extended layers do **not**
+learn, so window events cannot inflate `loss` or shift `bias`.
+
+An unknown window state is never read as closed. A window last seen open stays
+assumed open in forecasts; one last seen closed contributes no ventilation term.
+If a window closes while its sensor was unknown, the event's duration is reported
+as a range and the recovery term is not learned from it, because the closing time
+is not known.
 
 `vent_wind_dir` is bounded by `vent_wind` (|vent_wind_dir| ≤ vent_wind), so the
 combined wind term never goes negative: a leeward window still ventilates.
@@ -128,11 +135,20 @@ outlier rejection above 0.8 °C per 5 minutes.
 Each coefficient reports:
 
 * `confidence`: the share of its information that comes from data rather than the
-  prior (0 = prior only);
+  prior (0 = prior only), from its posterior variance. It is not a sample count:
+  it grows only while the data varies the coefficient's input independently of
+  the others, and with forgetting it falls again during long periods without such
+  variation. The base layer reports it too (`loss`, `heat`, `bias`);
 * `std`: its standard error, using a running estimate of the unexplained noise;
 * `samples`: how many accepted samples actually exercised it;
 * `identified`: at least 30 exercising samples and confidence ≥ 0.8.
 
+The base layer has no per-coefficient sample counts or standard errors; its
+`confidence` alone shows identification. In the tests, two weeks of samples with
+the relay always off and a steady indoor-outdoor difference leave `heat` at 0 and
+`loss` and `bias` below 0.8, while four days with the relay cycling and the
+weather changing bring all three above 0.8.
+
 A coefficient can move without being identified. Example from the tests: two rooms
 whose temperature difference stays fairly steady cannot separate exchange between
 them from each room's bias; the learner finds a positive coupling but correctly
@@ -154,6 +170,27 @@ A zone learns from the interval between two polls only when:
 No experiments are run and the boiler is never switched to gather data: everything
 is learned passively.
 
+## Configuration changes
+
+Every learned component stores a fingerprint of the configuration it was learned
+under (`custom_components/heating_mpc/compatibility.py`). At startup, components
+whose fingerprint changed are reset; the rest are kept.
+
+| Change | Effect |
+|---|---|
+| A zone's sensors, or the ratio of their weights | Zone recalibrated: coefficients kept as a starting point; uncertainty, counts and accuracy statistics reset. Its extended and ventilation layers start over too (their loss, heat and bias values kept as a starting point) |
+| Scaling all of a zone's weights equally | Nothing (the normalized measurement is unchanged) |
+| A heat source's sensor, heat share, baseline, response time or "included in" | `internal` of its zone; the source's baseline samples if its effective power changed |
+| A neighbour's sensors or weights | `couple:<neighbour>` |
+| A window's sensor, polarity or opening factor | that zone's ventilation terms; `vent_cross` of the other zones |
+| A window's facing direction | `vent_wind_dir` |
+| Recovery time constant | `recovery` |
+| Comfort weight, target offset, names | Nothing (not physical) |
+
+Models saved before fingerprints existed (0.4.0) are assumed to match the
+configuration they are loaded with, and are fingerprinted at once. Every reset is
+logged in `model_changes` with its reason.
+
 ## Forecasts
 
 Forecasts simulate all zones together, step by step, over the MPC horizon (60–180
@@ -177,18 +214,34 @@ they describe what would happen under the MPC's proposed plan.
 
 ## Validation
 
-The Forecast Error sensors score forecasts that are valid for comparison with
-measured temperatures. At every poll the model parameters are frozen; 15, 30, 60
-and 120 minutes later all zones are replayed from that poll using the relay,
-electrical heat and window history that actually followed (outdoor temperature held
-at its start value), and compared with the readings. The base layer and a
-no-change forecast are scored on exactly the same spans:
+Two separate measurements, because they answer different questions.
 
-* `mae`, `bias`: the full model's error;
+**Model accuracy (Model Error sensors)** scores the model, not a forecast. At
+every poll the model parameters are frozen; 15, 30, 60 and 120 minutes later all
+zones are replayed from that poll using the relay, electrical heat and window
+history that actually followed (outdoor temperature held at its start value),
+and compared with the readings. The base layer and a no-change forecast are
+scored on exactly the same spans:
+
+* `mae`, `rmse`, `bias`: the full model's error;
 * `persistence_mae`, `skill`: against assuming no change;
 * `base_model_mae`, `improvement_vs_base`: against the base layer alone. A feature
   that does not improve this should be switched off (for example the gust term).
 
+**Forecast accuracy (Forecast Error sensor)** scores forecasts as published: the
+start readings, trajectory, plan, model revision and assumptions of the moment
+they were made, nothing replaced with hindsight. Because the plan is the MPC's
+proposal while Generic Thermostat controls the relay, a forecast is scored at a
+horizon only if the relay was in the planned state for at least 12 minutes of
+every 15-minute block up to it; otherwise it is counted as `counterfactual` (or
+`unverifiable` when the relay history is incomplete) and never as an error.
+
+Both report how much evidence there is. Forecasts start every 5 minutes, so at
+horizon h consecutive forecasts share most of their span, and all zones share one
+boiler: `effective_samples = samples × 5 / h / zones`. Below 5 the quality is
+`insufficient` and the sensor's state stays unknown; below 30 it is
+`indicative`.
+
 Results on synthetic apartments (the tests; these check the method, not real-world
 accuracy):