diff --git a/AUDIT.md b/AUDIT.md index 5abdebb..40fda77 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -397,3 +397,73 @@ Remaining risks: ruff, 159 offline tests, 28 HA Core tests; the offline tests also pass on Python 3.9.6. `sh scripts/check.sh` additionally validates packaging, translations and the changelog. + +## Final stability fixes after 0.4.1 — 2026-10-27 + +Scope: four issues from an independent review of 0.4.1, plus a user-requested +*Reset learned model* action. Repository code and tests only (HA Core 2025.12.0 +and 2026.2.3 harness); no HAOS, device or deployment testing. Still no actuator +code; the static and runtime service-call guards pass. Version unchanged (0.4.1). + +Confirmed issues, root causes and fixes: + +1. **Model Error scored spans with unknown ventilation inputs** (2cfba02). + Reproduced: a window unknown for a whole 15-minute span gave a Model Error + sample (MAE 0.76 °C). Root cause: the replay used the prediction path, which + drops the ventilation term when the opening is unknown, i.e. assumes closed. + Fix: per zone and step, `inputs_complete` checks every input the zone's model + used (window opening and confirmed recovery; wind, direction and other zones' + windows while a window is open; electrical heat and neighbours for the extended + layer, including the neighbour's own completeness). Incomplete spans are + counted, not scored, with the base model's error kept apart; unreplayable + spans are counted as skipped. Learning also stopped substituting a zero facing + term or zero cross-ventilation input for unknown values. +2. **Window unknown at startup treated as closed** (13cb972). Reproduced + (0 % assumed opening). Root cause: last known states lived only in memory, and + "never observed" fell through to the closed case; a first reading of an open + window was also logged as a confirmed opening. Fix: confirmation only for + witnessed transitions; last known states, open events and recovery saved (also + on every window state change) and restored as unknown; per-window forecast + assumptions (observed / last known open, bounded by the open-window time after + it was last seen / last known closed / unobserved), exposed in Status and on + zone forecasts; durations as ranges when an end was not seen. Chosen over + open/closed scenario pairs to avoid a second plan simulation per poll. +3. **Lifecycle and storage races** (28ff953). Confirmed by reading HA's unload + order: the coordinator's own shutdown runs only after platforms unload, and the + save checked for shutdown before waiting for the storage lock (check-then-act). + Found during testing: unloading cancels a running scheduled update, which + released the lock while the file write continued in a worker thread, so a + deletion could run first and be undone. Fix: closing mark at the start of + unload (restored if unload fails); checks after acquiring the lock, including a + removed-model mark set under the lock by removal; the write runs as a shielded, + HA-tracked task that holds the lock until it finishes; failed setup shuts the + coordinator down; queued relay callbacks are ignored once closing. Mutation + checks: removing the locked check or the shield makes the deletion tests fail; + removing the early closing mark fails the ordering test (the HA-level unload + test cannot distinguish it, as HA's own shutdown also lands before release). +4. **Forecast Error accepted materially different relay behaviour** (0b82a8e). + Reproduced: 12 of 15 planned minutes was scored. Root cause: a per-block + on-share tolerance of 20 %, blind to timing within a block. Fix: deviation = + time in another state than planned, block by block from the relay's switch + times, with unobserved time as possible deviation; matched ≤ 2 min (scored), + approximate ≤ 5 min (separate), counterfactual > 5 min observed, otherwise + incomplete with a reason; forecasts made with assumed inputs are incomplete. + Ledger statistics are versioned; 0.4.1 figures are dropped. + +Added: *Reset learned model* (49565d6): unload, strip learned keys from the +stored model under the lock, set up again; configuration, heat-source baselines +and window history kept. + +Tests added (by area): Model Error incomplete/skipped (7), plan matching (10), +window startup/restart/mixed/strict inputs (9), coordinator window reload and +uncertain forecast (2), lifecycle offline (5) and in HA (4), reset flow (1). + +Remaining limitations: unknown-but-last-closed and never-seen windows still +produce a closed-window forecast (flagged, never scored); the 2 and 5 minute +limits rest on typical learned heat coefficients, not a per-forecast calculation; +spans lost when the replay window restarts are not counted; saved window state can +be up to one save old, which only makes the restored unknown period longer. + +**Verification.** `sh scripts/check.sh` passes in both environments (pip check, +ruff, 192 offline tests, 33 HA Core tests, compilation, package/translation/ +changelog checks, diff whitespace); the offline tests also pass on Python 3.9.6. diff --git a/CHANGELOG.md b/CHANGELOG.md index 4823f02..1e98135 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,78 @@ Engineering findings and test evidence behind each change are in [AUDIT.md](AUDIT.md). +## Unreleased — final stability fixes after 0.4.1 + +Four correctness fixes from an independent review of 0.4.1, and a **Reset learned +model** action you asked for. The version stays 0.4.1 until released. Still +**observation-only**: no service calls; the boiler and thermostat are untouched. +Tested in the Home Assistant Core harness with HA 2025.12.0 and 2026.2.3; not on +HAOS. + +### Upgrade notes + +- No reconfiguration needed. Zones, sensor and comfort weights, weather sources, + heat sources, windows, learned coefficients and fingerprints carry over; entity + IDs are unchanged. +- **Configure now opens a menu**: *Settings* (the previous form) or *Reset learned + model*. +- **Forecast Error statistics restart.** 0.4.1 accepted plans the relay followed + within 20 % per block; those figures are not comparable with the new rule and + are dropped on the first start (they covered at most a day). +- Model Error statistics carry over. Those from 0.4.0/0.4.1 may include a few + spans with unknown window states; they age out within 7 days. + +### Added + +- **Reset learned model** (Configure, with a confirmation): forgets every zone's + learned coefficients, models kept for removed zones, and the Model Error and + Forecast Error statistics; keeps the configuration, heat-source baselines and the + window event log, and logs the reset in `model_changes`. + +### Fixed + +- **Model Error no longer scores spans with unknown inputs.** A window unknown + during a span was replayed as closed and scored. Spans where a window state, an + open window's weather input, or the extended layer's electrical heat or + neighbour was unknown are now counted as `incomplete` (with the base model's + error kept apart as `incomplete_base_model_mae`); spans that cannot be replayed + are counted as `skipped`. +- **Window sensors unavailable at startup are unknown, not closed.** Last known + states, open events and recovery are saved and restored as unknown until the + sensor reports; openings and closings count as confirmed only when seen; an + unknown window last seen open is assumed open only until the open-window time + after it was last seen; other unknown windows are flagged as uncertain in + forecasts (`uncertain_windows`) and in Status (`forecast_assumption`). Event + durations are given as a range when an end was not seen. +- **Unload, reload and removal cannot publish or save stale work.** The + coordinator is marked closing before platforms unload; the model save checks + this, and a removed-model mark, after taking the storage lock and holds the lock + until the file write finishes, so a deletion can no longer be undone by a save + that was waiting or cancelled mid-write. A failed setup shuts the coordinator + down; relay callbacks queued before unload are ignored. +- **Forecast Error compares the relay's actual timing with the plan.** The + deviation is the time the relay spent in another state than planned (unobserved + time counts as possible deviation). Up to 2 minutes: matched and scored; up to 5: + approximate, scored separately; more: counterfactual; otherwise incomplete, with + a reason (relay history, assumed inputs, no poll at the due time, missing + reading). Forecasts made with an unknown window or electrical heat are never + scored. + +### Changed + +- Learning no longer substitutes values for an open window: samples where the + wind direction (for the directional term) or another zone's windows (for + cross-ventilation) were unknown are not learned. With every window closed, the + opening terms need no weather input, so recovery can be learned without wind. +- The gust option's hint refers to Model Error. + +### Persisted data + +- `schema_version` stays 2. `ventilation` gains `state` (last known window + states, open events, recovery); `evaluation` gains `excluded` (incomplete and + skipped spans); `operational_evaluation` gains `version` 2 and `approximate`. + Missing optional keys load as empty. + ## 0.4.1 — 2026-10-27 Stability and correctness: fixes four problems found in 0.4.0 and separates model diff --git a/README.md b/README.md index f649c66..f3a4d26 100644 --- a/README.md +++ b/README.md @@ -66,12 +66,24 @@ modeled; the persistent load is already part of the zone's learned behavior. **Windows** need a contact sensor (binary sensor), the zone, whether the sensor reports *on* when closed (inverted), the direction the outside of the window faces (degrees clockwise from north; default 345°, editable) and an opening factor -(1.0 normal, lower for tilted, higher for large). Unknown or unavailable states are -treated as unknown, never as closed: a window last seen open counts as possibly -open (the forecast assumes it open, the zone's base model does not learn), and an -opening that ended while the sensor was unknown is logged as unconfirmed, with its -longest possible duration. Recovery after closing is learned only from a confirmed -close. +(1.0 normal, lower for tilted, higher for large). + +A window is **open**, **closed** or **unknown** (unknown, unavailable or no state); +unknown never means closed. While any window of a zone is unknown, the zone does +not learn. What forecasts assume for an unknown window: + +- last seen open: open, but only until the configured open-window time after it was + last seen (never renewed), then closed and recovering; +- last seen closed, or never seen since setup: no ventilation, but the forecast is + flagged (`uncertain_windows` on the zone's forecast entity, `forecast_assumption` + in Status) and is never used to measure forecast accuracy. + +An opening or closing is *confirmed* only when it was seen happen: the previous +reading was the other state, with no unknown period or restart in between. Events +report a duration only when both ends are confirmed, otherwise a minimum and, where +possible, a maximum; recovery is learned only after a confirmed close. The last +known states, an open event and a recovery in progress are saved, so a restart or +reload continues them, but they come back as unknown until the sensor reports. Model options in **Configure**: how fast extra electrical heat is assumed to fade in forecasts (half-life, default 30 minutes), how long an open window is assumed to @@ -186,9 +198,10 @@ the plan is discarded and the next poll computes a fresh one. Useful attributes: included sources), baseline, variable power above it, thermal response, whether it has settled. - `windows` — per window: state (`open`, `closed` or `unknown`), the last known - state and for how long the sensor has been unknown, minutes open, wind direction - relative to the window's facing and the resulting exposure (1 = wind straight at - it). + state and for how long the sensor has been unknown, what forecasts assume + (`forecast_assumption`: `observed`, `last_known_open`, `last_known_closed` or + `unobserved`; `forecast_open_minutes`), minutes open, wind direction relative to + the window's facing and the resulting exposure (1 = wind straight at it). `ventilation` — per zone: current opening, recovery, temperature before opening, its contribution in °C/h and the ventilation coefficients. `ventilation_events` — the last five openings with temperatures before, lowest, at closing and 30 @@ -288,35 +301,57 @@ model is scored on the same spans. Each sensor's state is the mean absolute erro - `base_model_mae`, `improvement_vs_base` — the base model's error on the same spans, and the share of it the extra layers remove (0 = no gain, negative = they make it worse: switch that option off). +- `incomplete` — spans left out because an input the zone's model needed was not + observed at some step: a window state (or the start of a recovery), the wind, + wind direction or another zone's windows while a window was open, or, for the + extended layer, electrical heat or a neighbour (whose own span must be complete + too). The base model needs none of these, so its error on those spans is still + reported as `incomplete_base_model_mae`. +- `skipped` — spans that could not be replayed: the radiator-lag estimate was + uncertain along the way, or the zone had no reading at the end. - `zones` — the same numbers per zone (not written to the Recorder). -Spans are scored only where the inputs were known: polls 4.5–5.5 minutes apart, a -usable reading at the start and the end, and a well-known radiator-lag estimate -throughout. A missing relay, outdoor temperature or all zones restarts the replay -window, so scores resume 15–120 minutes later. Statistics are kept with the model -and survive restarts; recalibrating a zone or deleting the model resets them. +Spans need polls 4.5–5.5 minutes apart; a missing relay, outdoor temperature or +all zones restarts the replay window, so scores resume 15–120 minutes later (these +gaps are not counted). Statistics are kept with the model and survive restarts; +recalibrating a zone or resetting the learned model clears them. ### Forecast Error Each time a plan is published, its forecast is recorded as it was: start temperatures, the trajectory, the plan, the model revision and the assumptions (outdoor temperature and wind held, electrical heat fading, open windows closing). -When the forecast's horizon comes, it is scored against the reading, but only if -the relay actually followed the plan: in every 15-minute block up to that horizon -the relay must have been in the planned state for at least 12 of the 15 minutes. -Otherwise the forecast is not an error measurement, and is counted instead: +When the forecast's horizon comes, the relay's actual behaviour is compared with +the plan, block by block, from the relay's own switch times: the **deviation** is +how long the relay was in another state than planned, so the same on-time at other +times than planned counts as deviation. Relay time that was not observed counts as +possible deviation. Each forecast is then: -- `counterfactual` — the relay did something else: Generic Thermostat, which - controls it, disagreed with the plan. Expected in observation mode whenever the - two disagree. -- `unverifiable` — the relay history is incomplete for that span. -- `missing` — no poll near the due time, or no reading for a zone. +- `matched` — at most 2 minutes of deviation. Only these make the Forecast Error + figures. +- `approximate` — at most 5 minutes (of which at most 2 unobserved), for example + the thermostat switching a few minutes earlier than planned. Scored separately + in `approximate_stats`. +- `counterfactual` — more than 5 minutes: Generic Thermostat, which controls the + relay, did something else. Not an error of the forecast. Expected in observation + mode whenever the two disagree. +- `incomplete` — not comparable, with `incomplete_reasons`: `relay_history` (too + much relay time unknown), `uncertain_inputs` (the forecast had to assume an + unknown window or electrical heat), `no_poll` (no poll near the due time) or + `reading` (a zone had no reading; counted per zone). -The state is the 60-minute MAE once its quality is at least `indicative`; the -attribute `horizons` (not written to the Recorder) gives MAE, RMSE, bias, the -no-change error, `improvement_vs_persistence`, sample counts, quality, the counts -above and the per-zone figures for each horizon. Statistics cover 7 days and -survive restarts; forecasts not yet due when HA restarts are dropped. +Why these limits: d seconds of boiler heat that the plan did not assume move a room +by roughly *heat* × d / 300, where *heat* is the learned boiler coefficient +(typically 0.05–0.12 °C per 5 minutes with warm radiators). 2 minutes stay below +half a typical 0.1 °C sensor step and absorb timestamp jitter; 5 minutes are about +one step. + +The state is the 60-minute MAE of matched forecasts once its quality is at least +`indicative`; the attribute `horizons` (not written to the Recorder) gives MAE, +RMSE, bias, the no-change error, `improvement_vs_persistence`, sample counts, +quality, the counts above, `approximate_stats` and the per-zone figures for each +horizon. Statistics cover 7 days and survive restarts; forecasts not yet due when +HA restarts are dropped. What neither covers: outdoor temperature changes during the horizon (held constant, as in the MPC), and periods when the model was still learning (included, @@ -359,9 +394,10 @@ The brand icons in `custom_components/heating_mpc/brand/` are drawn by `python scripts/make_brand_icons.py` (uses Pillow from the test requirements); the package check verifies their sizes and transparency. -HA tests cover setup, config/options forms, migration, reload (a single relay -listener and polling coordinator after repeated reloads) and entity cleanup, -scheduled sampling and poll cancellation, the relay listener, stored model +HA tests cover setup, config/options forms (including the reset), migration, +reload (a single relay listener and polling coordinator after repeated reloads), +unload, reload and removal while an optimization or a save is still running, +entity cleanup, scheduled sampling and poll cancellation, the relay listener, stored model restoration, retention and deletion, missing inputs, manual off/presets, optimizer failure and Recorder exclusions. A fixture rejects all HA service calls throughout each test, and a static test fails if actuator service calls are added to the diff --git a/ROADMAP.md b/ROADMAP.md index 02cdd52..2f6e074 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -95,6 +95,18 @@ changed during optimization or by an overtaken run; and model accuracy reported as if it were forecast accuracy. Reload/unload tests now cover repeated reloads (one relay listener, one polling coordinator). Released as 0.4.1; not yet tested on HAOS. +### Final stability fixes after 0.4.1 — 2026-10-27 + +An independent review of 0.4.1 reported four problems; three were reproduced and +the fourth (lifecycle races) confirmed by reading Home Assistant's unload order. +Fixed with regression tests (see AUDIT.md): Model Error spans with unknown inputs +are now `incomplete`, not scored; window sensors unavailable at startup are +unknown, with last known states saved and restored as unknown; unload, reload +and removal can no longer publish stale results or recreate a deleted model; +Forecast Error scores only forecasts whose plan the relay matched within 2 +minutes. Also added: **Reset learned model** in Configure. Not released or tested +on HAOS yet. + ## Milestone 1 — Explainable forecasts and dashboards (target v0.3.0) **Objective:** Users can verify what the model predicts and whether it is actually accurate. @@ -103,7 +115,7 @@ as if it were forecast accuracy. Reload/unload tests now cover repeated reloads - [x] Expose predicted +15/+30/+60/+120 minute temperatures and full proposed ON/OFF schedule. *(0.3.0: `forecast_15min` and `plan_15min` on each room forecast entity.)* - [ ] Provide an HA dashboard example using native cards or a clearly documented optional chart card (do not bundle a heavyweight frontend unnecessarily). - [ ] Make clear that the **MPC plan forecast is counterfactual** while Generic Thermostat is controlling the relay; add a separate actual-policy forecast/evaluation pathway. *(0.3.0: plan forecasts labelled counterfactual; evaluation uses the actual relay history. Stabilization: published forecasts are scored only when the relay followed their plan and otherwise counted as `counterfactual`. An actual-policy forecast, simulating Generic Thermostat, is still open.)* -- [ ] Store a bounded history of predictions with snapshots of inputs, proposed plan and eventually observed outcome. *(Stabilization: each published forecast is kept with its start readings, plan, model revision and assumptions until scored, in memory only; the scored statistics persist. No browsable history yet.)* +- [ ] Store a bounded history of predictions with snapshots of inputs, proposed plan and eventually observed outcome. *(Stabilization: each published forecast is kept with its start readings, plan, model revision and assumptions, including any it had to assume, until scored, in memory only; the scored statistics persist, split into matched, approximate, counterfactual and incomplete. No browsable history yet.)* - [x] Implement out-of-sample/walk-forward MAE and bias at 15, 30, 60 and 120 minutes; include sample counts and confidence/quality labels. *(0.3.0: replay with parameters frozen at the forecast start and the actual relay history; `skill` against persistence. Stabilization: these are now the Model Error sensors, with RMSE, effective sample counts and quality labels; a separate Forecast Error sensor scores published forecasts. No confidence intervals yet.)* - [ ] Compare against simple benchmarks such as current-temperature persistence and a fixed thermal baseline. *(0.3.0: persistence done; fixed baseline open.)* - [ ] Add inspectable explanations: expected heat loss, expected heating response, target, room weights and reason for the next proposed action. @@ -187,9 +199,12 @@ Items 1–3 of the original backlog are done (see AUDIT.md and CHANGELOG.md 0.2. 3. Configure zones, the UPS heat source and the windows on HAOS; after a week of heating review Model Error `improvement_vs_base` and `quality`, per-zone bias, coefficient `confidence` (base layer included), `model_changes` after any - reconfiguration, and the Forecast Error counts (how many forecasts were scored - vs counterfactual). Switch off optional terms that do not help. Also try an - unavailable window sensor and a zone edit during HAOS acceptance. + reconfiguration, and the Forecast Error counts (matched, approximate, + counterfactual, incomplete and why). Switch off optional terms that do not + help. During HAOS acceptance also try: a window sensor unavailable across an HA + restart, a zone edit, a reload while the integration is running, *Reset learned + model* and removing a disposable entry (its `.storage/heating_mpc.` + file must be gone afterwards). 4. Milestone 1 (rest): a dashboard example; forecast timestamps/provenance; explanations for the proposed action; an actual-policy forecast that simulates Generic Thermostat. diff --git a/docs/MODEL.md b/docs/MODEL.md index e397e6d..a8ec9cd 100644 --- a/docs/MODEL.md +++ b/docs/MODEL.md @@ -98,11 +98,19 @@ known. While a window is open, possibly open (its sensor is unknown and it was l seen open) or the zone is recovering, the base and extended layers do **not** learn, so window events cannot inflate `loss` or shift `bias`. -An unknown window state is never read as closed. A window last seen open stays -assumed open in forecasts; one last seen closed contributes no ventilation term. -If a window closes while its sensor was unknown, the event's duration is reported -as a range and the recovery term is not learned from it, because the closing time -is not known. +An unknown window state is never read as closed, and an opening or closing counts +as confirmed only when it was seen: the previous reading was the other state with +no unknown period (or restart) in between. If a window closes while its sensor was +unknown, the event's duration is reported as a range and the recovery term is not +learned from it, because the closing time is not known. Last known states, open +events and recovery are saved and restored across restarts, as unknown until the +sensor reports again. + +An input that a term needs is never filled in for learning or for Model Error: +while a window is open, the wind, its direction (for `vent_wind_dir`) and the other +zones' windows (for `vent_cross`) must be known. With every window closed, the +opening terms are 0 and need no weather. Forecasts, which must produce a number, +assume calm air for a missing wind input. `vent_wind_dir` is bounded by `vent_wind` (|vent_wind_dir| ≤ vent_wind), so the combined wind term never goes negative: a leeward window still ventilates. @@ -202,7 +210,10 @@ minutes): its response time. A gaming PC drawing 600 W now is not assumed to run on for two hours. * **Windows:** an open window is assumed to stay open for a configurable time - (default 15 minutes), then close; the zone then recovers. + (default 15 minutes), then close; the zone then recovers. A window whose sensor + is unknown but was last seen open is assumed open only until that time after it + was last seen, never renewed. One last seen closed, or never seen since setup, is + assumed closed, and the forecast is flagged as uncertain (`uncertain_windows`). * **Relay:** the candidate plan being evaluated. For the plan search the step is compiled into exact per-zone coefficients (every @@ -228,13 +239,34 @@ scored on exactly the same spans: * `base_model_mae`, `improvement_vs_base`: against the base layer alone. A feature that does not improve this should be switched off (for example the gust term). +A span counts only if every input the zone's model used was observed at every +step (see *Ventilation layer*; for the extended layer also electrical heat and +neighbours, whose own replay must be complete). Otherwise it is counted as +`incomplete`, with the base model's error on it kept apart +(`incomplete_base_model_mae`), since the base model needs only the relay and the +outdoor temperature. Spans that cannot be replayed (uncertain radiator lag, no +reading at the end) are counted as `skipped`. + **Forecast accuracy (Forecast Error sensor)** scores forecasts as published: the start readings, trajectory, plan, model revision and assumptions of the moment they were made, nothing replaced with hindsight. Because the plan is the MPC's -proposal while Generic Thermostat controls the relay, a forecast is scored at a -horizon only if the relay was in the planned state for at least 12 minutes of -every 15-minute block up to it; otherwise it is counted as `counterfactual` (or -`unverifiable` when the relay history is incomplete) and never as an error. +proposal while Generic Thermostat controls the relay, a forecast is compared only +when the relay followed it. The deviation is the time, up to the horizon, the +relay spent in another state than planned, compared block by block from the +relay's own switch times; unobserved relay time counts as possible deviation. + +| Deviation (worst case) | Class | Used for | +|---|---|---| +| ≤ 2 min | matched | Forecast Error figures | +| ≤ 5 min, ≤ 2 min unobserved | approximate | `approximate_stats` only | +| > 5 min observed | counterfactual | counted, never an error | +| otherwise | incomplete | counted, with a reason | + +Forecasts made with an assumed input (an unknown window or electrical heat) are +incomplete as well. The limits follow from the model: d seconds of unplanned +boiler heat change a room by about heat × d / 300 with the learned `heat` +coefficient (typically 0.05–0.12 °C per 5 minutes), so 2 minutes stay below half a +0.1 °C sensor step and 5 minutes are about one step. Both report how much evidence there is. Forecasts start every 5 minutes, so at horizon h consecutive forecasts share most of their span, and all zones share one @@ -267,5 +299,8 @@ the contamination the separate ventilation layer prevents. variation their split is uncertain even when their combined effect is accurate. * Forecasts hold weather constant and use one scenario for electrical heat and windows; no forecast uncertainty band is computed yet. -* Window sensors with a permanently unknown state keep their zone out of learning; - disable such a window. +* Window sensors with a permanently unknown state keep their zone out of learning + and its Model Error spans incomplete; disable such a window. +* A window that is unknown and was last seen closed (or never seen) is assumed + closed in forecasts, flagged as uncertain; no open/closed scenario pair is + computed.