Document the final stability fixes and the model reset

README: window states and forecast assumptions, Model Error incomplete and
skipped spans, Forecast Error classes and why 2 and 5 minutes, and the
lifecycle tests. docs/MODEL.md: strict inputs, unknown-window assumptions,
the plan-matching table. CHANGELOG gains an Unreleased section (version
unchanged); ROADMAP and AUDIT record the findings, root causes, evidence and
remaining limitations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
buzeraandClaude Opus 5.5 committed 2026-10-27 04:54:19 +02:00
1 parent 0b82a8e26c
commit c4035951df
5 files changed
+275 -47

No files matched your search

+70
View File
@@ -397,3 +397,73 @@ Remaining risks:
ruff, 159 offline tests, 28 HA Core tests; the offline tests also pass on Python
3.9.6. `sh scripts/check.sh` additionally validates packaging, translations and
the changelog.
## Final stability fixes after 0.4.1 — 2026-10-27
Scope: four issues from an independent review of 0.4.1, plus a user-requested
*Reset learned model* action. Repository code and tests only (HA Core 2025.12.0
and 2026.2.3 harness); no HAOS, device or deployment testing. Still no actuator
code; the static and runtime service-call guards pass. Version unchanged (0.4.1).
Confirmed issues, root causes and fixes:
1. **Model Error scored spans with unknown ventilation inputs** (2cfba02).
Reproduced: a window unknown for a whole 15-minute span gave a Model Error
sample (MAE 0.76 °C). Root cause: the replay used the prediction path, which
drops the ventilation term when the opening is unknown, i.e. assumes closed.
Fix: per zone and step, `inputs_complete` checks every input the zone's model
used (window opening and confirmed recovery; wind, direction and other zones'
windows while a window is open; electrical heat and neighbours for the extended
layer, including the neighbour's own completeness). Incomplete spans are
counted, not scored, with the base model's error kept apart; unreplayable
spans are counted as skipped. Learning also stopped substituting a zero facing
term or zero cross-ventilation input for unknown values.
2. **Window unknown at startup treated as closed** (13cb972). Reproduced
(0 % assumed opening). Root cause: last known states lived only in memory, and
"never observed" fell through to the closed case; a first reading of an open
window was also logged as a confirmed opening. Fix: confirmation only for
witnessed transitions; last known states, open events and recovery saved (also
on every window state change) and restored as unknown; per-window forecast
assumptions (observed / last known open, bounded by the open-window time after
it was last seen / last known closed / unobserved), exposed in Status and on
zone forecasts; durations as ranges when an end was not seen. Chosen over
open/closed scenario pairs to avoid a second plan simulation per poll.
3. **Lifecycle and storage races** (28ff953). Confirmed by reading HA's unload
order: the coordinator's own shutdown runs only after platforms unload, and the
save checked for shutdown before waiting for the storage lock (check-then-act).
Found during testing: unloading cancels a running scheduled update, which
released the lock while the file write continued in a worker thread, so a
deletion could run first and be undone. Fix: closing mark at the start of
unload (restored if unload fails); checks after acquiring the lock, including a
removed-model mark set under the lock by removal; the write runs as a shielded,
HA-tracked task that holds the lock until it finishes; failed setup shuts the
coordinator down; queued relay callbacks are ignored once closing. Mutation
checks: removing the locked check or the shield makes the deletion tests fail;
removing the early closing mark fails the ordering test (the HA-level unload
test cannot distinguish it, as HA's own shutdown also lands before release).
4. **Forecast Error accepted materially different relay behaviour** (0b82a8e).
Reproduced: 12 of 15 planned minutes was scored. Root cause: a per-block
on-share tolerance of 20 %, blind to timing within a block. Fix: deviation =
time in another state than planned, block by block from the relay's switch
times, with unobserved time as possible deviation; matched ≤ 2 min (scored),
approximate ≤ 5 min (separate), counterfactual > 5 min observed, otherwise
incomplete with a reason; forecasts made with assumed inputs are incomplete.
Ledger statistics are versioned; 0.4.1 figures are dropped.
Added: *Reset learned model* (49565d6): unload, strip learned keys from the
stored model under the lock, set up again; configuration, heat-source baselines
and window history kept.
Tests added (by area): Model Error incomplete/skipped (7), plan matching (10),
window startup/restart/mixed/strict inputs (9), coordinator window reload and
uncertain forecast (2), lifecycle offline (5) and in HA (4), reset flow (1).
Remaining limitations: unknown-but-last-closed and never-seen windows still
produce a closed-window forecast (flagged, never scored); the 2 and 5 minute
limits rest on typical learned heat coefficients, not a per-forecast calculation;
spans lost when the replay window restarts are not counted; saved window state can
be up to one save old, which only makes the restored unknown period longer.
**Verification.** `sh scripts/check.sh` passes in both environments (pip check,
ruff, 192 offline tests, 33 HA Core tests, compilation, package/translation/
changelog checks, diff whitespace); the offline tests also pass on Python 3.9.6.
+72
View File
@@ -2,6 +2,78 @@
Engineering findings and test evidence behind each change are in [AUDIT.md](AUDIT.md).
## Unreleased — final stability fixes after 0.4.1
Four correctness fixes from an independent review of 0.4.1, and a **Reset learned
model** action you asked for. The version stays 0.4.1 until released. Still
**observation-only**: no service calls; the boiler and thermostat are untouched.
Tested in the Home Assistant Core harness with HA 2025.12.0 and 2026.2.3; not on
HAOS.
### Upgrade notes
- No reconfiguration needed. Zones, sensor and comfort weights, weather sources,
heat sources, windows, learned coefficients and fingerprints carry over; entity
IDs are unchanged.
- **Configure now opens a menu**: *Settings* (the previous form) or *Reset learned
model*.
- **Forecast Error statistics restart.** 0.4.1 accepted plans the relay followed
within 20 % per block; those figures are not comparable with the new rule and
are dropped on the first start (they covered at most a day).
- Model Error statistics carry over. Those from 0.4.0/0.4.1 may include a few
spans with unknown window states; they age out within 7 days.
### Added
- **Reset learned model** (Configure, with a confirmation): forgets every zone's
learned coefficients, models kept for removed zones, and the Model Error and
Forecast Error statistics; keeps the configuration, heat-source baselines and the
window event log, and logs the reset in `model_changes`.
### Fixed
- **Model Error no longer scores spans with unknown inputs.** A window unknown
during a span was replayed as closed and scored. Spans where a window state, an
open window's weather input, or the extended layer's electrical heat or
neighbour was unknown are now counted as `incomplete` (with the base model's
error kept apart as `incomplete_base_model_mae`); spans that cannot be replayed
are counted as `skipped`.
- **Window sensors unavailable at startup are unknown, not closed.** Last known
states, open events and recovery are saved and restored as unknown until the
sensor reports; openings and closings count as confirmed only when seen; an
unknown window last seen open is assumed open only until the open-window time
after it was last seen; other unknown windows are flagged as uncertain in
forecasts (`uncertain_windows`) and in Status (`forecast_assumption`). Event
durations are given as a range when an end was not seen.
- **Unload, reload and removal cannot publish or save stale work.** The
coordinator is marked closing before platforms unload; the model save checks
this, and a removed-model mark, after taking the storage lock and holds the lock
until the file write finishes, so a deletion can no longer be undone by a save
that was waiting or cancelled mid-write. A failed setup shuts the coordinator
down; relay callbacks queued before unload are ignored.
- **Forecast Error compares the relay's actual timing with the plan.** The
deviation is the time the relay spent in another state than planned (unobserved
time counts as possible deviation). Up to 2 minutes: matched and scored; up to 5:
approximate, scored separately; more: counterfactual; otherwise incomplete, with
a reason (relay history, assumed inputs, no poll at the due time, missing
reading). Forecasts made with an unknown window or electrical heat are never
scored.
### Changed
- Learning no longer substitutes values for an open window: samples where the
wind direction (for the directional term) or another zone's windows (for
cross-ventilation) were unknown are not learned. With every window closed, the
opening terms need no weather input, so recovery can be learned without wind.
- The gust option's hint refers to Model Error.
### Persisted data
- `schema_version` stays 2. `ventilation` gains `state` (last known window
states, open events, recovery); `evaluation` gains `excluded` (incomplete and
skipped spans); `operational_evaluation` gains `version` 2 and `approximate`.
Missing optional keys load as empty.
## 0.4.1 — 2026-10-27
Stability and correctness: fixes four problems found in 0.4.0 and separates model
+67 -31
View File
@@ -66,12 +66,24 @@ modeled; the persistent load is already part of the zone's learned behavior.
**Windows** need a contact sensor (binary sensor), the zone, whether the sensor
reports *on* when closed (inverted), the direction the outside of the window faces
(degrees clockwise from north; default 345°, editable) and an opening factor
(1.0 normal, lower for tilted, higher for large). Unknown or unavailable states are
treated as unknown, never as closed: a window last seen open counts as possibly
open (the forecast assumes it open, the zone's base model does not learn), and an
opening that ended while the sensor was unknown is logged as unconfirmed, with its
longest possible duration. Recovery after closing is learned only from a confirmed
close.
(1.0 normal, lower for tilted, higher for large).
A window is **open**, **closed** or **unknown** (unknown, unavailable or no state);
unknown never means closed. While any window of a zone is unknown, the zone does
not learn. What forecasts assume for an unknown window:
- last seen open: open, but only until the configured open-window time after it was
last seen (never renewed), then closed and recovering;
- last seen closed, or never seen since setup: no ventilation, but the forecast is
flagged (`uncertain_windows` on the zone's forecast entity, `forecast_assumption`
in Status) and is never used to measure forecast accuracy.
An opening or closing is *confirmed* only when it was seen happen: the previous
reading was the other state, with no unknown period or restart in between. Events
report a duration only when both ends are confirmed, otherwise a minimum and, where
possible, a maximum; recovery is learned only after a confirmed close. The last
known states, an open event and a recovery in progress are saved, so a restart or
reload continues them, but they come back as unknown until the sensor reports.
Model options in **Configure**: how fast extra electrical heat is assumed to fade in
forecasts (half-life, default 30 minutes), how long an open window is assumed to
@@ -186,9 +198,10 @@ the plan is discarded and the next poll computes a fresh one. Useful attributes:
included sources), baseline, variable power above it, thermal response, whether it
has settled.
- `windows` — per window: state (`open`, `closed` or `unknown`), the last known
state and for how long the sensor has been unknown, minutes open, wind direction
relative to the window's facing and the resulting exposure (1 = wind straight at
it).
state and for how long the sensor has been unknown, what forecasts assume
(`forecast_assumption`: `observed`, `last_known_open`, `last_known_closed` or
`unobserved`; `forecast_open_minutes`), minutes open, wind direction relative to
the window's facing and the resulting exposure (1 = wind straight at it).
`ventilation` — per zone: current opening, recovery, temperature before opening,
its contribution in °C/h and the ventilation coefficients. `ventilation_events` —
the last five openings with temperatures before, lowest, at closing and 30
@@ -288,35 +301,57 @@ model is scored on the same spans. Each sensor's state is the mean absolute erro
- `base_model_mae`, `improvement_vs_base` — the base model's error on the same
spans, and the share of it the extra layers remove (0 = no gain, negative = they
make it worse: switch that option off).
- `incomplete` — spans left out because an input the zone's model needed was not
observed at some step: a window state (or the start of a recovery), the wind,
wind direction or another zone's windows while a window was open, or, for the
extended layer, electrical heat or a neighbour (whose own span must be complete
too). The base model needs none of these, so its error on those spans is still
reported as `incomplete_base_model_mae`.
- `skipped` — spans that could not be replayed: the radiator-lag estimate was
uncertain along the way, or the zone had no reading at the end.
- `zones` — the same numbers per zone (not written to the Recorder).
Spans are scored only where the inputs were known: polls 4.5–5.5 minutes apart, a
usable reading at the start and the end, and a well-known radiator-lag estimate
throughout. A missing relay, outdoor temperature or all zones restarts the replay
window, so scores resume 15–120 minutes later. Statistics are kept with the model
and survive restarts; recalibrating a zone or deleting the model resets them.
Spans need polls 4.5–5.5 minutes apart; a missing relay, outdoor temperature or
all zones restarts the replay window, so scores resume 15–120 minutes later (these
gaps are not counted). Statistics are kept with the model and survive restarts;
recalibrating a zone or resetting the learned model clears them.
### Forecast Error
Each time a plan is published, its forecast is recorded as it was: start
temperatures, the trajectory, the plan, the model revision and the assumptions
(outdoor temperature and wind held, electrical heat fading, open windows closing).
When the forecast's horizon comes, it is scored against the reading, but only if
the relay actually followed the plan: in every 15-minute block up to that horizon
the relay must have been in the planned state for at least 12 of the 15 minutes.
Otherwise the forecast is not an error measurement, and is counted instead:
When the forecast's horizon comes, the relay's actual behaviour is compared with
the plan, block by block, from the relay's own switch times: the **deviation** is
how long the relay was in another state than planned, so the same on-time at other
times than planned counts as deviation. Relay time that was not observed counts as
possible deviation. Each forecast is then:
- `counterfactual` — the relay did something else: Generic Thermostat, which
controls it, disagreed with the plan. Expected in observation mode whenever the
two disagree.
- `unverifiable` — the relay history is incomplete for that span.
- `missing` — no poll near the due time, or no reading for a zone.
- `matched` — at most 2 minutes of deviation. Only these make the Forecast Error
figures.
- `approximate` — at most 5 minutes (of which at most 2 unobserved), for example
the thermostat switching a few minutes earlier than planned. Scored separately
in `approximate_stats`.
- `counterfactual` — more than 5 minutes: Generic Thermostat, which controls the
relay, did something else. Not an error of the forecast. Expected in observation
mode whenever the two disagree.
- `incomplete` — not comparable, with `incomplete_reasons`: `relay_history` (too
much relay time unknown), `uncertain_inputs` (the forecast had to assume an
unknown window or electrical heat), `no_poll` (no poll near the due time) or
`reading` (a zone had no reading; counted per zone).
The state is the 60-minute MAE once its quality is at least `indicative`; the
attribute `horizons` (not written to the Recorder) gives MAE, RMSE, bias, the
no-change error, `improvement_vs_persistence`, sample counts, quality, the counts
above and the per-zone figures for each horizon. Statistics cover 7 days and
survive restarts; forecasts not yet due when HA restarts are dropped.
Why these limits: d seconds of boiler heat that the plan did not assume move a room
by roughly *heat* × d / 300, where *heat* is the learned boiler coefficient
(typically 0.05–0.12 °C per 5 minutes with warm radiators). 2 minutes stay below
half a typical 0.1 °C sensor step and absorb timestamp jitter; 5 minutes are about
one step.
The state is the 60-minute MAE of matched forecasts once its quality is at least
`indicative`; the attribute `horizons` (not written to the Recorder) gives MAE,
RMSE, bias, the no-change error, `improvement_vs_persistence`, sample counts,
quality, the counts above, `approximate_stats` and the per-zone figures for each
horizon. Statistics cover 7 days and survive restarts; forecasts not yet due when
HA restarts are dropped.
What neither covers: outdoor temperature changes during the horizon (held
constant, as in the MPC), and periods when the model was still learning (included,
@@ -359,9 +394,10 @@ The brand icons in `custom_components/heating_mpc/brand/` are drawn by
`python scripts/make_brand_icons.py` (uses Pillow from the test requirements);
the package check verifies their sizes and transparency.
HA tests cover setup, config/options forms, migration, reload (a single relay
listener and polling coordinator after repeated reloads) and entity cleanup,
scheduled sampling and poll cancellation, the relay listener, stored model
HA tests cover setup, config/options forms (including the reset), migration,
reload (a single relay listener and polling coordinator after repeated reloads),
unload, reload and removal while an optimization or a save is still running,
entity cleanup, scheduled sampling and poll cancellation, the relay listener, stored model
restoration, retention and deletion, missing inputs, manual off/presets, optimizer
failure and Recorder exclusions. A fixture rejects all HA service calls throughout
each test, and a static test fails if actuator service calls are added to the
+19 -4
View File
@@ -95,6 +95,18 @@ changed during optimization or by an overtaken run; and model accuracy reported
as if it were forecast accuracy. Reload/unload tests now cover repeated reloads
(one relay listener, one polling coordinator). Released as 0.4.1; not yet tested on HAOS.
### Final stability fixes after 0.4.1 — 2026-10-27
An independent review of 0.4.1 reported four problems; three were reproduced and
the fourth (lifecycle races) confirmed by reading Home Assistant's unload order.
Fixed with regression tests (see AUDIT.md): Model Error spans with unknown inputs
are now `incomplete`, not scored; window sensors unavailable at startup are
unknown, with last known states saved and restored as unknown; unload, reload
and removal can no longer publish stale results or recreate a deleted model;
Forecast Error scores only forecasts whose plan the relay matched within 2
minutes. Also added: **Reset learned model** in Configure. Not released or tested
on HAOS yet.
## Milestone 1 — Explainable forecasts and dashboards (target v0.3.0)
**Objective:** Users can verify what the model predicts and whether it is actually accurate.
@@ -103,7 +115,7 @@ as if it were forecast accuracy. Reload/unload tests now cover repeated reloads
- [x] Expose predicted +15/+30/+60/+120 minute temperatures and full proposed ON/OFF schedule. *(0.3.0: `forecast_15min` and `plan_15min` on each room forecast entity.)*
- [ ] Provide an HA dashboard example using native cards or a clearly documented optional chart card (do not bundle a heavyweight frontend unnecessarily).
- [ ] Make clear that the **MPC plan forecast is counterfactual** while Generic Thermostat is controlling the relay; add a separate actual-policy forecast/evaluation pathway. *(0.3.0: plan forecasts labelled counterfactual; evaluation uses the actual relay history. Stabilization: published forecasts are scored only when the relay followed their plan and otherwise counted as `counterfactual`. An actual-policy forecast, simulating Generic Thermostat, is still open.)*
- [ ] Store a bounded history of predictions with snapshots of inputs, proposed plan and eventually observed outcome. *(Stabilization: each published forecast is kept with its start readings, plan, model revision and assumptions until scored, in memory only; the scored statistics persist. No browsable history yet.)*
- [ ] Store a bounded history of predictions with snapshots of inputs, proposed plan and eventually observed outcome. *(Stabilization: each published forecast is kept with its start readings, plan, model revision and assumptions, including any it had to assume, until scored, in memory only; the scored statistics persist, split into matched, approximate, counterfactual and incomplete. No browsable history yet.)*
- [x] Implement out-of-sample/walk-forward MAE and bias at 15, 30, 60 and 120 minutes; include sample counts and confidence/quality labels. *(0.3.0: replay with parameters frozen at the forecast start and the actual relay history; `skill` against persistence. Stabilization: these are now the Model Error sensors, with RMSE, effective sample counts and quality labels; a separate Forecast Error sensor scores published forecasts. No confidence intervals yet.)*
- [ ] Compare against simple benchmarks such as current-temperature persistence and a fixed thermal baseline. *(0.3.0: persistence done; fixed baseline open.)*
- [ ] Add inspectable explanations: expected heat loss, expected heating response, target, room weights and reason for the next proposed action.
@@ -187,9 +199,12 @@ Items 1–3 of the original backlog are done (see AUDIT.md and CHANGELOG.md 0.2.
3. Configure zones, the UPS heat source and the windows on HAOS; after a week of
heating review Model Error `improvement_vs_base` and `quality`, per-zone bias,
coefficient `confidence` (base layer included), `model_changes` after any
reconfiguration, and the Forecast Error counts (how many forecasts were scored
vs counterfactual). Switch off optional terms that do not help. Also try an
unavailable window sensor and a zone edit during HAOS acceptance.
reconfiguration, and the Forecast Error counts (matched, approximate,
counterfactual, incomplete and why). Switch off optional terms that do not
help. During HAOS acceptance also try: a window sensor unavailable across an HA
restart, a zone edit, a reload while the integration is running, *Reset learned
model* and removing a disposable entry (its `.storage/heating_mpc.<entry_id>`
file must be gone afterwards).
4. Milestone 1 (rest): a dashboard example; forecast timestamps/provenance;
explanations for the proposed action; an actual-policy forecast that simulates
Generic Thermostat.
+47 -12
View File
@@ -98,11 +98,19 @@ known. While a window is open, possibly open (its sensor is unknown and it was l
seen open) or the zone is recovering, the base and extended layers do **not**
learn, so window events cannot inflate `loss` or shift `bias`.
An unknown window state is never read as closed. A window last seen open stays
assumed open in forecasts; one last seen closed contributes no ventilation term.
If a window closes while its sensor was unknown, the event's duration is reported
as a range and the recovery term is not learned from it, because the closing time
is not known.
An unknown window state is never read as closed, and an opening or closing counts
as confirmed only when it was seen: the previous reading was the other state with
no unknown period (or restart) in between. If a window closes while its sensor was
unknown, the event's duration is reported as a range and the recovery term is not
learned from it, because the closing time is not known. Last known states, open
events and recovery are saved and restored across restarts, as unknown until the
sensor reports again.
An input that a term needs is never filled in for learning or for Model Error:
while a window is open, the wind, its direction (for `vent_wind_dir`) and the other
zones' windows (for `vent_cross`) must be known. With every window closed, the
opening terms are 0 and need no weather. Forecasts, which must produce a number,
assume calm air for a missing wind input.
`vent_wind_dir` is bounded by `vent_wind` (|vent_wind_dir| ≤ vent_wind), so the
combined wind term never goes negative: a leeward window still ventilates.
@@ -202,7 +210,10 @@ minutes):
its response time. A gaming PC drawing 600 W now is not assumed to run on for two
hours.
* **Windows:** an open window is assumed to stay open for a configurable time
(default 15 minutes), then close; the zone then recovers.
(default 15 minutes), then close; the zone then recovers. A window whose sensor
is unknown but was last seen open is assumed open only until that time after it
was last seen, never renewed. One last seen closed, or never seen since setup, is
assumed closed, and the forecast is flagged as uncertain (`uncertain_windows`).
* **Relay:** the candidate plan being evaluated.
For the plan search the step is compiled into exact per-zone coefficients (every
@@ -228,13 +239,34 @@ scored on exactly the same spans:
* `base_model_mae`, `improvement_vs_base`: against the base layer alone. A feature
that does not improve this should be switched off (for example the gust term).
A span counts only if every input the zone's model used was observed at every
step (see *Ventilation layer*; for the extended layer also electrical heat and
neighbours, whose own replay must be complete). Otherwise it is counted as
`incomplete`, with the base model's error on it kept apart
(`incomplete_base_model_mae`), since the base model needs only the relay and the
outdoor temperature. Spans that cannot be replayed (uncertain radiator lag, no
reading at the end) are counted as `skipped`.
**Forecast accuracy (Forecast Error sensor)** scores forecasts as published: the
start readings, trajectory, plan, model revision and assumptions of the moment
they were made, nothing replaced with hindsight. Because the plan is the MPC's
proposal while Generic Thermostat controls the relay, a forecast is scored at a
horizon only if the relay was in the planned state for at least 12 minutes of
every 15-minute block up to it; otherwise it is counted as `counterfactual` (or
`unverifiable` when the relay history is incomplete) and never as an error.
proposal while Generic Thermostat controls the relay, a forecast is compared only
when the relay followed it. The deviation is the time, up to the horizon, the
relay spent in another state than planned, compared block by block from the
relay's own switch times; unobserved relay time counts as possible deviation.
| Deviation (worst case) | Class | Used for |
|---|---|---|
| ≤ 2 min | matched | Forecast Error figures |
| ≤ 5 min, ≤ 2 min unobserved | approximate | `approximate_stats` only |
| > 5 min observed | counterfactual | counted, never an error |
| otherwise | incomplete | counted, with a reason |
Forecasts made with an assumed input (an unknown window or electrical heat) are
incomplete as well. The limits follow from the model: d seconds of unplanned
boiler heat change a room by about heat × d / 300 with the learned `heat`
coefficient (typically 0.05–0.12 °C per 5 minutes), so 2 minutes stay below half a
0.1 °C sensor step and 5 minutes are about one step.
Both report how much evidence there is. Forecasts start every 5 minutes, so at
horizon h consecutive forecasts share most of their span, and all zones share one
@@ -267,5 +299,8 @@ the contamination the separate ventilation layer prevents.
variation their split is uncertain even when their combined effect is accurate.
* Forecasts hold weather constant and use one scenario for electrical heat and
windows; no forecast uncertainty band is computed yet.
* Window sensors with a permanently unknown state keep their zone out of learning;
disable such a window.
* Window sensors with a permanently unknown state keep their zone out of learning
and its Model Error spans incomplete; disable such a window.
* A window that is unknown and was last seen closed (or never seen) is assumed
closed in forecasts, flagged as uncertain; no open/closed scenario pair is
computed.