Document the final stability fixes and the model reset
README: window states and forecast assumptions, Model Error incomplete and skipped spans, Forecast Error classes and why 2 and 5 minutes, and the lifecycle tests. docs/MODEL.md: strict inputs, unknown-window assumptions, the plan-matching table. CHANGELOG gains an Unreleased section (version unchanged); ROADMAP and AUDIT record the findings, root causes, evidence and remaining limitations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0b82a8e26c
commit
c4035951df
5 files changed
+275
-47
No files matched your search
@@ -397,3 +397,73 @@ Remaining risks:
|
||||
ruff, 159 offline tests, 28 HA Core tests; the offline tests also pass on Python
|
||||
3.9.6. `sh scripts/check.sh` additionally validates packaging, translations and
|
||||
the changelog.
|
||||
|
||||
## Final stability fixes after 0.4.1 — 2026-10-27
|
||||
|
||||
Scope: four issues from an independent review of 0.4.1, plus a user-requested
|
||||
*Reset learned model* action. Repository code and tests only (HA Core 2025.12.0
|
||||
and 2026.2.3 harness); no HAOS, device or deployment testing. Still no actuator
|
||||
code; the static and runtime service-call guards pass. Version unchanged (0.4.1).
|
||||
|
||||
Confirmed issues, root causes and fixes:
|
||||
|
||||
1. **Model Error scored spans with unknown ventilation inputs** (2cfba02).
|
||||
Reproduced: a window unknown for a whole 15-minute span gave a Model Error
|
||||
sample (MAE 0.76 °C). Root cause: the replay used the prediction path, which
|
||||
drops the ventilation term when the opening is unknown, i.e. assumes closed.
|
||||
Fix: per zone and step, `inputs_complete` checks every input the zone's model
|
||||
used (window opening and confirmed recovery; wind, direction and other zones'
|
||||
windows while a window is open; electrical heat and neighbours for the extended
|
||||
layer, including the neighbour's own completeness). Incomplete spans are
|
||||
counted, not scored, with the base model's error kept apart; unreplayable
|
||||
spans are counted as skipped. Learning also stopped substituting a zero facing
|
||||
term or zero cross-ventilation input for unknown values.
|
||||
2. **Window unknown at startup treated as closed** (13cb972). Reproduced
|
||||
(0 % assumed opening). Root cause: last known states lived only in memory, and
|
||||
"never observed" fell through to the closed case; a first reading of an open
|
||||
window was also logged as a confirmed opening. Fix: confirmation only for
|
||||
witnessed transitions; last known states, open events and recovery saved (also
|
||||
on every window state change) and restored as unknown; per-window forecast
|
||||
assumptions (observed / last known open, bounded by the open-window time after
|
||||
it was last seen / last known closed / unobserved), exposed in Status and on
|
||||
zone forecasts; durations as ranges when an end was not seen. Chosen over
|
||||
open/closed scenario pairs to avoid a second plan simulation per poll.
|
||||
3. **Lifecycle and storage races** (28ff953). Confirmed by reading HA's unload
|
||||
order: the coordinator's own shutdown runs only after platforms unload, and the
|
||||
save checked for shutdown before waiting for the storage lock (check-then-act).
|
||||
Found during testing: unloading cancels a running scheduled update, which
|
||||
released the lock while the file write continued in a worker thread, so a
|
||||
deletion could run first and be undone. Fix: closing mark at the start of
|
||||
unload (restored if unload fails); checks after acquiring the lock, including a
|
||||
removed-model mark set under the lock by removal; the write runs as a shielded,
|
||||
HA-tracked task that holds the lock until it finishes; failed setup shuts the
|
||||
coordinator down; queued relay callbacks are ignored once closing. Mutation
|
||||
checks: removing the locked check or the shield makes the deletion tests fail;
|
||||
removing the early closing mark fails the ordering test (the HA-level unload
|
||||
test cannot distinguish it, as HA's own shutdown also lands before release).
|
||||
4. **Forecast Error accepted materially different relay behaviour** (0b82a8e).
|
||||
Reproduced: 12 of 15 planned minutes was scored. Root cause: a per-block
|
||||
on-share tolerance of 20 %, blind to timing within a block. Fix: deviation =
|
||||
time in another state than planned, block by block from the relay's switch
|
||||
times, with unobserved time as possible deviation; matched ≤ 2 min (scored),
|
||||
approximate ≤ 5 min (separate), counterfactual > 5 min observed, otherwise
|
||||
incomplete with a reason; forecasts made with assumed inputs are incomplete.
|
||||
Ledger statistics are versioned; 0.4.1 figures are dropped.
|
||||
|
||||
Added: *Reset learned model* (49565d6): unload, strip learned keys from the
|
||||
stored model under the lock, set up again; configuration, heat-source baselines
|
||||
and window history kept.
|
||||
|
||||
Tests added (by area): Model Error incomplete/skipped (7), plan matching (10),
|
||||
window startup/restart/mixed/strict inputs (9), coordinator window reload and
|
||||
uncertain forecast (2), lifecycle offline (5) and in HA (4), reset flow (1).
|
||||
|
||||
Remaining limitations: unknown-but-last-closed and never-seen windows still
|
||||
produce a closed-window forecast (flagged, never scored); the 2 and 5 minute
|
||||
limits rest on typical learned heat coefficients, not a per-forecast calculation;
|
||||
spans lost when the replay window restarts are not counted; saved window state can
|
||||
be up to one save old, which only makes the restored unknown period longer.
|
||||
|
||||
**Verification.** `sh scripts/check.sh` passes in both environments (pip check,
|
||||
ruff, 192 offline tests, 33 HA Core tests, compilation, package/translation/
|
||||
changelog checks, diff whitespace); the offline tests also pass on Python 3.9.6.
|
||||
Reference in new issue
Block a user