Document the stabilization pass
README: window unknown states, what configuration changes do to the learned model, inputs_changed tolerances, coefficient confidence, model_changes, and Model Error vs Forecast Error. docs/MODEL.md: configuration-change table, confidence semantics, both validation modes. CHANGELOG gains an Unreleased section (version unchanged); AUDIT and ROADMAP record findings, evidence and remaining risks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8eef04c385
commit
35370958fc
5 files changed
+340
-58
No files matched your search
@@ -320,3 +320,80 @@ weather forecasts. Learners of removed zones keep only the base model.
|
||||
**Verification.** `sh scripts/check.sh` passes in both environments (lint, 125
|
||||
offline tests, 26 HA Core tests, compilation, package/translation/changelog checks).
|
||||
The offline tests also pass on Python 3.9.6.
|
||||
|
||||
## Stabilization after 0.4.0 — 2026-10-27
|
||||
|
||||
Scope: correctness of 0.4.0, no new features, version unchanged. Repository code
|
||||
and tests only (HA Core 2025.12.0 and 2026.2.3 harness); no HAOS, device or
|
||||
deployment testing. Still no actuator code; the static and runtime service-call
|
||||
guards pass.
|
||||
|
||||
Confirmed issues and fixes:
|
||||
|
||||
1. **Unknown window states read as closed** (2bddc74). Reproduced: an open window
|
||||
whose sensor became unavailable ended the ventilation event, started recovery
|
||||
at full weight and let the forecast assume a closed window. Windows are now
|
||||
tri-state with a remembered last known state: last seen open → possibly open
|
||||
(base and extended layers do not learn, the forecast assumes open); a close
|
||||
while unknown is logged as unconfirmed with its longest possible duration;
|
||||
recovery is learned only after a confirmed close. Seven transition tests
|
||||
(open/closed/unknown in every order, restart while unknown, polarity).
|
||||
2. **Learned coefficients reused after incompatible changes** (099c63c, 8eef04c).
|
||||
By construction: changing a zone's sensors or weight ratio, a heat source, a
|
||||
window or the recovery time kept every coefficient. New
|
||||
`compatibility.py` fingerprints each component's configuration; on restore
|
||||
only changed components are reset (a measurement change recalibrates the zone:
|
||||
coefficients kept as a starting point, uncertainty, counts and accuracy
|
||||
statistics reset). Comfort weight, target offset and names never invalidate.
|
||||
Heat-source baseline samples are dropped when the effective power's definition
|
||||
changes. 0.4.0 data has no fingerprints, is assumed compatible, and is
|
||||
fingerprinted at the first start. Resets are logged in `model_changes`.
|
||||
3. **Plans published from changed inputs** (d73e32a). The pre/post-optimization
|
||||
check covered only temperatures, relay and thermostat, and an overtaken update
|
||||
could still publish. The input snapshot now covers zone and outdoor
|
||||
temperatures, electrical power, wind, windows, relay and thermostat, with
|
||||
tolerances so sensor noise does not discard every plan; a run counter keeps an
|
||||
older run from overwriting a newer result; shutdown is checked after saving
|
||||
and after the worker; a per-entry storage lock serializes save, restore and
|
||||
delete (also fixing a delete-during-save race that could recreate the file).
|
||||
4. **Model accuracy reported as forecast accuracy** (26c415f). The Forecast Error
|
||||
sensors replayed the model with the actual future relay, electrical heat and
|
||||
windows. They are now **Model Error** (unique IDs unchanged) with RMSE,
|
||||
effective samples and a quality label, and their state stays unknown below 5
|
||||
effective samples. A new **Forecast Error** sensor scores published forecasts
|
||||
with the plan and assumptions of the moment, only when the relay followed the
|
||||
plan (≥ 12 of every 15 minutes in the planned state); otherwise the forecast is
|
||||
counted as counterfactual, unverifiable or missing, never as error.
|
||||
|
||||
Also reviewed:
|
||||
|
||||
- **Confidence** (a44163a). Extended and ventilation coefficients already used
|
||||
posterior variance. Base coefficients now report it too; a test shows two weeks
|
||||
of non-exciting data (relay off, steady difference) keep `heat` at 0 and `loss`
|
||||
and `bias` below 0.8, while four varied days bring all three above 0.8.
|
||||
Readiness still needs sample counts; it is a gate, not a confidence.
|
||||
- **Lifecycle** (4911b32). Three reloads leave one relay listener and one polling
|
||||
coordinator; after unload nothing is recorded or published. The test was
|
||||
checked to fail with the unsubscribe removed.
|
||||
- **Sensor weights UI** (9bb64e7, user report). Per-sensor weights existed but
|
||||
were an unlabelled second step; they are now numbered fields with the sensors
|
||||
listed, in English and Romanian.
|
||||
|
||||
Remaining risks:
|
||||
|
||||
- In observation mode the relay follows the MPC plan mainly when Generic
|
||||
Thermostat happens to agree, so Forecast Error may stay `insufficient` for a
|
||||
long time, and the scored subset is biased towards those periods.
|
||||
- `effective_samples` corrects for overlapping horizons and shared zones only; it
|
||||
is not an autocorrelation estimate. Quality thresholds (5, 30) are judgement.
|
||||
- Configuration changes made before this update cannot be detected: 0.4.0 models
|
||||
are fingerprinted as they are. A sensor replaced under the same entity ID, or a
|
||||
weather provider swap with a systematic offset, does not change any fingerprint.
|
||||
- Input-change tolerances are fixed constants, not options.
|
||||
- Published forecasts awaiting their horizon are lost on restart.
|
||||
- None of this has run on HAOS or with real devices.
|
||||
|
||||
**Verification.** In both environments (HA 2025.12.0 and 2026.2.3, Python 3.13):
|
||||
ruff, 159 offline tests, 28 HA Core tests; the offline tests also pass on Python
|
||||
3.9.6. `sh scripts/check.sh` additionally validates packaging, translations and
|
||||
the changelog.
|
||||
Reference in new issue
Block a user