From af0a564279e34b8ffa39d10dfb8988d661b85342 Mon Sep 17 00:00:00 2001 From: Andrei Buzera Date: Mon, 26 Oct 2026 21:21:43 +0200 Subject: [PATCH] Document second audit pass Record findings, evidence, compatibility notes and remaining gaps in AUDIT.md; add user-facing upgrade notes to README and update test counts; note progress against Milestone 0 in ROADMAP. No version bump. Co-Authored-By: Claude Opus 5.5 --- AUDIT.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 34 +++++++++++++++++++++++++++++++++- ROADMAP.md | 8 ++++++++ 3 files changed, 89 insertions(+), 1 deletion(-) diff --git a/AUDIT.md b/AUDIT.md index d79f59f..271f55b 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -135,3 +135,51 @@ Python 3.13 virtual environment, then runs the same dependency and test checks. GitHub cache services are disabled and no Gitea token is supplied to setup-uv. The runner label and integration runtime are unchanged. Remote execution of the corrected workflow remains to be verified. + + +## Follow-up: second source audit — lag, learner stability, options + +Scope as above: repository code, offline tests and the HA Core 2025.12.0 suite. +No HAOS, physical relay, thermostat or deployment testing was performed. The +manifest remains 0.2.1 and no actuator code was added. + +| Area | Finding | Repair / evidence | +|---|---|---| +| Learning and prediction, high | The radiator-lag proxy was reset to the new relay command at every relay change and after any skipped interval: it read 1.0 straight after ON and 0.0 straight after OFF, so the lag model was effectively disabled around every boiler cycle. On a synthetic lagged plant cycled by a hysteresis thermostat, the original code learned a heat gain 34 % low (0.079 vs 0.12) and a loss coefficient 43 % low. The optimizer also started each post-transition plan from the wrong lag. | Lag is tracked at every poll with a valid relay state, independently of room sensors. One change between polls is integrated at its `last_changed` time. Ambiguous cases (on/off round trip between polls, relay outage, restart, clock jump) restart from time-in-state with an explicit error half-width; learning waits until it is at most 0.1 (about 32 minutes after an HA restart). The same synthetic test now learns 0.1195 / 0.00389 (true 0.12 / 0.004). | +| Learning and persistence, high | RLS covariance windup: with forgetting 0.999 and no heat excitation (summer, long OFF periods) the heat variance grew by 1/0.999 per sample without limit, about 8.7×10¹² after 104 simulated OFF days. The first ON samples afterwards could swing coefficients to their bounds, and accumulated asymmetry (9×10⁻⁹ there) approached the absolute 10⁻⁸ restore tolerance, beyond which a restart would silently reset the model. | Covariance is symmetrized and its diagonal bounded by the prior after every update; row/column scaling preserves positive definiteness. Restore uses a relative round-off tolerance and applies the same bound, so wound-up older models load rather than reset. | +| Configuration, medium | Optional weather source fields used the current selection as a form default. A cleared field omitted from the submission re-inserted the old sources, and an options key absent from the submission fell back to the original setup data, so optional providers could not be removed in the UI. | Suggested values pre-fill the form instead; omitted optional lists are saved as empty and override older entry data. Covered by an HA Core options-flow test. | +| Prediction availability, medium | A finished plan was discarded when any watched State object changed during the calculation, including diagnostic-only weather sensors and the thermostat's own `current_temperature`/`hvac_action`. The proposed-heat entity could go unavailable for five minutes without cause. | Compare only values the prediction uses: room and outdoor-temperature states and units, thermostat mode and target, relay state and last change. | +| Lifecycle, low | The coordinator relied on HA's config-entry ContextVar; HA reports this, though it does not enforce it for custom integrations. | Pass `config_entry` explicitly. | +| Tests, medium | The shadow-only rule was checked only at runtime on exercised paths. | Static AST tripwire over the integration source: no service calls or registration, no actuator method names or service-name strings, and only sensor/binary_sensor platforms. | + +New Status attributes: `heat_lag`, `heat_lag_uncertainty` and `sample_result` +(`first_sample`, `learned`, `partly_rejected`, `interval_out_of_range`, +`relay_changed`, `outside_source_changed`, `lag_uncertain`). They explain why the +sample counters are or are not increasing. + +**Compatibility.** Persisted payload schema stays 2; existing models load and get +the covariance bound on load. Coefficients learned before this fix were fitted +with the wrong lag. With forgetting 0.999 they should re-converge over roughly +three to four days of heating data, but sample counts are kept, so `ready` does +not reflect this. Treat earlier forecasts as uncalibrated. For a clean model, +delete and re-add the integration (this deletes the learned data). + +**Remaining gaps found in this pass**, in addition to those above: + +1. Relay changes are still inferred from `last_changed` at each poll. An odd + number of changes greater than one between polls is treated as one change. + A read-only state-change listener for the relay would remove this. +2. The lag estimate is not persisted, so an options reload restarts it from + time-in-state (exact for a long-steady relay, approximate otherwise). +3. Learners of rooms removed in Options are dropped at the next save; re-adding a + room, or reverting an accidental relay change, starts fresh. +4. Status attributes change every poll (`age_seconds`, MAE), so Recorder stores a + new attribute row every five minutes. Consider `_unrecorded_attributes`. +5. The HA Core suite pins 2025.12.0; current HA releases are untested. + +**Verification.** `sh scripts/check.sh` in a Python 3.13 environment with +`requirements-test.txt`: pip check, 56 offline tests, 16 HA Core tests, +compilation, package/translation validation and `git diff --check` all pass. +`python3 -m unittest discover -s tests` also passes on Python 3.9.6. The new +synthetic-plant and windup tests were also run against the original code to confirm +they fail there. diff --git a/README.md b/README.md index ec1a905..5edf377 100644 --- a/README.md +++ b/README.md @@ -128,6 +128,38 @@ Options to match your sensors, and verify fallback/recovery with an unavailable source. Generic Thermostat remains the sole controller. No HAOS trial was performed. +## Unreleased: radiator-lag tracking and learner stability + +See the second-pass section of [AUDIT.md](AUDIT.md). Still shadow-only; no new +options, translations or entities; the manifest version is unchanged. + +- **Radiator lag now follows the relay.** Previously the lag estimate jumped to + fully hot or fully cold at each relay change, which biased the learned heating + coefficient low and gave wrong starting points to forecasts right after a boiler + cycle. A relay change between polls is now integrated at its timestamp. +- **Learning pauses briefly when the lag is unknown**: after an HA restart, a relay + outage, or two relay changes inside one five-minute poll. The Status attribute + `heat_lag_uncertainty` shows the error bound; learning resumes once it is at most + 0.1, about 32 minutes after a restart. Forecasts continue meanwhile. +- **Long OFF periods no longer destabilize learning.** The learner's covariance is + bounded, so a summer without heating cannot make coefficients jump when heating + resumes, and older saved models with large covariance are loaded rather than reset. +- **Optional weather sources can be removed in Configure.** Clearing wind, cloud or + any additional metric now saves an empty list instead of restoring the old sources. +- The proposed-heat entity is no longer marked unavailable when only a + diagnostic weather sensor or the thermostat's current temperature changed during + the calculation. +- New Status attributes: `heat_lag`, `heat_lag_uncertainty`, `sample_result` (why the + last poll was or was not learned). + +**After upgrading:** saved models load unchanged, but coefficients learned before +this change were fitted with the wrong lag. Expect them to re-converge over a few +days of heating and judge forecasts only on data after the upgrade, or delete and +re-add the integration for a clean model (this deletes learned data). After a +restart, check that `sample_result` moves from `lag_uncertain` to `learned` within +about 35 minutes. Generic Thermostat remains the sole relay controller. No HAOS +trial was performed. + ## Developer checks and Forgejo/Gitea Actions The HA Core test suite targets **Home Assistant 2025.12.0 on Python 3.13**, @@ -142,7 +174,7 @@ python -m pip install -r requirements-test.txt sh scripts/check.sh ``` -The check script runs dependency consistency checks, 42 offline tests, 14 tests +The check script runs dependency consistency checks, 56 offline tests, 16 tests using the real HA Core/fixture package, compilation, lightweight package metadata and translation checks, and diff whitespace checks. A compatible DNS dependency is pinned because pycares 5 breaks the older HA dependency's import. The script diff --git a/ROADMAP.md b/ROADMAP.md index f958b82..8a17be4 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -70,6 +70,14 @@ remote runner execution remains unverified. Next improve event-based sampling an identity/persistence migration tests, then extend the HA version matrix. Only after those reliability checks proceed to Milestone 1's recorded forecasts and walk-forward evaluation. Live control remains unauthorized. +A second audit pass (see AUDIT.md) fixed radiator-lag resets at every relay change, +RLS covariance windup and a restore tolerance that could reset long-running models, +optional weather sources that could not be cleared in Options, and spurious +prediction discards. It added `sample_result`/lag diagnostics and a static +shadow-only tripwire test. Relay transitions between polls are now integrated at +their timestamp; multiple changes within one poll still need event-based relay +history before the transition item below can be checked. + ## Milestone 1 — Explainable forecasts and dashboards (target v0.3.0) **Objective:** Users can verify what the model predicts and whether it is actually accurate.