Fix shadow observer lifecycle, learning and prediction correctness

AUDIT.md

@@ -0,0 +1,93 @@
+# v0.2.1 source audit — 2026-10-26
+
+Scope: repository implementation, offline tests and documented HA APIs. No HAOS,
+physical relay, thermostat, deployment or gas-consumption testing was performed.
+The manifest remains 0.2.1; these are unreleased reliability fixes, not a release
+acceptance declaration. Milestone 0 remains open.
+
+## Findings and repairs
+
+| Area | Baseline finding | Repair / evidence |
+|---|---|---|
+| Lifecycle, high | Config flow VERSION=2 had no migration hook; the weather helper's legacy fallback alone cannot migrate HA entries. | Migrate v1 data and options independently through HA's update API, retaining options precedence and existing prioritized selections. Reject newer config versions. |
+| Lifecycle, medium | Removed room forecast entities remained in the registry after options reload. | Remove only obsolete room forecast IDs for this integration and entry; retain all other entities and compatible IDs. |
+| Lifecycle, medium | Raised platform setup left the coordinator in hass.data. | Remove that reference on failure. HA's own partially loaded platform cleanup still requires real runtime testing. |
+| Persistence, high | Malformed roots/counters could crash restore; finite but nonphysical coefficients and invalid covariance could be accepted. | Validate shapes, coefficient bounds, symmetric positive-definite covariance, integer counts and finite error totals; reset malformed rooms atomically. |
+| Learning, high | Global ON/OFF counts increased even when fitting rejected outliers and could qualify a newly added room using other rooms' history. | Persist accepted ON/OFF counts per room; readiness requires every room to qualify. Retain compatible legacy coefficients/counts but collect fresh excitation evidence. |
+| Learning, high | A 4–7 minute interval was treated as exactly 5 minutes, including intervals containing relay transitions. | Accept 270–330 seconds, normalize temperature change to five minutes, compute lag using actual elapsed time; reject gaps/backward time, source changes and relay changes since the previous poll (including on/off round trips). |
+| Inputs, high | Thermostat targets were interpreted as Celsius even in a Fahrenheit HA installation. | Normalize using thermostat temperature_unit when present, otherwise HA's configured temperature unit. |
+| Inputs, medium | Invalid weights or malformed unit metadata could cause crashes/nonfinite arithmetic. | Reject invalid weights with unavailable proposal; reject non-string units as unusable sources. |
+| Prediction, high | Inputs could change while saving or optimizing, yet the old plan could be published as ready. | Compare captured HA State snapshots before publishing; discard results when any watched state changes. |
+| Prediction, medium | A 30/45-minute pure optimizer horizon returned its endpoint as a 60-minute forecast. | Return no 60-minute forecast for horizons below 60. UI already limits horizons to at least 60. Replay tests check the actual +60 and endpoint values. |
+| Prediction, medium | NaN weights/lag and negative/nonfinite optimizer parameters were not rejected consistently. | Add finite/range validation and regression tests. |
+
+The initial 10 tests passed. The expanded suite contains 33 offline tests covering
+model/weather logic plus coordinator and integration entry points using small HA
+boundary doubles. The doubles check our calls and state handling; they do not
+exercise HA scheduling, Store internals, entity registration or actual config forms.
+The test service-call spy raises if invoked. Source inspection found no actuator
+service calls, service registration, relay automations or network model inference.
+
+## Remaining gaps, in priority order
+
+1. **Freshness and alignment:** numeric but stale room/outdoor states are still
+   accepted. Define configurable freshness limits and reporting timestamps in the
+   UI, distinguish unchanged values from missing reports, expose rejection reasons,
+   and test fallback/recovery. Missing units still assume canonical units for
+   compatibility. Optional weather source lists lack uniform maximum-size checks.
+2. **Real HA test harness and HAOS acceptance:** config/options form submission,
+   reload listener/poller teardown, disabled entities, partial platform setup failure,
+   actual storage failures, restart/uninstall and registry cleanup need tests against
+   a declared supported HA version. Add CI, translation/schema and packaging checks.
+   Existing en/ro flow strings are unchanged by this patch.
+3. **Sampling and identification:** polling detects a relay change through
+   last_changed and skips that interval; it does not integrate the relay duty cycle.
+   Restart/gaps reset the lag proxy to the current command, which is only an
+   approximation. Use timestamped relay events and aligned sensor observations
+   before claiming robust radiator identification. Rejecting transition intervals
+   can slow learning. The jitter correction is a local linear approximation.
+4. **Persistence and identity:** Store envelope version stays 1; the additive model
+   payload is schema_version=2. Legacy payloads remain readable. Unsupported payload
+   schemas are ignored; comprehensive future-version downgrade protection remains
+   open. Renamed room entity IDs are treated as replacements (fresh model), not
+   automatically matched to entity-registry identity. Same-ID physical sensor
+   relocation and systematic weather provider offsets remain undetected.
+5. **Forecast validity:** forecasts remain counterfactual MPC plans with constant
+   outdoor temperature. Five-minute residual MAE is not held-out multi-horizon
+   accuracy. No actual-policy forecast, recorded trajectories, confidence estimate,
+   identifiability test or persistence benchmark exists. Prediction updates remain
+   five-minute polls; a later manual change does not instantly refresh the diagnostic.
+6. **Resource use:** optimization is offloaded, but no Pi 5 benchmark exists.
+   Model state is written on each eligible interval (roughly 288 writes/day).
+   Rejected-fit reasons and aggregate failure counters are not yet exposed.
+
+## Compatibility and safe trial
+
+Back up HA before installing. Replace the component directory and restart Core;
+keep Generic Thermostat as the only relay writer. Compatible room coefficients and
+forecast unique IDs remain stable. Older saved models need at least 30 newly
+accepted ON and 30 OFF samples per room before becoming ready again, plus the
+existing 200-total-sample condition. No artificial heating cycles are generated.
+
+Observe status, normalized target, per-room model counts and weather provenance.
+Test manual heat/off and preset changes, a missing sensor, an options reload, room
+removal and a restart. The proposed-heat entity must remain diagnostic only. Use a
+disposable config entry for uninstall testing because deletion removes its models.
+For rollback, restore the component and HA backup together; older code ignores the
+new per-room counters and retains its original defects. No installation was done
+as part of this audit.
+
+## Verification commands
+
+- `python3 -m unittest discover -s tests -v` — 33 tests pass (Python 3.9.6).
+- `PYTHONPYCACHEPREFIX=/tmp/heating-mpc-audit-pycache python3 -m compileall -q custom_components/heating_mpc` — passes.
+- `git diff --check` — passes.
+
+The initial unredirected compile check failed because macOS's Python cache path
+was outside the writable sandbox; the temporary-cache rerun succeeded. No full HA
+suite, type checker, linter, HAOS or physical-device checks were run.
+
+API references used to check migration and entity-registry calls:
+[HA config entries](https://developers.home-assistant.io/docs/config_entries_index/),
+[config migration](https://developers.home-assistant.io/docs/config_entries_config_flow_handler/#config-entry-migration),
+[entity registry](https://contributors.home-assistant.io/core/integrations/entity-registry/).