From e59c08f586aeb15f4d9861cdf32d119535a3820d Mon Sep 17 00:00:00 2001 From: Andrei Buzera Date: Mon, 26 Oct 2026 22:39:25 +0200 Subject: [PATCH] Release 0.3.0 First part of Milestone 1: Forecast Error sensors scoring the thermal model against reality, forecast trajectories, Learning Progress and Last Sample sensors, rejected-sample and poll-outcome counters, plus lint and rollback-contract tests. README explains forecast accuracy, the 30-minute learning wait after a restart and where Developer Tools now lives. ROADMAP marks the items now backed by tests; AUDIT records the evaluation design and evidence. Still observation-only. Co-Authored-By: Claude Opus 5.5 --- AUDIT.md | 44 +++++++++++++++++++++ CHANGELOG.md | 44 +++++++++++++++++++++ README.md | 23 ++++++++++- ROADMAP.md | 21 +++++----- custom_components/heating_mpc/manifest.json | 2 +- 5 files changed, 122 insertions(+), 12 deletions(-) diff --git a/AUDIT.md b/AUDIT.md index e7ddc68..08075b5 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -225,3 +225,47 @@ frozen at 7) and the bedroom resumed one poll after reporting again. **Verification.** `sh scripts/check.sh` passes for HA 2025.12.0 and 2026.2.3 (69 offline tests, 18 HA Core tests). The offline tests also pass on Python 3.9.6. + + +## 0.3.0 — forecast evaluation + +Scope: repository code and tests (HA 2025.12.0 and 2026.2.3 harness); no HAOS. + +Design choices and why: + +- **Score the thermal model, not the MPC plan.** Generic Thermostat controls the + relay, so the plan is never executed and cannot be checked. Each forecast is + replayed with the radiator-lag values that the relay's actual history produced, + which tests the model under known heat input. +- **Out of sample.** Each replay uses the parameters stored at the forecast start; + later learning never influences the forecast it is scored on. +- **Only known inputs.** Spans with irregular polls, an uncertain lag estimate, or a + missing relay/outdoor/room reading are not scored, and a missing shared input + restarts the window. +- **Baseline.** Persistence (no change) is scored on exactly the same spans; `skill` + reports the share of its error the model removes. +- **Known limits.** The outdoor temperature is held at its start value (as the MPC + does); consecutive forecasts overlap, so samples are not independent; the 7-day + window includes learning-phase forecasts. + +Evidence: on the synthetic lagged plant (5 simulated days, thermostat cycling, +outdoor ±5 °C daily swing) the 60-minute error is 0.03 °C against 0.39 °C for +persistence (skill 0.92); the 120-minute error grows to 0.09 °C from the +constant-outdoor assumption. A model matching the plant exactly scores zero error; +a model without heat gain shows the expected negative bias. These are checks of +the evaluation itself, not of real-world accuracy. + +Also in this release: per-room rejected-sample counters and per-outcome poll +counters (Milestone 0 "failed fitting counters"), a rollback contract test against +the 0.2.1 reader, ruff bug-finding lint, and removal of tracked `.DS_Store` files. + +User feedback during testing: after a reset the Status entity only said +`learning`, and with Developer Tools moved under Settings in recent HA versions the +attributes were hard to find, so learning activity was invisible for the first ~30 +minutes. Two sensors now show it directly: Learning Progress (%) and Last Sample +(an enum with translated states, including *Waiting for radiators to settle*). A +poll with missing shared inputs now reports `waiting_for_inputs` instead of nothing. + +**Verification.** `sh scripts/check.sh` passes in both environments: lint, 80 +offline tests, 20 HA Core tests, compilation, package checks. The offline tests +also pass on Python 3.9.6. diff --git a/CHANGELOG.md b/CHANGELOG.md index a884cb3..0f90548 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,50 @@ Engineering findings and test evidence behind each change are in [AUDIT.md](AUDIT.md). +## 0.3.0 — 2026-10-26 + +First part of Milestone 1: the integration now measures how accurate its thermal +model is. Still **observation-only**; no new options. Upgrading from an earlier +version: also read the notes of each version in between. + +### Added + +- Four sensors, **Heating MPC Forecast Error 15/30/60/120 min**: the thermal + model's mean absolute error over the last 7 days, scored out of sample against + what actually happened, using the relay's real ON/OFF history. Attributes show + the bias, the error of a no-change forecast (`persistence_mae`), `skill` and + per-room details. See README, "Forecast accuracy". +- **Heating MPC Learning Progress** (% towards forecasts being used) and **Heating + MPC Last Sample** (what the last poll did for learning, with English and Romanian + state names), so learning activity is visible without opening attributes. +- Room forecast entities: `forecast_15min` (temperature every 15 minutes over the + horizon) and `plan_15min` (the proposed ON/OFF plan), labelled counterfactual. +- Status: `rejected_samples` per room in `models`, `sample_results_since_start` + counting how each poll was handled, and `sample_result: waiting_for_inputs` when + the relay, thermostat, outdoor temperature or every room is missing. + +### Changed + +- The model is saved after each poll's forecasts are scored, with the forecast + statistics (about 8 KB for five rooms, at most 18 KB). + +### Upgrade notes + +- Six new entities appear (four forecast-error sensors, learning progress, last + sample). The error sensors stay unknown until forecasts can be scored: 15 minutes after + start for the 15-minute sensor, 2 hours for the 120-minute one. Early numbers + include the learning phase and are pessimistic. +- Rollback to 0.2.x is safe: older versions ignore the new saved keys, and the six + new sensors become unavailable (remove them in the entity settings if you roll back + permanently). + +### Development + +- Bug-finding lint (ruff: pyflakes, bugbear) in the checks and CI. +- Tests for restoring a model saved by 0.2.1 and for 0.2.1 being able to read + today's saved model. +- `.DS_Store` files are ignored and rejected by the package check. + ## 0.2.3 — 2026-10-26 A stale or failed room sensor no longer pauses the whole integration. Still diff --git a/README.md b/README.md index eeeadf5..2d6406e 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Heating MPC 0.2.3 — custom integration for Home Assistant OS +# Heating MPC 0.3.0 — custom integration for Home Assistant OS **Experimental, observation-only (shadow mode).** It **never** changes the Generic Thermostat, the physical relay, the boiler's setpoint, `heat/off`, or Comfort/Away/Sleep. No Docker add-on, MQTT or InfluxDB connection is needed. @@ -55,6 +55,12 @@ Entities (entity IDs may vary): ON/OFF plan). These assume the MPC's plan is executed; while Generic Thermostat controls the relay they are **counterfactual**. - `binary_sensor.heating_mpc_proposed_heat` — **hypothetical** heating demand (unavailable until ready). +- **Heating MPC Learning Progress** — how close the least trained room is to being + used for forecasts (100 % = ready); attributes show the sample counts and what is + required. +- **Heating MPC Last Sample** — what the last poll did for learning (for example + *Learned*, *Relay switched*, *Waiting for radiators to settle*); its history shows + when learning happens. - **Heating MPC Forecast Error 15/30/60/120 min** — how far the thermal model's forecasts were from what actually happened (see *Forecast accuracy*). @@ -71,6 +77,21 @@ Entities (entity IDs may vary): ## Diagnostics +Start with **Learning Progress** and **Last Sample**. For details, open the +**Heating MPC Status** entity's attributes. In recent Home Assistant versions +Developer Tools is at the bottom of **Settings** (or press Cmd/Ctrl+K and type +"developer"); its **States** tab lists every attribute. + +**Why learning starts about 30 minutes after a restart.** The model learns how +much the radiators warm each room, so it needs to know how warm the radiators are. +It estimates this from when the relay last switched (20-minute time constant). +After a Home Assistant restart that time is unknown, so learning waits until the +estimate is within 10 % (*Waiting for radiators to settle*), about 32 minutes. +Forecasts continue meanwhile. Reloading the integration usually does not cause this +wait, because the relay's last change is still known. *Ready* then needs 200 +accepted samples (about 17 hours) per room, including 30 with the relay ON and 30 +with it OFF, which can take a few days depending on how often the boiler runs. + The **Heating MPC Status** entity's state is one of `waiting_for_sensors`, `learning`, `paused` (thermostat not in `heat`), `waiting_for_target`, `invalid_weights`, `ready`, `inputs_changed` or `prediction_error`. Proposed heat diff --git a/ROADMAP.md b/ROADMAP.md index f773976..6f6aff4 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -10,7 +10,7 @@ Success means predictable comfort, understandable decisions, safe operation and ## Current baseline — v0.2.1 -> **Current release: 0.2.3** — Milestone 0 reliability work on top of this +> **Current release: 0.3.0** — Milestone 0 reliability work and the first part of Milestone 1 on top of this > baseline; see [CHANGELOG.md](CHANGELOG.md). The lists below describe 0.2.1. **Implemented (experimental):** @@ -49,7 +49,7 @@ Current behavior: **shadow mode only**. `generic_thermostat` remains the one and - [x] Add input freshness thresholds and diagnostics that identify exactly which source is missing/stale/invalid. - [x] Verify that source switching, entity renaming, adding/removing rooms and replacing the relay do not silently reuse incompatible fitted coefficients. *(0.2.2: tests for outdoor-source switching, adding, removing and re-adding rooms, and relay replacement. A renamed sensor is handled as removing one room and adding another; its old model is retained under the old ID.)* - [ ] Version persisted model schema; test corrupted/older state and migration/rollback behavior. -- [ ] Verify calculated sample totals and successful/failed fitting counters; expose reasons why training/prediction is unavailable. +- [x] Verify calculated sample totals and successful/failed fitting counters; expose reasons why training/prediction is unavailable. *(0.3.0: per-room accepted/ON/OFF/rejected counts, per-outcome poll counters, Learning Progress and Last Sample sensors, Status reasons; covered by tests.)* - [ ] Add CI, linting, tests and packaging validation (including HACS metadata/translations). - [x] Update installation/release documentation and remove packaged `__pycache__`/`.pyc` files if present. *(0.2.2: CHANGELOG with upgrade/rollback notes; package check rejects tracked bytecode.)* @@ -89,13 +89,13 @@ recording volatile attributes, and tests against HA 2025.12.0 and 2026.2.3. **Objective:** Users can verify what the model predicts and whether it is actually accurate. -- [ ] Expose trajectories for each room at 5–15-minute resolution for 2–3 hours, with forecast generation timestamp, horizon and input provenance. -- [ ] Expose predicted +15/+30/+60/+120 minute temperatures and full proposed ON/OFF schedule. +- [ ] Expose trajectories for each room at 5–15-minute resolution for 2–3 hours, with forecast generation timestamp, horizon and input provenance. *(0.3.0: 15-minute trajectory over the horizon; generation timestamp and provenance still to add.)* +- [x] Expose predicted +15/+30/+60/+120 minute temperatures and full proposed ON/OFF schedule. *(0.3.0: `forecast_15min` and `plan_15min` on each room forecast entity.)* - [ ] Provide an HA dashboard example using native cards or a clearly documented optional chart card (do not bundle a heavyweight frontend unnecessarily). -- [ ] Make clear that the **MPC plan forecast is counterfactual** while Generic Thermostat is controlling the relay; add a separate actual-policy forecast/evaluation pathway. +- [ ] Make clear that the **MPC plan forecast is counterfactual** while Generic Thermostat is controlling the relay; add a separate actual-policy forecast/evaluation pathway. *(0.3.0: plan forecasts labelled counterfactual; evaluation uses the actual relay history. An actual-policy forecast, simulating Generic Thermostat, is still open.)* - [ ] Store a bounded history of predictions with snapshots of inputs, proposed plan and eventually observed outcome. -- [ ] Implement out-of-sample/walk-forward MAE and bias at 15, 30, 60 and 120 minutes; include sample counts and confidence/quality labels. -- [ ] Compare against simple benchmarks such as current-temperature persistence and a fixed thermal baseline. +- [x] Implement out-of-sample/walk-forward MAE and bias at 15, 30, 60 and 120 minutes; include sample counts and confidence/quality labels. *(0.3.0: Forecast Error sensors; replay with parameters frozen at the forecast start and the actual relay history; `skill` against persistence. No confidence intervals yet.)* +- [ ] Compare against simple benchmarks such as current-temperature persistence and a fixed thermal baseline. *(0.3.0: persistence done; fixed baseline open.)* - [ ] Add inspectable explanations: expected heat loss, expected heating response, target, room weights and reason for the next proposed action. **Candidate quality gate:** Sustain **about 0.3–0.4°C or better 60-minute MAE** on representative held-out data *under comparable known relay input sequences*, without systematic room bias. This is a starting engineering goal to validate, not a guarantee or an arbitrary gate to waive. Compare to baselines. @@ -174,9 +174,10 @@ Items 1–3 of the original backlog are done (see AUDIT.md and CHANGELOG.md 0.2. room out, or optionally keep its last value for up to 24 hours). Before live control, decide a minimum `weight_coverage` and the outdoor-temperature outage policy, which still pauses. -3. Milestone 1: record forecasts with their input snapshot and evaluate them - walk-forward at 15/30/60/120 minutes against persistence, replaying the actual - relay sequence; then expose trajectories for a dashboard. +3. Milestone 1 (rest, 0.3.x): let the Forecast Error sensors collect a few weeks + of heating data and review per-room bias; a dashboard example; forecast + timestamps/provenance; a fixed-parameter baseline; explanations for the proposed + action; an actual-policy forecast that simulates Generic Thermostat. 4. Only then consider richer weather inputs and the live-control **design**; do not implement live control without separate authorization. diff --git a/custom_components/heating_mpc/manifest.json b/custom_components/heating_mpc/manifest.json index 592f529..03bfdc2 100644 --- a/custom_components/heating_mpc/manifest.json +++ b/custom_components/heating_mpc/manifest.json @@ -9,5 +9,5 @@ "iot_class": "local_polling", "requirements": [], "single_config_entry": true, - "version": "0.2.3" + "version": "0.3.0" }