Document the final stability fixes and the model reset
README: window states and forecast assumptions, Model Error incomplete and skipped spans, Forecast Error classes and why 2 and 5 minutes, and the lifecycle tests. docs/MODEL.md: strict inputs, unknown-window assumptions, the plan-matching table. CHANGELOG gains an Unreleased section (version unchanged); ROADMAP and AUDIT record the findings, root causes, evidence and remaining limitations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0b82a8e26c
commit
c4035951df
5 files changed
+275
-47
No files matched your search
@@ -397,3 +397,73 @@ Remaining risks:
|
|||||||
ruff, 159 offline tests, 28 HA Core tests; the offline tests also pass on Python
|
ruff, 159 offline tests, 28 HA Core tests; the offline tests also pass on Python
|
||||||
3.9.6. `sh scripts/check.sh` additionally validates packaging, translations and
|
3.9.6. `sh scripts/check.sh` additionally validates packaging, translations and
|
||||||
the changelog.
|
the changelog.
|
||||||
|
|
||||||
|
## Final stability fixes after 0.4.1 — 2026-10-27
|
||||||
|
|
||||||
|
Scope: four issues from an independent review of 0.4.1, plus a user-requested
|
||||||
|
*Reset learned model* action. Repository code and tests only (HA Core 2025.12.0
|
||||||
|
and 2026.2.3 harness); no HAOS, device or deployment testing. Still no actuator
|
||||||
|
code; the static and runtime service-call guards pass. Version unchanged (0.4.1).
|
||||||
|
|
||||||
|
Confirmed issues, root causes and fixes:
|
||||||
|
|
||||||
|
1. **Model Error scored spans with unknown ventilation inputs** (2cfba02).
|
||||||
|
Reproduced: a window unknown for a whole 15-minute span gave a Model Error
|
||||||
|
sample (MAE 0.76 °C). Root cause: the replay used the prediction path, which
|
||||||
|
drops the ventilation term when the opening is unknown, i.e. assumes closed.
|
||||||
|
Fix: per zone and step, `inputs_complete` checks every input the zone's model
|
||||||
|
used (window opening and confirmed recovery; wind, direction and other zones'
|
||||||
|
windows while a window is open; electrical heat and neighbours for the extended
|
||||||
|
layer, including the neighbour's own completeness). Incomplete spans are
|
||||||
|
counted, not scored, with the base model's error kept apart; unreplayable
|
||||||
|
spans are counted as skipped. Learning also stopped substituting a zero facing
|
||||||
|
term or zero cross-ventilation input for unknown values.
|
||||||
|
2. **Window unknown at startup treated as closed** (13cb972). Reproduced
|
||||||
|
(0 % assumed opening). Root cause: last known states lived only in memory, and
|
||||||
|
"never observed" fell through to the closed case; a first reading of an open
|
||||||
|
window was also logged as a confirmed opening. Fix: confirmation only for
|
||||||
|
witnessed transitions; last known states, open events and recovery saved (also
|
||||||
|
on every window state change) and restored as unknown; per-window forecast
|
||||||
|
assumptions (observed / last known open, bounded by the open-window time after
|
||||||
|
it was last seen / last known closed / unobserved), exposed in Status and on
|
||||||
|
zone forecasts; durations as ranges when an end was not seen. Chosen over
|
||||||
|
open/closed scenario pairs to avoid a second plan simulation per poll.
|
||||||
|
3. **Lifecycle and storage races** (28ff953). Confirmed by reading HA's unload
|
||||||
|
order: the coordinator's own shutdown runs only after platforms unload, and the
|
||||||
|
save checked for shutdown before waiting for the storage lock (check-then-act).
|
||||||
|
Found during testing: unloading cancels a running scheduled update, which
|
||||||
|
released the lock while the file write continued in a worker thread, so a
|
||||||
|
deletion could run first and be undone. Fix: closing mark at the start of
|
||||||
|
unload (restored if unload fails); checks after acquiring the lock, including a
|
||||||
|
removed-model mark set under the lock by removal; the write runs as a shielded,
|
||||||
|
HA-tracked task that holds the lock until it finishes; failed setup shuts the
|
||||||
|
coordinator down; queued relay callbacks are ignored once closing. Mutation
|
||||||
|
checks: removing the locked check or the shield makes the deletion tests fail;
|
||||||
|
removing the early closing mark fails the ordering test (the HA-level unload
|
||||||
|
test cannot distinguish it, as HA's own shutdown also lands before release).
|
||||||
|
4. **Forecast Error accepted materially different relay behaviour** (0b82a8e).
|
||||||
|
Reproduced: 12 of 15 planned minutes was scored. Root cause: a per-block
|
||||||
|
on-share tolerance of 20 %, blind to timing within a block. Fix: deviation =
|
||||||
|
time in another state than planned, block by block from the relay's switch
|
||||||
|
times, with unobserved time as possible deviation; matched ≤ 2 min (scored),
|
||||||
|
approximate ≤ 5 min (separate), counterfactual > 5 min observed, otherwise
|
||||||
|
incomplete with a reason; forecasts made with assumed inputs are incomplete.
|
||||||
|
Ledger statistics are versioned; 0.4.1 figures are dropped.
|
||||||
|
|
||||||
|
Added: *Reset learned model* (49565d6): unload, strip learned keys from the
|
||||||
|
stored model under the lock, set up again; configuration, heat-source baselines
|
||||||
|
and window history kept.
|
||||||
|
|
||||||
|
Tests added (by area): Model Error incomplete/skipped (7), plan matching (10),
|
||||||
|
window startup/restart/mixed/strict inputs (9), coordinator window reload and
|
||||||
|
uncertain forecast (2), lifecycle offline (5) and in HA (4), reset flow (1).
|
||||||
|
|
||||||
|
Remaining limitations: unknown-but-last-closed and never-seen windows still
|
||||||
|
produce a closed-window forecast (flagged, never scored); the 2 and 5 minute
|
||||||
|
limits rest on typical learned heat coefficients, not a per-forecast calculation;
|
||||||
|
spans lost when the replay window restarts are not counted; saved window state can
|
||||||
|
be up to one save old, which only makes the restored unknown period longer.
|
||||||
|
|
||||||
|
**Verification.** `sh scripts/check.sh` passes in both environments (pip check,
|
||||||
|
ruff, 192 offline tests, 33 HA Core tests, compilation, package/translation/
|
||||||
|
changelog checks, diff whitespace); the offline tests also pass on Python 3.9.6.
|
||||||
@@ -2,6 +2,78 @@
|
|||||||
|
|
||||||
Engineering findings and test evidence behind each change are in [AUDIT.md](AUDIT.md).
|
Engineering findings and test evidence behind each change are in [AUDIT.md](AUDIT.md).
|
||||||
|
|
||||||
|
## Unreleased — final stability fixes after 0.4.1
|
||||||
|
|
||||||
|
Four correctness fixes from an independent review of 0.4.1, and a **Reset learned
|
||||||
|
model** action you asked for. The version stays 0.4.1 until released. Still
|
||||||
|
**observation-only**: no service calls; the boiler and thermostat are untouched.
|
||||||
|
Tested in the Home Assistant Core harness with HA 2025.12.0 and 2026.2.3; not on
|
||||||
|
HAOS.
|
||||||
|
|
||||||
|
### Upgrade notes
|
||||||
|
|
||||||
|
- No reconfiguration needed. Zones, sensor and comfort weights, weather sources,
|
||||||
|
heat sources, windows, learned coefficients and fingerprints carry over; entity
|
||||||
|
IDs are unchanged.
|
||||||
|
- **Configure now opens a menu**: *Settings* (the previous form) or *Reset learned
|
||||||
|
model*.
|
||||||
|
- **Forecast Error statistics restart.** 0.4.1 accepted plans the relay followed
|
||||||
|
within 20 % per block; those figures are not comparable with the new rule and
|
||||||
|
are dropped on the first start (they covered at most a day).
|
||||||
|
- Model Error statistics carry over. Those from 0.4.0/0.4.1 may include a few
|
||||||
|
spans with unknown window states; they age out within 7 days.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Reset learned model** (Configure, with a confirmation): forgets every zone's
|
||||||
|
learned coefficients, models kept for removed zones, and the Model Error and
|
||||||
|
Forecast Error statistics; keeps the configuration, heat-source baselines and the
|
||||||
|
window event log, and logs the reset in `model_changes`.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **Model Error no longer scores spans with unknown inputs.** A window unknown
|
||||||
|
during a span was replayed as closed and scored. Spans where a window state, an
|
||||||
|
open window's weather input, or the extended layer's electrical heat or
|
||||||
|
neighbour was unknown are now counted as `incomplete` (with the base model's
|
||||||
|
error kept apart as `incomplete_base_model_mae`); spans that cannot be replayed
|
||||||
|
are counted as `skipped`.
|
||||||
|
- **Window sensors unavailable at startup are unknown, not closed.** Last known
|
||||||
|
states, open events and recovery are saved and restored as unknown until the
|
||||||
|
sensor reports; openings and closings count as confirmed only when seen; an
|
||||||
|
unknown window last seen open is assumed open only until the open-window time
|
||||||
|
after it was last seen; other unknown windows are flagged as uncertain in
|
||||||
|
forecasts (`uncertain_windows`) and in Status (`forecast_assumption`). Event
|
||||||
|
durations are given as a range when an end was not seen.
|
||||||
|
- **Unload, reload and removal cannot publish or save stale work.** The
|
||||||
|
coordinator is marked closing before platforms unload; the model save checks
|
||||||
|
this, and a removed-model mark, after taking the storage lock and holds the lock
|
||||||
|
until the file write finishes, so a deletion can no longer be undone by a save
|
||||||
|
that was waiting or cancelled mid-write. A failed setup shuts the coordinator
|
||||||
|
down; relay callbacks queued before unload are ignored.
|
||||||
|
- **Forecast Error compares the relay's actual timing with the plan.** The
|
||||||
|
deviation is the time the relay spent in another state than planned (unobserved
|
||||||
|
time counts as possible deviation). Up to 2 minutes: matched and scored; up to 5:
|
||||||
|
approximate, scored separately; more: counterfactual; otherwise incomplete, with
|
||||||
|
a reason (relay history, assumed inputs, no poll at the due time, missing
|
||||||
|
reading). Forecasts made with an unknown window or electrical heat are never
|
||||||
|
scored.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Learning no longer substitutes values for an open window: samples where the
|
||||||
|
wind direction (for the directional term) or another zone's windows (for
|
||||||
|
cross-ventilation) were unknown are not learned. With every window closed, the
|
||||||
|
opening terms need no weather input, so recovery can be learned without wind.
|
||||||
|
- The gust option's hint refers to Model Error.
|
||||||
|
|
||||||
|
### Persisted data
|
||||||
|
|
||||||
|
- `schema_version` stays 2. `ventilation` gains `state` (last known window
|
||||||
|
states, open events, recovery); `evaluation` gains `excluded` (incomplete and
|
||||||
|
skipped spans); `operational_evaluation` gains `version` 2 and `approximate`.
|
||||||
|
Missing optional keys load as empty.
|
||||||
|
|
||||||
## 0.4.1 — 2026-10-27
|
## 0.4.1 — 2026-10-27
|
||||||
|
|
||||||
Stability and correctness: fixes four problems found in 0.4.0 and separates model
|
Stability and correctness: fixes four problems found in 0.4.0 and separates model
|
||||||
|
|||||||
@@ -66,12 +66,24 @@ modeled; the persistent load is already part of the zone's learned behavior.
|
|||||||
**Windows** need a contact sensor (binary sensor), the zone, whether the sensor
|
**Windows** need a contact sensor (binary sensor), the zone, whether the sensor
|
||||||
reports *on* when closed (inverted), the direction the outside of the window faces
|
reports *on* when closed (inverted), the direction the outside of the window faces
|
||||||
(degrees clockwise from north; default 345°, editable) and an opening factor
|
(degrees clockwise from north; default 345°, editable) and an opening factor
|
||||||
(1.0 normal, lower for tilted, higher for large). Unknown or unavailable states are
|
(1.0 normal, lower for tilted, higher for large).
|
||||||
treated as unknown, never as closed: a window last seen open counts as possibly
|
|
||||||
open (the forecast assumes it open, the zone's base model does not learn), and an
|
A window is **open**, **closed** or **unknown** (unknown, unavailable or no state);
|
||||||
opening that ended while the sensor was unknown is logged as unconfirmed, with its
|
unknown never means closed. While any window of a zone is unknown, the zone does
|
||||||
longest possible duration. Recovery after closing is learned only from a confirmed
|
not learn. What forecasts assume for an unknown window:
|
||||||
close.
|
|
||||||
|
- last seen open: open, but only until the configured open-window time after it was
|
||||||
|
last seen (never renewed), then closed and recovering;
|
||||||
|
- last seen closed, or never seen since setup: no ventilation, but the forecast is
|
||||||
|
flagged (`uncertain_windows` on the zone's forecast entity, `forecast_assumption`
|
||||||
|
in Status) and is never used to measure forecast accuracy.
|
||||||
|
|
||||||
|
An opening or closing is *confirmed* only when it was seen happen: the previous
|
||||||
|
reading was the other state, with no unknown period or restart in between. Events
|
||||||
|
report a duration only when both ends are confirmed, otherwise a minimum and, where
|
||||||
|
possible, a maximum; recovery is learned only after a confirmed close. The last
|
||||||
|
known states, an open event and a recovery in progress are saved, so a restart or
|
||||||
|
reload continues them, but they come back as unknown until the sensor reports.
|
||||||
|
|
||||||
Model options in **Configure**: how fast extra electrical heat is assumed to fade in
|
Model options in **Configure**: how fast extra electrical heat is assumed to fade in
|
||||||
forecasts (half-life, default 30 minutes), how long an open window is assumed to
|
forecasts (half-life, default 30 minutes), how long an open window is assumed to
|
||||||
@@ -186,9 +198,10 @@ the plan is discarded and the next poll computes a fresh one. Useful attributes:
|
|||||||
included sources), baseline, variable power above it, thermal response, whether it
|
included sources), baseline, variable power above it, thermal response, whether it
|
||||||
has settled.
|
has settled.
|
||||||
- `windows` — per window: state (`open`, `closed` or `unknown`), the last known
|
- `windows` — per window: state (`open`, `closed` or `unknown`), the last known
|
||||||
state and for how long the sensor has been unknown, minutes open, wind direction
|
state and for how long the sensor has been unknown, what forecasts assume
|
||||||
relative to the window's facing and the resulting exposure (1 = wind straight at
|
(`forecast_assumption`: `observed`, `last_known_open`, `last_known_closed` or
|
||||||
it).
|
`unobserved`; `forecast_open_minutes`), minutes open, wind direction relative to
|
||||||
|
the window's facing and the resulting exposure (1 = wind straight at it).
|
||||||
`ventilation` — per zone: current opening, recovery, temperature before opening,
|
`ventilation` — per zone: current opening, recovery, temperature before opening,
|
||||||
its contribution in °C/h and the ventilation coefficients. `ventilation_events` —
|
its contribution in °C/h and the ventilation coefficients. `ventilation_events` —
|
||||||
the last five openings with temperatures before, lowest, at closing and 30
|
the last five openings with temperatures before, lowest, at closing and 30
|
||||||
@@ -288,35 +301,57 @@ model is scored on the same spans. Each sensor's state is the mean absolute erro
|
|||||||
- `base_model_mae`, `improvement_vs_base` — the base model's error on the same
|
- `base_model_mae`, `improvement_vs_base` — the base model's error on the same
|
||||||
spans, and the share of it the extra layers remove (0 = no gain, negative = they
|
spans, and the share of it the extra layers remove (0 = no gain, negative = they
|
||||||
make it worse: switch that option off).
|
make it worse: switch that option off).
|
||||||
|
- `incomplete` — spans left out because an input the zone's model needed was not
|
||||||
|
observed at some step: a window state (or the start of a recovery), the wind,
|
||||||
|
wind direction or another zone's windows while a window was open, or, for the
|
||||||
|
extended layer, electrical heat or a neighbour (whose own span must be complete
|
||||||
|
too). The base model needs none of these, so its error on those spans is still
|
||||||
|
reported as `incomplete_base_model_mae`.
|
||||||
|
- `skipped` — spans that could not be replayed: the radiator-lag estimate was
|
||||||
|
uncertain along the way, or the zone had no reading at the end.
|
||||||
- `zones` — the same numbers per zone (not written to the Recorder).
|
- `zones` — the same numbers per zone (not written to the Recorder).
|
||||||
|
|
||||||
Spans are scored only where the inputs were known: polls 4.5–5.5 minutes apart, a
|
Spans need polls 4.5–5.5 minutes apart; a missing relay, outdoor temperature or
|
||||||
usable reading at the start and the end, and a well-known radiator-lag estimate
|
all zones restarts the replay window, so scores resume 15–120 minutes later (these
|
||||||
throughout. A missing relay, outdoor temperature or all zones restarts the replay
|
gaps are not counted). Statistics are kept with the model and survive restarts;
|
||||||
window, so scores resume 15–120 minutes later. Statistics are kept with the model
|
recalibrating a zone or resetting the learned model clears them.
|
||||||
and survive restarts; recalibrating a zone or deleting the model resets them.
|
|
||||||
|
|
||||||
### Forecast Error
|
### Forecast Error
|
||||||
|
|
||||||
Each time a plan is published, its forecast is recorded as it was: start
|
Each time a plan is published, its forecast is recorded as it was: start
|
||||||
temperatures, the trajectory, the plan, the model revision and the assumptions
|
temperatures, the trajectory, the plan, the model revision and the assumptions
|
||||||
(outdoor temperature and wind held, electrical heat fading, open windows closing).
|
(outdoor temperature and wind held, electrical heat fading, open windows closing).
|
||||||
When the forecast's horizon comes, it is scored against the reading, but only if
|
When the forecast's horizon comes, the relay's actual behaviour is compared with
|
||||||
the relay actually followed the plan: in every 15-minute block up to that horizon
|
the plan, block by block, from the relay's own switch times: the **deviation** is
|
||||||
the relay must have been in the planned state for at least 12 of the 15 minutes.
|
how long the relay was in another state than planned, so the same on-time at other
|
||||||
Otherwise the forecast is not an error measurement, and is counted instead:
|
times than planned counts as deviation. Relay time that was not observed counts as
|
||||||
|
possible deviation. Each forecast is then:
|
||||||
|
|
||||||
- `counterfactual` — the relay did something else: Generic Thermostat, which
|
- `matched` — at most 2 minutes of deviation. Only these make the Forecast Error
|
||||||
controls it, disagreed with the plan. Expected in observation mode whenever the
|
figures.
|
||||||
two disagree.
|
- `approximate` — at most 5 minutes (of which at most 2 unobserved), for example
|
||||||
- `unverifiable` — the relay history is incomplete for that span.
|
the thermostat switching a few minutes earlier than planned. Scored separately
|
||||||
- `missing` — no poll near the due time, or no reading for a zone.
|
in `approximate_stats`.
|
||||||
|
- `counterfactual` — more than 5 minutes: Generic Thermostat, which controls the
|
||||||
|
relay, did something else. Not an error of the forecast. Expected in observation
|
||||||
|
mode whenever the two disagree.
|
||||||
|
- `incomplete` — not comparable, with `incomplete_reasons`: `relay_history` (too
|
||||||
|
much relay time unknown), `uncertain_inputs` (the forecast had to assume an
|
||||||
|
unknown window or electrical heat), `no_poll` (no poll near the due time) or
|
||||||
|
`reading` (a zone had no reading; counted per zone).
|
||||||
|
|
||||||
The state is the 60-minute MAE once its quality is at least `indicative`; the
|
Why these limits: d seconds of boiler heat that the plan did not assume move a room
|
||||||
attribute `horizons` (not written to the Recorder) gives MAE, RMSE, bias, the
|
by roughly *heat* × d / 300, where *heat* is the learned boiler coefficient
|
||||||
no-change error, `improvement_vs_persistence`, sample counts, quality, the counts
|
(typically 0.05–0.12 °C per 5 minutes with warm radiators). 2 minutes stay below
|
||||||
above and the per-zone figures for each horizon. Statistics cover 7 days and
|
half a typical 0.1 °C sensor step and absorb timestamp jitter; 5 minutes are about
|
||||||
survive restarts; forecasts not yet due when HA restarts are dropped.
|
one step.
|
||||||
|
|
||||||
|
The state is the 60-minute MAE of matched forecasts once its quality is at least
|
||||||
|
`indicative`; the attribute `horizons` (not written to the Recorder) gives MAE,
|
||||||
|
RMSE, bias, the no-change error, `improvement_vs_persistence`, sample counts,
|
||||||
|
quality, the counts above, `approximate_stats` and the per-zone figures for each
|
||||||
|
horizon. Statistics cover 7 days and survive restarts; forecasts not yet due when
|
||||||
|
HA restarts are dropped.
|
||||||
|
|
||||||
What neither covers: outdoor temperature changes during the horizon (held
|
What neither covers: outdoor temperature changes during the horizon (held
|
||||||
constant, as in the MPC), and periods when the model was still learning (included,
|
constant, as in the MPC), and periods when the model was still learning (included,
|
||||||
@@ -359,9 +394,10 @@ The brand icons in `custom_components/heating_mpc/brand/` are drawn by
|
|||||||
`python scripts/make_brand_icons.py` (uses Pillow from the test requirements);
|
`python scripts/make_brand_icons.py` (uses Pillow from the test requirements);
|
||||||
the package check verifies their sizes and transparency.
|
the package check verifies their sizes and transparency.
|
||||||
|
|
||||||
HA tests cover setup, config/options forms, migration, reload (a single relay
|
HA tests cover setup, config/options forms (including the reset), migration,
|
||||||
listener and polling coordinator after repeated reloads) and entity cleanup,
|
reload (a single relay listener and polling coordinator after repeated reloads),
|
||||||
scheduled sampling and poll cancellation, the relay listener, stored model
|
unload, reload and removal while an optimization or a save is still running,
|
||||||
|
entity cleanup, scheduled sampling and poll cancellation, the relay listener, stored model
|
||||||
restoration, retention and deletion, missing inputs, manual off/presets, optimizer
|
restoration, retention and deletion, missing inputs, manual off/presets, optimizer
|
||||||
failure and Recorder exclusions. A fixture rejects all HA service calls throughout
|
failure and Recorder exclusions. A fixture rejects all HA service calls throughout
|
||||||
each test, and a static test fails if actuator service calls are added to the
|
each test, and a static test fails if actuator service calls are added to the
|
||||||
|
|||||||
+19
-4
@@ -95,6 +95,18 @@ changed during optimization or by an overtaken run; and model accuracy reported
|
|||||||
as if it were forecast accuracy. Reload/unload tests now cover repeated reloads
|
as if it were forecast accuracy. Reload/unload tests now cover repeated reloads
|
||||||
(one relay listener, one polling coordinator). Released as 0.4.1; not yet tested on HAOS.
|
(one relay listener, one polling coordinator). Released as 0.4.1; not yet tested on HAOS.
|
||||||
|
|
||||||
|
### Final stability fixes after 0.4.1 — 2026-10-27
|
||||||
|
|
||||||
|
An independent review of 0.4.1 reported four problems; three were reproduced and
|
||||||
|
the fourth (lifecycle races) confirmed by reading Home Assistant's unload order.
|
||||||
|
Fixed with regression tests (see AUDIT.md): Model Error spans with unknown inputs
|
||||||
|
are now `incomplete`, not scored; window sensors unavailable at startup are
|
||||||
|
unknown, with last known states saved and restored as unknown; unload, reload
|
||||||
|
and removal can no longer publish stale results or recreate a deleted model;
|
||||||
|
Forecast Error scores only forecasts whose plan the relay matched within 2
|
||||||
|
minutes. Also added: **Reset learned model** in Configure. Not released or tested
|
||||||
|
on HAOS yet.
|
||||||
|
|
||||||
## Milestone 1 — Explainable forecasts and dashboards (target v0.3.0)
|
## Milestone 1 — Explainable forecasts and dashboards (target v0.3.0)
|
||||||
|
|
||||||
**Objective:** Users can verify what the model predicts and whether it is actually accurate.
|
**Objective:** Users can verify what the model predicts and whether it is actually accurate.
|
||||||
@@ -103,7 +115,7 @@ as if it were forecast accuracy. Reload/unload tests now cover repeated reloads
|
|||||||
- [x] Expose predicted +15/+30/+60/+120 minute temperatures and full proposed ON/OFF schedule. *(0.3.0: `forecast_15min` and `plan_15min` on each room forecast entity.)*
|
- [x] Expose predicted +15/+30/+60/+120 minute temperatures and full proposed ON/OFF schedule. *(0.3.0: `forecast_15min` and `plan_15min` on each room forecast entity.)*
|
||||||
- [ ] Provide an HA dashboard example using native cards or a clearly documented optional chart card (do not bundle a heavyweight frontend unnecessarily).
|
- [ ] Provide an HA dashboard example using native cards or a clearly documented optional chart card (do not bundle a heavyweight frontend unnecessarily).
|
||||||
- [ ] Make clear that the **MPC plan forecast is counterfactual** while Generic Thermostat is controlling the relay; add a separate actual-policy forecast/evaluation pathway. *(0.3.0: plan forecasts labelled counterfactual; evaluation uses the actual relay history. Stabilization: published forecasts are scored only when the relay followed their plan and otherwise counted as `counterfactual`. An actual-policy forecast, simulating Generic Thermostat, is still open.)*
|
- [ ] Make clear that the **MPC plan forecast is counterfactual** while Generic Thermostat is controlling the relay; add a separate actual-policy forecast/evaluation pathway. *(0.3.0: plan forecasts labelled counterfactual; evaluation uses the actual relay history. Stabilization: published forecasts are scored only when the relay followed their plan and otherwise counted as `counterfactual`. An actual-policy forecast, simulating Generic Thermostat, is still open.)*
|
||||||
- [ ] Store a bounded history of predictions with snapshots of inputs, proposed plan and eventually observed outcome. *(Stabilization: each published forecast is kept with its start readings, plan, model revision and assumptions until scored, in memory only; the scored statistics persist. No browsable history yet.)*
|
- [ ] Store a bounded history of predictions with snapshots of inputs, proposed plan and eventually observed outcome. *(Stabilization: each published forecast is kept with its start readings, plan, model revision and assumptions, including any it had to assume, until scored, in memory only; the scored statistics persist, split into matched, approximate, counterfactual and incomplete. No browsable history yet.)*
|
||||||
- [x] Implement out-of-sample/walk-forward MAE and bias at 15, 30, 60 and 120 minutes; include sample counts and confidence/quality labels. *(0.3.0: replay with parameters frozen at the forecast start and the actual relay history; `skill` against persistence. Stabilization: these are now the Model Error sensors, with RMSE, effective sample counts and quality labels; a separate Forecast Error sensor scores published forecasts. No confidence intervals yet.)*
|
- [x] Implement out-of-sample/walk-forward MAE and bias at 15, 30, 60 and 120 minutes; include sample counts and confidence/quality labels. *(0.3.0: replay with parameters frozen at the forecast start and the actual relay history; `skill` against persistence. Stabilization: these are now the Model Error sensors, with RMSE, effective sample counts and quality labels; a separate Forecast Error sensor scores published forecasts. No confidence intervals yet.)*
|
||||||
- [ ] Compare against simple benchmarks such as current-temperature persistence and a fixed thermal baseline. *(0.3.0: persistence done; fixed baseline open.)*
|
- [ ] Compare against simple benchmarks such as current-temperature persistence and a fixed thermal baseline. *(0.3.0: persistence done; fixed baseline open.)*
|
||||||
- [ ] Add inspectable explanations: expected heat loss, expected heating response, target, room weights and reason for the next proposed action.
|
- [ ] Add inspectable explanations: expected heat loss, expected heating response, target, room weights and reason for the next proposed action.
|
||||||
@@ -187,9 +199,12 @@ Items 1–3 of the original backlog are done (see AUDIT.md and CHANGELOG.md 0.2.
|
|||||||
3. Configure zones, the UPS heat source and the windows on HAOS; after a week of
|
3. Configure zones, the UPS heat source and the windows on HAOS; after a week of
|
||||||
heating review Model Error `improvement_vs_base` and `quality`, per-zone bias,
|
heating review Model Error `improvement_vs_base` and `quality`, per-zone bias,
|
||||||
coefficient `confidence` (base layer included), `model_changes` after any
|
coefficient `confidence` (base layer included), `model_changes` after any
|
||||||
reconfiguration, and the Forecast Error counts (how many forecasts were scored
|
reconfiguration, and the Forecast Error counts (matched, approximate,
|
||||||
vs counterfactual). Switch off optional terms that do not help. Also try an
|
counterfactual, incomplete and why). Switch off optional terms that do not
|
||||||
unavailable window sensor and a zone edit during HAOS acceptance.
|
help. During HAOS acceptance also try: a window sensor unavailable across an HA
|
||||||
|
restart, a zone edit, a reload while the integration is running, *Reset learned
|
||||||
|
model* and removing a disposable entry (its `.storage/heating_mpc.<entry_id>`
|
||||||
|
file must be gone afterwards).
|
||||||
4. Milestone 1 (rest): a dashboard example; forecast timestamps/provenance;
|
4. Milestone 1 (rest): a dashboard example; forecast timestamps/provenance;
|
||||||
explanations for the proposed action; an actual-policy forecast that simulates
|
explanations for the proposed action; an actual-policy forecast that simulates
|
||||||
Generic Thermostat.
|
Generic Thermostat.
|
||||||
|
|||||||
+47
-12
@@ -98,11 +98,19 @@ known. While a window is open, possibly open (its sensor is unknown and it was l
|
|||||||
seen open) or the zone is recovering, the base and extended layers do **not**
|
seen open) or the zone is recovering, the base and extended layers do **not**
|
||||||
learn, so window events cannot inflate `loss` or shift `bias`.
|
learn, so window events cannot inflate `loss` or shift `bias`.
|
||||||
|
|
||||||
An unknown window state is never read as closed. A window last seen open stays
|
An unknown window state is never read as closed, and an opening or closing counts
|
||||||
assumed open in forecasts; one last seen closed contributes no ventilation term.
|
as confirmed only when it was seen: the previous reading was the other state with
|
||||||
If a window closes while its sensor was unknown, the event's duration is reported
|
no unknown period (or restart) in between. If a window closes while its sensor was
|
||||||
as a range and the recovery term is not learned from it, because the closing time
|
unknown, the event's duration is reported as a range and the recovery term is not
|
||||||
is not known.
|
learned from it, because the closing time is not known. Last known states, open
|
||||||
|
events and recovery are saved and restored across restarts, as unknown until the
|
||||||
|
sensor reports again.
|
||||||
|
|
||||||
|
An input that a term needs is never filled in for learning or for Model Error:
|
||||||
|
while a window is open, the wind, its direction (for `vent_wind_dir`) and the other
|
||||||
|
zones' windows (for `vent_cross`) must be known. With every window closed, the
|
||||||
|
opening terms are 0 and need no weather. Forecasts, which must produce a number,
|
||||||
|
assume calm air for a missing wind input.
|
||||||
|
|
||||||
`vent_wind_dir` is bounded by `vent_wind` (|vent_wind_dir| ≤ vent_wind), so the
|
`vent_wind_dir` is bounded by `vent_wind` (|vent_wind_dir| ≤ vent_wind), so the
|
||||||
combined wind term never goes negative: a leeward window still ventilates.
|
combined wind term never goes negative: a leeward window still ventilates.
|
||||||
@@ -202,7 +210,10 @@ minutes):
|
|||||||
its response time. A gaming PC drawing 600 W now is not assumed to run on for two
|
its response time. A gaming PC drawing 600 W now is not assumed to run on for two
|
||||||
hours.
|
hours.
|
||||||
* **Windows:** an open window is assumed to stay open for a configurable time
|
* **Windows:** an open window is assumed to stay open for a configurable time
|
||||||
(default 15 minutes), then close; the zone then recovers.
|
(default 15 minutes), then close; the zone then recovers. A window whose sensor
|
||||||
|
is unknown but was last seen open is assumed open only until that time after it
|
||||||
|
was last seen, never renewed. One last seen closed, or never seen since setup, is
|
||||||
|
assumed closed, and the forecast is flagged as uncertain (`uncertain_windows`).
|
||||||
* **Relay:** the candidate plan being evaluated.
|
* **Relay:** the candidate plan being evaluated.
|
||||||
|
|
||||||
For the plan search the step is compiled into exact per-zone coefficients (every
|
For the plan search the step is compiled into exact per-zone coefficients (every
|
||||||
@@ -228,13 +239,34 @@ scored on exactly the same spans:
|
|||||||
* `base_model_mae`, `improvement_vs_base`: against the base layer alone. A feature
|
* `base_model_mae`, `improvement_vs_base`: against the base layer alone. A feature
|
||||||
that does not improve this should be switched off (for example the gust term).
|
that does not improve this should be switched off (for example the gust term).
|
||||||
|
|
||||||
|
A span counts only if every input the zone's model used was observed at every
|
||||||
|
step (see *Ventilation layer*; for the extended layer also electrical heat and
|
||||||
|
neighbours, whose own replay must be complete). Otherwise it is counted as
|
||||||
|
`incomplete`, with the base model's error on it kept apart
|
||||||
|
(`incomplete_base_model_mae`), since the base model needs only the relay and the
|
||||||
|
outdoor temperature. Spans that cannot be replayed (uncertain radiator lag, no
|
||||||
|
reading at the end) are counted as `skipped`.
|
||||||
|
|
||||||
**Forecast accuracy (Forecast Error sensor)** scores forecasts as published: the
|
**Forecast accuracy (Forecast Error sensor)** scores forecasts as published: the
|
||||||
start readings, trajectory, plan, model revision and assumptions of the moment
|
start readings, trajectory, plan, model revision and assumptions of the moment
|
||||||
they were made, nothing replaced with hindsight. Because the plan is the MPC's
|
they were made, nothing replaced with hindsight. Because the plan is the MPC's
|
||||||
proposal while Generic Thermostat controls the relay, a forecast is scored at a
|
proposal while Generic Thermostat controls the relay, a forecast is compared only
|
||||||
horizon only if the relay was in the planned state for at least 12 minutes of
|
when the relay followed it. The deviation is the time, up to the horizon, the
|
||||||
every 15-minute block up to it; otherwise it is counted as `counterfactual` (or
|
relay spent in another state than planned, compared block by block from the
|
||||||
`unverifiable` when the relay history is incomplete) and never as an error.
|
relay's own switch times; unobserved relay time counts as possible deviation.
|
||||||
|
|
||||||
|
| Deviation (worst case) | Class | Used for |
|
||||||
|
|---|---|---|
|
||||||
|
| ≤ 2 min | matched | Forecast Error figures |
|
||||||
|
| ≤ 5 min, ≤ 2 min unobserved | approximate | `approximate_stats` only |
|
||||||
|
| > 5 min observed | counterfactual | counted, never an error |
|
||||||
|
| otherwise | incomplete | counted, with a reason |
|
||||||
|
|
||||||
|
Forecasts made with an assumed input (an unknown window or electrical heat) are
|
||||||
|
incomplete as well. The limits follow from the model: d seconds of unplanned
|
||||||
|
boiler heat change a room by about heat × d / 300 with the learned `heat`
|
||||||
|
coefficient (typically 0.05–0.12 °C per 5 minutes), so 2 minutes stay below half a
|
||||||
|
0.1 °C sensor step and 5 minutes are about one step.
|
||||||
|
|
||||||
Both report how much evidence there is. Forecasts start every 5 minutes, so at
|
Both report how much evidence there is. Forecasts start every 5 minutes, so at
|
||||||
horizon h consecutive forecasts share most of their span, and all zones share one
|
horizon h consecutive forecasts share most of their span, and all zones share one
|
||||||
@@ -267,5 +299,8 @@ the contamination the separate ventilation layer prevents.
|
|||||||
variation their split is uncertain even when their combined effect is accurate.
|
variation their split is uncertain even when their combined effect is accurate.
|
||||||
* Forecasts hold weather constant and use one scenario for electrical heat and
|
* Forecasts hold weather constant and use one scenario for electrical heat and
|
||||||
windows; no forecast uncertainty band is computed yet.
|
windows; no forecast uncertainty band is computed yet.
|
||||||
* Window sensors with a permanently unknown state keep their zone out of learning;
|
* Window sensors with a permanently unknown state keep their zone out of learning
|
||||||
disable such a window.
|
and its Model Error spans incomplete; disable such a window.
|
||||||
|
* A window that is unknown and was last seen closed (or never seen) is assumed
|
||||||
|
closed in forecasts, flagged as uncertain; no open/closed scenario pair is
|
||||||
|
computed.
|
||||||
Reference in new issue
Block a user