Skip to content

CHS: freeze states after each person's last observed measurement - #104

Open
hmgaudecker wants to merge 1 commit into
asymmetric-translogfrom
frozen-after-last-observation
Open

hmgaudecker wants to merge 1 commit into
asymmetric-translogfrom
frozen-after-last-observation

Conversation

@hmgaudecker

@hmgaudecker hmgaudecker commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Stacked on #102 (asymmetric-translog).

Problem

Persons with no later observed measurement (death, attrition) are predicted through every remaining period. In a translog, their predicted covariance explodes (about 1e19 at ages 90+ in health-cognition). The QR of the missing-measurement update then has a zero R diagonal, the floored QR derivative overflows, and the where-mask turns 0 · inf into a NaN gradient. The log-likelihood stays finite.

Change

  • predict_if_observed_later (in chs/likelihood.py, used by both scans): the predict step runs only for persons with an observed measurement in a later update. All others keep their last filtered states and covariance. Their predict step runs on placeholder inputs (zero states, identity covariance) and is discarded (double-where).
  • kalman_update: the QR for missing measurements is built from an identity covariance. The result is discarded as before.
  • debug_loglike: filtered states are NaN for the periods after a person's last observation.

The log-likelihood does not change, because the frozen updates contribute exactly zero, before and after the change.

Tests (tests/test_frozen_after_last_observation.py)

  • Explosive translog with a person last observed in period 1: the gradient matches central finite differences. Before the change, it was non-finite.
  • loglikeobs and the gradients at three random parameter vectors are equal to the values recorded on the unchanged code (9863a78).
  • Filtered states are NaN exactly after the last observation. A gap followed by more observations stays finite.

Full suite: 870 passed.

https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq

Persons with no later observed measurement (death, attrition) were predicted
through every remaining period. In a translog their predicted covariance
explodes (about 1e19 at ages 90+ in health-cognition), the QR of the
missing-measurement update gets a zero R diagonal, and the floored QR
derivative overflows; the where-mask then turns 0 * inf into a NaN gradient
while the log-likelihood stays finite.

- The predict step runs only for persons with an observed measurement in a
  later update. All others keep their last filtered states and covariance; their
  predict step runs on placeholder inputs (zero states, identity covariance)
  and is discarded.
- kalman_update builds the QR for missing measurements from an identity
  covariance; the result is discarded as before.
- debug_loglike reports filtered states as NaN for the periods after a
  person's last observation.

The log-likelihood is unchanged: the frozen updates contribute exactly zero
before and after. Tests pin loglikeobs and gradients from the unchanged code
and check the gradient against finite differences where it overflowed before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y3CuwvNUhYTSpriKYSwqmq
@codecov

codecov Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.07%. Comparing base (9863a78) to head (ccbf9b5).

Additional details and impacted files
@@                   Coverage Diff                   @@
##           asymmetric-translog     #104      +/-   ##
=======================================================
+ Coverage                96.05%   96.07%   +0.02%     
=======================================================
  Files                      147      148       +1     
  Lines                    14998    15089      +91     
=======================================================
+ Hits                     14406    14497      +91     
  Misses                     592      592              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hmgaudecker
hmgaudecker requested a review from janosg October 24, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant