Skip to content

Fix multiply's and matmul's Hessians when their children share a subexpression - #169

Merged
dance858 merged 2 commits into
mainfrom
shared-subexpr-mul-matmul
Oct 14, 2026
Merged

dance858 merged 2 commits into
mainfrom
shared-subexpr-mul-matmul

Conversation

@dance858

Copy link
Copy Markdown
Collaborator

Fixes #160.

Summary

  • Bug: multiply and matmul (chain-rule path) evaluated both children's Hessians and only then summed C + C^T + H_x + H_y with sum_4_matrices_fill_values. When the children share a node, evaluating the second child re-evaluates that node with other weights and overwrites the first child's Hessian before the sum reads it.
  • Fix: same as Fix add's Hessian when its children share a subexpression #159 for add. The Hessian pattern comes from sum_n_matrices_alloc with n = 4, and each term is added with sum_n_matrices_fill_add right after it is final (C and C^T first, then each child right after its eval_wsum_hess).
  • Not changed: problem.c's Lagrangian Hessian already fills incrementally. The variable × variable fast paths evaluate no child Hessians.
  • Performance: the fill does the same work as before (one memset plus four accumulator passes over the same bound maps), plus three inlined values_version++. In a single A/B run against main (Release), multiply (n = 200k, elementwise children) took 1.18 vs 1.16 ms and chain-rule matmul (exp(X) @ log(Y), 60×60) took 20.5 vs 21.5 ms. That's one run each and within the usual layout noise; no repeated runs were done.
  • Follow-up: sum_4_matrices_* now has no engine callers and can be dropped from the library later.

Tests

  • New test_wsum_hess_multiply_shared_subexpr (mul(log x, exp(log x)) and mul(exp(log x), log x)) and test_wsum_hess_matmul_shared_subexpr (log X @ exp(log X) and exp(log X) @ log X, X 2×2) check against the numerical Hessian with one shared log node.
  • Both fail on the old code (multiply: analytical 5.39 vs numerical 2.00; matmul: 5.39 vs −16.0) and pass with the fix.
  • All tests pass in Debug (306, SP_TRACK_MEMORY=ON) and Release (307).

🤖 Generated with Claude Code

dance858 and others added 2 commits October 14, 2026 11:15
…xpression

Both evaluated the two children's Hessians and summed afterwards, so a node
shared by the children could be re-evaluated with other weights before the
sum read it. Fill the sum incrementally with sum_n_matrices_fill_zero /
sum_n_matrices_fill_add, as add does since #159.

Fixes #160.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dance858
dance858 merged commit 367a77b into main Oct 14, 2026
14 checks passed
@dance858
dance858 deleted the shared-subexpr-mul-matmul branch October 15, 2026 05:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

multiply's Hessian is wrong when its children share a subexpression

1 participant