Skip to content

feat(trajectory): show LoopX tool context in research usage - #7758

Open
shangzh0 wants to merge 3 commits into
loopx-project:mainfrom
shangzh0:codex/trajectory-context-cost-20261101
Open

shangzh0 wants to merge 3 commits into
loopx-project:mainfrom
shangzh0:codex/trajectory-context-cost-20261101

Conversation

@shangzh0

@shangzh0 shangzh0 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Goal And Delivered Outcome

Research usage can show a recorded Turn-packet token estimate while omitting LoopX CLI tool returns. Context reads, state readback and settlement returns therefore remain outside that delivery comparison.

Add --include-tool-context to the existing source-bound trajectory registration. The Research workbench displays the saved byte inventory in its existing Sources and usage disclosure: appended input text, recognized LoopX return bytes by operation, repeated identical content, unassigned text and unmeasured parts. Viewing does not invoke analysis.

Outcome basis: this self-contained observation gap. Intended base: loopx-project/loopx:main at 04c5799b88f6c4cf12a224b7fa679f272e1e794e. Existing provider-token attribution and recorded-packet estimates retain their own units and denominators.

Author Declaration

Written by: model_agent — OpenAI Codex, GPT-6 family.

Implemented against docs/reference/trajectory-assessment.md at the base above: source-verified registration, separate facts/judgments/cost, unknown coverage and effect-free reading. This PR defines the added inventory's bounded measurement contract in that same document.

Criterion Disposition Owner Decisive evidence
Source-bound, opt-in preparation with unchanged defaults implemented Existing assessment store / CLI Real CLI registration, optional-field parity and capture digest checks
Keep text bytes separate from request tokens and model judgments implemented TS trajectory types/reducer Independent denominator, overflow, unknown, mirror and attribution-mutation tests
Preserve honest attribution limits implemented Codex IO adapter / TS reducer Compound/indirect commands unassigned; literal wrapper and non-text coverage tests
Read measured facts without invoking analysis implemented Research Sources and usage Packaged page, real source verifier/registry/HTTP owner, no analysis outputs or model calls

Self-check: full diff and existing request/token owners reviewed; the new shared reduction stays in TypeScript. Python performs Codex IO and lexical parsing only. Observation buckets do not create execution/settlement states. Related campaign-effort, context-read diagnostics and quota-projection work retain their existing owners.

Scope And Continuation

Complete within the recorded-text inventory scope. Only simple POSIX argv, Python module invocation and an exact single literal exec wrapper are recognized. Compound scripts, indirect/file reads and other content remain unassigned. Duplicate records for one call count once; repeated content from different calls remains recorded input and is not automatically waste.

This is neither whole control-plane overhead nor per-request replay, provider-token billing, avoidable cost or causal score benefit. Injected instructions may remain unassigned, and useful research evidence can occur inside LoopX returns. Historical captures cannot supply missing request lineage; per-episode replay attribution remains separate work. Solver prompts, scoring and continuation policy are unchanged. The independent streaming proposal #7757 reduces capture-read allocation without being a prerequisite here.

Validation

Tested revision: 4e04198a793be7650d0ccf545a89019e48ad8f16. Run state: finished. Input classes: synthetic; authorized private captures were separately read locally and are excluded from this proposal.

Check kind Result Evidence / limitation
unit / integration passed 82 Python tests: tool context, input capture, assessment store and HTTP analysis
unit / regression_parity passed 29 TS tests: inventory, session counters, request reconciliation and input attribution; omitted option preserves prior projection, invalid supplementary observation preserves native counters
static passed Control-plane typecheck, Ruff, semantic advisory, diff check and explicit public-boundary scan of all 16 changed files; absent local Goal registry reported separately
real_entrypoint / real_backend passed Packaged research-tool-context scenario: actual source verifier, typed reducer, registry and HTTP owner with isolated SQLite 3.53.4; opt-out, measured share, conflicting capture, keyboard, desktop/phone and English; zero model calls and unchanged synthetic sources
real_entrypoint passed Dashboard build; clean wheel install/help/provider import, typed observation from that installed wheel and dependency check
integration not_run Full CI, installed App adoption, whole-run billing and solver effects are not qualified by these checks

The first browser attempt used an unqualified Node 24.14 SQLite driver. Final acceptance used Node 24.22.0 / SQLite 3.53.4. Test-harness disclosure/locale handling was corrected before the passing run. No live solver run was launched.

Frontend / Visual Evidence

UI impact: changed. The optional result stays inside the existing Sources and usage disclosure; the normal page remains unchanged while it is closed. The owner can read the inventory in place without a new page, form or model call. Byte units and captured-text scope remain visible; method limitations are available on demand.

Source data: synthetic. These are packaged source checks for the tested revision above, not installed App or scientific qualification. The revision-pinned visual artifact is on a dedicated evidence branch and excluded from the proposed product diff. No public running preview is provided; the executable scenario reproduces the journey.

Before, optional inventory disabled:

Before: existing research page

After, desktop Sources and usage readback:

After: packaged Research usage with tool context

Phone detail:

Phone: tool context byte inventory

Attention/craft review: the requested metric sits beside existing token attribution and uses existing typography/disclosures. Raw commands, output text and runtime paths are not rendered. Unknown/conflicting capture stays unavailable rather than zero; narrow detail fits without horizontal overflow. No new decorative cards, color constants or motion were added.

Type of Change

  • New feature
  • Documentation update
  • Test update

LoopX Area

  • Capability or extension (existing benchmark-toolkit)
  • Benchmark boundary (read-only evidence observation)
  • Public docs or presentation surface

Technical direction: existing S5/S11 Research observer and complete-cost measurement. Shared-authority RFC fixture impact: N/A; this does not promote or replace an authority provider.

Boundary Checklist

  • No private state, credentials, raw trajectories, local paths or internal links appear in the diff/body/visuals.
  • Reuses existing trajectory, input attribution and Research owners; related campaign reporting is not duplicated.
  • Scoped to the observation gap above; no paired RFC changed.
  • Visual evidence and tested packaged boundary are included.
  • Commit includes DCO sign-off.

Lifecycle inventory follow-up

At e773efcf745d5c110b8e8e13f4492a358bf8888b, split task-lease acquire/inspect/renew/release/transfer and common Todo add/list/claim/complete/update replies into separate operation rows. Recognize the real python -m loopx.cli entrypoint alongside the existing module forms. Unknown subcommands keep their old fallback bucket; no command/output contents are exposed. Categories still describe observed replies, not successful mutations or avoidable work. Historical saved inventories are unchanged.

Validation for this follow-up: 12 TypeScript inventory/native-counter tests and 58 Python capture/store tests passed; control-plane typecheck, semantic advisory, diff/public-boundary and bilingual checks passed. The new lease grouping regression fails on the preceding PR head. A newly built wheel was installed in an isolated environment and its shipped TypeScript owner exercised outside the checkout. No screen or layout changes; the earlier public-safe screenshots remain evidence for the unchanged renderer at their explicitly recorded revision. No live model run or benchmark score gain is claimed.

Review follow-up — head 5d3a5d1

Addressed the canonical digest consumer finding in commit 5d3a5d1ebb3ab9f6050449ee0bed0ae091d1ce33: added control_plane/capabilities/trajectory_tool_context.ts to the existing CANONICAL_CONSUMERS inventory. The follow-up changes one test-manifest line. The shared digest import, strict consumer equality check and production code are unchanged.

Validation of this source tree:

  • Reproduced the review finding before repair: the named pinned-consumer guard failed with exactly the missing consumer.
  • Complete content_digest_single_owner.test.ts: 17 passed, including the repaired strict guard.
  • Complete existing CI-selected TypeScript shard 2/3: 1,979 passed, 0 failed, 31 reported skips, across 118 files. No skip, assertion or threshold was changed.
  • Three existing Python trajectory capture/context/store suites: 58 passed.
  • Control-plane TypeScript no-emit check and git diff --check: passed.

The first two complete-shard attempts each had one failure because a Python CLI child reused an already-running Node 24.14.0 / SQLite 3.51.2 runtime. Pinning only the parent Node was insufficient. A disposable temp namespace with the qualified Node fixed that environment mismatch: the original failed CLI case passed, then the entire unchanged shard passed. The older runtime was not restarted or modified; the failed attempts remain recorded, not relabelled passes.

The earlier Windows locator-publication failure remains a separate platform verification risk. Its original Python/server/test sources are unchanged from base, and all three relevant Linux cases pass on both base and this tree; that does not establish Windows success or an infrastructure cause. New-head remote CI is pending, not declared green. The related browser budget investigation and full scenario comparison are recorded in #7759's review follow-up; no budget was loosened here.

No additional UI or production behavior changed in this follow-up. Earlier visual and installed-wheel evidence retain their stated original revisions. This update requests re-review; it neither dismisses the existing review nor claims merge readiness or benchmark gains.

Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>
Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>

@songoow songoow left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

动机 / Motivation

发现:P2,需要补齐 canonical digest consumer 清单。 新文件 trajectory_tool_context.ts:2 合理地复用了 BARE_SHA256_PATTERN,但 tests/control_plane_ts/content_digest_single_owner.test.ts 的 CANONICAL_CONSUMERS 没加入它。因此仓库已有的消费者守卫稳定失败,与旧 head 的 typescript-core (2/3) CI 一致。功能测试通过不能消除这项集成回归。

受影响的是在 Research 中检查已冻结 Codex 用量记录的操作者和代理。此前只能看到原生用量与已有 packet 归因;启用本项后可以另看已记录工具返回的字节占比和操作明细。可观察增量是单独的工具文本清单,未归因与未计量内容仍显式保留,原生 token 计数不被替换。本项不负责降低 token、压缩上下文、推导 provider 账单或证明任务性能收益。这个增量有实际诊断价值:可以先找出哪些工具返回占据已记录文本,再决定是否值得优化。它本身不证明已经节省 token,也不能把重复返回全部认定为浪费。

契约依据:docs/reference/trajectory-assessment.md,固定版本 04c5799b88f6c4cf12a224b7fa679f272e1e794e;重点核对 Reconcile request and compaction coverage 和 LoopX packet share of model input 两节:有效原生计数应独立保留,字节不能冒充 provider token 或因果收益。

改动思路 / Approach

保留 Python 的文件与字面量解析、TypeScript 的归类和归约、UI 的只读展示分工;复用摘要校验所有者是正确方向,缺失的是已有消费者清单的同步。当前 PR 的边界应保持为可选工具文本观测,先补齐 canonical consumer 清单并完成相关验证,不扩大为 token 计费或自动优化策略。不做此改动会继续缺少工具文本视图,而把字节混入既有 token 归因会改变计量含义。独立的可选投影是合适的设计;不需要增加计费状态或执行任意 shell/JavaScript 来提高识别率。

实际路径是已有登记命令显式传入 --include-tool-context,Python 从原生记录提取长度、摘要和调用标识,TypeScript 校验、去重并聚合,Research 读取保存的投影。来源仍由既有 hash 和历史状态约束,日常页面读取不调用模型。未启用时保留字段缺失;不支持的编排形式保持未归因,不以自然语言包含 “loopx” 作为识别依据。

具体改动 / Changes

审阅了当前完整 16 文件 diff(+575/-2)及周边调用方。发表前 head 从 4e04198a793be7650d0ccf545a89019e48ad8f16 更新为 e773efcf745d5c110b8e8e13f4492a358bf8888b,已重读新增提交并重新验证:它细分租约/Todo 返回类别、支持 python -m loopx.cli,没有补齐消费者清单;这些类别仍只表示观测到的回复,不表示成功写入或应被省略的恢复。project_tool_context 只提取原生 response_item;镜像事件和模型 prose 不作为额外输入。字面量解析不执行命令,复杂脚本、展开和管道保留未知。summarizeTrajectoryToolContext 校验行序、调用标识、非负安全整数及摘要;同一输出镜像去重,不同调用的相同内容保留为重复输入,零分母保持未知。ToolContextInventory 把字节比例与 provider token 单独呈现,详细计数可以展开。

实际验证:四个相关 TypeScript 文件 30 项通过;三个 Python 文件 58 项通过,覆盖真实 CLI 登记、来源变化、可选观测失败和原生计数保留。Python 执行使用固定 head 的源码及真实本地 Effect runtime;没有模型调用。

独立回归对照在相同环境运行既有消费者测试:base 04c5799b88f6c4cf12a224b7fa679f272e1e794e 1 pass,head e773efcf745d5c110b8e8e13f4492a358bf8888b 1 fail,差异只有新增消费者 control_plane/capabilities/trajectory_tool_context.ts。可复现命令:

node --no-warnings --experimental-sqlite --experimental-strip-types --test --test-isolation=none --test-name-pattern=the.owner.s.consumers.are.the.pinned.set tests/control_plane_ts/content_digest_single_owner.test.ts

请把该路径加入既有 CANONICAL_CONSUMERS 后重跑上述守卫及受影响 TS shard。应保留 canonical import 和严格断言,避免用复制正则或放松测试绕过所有权约束。两端 diff hygiene 已通过;语义增量扫描检查六个 source snapshot,未发现其支持范围内的新词汇载体,这只是有限静态 advisory。

对主干的风险 / Risks

已复现风险是当前主干集成门会失败,未观察到因此导致运行时计数损坏。字节库存和重复文本仍可能被读者误解为完整模型输入或可避免开销,现有单位、未知值及说明应保留。仅相同 head 的开关对照已验证 native projection 一致;本次未完成完整 baseline/head 的同输入 UI 语义矩阵,不宣称全路径等价。

另外保留了旧 head 4e04198a793be7650d0ccf545a89019e48ad8f16 的两个浏览器失败日志:dashboard-browser (3) 的一次 lens switch 为约 1142 ms,超过既有 1000 ms 上限;chat-bundle-browser (1) 为 delegation smoke 超时。新 head 的 CI 刚开始,当前没有已完成的失败检查;上述旧失败仍没有同条件 base/head 对照或独立归因。本次没有把历史失败冒称为新 head 的红灯,也不以 pending 本身阻止批准。它们是重新评审时需考虑的浏览器风险;不得靠放宽预算消除。当前阻断代码证据来自新 head 上独立重现的消费者守卫失败。

我的整体评价 / Verdict

REQUEST_CHANGES。功能边界与分工合理,已有通过测试支持其作为可选诊断增量;当前不批准的明确代码原因是消费者清单缺失。修复后还应核对当前浏览器验证和相关隔离证据,不能只选取绿色运行。这里不要求该 PR 顺带实现上下文压缩或证明实验收益。

已目视检查作者公开的 before、desktop、phone 截图(原测试 revision 为 4e04198;新增提交未修改其 UI 组件与布局):字节占比与 token 归因分列,桌面和窄屏展开明细可读;这些是合成数据截图,不能证明真实浏览器交互或历史来源恢复。尚未独立执行完整页面的长列表、未知值与来源失效恢复交互;旧 head 的浏览器失败仍保留为风险,新 head 的 CI 正在运行。上述不足是验证范围声明,不是凭截图推断出的 UI 缺陷。

Reviewer: model_agent; GPT-6; OpenAI; self_reported.

English verdict: REQUEST_CHANGES for exact head e773efcf745d5c110b8e8e13f4492a358bf8888b. Add the new canonical digest consumer to the pinned inventory and rerun the guard; independently reproduced base PASS / head FAIL. Keep the optional byte inventory separate from native token accounting. Earlier-head browser failures remain a disclosed risk; current CI is pending. Complete baseline/head UI parity has not been established.

Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>

songoow commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

后续 token 优化评估建议:复用已有请求账本,对照完整任务过程

基于当前 head 5d3a5d1ebb3ab9f6050449ee0bed0ae091d1ce33 的非阻塞建议。这个 PR 的 tool-context inventory 适合定位值得调查的操作及大回执;下一步可以把它与已有真实 request usage 并排使用,验证其他优化是否产生净收益。

现有 trajectory_request_usage.ts 已按 response ID 核对请求、compaction 及镜像,输出 request_coverage.all_requests 和 compaction,并保留旧记录 coverage 不明的状态。因此建议复用这份 owner,不再另建压缩 token 计数器,也不要把新 inventory 的 bytes 换算成已验证 token。

具体可以先做一个离线对照或小的现有报告扩展:固定源码、模型配置、任务和可比工作阶段,按完整 entry → 选读/实际工作 → 写回/结算 展示下列数据:

  • 现有请求账本的 input、cached input、uncached input、output、request count;compaction 单列。output 已包含的 reasoning 不重复相加。
  • 本 PR 的操作类别、回执字节、相同内容重复量,外加实际补读、失败恢复和分页调用。相同内容重复可帮助定位调查对象,不能直接判定可删除;压缩恢复或当前契约要求的重新读取可能是必要的。
  • 最终有效产出、失败/重试和墙钟时间。只有首个响应变短、后续又读全量时,应在完整过程里显示抵消后的成本。

如果缺少明确请求关联,就先做同一可比阶段内的并列统计;不要按时间相邻、字节比例或字符串相同,推断某个工具贡献了多少 provider token。已有明确 input-attribution 证据时沿用其 coverage/status,缺失仍报告 unknown。缓存是否受益也由真实 cached/uncached counters 说明,不能仅由提示词变短推断。

这样可以直接检验 #7761 的目录是否被实际消费、#7711 的 focused receipt 是否避免了后续全板读取,以及未来分阶段 context 的收益是否被补读或 compaction 抵消。建议作为测量工具的后续应用,不要求本 PR 扩大为 token 优化实现;本 PR 自身也不能单独证明模型 token 已下降。

本条仅为计量与实验设计建议,不重新裁决当前 head 的完整 CI 或合并资格。此前 canonical consumer finding 的验证会在原线程单独回复。

Reviewer: actor_kind=model_agent; model=GPT-6 Astra Pro; provider=OpenAI; declaration_source=self_reported.

@songoow songoow left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | GPT-6 Astra Pro | OpenAI | self_reported

审阅固定在 5d3a5d1ebb3ab9f6050449ee0bed0ae091d1ce33,不可变基线 04c5799b88f6c4cf12a224b7fa679f272e1e794e。上一轮 canonical digest consumer 清单缺项已修复;本轮不重复该旧 finding。

动机 / Motivation

受影响的是在 Research 页面核对原生会话成本、需要区分 LoopX 工具返回内容与真实 token 计数的研究人员。 旧页面保留原生 token 总量,却无法把其中可归属的工具返回文本单独列出;本次显式登记工具清单后,读者能查看字节占比和具体命令,同时仍看到缺失或冲突的测量是未知。 真实登记、独立回读和打包页面验证了可选字节清单:60 B 占记录文本 200 B 的 30%,原生输入 100、缓存 60、输出 5 tokens 保持分开。 范围是已记录轨迹的可选只读观测,不估计未记录 token、不计算因果收益、不启动模型分析,也不改变评分、准入或原始证据。

改动思路 / Approach

Python只投影已记录文本,TypeScript拥有校验与归约,React只读显示;分离字节清单和原生计费避免把可选观测变成评分权威。 本PR交付登记、保存、回读和已有Research页面的完整可选清单,不新增模型调用或通用账务系统。 只做页面字段会失去来源和冲突判定,只用原生token总量也不能回答已记录工具文本由哪些命令组成。当前方案复用原capture、来源验证和Sources/usage展开区,没有增加第二个可写账务owner。动态命令无法被字面解析时保留未归属;非文本内容保持未计量,不为了得到百分比补造数据。

具体改动 / Changes

关键代码讲解

  • project_tool_context 读取原生call/output和字面命令,仅输出计量记录。它不会执行轨迹中的shell或Python代码;无法静态识别的命令保持未归属。
  • summarizeTrajectoryToolContext 归约已记录文本,区分重复记录与真正重复返回,并在call身份冲突时保留未知。原生token计费仍由现有usage owner负责。
  • ToolContextInventory 在已有用量展开区显示字节占比、统计口径和逐命令细节。无inventory的历史记录不显示该区域,token占比未知仍用未知表示。

规范依据是基线 docs/reference/trajectory-assessment.md,revision 04c5799b88f6c4cf12a224b7fa679f272e1e794e;本评审以 source-bound-reading 标识原有“来源绑定、阅读不执行”要求,以 native-usage-truth 标识原生用量与未知边界。这些是评审定位标签,原文没有相应编号。两项在当前登记/回读路径已实现;本PR新添的字节说明没有被用作自证标准。17files +576/-2主要覆盖适配器、typed归约、展示与测试,最新复审提交只增加一行canonical consumer登记。

独立验证

47项TypeScript测试通过,包括原摘要所有权守卫;tsc --noEmit、Ruff和git diff --check通过。Python三套件58个断言通过,但session因并行审查进程出现默认usage-ping目录触发隔离guard而exit1,保留该失败,不称整套全绿。

同一真实CLI与合成来源在base/head分别登记并独立inspect:默认关闭时完整native projection指纹相同;原生100输入、60缓存、5输出和原归属指标保留,分析job/result均为空。显式开关在base被拒绝,head只增加可选inventory。实际打包构建后的 research-tool-context 浏览器场景通过,覆盖键盘展开、1440/390视口、英文reload、冲突后未知、原始来源不变和零mutation/model调用。浏览器是本地Chromium138,与CI151不同。

对主干的风险 / Risks

没有发现新的可复现阻断性代码缺陷;原digest登记问题已经关闭。最大语义风险是将字节占比误读为token或因果成本,当前typed字段、分离展示和文档均保留边界。默认关闭路径有实际base/head全对象对照,不能把仅缺少一个feature对象当作隔离证据。

最新dashboard-browser (3)在 goal-workspace-m2.mjs:206 触发既有1秒loaded-lens预算:一次切换1074.907ms。上面的工具清单场景不覆盖该失败条件,也没有相同CI条件的不可变base对照,所以当前无法把红灯归为无关。最小补证是保留失败签名并提供该共享页面条件的有效当前覆盖或同条件base归因;不要求修改无关生产代码,也不放宽冻结预算。

我的整体评价 / Verdict

该可选观测切片有真实用途、复用现有owner且保存原始事实,重复阅读不会产生额外模型工作;登记到页面的主要路径已被独立执行。当前仍保留验证hold,针对尚未归因的dashboard失败和完整视觉资格。已目视检查本次打包页面的默认桌面、展开桌面和390像素展开截图,并复用原作者同UI修订的清晰截图;本地Chromium缺少中文字体,无法据本地截图确认中文正文排版质量。 长命令清单和混合状态整页的阅读密度仍未独立验证;本地中文字体缺失,Chromium138也不是远端CI的151。 自动检查通过证明字段、交互和无溢出,不等同于整页可读性已经通过;不会把本机字体问题归咎于PR。

English verdict: REQUEST_CHANGES — exact head 5d3a5d1; the previous digest-consumer issue is fixed and the real opt-in reading path passes, while current dashboard failure attribution and representative visual qualification remain incomplete.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants