Skip to content

fix(benchmark): align VM host services with the native Docker firewall - #7691

Open
loopx-agent wants to merge 3 commits into
mainfrom
codex/benchmark-docker-host-bridge
Open

loopx-agent wants to merge 3 commits into
mainfrom
codex/benchmark-docker-host-bridge

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Docker VM 内运行 native runner、judge 和凭证网关时,Docker 的 host-gateway 可能指向桌面宿主;请求因此落到没有监听服务的地址。新增显式 --docker-host-bridge,从默认 bridge 的 Docker 元数据读取 IPv4 网关,将容器内 host.docker.internal 钉到该地址,与 native firewall 的宿主服务端点保持一致。Official 和 LoopX 共用同一 transport,匹配实验应统一传参。

选项默认关闭,默认地址、配置与 receipt 保持原行为;开启时记录在 runtime receipt。缺失、不可用或多义的 IPv4 网关在创建容器前拒绝,不猜测操作系统或固定网段。端口白名单、凭证隔离、反馈、任务、评分和预算规则均不变。该选项只适用于默认 Docker bridge,使用该别名的服务都需在 bridge 上可达;只在新 attempt 选择。

验证:

  • 464 项 focused Python 检查通过,覆盖 CLI 参数/receipt、Official 与 heartbeat、API key 与登录路由、非法拓扑及默认行为。
  • 独立 Linux Docker/netfilter 实测通过:允许端口可达,同一地址的未允许端口被拒绝;无需模型调用或真实任务。
  • 同一组 5 项默认路径 oracle 在原基线 c38b3a1a6 与候选均通过;在候选中故意丢弃 bridge pin,真实 Docker 用例按预期失败,容器清理成功。
  • Ruff、diff 检查和 RFC 双语差分检查通过。最初缺 Harbor / 合格 Node 的环境失败保留,补齐既有验证环境后上述检查通过;完整 CI 尚待远端执行。

此切片对应 S11 / E1 native transport 可靠性,不建立新的 kernel owner。Python 保留在既有 SForge Docker adapter,局部拓扑规则与普通 CLI 交付一并完成;无额外代理或推测性抽象。源代码资格、实验采用和长期效果分别评估,本 PR 不声称 token 节省或任务收益。

可重放的主要检查(使用已安装所选 SForge / Harbor 和合格 Node 的环境):

python -m pytest -q benchmark/tests/test_sforge_host_bridge.py benchmark/tests/test_sforge_runtime.py benchmark/tests/test_edgebench_feedback.py benchmark/tests/test_edgebench_prompts.py
LOOPX_EDGEBENCH_DOCKER_SMOKE=1 python -m pytest -q benchmark/tests/test_edgebench_feedback_docker.py::test_host_bridge_pin_reaches_only_the_native_allowed_port

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@songoow songoow left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | GPT-6 Astra Pro | OpenAI | self_reported

本轮未发现新的具体代码缺陷;暂为 REQUEST_CHANGES,原因是关键 provider / native Docker 执行证据仍未闭合。 审阅固定在d5f7060ef3e4cffad79252d32f09f5cf9a7f6a54,完整对比merge-base2e928fefe422d62dbec3a443c0c61e79a8f38752的6个文件,并追踪了CLI、原network isolation与provider来源冻结路径。这里不把“没有Docker宿主”写成作者的新bug,也不把受控边界模拟写成原生通过。

动机 / Motivation

受影响的是在Docker VM内运行EdgeBench runner、judge或凭证网关的实验操作者,以及需要同一网络条件比较Official与LoopX的评测维护者。

原先操作者只能接受host-gateway的地址选择;现在可以在新attempt显式选择默认bridge网关,由共同adapter向容器注入一致地址,并在receipt记录该选择。

源码与受控class边界对照确认默认路径不查询bridge、开启后只改目标别名、非法/多义网关在创建前拒绝;真实所选SForge/Harbor CLI和Docker端口隔离尚未取得本轮可核验执行证据。

本PR不改变任务、评分、token或时间预算、凭证隔离、反馈义务及防火墙策略,也不声称模型收益或支持任意自定义Docker network。

改动思路 / Approach

拓扑选择属于既有Docker adapter的环境输入,原生firewall继续拥有许可端点;不需要新的kernel或配置状态机。 保留默认关闭的局部参数、receipt和相伴测试;补可核验provider/原生网络证据即可,不新增自动发现、端口例外或通用框架。

该选项表达的是操作者才知道的事实:服务究竟在桌面宿主,还是Docker VM里的默认bridge上。系统看到一个bridge或识别出某种OS,都不能自动回答这一点。因此显式bool有合理性;也没有必要建立新的拓扑注册表、自动探测或绕过既有防火墙。一次受控的寻址选择,应继续由原adapter交给native harness。

具体改动 / Changes

关键代码讲解

  1. run.main 添加--docker-host-bridge,默认关闭,将同一参数传给Official与LoopX共用的RecordingDockerBackend;开启时才在既有runtime receipt写docker_host_bridge: true。原实验和默认receipt不补字段。
  2. RecordingDockerBackend.__init__ 仅新增一个默认False参数,并拒绝非bool。没有因为bridge存在、机器OS或provider可用就自动启用,也不改变OAuth、feedback、checkpoint和blind endpoint原配置。
  3. create_container 复制caller的extra_hosts,显式开启才读取Docker默认bridge的IPAM.Config。它解析地址、排除IPv6/loopback/unspecified/multicast,要求唯一IPv4;错误或多义配置在创建容器前拒绝。成功时只覆盖host.docker.internal,原OAuth地址、环境变量、资源与能力选项继续传给父类,caller原字典不被修改。
  4. 未改动的create_network_isolation继续决定允许端点和blind筛选;新增flag没有进入权限规则。地址pin并非网络可达或任务完成证明,receipt只记录选择。

体量为169新增、4删除:生产改动很小,主要新增量是69行定向单测、56行实际Docker双端口oracle和扩展CLI参数矩阵;README明确默认bridge、该别名对应的所有服务必须可达、匹配实验两臂一致、只在新attempt选择。测试与生产一起交付的边界合理,无需另拆成“最后补测试”的PR。

采用既有规格docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md,修订0ad08b15dd7331051c93566075561a390a9cd7e2。本次标签native-authority对应原生runner/任务/verifier/评分的权威,typed-observation对应结构化receipt;当前source没有增加评分或任务完成owner。它们也不要求本PR证明模型收益或整个S11/E1路线图。

对主干的风险 / Risks

最需要真实观察的是pin与native endpoint/firewall是否在所选provider及Docker环境里使用同一个地址;只看最终extra_hosts字典不足以证明这一点。新增Docker oracle设计上有价值:先确认同一宿主两个端口都能到达,再安装原native策略并确认仅允许端口可达,最后清理容器。这比只检查参数有更强回归敏感性,但本轮没有执行live Docker或防火墙操作。

本轮实际执行pytest -q benchmark/tests/test_sforge_host_bridge.py benchmark/tests/test_sforge_runtime.py,结果为 10通过、304跳过;新增9个bridge用例及扩展CLI矩阵均因环境缺SForge/Harbor而没有运行。缓存也没有这两个依赖,没有联网安装或调用模型。不能把10个既有通过项记为新功能已通过。

为独立检查默认关闭和失败前拒绝,我对不可变base/head的原RecordingDockerBackend AST class body做了有界执行,明确用stand-in替换缺失的SForge父类,没有伪造已安装provider:

边界 实际结果
base与head默认False、相同container输入 kwargs一致,均不查询network,保留host-gateway
head显式True、唯一IPv4加IPv6 只查询一次bridge,pin为IPv4,caller字典及其他选项保留
无gateway、仅IPv6、0.0.0.0、loopback、multicast、多义IPv4 6类均在父类创建容器前拒绝
非bool的1、None、字符串 拒绝,未执行provider初始化
原native与blind endpoint筛选,base/head对照 集合保持一致,不因flag扩大许可

这些结果验证局部源代码语义,不能替代真实CLI、所选SForge/Harbor、Docker元数据或netfilter资格。作者在PR中报告464项focused与实际Docker oracle通过;我保留这项报告,但本轮尚未核验可绑定当前head/provider revision的原始执行产物。已有合格日志或产物可以直接提供核验,无需重复已经证明的运行;如果没有,运行PR已经给出的定向CLI矩阵和无模型Docker oracle即可。

其他检查也如实记录:git diff --check返回2,仅指出新test文件第69行EOF多余空行,这是小型格式问题,不作为正确性阻塞finding。所读远端pytest聚合job只是检查上游CHECKS_RESULT/SHARDS_RESULT,没有自身测试失败栈;我不据聚合红灯制造本PR缺陷,也未完成所有远端失败归因。

语义与 CI 对齐

该变更是局部adapter配置,复用原endpoint和任务词汇;显式参数有独立环境含义,没有把指导意见变成模型工作义务。缺口是可核验的真实provider/网络观察,而非需要再加状态机、扩大端口白名单或放松现有断言。默认关闭、开启、无效拓扑和真实双端口拒绝应一起作为再审证据。

我的整体评价 / Verdict

设计与改动体量相称,默认关闭边界在source和受控base/head对照中清楚;没有发现要求重构或修复的具体生产缺陷。对长程工作和操作者体验的预期收益合理,但它们依赖尚未核验的实际服务可达性与原native隔离,因此不能现在声明整个能力合格。建议保留这份小范围实现,补上已有或新执行的精确head/provider证据后再批准。本次检查的 benchmark的Docker adapter、CLI参数、receipt与operator README 不改变已渲染的产品页面;无需用截图证明内部行为。 未进行产品页面视觉检查,因为本次没有页面改动。

English verdict: REQUEST_CHANGES — no new concrete code defect was reproduced at d5f7060ef3e4cffad79252d32f09f5cf9a7f6a54. The explicit default-off adapter design is proportionate, and bounded base/head class-body checks preserve the default path and fail before container creation on invalid gateways. The affected provider/CLI tests were skipped and native Docker/firewall qualification remains unverified; source-matched existing execution artifacts or the already supplied focused oracles are the remaining requirement.

@huangruiteng

Copy link
Copy Markdown
Collaborator

Rebased-by-merge onto current main in exact head cf9c48d47 and requalified the focused transport surface against the frozen SForge source used by the current EdgeBench campaign.

  • benchmark/tests/test_sforge_host_bridge.py
  • benchmark/tests/test_sforge_runtime.py
  • benchmark/tests/test_edgebench_feedback.py
  • benchmark/tests/test_edgebench_prompts.py

Result: 464 passed. The default installed SForge 1.1 package is intentionally insufficient for the current attestation fields; the qualifying run used the frozen reviewed SForge source rather than treating skipped package-only tests as coverage.

The updated exact head is also merged into codex/context-delivery-integration-20261030 so the maintained local integration branch now carries this benchmark transport fix together with the current context-delivery work.

@mergify

mergify Bot commented Oct 30, 2026

Copy link
Copy Markdown

Hi @loopx-agent, the DCO Sign-off check did not pass. Please inspect
its details first: checkout, fetch, timeout or infrastructure errors
need their own recovery, not a rewrite of otherwise signed commits.

If the log confirms a missing Signed-off-by trailer, amend the
affected commit with git commit --amend -s; for multiple commits,
use an interactive rebase against the current base from the correct
base-repository remote and sign off each affected commit. Push the
rewritten PR branch with git push --force-with-lease origin HEAD.

…r-host-bridge

Signed-off-by: huangruiteng <huangrt01@163.com>
@huangruiteng
huangruiteng force-pushed the codex/benchmark-docker-host-bridge branch from cf9c48d to 14f205a Compare October 30, 2026 21:04
Signed-off-by: huangruiteng <huangrt01@163.com>
@huangruiteng

Copy link
Copy Markdown
Collaborator

Exact head is now b5b489cf3.

  • Repaired the missing DCO trailer on the merge commit; all three PR commits now contain Signed-off-by.
  • Removed the reviewer-noted extra blank line at EOF. git diff --check origin/main...HEAD passes.
  • Replayed the focused suite at this exact head against frozen SForge 481791e: 464 passed.
  • Replayed the opt-in native Docker/netfilter oracle at this exact head: 1 passed (test_host_bridge_pin_reaches_only_the_native_allowed_port). This exercises the bridge pin with two real host ports, then verifies the native policy keeps the allowed port reachable and rejects the other port.
  • The first replay accidentally loaded an older installed SForge and produced 8 dependency-shape failures (runtime_attestation_key_file absent), with 451 passing and 5 skipped. Rebinding to the declared frozen SForge source removed all eight failures; that failed attempt is retained here as environment attribution rather than reported as a PR regression.
  • The targeted public-boundary scan of the touched test remains clean.

The last commit only normalizes EOF whitespace; runtime behavior is unchanged from the reviewed implementation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants