Repository navigation
benchmarks: move adaptive prefill calibration to Examples - #1018
SuperGoodGame wants to merge 4 commits into
Conversation
Signed-off-by: SuperGoodGame <985236470@qq.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dcdde0bf7f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: SuperGoodGame <985236470@qq.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79b2140eaf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: SuperGoodGame <985236470@qq.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: beb3401dce
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: SuperGoodGame <985236470@qq.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3fd685fc55
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 'allow_tf32', 'source_sha256'): | ||
| if calibration['metadata'].get(key) != metadata[key]: | ||
| parser.error(f'calibration differs in {key}; recalibrate for this configuration') | ||
| costs = calibration.get('costs', {}) |
There was a problem hiding this comment.
When --cost-config points to an otherwise matching calibration artifact whose costs member is absent or has lost one of the five emitted coefficients, this fallback silently supplies HybridEngineRolloutConfig defaults for the missing values and the sweep completes with current metadata as though the calibrated policy was evaluated. Since --calibrate always emits the complete map (and hand-tuned partial configurations are documented for direct application use), validate the required cost keys before running the sweep.
Useful? React with 👍 / 👎.
1 participant
Adds adaptive prefill calibration and evaluation tooling alongside the existing rollout benchmark in
benchmarks/opsd. Requires Transformers >=4.56 and DeepSpeed PR #8786; the README explains configuration loading, the short-context calibration range and deployment validation.Calibration derives ordinary MHA KV heads from query heads when the dedicated field is absent. The CLI requires a complete metadata object for supplied calibration files; cost-only artifacts are rejected, while applications can still pass hand-tuned coefficients directly to the rollout configuration. Saved configurations are checked against the model/device/library metadata and a combined source hash of rollout, bucket planning and static-cache implementations; stale or missing hashes require recalibration.
Validation: three regression tests pass, covering non-negative fitting, real tiny-OPT calibration on a GPU, and CLI acceptance/rejection of matching, changed, missing and cache-only-changed source hashes. Six targeted GPU oracle checks in the DeepSpeed companion validate OPT/Qwen generation through the static cache. Python lint and diff checks pass. The original seeded tiny-Qwen2 calibration/sweep smoke check also completed all four workloads with matching output IDs on every path.