From 159de845c76559b587df56efb72be16dd9c052b3 Mon Sep 17 00:00:00 2001 From: guohaotian Date: Tue, 17 Nov 2026 18:34:36 +0800 Subject: [PATCH] fix(oga-hipep): no vision warm-up and no shared image budget on Qwen3.5 (WP-23) The vision warm-up and the shared 8100-patch budget worked around wrong image features in hip-ep 0.5.2 (onnxruntime-genai 0.17). FLM requires 0.19, where neither is needed. - Remove OgaEngine::WarmUpVision and the per-prompt patch limit (kMaxImagePatches, OgaVisionConfig::max_patches). - Size each image on its own, by --img-pre-resize and the package's max_pixels, so an image keeps its tokens when another one joins. - On OGA, return HTTP 400 at the image that makes the images alone reach the context, before the remaining images are preprocessed. - Document the per-image sizing, its measured times and the 400 in cli.md. --- docs/docs/instructions/cli.md | 2 +- src/common/AutoModel/modeling_qwen3_5vl.cpp | 29 ++++++-- src/common/oga/oga_engine.cpp | 37 +--------- src/include/oga/oga_engine.hpp | 5 -- src/include/oga/oga_vision.hpp | 9 --- src/test/oga_hipep/test_oga_hipep.cpp | 80 +++++++++------------ src/test/oga_hipep/test_qwen_frontend.cpp | 49 ++++++++++++- 7 files changed, 105 insertions(+), 106 deletions(-) diff --git a/docs/docs/instructions/cli.md b/docs/docs/instructions/cli.md index 9cdfe4415..acddfee4f 100644 --- a/docs/docs/instructions/cli.md +++ b/docs/docs/instructions/cli.md @@ -295,7 +295,7 @@ flm run qwen3.5-oga-hipep:9b - **Manual and offline use.** You can still unzip a package yourself into a folder under `%LOCALAPPDATA%\hip-ep` or `%ProgramFiles%\hip-ep`. The folder must hold `onnxruntime-genai.dll` (0.19 or later), `onnxruntime.dll` and `hipgpu.dll`, at its root or in a `bin` directory. `amdgpu-0.5.3.zip` already holds them in an `amdgpu-0.5.3` folder, so extract it to `%LOCALAPPDATA%\hip-ep` itself: Windows' Extract All suggests a destination named after the zip, which nests that folder one level too deep. The version may sit anywhere in the folder name, such as `amdgpu-0.5.3` or `hip-ep-0.5.3`; a folder without one, such as `amdgpu`, is tried after the versioned ones. Or set `HIPEP_PKG_ROOT` to the package root. When `HIPEP_PKG_ROOT` is set it is the only location used, and a package it points to that is rejected is an error, not a fallback to the default locations or to an automatic download. - **Uninstall.** `flm uninstall hipep` removes what the automatic install added: `%LOCALAPPDATA%\hip-ep\flm-` and the installer's files, including the cached zip, under `%LOCALAPPDATA%\hip-ep-install`. A package you unzipped yourself, any package under `%ProgramFiles%`, the package `HIPEP_PKG_ROOT` names, and every model stay; `flm remove ` removes a model. While a running `flm` has the runtime loaded for an oga-hipep model, the command removes nothing and says so: stop that process first. The next `run` or `serve` of an oga-hipep model installs hip-ep again when it finds no other package. - **Models.** `qwen3.5-oga-hipep` (`9b`, `35b-a3b`, `122b-a10b`), `qwen3.6-oga-hipep` (`27b`, `35b-a3b`), `qwen3.8-oga-hipep:27b` and `qwen3.8-flash-next-oga-hipep:125b-a6b` take images as well as text. `qwen2.5-it-oga-hipep:14b`, `qwen2.5-coder-it-oga-hipep:14b`, `gemma3-oga-hipep:4b`, `gemma4-it-oga-hipep` (`12b`, `26b-a4b`) and `gpt-oss-oga-hipep:120b` take text only: the Gemma packages ship a vision model, and `gemma4-it-oga-hipep:12b` an audio model too, but FLM does not run them, so a request with an image or audio gets HTTP 400. The packages are 4 to 124 GB each. -- **Images.** The images of one prompt share a budget of 8100 patches (one 1440x1440 image, 2025 image tokens), so a prompt with several images resizes each one smaller. The first image after the model loads takes a few seconds longer: FLM first runs the vision model once at that full size, so that every size after it gets correct image features (hip-ep 0.5.2 gave wrong ones without it). +- **Images.** Each image is resized on its own, as on the NPU: to at most the height `--img-pre-resize` sets (720 by default), within the package's pixel limit. Another image in the prompt does not make it smaller, so an image keeps its size and its tokens for the rest of the conversation: a 1280x720 screenshot is 882 tokens. The time of the images adds up. On a Radeon 8060S with `qwen3.5-oga-hipep:9b`, the first token of a prompt with 1280x720 screenshots comes after about 5 s for 3 of them, 8 s for 4, 10 s for 6 and 14 s for 8, and a conversation pays that again each time it gets a new image (see "Conversation state"). One large image costs more than the same pixels in several small ones: a 2880x2880 image at `--img-pre-resize 0` (original size) takes about 40 s, and a 4096x4096 image, or a 23300x720 panorama at the default setting, about 2 minutes. Images count toward the context length as text does, so the default 32768 tokens hold at most 37 such screenshots, and a prompt past the context length gets HTTP 400. - **The host must have an NPU.** FLM still opens the NPU for this backend, so a host without an NPU is not supported yet. - **Conversation state.** FLM keeps up to four conversations on the GPU, one OGA generator each, so that two chats, or an agent and its sub-agent, that take turns do not replay each other's history. A request goes to the generator holding the longest part of its prompt; with none, a new generator replays the whole prompt, in place of the least recently used one once all are in use. Exact token prefixes continue a generator, so a request does not run its tokens twice. For Qwen3.5, Qwen3.6, Qwen3.8 and gpt-oss, FLM also keeps a checkpoint at the end of each prompt, as on the NPU: a follow-up turn whose prompt starts with the previous prompt rewinds the generator to that checkpoint and runs only the rest, even when the chat template leaves the reasoning of earlier replies out, and the same prompt sent again (a retry, "regenerate") runs no prompt token at all. onnxruntime-genai 0.19 cannot rewind the Gemma 4 packages, so with thinking on a Gemma 4 follow-up replays the conversation; FLM warns at the first refused rewind and does not try again. Any other history, an edited earlier message, and every new or changed image replay the whole conversation (OGA takes images only when it starts a generator); `usage.prompt_tokens_details.cached_tokens` counts the tokens that were not run again. - **How many conversations.** `--prefix-slots N` sets how many generators to keep (default `4`, at most `32`; `0` keeps one, as before). Each holds the KV cache of the full context length: for `qwen3.5-oga-hipep:9b`, about 1.1 GiB at the default 32768 tokens and 4.1 GiB at 131072. The generators beyond the first must fit in `--prefix-cache-mb` (default `4096` MiB), so at a long `--ctx-len` FLM keeps fewer; the load log says how many (`OGA generators: up to N conversation(s) kept`). A new generator is made for each prompt that no kept one holds, and it stays until a later one replaces it, so after a few chats, edited messages or Gemma 4 thinking turns all of them hold their memory. On a machine with 32 GB or less, `--prefix-slots 2` or `--prefix-cache-mb 2048` leaves more for other programs. diff --git a/src/common/AutoModel/modeling_qwen3_5vl.cpp b/src/common/AutoModel/modeling_qwen3_5vl.cpp index 599ae14a1..39c98d572 100644 --- a/src/common/AutoModel/modeling_qwen3_5vl.cpp +++ b/src/common/AutoModel/modeling_qwen3_5vl.cpp @@ -227,13 +227,26 @@ bool Qwen3_5VL::_build_tokens(chat_meta_info_t& meta_info, lm_uniform_input_t& i "invalid_request_error"); } const bool oga = this->backend_id() == flm::backend::kOgaHipepBackendId; - unsigned max_pixels = this->vision_geometry.max_pixels; - if (oga && requested_images > 0) { - // OGA runs all the images of a prompt in one vision pass, so they share its patch budget. - const uint64_t patch_pixels = static_cast(this->vision_geometry.patch_size) * this->vision_geometry.patch_size; - const uint64_t share = static_cast(flm::oga::kMaxImagePatches) / requested_images * patch_pixels; - max_pixels = static_cast(std::min(max_pixels, share)); - } + // Each image is sized on its own, by --img-pre-resize and the model's max_pixels, so an + // image keeps its tokens when another one joins the conversation. + const unsigned max_pixels = this->vision_geometry.max_pixels; + // Without a shared budget, a prompt of many large images would be preprocessed whole before + // the context check refuses it, and preprocess_image counts their pixel values in 32 bits. + // On OGA, refuse once the images alone pass the context. + std::size_t image_tokens = 0; + std::size_t counted_images = 0; + const auto count_image_tokens = [&](const qwen3_5vl_image_t& image) { + if (!oga) return; + const std::size_t merge = this->vision_geometry.merge_size; + image_tokens += static_cast(image.grid_h) * image.grid_w / (merge * merge); + ++counted_images; + const std::size_t cap = this->decode_cap(); + if (image_tokens >= cap) + throw ModelRequestError(400, false, + "Request exceeds the " + std::to_string(cap) + "-token limit of backend '" + this->backend_id() + + "': the first " + std::to_string(counted_images) + " image(s) alone have " + + std::to_string(image_tokens) + " image tokens"); + }; // preprocess constexpr int image_soft_token_id = 248056; @@ -264,6 +277,7 @@ bool Qwen3_5VL::_build_tokens(chat_meta_info_t& meta_info, lm_uniform_input_t& i header_print("ERROR", "Skipping image that failed to preprocess: " << img_str); continue; } + count_image_tokens(image); // Push the image AFTER preprocessing so grid_h and grid_w are set image_payload.images.push_back(image); image_payload.num_images++; @@ -298,6 +312,7 @@ bool Qwen3_5VL::_build_tokens(chat_meta_info_t& meta_info, lm_uniform_input_t& i header_print("ERROR", "Skipping image that failed to preprocess"); continue; } + count_image_tokens(image); image_payload.images.push_back(image); image_payload.num_images++; diff --git a/src/common/oga/oga_engine.cpp b/src/common/oga/oga_engine.cpp index e4501229a..c7203a882 100644 --- a/src/common/oga/oga_engine.cpp +++ b/src/common/oga/oga_engine.cpp @@ -8,8 +8,6 @@ #include #include -#include -#include #include #include #include @@ -140,7 +138,7 @@ OgaEngine::OgaEngine(std::shared_ptr api, OgaApi::Owned }; if (!in_vocabulary(v.image_token_id) || !in_vocabulary(v.vision_start_token_id) || v.image_token_id == v.vision_start_token_id || v.spatial_merge_size <= 0 || - v.patch_values <= 0 || v.max_patches < v.spatial_merge_size * v.spatial_merge_size) + v.patch_values <= 0) throw std::invalid_argument(display_name_ + " vision configuration is invalid"); } const OgaFunctions& f = api_->functions(); @@ -315,9 +313,6 @@ std::vector OgaEngine::ImageSpans(const std::vector& const OgaVisionConfig& config = *vision_; const std::size_t images = vision ? vision->images.size() : 0; const std::int64_t merged = config.spatial_merge_size * config.spatial_merge_size; - std::int64_t patches = 0; - for (const ImageSpan& span : images_) - patches += span.image->grid_t * span.image->grid_h * span.image->grid_w; for (std::size_t i = 0; i < fresh.size();) { if (fresh[i] != config.image_token_id) { ++i; @@ -344,11 +339,6 @@ std::vector OgaEngine::ImageSpans(const std::vector& throw std::invalid_argument(which + " has " + std::to_string(grid / merged) + " image tokens, but its run in the prompt has " + std::to_string(end - i)); - patches += grid; - if (patches > config.max_patches) - throw std::invalid_argument(display_name_ + " prompt images have " + std::to_string(patches) + - " patches, more than the " + std::to_string(config.max_patches) + - " the vision model runs at once"); spans.push_back({offset + i, end - i, std::make_shared(image), ImageHash(image)}); i = end; } @@ -364,7 +354,6 @@ bool OgaEngine::ImageAfter(std::size_t position) const { std::pair OgaEngine::Append(const char** why) { active_.reset(); // so that a failure before Resume drops no other conversation's slot - if (!images_.empty() && !vision_warm_) WarmUpVision(); const Reuse reuse = Resume(why); Live& run = Active(); const std::size_t from = run.ran.size(); @@ -546,30 +535,6 @@ void OgaEngine::SetInputs(OgaGenerator* generator, const std::vector& ids, api_->Check(f.generator_set_inputs(generator, named.get()), "OgaGenerator_SetInputs"); } -void OgaEngine::WarmUpVision() { - const OgaVisionConfig& config = *vision_; - auto side = static_cast(std::sqrt(static_cast(config.max_patches))); - while (side * side > config.max_patches) --side; - side -= side % config.spatial_merge_size; - OgaImage image; - image.grid_h = side; - image.grid_w = side; - image.pixels.assign(static_cast(side * side * config.patch_values), 0); - std::vector ids(static_cast( - 1 + side * side / (config.spatial_merge_size * config.spatial_merge_size)), - config.image_token_id); - ids.front() = config.vision_start_token_id; - const auto start = std::chrono::steady_clock::now(); - const OgaApi::Owned generator = NewGenerator(static_cast(ids.size())); - SetInputs(generator.get(), ids, {&image}); - Logits(generator.get()); - vision_warm_ = true; - const auto ms = std::chrono::duration_cast( - std::chrono::steady_clock::now() - start).count(); - header_print("FLM", "OGA vision: warmed up with a " << side << "x" << side << "-patch image in " - << ms << " ms"); -} - OgaApi::Owned OgaEngine::NewTensor(const std::vector& shape, OgaElementType type, const void* data, std::size_t bytes) const { diff --git a/src/include/oga/oga_engine.hpp b/src/include/oga/oga_engine.hpp index eb9e1cfb5..201dafe81 100644 --- a/src/include/oga/oga_engine.hpp +++ b/src/include/oga/oga_engine.hpp @@ -28,8 +28,6 @@ struct OgaVisionConfig { std::int64_t spatial_merge_size = 0; /// \brief values in one patch row of pixel_values std::int64_t patch_values = 0; - /// \brief the most patches the images of one prompt may have together - std::int64_t max_patches = kMaxImagePatches; }; /// \brief how many generators an engine keeps, one per conversation, and the memory they may take @@ -213,8 +211,6 @@ class OgaEngine final : public causal_lm { /// \note No images: the batch of one image without pixels (empty_image_batch) void SetInputs(OgaGenerator* generator, const std::vector& ids, const std::vector& images) const; - /// \brief run the vision model once at max_patches, before the first real image - void WarmUpVision(); /// \brief a tensor that OGA owns, with `bytes` of `data` copied in when given OgaApi::Owned NewTensor(const std::vector& shape, OgaElementType type, const void* data, @@ -233,7 +229,6 @@ class OgaEngine final : public causal_lm { std::optional vision_; bool empty_image_batch_ = false; bool rewinds_ = false; - bool vision_warm_ = false; OgaSlotLimits limits_; std::uint64_t replayed_ = 0; std::uint64_t reused_ = 0; diff --git a/src/include/oga/oga_vision.hpp b/src/include/oga/oga_vision.hpp index 9076fee45..990588285 100644 --- a/src/include/oga/oga_vision.hpp +++ b/src/include/oga/oga_vision.hpp @@ -12,15 +12,6 @@ namespace flm::oga { -/// \brief the most patches the images of one prompt may have together -/// \note OGA runs every image of a prompt in one pass of the vision model. On -/// hip-ep 0.5.2 (OGA 0.17.0) that pass gives wrong image features when it -/// has many more patches than the first pass the model ran, or when that -/// first pass had only a few hundred. The engine therefore runs one pass of -/// this size before its first real image, and no prompt may have more. -/// 8100 patches is a 90x90 grid (1440x1440 pixels), 2025 image tokens. -inline constexpr std::int64_t kMaxImagePatches = 90 * 90; - /// \brief one image, preprocessed for the vision model struct OgaImage { std::int64_t grid_t = 1; diff --git a/src/test/oga_hipep/test_oga_hipep.cpp b/src/test/oga_hipep/test_oga_hipep.cpp index f19d3bb27..06bd75446 100644 --- a/src/test/oga_hipep/test_oga_hipep.cpp +++ b/src/test/oga_hipep/test_oga_hipep.cpp @@ -471,14 +471,13 @@ std::unique_ptr MakeEngineOn(flm::oga::FileVersion version, return ConstructOn(FakeApi(std::move(symbols), {"test", {}, version})); } -/// \brief 2x2 patches to an image token, 4 values to a patch, 64 patches to a prompt +/// \brief 2x2 patches to an image token, 4 values to a patch flm::oga::OgaVisionConfig VisionConfig() { flm::oga::OgaVisionConfig config; config.image_token_id = kImageToken; config.vision_start_token_id = kVisionStart; config.spatial_merge_size = 2; config.patch_values = kPatchValues; - config.max_patches = 64; return config; } @@ -1088,30 +1087,22 @@ void TestTheSlotCountFitsTheBudget() { TEST_REQUIRE(g_fake.last_max_length == 32); } -void TestAnImagePromptRunsInOneSetInputsAfterAWarmUp() { +void TestAnImagePromptRunsInOneSetInputs() { auto engine = MakeVisionEngine(); const auto image = Image(4, 4); flm::oga::OgaVisionInput vision{{image}}; std::vector prompt = WithImage({1}, image, {2}); RequireRan(engine->prefill(prompt, &vision), prompt); - TEST_REQUIRE(g_fake.set_inputs.size() == 2); - // the warm-up: blank pixels, at the most patches a prompt may have - const auto& warm = g_fake.set_inputs[0]; - TEST_REQUIRE(warm.ids.size() == 17); - TEST_REQUIRE(warm.ids.front() == kVisionStart); - TEST_REQUIRE(std::count(warm.ids.begin(), warm.ids.end(), kImageToken) == 16); - TEST_REQUIRE(warm.pixel_shape == (std::vector{64, kPatchValues})); - TEST_REQUIRE(std::all_of(warm.pixels.begin(), warm.pixels.end(), [](std::uint16_t v) { return v == 0; })); - TEST_REQUIRE(warm.grids == (std::vector{1, 8, 8})); - TEST_REQUIRE(warm.image_tokens == (std::vector{16})); - const auto& run = g_fake.set_inputs[1]; + // no warm-up pass before it: OGA 0.19 gives the same image features without one + TEST_REQUIRE(g_fake.set_inputs.size() == 1); + const auto& run = g_fake.set_inputs[0]; TEST_REQUIRE(run.ids == prompt); TEST_REQUIRE(run.pixel_shape == (std::vector{16, kPatchValues})); TEST_REQUIRE(run.pixels == image.pixels); TEST_REQUIRE(run.grids == (std::vector{1, 4, 4})); TEST_REQUIRE(run.image_tokens == (std::vector{4})); TEST_REQUIRE(g_fake.appends.empty()); - TEST_REQUIRE(g_fake.generators_created == 2); + TEST_REQUIRE(g_fake.generators_created == 1); TEST_REQUIRE(g_fake.generators_live == 1); // what follows the image continues the same generator @@ -1119,7 +1110,7 @@ void TestAnImagePromptRunsInOneSetInputsAfterAWarmUp() { RequireRan(engine->prefill(more), Concat(prompt, more)); RequireRan(engine->forward(5), Concat(prompt, {3, 4, 5})); TEST_REQUIRE(g_fake.appends == (std::vector{{3, 4}, {5}})); - TEST_REQUIRE(g_fake.generators_created == 2); + TEST_REQUIRE(g_fake.generators_created == 1); TEST_REQUIRE(engine->replayed_tokens() == 0); } @@ -1147,8 +1138,8 @@ void TestANewImageReplaysTheHistoryWithEveryImage() { std::vector turn2 = WithImage({4}, second, {5}); const auto all = Concat(Concat(turn1, {3}), turn2); RequireRan(engine->prefill(turn2, &two), all); - // the warm-up, the first turn and the replay: the vision model warms up once - TEST_REQUIRE(g_fake.set_inputs.size() == 3); + // the first turn and the replay + TEST_REQUIRE(g_fake.set_inputs.size() == 2); const auto& replay = g_fake.set_inputs.back(); TEST_REQUIRE(replay.ids == all); TEST_REQUIRE(replay.grids == (std::vector{1, 4, 4, 1, 2, 4})); @@ -1174,7 +1165,7 @@ void TestARestoredCheckpointBringsBackItsImages() { std::vector turn{5}; const auto expected = Concat(prompt, turn); RequireRan(engine->prefill(turn), expected); - TEST_REQUIRE(g_fake.set_inputs.size() == 3); + TEST_REQUIRE(g_fake.set_inputs.size() == 2); TEST_REQUIRE(g_fake.set_inputs.back().ids == expected); TEST_REQUIRE(g_fake.set_inputs.back().pixels == image.pixels); TEST_REQUIRE(engine->replayed_tokens() == prompt.size()); @@ -1183,7 +1174,7 @@ void TestARestoredCheckpointBringsBackItsImages() { engine->clear_context(); std::vector text{1, 2}; RequireRan(engine->prefill(text), text); - TEST_REQUIRE(g_fake.set_inputs.size() == 3); + TEST_REQUIRE(g_fake.set_inputs.size() == 2); TEST_REQUIRE(g_fake.appends.back() == (Tokens{1, 2})); } @@ -1217,7 +1208,7 @@ void TestAnotherImageOfTheSameSizeRebuildsTheGenerator() { engine->clear_context(); std::vector again = Concat(prompt, {3, 4}); RequireRan(engine->prefill(again, &two), again); - TEST_REQUIRE(g_fake.set_inputs.size() == 3); + TEST_REQUIRE(g_fake.set_inputs.size() == 2); TEST_REQUIRE(g_fake.set_inputs.back().pixels == second.pixels); TEST_REQUIRE(g_fake.generators_live == 1); TEST_REQUIRE(engine->replayed_tokens() == 0); @@ -1227,7 +1218,7 @@ void TestAnotherImageOfTheSameSizeRebuildsTheGenerator() { flm::oga::OgaVisionInput copy{{second}}; std::vector more = Concat(again, {5}); RequireRan(engine->prefill(more, ©), more); - TEST_REQUIRE(g_fake.set_inputs.size() == 3); + TEST_REQUIRE(g_fake.set_inputs.size() == 2); TEST_REQUIRE(g_fake.appends.back() == (Tokens{5})); } @@ -1273,24 +1264,22 @@ void TestImageRunsAndImagesMustAgree() { TEST_REQUIRE(engine->get_current_context_length() == 0); } -void TestTheImagesOfAPromptShareThePatchBudget() { - auto engine = MakeVisionEngine(); - const auto big = Image(6, 6); +void TestTheImagesOfAPromptHaveNoSharedPatchBudget() { + g_fake = FakeOga{}; + auto engine = ConstructOn(FakeApi(FakeSymbols(), {"test", {}, {0, 19, 0, 0}}), 4096, 4096, VisionConfig()); + // 4096 patches each, 8192 in one vision pass + const auto big = Image(64, 64); flm::oga::OgaVisionInput one{{big}}; std::vector turn1 = WithImage({1}, big, {2}); engine->prefill(turn1, &one); - // the earlier image counts too: a rebuilt generator runs both at once + // a second image rebuilds the generator with both, each as it came std::vector turn2 = WithImage({3}, big, {4}); - RequireContains(RequireThrows([&] { engine->prefill(turn2, &one); }), - "prompt images have 72 patches, more than the 64"); - TEST_REQUIRE(engine->get_current_context_length() == static_cast(turn1.size())); - TEST_REQUIRE(g_fake.set_inputs.size() == 2); - - engine->clear_context(); - flm::oga::OgaVisionInput two{{Image(4, 4), Image(6, 6)}}; - std::vector both = WithImage(WithImage({1}, two.images[0], {2}), two.images[1], {3}); - RequireRan(engine->prefill(both, &two), both); - TEST_REQUIRE(g_fake.set_inputs.back().image_tokens == (std::vector{4, 9})); + engine->prefill(turn2, &one); + // (the fake's bf16 logits cannot count this many tokens exactly: check the inputs) + TEST_REQUIRE(engine->get_current_context_length() == static_cast(turn1.size() + turn2.size())); + TEST_REQUIRE(g_fake.set_inputs.back().ids == Concat(turn1, turn2)); + TEST_REQUIRE(g_fake.set_inputs.back().image_tokens == (std::vector{1024, 1024})); + TEST_REQUIRE(g_fake.set_inputs.back().pixel_shape == (std::vector{8192, kPatchValues})); } void TestSetContextLengthKeepsWholeImagesOnly() { @@ -1331,7 +1320,10 @@ void TestAnInvalidVisionConfigurationIsRefused() { config.vision_start_token_id = config.image_token_id; RequireThrows([&] { ConstructOn(FakeApi(), 32, 64, config); }); config = VisionConfig(); - config.max_patches = 3; + config.spatial_merge_size = 0; + RequireThrows([&] { ConstructOn(FakeApi(), 32, 64, config); }); + config = VisionConfig(); + config.patch_values = 0; RequireThrows([&] { ConstructOn(FakeApi(), 32, 64, config); }); } @@ -2488,12 +2480,10 @@ void TestTheBackendRunsThePackagesImages() { flm::oga::OgaVisionInput vision{{image}}; std::vector prompt = WithImage({1}, image, {2}); backend->engine().prefill(prompt, &vision); - TEST_REQUIRE(g_fake.set_inputs.size() == 2); - // the package's warm-up is the full budget: 90x90 patches - TEST_REQUIRE(g_fake.set_inputs[0].grids == (std::vector{1, 90, 90})); - TEST_REQUIRE(g_fake.set_inputs[0].pixel_shape == (std::vector{8100, 4})); - TEST_REQUIRE(g_fake.set_inputs[1].ids == prompt); - TEST_REQUIRE(g_fake.set_inputs[1].pixel_shape == (std::vector{4, 4})); + // the image runs in the prompt's own SetInputs, with no warm-up before it + TEST_REQUIRE(g_fake.set_inputs.size() == 1); + TEST_REQUIRE(g_fake.set_inputs[0].ids == prompt); + TEST_REQUIRE(g_fake.set_inputs[0].pixel_shape == (std::vector{4, 4})); TEST_REQUIRE(backend->replayed_tokens() == 0); } @@ -2674,7 +2664,7 @@ int main(int argc, char** argv) { RUN(TestAChunkedPromptReachesALongerSlot); RUN(TestAnOgaErrorDropsOnlyItsSlot); RUN(TestTheSlotCountFitsTheBudget); - RUN(TestAnImagePromptRunsInOneSetInputsAfterAWarmUp); + RUN(TestAnImagePromptRunsInOneSetInputs); RUN(TestAnImageRunCanStartAChunk); RUN(TestANewImageReplaysTheHistoryWithEveryImage); RUN(TestARestoredCheckpointBringsBackItsImages); @@ -2682,7 +2672,7 @@ int main(int argc, char** argv) { RUN(TestAnotherImageOfTheSameSizeRebuildsTheGenerator); RUN(TestOga019DoesNotRewindToAnotherImageOfTheSameSize); RUN(TestImageRunsAndImagesMustAgree); - RUN(TestTheImagesOfAPromptShareThePatchBudget); + RUN(TestTheImagesOfAPromptHaveNoSharedPatchBudget); RUN(TestSetContextLengthKeepsWholeImagesOnly); RUN(TestAPackageWithoutVisionTakesNoImages); RUN(TestAnInvalidVisionConfigurationIsRefused); diff --git a/src/test/oga_hipep/test_qwen_frontend.cpp b/src/test/oga_hipep/test_qwen_frontend.cpp index f699f206d..d53c0f313 100644 --- a/src/test/oga_hipep/test_qwen_frontend.cpp +++ b/src/test/oga_hipep/test_qwen_frontend.cpp @@ -619,9 +619,9 @@ void OgaImagesReachTheEngineAsFloat16() { auto input = ImageInput(2); Require(model.insert(meta, input), "image prompt insert failed"); - // OGA runs both in one vision pass of at most 8100 patches of 16x16 pixels - Require(g_max_pixels == (std::vector{4050 * 256, 4050 * 256}), - "the images of a prompt must share the vision model's patch budget"); + // each image is sized on its own by the package's max_pixels: another image does not shrink it + Require(g_max_pixels == (std::vector{16777216, 16777216}), + "each image of a prompt must keep the package's max_pixels"); const FakeEngine& engine = model.engine(); Require(engine.vision.images.size() == 2, "both images must reach the engine"); const std::size_t values = 4 * 6 * 3 * 2 * 16 * 16; @@ -655,6 +655,47 @@ void ImagesAreRefusedWithoutImagePreprocessing() { throw std::runtime_error("an engine without image preprocessing must refuse images"); } +void OgaImagesPastTheContextAreRefusedBeforeAllArePreprocessed() { + Reset({10, kVisionStart, kImagePad, kVisionEnd, 12, kThinkStart, 271, kThinkEnd, 271}, {7}); + g_grid_h = 4; + g_grid_w = 6; // 6 image tokens each + Harness model(16, 64); // decode cap 15 + model.set_vision_geometry(Geometry()); + auto meta = Meta(); + auto input = ImageInput(5); + try { + model.insert(meta, input); + } catch (const ModelRequestError& error) { + Require(error.http_code() == 400, "images past the context must be an HTTP 400"); + // 6, 12, then 18 tokens: the third image passes the cap, the last two are never preprocessed + Require(g_max_pixels.size() == 3, "preprocessing must stop at the image that passes the context"); + Require(model.engine().prefills.empty(), "nothing may run for a refused prompt"); + return; + } + throw std::runtime_error("images past the context must be refused"); +} + +void OgaImagesAreRefusedAtTheImageThatReachesTheContext() { + // 6 image tokens each. Decode cap 12: the second image reaches it. Cap 13: only the third passes it. + for (const auto& [max_length, preprocessed] : {std::pair{13, 2}, {14, 3}}) { + Reset({10, kVisionStart, kImagePad, kVisionEnd, 12, kThinkStart, 271, kThinkEnd, 271}, {7}); + g_grid_h = 4; + g_grid_w = 6; + Harness model(max_length, 64); + model.set_vision_geometry(Geometry()); + auto meta = Meta(); + auto input = ImageInput(3); + bool refused = false; + try { + model.insert(meta, input); + } catch (const ModelRequestError& error) { + refused = error.http_code() == 400; + } + Require(refused, "images that reach the context must be an HTTP 400"); + Require(g_max_pixels.size() == preprocessed, "preprocessing must stop at the image that reaches the context"); + } +} + void MorePlaceholdersThanImagesIsARequestError() { Reset({10, kVisionStart, kImagePad, kVisionEnd, kVisionStart, kImagePad, kVisionEnd, 12, kThinkStart, 271, kThinkEnd, 271}, @@ -874,6 +915,8 @@ int main() { run("a second continuation reuses the first", ASecondContinuationReusesTheFirst); run("OGA images reach the engine as float16", OgaImagesReachTheEngineAsFloat16); run("images refused without preprocessing", ImagesAreRefusedWithoutImagePreprocessing); + run("OGA images past the context are refused early", OgaImagesPastTheContextAreRefusedBeforeAllArePreprocessed); + run("OGA images are refused at the image that reaches the context", OgaImagesAreRefusedAtTheImageThatReachesTheContext); run("more placeholders than images", MorePlaceholdersThanImagesIsARequestError); run("another image of the same size runs the prompt again", AnotherImageOfTheSameSizeRunsTheWholePromptAgain); run("the same image prompt again carries its image", TheSameImagePromptAgainCarriesItsImage);