Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/instructions/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,7 @@ flm run qwen3.5-oga-hipep:9b
- **Manual and offline use.** You can still unzip a package yourself into a folder under `%LOCALAPPDATA%\hip-ep` or `%ProgramFiles%\hip-ep`. The folder must hold `onnxruntime-genai.dll` (0.19 or later), `onnxruntime.dll` and `hipgpu.dll`, at its root or in a `bin` directory. `amdgpu-0.5.3.zip` already holds them in an `amdgpu-0.5.3` folder, so extract it to `%LOCALAPPDATA%\hip-ep` itself: Windows' Extract All suggests a destination named after the zip, which nests that folder one level too deep. The version may sit anywhere in the folder name, such as `amdgpu-0.5.3` or `hip-ep-0.5.3`; a folder without one, such as `amdgpu`, is tried after the versioned ones. Or set `HIPEP_PKG_ROOT` to the package root. When `HIPEP_PKG_ROOT` is set it is the only location used, and a package it points to that is rejected is an error, not a fallback to the default locations or to an automatic download.
- **Uninstall.** `flm uninstall hipep` removes what the automatic install added: `%LOCALAPPDATA%\hip-ep\flm-<version>` and the installer's files, including the cached zip, under `%LOCALAPPDATA%\hip-ep-install`. A package you unzipped yourself, any package under `%ProgramFiles%`, the package `HIPEP_PKG_ROOT` names, and every model stay; `flm remove <model_tag>` removes a model. While a running `flm` has the runtime loaded for an oga-hipep model, the command removes nothing and says so: stop that process first. The next `run` or `serve` of an oga-hipep model installs hip-ep again when it finds no other package.
- **Models.** `qwen3.5-oga-hipep` (`9b`, `35b-a3b`, `122b-a10b`), `qwen3.6-oga-hipep` (`27b`, `35b-a3b`), `qwen3.8-oga-hipep:27b` and `qwen3.8-flash-next-oga-hipep:125b-a6b` take images as well as text. `qwen2.5-it-oga-hipep:14b`, `qwen2.5-coder-it-oga-hipep:14b`, `gemma3-oga-hipep:4b`, `gemma4-it-oga-hipep` (`12b`, `26b-a4b`) and `gpt-oss-oga-hipep:120b` take text only: the Gemma packages ship a vision model, and `gemma4-it-oga-hipep:12b` an audio model too, but FLM does not run them, so a request with an image or audio gets HTTP 400. The packages are 4 to 124 GB each.
- **Images.** The images of one prompt share a budget of 8100 patches (one 1440x1440 image, 2025 image tokens), so a prompt with several images resizes each one smaller. The first image after the model loads takes a few seconds longer: FLM first runs the vision model once at that full size, so that every size after it gets correct image features (hip-ep 0.5.2 gave wrong ones without it).
- **Images.** Each image is resized on its own, as on the NPU: to at most the height `--img-pre-resize` sets (720 by default), within the package's pixel limit. Another image in the prompt does not make it smaller, so an image keeps its size and its tokens for the rest of the conversation: a 1280x720 screenshot is 882 tokens. The time of the images adds up. On a Radeon 8060S with `qwen3.5-oga-hipep:9b`, the first token of a prompt with 1280x720 screenshots comes after about 5 s for 3 of them, 8 s for 4, 10 s for 6 and 14 s for 8, and a conversation pays that again each time it gets a new image (see "Conversation state"). One large image costs more than the same pixels in several small ones: a 2880x2880 image at `--img-pre-resize 0` (original size) takes about 40 s, and a 4096x4096 image, or a 23300x720 panorama at the default setting, about 2 minutes. Images count toward the context length as text does, so the default 32768 tokens hold at most 37 such screenshots, and a prompt past the context length gets HTTP 400.
- **The host must have an NPU.** FLM still opens the NPU for this backend, so a host without an NPU is not supported yet.
- **Conversation state.** FLM keeps up to four conversations on the GPU, one OGA generator each, so that two chats, or an agent and its sub-agent, that take turns do not replay each other's history. A request goes to the generator holding the longest part of its prompt; with none, a new generator replays the whole prompt, in place of the least recently used one once all are in use. Exact token prefixes continue a generator, so a request does not run its tokens twice. For Qwen3.5, Qwen3.6, Qwen3.8 and gpt-oss, FLM also keeps a checkpoint at the end of each prompt, as on the NPU: a follow-up turn whose prompt starts with the previous prompt rewinds the generator to that checkpoint and runs only the rest, even when the chat template leaves the reasoning of earlier replies out, and the same prompt sent again (a retry, "regenerate") runs no prompt token at all. onnxruntime-genai 0.19 cannot rewind the Gemma 4 packages, so with thinking on a Gemma 4 follow-up replays the conversation; FLM warns at the first refused rewind and does not try again. Any other history, an edited earlier message, and every new or changed image replay the whole conversation (OGA takes images only when it starts a generator); `usage.prompt_tokens_details.cached_tokens` counts the tokens that were not run again.
- **How many conversations.** `--prefix-slots N` sets how many generators to keep (default `4`, at most `32`; `0` keeps one, as before). Each holds the KV cache of the full context length: for `qwen3.5-oga-hipep:9b`, about 1.1 GiB at the default 32768 tokens and 4.1 GiB at 131072. The generators beyond the first must fit in `--prefix-cache-mb` (default `4096` MiB), so at a long `--ctx-len` FLM keeps fewer; the load log says how many (`OGA generators: up to N conversation(s) kept`). A new generator is made for each prompt that no kept one holds, and it stays until a later one replaces it, so after a few chats, edited messages or Gemma 4 thinking turns all of them hold their memory. On a machine with 32 GB or less, `--prefix-slots 2` or `--prefix-cache-mb 2048` leaves more for other programs.
Expand Down
29 changes: 22 additions & 7 deletions src/common/AutoModel/modeling_qwen3_5vl.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -227,13 +227,26 @@ bool Qwen3_5VL::_build_tokens(chat_meta_info_t& meta_info, lm_uniform_input_t& i
"invalid_request_error");
}
const bool oga = this->backend_id() == flm::backend::kOgaHipepBackendId;
unsigned max_pixels = this->vision_geometry.max_pixels;
if (oga && requested_images > 0) {
// OGA runs all the images of a prompt in one vision pass, so they share its patch budget.
const uint64_t patch_pixels = static_cast<uint64_t>(this->vision_geometry.patch_size) * this->vision_geometry.patch_size;
const uint64_t share = static_cast<uint64_t>(flm::oga::kMaxImagePatches) / requested_images * patch_pixels;
max_pixels = static_cast<unsigned>(std::min<uint64_t>(max_pixels, share));
}
// Each image is sized on its own, by --img-pre-resize and the model's max_pixels, so an
// image keeps its tokens when another one joins the conversation.
const unsigned max_pixels = this->vision_geometry.max_pixels;
// Without a shared budget, a prompt of many large images would be preprocessed whole before
// the context check refuses it, and preprocess_image counts their pixel values in 32 bits.
// On OGA, refuse once the images alone pass the context.
std::size_t image_tokens = 0;
std::size_t counted_images = 0;
const auto count_image_tokens = [&](const qwen3_5vl_image_t& image) {
if (!oga) return;
const std::size_t merge = this->vision_geometry.merge_size;
image_tokens += static_cast<std::size_t>(image.grid_h) * image.grid_w / (merge * merge);
++counted_images;
const std::size_t cap = this->decode_cap();
if (image_tokens >= cap)
throw ModelRequestError(400, false,
"Request exceeds the " + std::to_string(cap) + "-token limit of backend '" + this->backend_id() +
"': the first " + std::to_string(counted_images) + " image(s) alone have " +
std::to_string(image_tokens) + " image tokens");
};

// preprocess
constexpr int image_soft_token_id = 248056;
Expand Down Expand Up @@ -264,6 +277,7 @@ bool Qwen3_5VL::_build_tokens(chat_meta_info_t& meta_info, lm_uniform_input_t& i
header_print("ERROR", "Skipping image that failed to preprocess: " << img_str);
continue;
}
count_image_tokens(image);
// Push the image AFTER preprocessing so grid_h and grid_w are set
image_payload.images.push_back(image);
image_payload.num_images++;
Expand Down Expand Up @@ -298,6 +312,7 @@ bool Qwen3_5VL::_build_tokens(chat_meta_info_t& meta_info, lm_uniform_input_t& i
header_print("ERROR", "Skipping image that failed to preprocess");
continue;
}
count_image_tokens(image);

image_payload.images.push_back(image);
image_payload.num_images++;
Expand Down
37 changes: 1 addition & 36 deletions src/common/oga/oga_engine.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,6 @@

#include <algorithm>
#include <bit>
#include <chrono>
#include <cmath>
#include <cstring>
#include <iterator>
#include <limits>
Expand Down Expand Up @@ -140,7 +138,7 @@ OgaEngine::OgaEngine(std::shared_ptr<const OgaApi> api, OgaApi::Owned<OgaModel>
};
if (!in_vocabulary(v.image_token_id) || !in_vocabulary(v.vision_start_token_id) ||
v.image_token_id == v.vision_start_token_id || v.spatial_merge_size <= 0 ||
v.patch_values <= 0 || v.max_patches < v.spatial_merge_size * v.spatial_merge_size)
v.patch_values <= 0)
throw std::invalid_argument(display_name_ + " vision configuration is invalid");
}
const OgaFunctions& f = api_->functions();
Expand Down Expand Up @@ -315,9 +313,6 @@ std::vector<OgaEngine::ImageSpan> OgaEngine::ImageSpans(const std::vector<int>&
const OgaVisionConfig& config = *vision_;
const std::size_t images = vision ? vision->images.size() : 0;
const std::int64_t merged = config.spatial_merge_size * config.spatial_merge_size;
std::int64_t patches = 0;
for (const ImageSpan& span : images_)
patches += span.image->grid_t * span.image->grid_h * span.image->grid_w;
for (std::size_t i = 0; i < fresh.size();) {
if (fresh[i] != config.image_token_id) {
++i;
Expand All @@ -344,11 +339,6 @@ std::vector<OgaEngine::ImageSpan> OgaEngine::ImageSpans(const std::vector<int>&
throw std::invalid_argument(which + " has " + std::to_string(grid / merged) +
" image tokens, but its run in the prompt has " +
std::to_string(end - i));
patches += grid;
if (patches > config.max_patches)
throw std::invalid_argument(display_name_ + " prompt images have " + std::to_string(patches) +
" patches, more than the " + std::to_string(config.max_patches) +
" the vision model runs at once");
spans.push_back({offset + i, end - i, std::make_shared<const OgaImage>(image), ImageHash(image)});
i = end;
}
Expand All @@ -364,7 +354,6 @@ bool OgaEngine::ImageAfter(std::size_t position) const {

std::pair<OgaEngine::Reuse, std::size_t> OgaEngine::Append(const char** why) {
active_.reset(); // so that a failure before Resume drops no other conversation's slot
if (!images_.empty() && !vision_warm_) WarmUpVision();
const Reuse reuse = Resume(why);
Live& run = Active();
const std::size_t from = run.ran.size();
Expand Down Expand Up @@ -546,30 +535,6 @@ void OgaEngine::SetInputs(OgaGenerator* generator, const std::vector<int>& ids,
api_->Check(f.generator_set_inputs(generator, named.get()), "OgaGenerator_SetInputs");
}

void OgaEngine::WarmUpVision() {
const OgaVisionConfig& config = *vision_;
auto side = static_cast<std::int64_t>(std::sqrt(static_cast<double>(config.max_patches)));
while (side * side > config.max_patches) --side;
side -= side % config.spatial_merge_size;
OgaImage image;
image.grid_h = side;
image.grid_w = side;
image.pixels.assign(static_cast<std::size_t>(side * side * config.patch_values), 0);
std::vector<int> ids(static_cast<std::size_t>(
1 + side * side / (config.spatial_merge_size * config.spatial_merge_size)),
config.image_token_id);
ids.front() = config.vision_start_token_id;
const auto start = std::chrono::steady_clock::now();
const OgaApi::Owned<OgaGenerator> generator = NewGenerator(static_cast<std::uint32_t>(ids.size()));
SetInputs(generator.get(), ids, {&image});
Logits(generator.get());
vision_warm_ = true;
const auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now() - start).count();
header_print("FLM", "OGA vision: warmed up with a " << side << "x" << side << "-patch image in "
<< ms << " ms");
}

OgaApi::Owned<OgaTensor> OgaEngine::NewTensor(const std::vector<std::int64_t>& shape,
OgaElementType type, const void* data,
std::size_t bytes) const {
Expand Down
5 changes: 0 additions & 5 deletions src/include/oga/oga_engine.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,6 @@ struct OgaVisionConfig {
std::int64_t spatial_merge_size = 0;
/// \brief values in one patch row of pixel_values
std::int64_t patch_values = 0;
/// \brief the most patches the images of one prompt may have together
std::int64_t max_patches = kMaxImagePatches;
};

/// \brief how many generators an engine keeps, one per conversation, and the memory they may take
Expand Down Expand Up @@ -213,8 +211,6 @@ class OgaEngine final : public causal_lm {
/// \note No images: the batch of one image without pixels (empty_image_batch)
void SetInputs(OgaGenerator* generator, const std::vector<int>& ids,
const std::vector<const OgaImage*>& images) const;
/// \brief run the vision model once at max_patches, before the first real image
void WarmUpVision();
/// \brief a tensor that OGA owns, with `bytes` of `data` copied in when given
OgaApi::Owned<OgaTensor> NewTensor(const std::vector<std::int64_t>& shape,
OgaElementType type, const void* data,
Expand All @@ -233,7 +229,6 @@ class OgaEngine final : public causal_lm {
std::optional<OgaVisionConfig> vision_;
bool empty_image_batch_ = false;
bool rewinds_ = false;
bool vision_warm_ = false;
OgaSlotLimits limits_;
std::uint64_t replayed_ = 0;
std::uint64_t reused_ = 0;
Expand Down
9 changes: 0 additions & 9 deletions src/include/oga/oga_vision.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,6 @@

namespace flm::oga {

/// \brief the most patches the images of one prompt may have together
/// \note OGA runs every image of a prompt in one pass of the vision model. On
/// hip-ep 0.5.2 (OGA 0.17.0) that pass gives wrong image features when it
/// has many more patches than the first pass the model ran, or when that
/// first pass had only a few hundred. The engine therefore runs one pass of
/// this size before its first real image, and no prompt may have more.
/// 8100 patches is a 90x90 grid (1440x1440 pixels), 2025 image tokens.
inline constexpr std::int64_t kMaxImagePatches = 90 * 90;

/// \brief one image, preprocessed for the vision model
struct OgaImage {
std::int64_t grid_t = 1;
Expand Down
Loading
Loading