From f48acf39ec1c7d352bd421405803e9eb4fcf0a04 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:10 +0000 Subject: [PATCH 01/16] Deploy Doctum docs with pnpm dlx Every Doctum deployment since the pnpm 12 upgrade has failed. The deploy step used wrangler-action, which installs Wrangler with `pnpm add` at the workspace root. pnpm 12 rejects that twice: the global ignore-workspace-root-check setting is no longer supported, and once it is bypassed the install fails on unapproved esbuild and workerd build scripts. Run Wrangler through `pnpm dlx wrangler@4 pages deploy`, passing the Cloudflare credentials and project name through the step environment. This installs only Wrangler, honors the workspace's minimum release age and leaves the checkout untouched. A local run of the command reaches Wrangler's credentials check. --- .github/workflows/doctum.yml | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/.github/workflows/doctum.yml b/.github/workflows/doctum.yml index 74461e9d69..c286add783 100644 --- a/.github/workflows/doctum.yml +++ b/.github/workflows/doctum.yml @@ -57,12 +57,11 @@ jobs: run: | corepack enable corepack prepare pnpm@12.10.1 --activate - pnpm config set ignore-workspace-root-check true pnpm --version - name: Deploy - uses: cloudflare/wrangler-action@953926a2e2182532811c01a25e53647d93bf07c0 # v4.1.3 - with: - apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }} - accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} - command: pages deploy doctum/build --project-name=${{ secrets.CLOUDFLARE_PAGE_PROJECT_NAME }} + env: + CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} + CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} + CLOUDFLARE_PAGE_PROJECT_NAME: ${{ secrets.CLOUDFLARE_PAGE_PROJECT_NAME }} + run: pnpm dlx wrangler@4 pages deploy doctum/build --project-name="$CLOUDFLARE_PAGE_PROJECT_NAME" From 7acdfa5c5db048eda83e734ad017cd18924993be Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:10 +0000 Subject: [PATCH 02/16] Explain retained Hypervel behavior when reconciling upstream changes Ask contributors reconciling upstream changes to add missing explanations for non-obvious Hypervel behavior they keep, so later syncs can see why it exists. --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index c23d4b11a1..51abb42f96 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -192,7 +192,7 @@ Build complete, long-term solutions, not MVPs or local workarounds. A broad chan - **Use Laravel observer conventions** — place Eloquent observers in a top-level `Observers/` directory. Register model-specific observers with `#[ObservedBy(...)]`; use `observe()` only for dynamic registration or observers supplied automatically by a reusable concern. - **Use attributed local scopes** — define local Eloquent query scopes as protected methods marked with `#[Scope]`, rather than legacy public `scopeFoo()` methods. Use separate scope classes only for genuine global scopes. - **No class docblocks unless warranted** — only add a class-level docblock if something unusual or complex needs explanation: purpose, architectural role, usage patterns. Never write one that inventories the class — trait lists, method summaries, "registers X, configures Y" — that duplicates the members' own docblocks and goes stale. Method docblocks (title only, Laravel-style, imperative mood: "Return", not "Returns") are always added, except on test methods, whether identified by a test name prefix or PHPUnit’s `#[Test]` attribute. A body can accompany the title for complex methods that need further explanation. -- **Add comments only where they're genuinely useful** (required Laravel-style method title docblocks and config section comment blocks still apply) — a short WHY for logic that isn't obvious from reading the code, the reason behind a bug fix, or logic that's coupled to code in other files and hard to understand in isolation. Avoid unnecessary code comments; don't comment what the code does, and don't annotate framework divergences, routine casts, or type normalizations. +- **Add comments only where they're genuinely useful** (required Laravel-style method title docblocks and config section comment blocks still apply) — a short WHY for logic that isn't obvious from reading the code, the reason behind a bug fix, or logic that's coupled to code in other files and hard to understand in isolation. Avoid unnecessary code comments; don't comment what the code does, and don't annotate framework divergences, routine casts, or type normalizations. When reconciling upstream changes, add missing explanations for non-obvious Hypervel behavior being retained, so future syncs can understand why it exists. - **Don't make classes final by default** — keep classes open; add `final` only when it protects a real invariant or avoids a concrete framework/API problem, e.g. immutability, coroutine-safety, or a security guarantee. - **Place methods logically, not at the end** — group new methods with related ones (getters with getters, setters with setters). Two exceptions: preserve upstream order when merging ported code (see Porting rules), and `flushState()` has its own placement rule (see Static state and test cleanup). - **Only extract methods when justified** — extract only when the logic is complex enough to benefit from a name, it's likely to be reused, or two or more methods call it. Don't extract a simple one-liner with a single caller. From de4ec0dcfbc7c36250041f457c00c6f57f39493d Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:10 +0000 Subject: [PATCH 03/16] Dispatch failure events for AI operations Port laravel/ai#1090 (laravel/ai 1.x at d44ae14d17fb). Embeddings, image, audio, transcription, reranking and classification requests, file uploads and store changes dispatched a starting and a success event but nothing on failure, so listeners could not close a failed call. Add AudioFailed, ImageFailed, TranscriptionFailed, RerankingFailed, StoringFileFailed, CreatingStoreFailed, AddingFileToStoreFailed and RemovingFileFromStoreFailed, dispatched through Provider::dispatchingFailure() for each failed attempt, including attempts failover recovers from. The existing EmbeddingsFailed and ClassificationFailed events move onto the same helper. Errors thrown by success listeners still do not produce failure events. The helper keeps Hypervel's hasListeners() guard, so passive observers alone do not trigger optional failure events. The new events type the provider as the contract, like their starting events. Classifies also carries laravel/ai#1134's state and attachment docblock types. Port upstream's operation, file and store event tests and remove the Hypervel embeddings tests they duplicate, keeping the check that a failed attempt dispatches no success event. Formatting, static analysis and the AI and Sentry suites pass. --- src/ai/src/Events/AddingFileToStoreFailed.php | 23 +++ src/ai/src/Events/AudioFailed.php | 27 +++ src/ai/src/Events/ClassificationFailed.php | 3 + src/ai/src/Events/CreatingStoreFailed.php | 27 +++ src/ai/src/Events/EmbeddingsFailed.php | 3 + src/ai/src/Events/ImageFailed.php | 27 +++ .../Events/RemovingFileFromStoreFailed.php | 23 +++ src/ai/src/Events/RerankingFailed.php | 27 +++ src/ai/src/Events/StoringFileFailed.php | 23 +++ src/ai/src/Events/TranscriptionFailed.php | 27 +++ src/ai/src/Providers/Concerns/Classifies.php | 21 +-- .../src/Providers/Concerns/GeneratesAudio.php | 21 ++- .../Concerns/GeneratesEmbeddings.php | 17 +- .../Providers/Concerns/GeneratesImages.php | 23 ++- .../Concerns/GeneratesTranscriptions.php | 21 ++- .../src/Providers/Concerns/ManagesFiles.php | 7 +- .../src/Providers/Concerns/ManagesStores.php | 19 +- src/ai/src/Providers/Concerns/Reranks.php | 21 ++- src/ai/src/Providers/Provider.php | 27 +++ tests/Ai/Feature/EmbeddingsFakeTest.php | 28 +-- tests/Ai/Feature/FileAndStoreEventsTest.php | 93 ++++++++++ tests/Ai/Feature/OperationEventsTest.php | 174 ++++++++++++++++++ 22 files changed, 592 insertions(+), 90 deletions(-) create mode 100644 src/ai/src/Events/AddingFileToStoreFailed.php create mode 100644 src/ai/src/Events/AudioFailed.php create mode 100644 src/ai/src/Events/CreatingStoreFailed.php create mode 100644 src/ai/src/Events/ImageFailed.php create mode 100644 src/ai/src/Events/RemovingFileFromStoreFailed.php create mode 100644 src/ai/src/Events/RerankingFailed.php create mode 100644 src/ai/src/Events/StoringFileFailed.php create mode 100644 src/ai/src/Events/TranscriptionFailed.php create mode 100644 tests/Ai/Feature/FileAndStoreEventsTest.php create mode 100644 tests/Ai/Feature/OperationEventsTest.php diff --git a/src/ai/src/Events/AddingFileToStoreFailed.php b/src/ai/src/Events/AddingFileToStoreFailed.php new file mode 100644 index 0000000000..065843cdae --- /dev/null +++ b/src/ai/src/Events/AddingFileToStoreFailed.php @@ -0,0 +1,23 @@ +|string $state + * @param array|string $state * @param array $questions * @param array $providerOptions - * @param array $attachments + * @param array $attachments */ public function classify(string|array $state, array $questions, ?string $model = null, int $timeout = 30, array $providerOptions = [], array $attachments = []): ClassificationResponse { @@ -52,8 +52,8 @@ public function classify(string|array $state, array $questions, ?string $model = )); } - try { - $response = $this->classificationGateway()->classify( + return tap($this->dispatchingFailure( + fn () => $this->classificationGateway()->classify( $this, $model, $state, @@ -61,16 +61,9 @@ public function classify(string|array $state, array $questions, ?string $model = $timeout, $providerOptions, $attachments, - ); - } catch (Throwable $exception) { - if ($this->events->hasListeners(ClassificationFailed::class)) { - $this->events->dispatch(new ClassificationFailed($invocationId, $this, $model, $prompt, $exception)); - } - - throw $exception; - } - - return tap($response, function (ClassificationResponse $response) use ($invocationId, $model, $prompt): void { + ), + fn (Throwable $exception) => new ClassificationFailed($invocationId, $this, $model, $prompt, $exception), + ), function (ClassificationResponse $response) use ($invocationId, $model, $prompt): void { if ($this->events->hasListeners(Classified::class)) { $this->events->dispatch(new Classified( $invocationId, diff --git a/src/ai/src/Providers/Concerns/GeneratesAudio.php b/src/ai/src/Providers/Concerns/GeneratesAudio.php index d113b6d216..6f213d50bc 100644 --- a/src/ai/src/Providers/Concerns/GeneratesAudio.php +++ b/src/ai/src/Providers/Concerns/GeneratesAudio.php @@ -5,11 +5,13 @@ namespace Hypervel\Ai\Providers\Concerns; use Hypervel\Ai\Ai; +use Hypervel\Ai\Events\AudioFailed; use Hypervel\Ai\Events\AudioGenerated; use Hypervel\Ai\Events\GeneratingAudio; use Hypervel\Ai\Prompts\AudioPrompt; use Hypervel\Ai\Responses\AudioResponse; use Hypervel\Support\Str; +use Throwable; trait GeneratesAudio { @@ -45,14 +47,17 @@ public function audio( )); } - return tap($this->audioGateway()->generateAudio( - $this, - $model, - $prompt->text, - $prompt->voice, - $prompt->instructions, - $timeout, - $prompt->providerOptions, + return tap($this->dispatchingFailure( + fn () => $this->audioGateway()->generateAudio( + $this, + $model, + $prompt->text, + $prompt->voice, + $prompt->instructions, + $timeout, + $prompt->providerOptions, + ), + fn (Throwable $exception) => new AudioFailed($invocationId, $this, $model, $prompt, $exception), ), function (AudioResponse $response) use ($invocationId, $model, $prompt): void { if ($this->events->hasListeners(AudioGenerated::class)) { $this->events->dispatch(new AudioGenerated( diff --git a/src/ai/src/Providers/Concerns/GeneratesEmbeddings.php b/src/ai/src/Providers/Concerns/GeneratesEmbeddings.php index a675b388d3..428c6e52de 100644 --- a/src/ai/src/Providers/Concerns/GeneratesEmbeddings.php +++ b/src/ai/src/Providers/Concerns/GeneratesEmbeddings.php @@ -58,24 +58,17 @@ public function embeddings(array $inputs, ?int $dimensions = null, ?string $mode )); } - try { - $response = $this->embeddingGateway()->generateEmbeddings( + return tap($this->dispatchingFailure( + fn () => $this->embeddingGateway()->generateEmbeddings( $this, $model, $inputs, $dimensions, $timeout, $providerOptions, - ); - } catch (Throwable $exception) { - if ($this->events->hasListeners(EmbeddingsFailed::class)) { - $this->events->dispatch(new EmbeddingsFailed($invocationId, $this, $model, $prompt, $exception)); - } - - throw $exception; - } - - return tap($response, function (EmbeddingsResponse $response) use ($invocationId, $model, $prompt): void { + ), + fn (Throwable $exception) => new EmbeddingsFailed($invocationId, $this, $model, $prompt, $exception), + ), function (EmbeddingsResponse $response) use ($invocationId, $model, $prompt): void { if ($this->events->hasListeners(EmbeddingsGenerated::class)) { $this->events->dispatch(new EmbeddingsGenerated( $invocationId, diff --git a/src/ai/src/Providers/Concerns/GeneratesImages.php b/src/ai/src/Providers/Concerns/GeneratesImages.php index c2e739f05b..a201634f77 100644 --- a/src/ai/src/Providers/Concerns/GeneratesImages.php +++ b/src/ai/src/Providers/Concerns/GeneratesImages.php @@ -6,11 +6,13 @@ use Hypervel\Ai\Ai; use Hypervel\Ai\Events\GeneratingImage; +use Hypervel\Ai\Events\ImageFailed; use Hypervel\Ai\Events\ImageGenerated; use Hypervel\Ai\Files\Image; use Hypervel\Ai\Prompts\ImagePrompt; use Hypervel\Ai\Responses\ImageResponse; use Hypervel\Support\Str; +use Throwable; trait GeneratesImages { @@ -49,15 +51,18 @@ public function image( )); } - return tap($this->imageGateway()->generateImage( - $this, - $model, - $prompt->prompt, - $prompt->attachments->all(), - $prompt->size, - $prompt->quality, - $timeout, - $prompt->providerOptions, + return tap($this->dispatchingFailure( + fn () => $this->imageGateway()->generateImage( + $this, + $model, + $prompt->prompt, + $prompt->attachments->all(), + $prompt->size, + $prompt->quality, + $timeout, + $prompt->providerOptions, + ), + fn (Throwable $exception) => new ImageFailed($invocationId, $this, $model, $prompt, $exception), ), function (ImageResponse $response) use ($invocationId, $prompt, $model): void { if ($this->events->hasListeners(ImageGenerated::class)) { $this->events->dispatch(new ImageGenerated( diff --git a/src/ai/src/Providers/Concerns/GeneratesTranscriptions.php b/src/ai/src/Providers/Concerns/GeneratesTranscriptions.php index 2f6fbde758..5486e4661a 100644 --- a/src/ai/src/Providers/Concerns/GeneratesTranscriptions.php +++ b/src/ai/src/Providers/Concerns/GeneratesTranscriptions.php @@ -7,10 +7,12 @@ use Hypervel\Ai\Ai; use Hypervel\Ai\Contracts\Files\TranscribableAudio; use Hypervel\Ai\Events\GeneratingTranscription; +use Hypervel\Ai\Events\TranscriptionFailed; use Hypervel\Ai\Events\TranscriptionGenerated; use Hypervel\Ai\Prompts\TranscriptionPrompt; use Hypervel\Ai\Responses\TranscriptionResponse; use Hypervel\Support\Str; +use Throwable; trait GeneratesTranscriptions { @@ -46,14 +48,17 @@ public function transcribe( )); } - return tap($this->transcriptionGateway()->generateTranscription( - $this, - $model, - $prompt->audio, - $prompt->language, - $prompt->diarize, - $prompt->timeout ?? 30, - $prompt->providerOptions + return tap($this->dispatchingFailure( + fn () => $this->transcriptionGateway()->generateTranscription( + $this, + $model, + $prompt->audio, + $prompt->language, + $prompt->diarize, + $prompt->timeout ?? 30, + $prompt->providerOptions + ), + fn (Throwable $exception) => new TranscriptionFailed($invocationId, $this, $model, $prompt, $exception), ), function (TranscriptionResponse $response) use ($invocationId, $model, $prompt): void { if ($this->events->hasListeners(TranscriptionGenerated::class)) { $this->events->dispatch(new TranscriptionGenerated( diff --git a/src/ai/src/Providers/Concerns/ManagesFiles.php b/src/ai/src/Providers/Concerns/ManagesFiles.php index 14348deacf..3d23496097 100644 --- a/src/ai/src/Providers/Concerns/ManagesFiles.php +++ b/src/ai/src/Providers/Concerns/ManagesFiles.php @@ -9,9 +9,11 @@ use Hypervel\Ai\Events\FileDeleted; use Hypervel\Ai\Events\FileStored; use Hypervel\Ai\Events\StoringFile; +use Hypervel\Ai\Events\StoringFileFailed; use Hypervel\Ai\Responses\FileResponse; use Hypervel\Ai\Responses\StoredFileResponse; use Hypervel\Support\Str; +use Throwable; trait ManagesFiles { @@ -43,7 +45,10 @@ public function putFile(StorableFile $file): StoredFileResponse } return tap( - $this->fileGateway()->putFile($this, $file), + $this->dispatchingFailure( + fn () => $this->fileGateway()->putFile($this, $file), + fn (Throwable $exception) => new StoringFileFailed($invocationId, $this, $file, $exception), + ), function (StoredFileResponse $response) use ($invocationId, $file): void { if ($this->events->hasListeners(FileStored::class)) { $this->events->dispatch(new FileStored( diff --git a/src/ai/src/Providers/Concerns/ManagesStores.php b/src/ai/src/Providers/Concerns/ManagesStores.php index 8d5422dfa4..f1f9eb07e9 100644 --- a/src/ai/src/Providers/Concerns/ManagesStores.php +++ b/src/ai/src/Providers/Concerns/ManagesStores.php @@ -8,15 +8,19 @@ use Hypervel\Ai\Ai; use Hypervel\Ai\Contracts\Files\HasProviderId; use Hypervel\Ai\Events\AddingFileToStore; +use Hypervel\Ai\Events\AddingFileToStoreFailed; use Hypervel\Ai\Events\CreatingStore; +use Hypervel\Ai\Events\CreatingStoreFailed; use Hypervel\Ai\Events\FileAddedToStore; use Hypervel\Ai\Events\FileRemovedFromStore; use Hypervel\Ai\Events\RemovingFileFromStore; +use Hypervel\Ai\Events\RemovingFileFromStoreFailed; use Hypervel\Ai\Events\StoreCreated; use Hypervel\Ai\Events\StoreDeleted; use Hypervel\Ai\Store; use Hypervel\Support\Collection; use Hypervel\Support\Str; +use Throwable; trait ManagesStores { @@ -57,7 +61,10 @@ public function createStore( } return tap( - $this->storeGateway()->createStore($this, $name, $description, $fileIds, $expiresWhenIdleFor), + $this->dispatchingFailure( + fn () => $this->storeGateway()->createStore($this, $name, $description, $fileIds, $expiresWhenIdleFor), + fn (Throwable $exception) => new CreatingStoreFailed($invocationId, $this, $name, $description, $fileIds, $expiresWhenIdleFor, $exception), + ), function (Store $store) use ($invocationId, $name, $description, $fileIds, $expiresWhenIdleFor): void { if ($this->events->hasListeners(StoreCreated::class)) { $this->events->dispatch(new StoreCreated( @@ -91,7 +98,10 @@ public function addFileToStore(string $storeId, HasProviderId $file, array $meta } return tap( - $this->storeGateway()->addFile($this, $storeId, $file->id(), $metadata), + $this->dispatchingFailure( + fn () => $this->storeGateway()->addFile($this, $storeId, $file->id(), $metadata), + fn (Throwable $exception) => new AddingFileToStoreFailed($invocationId, $this, $storeId, $file->id(), $exception), + ), function (string $documentId) use ($invocationId, $storeId, $file): void { if ($this->events->hasListeners(FileAddedToStore::class)) { $this->events->dispatch(new FileAddedToStore( @@ -129,7 +139,10 @@ public function removeFileFromStore(string $storeId, HasProviderId|string $docum } return tap( - $this->storeGateway()->removeFile($this, $storeId, $documentId), + $this->dispatchingFailure( + fn () => $this->storeGateway()->removeFile($this, $storeId, $documentId), + fn (Throwable $exception) => new RemovingFileFromStoreFailed($invocationId, $this, $storeId, $documentId, $exception), + ), function () use ($invocationId, $storeId, $documentId): void { if ($this->events->hasListeners(FileRemovedFromStore::class)) { $this->events->dispatch(new FileRemovedFromStore( diff --git a/src/ai/src/Providers/Concerns/Reranks.php b/src/ai/src/Providers/Concerns/Reranks.php index 6562e4d52b..75632aff1f 100644 --- a/src/ai/src/Providers/Concerns/Reranks.php +++ b/src/ai/src/Providers/Concerns/Reranks.php @@ -7,9 +7,11 @@ use Hypervel\Ai\Ai; use Hypervel\Ai\Events\Reranked; use Hypervel\Ai\Events\Reranking; +use Hypervel\Ai\Events\RerankingFailed; use Hypervel\Ai\Prompts\RerankingPrompt; use Hypervel\Ai\Responses\RerankingResponse; use Hypervel\Support\Str; +use Throwable; trait Reranks { @@ -40,14 +42,17 @@ public function rerank(array $documents, string $query, ?int $limit = null, ?str )); } - return tap($this->rerankingGateway()->rerank( - $this, - $model, - $documents, - $query, - $limit, - $timeout, - $providerOptions, + return tap($this->dispatchingFailure( + fn () => $this->rerankingGateway()->rerank( + $this, + $model, + $documents, + $query, + $limit, + $timeout, + $providerOptions, + ), + fn (Throwable $exception) => new RerankingFailed($invocationId, $this, $model, $prompt, $exception), ), function (RerankingResponse $response) use ($invocationId, $model, $prompt): void { if ($this->events->hasListeners(Reranked::class)) { $this->events->dispatch(new Reranked( diff --git a/src/ai/src/Providers/Provider.php b/src/ai/src/Providers/Provider.php index ad19b877c4..3e57aa3c5a 100644 --- a/src/ai/src/Providers/Provider.php +++ b/src/ai/src/Providers/Provider.php @@ -4,6 +4,7 @@ namespace Hypervel\Ai\Providers; +use Closure; use Hypervel\Ai\AiManager; use Hypervel\Ai\Contracts\Gateway\Gateway; use Hypervel\Ai\Contracts\Providers\Provider as ProviderContract; @@ -13,6 +14,7 @@ use Hypervel\Contracts\Events\Dispatcher; use Hypervel\Support\Collection; use Stringable; +use Throwable; abstract class Provider implements Stringable, ProviderContract { @@ -78,6 +80,31 @@ public function withHeaders(array $headers): static }); } + /** + * Run the given gateway call, dispatching the event built from its exception before rethrowing. + * + * @template TResult + * + * @param Closure(): TResult $call + * @param Closure(Throwable): object $failed + * @return TResult + */ + protected function dispatchingFailure(Closure $call, Closure $failed): mixed + { + try { + return $call(); + } catch (Throwable $exception) { + $event = $failed($exception); + + // Passive observers alone must not trigger optional failure events; hasListeners() excludes them. + if ($this->events->hasListeners($event::class)) { + $this->events->dispatch($event); + } + + throw $exception; + } + } + /** * Format the given provider / model list. */ diff --git a/tests/Ai/Feature/EmbeddingsFakeTest.php b/tests/Ai/Feature/EmbeddingsFakeTest.php index 88333f5ed4..a3df4a49fb 100644 --- a/tests/Ai/Feature/EmbeddingsFakeTest.php +++ b/tests/Ai/Feature/EmbeddingsFakeTest.php @@ -6,9 +6,7 @@ use Hypervel\Ai\Embeddings; use Hypervel\Ai\Enums\Lab; -use Hypervel\Ai\Events\EmbeddingsFailed; use Hypervel\Ai\Events\EmbeddingsGenerated; -use Hypervel\Ai\Events\GeneratingEmbeddings; use Hypervel\Ai\Files\Audio; use Hypervel\Ai\Files\Document; use Hypervel\Ai\Files\Image; @@ -561,7 +559,7 @@ public function testEmptyVectorsCanBeFaked(): void public function testThrowingSequenceEntryIsConsumed(): void { - Event::fake(); + Event::fake([EmbeddingsGenerated::class]); $exception = new RuntimeException('Embedding failed.'); Embeddings::fake([fn (): never => throw $exception, [[0.1, 0.2]]]); @@ -573,32 +571,8 @@ public function testThrowingSequenceEntryIsConsumed(): void $this->assertSame($exception, $caught); } - $started = Event::dispatched(GeneratingEmbeddings::class)->first()[0]; - Event::assertDispatched(EmbeddingsFailed::class, fn (EmbeddingsFailed $event): bool => $event->exception === $exception - && $event->invocationId === $started->invocationId - && $event->provider === $started->provider - && $event->model === $started->model - && $event->prompt === $started->prompt); Event::assertNotDispatched(EmbeddingsGenerated::class); $this->assertSame([[0.1, 0.2]], Embeddings::for(['Second'])->dimensions(2)->generate(provider: Lab::Jina)->embeddings); - Event::assertDispatchedTimes(EmbeddingsFailed::class, 1); - } - - public function testSuccessListenerFailureIsNotReportedAsAnEmbeddingFailure(): void - { - Event::fake([EmbeddingsFailed::class]); - Embeddings::fake(); - $exception = new RuntimeException('Listener failed.'); - Event::listen(EmbeddingsGenerated::class, fn (): never => throw $exception); - - try { - Embeddings::for(['Document'])->dimensions(2)->generate(provider: Lab::Jina); - $this->fail('The listener must throw.'); - } catch (RuntimeException $caught) { - $this->assertSame($exception, $caught); - } - - Event::assertNotDispatched(EmbeddingsFailed::class); } } diff --git a/tests/Ai/Feature/FileAndStoreEventsTest.php b/tests/Ai/Feature/FileAndStoreEventsTest.php new file mode 100644 index 0000000000..697c85bde1 --- /dev/null +++ b/tests/Ai/Feature/FileAndStoreEventsTest.php @@ -0,0 +1,93 @@ +make('config')->set('ai.providers.openai.key', 'test-key'); + } + + #[DataProvider('operations')] + public function testFileAndStoreFailuresDispatchAnEventMirroringTheStartingEventAndRethrow(Closure $invoke, string $startingEvent, string $failureEvent): void + { + Event::fake(); + + Http::fake([ + 'api.openai.com/*' => Http::response(['error' => ['message' => 'Server error.']], 500), + ]); + + $caught = null; + + try { + $invoke($this->app->make(AiManager::class)->instance('openai')); + } catch (Throwable $e) { + $caught = $e; + } + + Event::assertDispatchedTimes($startingEvent, 1); + Event::assertDispatchedTimes($failureEvent, 1); + + $started = Event::dispatched($startingEvent)->first()[0]; + $failed = Event::dispatched($failureEvent)->first()[0]; + + $this->assertNotNull($caught); + $this->assertSame($caught, $failed->exception); + $this->assertSame(get_object_vars($started), Arr::except(get_object_vars($failed), 'exception')); + } + + /** + * Provide each file and store operation with its starting and failure events. + */ + public static function operations(): array + { + return [ + 'storing a file' => [ + fn ($provider) => $provider->putFile(Document::fromString('Hello, World!', 'text/plain')->as('hello.txt')), + StoringFile::class, + StoringFileFailed::class, + ], + 'creating a store' => [ + fn ($provider) => $provider->createStore('Knowledge Base', 'Product docs'), + CreatingStore::class, + CreatingStoreFailed::class, + ], + 'adding a file to a store' => [ + fn ($provider) => $provider->addFileToStore('vs-123', new ProviderDocument('file-abc123')), + AddingFileToStore::class, + AddingFileToStoreFailed::class, + ], + 'removing a file from a store' => [ + fn ($provider) => $provider->removeFileFromStore('vs-123', 'file-abc123'), + RemovingFileFromStore::class, + RemovingFileFromStoreFailed::class, + ], + ]; + } +} diff --git a/tests/Ai/Feature/OperationEventsTest.php b/tests/Ai/Feature/OperationEventsTest.php new file mode 100644 index 0000000000..f4659cf5e9 --- /dev/null +++ b/tests/Ai/Feature/OperationEventsTest.php @@ -0,0 +1,174 @@ + throw $exception); + + $caught = null; + + try { + $invoke(); + } catch (Throwable $e) { + $caught = $e; + } + + Event::assertDispatchedTimes($startingEvent, 1); + Event::assertDispatchedTimes($failureEvent, 1); + + $started = Event::dispatched($startingEvent)->first()[0]; + $failed = Event::dispatched($failureEvent)->first()[0]; + + $this->assertSame($exception, $caught); + $this->assertSame($started->invocationId, $failed->invocationId); + $this->assertSame($started->provider, $failed->provider); + $this->assertSame($started->model, $failed->model); + $this->assertSame($started->prompt, $failed->prompt); + $this->assertSame($exception, $failed->exception); + } + + #[DataProvider('operations')] + public function testSuccessfulOperationsDoNotDispatchFailureEvents(Closure $fake, Closure $invoke, string $startingEvent, string $failureEvent): void + { + Event::fake(); + + $fake([]); + $invoke(); + + Event::assertDispatchedTimes($startingEvent, 1); + Event::assertNotDispatched($failureEvent); + } + + /** + * Provide each model operation with its fake, invocation, and starting and failure events. + */ + public static function operations(): array + { + return [ + 'embeddings' => [ + fn ($responses) => Embeddings::fake($responses), + fn () => Embeddings::for(['Hello world'])->generate(), + GeneratingEmbeddings::class, + EmbeddingsFailed::class, + ], + 'image' => [ + fn ($responses) => Image::fake($responses), + fn () => Image::of('A sunset')->generate(), + GeneratingImage::class, + ImageFailed::class, + ], + 'audio' => [ + fn ($responses) => Audio::fake($responses), + fn () => Audio::of('Hello world')->generate(), + GeneratingAudio::class, + AudioFailed::class, + ], + 'transcription' => [ + fn ($responses) => Transcription::fake($responses), + fn () => Transcription::of(base64_encode('audio'))->generate(), + GeneratingTranscription::class, + TranscriptionFailed::class, + ], + 'reranking' => [ + fn ($responses) => RerankingOperation::fake($responses), + fn () => RerankingOperation::of(['Hypervel is a framework'])->rerank('What is Hypervel?'), + Reranking::class, + RerankingFailed::class, + ], + 'classification' => [ + fn ($responses) => Classification::fake($responses), + fn () => Classification::of('Urgent request')->question('urgent', new Boolean('Is it urgent?'))->classify(), + Classifying::class, + ClassificationFailed::class, + ], + ]; + } + + public function testAThrowingSuccessListenerDoesNotReportTheOperationAsFailed(): void + { + Embeddings::fake(); + + $failed = []; + + Event::listen(EmbeddingsFailed::class, function (EmbeddingsFailed $event) use (&$failed): void { + $failed[] = $event; + }); + + $exception = new RuntimeException('Listener failed.'); + + Event::listen(EmbeddingsGenerated::class, fn () => throw $exception); + + try { + Embeddings::for(['Hello world'])->generate(); + $this->fail('The listener must throw.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + + $this->assertSame([], $failed); + } + + public function testARecoveredFailoverReportsOnlyTheAttemptThatFailed(): void + { + Event::fake(); + + $attempts = 0; + + Embeddings::fake(function () use (&$attempts): array { + if ($attempts++ === 0) { + throw RateLimitedException::forProvider('openai'); + } + + return [[0.1, 0.2]]; + }); + + $response = Embeddings::for(['Hello world'])->generate(['openai', 'gemini']); + + [$failedAttempt, $recoveredAttempt] = Event::dispatched(GeneratingEmbeddings::class) + ->map(fn (array $event): string => $event[0]->invocationId) + ->all(); + + $this->assertSame([[0.1, 0.2]], $response->embeddings); + + Event::assertDispatchedTimes(EmbeddingsFailed::class, 1); + Event::assertDispatched(EmbeddingsFailed::class, fn (EmbeddingsFailed $event): bool => $event->invocationId === $failedAttempt); + Event::assertDispatched(EmbeddingsGenerated::class, fn (EmbeddingsGenerated $event): bool => $event->invocationId === $recoveredAttempt); + } +} From df76d463ec3332194dac36a15507df9466c2ea77 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:10 +0000 Subject: [PATCH 04/16] Name the arguments of interrupted tool results Port laravel/ai#1133 (laravel/ai 1.x at d44ae14d17fb). A failed turn replays its unanswered tool calls with an interrupted placeholder result. Hypervel already kept the stored result ID, so OpenAI, Azure OpenAI and xAI receive the call_id their output is paired by. Match upstream's named arguments and port its regression test, which fails without the result ID. --- .../src/Storage/DatabaseConversationStore.php | 10 ++-- tests/Ai/Feature/FailedTurnRecordingTest.php | 48 +++++++++++++++++++ 2 files changed, 53 insertions(+), 5 deletions(-) diff --git a/src/ai/src/Storage/DatabaseConversationStore.php b/src/ai/src/Storage/DatabaseConversationStore.php index f29b42b215..15fe455b1a 100644 --- a/src/ai/src/Storage/DatabaseConversationStore.php +++ b/src/ai/src/Storage/DatabaseConversationStore.php @@ -657,11 +657,11 @@ protected function assistantTurnFrom(object $record): array protected function interruptedResultFor(array $toolCall): ToolResult { return new ToolResult( - $toolCall['id'], - $toolCall['name'], - $toolCall['arguments'] ?? [], - 'This tool call was interrupted before a result was recorded, so it may or may not have run.', - $toolCall['result_id'] ?? null, + id: $toolCall['id'], + name: $toolCall['name'], + arguments: $toolCall['arguments'] ?? [], + result: 'This tool call was interrupted before a result was recorded, so it may or may not have run.', + resultId: $toolCall['result_id'] ?? null, ); } diff --git a/tests/Ai/Feature/FailedTurnRecordingTest.php b/tests/Ai/Feature/FailedTurnRecordingTest.php index 88f42c25ac..0e7911a720 100644 --- a/tests/Ai/Feature/FailedTurnRecordingTest.php +++ b/tests/Ai/Feature/FailedTurnRecordingTest.php @@ -18,6 +18,7 @@ use Hypervel\Tests\Ai\Fixtures\Agents\RememberingFailingToolAgent; use Hypervel\Tests\Ai\Fixtures\Agents\RememberingToolUsingAgent; use Hypervel\Tests\Ai\TestCase; +use PHPUnit\Framework\Attributes\TestWith; #[WithConfig('ai.conversations.generate_title', false)] #[WithConfig('ai.providers.anthropic', ['driver' => 'anthropic', 'key' => 'test-key'])] @@ -280,4 +281,51 @@ public function testAConversationOpenedByAFailedTurnIsTitledTheWayACompletedOneI $this->assertStringContainsString('arrived at it', DB::table('agent_conversations')->value('title')); } + + #[TestWith(['openai'])] + #[TestWith(['xai'])] + public function testAReplayedInterruptedCallKeepsTheCallIdTheResponsesApiPairsItsOutputBy(string $provider): void + { + config(["ai.providers.{$provider}.key" => 'test-key']); + + Http::fake(['*' => Http::sequence() + ->push([ + 'id' => 'resp_1', + 'object' => 'response', + 'status' => 'completed', + 'output' => [[ + 'type' => 'function_call', + 'id' => 'fc_1', + 'call_id' => 'call_1', + 'name' => 'FixedNumberGenerator', + 'arguments' => '{}', + 'status' => 'completed', + ]], + 'usage' => ['input_tokens' => 10, 'output_tokens' => 5], + ]) + ->push([ + 'id' => 'resp_2', + 'object' => 'response', + 'status' => 'completed', + 'output' => [[ + 'type' => 'message', + 'id' => 'msg_2', + 'role' => 'assistant', + 'content' => [['type' => 'output_text', 'text' => 'Done.', 'annotations' => []]], + ]], + 'usage' => ['input_tokens' => 10, 'output_tokens' => 5], + ]), + ]); + + $agent = (new RememberingFailingToolAgent)->forUser((object) ['id' => 1]); + + $this->assertThrows(fn () => $agent->prompt('Go', provider: $provider), Exception::class, 'Forced to throw exception.'); + + (new RememberingFailingToolAgent)->continueLastConversation((object) ['id' => 1])->prompt('Again', provider: $provider); + + $input = collect(json_decode((string) Http::recorded()[1][0]->body(), true)['input']); + + $this->assertSame('call_1', $input->firstWhere('type', 'function_call')['call_id']); + $this->assertSame('call_1', $input->firstWhere('type', 'function_call_output')['call_id']); + } } From 50e40033c4b9eab3325526b7fd7a6b5019da564e Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:10 +0000 Subject: [PATCH 05/16] Cover string keys on created conversation models Port laravel/ai#1137's test (laravel/ai 1.x at d44ae14d17fb). Its source change restores $incrementing for Laravel 12, which lacks the WithoutIncrementing attribute. Hypervel's conversation models use HasUuids and are already non-incrementing, so only the test applies: a conversation and message created through Eloquent keep their string keys. --- .../Feature/ConversationRelationshipTest.php | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/tests/Ai/Feature/ConversationRelationshipTest.php b/tests/Ai/Feature/ConversationRelationshipTest.php index 0751be643e..0002e2db53 100644 --- a/tests/Ai/Feature/ConversationRelationshipTest.php +++ b/tests/Ai/Feature/ConversationRelationshipTest.php @@ -179,6 +179,35 @@ public function testConversationCanRetrieveItsParticipantUsingRelationship(): vo $this->assertTrue($conversation->participant->is($user)); } + public function testConversationAndMessageKeepTheirStringKeysWhenCreated(): void + { + $user = ConversationRelationshipUser::create(['name' => 'Taylor']); + + $conversation = Conversation::create([ + 'id' => 'conversation-1', + 'participant_type' => $user->getMorphClass(), + 'participant_id' => $user->id, + 'title' => 'Conversation', + ]); + + $message = $conversation->messages()->create([ + 'id' => 'message-1', + 'participant_type' => $user->getMorphClass(), + 'participant_id' => $user->id, + 'agent' => 'Agent', + 'role' => 'user', + 'content' => 'Hello', + 'attachments' => [], + 'steps' => [], + 'usage' => [], + 'meta' => [], + 'status' => MessageStatus::Completed, + ]); + + $this->assertSame('conversation-1', $conversation->id); + $this->assertSame('message-1', $message->id); + } + public function testDeletingAConversationRemovesItsMessages(): void { $conversation = Conversation::create(['id' => (string) Str::uuid7(), 'title' => 'Conversation']); From 121891c42b0de1e7af9ef1e51b0465403be13dc4 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:21 +0000 Subject: [PATCH 06/16] Classify image attachments with TypeSafe and OpenRouter Port laravel/ai#1134 (laravel/ai 1.x at d44ae14d17fb). TypeSafe and OpenRouter classification rejected image attachments. TypeSafe now sends images as referenced assets with its images header, OpenRouter appends data URL image parts to the state, and OpenAI and OpenAI-compatible classification label images given string keys, so questions can refer to them by name. ResolvesInlineImages shares the inline image checks. Fix two defects. Every classification state docblock said array, although list states are supported, so list states failed static analysis; widen them to array, pinned by a types fixture. TypeSafe also dropped an attachment named "images" from a structured state when unnamed attachments were given, while still uploading it; that combination is now rejected before any request. Port upstream's TypeSafe, OpenRouter, OpenAI, fake and integration tests with the updated fixture images and the text-only classifier helper. ClassificationFakeTest also drops the classification failure assertions that duplicate laravel/ai#1090's tests. Formatting, static analysis and the AI suite pass. --- src/ai/src/Classification.php | 4 +- .../Gateway/ClassificationGateway.php | 4 +- .../Providers/ClassificationProvider.php | 4 +- .../src/Gateway/Concerns/AnswersQuestions.php | 4 +- .../Gateway/Concerns/ResolvesInlineImages.php | 43 ++++ .../src/Gateway/FakeClassificationGateway.php | 4 +- .../OpenAi/OpenAiClassificationGateway.php | 53 ++--- .../OpenRouterClassificationGateway.php | 78 ++++++- src/ai/src/Gateway/TypeSafeGateway.php | 97 +++++++- .../PendingClassification.php | 4 +- src/ai/src/Prompts/ClassificationPrompt.php | 4 +- src/ai/src/Providers/OpenRouterProvider.php | 8 + src/ai/src/Providers/TypeSafeProvider.php | 8 + tests/Ai/Feature/ClassificationFakeTest.php | 20 +- .../Providers/OpenAi/ClassificationTest.php | 14 +- .../OpenRouter/ClassificationTest.php | 91 +++++++- .../Providers/TypeSafe/ClassificationTest.php | 208 ++++++++++++++++++ tests/Ai/Fixtures/Images/blue.jpg | Bin 755 -> 2394 bytes tests/Ai/Fixtures/Images/red.png | Bin 176 -> 2019 bytes tests/Ai/Fixtures/helpers.php | 19 ++ tests/Integration/Ai/ClassificationTest.php | 18 ++ tests/Integration/Ai/Fixtures/Providers.php | 2 + types/Ai/Classification.php | 16 ++ 23 files changed, 635 insertions(+), 68 deletions(-) create mode 100644 src/ai/src/Gateway/Concerns/ResolvesInlineImages.php create mode 100644 types/Ai/Classification.php diff --git a/src/ai/src/Classification.php b/src/ai/src/Classification.php index eb8ea084c2..d0a750eed5 100644 --- a/src/ai/src/Classification.php +++ b/src/ai/src/Classification.php @@ -16,8 +16,8 @@ class Classification /** * Create a new pending classification for the given state. * - * @param array|string $state - * @param array $attachments + * @param array|string $state + * @param array $attachments * * @throws InvalidArgumentException if the state is blank */ diff --git a/src/ai/src/Contracts/Gateway/ClassificationGateway.php b/src/ai/src/Contracts/Gateway/ClassificationGateway.php index 2c14729f00..3d6704d584 100644 --- a/src/ai/src/Contracts/Gateway/ClassificationGateway.php +++ b/src/ai/src/Contracts/Gateway/ClassificationGateway.php @@ -15,10 +15,10 @@ interface ClassificationGateway /** * Answer the given questions about the state. * - * @param array|string $state + * @param array|string $state * @param array $questions * @param array $providerOptions - * @param array $attachments + * @param array $attachments */ public function classify( ClassificationProvider $provider, diff --git a/src/ai/src/Contracts/Providers/ClassificationProvider.php b/src/ai/src/Contracts/Providers/ClassificationProvider.php index 47bf6be2a6..07bceed379 100644 --- a/src/ai/src/Contracts/Providers/ClassificationProvider.php +++ b/src/ai/src/Contracts/Providers/ClassificationProvider.php @@ -15,10 +15,10 @@ interface ClassificationProvider extends Provider /** * Answer the given questions about the state. * - * @param array|string $state + * @param array|string $state * @param array $questions * @param array $providerOptions - * @param array $attachments + * @param array $attachments */ public function classify(string|array $state, array $questions, ?string $model = null, int $timeout = 30, array $providerOptions = [], array $attachments = []): ClassificationResponse; diff --git a/src/ai/src/Gateway/Concerns/AnswersQuestions.php b/src/ai/src/Gateway/Concerns/AnswersQuestions.php index 4a88f1f729..d1e8198597 100644 --- a/src/ai/src/Gateway/Concerns/AnswersQuestions.php +++ b/src/ai/src/Gateway/Concerns/AnswersQuestions.php @@ -31,10 +31,10 @@ abstract protected function classificationEndpoint(): string; /** * Answer the given questions about the state. * - * @param array|string $state + * @param array|string $state * @param array $questions * @param array $providerOptions - * @param array $attachments + * @param array $attachments */ public function classify( ClassificationProvider $provider, diff --git a/src/ai/src/Gateway/Concerns/ResolvesInlineImages.php b/src/ai/src/Gateway/Concerns/ResolvesInlineImages.php new file mode 100644 index 0000000000..f39d76cb9c --- /dev/null +++ b/src/ai/src/Gateway/Concerns/ResolvesInlineImages.php @@ -0,0 +1,43 @@ + $formats + * @return array{Image&StorableFile, string, string} + * + * @throws InvalidArgumentException if the attachment is not an image with inline content in one of the formats + */ + protected function resolveInlineImage(File|UploadedFile $attachment, string $endpoint, array $formats): array + { + $image = $attachment instanceof UploadedFile ? Image::fromUpload($attachment) : $attachment; + + if (! $image instanceof Image || ! $image instanceof StorableFile) { + throw new InvalidArgumentException("{$endpoint} only accept images with inline content; [" . get_debug_type($attachment) . '] given.'); + } + + $content = $image->content(); + + $mime = $image->mimeType() ?? (new finfo(FILEINFO_MIME_TYPE))->buffer($content); + + if (! array_key_exists($mime, $formats)) { + throw new InvalidArgumentException("{$endpoint} only accept " . Arr::join($formats, ', ', count($formats) > 2 ? ', or ' : ' or ') . " images; [{$mime}] given."); + } + + return [$image, $content, $mime]; + } +} diff --git a/src/ai/src/Gateway/FakeClassificationGateway.php b/src/ai/src/Gateway/FakeClassificationGateway.php index 306a84505b..16e5738222 100644 --- a/src/ai/src/Gateway/FakeClassificationGateway.php +++ b/src/ai/src/Gateway/FakeClassificationGateway.php @@ -40,10 +40,10 @@ public function __construct( /** * Answer the given questions about the state. * - * @param array|string $state + * @param array|string $state * @param array $questions * @param array $providerOptions - * @param array $attachments + * @param array $attachments */ public function classify( ClassificationProvider $provider, diff --git a/src/ai/src/Gateway/OpenAi/OpenAiClassificationGateway.php b/src/ai/src/Gateway/OpenAi/OpenAiClassificationGateway.php index 9cb9807867..20da95169c 100644 --- a/src/ai/src/Gateway/OpenAi/OpenAiClassificationGateway.php +++ b/src/ai/src/Gateway/OpenAi/OpenAiClassificationGateway.php @@ -4,18 +4,16 @@ namespace Hypervel\Ai\Gateway\OpenAi; -use finfo; use Hypervel\Ai\Classification\Boolean; use Hypervel\Ai\Classification\Choice; use Hypervel\Ai\Classification\Score; -use Hypervel\Ai\Contracts\Files\StorableFile; use Hypervel\Ai\Contracts\Gateway\ClassificationGateway; use Hypervel\Ai\Contracts\Providers\ClassificationProvider; use Hypervel\Ai\Contracts\Question; use Hypervel\Ai\Enums\Lab; use Hypervel\Ai\Files\File; -use Hypervel\Ai\Files\Image; use Hypervel\Ai\Gateway\Concerns\HandlesFailoverErrors; +use Hypervel\Ai\Gateway\Concerns\ResolvesInlineImages; use Hypervel\Ai\Gateway\OpenAi\Concerns\CreatesOpenAiClient; use Hypervel\Ai\Providers\Provider; use Hypervel\Ai\Responses\ClassificationResponse; @@ -32,14 +30,15 @@ class OpenAiClassificationGateway implements ClassificationGateway { use CreatesOpenAiClient; use HandlesFailoverErrors; + use ResolvesInlineImages; /** * Answer the given questions about the state. * - * @param array|string $state + * @param array|string $state * @param array $questions * @param array $providerOptions - * @param array $attachments + * @param array $attachments */ public function classify( ClassificationProvider $provider, @@ -87,8 +86,8 @@ public function classify( /** * Map the state and attachments to the decisions input. * - * @param array|string $state - * @param array $attachments + * @param array|string $state + * @param array $attachments */ protected function mapInput(string|array $state, array $attachments): string|array { @@ -98,13 +97,17 @@ protected function mapInput(string|array $state, array $attachments): string|arr return $text; } - return [[ - 'role' => 'user', - 'content' => [ - ['type' => 'input_text', 'text' => $text], - ...array_map($this->mapImage(...), array_values($attachments)), - ], - ]]; + $content = [['type' => 'input_text', 'text' => $text]]; + + foreach ($attachments as $name => $attachment) { + if (is_string($name)) { + $content[] = ['type' => 'input_text', 'text' => "{$name}:"]; + } + + $content[] = $this->mapImage($attachment); + } + + return [['role' => 'user', 'content' => $content]]; } /** @@ -112,23 +115,11 @@ protected function mapInput(string|array $state, array $attachments): string|arr * * @throws InvalidArgumentException if the attachment is not a JPEG, PNG, GIF, or WebP image with inline content */ - protected function mapImage(File|UploadedFile $image): array + protected function mapImage(File|UploadedFile $attachment): array { - if ($image instanceof UploadedFile) { - $image = Image::fromUpload($image); - } - - if (! $image instanceof Image || ! $image instanceof StorableFile) { - throw new InvalidArgumentException('OpenAI decisions only accept images with inline content; [' . get_debug_type($image) . '] given.'); - } - - $content = $image->content(); - - $mime = $image->mimeType() ?? (new finfo(FILEINFO_MIME_TYPE))->buffer($content); - - if (! in_array($mime, ['image/jpeg', 'image/png', 'image/gif', 'image/webp'], true)) { - throw new InvalidArgumentException("OpenAI decisions only accept JPEG, PNG, GIF, or WebP images; [{$mime}] given."); - } + [$image, $content, $mime] = $this->resolveInlineImage($attachment, 'OpenAI decisions', [ + 'image/jpeg' => 'JPEG', 'image/png' => 'PNG', 'image/gif' => 'GIF', 'image/webp' => 'WebP', + ]); return array_merge($image->providerOptions($this->providerOptionsKey()), [ 'type' => 'input_image', @@ -228,7 +219,7 @@ protected function mapAnswer(array $answer, ?Question $question): ?Answer /** * Convert structured content to the text the endpoint accepts. * - * @param array|string $value + * @param array|string $value */ protected function toText(string|array $value): string { diff --git a/src/ai/src/Gateway/OpenRouter/OpenRouterClassificationGateway.php b/src/ai/src/Gateway/OpenRouter/OpenRouterClassificationGateway.php index d296419002..1a2a7171a3 100644 --- a/src/ai/src/Gateway/OpenRouter/OpenRouterClassificationGateway.php +++ b/src/ai/src/Gateway/OpenRouter/OpenRouterClassificationGateway.php @@ -5,19 +5,95 @@ namespace Hypervel\Ai\Gateway\OpenRouter; use Hypervel\Ai\Contracts\Gateway\ClassificationGateway; +use Hypervel\Ai\Contracts\Providers\ClassificationProvider; +use Hypervel\Ai\Contracts\Question; +use Hypervel\Ai\Enums\Lab; +use Hypervel\Ai\Files\File; use Hypervel\Ai\Gateway\Concerns\AnswersQuestions; use Hypervel\Ai\Gateway\Concerns\HandlesFailoverErrors; +use Hypervel\Ai\Gateway\Concerns\ResolvesInlineImages; use Hypervel\Ai\Gateway\OpenRouter\Concerns\CreatesOpenRouterClient; use Hypervel\Ai\Providers\Provider; +use Hypervel\Ai\Responses\ClassificationResponse; +use Hypervel\Http\UploadedFile; use Hypervel\Support\Str; +use InvalidArgumentException; class OpenRouterClassificationGateway implements ClassificationGateway { - use AnswersQuestions; + use AnswersQuestions { + classify as answerQuestions; + } use CreatesOpenRouterClient { baseUrl as openRouterBaseUrl; } use HandlesFailoverErrors; + use ResolvesInlineImages; + + /** + * Answer the given questions about the state, sending image attachments as image parts of the state. + * + * @param array|string $state + * @param array $questions + * @param array $providerOptions + * @param array $attachments + */ + public function classify( + ClassificationProvider $provider, + string $model, + string|array $state, + array $questions, + int $timeout = 30, + array $providerOptions = [], + array $attachments = [], + ): ClassificationResponse { + return $this->answerQuestions( + $provider, + $model, + $attachments === [] ? $state : $this->stateWithImages($state, $attachments), + $questions, + $timeout, + $providerOptions, + ); + } + + /** + * Append the images to the state as data URL image parts, each named image preceded by its name. + * + * @param array|string $state + * @param array $attachments + */ + protected function stateWithImages(string|array $state, array $attachments): array + { + $state = is_array($state) && array_is_list($state) ? $state : [$state]; + + foreach ($attachments as $name => $attachment) { + if (is_string($name)) { + $state[] = "{$name}:"; + } + + $state[] = $this->mapImage($attachment); + } + + return $state; + } + + /** + * Map an image to a data URL image part. + * + * @throws InvalidArgumentException if the attachment is not a JPEG, PNG, or WebP image with inline content + */ + protected function mapImage(File|UploadedFile $attachment): array + { + [$image, $content, $mime] = $this->resolveInlineImage($attachment, 'OpenRouter decisions', [ + 'image/jpeg' => 'JPEG', 'image/png' => 'PNG', 'image/webp' => 'WebP', + ]); + + return array_merge($image->providerOptions(Lab::OpenRouter), [ + 'type' => 'image_url', + 'image_url' => ['url' => 'data:' . $mime . ';base64,' . base64_encode($content)], + ]); + } /** * Get the path of the endpoint that answers questions. diff --git a/src/ai/src/Gateway/TypeSafeGateway.php b/src/ai/src/Gateway/TypeSafeGateway.php index 416ff00aa2..b79947b11f 100644 --- a/src/ai/src/Gateway/TypeSafeGateway.php +++ b/src/ai/src/Gateway/TypeSafeGateway.php @@ -6,16 +6,111 @@ use Hypervel\Ai\Contracts\Gateway\ClassificationGateway; use Hypervel\Ai\Contracts\Providers\ClassificationProvider; +use Hypervel\Ai\Contracts\Question; +use Hypervel\Ai\Enums\Lab; +use Hypervel\Ai\Files\File; use Hypervel\Ai\Gateway\Concerns\AnswersQuestions; use Hypervel\Ai\Gateway\Concerns\CreatesClient; use Hypervel\Ai\Gateway\Concerns\HandlesFailoverErrors; +use Hypervel\Ai\Gateway\Concerns\ResolvesInlineImages; +use Hypervel\Ai\Responses\ClassificationResponse; use Hypervel\Http\Client\PendingRequest; +use Hypervel\Http\UploadedFile; +use Hypervel\Support\Str; +use InvalidArgumentException; class TypeSafeGateway implements ClassificationGateway { - use AnswersQuestions; + use AnswersQuestions { + classify as answerQuestions; + } use CreatesClient; use HandlesFailoverErrors; + use ResolvesInlineImages; + + /** + * Answer the given questions about the state, sending image attachments as referenced assets. + * + * @param array|string $state + * @param array $questions + * @param array $providerOptions + * @param array $attachments + */ + public function classify( + ClassificationProvider $provider, + string $model, + string|array $state, + array $questions, + int $timeout = 30, + array $providerOptions = [], + array $attachments = [], + ): ClassificationResponse { + if ($attachments === []) { + return $this->answerQuestions($provider, $model, $state, $questions, $timeout, $providerOptions); + } + + $references = array_map(fn (): string => (string) Str::uuid(), $attachments); + + return $this->answerQuestions( + $provider->withHeaders(['TypeSafe-Danger-Zone-Images' => 'true']), + $model, + $this->stateWithImages($state, $references), + $questions, + $timeout, + [...$providerOptions, 'assets' => array_combine($references, array_map($this->mapImage(...), $attachments))], + ); + } + + /** + * Place the image references in the state: keyed by name into a structured state, or labeled by name in a list. + * + * @param array|string $state + * @param array $references + * + * @throws InvalidArgumentException if a structured state already uses a key the images need + */ + protected function stateWithImages(string|array $state, array $references): array + { + if (is_string($state) || array_is_list($state)) { + $labeled = array_map(fn (int|string $name, string $reference): string => is_string($name) ? "{$name}: {$reference}" : $reference, array_keys($references), $references); + + return [...(is_string($state) ? [$state] : $state), ...$labeled]; + } + + $images = array_filter($references, is_string(...), ARRAY_FILTER_USE_KEY); + + if ($unnamed = array_values(array_filter($references, is_int(...), ARRAY_FILTER_USE_KEY))) { + if (array_key_exists('images', $images)) { + throw new InvalidArgumentException('TypeSafe adds unnamed images to a structured state under [images], so no attachment may be named [images] alongside them.'); + } + + $images['images'] = $unnamed; + } + + if ($duplicates = array_intersect_key($state, $images)) { + throw new InvalidArgumentException('TypeSafe adds images to a structured state under their attachment names, or [images] when unnamed; [' . implode(', ', array_keys($duplicates)) . '] already in the state.'); + } + + return [...$state, ...$images]; + } + + /** + * Map an image to a base64 System One asset. + * + * @throws InvalidArgumentException if the attachment is not a JPEG or PNG image with inline content + */ + protected function mapImage(File|UploadedFile $attachment): array + { + [$image, $content, $mime] = $this->resolveInlineImage($attachment, 'TypeSafe decisions', [ + 'image/jpeg' => 'JPEG', 'image/png' => 'PNG', + ]); + + return array_merge($image->providerOptions(Lab::TypeSafe), [ + 'type' => 'image', + 'format' => $mime === 'image/png' ? 'png' : 'jpg', + 'data_base64' => base64_encode($content), + ]); + } /** * Get the path of the endpoint that answers questions. diff --git a/src/ai/src/PendingResponses/PendingClassification.php b/src/ai/src/PendingResponses/PendingClassification.php index 8c3c9ed895..bdbf5447bd 100644 --- a/src/ai/src/PendingResponses/PendingClassification.php +++ b/src/ai/src/PendingResponses/PendingClassification.php @@ -32,8 +32,8 @@ class PendingClassification /** * Create a new pending classification instance. * - * @param array|string $state - * @param array $attachments + * @param array|string $state + * @param array $attachments * * @throws InvalidArgumentException if the state is blank */ diff --git a/src/ai/src/Prompts/ClassificationPrompt.php b/src/ai/src/Prompts/ClassificationPrompt.php index 418f077b30..7d89420476 100644 --- a/src/ai/src/Prompts/ClassificationPrompt.php +++ b/src/ai/src/Prompts/ClassificationPrompt.php @@ -16,10 +16,10 @@ class ClassificationPrompt implements Countable /** * Create a new classification prompt instance. * - * @param array|string $state + * @param array|string $state * @param array $questions * @param array $providerOptions - * @param array $attachments + * @param array $attachments */ public function __construct( public readonly string|array $state, diff --git a/src/ai/src/Providers/OpenRouterProvider.php b/src/ai/src/Providers/OpenRouterProvider.php index 70dff31db8..14090a4d2b 100644 --- a/src/ai/src/Providers/OpenRouterProvider.php +++ b/src/ai/src/Providers/OpenRouterProvider.php @@ -265,6 +265,14 @@ public function defaultClassificationModel(): string return $this->config['models']['classification']['default'] ?? '~typesafe/jev-latest'; } + /** + * Determine if the provider can classify attachments alongside the state. + */ + public function supportsClassificationAttachments(): bool + { + return true; + } + /** * Get the provider's reranking gateway. */ diff --git a/src/ai/src/Providers/TypeSafeProvider.php b/src/ai/src/Providers/TypeSafeProvider.php index 8a7dae88d6..5fafff55fc 100644 --- a/src/ai/src/Providers/TypeSafeProvider.php +++ b/src/ai/src/Providers/TypeSafeProvider.php @@ -33,6 +33,14 @@ public function defaultClassificationModel(): string return $this->config['models']['classification']['default'] ?? 'jev-latest'; } + /** + * Determine if the provider can classify attachments alongside the state. + */ + public function supportsClassificationAttachments(): bool + { + return true; + } + /** * Get the provider's classification gateway. */ diff --git a/tests/Ai/Feature/ClassificationFakeTest.php b/tests/Ai/Feature/ClassificationFakeTest.php index 321774b8cf..801b26158f 100644 --- a/tests/Ai/Feature/ClassificationFakeTest.php +++ b/tests/Ai/Feature/ClassificationFakeTest.php @@ -12,7 +12,6 @@ use Hypervel\Ai\Enums\Lab; use Hypervel\Ai\Events\ClassificationFailed; use Hypervel\Ai\Events\Classified; -use Hypervel\Ai\Events\Classifying; use Hypervel\Ai\Files\Image; use Hypervel\Ai\Prompts\ClassificationPrompt; use Hypervel\Ai\Responses\Data\BooleanAnswer; @@ -24,6 +23,10 @@ use LogicException; use RuntimeException; +use function Hypervel\Tests\Ai\Fixtures\registerTextOnlyClassifier; + +require_once __DIR__ . '/../Fixtures/helpers.php'; + class ClassificationFakeTest extends TestCase { public function testClassifyRejectsBlankState(): void @@ -175,13 +178,15 @@ public function testFakeRecordsTheAttachmentsAClassificationWasGiven(): void public function testFakesRejectAttachmentsForProvidersThatCannotClassifyThem(): void { + registerTextOnlyClassifier(); + Classification::fake(); $this->expectException(LogicException::class); - $this->expectExceptionMessage('Provider [typesafe] does not support classification attachments.'); + $this->expectExceptionMessage('Provider [text-only] does not support classification attachments.'); Classification::of('Inspect this.', [Image::fromBase64(base64_encode('photo'), 'image/png')]) ->question('damaged', new Boolean('Damaged?')) - ->classify(provider: Lab::TypeSafe); + ->classify(provider: 'text-only'); } public function testCanAssertNothingClassified(): void @@ -239,7 +244,7 @@ public function toArray(): array public function testThrowingSequenceEntryIsConsumed(): void { - Event::fake(); + Event::fake([Classified::class]); $exception = new RuntimeException('Classification failed.'); Classification::fake([fn (): never => throw $exception, ['urgent' => new BooleanAnswer(1.0)]]); @@ -251,16 +256,9 @@ public function testThrowingSequenceEntryIsConsumed(): void $this->assertSame($exception, $caught); } - $started = Event::dispatched(Classifying::class)->first()[0]; - Event::assertDispatched(ClassificationFailed::class, fn (ClassificationFailed $event): bool => $event->exception === $exception - && $event->invocationId === $started->invocationId - && $event->provider === $started->provider - && $event->model === $started->model - && $event->prompt === $started->prompt); Event::assertNotDispatched(Classified::class); $this->assertTrue(Classification::of('Second')->question('urgent', new Boolean('Urgent?'))->classify()->answer('urgent')->isTrue()); - Event::assertDispatchedTimes(ClassificationFailed::class, 1); } public function testSuccessListenerFailureIsNotReportedAsAClassificationFailure(): void diff --git a/tests/Ai/Feature/Providers/OpenAi/ClassificationTest.php b/tests/Ai/Feature/Providers/OpenAi/ClassificationTest.php index c89b189c01..f05059d4e7 100644 --- a/tests/Ai/Feature/Providers/OpenAi/ClassificationTest.php +++ b/tests/Ai/Feature/Providers/OpenAi/ClassificationTest.php @@ -20,6 +20,10 @@ use LogicException; use PHPUnit\Framework\Attributes\DataProvider; +use function Hypervel\Tests\Ai\Fixtures\registerTextOnlyClassifier; + +require_once __DIR__ . '/../../../Fixtures/helpers.php'; + class ClassificationTest extends TestCase { /** @@ -95,7 +99,7 @@ public function testImageAttachmentsAreSentInlineWithTheStateAsAUserMessage(): v ]]); } - public function testRemoteAndStoredImagesAreDownloadedAndSentInline(): void + public function testRemoteAndStoredImagesAreDownloadedAndSentInlineAfterTheirNameLabels(): void { Http::fake([ 'example.com/*' => Http::response('remote-bytes', 200, ['Content-Type' => 'image/gif']), @@ -109,7 +113,9 @@ public function testRemoteAndStoredImagesAreDownloadedAndSentInline(): void Http::assertSent(fn (Request $request): bool => $request->url() === 'https://api.openai.com/v1/decisions' && array_slice(json_decode($request->body(), true)['input'][0]['content'], 1) === [ + ['type' => 'input_text', 'text' => 'front:'], ['type' => 'input_image', 'image_url' => 'data:image/gif;base64,' . base64_encode('remote-bytes')], + ['type' => 'input_text', 'text' => 'back:'], ['type' => 'input_image', 'image_url' => 'data:image/png;base64,' . base64_encode('stored-bytes')], ]); } @@ -181,19 +187,19 @@ public function testClassificationUsesTheConfiguredClassificationModel(): void public function testAttachmentsFailLoudlyInsteadOfFailingOverToAProviderThatCannotClassifyThem(): void { - config(['ai.providers.typesafe.key' => 'test-key']); + registerTextOnlyClassifier(); Http::fake(['api.openai.com/*' => Http::response([], 503)]); $error = null; try { Classification::of('Inspect this.', [Image::fromBase64(base64_encode('photo'), 'image/png')]) - ->question('damaged', new Boolean('Damaged?'))->classify(provider: ['openai', 'typesafe']); + ->question('damaged', new Boolean('Damaged?'))->classify(provider: ['openai', 'text-only']); } catch (LogicException $exception) { $error = $exception; } $this->assertInstanceOf(LogicException::class, $error); - $this->assertStringContainsString('Provider [typesafe] does not support classification attachments.', $error->getMessage()); + $this->assertStringContainsString('Provider [text-only] does not support classification attachments.', $error->getMessage()); Http::assertSentCount(1); } diff --git a/tests/Ai/Feature/Providers/OpenRouter/ClassificationTest.php b/tests/Ai/Feature/Providers/OpenRouter/ClassificationTest.php index 52ac29c1ac..23e02410b9 100644 --- a/tests/Ai/Feature/Providers/OpenRouter/ClassificationTest.php +++ b/tests/Ai/Feature/Providers/OpenRouter/ClassificationTest.php @@ -4,19 +4,23 @@ namespace Hypervel\Tests\Ai\Feature\Providers\OpenRouter; +use Closure; use Hypervel\Ai\Classification; use Hypervel\Ai\Classification\Boolean; use Hypervel\Ai\Classification\Choice; use Hypervel\Ai\Classification\Score; +use Hypervel\Ai\Enums\Lab; use Hypervel\Ai\Files\Image; use Hypervel\Ai\Responses\Data\BooleanAnswer; use Hypervel\Ai\Responses\Data\ChoiceAnswer; use Hypervel\Ai\Responses\Data\ScoreAnswer; use Hypervel\Contracts\Foundation\Application; use Hypervel\Http\Client\Request; +use Hypervel\Http\UploadedFile; use Hypervel\Support\Facades\Http; use Hypervel\Tests\Ai\TestCase; -use LogicException; +use InvalidArgumentException; +use PHPUnit\Framework\Attributes\DataProvider; class ClassificationTest extends TestCase { @@ -172,23 +176,98 @@ public function testAnswersWithoutTheOptionalDistributionFieldsFallBackToTheQues $this->assertSame(1.0, $response['urgency']->normalized()); } - public function testAttachmentsAreRejectedBeforeAnyRequest(): void + public function testImageAttachmentsAreAppendedToTheStateAsDataUrlImagePartsLabeledByName(): void + { + Http::fake(['*' => Http::response($this->fakeDecisionsResponse())]); + + $jpeg = file_get_contents(__DIR__ . '/../../../Fixtures/Images/blue.jpg'); + + Classification::of('A product photo uploaded by a customer.', [ + 'front' => Image::fromPath(__DIR__ . '/../../../Fixtures/Images/blue.jpg')->withProviderOptions(fn ($provider) => $provider === Lab::OpenRouter ? ['custom' => 'value'] : null), + 'back' => UploadedFile::fake()->createWithContent('back.png', 'upload-bytes')->mimeType('image/png'), + ]) + ->question('damaged', new Boolean('Is the product damaged?')) + ->classify(provider: 'openrouter'); + + Http::assertSent(fn (Request $request): bool => $request->url() === 'https://openrouter.ai/api/alpha/decisions' + && ! $request->hasHeader('TypeSafe-Danger-Zone-Images') + && json_decode($request->body(), true)['state'] === [ + 'A product photo uploaded by a customer.', + 'front:', + ['custom' => 'value', 'type' => 'image_url', 'image_url' => ['url' => 'data:image/jpeg;base64,' . base64_encode($jpeg)]], + 'back:', + ['type' => 'image_url', 'image_url' => ['url' => 'data:image/png;base64,' . base64_encode('upload-bytes')]], + ]); + } + + #[DataProvider('statesWithImages')] + public function testImagePartsFollowTheItemsOfAListStateAndTheObjectOfAStructuredState(array $state, array $expected): void + { + Http::fake(['*' => Http::response($this->fakeDecisionsResponse())]); + + Classification::of($state, [Image::fromBase64(base64_encode('photo'), 'image/png')]) + ->question('damaged', new Boolean('Damaged?')) + ->classify(provider: 'openrouter'); + + $part = ['type' => 'image_url', 'image_url' => ['url' => 'data:image/png;base64,' . base64_encode('photo')]]; + + Http::assertSent(fn (Request $request): bool => json_decode($request->body(), true)['state'] === [...$expected, $part]); + } + + /** + * Provide list and structured states with the state items that precede the image parts. + */ + public static function statesWithImages(): array + { + return [ + 'list' => [['first message', 'second message'], ['first message', 'second message']], + 'object' => [['subject' => 'Broken lamp'], [['subject' => 'Broken lamp']]], + ]; + } + + public function testWebpImagesAreSentAsWebpDataUrls(): void + { + Http::fake(['*' => Http::response($this->fakeDecisionsResponse())]); + + Classification::of('Inspect this.', [UploadedFile::fake()->createWithContent('photo.webp', 'webp-bytes')->mimeType('image/webp')]) + ->question('damaged', new Boolean('Damaged?')) + ->classify(provider: 'openrouter', model: 'cloudflare/clef-flash'); + + Http::assertSent(fn (Request $request): bool => json_decode($request->body(), true)['state'] === [ + 'Inspect this.', + ['type' => 'image_url', 'image_url' => ['url' => 'data:image/webp;base64,' . base64_encode('webp-bytes')]], + ]); + } + + #[DataProvider('unsupportedAttachments')] + public function testAttachmentsThatAreNotInlineJpegPngOrWebpImagesAreRejectedBeforeAnyRequest(Closure $attachment, string $message): void { Http::fake(); $exception = null; try { - Classification::of('Inspect this.', [Image::fromBase64(base64_encode('photo'), 'image/png')]) + Classification::of('Inspect this.', [$attachment()]) ->question('damaged', new Boolean('Damaged?')) ->classify(provider: 'openrouter'); - } catch (LogicException $caught) { + } catch (InvalidArgumentException $caught) { $exception = $caught; } - $this->assertInstanceOf(LogicException::class, $exception); - $this->assertSame('Provider [openrouter] does not support classification attachments.', $exception->getMessage()); + $this->assertInstanceOf(InvalidArgumentException::class, $exception); + $this->assertSame($message, $exception->getMessage()); Http::assertNothingSent(); } + + /** + * Provide attachments the OpenRouter decisions endpoint does not accept. + */ + public static function unsupportedAttachments(): array + { + return [ + 'provider image' => [fn () => Image::fromId('file_123'), 'OpenRouter decisions only accept images with inline content; [Hypervel\Ai\Files\ProviderImage] given.'], + 'gif upload' => [fn () => UploadedFile::fake()->createWithContent('photo.gif', 'gif-bytes')->mimeType('image/gif'), 'OpenRouter decisions only accept JPEG, PNG, or WebP images; [image/gif] given.'], + ]; + } } diff --git a/tests/Ai/Feature/Providers/TypeSafe/ClassificationTest.php b/tests/Ai/Feature/Providers/TypeSafe/ClassificationTest.php index af5ed0e029..02e2d0a636 100644 --- a/tests/Ai/Feature/Providers/TypeSafe/ClassificationTest.php +++ b/tests/Ai/Feature/Providers/TypeSafe/ClassificationTest.php @@ -4,20 +4,25 @@ namespace Hypervel\Tests\Ai\Feature\Providers\TypeSafe; +use Closure; use Hypervel\Ai\Classification; use Hypervel\Ai\Classification\Boolean; use Hypervel\Ai\Classification\Choice; use Hypervel\Ai\Classification\Score; use Hypervel\Ai\Exceptions\ProviderOverloadedException; use Hypervel\Ai\Exceptions\RateLimitedException; +use Hypervel\Ai\Files\Image; use Hypervel\Ai\Responses\Data\BooleanAnswer; use Hypervel\Ai\Responses\Data\ChoiceAnswer; use Hypervel\Ai\Responses\Data\ScoreAnswer; use Hypervel\Contracts\Foundation\Application; use Hypervel\Http\Client\Request; use Hypervel\Http\Client\RequestException; +use Hypervel\Http\UploadedFile; use Hypervel\Support\Facades\Http; +use Hypervel\Support\Facades\Storage; use Hypervel\Tests\Ai\TestCase; +use InvalidArgumentException; use PHPUnit\Framework\Attributes\DataProvider; /** @@ -274,4 +279,207 @@ public static function overloadStatuses(): array { return [[529], [503]]; } + + public function testImageAttachmentsAreSentAsReferencedAssetsWithTheImagesHeader(): void + { + Http::fake(['*' => Http::response(fakeTypeSafeResponse())]); + + $jpeg = file_get_contents(__DIR__ . '/../../../Fixtures/Images/blue.jpg'); + $png = file_get_contents(__DIR__ . '/../../../Fixtures/Images/red.png'); + + Classification::of('Inspect this.', [ + Image::fromBase64(base64_encode('base64-bytes'), 'image/jpeg')->withProviderOptions(['custom' => 'value']), + Image::fromPath(__DIR__ . '/../../../Fixtures/Images/blue.jpg'), + UploadedFile::fake()->createWithContent('upload.png', 'upload-bytes')->mimeType('image/png'), + Image::fromBase64(base64_encode($png)), + ]) + ->question('visible_damage', new Boolean('Is the product in the photos damaged?')) + ->classify(provider: 'typesafe'); + + Http::assertSent(function (Request $request) use ($jpeg, $png): bool { + $body = json_decode($request->body(), true); + + return $request->hasHeader('TypeSafe-Danger-Zone-Images', 'true') + && $body['state'] === ['Inspect this.', ...array_keys($body['assets'])] + && array_values($body['assets']) === [ + ['custom' => 'value', 'type' => 'image', 'format' => 'jpg', 'data_base64' => base64_encode('base64-bytes')], + ['type' => 'image', 'format' => 'jpg', 'data_base64' => base64_encode($jpeg)], + ['type' => 'image', 'format' => 'png', 'data_base64' => base64_encode('upload-bytes')], + ['type' => 'image', 'format' => 'png', 'data_base64' => base64_encode($png)], + ]; + }); + } + + public function testRemoteAndStoredImagesAreDownloadedAndLabeledInTheStateByAttachmentName(): void + { + Http::fake([ + 'example.com/*' => Http::response('remote-bytes', 200, ['Content-Type' => 'image/jpeg']), + '*' => Http::response(fakeTypeSafeResponse()), + ]); + + Storage::fake('photos')->put('product.png', 'stored-bytes'); + + Classification::of('Inspect this.', [ + 'front' => Image::fromUrl('https://example.com/product.jpg'), + 'back' => Image::fromStorage('product.png', 'photos'), + ]) + ->question('visible_damage', new Boolean('Is the product in `front` or `back` damaged?')) + ->classify(provider: 'typesafe'); + + Http::assertSent(function (Request $request): bool { + if ($request->url() !== 'https://api.typesafe.ai/v1/systemone') { + return false; + } + + $body = json_decode($request->body(), true); + + [$front, $back] = array_keys($body['assets']); + + return $body['assets'][$front] === ['type' => 'image', 'format' => 'jpg', 'data_base64' => base64_encode('remote-bytes')] + && $body['assets'][$back] === ['type' => 'image', 'format' => 'png', 'data_base64' => base64_encode('stored-bytes')] + && $body['state'] === ['Inspect this.', "front: {$front}", "back: {$back}"]; + }); + } + + #[DataProvider('imageReferenceStates')] + public function testImageReferencesFollowTheShapeOfTheState(string|array $state, array $attachments, Closure $expected): void + { + Http::fake(['*' => Http::response(fakeTypeSafeResponse())]); + + Classification::of($state, $attachments) + ->question('visible_damage', new Boolean('Damaged?')) + ->classify(provider: 'typesafe'); + + Http::assertSent(function (Request $request) use ($expected): bool { + $body = json_decode($request->body(), true); + + return $body['state'] === $expected(...array_keys($body['assets'])); + }); + } + + /** + * Provide states with attachments and the state expected for the generated image references. + */ + public static function imageReferenceStates(): array + { + return [ + 'unnamed images under images in a structured state' => [ + ['subject' => 'Broken lamp'], + [Image::fromBase64(base64_encode('a'), 'image/png')], + fn (string $a): array => ['subject' => 'Broken lamp', 'images' => [$a]], + ], + 'named and unnamed images in a structured state' => [ + ['subject' => 'Broken lamp'], + ['front' => Image::fromBase64(base64_encode('a'), 'image/png'), Image::fromBase64(base64_encode('b'), 'image/png')], + fn (string $a, string $b): array => ['subject' => 'Broken lamp', 'front' => $a, 'images' => [$b]], + ], + 'named and unnamed images in a list state' => [ + ['first', 'second'], + ['front' => Image::fromBase64(base64_encode('a'), 'image/png'), 3 => Image::fromBase64(base64_encode('b'), 'image/png')], + fn (string $a, string $b): array => ['first', 'second', "front: {$a}", $b], + ], + 'an image named images in a structured state' => [ + ['subject' => 'Broken lamp'], + ['images' => Image::fromBase64(base64_encode('a'), 'image/png')], + fn (string $a): array => ['subject' => 'Broken lamp', 'images' => $a], + ], + 'an image named images alongside unnamed images in a list state' => [ + ['first', 'second'], + ['images' => Image::fromBase64(base64_encode('a'), 'image/png'), Image::fromBase64(base64_encode('b'), 'image/png')], + fn (string $a, string $b): array => ['first', 'second', "images: {$a}", $b], + ], + ]; + } + + public function testRequestsWithoutAttachmentsDoNotSendAssetsOrTheImagesHeader(): void + { + Http::fake(['*' => Http::response(fakeTypeSafeResponse())]); + + Classification::of('text')->question('is_urgent', new Boolean('Urgent?'))->classify(provider: 'typesafe'); + + Http::assertSent(fn (Request $request): bool => ! $request->hasHeader('TypeSafe-Danger-Zone-Images') + && ! array_key_exists('assets', json_decode($request->body(), true))); + } + + #[DataProvider('conflictingImageKeys')] + public function testImagesThatNeedAKeyTheStructuredStateAlreadyUsesAreRejectedBeforeAnyRequest(array $state, array $attachments, string $message): void + { + Http::fake(); + + $exception = null; + + try { + Classification::of($state, $attachments) + ->question('damaged', new Boolean('Damaged?')) + ->classify(provider: 'typesafe'); + } catch (InvalidArgumentException $caught) { + $exception = $caught; + } + + $this->assertInstanceOf(InvalidArgumentException::class, $exception); + $this->assertSame($message, $exception->getMessage()); + + Http::assertNothingSent(); + } + + /** + * Provide structured states and attachments that need the same key for different values. + */ + public static function conflictingImageKeys(): array + { + return [ + 'named' => [ + ['photo' => 'caption'], + ['photo' => Image::fromBase64(base64_encode('photo'), 'image/png')], + 'TypeSafe adds images to a structured state under their attachment names, or [images] when unnamed; [photo] already in the state.', + ], + 'unnamed' => [ + ['images' => 'none attached'], + [Image::fromBase64(base64_encode('photo'), 'image/png')], + 'TypeSafe adds images to a structured state under their attachment names, or [images] when unnamed; [images] already in the state.', + ], + 'an image named images alongside unnamed images' => [ + ['subject' => 'Compare photos'], + ['images' => Image::fromBase64(base64_encode('a'), 'image/png'), Image::fromBase64(base64_encode('b'), 'image/png')], + 'TypeSafe adds unnamed images to a structured state under [images], so no attachment may be named [images] alongside them.', + ], + ]; + } + + #[DataProvider('unsupportedAttachments')] + public function testAttachmentsThatAreNotSupportedImagesAreRejectedBeforeAnyRequest(Closure $attachment, string $message): void + { + Http::fake([ + 'example.com/*' => Http::response('', 200, ['Content-Type' => 'text/html']), + ]); + + $exception = null; + + try { + Classification::of('Inspect this.', [$attachment()]) + ->question('visible_damage', new Boolean('Damaged?')) + ->classify(provider: 'typesafe'); + } catch (InvalidArgumentException $caught) { + $exception = $caught; + } + + $this->assertInstanceOf(InvalidArgumentException::class, $exception); + $this->assertStringContainsString($message, $exception->getMessage()); + + Http::assertNotSent(fn (Request $request): bool => str_contains($request->url(), 'api.typesafe.ai')); + } + + /** + * Provide attachments the TypeSafe decisions endpoint does not accept. + */ + public static function unsupportedAttachments(): array + { + return [ + 'provider image' => [fn () => Image::fromId('file_123'), 'TypeSafe decisions only accept images with inline content; [Hypervel\Ai\Files\ProviderImage] given.'], + 'pdf upload' => [fn () => UploadedFile::fake()->createWithContent('invoice.pdf', 'pdf-bytes')->mimeType('application/pdf'), '[application/pdf] given.'], + 'webp upload' => [fn () => UploadedFile::fake()->createWithContent('photo.webp', 'webp-bytes')->mimeType('image/webp'), 'TypeSafe decisions only accept JPEG or PNG images; [image/webp] given.'], + 'gif file' => [fn () => Image::fromPath(__DIR__ . '/../../../Fixtures/Images/red.png', 'image/gif'), '[image/gif] given.'], + 'html served as a remote image' => [fn () => Image::fromUrl('https://example.com/product.png'), '[text/html] given.'], + ]; + } } diff --git a/tests/Ai/Fixtures/Images/blue.jpg b/tests/Ai/Fixtures/Images/blue.jpg index afe59ded8eb6fc9bbbba670e7c713ccb3aa79915..58ccbcfa9b12ddfa7399513ca6949fd9d49ee824 100644 GIT binary patch literal 2394 zcmex=#*GyH$c zU=fg!Uy@&(kzb%-tY^Sr;pFMd0x}&4z-(18Q{akh&&_^@O)M`PIA^^1;Z*nk0D~Y0 zqYk4EGoum%lOQ9rAmjfd4Dvu%urh)HSR0UFWMXDvWn<^y|HF0u@iAzXIsj8`K zXlj|5nweWzS~We&gn?hmRgV zdHU@6i$mSee*Oaa3mBddZvjb&&(QoO2=o^d3kx#~JIG&*OywYt3$m~( z8nOvF2C^p>3M&~ka)>xhT)6Qdr?PR-2hpUWi(FzVCJ$9Vg1iRy8F3zKBFkrRk0JbZ zo57iZ2@IH-n3!3anVDJHSb>0zgAD{YxH!Oo3krC^fDZ)t;ov{e5JsRGEG#VSKqSD; z$t^%K_%Rv4vC&x6#Xt@G{{r=myhwc`FHzsf m%hWgW3iXY=N_``*QQyex)HiZ8T~aTPkEToNh2_vlmp1`x<|VWM delta 323 zcmca5^qIB(|Be3-7zDiBJlz-=85tQ8fRN$;KL%^(AXmo_{~#-c^c3B!g4A?{GDAHB z1&z|;%)E33PcL@`uK-tfg)%cEO&x{8(!`w1l1c?z1xo|2|F;>O8JL-vKnMs}fQXfY zm6e5sm5ZI7jf01ahliVso12$UNPw45FqMy+TR==eP*_A%RFsEbTtZAlLP$hZ1Y`;$ zGf)!?D<>-}rwA`MuL#ND{{aR;4h9DXhmALPGltw^;9+KDU=m~&WUy!W+0~`7{+;n> Ohecf)3}j;F|2F}KNk^gp diff --git a/tests/Ai/Fixtures/Images/red.png b/tests/Ai/Fixtures/Images/red.png index e735c5e42bf2ec91e8b8ddac4983ec4ebebba2be..3446e015f998cb071233ff8aff64568d0f375816 100644 GIT binary patch literal 2019 zcmeAS@N?(olHy`uVBq!ia0y~yVAKI&4kn<;H+e}VAjMc5NmqCHY z+0o&|pKd1}$D_Y@N@f(rexDoJ$I$5apyc8tPJtgxZZG&bjxf1Nh%3x%5d6i+w6Q_( z7;A@mLQ7qPgHA$AA>$)XiG}hC0^AabF%1b5sYS!(UM@aj2x`h!2#x* VcFYB~4{w31Sx;9#mvv4FO#pG6Tgm_c literal 176 zcmeAS@N?(olHy`uVBq!ia0vp^4j|0I1SD0tpLGH$&H|6fVg?4jBOuH;Rhv&5D468w z;uw ['driver' => 'text-only', 'key' => 'test-key']]); + + Ai::extend('text-only', fn (Application $app, array $config): TypeSafeProvider => new class($config, $app->make(Dispatcher::class)) extends TypeSafeProvider { + /** + * Determine if the provider can classify attachments alongside the state. + */ + public function supportsClassificationAttachments(): bool + { + return false; + } + }); +} + function sentRequest(): Request { [$request] = Http::recorded()->first(); diff --git a/tests/Integration/Ai/ClassificationTest.php b/tests/Integration/Ai/ClassificationTest.php index 749da318ff..d852de196f 100644 --- a/tests/Integration/Ai/ClassificationTest.php +++ b/tests/Integration/Ai/ClassificationTest.php @@ -83,4 +83,22 @@ public static function imageProviders(): iterable yield $name . '-jpeg' => [...$provider, 'blue.jpg', 'blue']; } } + + #[DataProviderExternal(Providers::class, 'imageClassification')] + public function testNamedImagesCanBeReferredToByName(string $provider, string $apiKey): void + { + requiresApiKey($apiKey); + + $response = Classification::of('Two product photos uploaded by a customer.', [ + 'front' => Image::fromPath(__DIR__ . '/../../Ai/Fixtures/Images/red.png'), + 'back' => Image::fromPath(__DIR__ . '/../../Ai/Fixtures/Images/blue.jpg'), + ]) + ->questions([ + 'front_color' => new Choice('What color is the background of `front`?', ['red' => null, 'green' => null, 'blue' => null]), + 'back_color' => new Choice('What color is the background of `back`?', ['red' => null, 'green' => null, 'blue' => null]), + ])->classify(provider: $provider); + + $this->assertSame('red', $response['front_color']->choice); + $this->assertSame('blue', $response['back_color']->choice); + } } diff --git a/tests/Integration/Ai/Fixtures/Providers.php b/tests/Integration/Ai/Fixtures/Providers.php index 3e6c27f81a..6701806878 100644 --- a/tests/Integration/Ai/Fixtures/Providers.php +++ b/tests/Integration/Ai/Fixtures/Providers.php @@ -138,6 +138,8 @@ public static function imageClassification(): array { return [ 'openai' => ['openai', 'OPENAI_API_KEY'], + 'openrouter' => ['openrouter', 'OPENROUTER_API_KEY'], + 'typesafe' => ['typesafe', 'TYPESAFE_API_KEY'], ]; } diff --git a/types/Ai/Classification.php b/types/Ai/Classification.php new file mode 100644 index 0000000000..657a71e3b4 --- /dev/null +++ b/types/Ai/Classification.php @@ -0,0 +1,16 @@ + 'Broken lamp'])); +assertType('Hypervel\Ai\PendingResponses\PendingClassification', Classification::of(['first message', 'second message'])); +assertType('Hypervel\Ai\PendingResponses\PendingClassification', Classification::of('Two product photos.', [ + 'front' => Image::fromBase64('cGhvdG8=', 'image/png'), + Image::fromBase64('cGhvdG8=', 'image/png'), +])); From a05bb7d0b9eefdf9af08bbf80701aedf6864fe29 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:21 +0000 Subject: [PATCH 07/16] Bump source-map-js in the AI workbench Port laravel/ai#1142 (laravel/ai 1.x at d44ae14d17fb), updating the workbench lockfile to source-map-js 1.2.2 with npm. --- dogfood/ai/workbench/package-lock.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dogfood/ai/workbench/package-lock.json b/dogfood/ai/workbench/package-lock.json index de24e125b8..1ddad7b1ec 100644 --- a/dogfood/ai/workbench/package-lock.json +++ b/dogfood/ai/workbench/package-lock.json @@ -3005,9 +3005,9 @@ "license": "MIT" }, "node_modules/source-map-js": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", - "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.2.tgz", + "integrity": "sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==", "dev": true, "license": "BSD-3-Clause", "engines": { From bbba0eedcb5038388659cd10169c9c4135f9cd0c Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:22 +0000 Subject: [PATCH 08/16] Encode provider resource IDs and keep raw file responses Port laravel/ai#1140 and laravel/ai#1146 (laravel/ai 1.x at d44ae14d17fb), which change the same file gateways. File, store and document IDs went into request paths unchanged, so an ID containing "/", "?", "#" or ".." could reach a different endpoint, such as a file ID that deletes a vector store. EncodesResourceIds now encodes each ID as one path segment and rejects empty and dot-segment IDs, in the OpenAI, Azure OpenAI, Anthropic, OpenRouter and Gemini gateways. Gemini still strips its own prefixes, but a document path from another store no longer names that document. Stored and retrieved file responses now keep the provider's raw HTTP response, which is dropped when they are serialized, so provider metadata such as a file's size is available. Port upstream's resource ID and gateway tests. Formatting, static analysis and the AI suite pass. --- .../Anthropic/AnthropicFileGateway.php | 12 +- .../Gateway/Concerns/EncodesResourceIds.php | 24 +++ .../src/Gateway/Gemini/GeminiFileGateway.php | 21 ++- .../src/Gateway/Gemini/GeminiStoreGateway.php | 24 +-- .../src/Gateway/OpenAi/OpenAiFileGateway.php | 12 +- .../src/Gateway/OpenAi/OpenAiStoreGateway.php | 10 +- .../OpenRouter/OpenRouterFileGateway.php | 12 +- src/ai/src/Responses/FileResponse.php | 4 + src/ai/src/Responses/StoredFileResponse.php | 3 + tests/Ai/Feature/ProviderResourceIdTest.php | 172 ++++++++++++++++++ .../Providers/Anthropic/FileGatewayTest.php | 14 ++ .../Providers/AzureOpenAi/FileGatewayTest.php | 13 ++ .../AzureOpenAi/StoreGatewayTest.php | 13 ++ .../Providers/Gemini/FileGatewayTest.php | 29 +++ .../Providers/Gemini/StoreGatewayTest.php | 38 ++++ .../Providers/OpenAi/FileGatewayTest.php | 70 +++++++ .../Providers/OpenAi/StoreGatewayTest.php | 25 +++ .../Providers/OpenRouter/FileGatewayTest.php | 2 + 18 files changed, 456 insertions(+), 42 deletions(-) create mode 100644 src/ai/src/Gateway/Concerns/EncodesResourceIds.php create mode 100644 tests/Ai/Feature/ProviderResourceIdTest.php diff --git a/src/ai/src/Gateway/Anthropic/AnthropicFileGateway.php b/src/ai/src/Gateway/Anthropic/AnthropicFileGateway.php index cec2bf4fd5..82ee376d88 100644 --- a/src/ai/src/Gateway/Anthropic/AnthropicFileGateway.php +++ b/src/ai/src/Gateway/Anthropic/AnthropicFileGateway.php @@ -9,6 +9,7 @@ use Hypervel\Ai\Contracts\Providers\FileProvider; use Hypervel\Ai\Enums\Lab; use Hypervel\Ai\Gateway\Anthropic\Concerns\CreatesAnthropicClient; +use Hypervel\Ai\Gateway\Concerns\EncodesResourceIds; use Hypervel\Ai\Gateway\Concerns\HandlesFailoverErrors; use Hypervel\Ai\Gateway\Concerns\PreparesStorableFiles; use Hypervel\Ai\Providers\Provider; @@ -19,6 +20,7 @@ class AnthropicFileGateway implements FileGateway { use CreatesAnthropicClient; + use EncodesResourceIds; use HandlesFailoverErrors; use PreparesStorableFiles; @@ -30,13 +32,13 @@ public function getFile(FileProvider $provider, string $fileId): FileResponse /** @var FileProvider&Provider $provider */ $response = $this->withErrorHandling( $provider->name(), - fn () => $this->client($provider)->get("files/{$fileId}"), + fn () => $this->client($provider)->get("files/{$this->resourceId($fileId)}"), ); - return new FileResponse( + return (new FileResponse( id: $response->json('id'), mimeType: $response->json('mime_type'), - ); + ))->withRawResponse($response); } /** @@ -60,7 +62,7 @@ public function putFile( ->post('files', $providerOptions), ); - return new StoredFileResponse($response->json('id')); + return (new StoredFileResponse($response->json('id')))->withRawResponse($response); } /** @@ -71,7 +73,7 @@ public function deleteFile(FileProvider $provider, string $fileId): void /** @var FileProvider&Provider $provider */ $this->withErrorHandling( $provider->name(), - fn () => $this->client($provider)->delete("files/{$fileId}"), + fn () => $this->client($provider)->delete("files/{$this->resourceId($fileId)}"), ); } diff --git a/src/ai/src/Gateway/Concerns/EncodesResourceIds.php b/src/ai/src/Gateway/Concerns/EncodesResourceIds.php new file mode 100644 index 0000000000..5e93b88fbd --- /dev/null +++ b/src/ai/src/Gateway/Concerns/EncodesResourceIds.php @@ -0,0 +1,24 @@ +normalizeFileId($fileId); $response = $this->withErrorHandling( $provider->name(), fn () => $this->client($provider)->get($this->baseUrl($provider) . "/{$fileId}")->throw(), ); - return new FileResponse( + return (new FileResponse( id: $response->json('name'), mimeType: $response->json('mimeType'), - ); + ))->withRawResponse($response); } /** @@ -66,7 +69,7 @@ public function putFile( 'file' => ['display_name' => $name], ], $providerOptions))->throw()); - return new StoredFileResponse($response->json('file.name')); + return (new StoredFileResponse($response->json('file.name')))->withRawResponse($response); } /** @@ -75,7 +78,7 @@ public function putFile( public function deleteFile(FileProvider $provider, string $fileId): void { /** @var FileProvider&Provider $provider */ - $fileId = str_starts_with($fileId, 'files/') ? $fileId : "files/{$fileId}"; + $fileId = $this->normalizeFileId($fileId); $this->withErrorHandling( $provider->name(), @@ -83,6 +86,14 @@ public function deleteFile(FileProvider $provider, string $fileId): void ); } + /** + * Normalize the file ID to include the resource prefix. + */ + protected function normalizeFileId(string $fileId): string + { + return 'files/' . $this->resourceId(Str::chopStart($fileId, 'files/')); + } + /** * Get an HTTP client for the Gemini Files API. */ diff --git a/src/ai/src/Gateway/Gemini/GeminiStoreGateway.php b/src/ai/src/Gateway/Gemini/GeminiStoreGateway.php index 23b785e638..e48f3f09fb 100644 --- a/src/ai/src/Gateway/Gemini/GeminiStoreGateway.php +++ b/src/ai/src/Gateway/Gemini/GeminiStoreGateway.php @@ -10,6 +10,7 @@ use Hypervel\Ai\Contracts\Providers\StoreProvider; use Hypervel\Ai\Exceptions\AiException; use Hypervel\Ai\Gateway\Concerns\CreatesClient; +use Hypervel\Ai\Gateway\Concerns\EncodesResourceIds; use Hypervel\Ai\Gateway\Concerns\HandlesFailoverErrors; use Hypervel\Ai\Providers\Provider; use Hypervel\Ai\Responses\Data\StoreFileCounts; @@ -19,10 +20,12 @@ use Hypervel\Http\Client\Response; use Hypervel\Support\Collection; use Hypervel\Support\Sleep; +use Hypervel\Support\Str; class GeminiStoreGateway implements StoreGateway { use CreatesClient; + use EncodesResourceIds; use HandlesFailoverErrors; /** @@ -209,9 +212,7 @@ protected function baseUrl(Provider $provider): string */ protected function normalizeStoreId(string $storeId): string { - return str_starts_with($storeId, 'fileSearchStores/') - ? $storeId - : "fileSearchStores/{$storeId}"; + return 'fileSearchStores/' . $this->resourceId(Str::chopStart($storeId, 'fileSearchStores/')); } /** @@ -219,9 +220,7 @@ protected function normalizeStoreId(string $storeId): string */ protected function normalizeFileId(string $fileId): string { - return str_starts_with($fileId, 'files/') - ? $fileId - : "files/{$fileId}"; + return 'files/' . $this->resourceId(Str::chopStart($fileId, 'files/')); } /** @@ -229,17 +228,6 @@ protected function normalizeFileId(string $fileId): string */ protected function normalizeDocumentId(string $storeId, string $documentId): string { - // Already a full document path... - if (str_starts_with($documentId, 'fileSearchStores/')) { - return $documentId; - } - - $documentId = match (true) { - str_starts_with($documentId, 'documents/') => substr($documentId, 10), - str_starts_with($documentId, 'files/') => substr($documentId, 6), - default => $documentId, - }; - - return "{$storeId}/documents/{$documentId}"; + return "{$storeId}/documents/" . $this->resourceId(Str::chopStart($documentId, ["{$storeId}/documents/", 'documents/', 'files/'])); } } diff --git a/src/ai/src/Gateway/OpenAi/OpenAiFileGateway.php b/src/ai/src/Gateway/OpenAi/OpenAiFileGateway.php index eef06bdc1e..3952fb2a1c 100644 --- a/src/ai/src/Gateway/OpenAi/OpenAiFileGateway.php +++ b/src/ai/src/Gateway/OpenAi/OpenAiFileGateway.php @@ -8,6 +8,7 @@ use Hypervel\Ai\Contracts\Gateway\FileGateway; use Hypervel\Ai\Contracts\Providers\FileProvider; use Hypervel\Ai\Enums\Lab; +use Hypervel\Ai\Gateway\Concerns\EncodesResourceIds; use Hypervel\Ai\Gateway\Concerns\HandlesFailoverErrors; use Hypervel\Ai\Gateway\Concerns\PreparesStorableFiles; use Hypervel\Ai\Gateway\OpenAi\Concerns\CreatesOpenAiClient; @@ -18,6 +19,7 @@ class OpenAiFileGateway implements FileGateway { use CreatesOpenAiClient; + use EncodesResourceIds; use HandlesFailoverErrors; use PreparesStorableFiles; @@ -30,12 +32,12 @@ public function getFile(FileProvider $provider, string $fileId): FileResponse $response = $this->withErrorHandling( $provider->name(), fn () => $this->client($provider) - ->get("files/{$fileId}") + ->get("files/{$this->resourceId($fileId)}") ); - return new FileResponse( + return (new FileResponse( id: $response->json('id'), - ); + ))->withRawResponse($response); } /** @@ -62,7 +64,7 @@ public function putFile( )) ); - return new StoredFileResponse($response->json('id')); + return (new StoredFileResponse($response->json('id')))->withRawResponse($response); } /** @@ -74,7 +76,7 @@ public function deleteFile(FileProvider $provider, string $fileId): void $this->withErrorHandling( $provider->name(), fn () => $this->client($provider) - ->delete("files/{$fileId}") + ->delete("files/{$this->resourceId($fileId)}") ); } diff --git a/src/ai/src/Gateway/OpenAi/OpenAiStoreGateway.php b/src/ai/src/Gateway/OpenAi/OpenAiStoreGateway.php index dd3b26a127..fffcbd074e 100644 --- a/src/ai/src/Gateway/OpenAi/OpenAiStoreGateway.php +++ b/src/ai/src/Gateway/OpenAi/OpenAiStoreGateway.php @@ -8,6 +8,7 @@ use Hypervel\Ai\Contracts\Gateway\StoreGateway; use Hypervel\Ai\Contracts\Providers\FileProvider; use Hypervel\Ai\Contracts\Providers\StoreProvider; +use Hypervel\Ai\Gateway\Concerns\EncodesResourceIds; use Hypervel\Ai\Gateway\Concerns\HandlesFailoverErrors; use Hypervel\Ai\Gateway\OpenAi\Concerns\CreatesOpenAiClient; use Hypervel\Ai\Providers\Provider; @@ -19,6 +20,7 @@ class OpenAiStoreGateway implements StoreGateway { use CreatesOpenAiClient; + use EncodesResourceIds; use HandlesFailoverErrors; /** @@ -30,7 +32,7 @@ public function getStore(StoreProvider $provider, string $storeId): Store $response = $this->withErrorHandling( $provider->name(), fn () => $this->client($provider) - ->get("vector_stores/{$storeId}") + ->get("vector_stores/{$this->resourceId($storeId)}") ); return new Store( @@ -87,7 +89,7 @@ public function addFile(StoreProvider $provider, string $storeId, string $fileId $response = $this->withErrorHandling( $provider->name(), fn () => $this->client($provider) - ->post("vector_stores/{$storeId}/files", array_filter([ + ->post("vector_stores/{$this->resourceId($storeId)}/files", array_filter([ 'file_id' => $fileId, 'attributes' => filled($metadata) ? (object) $metadata : null, ])) @@ -105,7 +107,7 @@ public function removeFile(StoreProvider $provider, string $storeId, string $doc $response = $this->withErrorHandling( $provider->name(), fn () => $this->client($provider) - ->delete("vector_stores/{$storeId}/files/{$documentId}") + ->delete("vector_stores/{$this->resourceId($storeId)}/files/{$this->resourceId($documentId)}") ); return $response->json('deleted', false); @@ -120,7 +122,7 @@ public function deleteStore(StoreProvider $provider, string $storeId): bool $response = $this->withErrorHandling( $provider->name(), fn () => $this->client($provider) - ->delete("vector_stores/{$storeId}") + ->delete("vector_stores/{$this->resourceId($storeId)}") ); return $response->json('deleted', false); diff --git a/src/ai/src/Gateway/OpenRouter/OpenRouterFileGateway.php b/src/ai/src/Gateway/OpenRouter/OpenRouterFileGateway.php index c55f599151..2a5c2bd4f1 100644 --- a/src/ai/src/Gateway/OpenRouter/OpenRouterFileGateway.php +++ b/src/ai/src/Gateway/OpenRouter/OpenRouterFileGateway.php @@ -8,6 +8,7 @@ use Hypervel\Ai\Contracts\Gateway\FileGateway; use Hypervel\Ai\Contracts\Providers\FileProvider; use Hypervel\Ai\Enums\Lab; +use Hypervel\Ai\Gateway\Concerns\EncodesResourceIds; use Hypervel\Ai\Gateway\Concerns\HandlesFailoverErrors; use Hypervel\Ai\Gateway\Concerns\PreparesStorableFiles; use Hypervel\Ai\Gateway\OpenRouter\Concerns\CreatesOpenRouterClient; @@ -18,6 +19,7 @@ class OpenRouterFileGateway implements FileGateway { use CreatesOpenRouterClient; + use EncodesResourceIds; use HandlesFailoverErrors; use PreparesStorableFiles; @@ -30,13 +32,13 @@ public function getFile(FileProvider $provider, string $fileId): FileResponse $response = $this->withErrorHandling( $provider->name(), fn () => $this->client($provider) - ->get("files/{$fileId}") + ->get("files/{$this->resourceId($fileId)}") ); - return new FileResponse( + return (new FileResponse( id: $response->json('id'), mimeType: $response->json('mime_type'), - ); + ))->withRawResponse($response); } /** @@ -60,7 +62,7 @@ public function putFile( ->post('files', $providerOptions) ); - return new StoredFileResponse($response->json('id')); + return (new StoredFileResponse($response->json('id')))->withRawResponse($response); } /** @@ -80,7 +82,7 @@ public function deleteFile(FileProvider $provider, string $fileId): void $this->withErrorHandling( $provider->name(), fn () => $this->client($provider) - ->delete("files/{$fileId}") + ->delete("files/{$this->resourceId($fileId)}") ); } } diff --git a/src/ai/src/Responses/FileResponse.php b/src/ai/src/Responses/FileResponse.php index f6ce021043..2e137fb2df 100644 --- a/src/ai/src/Responses/FileResponse.php +++ b/src/ai/src/Responses/FileResponse.php @@ -4,8 +4,12 @@ namespace Hypervel\Ai\Responses; +use Hypervel\Ai\Responses\Concerns\HasRawResponse; + class FileResponse { + use HasRawResponse; + public readonly ?string $mime; /** diff --git a/src/ai/src/Responses/StoredFileResponse.php b/src/ai/src/Responses/StoredFileResponse.php index d429ecc8a3..7743909545 100644 --- a/src/ai/src/Responses/StoredFileResponse.php +++ b/src/ai/src/Responses/StoredFileResponse.php @@ -5,9 +5,12 @@ namespace Hypervel\Ai\Responses; use Hypervel\Ai\Contracts\Files\HasProviderId; +use Hypervel\Ai\Responses\Concerns\HasRawResponse; class StoredFileResponse implements HasProviderId { + use HasRawResponse; + /** * Create a stored file response. */ diff --git a/tests/Ai/Feature/ProviderResourceIdTest.php b/tests/Ai/Feature/ProviderResourceIdTest.php new file mode 100644 index 0000000000..31977addbf --- /dev/null +++ b/tests/Ai/Feature/ProviderResourceIdTest.php @@ -0,0 +1,172 @@ +make('config'); + + foreach (['openai', 'anthropic', 'openrouter', 'gemini'] as $provider) { + $config->set("ai.providers.{$provider}.key", 'test-key'); + } + + $config->set('ai.providers.azure.key', 'test-key'); + $config->set('ai.providers.azure.url', 'https://test-resource.openai.azure.com'); + } + + /** + * Fake every provider request. + */ + protected function setUp(): void + { + parent::setUp(); + + Http::fake(); + } + + #[DataProvider('fileIds')] + public function testFileIdsAreSentAsASinglePathSegment(string $provider, string $url, string $id, string $encoded): void + { + Files::delete($id, provider: $provider); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' && $request->url() === sprintf($url, $encoded)); + } + + /** + * Provide each file gateway's delete URL with every unsafe ID. + */ + public static function fileIds(): iterable + { + yield from self::withUnsafeIds([ + 'openai' => ['openai', 'https://api.openai.com/v1/files/%s'], + 'azure' => ['azure', 'https://test-resource.openai.azure.com/openai/v1/files/%s'], + 'anthropic' => ['anthropic', 'https://api.anthropic.com/v1/files/%s'], + 'openrouter' => ['openrouter', 'https://openrouter.ai/api/v1/files/%s'], + 'gemini' => ['gemini', 'https://generativelanguage.googleapis.com/v1beta/files/%s'], + ]); + } + + #[DataProvider('storeIds')] + public function testStoreIdsAreSentAsASinglePathSegment(string $provider, string $url, string $id, string $encoded): void + { + Stores::delete($id, provider: $provider); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' && $request->url() === sprintf($url, $encoded)); + } + + /** + * Provide each store gateway's delete URL with every unsafe ID. + */ + public static function storeIds(): iterable + { + yield from self::withUnsafeIds([ + 'openai' => ['openai', 'https://api.openai.com/v1/vector_stores/%s'], + 'azure' => ['azure', 'https://test-resource.openai.azure.com/openai/v1/vector_stores/%s'], + 'gemini' => ['gemini', 'https://generativelanguage.googleapis.com/v1beta/fileSearchStores/%s?force=true'], + ]); + } + + #[DataProvider('storeDocumentIds')] + public function testStoreDocumentIdsAreSentAsASinglePathSegment(string $provider, string $url, string $id, string $encoded): void + { + $instance = $this->app->make(AiManager::class)->instance($provider); + + $instance->storeGateway()->removeFile($instance, 'store123', $id); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' && $request->url() === sprintf($url, $encoded)); + } + + /** + * Provide each store gateway's document removal URL with every unsafe ID. + */ + public static function storeDocumentIds(): iterable + { + yield from self::withUnsafeIds([ + 'openai' => ['openai', 'https://api.openai.com/v1/vector_stores/store123/files/%s'], + 'azure' => ['azure', 'https://test-resource.openai.azure.com/openai/v1/vector_stores/store123/files/%s'], + 'gemini' => ['gemini', 'https://generativelanguage.googleapis.com/v1beta/fileSearchStores/store123/documents/%s?force=true'], + ]); + } + + #[DataProvider('prefixedGeminiIds')] + public function testGeminiResourceIdsCannotNameADifferentResourceThroughTheirPrefix(string $operation, string $id, string $url): void + { + $gemini = $this->app->make(AiManager::class)->instance('gemini'); + + match ($operation) { + 'file' => Files::delete($id, provider: 'gemini'), + 'store' => Stores::delete($id, provider: 'gemini'), + 'document' => $gemini->storeGateway()->removeFile($gemini, 'store123', $id), + }; + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' && $request->url() === "https://generativelanguage.googleapis.com/v1beta/{$url}"); + } + + /** + * Provide Gemini IDs whose prefix names a different resource. + */ + public static function prefixedGeminiIds(): array + { + return [ + 'file pointing at a store' => ['file', 'files/../fileSearchStores/victim', 'files/..%2FfileSearchStores%2Fvictim'], + 'store pointing at a document' => ['store', 'fileSearchStores/victim/documents/doc456', 'fileSearchStores/victim%2Fdocuments%2Fdoc456?force=true'], + 'document pointing at a store' => ['document', 'fileSearchStores/victim', 'fileSearchStores/store123/documents/fileSearchStores%2Fvictim?force=true'], + 'document in a different store' => ['document', 'fileSearchStores/other/documents/doc456', 'fileSearchStores/store123/documents/fileSearchStores%2Fother%2Fdocuments%2Fdoc456?force=true'], + ]; + } + + #[TestWith([''])] + #[TestWith(['.'])] + #[TestWith(['..'])] + public function testDotSegmentIdsAreRejectedBeforeARequestIsSent(string $id): void + { + try { + Files::delete($id, provider: 'openai'); + $this->fail('The ID must be rejected.'); + } catch (InvalidArgumentException $exception) { + $this->assertSame("[{$id}] is not a valid provider resource ID.", $exception->getMessage()); + } + + Http::assertNothingSent(); + } + + /** + * Combine each gateway URL with every unsafe ID and its single-segment encoding. + * + * @param array $gateways + */ + private static function withUnsafeIds(array $gateways): iterable + { + $ids = [ + 'parent traversal' => ['../../vector_stores/victim', '..%2F..%2Fvector_stores%2Fvictim'], + 'nested path' => ['abc/def', 'abc%2Fdef'], + 'query injection' => ['abc?force=true', 'abc%3Fforce%3Dtrue'], + 'fragment' => ['abc#x', 'abc%23x'], + 'encoded slash' => ['abc%2F..', 'abc%252F..'], + ]; + + foreach ($gateways as $gateway => $route) { + foreach ($ids as $name => $id) { + yield "{$gateway} {$name}" => [...$route, ...$id]; + } + } + } +} diff --git a/tests/Ai/Feature/Providers/Anthropic/FileGatewayTest.php b/tests/Ai/Feature/Providers/Anthropic/FileGatewayTest.php index 8d3ddefcf0..7210332818 100644 --- a/tests/Ai/Feature/Providers/Anthropic/FileGatewayTest.php +++ b/tests/Ai/Feature/Providers/Anthropic/FileGatewayTest.php @@ -35,6 +35,7 @@ public function testGetFileSendsCorrectRequest(): void $response = Files::get('file-abc123', provider: 'anthropic'); $this->assertSame('file-abc123', $response->id); + $this->assertNotNull($response->raw); Http::assertSent(fn (Request $request): bool => $request->method() === 'GET' && $request->url() === 'https://api.anthropic.com/v1/files/file-abc123' @@ -52,6 +53,7 @@ public function testPutFileSendsMultipartUpload(): void ); $this->assertSame('file-uploaded123', $response->id); + $this->assertNotNull($response->raw); $request = sentRequest(); @@ -86,4 +88,16 @@ public function testDeleteFileSendsCorrectRequest(): void && $request->url() === 'https://api.anthropic.com/v1/files/file-abc123' && $request->hasHeader('x-api-key', 'test-key')); } + + public function testDeleteFileEncodesTheIdAsASinglePathSegment(): void + { + Http::fake([ + 'api.anthropic.com/*' => Http::response(['id' => 'file-abc123']), + ]); + + Files::delete('../skills/skill-123?beta=true', provider: 'anthropic'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && $request->url() === 'https://api.anthropic.com/v1/files/..%2Fskills%2Fskill-123%3Fbeta%3Dtrue'); + } } diff --git a/tests/Ai/Feature/Providers/AzureOpenAi/FileGatewayTest.php b/tests/Ai/Feature/Providers/AzureOpenAi/FileGatewayTest.php index 48cfc51e6b..f34cd5b159 100644 --- a/tests/Ai/Feature/Providers/AzureOpenAi/FileGatewayTest.php +++ b/tests/Ai/Feature/Providers/AzureOpenAi/FileGatewayTest.php @@ -5,6 +5,7 @@ namespace Hypervel\Tests\Ai\Feature\Providers\AzureOpenAi; use Hypervel\Ai\Enums\Lab; +use Hypervel\Ai\Files; use Hypervel\Ai\Files\Document; use Hypervel\Contracts\Foundation\Application; use Hypervel\Http\Client\Request; @@ -66,4 +67,16 @@ public function testProviderOptionsAreResolvedWithTheAzureKeyNotOpenAi(): void $this->assertSame('batch', multipartField(sentRequest(), 'purpose')); } + + public function testDeleteFileEncodesTheIdAsASinglePathSegment(): void + { + Http::fake([ + 'test-resource.openai.azure.com/*' => Http::response(['deleted' => true]), + ]); + + Files::delete('../vector_stores/vs-123', provider: 'azure'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && $request->url() === 'https://test-resource.openai.azure.com/openai/v1/files/..%2Fvector_stores%2Fvs-123'); + } } diff --git a/tests/Ai/Feature/Providers/AzureOpenAi/StoreGatewayTest.php b/tests/Ai/Feature/Providers/AzureOpenAi/StoreGatewayTest.php index 485f233952..f412fd649c 100644 --- a/tests/Ai/Feature/Providers/AzureOpenAi/StoreGatewayTest.php +++ b/tests/Ai/Feature/Providers/AzureOpenAi/StoreGatewayTest.php @@ -86,4 +86,17 @@ public function testFileSearchWithoutFiltersReturnsVectorStoreIds(): void $this->assertSame(['vector_store_ids' => ['vs-123']], Ai::storeProvider('azure')->fileSearchToolOptions($search)); } + + public function testRemoveFileEncodesTheStoreAndDocumentIdsAsSinglePathSegments(): void + { + Http::fake([ + 'test-resource.openai.azure.com/*' => Http::response(['deleted' => true]), + ]); + + $provider = Ai::storeProvider('azure'); + $provider->storeGateway()->removeFile($provider, 'vs-123', '../..'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && $request->url() === 'https://test-resource.openai.azure.com/openai/v1/vector_stores/vs-123/files/..%2F..'); + } } diff --git a/tests/Ai/Feature/Providers/Gemini/FileGatewayTest.php b/tests/Ai/Feature/Providers/Gemini/FileGatewayTest.php index 5a2ec11bb8..a0fbe2e2fa 100644 --- a/tests/Ai/Feature/Providers/Gemini/FileGatewayTest.php +++ b/tests/Ai/Feature/Providers/Gemini/FileGatewayTest.php @@ -10,6 +10,7 @@ use Hypervel\Http\Client\Request; use Hypervel\Support\Facades\Http; use Hypervel\Tests\Ai\TestCase; +use InvalidArgumentException; use function Hypervel\Tests\Ai\Fixtures\multipartField; use function Hypervel\Tests\Ai\Fixtures\multipartNestedField; @@ -39,6 +40,7 @@ public function testGetFileSendsCorrectRequest(): void $response = Files::get('abc123', provider: 'gemini'); $this->assertSame('files/abc123', $response->id); + $this->assertNotNull($response->raw); $this->assertSame('text/plain', $response->mime); Http::assertSent(fn (Request $request): bool => $request->method() === 'GET' @@ -76,6 +78,7 @@ public function testPutFileSendsMultipartUpload(): void ); $this->assertSame('files/uploaded123', $response->id); + $this->assertNotNull($response->raw); Http::assertSent(fn (Request $request): bool => $request->method() === 'POST' && str_contains((string) $request->url(), '/upload/v1beta/files') @@ -168,4 +171,30 @@ public function testFileGatewayUsesCustomBaseUrl(): void Http::assertSent(fn (Request $request): bool => str_contains((string) $request->url(), 'custom.api.example.com/v1beta/files/abc123')); } + + public function testDeleteFileEncodesTheIdAsASinglePathSegment(): void + { + Http::fake([ + 'generativelanguage.googleapis.com/*' => Http::response([], 200), + ]); + + Files::delete('files/../fileSearchStores/store123?force=true', provider: 'gemini'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && $request->url() === 'https://generativelanguage.googleapis.com/v1beta/files/..%2FfileSearchStores%2Fstore123%3Fforce%3Dtrue'); + } + + public function testDeleteFileRejectsAPrefixedDotSegmentId(): void + { + Http::fake(); + + try { + Files::delete('files/..', provider: 'gemini'); + $this->fail('The ID must be rejected.'); + } catch (InvalidArgumentException $exception) { + $this->assertSame('[..] is not a valid provider resource ID.', $exception->getMessage()); + } + + Http::assertNothingSent(); + } } diff --git a/tests/Ai/Feature/Providers/Gemini/StoreGatewayTest.php b/tests/Ai/Feature/Providers/Gemini/StoreGatewayTest.php index fe215fde0b..ce02c0e950 100644 --- a/tests/Ai/Feature/Providers/Gemini/StoreGatewayTest.php +++ b/tests/Ai/Feature/Providers/Gemini/StoreGatewayTest.php @@ -419,4 +419,42 @@ public function testRemovingADocumentSendsForceAsAQueryParameter(): void $this->assertStringEndsWith('/fileSearchStores/store123/documents/doc456?force=true', sentRequest()->url()); $this->assertSame('', sentRequest()->body()); } + + public function testDeleteStoreEncodesTheIdAsASinglePathSegment(): void + { + Http::fake([ + 'generativelanguage.googleapis.com/*' => Http::response([], 200), + ]); + + Stores::delete('fileSearchStores/../files/abc123#', provider: 'gemini'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && $request->url() === 'https://generativelanguage.googleapis.com/v1beta/fileSearchStores/..%2Ffiles%2Fabc123%23?force=true'); + } + + public function testRemoveFileEncodesTheDocumentIdAsASinglePathSegment(): void + { + Http::fake([ + 'generativelanguage.googleapis.com/*' => Http::response([], 200), + ]); + + $provider = geminiProvider(); + $provider->storeGateway()->removeFile($provider, 'store123', 'documents/../../store456'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && str_starts_with((string) $request->url(), 'https://generativelanguage.googleapis.com/v1beta/fileSearchStores/store123/documents/..%2F..%2Fstore456')); + } + + public function testRemoveFileRebuildsAFullDocumentPathFromEncodedSegments(): void + { + Http::fake([ + 'generativelanguage.googleapis.com/*' => Http::response([], 200), + ]); + + $provider = geminiProvider(); + $provider->storeGateway()->removeFile($provider, 'store123', 'fileSearchStores/store123/documents/doc456?force=false'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && str_starts_with((string) $request->url(), 'https://generativelanguage.googleapis.com/v1beta/fileSearchStores/store123/documents/doc456%3Fforce%3Dfalse')); + } } diff --git a/tests/Ai/Feature/Providers/OpenAi/FileGatewayTest.php b/tests/Ai/Feature/Providers/OpenAi/FileGatewayTest.php index 2070e394b3..c3a197be79 100644 --- a/tests/Ai/Feature/Providers/OpenAi/FileGatewayTest.php +++ b/tests/Ai/Feature/Providers/OpenAi/FileGatewayTest.php @@ -11,6 +11,8 @@ use Hypervel\Http\Client\Request; use Hypervel\Support\Facades\Http; use Hypervel\Tests\Ai\TestCase; +use InvalidArgumentException; +use PHPUnit\Framework\Attributes\TestWith; use function Hypervel\Tests\Ai\Fixtures\multipartField; use function Hypervel\Tests\Ai\Fixtures\sentRequest; @@ -90,4 +92,72 @@ public function testDeleteFileSendsCorrectRequest(): void && $request->url() === 'https://api.openai.com/v1/files/file-abc123' && $request->hasHeader('Authorization', 'Bearer test-key')); } + + public function testDeleteFileEncodesTheIdAsASinglePathSegment(): void + { + Http::fake([ + 'api.openai.com/*' => Http::response(['deleted' => true]), + ]); + + Files::delete('../vector_stores/vs-123', provider: 'openai'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && $request->url() === 'https://api.openai.com/v1/files/..%2Fvector_stores%2Fvs-123'); + } + + #[TestWith([''])] + #[TestWith(['.'])] + #[TestWith(['..'])] + public function testDeleteFileRejectsDotSegmentIds(string $fileId): void + { + Http::fake(); + + try { + Files::delete($fileId, provider: 'openai'); + $this->fail('The ID must be rejected.'); + } catch (InvalidArgumentException $exception) { + $this->assertSame("[{$fileId}] is not a valid provider resource ID.", $exception->getMessage()); + } + + Http::assertNothingSent(); + } + + public function testGetFilePreservesRawResponse(): void + { + Http::fake([ + 'api.openai.com/*' => Http::response(['id' => 'file-abc123', 'bytes' => 1024]), + ]); + + $response = Files::get('file-abc123', provider: 'openai'); + + $this->assertNotNull($response->raw); + $this->assertSame(1024, $response->raw->json('bytes')); + } + + public function testPutFilePreservesRawResponse(): void + { + Http::fake([ + 'api.openai.com/*' => Http::response(['id' => 'file-uploaded123', 'bytes' => 2048]), + ]); + + $response = Document::fromString('Hello, World!', 'text/plain')->as('hello.txt')->put( + provider: 'openai', + ); + + $this->assertNotNull($response->raw); + $this->assertSame(2048, $response->raw->json('bytes')); + } + + public function testRawResponseIsExcludedWhenSerialized(): void + { + Http::fake([ + 'api.openai.com/*' => Http::response(['id' => 'file-abc123']), + ]); + + $response = Files::get('file-abc123', provider: 'openai'); + $unserialized = unserialize(serialize($response)); + + $this->assertSame('file-abc123', $unserialized->id); + $this->assertNull($unserialized->raw); + } } diff --git a/tests/Ai/Feature/Providers/OpenAi/StoreGatewayTest.php b/tests/Ai/Feature/Providers/OpenAi/StoreGatewayTest.php index fb8b964bfb..f39c396c43 100644 --- a/tests/Ai/Feature/Providers/OpenAi/StoreGatewayTest.php +++ b/tests/Ai/Feature/Providers/OpenAi/StoreGatewayTest.php @@ -162,6 +162,31 @@ public function testDeleteStoreSendsCorrectRequest(): void && $request->hasHeader('Authorization', 'Bearer test-key')); } + public function testRemoveFileEncodesTheStoreAndDocumentIdsAsSinglePathSegments(): void + { + Http::fake([ + 'api.openai.com/*' => Http::response(['deleted' => true]), + ]); + + $provider = $this->openAiProvider(); + $provider->storeGateway()->removeFile($provider, 'vs-123#', '../../files/file-456'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && $request->url() === 'https://api.openai.com/v1/vector_stores/vs-123%23/files/..%2F..%2Ffiles%2Ffile-456'); + } + + public function testDeleteStoreEncodesTheIdAsASinglePathSegment(): void + { + Http::fake([ + 'api.openai.com/*' => Http::response(['deleted' => true]), + ]); + + Stores::delete('../files/file-456', provider: 'openai'); + + Http::assertSent(fn (Request $request): bool => $request->method() === 'DELETE' + && $request->url() === 'https://api.openai.com/v1/vector_stores/..%2Ffiles%2Ffile-456'); + } + /** * Resolve the OpenAI provider. */ diff --git a/tests/Ai/Feature/Providers/OpenRouter/FileGatewayTest.php b/tests/Ai/Feature/Providers/OpenRouter/FileGatewayTest.php index 20ad4ecbe6..60153458ce 100644 --- a/tests/Ai/Feature/Providers/OpenRouter/FileGatewayTest.php +++ b/tests/Ai/Feature/Providers/OpenRouter/FileGatewayTest.php @@ -39,6 +39,7 @@ public function testGetFileSendsCorrectRequestAndExposesTheMimeType(): void $response = Files::get('or_file_abc123', provider: 'openrouter'); $this->assertSame('or_file_abc123', $response->id); + $this->assertNotNull($response->raw); $this->assertSame('text/csv', $response->mimeType()); Http::assertSent(fn (Request $request): bool => $request->method() === 'GET' @@ -66,6 +67,7 @@ public function testPutFileSendsTheFileAsAMultipartUpload(): void ); $this->assertSame('or_file_uploaded123', $response->id); + $this->assertNotNull($response->raw); $request = sentRequest(); $file = collect($request->data())->firstWhere('name', 'file'); From 281c15019b8388da1538879b1edde60a24e95d05 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:31 +0000 Subject: [PATCH 09/16] Escape image attributes in image responses Port laravel/ai#1149 (laravel/ai 1.x at d44ae14d17fb). ImageResponse::toHtml() escaped only the alt text, so a MIME type or image payload containing quotes could break out of the src attribute. Escape both, with upstream's test. --- src/ai/src/Responses/ImageResponse.php | 4 ++-- tests/Ai/Unit/Responses/ImageResponseTest.php | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/ai/src/Responses/ImageResponse.php b/src/ai/src/Responses/ImageResponse.php index 2ef6edd9d0..9cbf197879 100644 --- a/src/ai/src/Responses/ImageResponse.php +++ b/src/ai/src/Responses/ImageResponse.php @@ -81,8 +81,8 @@ public function toHtml(string $alt = ''): string return sprintf( '%s', - $image->mime(), - $image->image, + e($image->mime()), + e($image->image), e($alt), ); } diff --git a/tests/Ai/Unit/Responses/ImageResponseTest.php b/tests/Ai/Unit/Responses/ImageResponseTest.php index a193b7432b..dfcf4abc03 100644 --- a/tests/Ai/Unit/Responses/ImageResponseTest.php +++ b/tests/Ai/Unit/Responses/ImageResponseTest.php @@ -48,16 +48,16 @@ public function testToHtmlFallsBackToImagePngWhenTheImageHasNoMimeType(): void ); } - public function testToHtmlEscapesTheAltAttribute(): void + public function testToHtmlEscapesTheImageAttributes(): void { $response = new ImageResponse( - new Collection([new GeneratedImage('aGVsbG8=', 'image/png')]), + new Collection([new GeneratedImage('aGVsbG8=" onload="alert(2)', 'image/png" onerror="alert(1)')]), new ImageUsage, new Meta('openai', 'gpt-image-1'), ); $this->assertSame( - '"><script>alert(1)</script>', + '"><script>alert(1)</script>', $response->toHtml('">'), ); } From 10d094fb0a7f4694d636df05736de3215b4d6407 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:31 +0000 Subject: [PATCH 10/16] Match upstream's schema normalizer node budget Port laravel/ai#1145 (laravel/ai 1.x at d44ae14d17fb). Inlining a $ref shared by sibling properties at every use can expand a small MCP tool schema exponentially. Hypervel's normalizer already bounded each call; adopt upstream's 10,000-node limit and message, which bound the same work. McpTool::schema() still drops the parameters of a schema that exceeds the limit. Port upstream's normalizer and MCP tool tests, replacing the Hypervel test they duplicate. --- src/ai/src/Schema/SchemaNormalizer.php | 12 ++-- tests/Ai/Unit/Schema/SchemaNormalizerTest.php | 72 ++++++++++++++----- tests/Ai/Unit/Tools/McpToolTest.php | 19 +++++ 3 files changed, 79 insertions(+), 24 deletions(-) diff --git a/src/ai/src/Schema/SchemaNormalizer.php b/src/ai/src/Schema/SchemaNormalizer.php index 3d13f58c4f..61d88ffd0d 100644 --- a/src/ai/src/Schema/SchemaNormalizer.php +++ b/src/ai/src/Schema/SchemaNormalizer.php @@ -9,11 +9,6 @@ class SchemaNormalizer { - /** - * The maximum number of schema nodes normalized in one call. - */ - private const int MAX_NODES = 20000; - /** * Type-specific constraint keywords the deserializer rejects on a multi-type union. */ @@ -40,6 +35,11 @@ class SchemaNormalizer */ private const array TYPES = ['string', 'integer', 'number', 'boolean', 'object', 'array', 'null']; + /** + * The most nodes a single schema may expand to once every "$ref" is inlined. + */ + private const int MAX_NODES = 10_000; + /** * The number of nodes normalized so far. */ @@ -70,7 +70,7 @@ private function node(array $schema, array $root, array $seen): array { // Reference expansion happens before the deserializer can apply its own work limit. if (++$this->nodes > self::MAX_NODES) { - throw new InvalidArgumentException('The JSON Schema is too large to normalize; it expands beyond [' . self::MAX_NODES . '] nodes.'); + throw new InvalidArgumentException('Schema expands to more than ' . self::MAX_NODES . ' nodes.'); } [$schema, $seen] = $this->inlineRefs($schema, $root, $seen); diff --git a/tests/Ai/Unit/Schema/SchemaNormalizerTest.php b/tests/Ai/Unit/Schema/SchemaNormalizerTest.php index b6bb4626ce..926751d059 100644 --- a/tests/Ai/Unit/Schema/SchemaNormalizerTest.php +++ b/tests/Ai/Unit/Schema/SchemaNormalizerTest.php @@ -992,6 +992,60 @@ public function testItDeepMergesOverlappingPropertiesAcrossAllOfBranches(): void ], $normalized['properties']['value']); } + public function testItInlinesARefSharedBySiblingPropertiesAtEveryUse(): void + { + $normalized = SchemaNormalizer::normalize([ + 'type' => 'object', + 'properties' => [ + 'billing' => ['$ref' => '#/$defs/Address'], + 'shipping' => ['$ref' => '#/$defs/Address', 'description' => 'Ship to.'], + 'returns' => ['type' => 'array', 'items' => ['$ref' => '#/$defs/Address']], + ], + '$defs' => [ + 'Address' => [ + 'type' => 'object', + 'properties' => ['city' => ['type' => 'string']], + 'required' => ['city'], + ], + ], + ]); + + $address = [ + 'type' => 'object', + 'properties' => ['city' => ['type' => 'string']], + 'required' => ['city'], + ]; + + $this->assertSame([ + 'type' => 'object', + 'properties' => [ + 'billing' => $address, + 'shipping' => [...$address, 'description' => 'Ship to.'], + 'returns' => ['type' => 'array', 'items' => $address], + ], + ], $normalized); + } + + public function testItRejectsASchemaWhoseSharedRefExpansionExceedsTheNodeBudget(): void + { + $defs = ['a13' => ['type' => 'string']]; + + for ($i = 0; $i < 13; ++$i) { + $defs["a{$i}"] = [ + 'type' => 'object', + 'properties' => [ + 'left' => ['$ref' => '#/$defs/a' . ($i + 1)], + 'right' => ['$ref' => '#/$defs/a' . ($i + 1)], + ], + ]; + } + + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('Schema expands to more than 10000 nodes.'); + + SchemaNormalizer::normalize(['$ref' => '#/$defs/a0', '$defs' => $defs]); + } + public function testItPreservesANestedNullableAnyOf(): void { $raw = [ @@ -1099,22 +1153,4 @@ public function testNestedOneOfKeepsValuesDescriptionsAndNonObjectAlternatives() ['type' => 'array', 'items' => ['type' => 'string']], ], $normalized['properties']['target']['anyOf']); } - - public function testItBoundsRepeatedReferenceExpansion(): void - { - $definitions = ['level14' => ['type' => 'string']]; - - for ($depth = 13; $depth >= 0; --$depth) { - $reference = ['$ref' => '#/$defs/level' . ($depth + 1)]; - $definitions['level' . $depth] = [ - 'type' => 'object', - 'properties' => ['left' => $reference, 'right' => $reference], - ]; - } - - $this->expectException(InvalidArgumentException::class); - $this->expectExceptionMessage('The JSON Schema is too large to normalize'); - - SchemaNormalizer::normalize(['$ref' => '#/$defs/level0', '$defs' => $definitions]); - } } diff --git a/tests/Ai/Unit/Tools/McpToolTest.php b/tests/Ai/Unit/Tools/McpToolTest.php index 3e0edd81e3..b8ae371810 100644 --- a/tests/Ai/Unit/Tools/McpToolTest.php +++ b/tests/Ai/Unit/Tools/McpToolTest.php @@ -171,6 +171,25 @@ public function testItDropsAnUnresolvableRefInsteadOfFailingTheWholeTool(): void $this->assertSame(['type' => 'string'], $schema['properties']['name']); } + public function testItDropsTheParameterSchemaWhenSharedRefExpansionExceedsTheNodeBudget(): void + { + $defs = ['a13' => ['type' => 'string']]; + + for ($i = 0; $i < 13; ++$i) { + $defs["a{$i}"] = [ + 'type' => 'object', + 'properties' => [ + 'left' => ['$ref' => '#/$defs/a' . ($i + 1)], + 'right' => ['$ref' => '#/$defs/a' . ($i + 1)], + ], + ]; + } + + $tool = new McpTool(mcpTool(new FakeMcpClient, inputSchema: ['$ref' => '#/$defs/a0', '$defs' => $defs])); + + $this->assertSame([], $tool->schema(new JsonSchemaTypeFactory)); + } + public function testItKeepsThePropertiesOfANullableObjectRoot(): void { $tool = new McpTool(mcpTool(new FakeMcpClient, inputSchema: [ From d7e1952599ba844416b2fcfd52d8f4d8d7f90eb3 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:31 +0000 Subject: [PATCH 11/16] Separate Anthropic text around server tools Port laravel/ai#1143 (laravel/ai 1.x at d44ae14d17fb). Text an Anthropic response wrote before and after a server tool, such as a web search, ran together. Put a blank line between those remarks unless the line already breaks there, and keep citation-split text joined. Streaming adds the separator to the first delta after the tool, while replayed blocks keep their original text, so streamed and final text match. Port upstream's response and streaming tests. --- .../Concerns/HandlesTextStreaming.php | 20 +++--- .../Concerns/ParsesTextResponses.php | 43 ++++++++++- .../Anthropic/ProviderToolCallsTest.php | 72 +++++++++++++++++++ .../Providers/Anthropic/StreamingTest.php | 68 ++++++++++++++++++ 4 files changed, 191 insertions(+), 12 deletions(-) diff --git a/src/ai/src/Gateway/Anthropic/Concerns/HandlesTextStreaming.php b/src/ai/src/Gateway/Anthropic/Concerns/HandlesTextStreaming.php index dd396a8447..74b1a903cf 100644 --- a/src/ai/src/Gateway/Anthropic/Concerns/HandlesTextStreaming.php +++ b/src/ai/src/Gateway/Anthropic/Concerns/HandlesTextStreaming.php @@ -50,6 +50,8 @@ protected function processTextStream( $currentServerToolInput = ''; $pendingToolCalls = []; $responseContent = []; + $afterTool = false; + $lastTextDelta = ''; $messageUsage = []; $usage = null; @@ -119,6 +121,10 @@ protected function processTextStream( $currentBlockType = $blockType; $currentBlockIndex = $data['index'] ?? count($responseContent); + if ($this->isToolBlock($blockType)) { + $afterTool = true; + } + if ($blockType === 'text') { $currentBlockText = ''; @@ -190,6 +196,10 @@ protected function processTextStream( $textDelta = (string) ($data['delta']['text'] ?? ''); if ($textDelta !== '') { + $separator = $afterTool && $lastTextDelta !== '' ? $this->textSeparator($lastTextDelta, $textDelta) : ''; + $afterTool = false; + $lastTextDelta = $textDelta; + if (($event = $emitTextStart()) instanceof StreamEvent) { yield $event; } @@ -199,7 +209,7 @@ protected function processTextStream( yield (new TextDelta( $this->generateEventId(), $messageId, - $textDelta, + $separator . $textDelta, time(), ))->withInvocationId($invocationId); } @@ -351,14 +361,6 @@ protected function processTextStream( ); } - /** - * Determine if the given block type is a provider tool result. - */ - protected function isProviderToolResultBlock(string $blockType): bool - { - return str_ends_with($blockType, '_tool_result'); - } - /** * Generate a lowercase UUID v7 for use as a stream event ID. */ diff --git a/src/ai/src/Gateway/Anthropic/Concerns/ParsesTextResponses.php b/src/ai/src/Gateway/Anthropic/Concerns/ParsesTextResponses.php index 2f32b94976..103b187d16 100644 --- a/src/ai/src/Gateway/Anthropic/Concerns/ParsesTextResponses.php +++ b/src/ai/src/Gateway/Anthropic/Concerns/ParsesTextResponses.php @@ -108,9 +108,46 @@ protected function buildStepResponse( */ protected function extractText(array $content): string { - $textBlocks = array_filter($content, fn (array $block): bool => ($block['type'] ?? '') === 'text'); + $text = ''; + $afterTool = false; - return implode('', array_column($textBlocks, 'text')); + foreach ($content as $block) { + $type = $block['type'] ?? ''; + + if ($type === 'text') { + $blockText = (string) ($block['text'] ?? ''); + $text .= ($afterTool && $text !== '' ? $this->textSeparator($text, $blockText) : '') . $blockText; + $afterTool = false; + } elseif ($this->isToolBlock($type)) { + $afterTool = true; + } + } + + return $text; + } + + /** + * Get the separator between text written before a tool block and text written after it. + */ + protected function textSeparator(string $before, string $after): string + { + return str_ends_with($before, "\n") || str_starts_with($after, "\n") ? '' : "\n\n"; + } + + /** + * Determine if the given block type is a server tool call or its result. + */ + protected function isToolBlock(string $type): bool + { + return $type === 'server_tool_use' || $this->isProviderToolResultBlock($type); + } + + /** + * Determine if the given block type is a provider tool result. + */ + protected function isProviderToolResultBlock(string $type): bool + { + return str_ends_with($type, '_tool_result'); } /** @@ -132,7 +169,7 @@ protected function extractProviderToolCalls(array $content): array { return array_values(array_map( fn (array $block): ProviderToolCall => new ProviderToolCall($block['tool_use_id'] ?? $block['id'] ?? '', $block['type'], $block), - array_filter($content, fn (array $block): bool => ($block['type'] ?? '') === 'server_tool_use' || str_ends_with((string) ($block['type'] ?? ''), '_tool_result')), + array_filter($content, fn (array $block): bool => ($block['type'] ?? '') === 'server_tool_use' || $this->isProviderToolResultBlock((string) ($block['type'] ?? ''))), )); } diff --git a/tests/Ai/Feature/Providers/Anthropic/ProviderToolCallsTest.php b/tests/Ai/Feature/Providers/Anthropic/ProviderToolCallsTest.php index db38b8b76b..a8651b945f 100644 --- a/tests/Ai/Feature/Providers/Anthropic/ProviderToolCallsTest.php +++ b/tests/Ai/Feature/Providers/Anthropic/ProviderToolCallsTest.php @@ -34,4 +34,76 @@ public function testServerToolUseAndResultBlocksLandOnTheStepKeyedByTheToolUse() $this->assertSame([['srvtoolu_1', 'server_tool_use'], ['srvtoolu_1', 'web_search_tool_result']], array_map(fn (ProviderToolCall $call): array => [$call->id, $call->type], $response->steps[0]->providerToolCalls)); $this->assertSame(['query' => 'hypervel ai'], $response->steps[0]->providerToolCalls[0]->data['input']); } + + public function testTextBeforeAndAfterAServerToolIsSeparatedByABlankLineWhileCitationSplitTextStaysJoined(): void + { + Http::fake([ + 'api.anthropic.com/*' => Http::response([ + 'id' => 'msg_1', + 'type' => 'message', + 'role' => 'assistant', + 'model' => 'claude-sonnet-4-6', + 'content' => [ + ['type' => 'text', 'text' => 'Let me check.'], + ['type' => 'server_tool_use', 'id' => 'srvtoolu_1', 'name' => 'web_search', 'input' => ['query' => 'hypervel ai']], + ['type' => 'web_search_tool_result', 'tool_use_id' => 'srvtoolu_1', 'content' => []], + ['type' => 'text', 'text' => 'The latest stable version is '], + ['type' => 'text', 'text' => 'Hypervel 0.4', 'citations' => [['type' => 'web_search_result_location', 'url' => 'https://hypervel.org', 'title' => 'Hypervel', 'cited_text' => 'Hypervel 0.4']]], + ], + 'stop_reason' => 'end_turn', + 'usage' => ['input_tokens' => 10, 'output_tokens' => 5], + ]), + ]); + + $response = (new AssistantAgent)->prompt('Search', provider: 'anthropic'); + + $this->assertSame("Let me check.\n\nThe latest stable version is Hypervel 0.4", $response->text); + } + + public function testTextAroundARedactedThinkingBlockIsNotSeparated(): void + { + Http::fake([ + 'api.anthropic.com/*' => Http::response([ + 'id' => 'msg_1', + 'type' => 'message', + 'role' => 'assistant', + 'model' => 'claude-sonnet-4-6', + 'content' => [ + ['type' => 'text', 'text' => 'First'], + ['type' => 'redacted_thinking', 'data' => 'encrypted'], + ['type' => 'text', 'text' => 'Second'], + ], + 'stop_reason' => 'end_turn', + 'usage' => ['input_tokens' => 10, 'output_tokens' => 5], + ]), + ]); + + $response = (new AssistantAgent)->prompt('Hello', provider: 'anthropic'); + + $this->assertSame('FirstSecond', $response->text); + } + + public function testTextAfterAServerToolIsNotSeparatedWhenTheModelAlreadyEndedTheLine(): void + { + Http::fake([ + 'api.anthropic.com/*' => Http::response([ + 'id' => 'msg_1', + 'type' => 'message', + 'role' => 'assistant', + 'model' => 'claude-sonnet-4-6', + 'content' => [ + ['type' => 'text', 'text' => "Let me check.\n"], + ['type' => 'server_tool_use', 'id' => 'srvtoolu_1', 'name' => 'web_search', 'input' => ['query' => 'hypervel ai']], + ['type' => 'web_search_tool_result', 'tool_use_id' => 'srvtoolu_1', 'content' => []], + ['type' => 'text', 'text' => 'Done'], + ], + 'stop_reason' => 'end_turn', + 'usage' => ['input_tokens' => 10, 'output_tokens' => 5], + ]), + ]); + + $response = (new AssistantAgent)->prompt('Search', provider: 'anthropic'); + + $this->assertSame("Let me check.\nDone", $response->text); + } } diff --git a/tests/Ai/Feature/Providers/Anthropic/StreamingTest.php b/tests/Ai/Feature/Providers/Anthropic/StreamingTest.php index ab1be3b74a..0f4e5835f5 100644 --- a/tests/Ai/Feature/Providers/Anthropic/StreamingTest.php +++ b/tests/Ai/Feature/Providers/Anthropic/StreamingTest.php @@ -94,6 +94,8 @@ public function testStreamingReportsTheTextBlocksOfOneStepAsASingleMessage(): vo $this->assertSame($textStarts[0]->messageId, $textEnds[0]->messageId); $this->assertSame($textStarts[0]->messageId, $textDeltas[0]->messageId); $this->assertSame($textStarts[0]->messageId, $textDeltas[1]->messageId); + $this->assertSame('First', $textDeltas[0]->delta); + $this->assertSame("\n\nSecond", $textDeltas[1]->delta); $this->assertGreaterThan(array_search($textDeltas[1], $events, true), array_search($textEnds[0], $events, true)); } @@ -312,6 +314,72 @@ public function testStreamingPauseTurnTriggersFollowUpStreamWithAssistantReplaye $this->assertSame('Resumed', $textDeltas[0]->delta); } + public function testStreamingReplaysTextBlocksAroundAServerToolWithoutTheSeparator(): void + { + Http::fake([ + 'api.anthropic.com/*' => Http::sequence([ + Http::response( + body: $this->ssePayload([ + $this->messageStart(), + $this->contentBlockStart(0, ['type' => 'text', 'text' => '']), + $this->contentBlockDelta(0, ['type' => 'text_delta', 'text' => 'First']), + $this->contentBlockStop(0), + $this->contentBlockStart(1, ['type' => 'server_tool_use', 'id' => 'srvtoolu_1', 'name' => 'web_search']), + $this->contentBlockStop(1), + $this->contentBlockStart(2, ['type' => 'text', 'text' => '']), + $this->contentBlockDelta(2, ['type' => 'text_delta', 'text' => 'Second']), + $this->contentBlockStop(2), + $this->messageDelta('pause_turn', 5), + ]), + status: 200, + headers: ['Content-Type' => 'text/event-stream'], + ), + Http::response( + body: $this->ssePayload([ + $this->messageStart(), + $this->messageDelta('end_turn', 5), + ]), + status: 200, + headers: ['Content-Type' => 'text/event-stream'], + ), + ]), + ]); + + $events = $this->collectStreamEvents(); + + $followUpBody = Http::recorded()[1][0]->data(); + $replayed = end($followUpBody['messages'])['content']; + + $this->assertSame(['First', 'Second'], [$replayed[0]['text'], $replayed[2]['text']]); + $this->assertSame("First\n\nSecond", TextDelta::combine($events)); + } + + public function testStreamingDoesNotSeparateTextAfterAServerToolWhenTheModelAlreadyEndedTheLine(): void + { + Http::fake([ + 'api.anthropic.com/*' => Http::response( + body: $this->ssePayload([ + $this->messageStart(), + $this->contentBlockStart(0, ['type' => 'text', 'text' => '']), + $this->contentBlockDelta(0, ['type' => 'text_delta', 'text' => "First\n"]), + $this->contentBlockStop(0), + $this->contentBlockStart(1, ['type' => 'server_tool_use', 'id' => 'srvtoolu_1', 'name' => 'web_search']), + $this->contentBlockStop(1), + $this->contentBlockStart(2, ['type' => 'text', 'text' => '']), + $this->contentBlockDelta(2, ['type' => 'text_delta', 'text' => 'Second']), + $this->contentBlockStop(2), + $this->messageDelta('end_turn', 10), + ]), + status: 200, + headers: ['Content-Type' => 'text/event-stream'], + ), + ]); + + $textDeltas = array_values(array_filter($this->collectStreamEvents(), fn ($e): bool => $e instanceof TextDelta)); + + $this->assertSame(["First\n", 'Second'], array_map(fn (TextDelta $e): string => $e->delta, $textDeltas)); + } + public function testStreamingErrorEventStopsStream(): void { Http::fake([ From 77ccf78d59875d69232519c235986e80ce26375d Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:31 +0000 Subject: [PATCH 12/16] Return structured responses for structured agents Port laravel/ai#1138 (laravel/ai 1.x at d44ae14d17fb). prompt() was declared as returning AgentResponse, so static analysis rejected array access on a structured agent's response. Promptable::prompt() and agent() now carry conditional return types, pinned by a types fixture. Structured agents now always return a StructuredAgentResponse, with an empty payload when a run produces none, such as a stopped run, a string fake or a Bedrock approval pause. Unlike upstream, the response type is chosen by HasStructuredOutput rather than by a filled schema, so agents with an empty schema also match the declared type; their text remains available and the response casts to an empty payload. Port upstream's tests and fixture agent, adapting the empty-schema test to the corrected behavior. Formatting, static analysis and the AI suite pass. --- .../Gateway/Bedrock/BedrockTextGateway.php | 2 +- src/ai/src/Promptable.php | 4 ++ .../src/Providers/Concerns/GeneratesText.php | 11 ++++- src/ai/src/functions.php | 2 + tests/Ai/Feature/AgentFakeTest.php | 19 ++++++-- tests/Ai/Feature/AgentMiddlewareTest.php | 7 +-- .../Providers/Bedrock/ToolCallLoopTest.php | 27 +++++++++++ tests/Ai/Feature/ToolApprovalResumeTest.php | 35 ++++++++++++++ .../RememberingStructuredApprovableAgent.php | 46 +++++++++++++++++++ types/Ai/Promptable.php | 43 +++++++++++++++++ 10 files changed, 187 insertions(+), 9 deletions(-) create mode 100644 tests/Ai/Fixtures/Agents/RememberingStructuredApprovableAgent.php create mode 100644 types/Ai/Promptable.php diff --git a/src/ai/src/Gateway/Bedrock/BedrockTextGateway.php b/src/ai/src/Gateway/Bedrock/BedrockTextGateway.php index 6596650388..1c174efa58 100644 --- a/src/ai/src/Gateway/Bedrock/BedrockTextGateway.php +++ b/src/ai/src/Gateway/Bedrock/BedrockTextGateway.php @@ -266,7 +266,7 @@ protected function parseTextResponse(array $result, TextProvider $provider, stri finishReason: $finishReason, usage: $usage, meta: new Meta($provider->name(), $model), - structured: $structuredOutput !== null ? $this->decodeStructuredOutput($structuredOutput) : null, + structured: $structured ? $this->decodeStructuredOutput($structuredOutput ?? $output) : null, replayBlocks: $replayBlocks, reasoning: $this->extractReasoning($replayBlocks), ); diff --git a/src/ai/src/Promptable.php b/src/ai/src/Promptable.php index a9ad6d882f..7bcf471042 100644 --- a/src/ai/src/Promptable.php +++ b/src/ai/src/Promptable.php @@ -17,6 +17,7 @@ use Hypervel\Ai\Contracts\AgentInput; use Hypervel\Ai\Contracts\Conversational; use Hypervel\Ai\Contracts\HasSkills; +use Hypervel\Ai\Contracts\HasStructuredOutput; use Hypervel\Ai\Contracts\HasTools; use Hypervel\Ai\Contracts\Providers\TextProvider; use Hypervel\Ai\Contracts\Tool; @@ -37,6 +38,7 @@ use Hypervel\Ai\Responses\QueuedAgentResponse; use Hypervel\Ai\Responses\StreamableAgentResponse; use Hypervel\Ai\Responses\StreamedAgentResponse; +use Hypervel\Ai\Responses\StructuredAgentResponse; use Hypervel\Ai\Support\ConversationSelection; use Hypervel\Ai\Tools\LoadSkill; use Hypervel\Ai\Vercel\Vercel; @@ -90,6 +92,8 @@ public static function make(mixed ...$arguments): static /** * Invoke the agent with a given prompt, or resume a paused run with tool approval decisions. + * + * @return ($this is HasStructuredOutput ? StructuredAgentResponse : AgentResponse) */ public function prompt( AgentInput|UserMessage|Decisions|string $prompt, diff --git a/src/ai/src/Providers/Concerns/GeneratesText.php b/src/ai/src/Providers/Concerns/GeneratesText.php index 75da7905f7..71dd1f1d3d 100644 --- a/src/ai/src/Providers/Concerns/GeneratesText.php +++ b/src/ai/src/Providers/Concerns/GeneratesText.php @@ -104,8 +104,15 @@ public function prompt(AgentPrompt $prompt): AgentResponse $this->throwIfNotResumable($prompt); } - $agentResponse = $response instanceof StructuredTextResponse - ? (new StructuredAgentResponse($invocationId, $response->structured, $response->text, $response->usage, $response->meta)) + // Empty schemas included, so prompt()'s declared structured return type always holds... + $agentResponse = $agent instanceof HasStructuredOutput + ? (new StructuredAgentResponse( + $invocationId, + $response instanceof StructuredTextResponse ? $response->structured : [], + $response->text, + $response->usage, + $response->meta, + )) ->withMessages($response->messages) ->withToolCallsAndResults($response->toolCalls, $response->toolResults) ->withSteps($response->steps) diff --git a/src/ai/src/functions.php b/src/ai/src/functions.php index 681118613f..d50da10693 100644 --- a/src/ai/src/functions.php +++ b/src/ai/src/functions.php @@ -13,6 +13,8 @@ /** * Get an ad-hoc agent instance. + * + * @return ($schema is null ? AnonymousAgent : StructuredAnonymousAgent) */ function agent( string $instructions = '', diff --git a/tests/Ai/Feature/AgentFakeTest.php b/tests/Ai/Feature/AgentFakeTest.php index f12620ba58..bfc5474f8e 100644 --- a/tests/Ai/Feature/AgentFakeTest.php +++ b/tests/Ai/Feature/AgentFakeTest.php @@ -163,6 +163,17 @@ public function testAgentsWithStructuredOutputCanBeFakedWithNoPredefinedResponse $this->assertIsString($response['symbol']); } + public function testAgentsWithStructuredOutputFakedWithAStringStillReturnAStructuredResponse(): void + { + StructuredAgent::fake(['Plain text']); + + $response = (new StructuredAgent)->prompt('Gold prompt'); + + $this->assertInstanceOf(StructuredAgentResponse::class, $response); + $this->assertSame([], $response->structured); + $this->assertSame('Plain text', $response->text); + } + public function testFakeClosuresCanThrowExceptions(): void { AssistantAgent::fake(function (): void { @@ -173,7 +184,7 @@ public function testFakeClosuresCanThrowExceptions(): void (new AssistantAgent)->prompt('Test prompt'); } - public function testStructuredAgentsWithEmptySchemasFallBackToATextResponse(): void + public function testStructuredAgentsWithEmptySchemasReturnAStructuredResponseKeepingTheirText(): void { EmptySchemaStructuredAgent::fake([ new TextResponse('Hello', new TextUsage, new Meta), @@ -181,9 +192,11 @@ public function testStructuredAgentsWithEmptySchemasFallBackToATextResponse(): v $response = (new EmptySchemaStructuredAgent)->prompt('Anything'); - $this->assertInstanceOf(AgentResponse::class, $response); - $this->assertNotInstanceOf(StructuredAgentResponse::class, $response); + // Upstream falls back to a plain AgentResponse here, contradicting prompt()'s structured return type. + $this->assertInstanceOf(StructuredAgentResponse::class, $response); + $this->assertSame([], $response->structured); $this->assertSame('Hello', $response->text); + $this->assertSame('[]', (string) $response); } public function testAgentsCanFakePausedApprovalResponsesAndAssertResumePrompts(): void diff --git a/tests/Ai/Feature/AgentMiddlewareTest.php b/tests/Ai/Feature/AgentMiddlewareTest.php index 6730752a11..87c5819c75 100644 --- a/tests/Ai/Feature/AgentMiddlewareTest.php +++ b/tests/Ai/Feature/AgentMiddlewareTest.php @@ -31,13 +31,13 @@ use Hypervel\Ai\PendingStep; use Hypervel\Ai\Promptable; use Hypervel\Ai\Providers\Tools\WebSearch; -use Hypervel\Ai\Responses\AgentResponse; use Hypervel\Ai\Responses\Data\FinishReason; use Hypervel\Ai\Responses\Data\Meta; use Hypervel\Ai\Responses\Data\TextUsage; use Hypervel\Ai\Responses\Data\ToolCall; use Hypervel\Ai\Responses\Data\UrlCitation; use Hypervel\Ai\Responses\StreamedAgentResponse; +use Hypervel\Ai\Responses\StructuredAgentResponse; use Hypervel\Ai\Storage\DatabaseConversationStore; use Hypervel\Ai\Streaming\Events\Citation; use Hypervel\Ai\Streaming\Events\ReasoningDelta; @@ -346,7 +346,7 @@ public function testAStopMiddlewareOnTheFirstStepReturnsAnEmptyResponseWithoutAS $this->assertEmpty($response->steps); } - public function testAStructuredAgentStoppedBeforeItsFirstStepReturnsAnUnstructuredResponse(): void + public function testAStructuredAgentStoppedBeforeItsFirstStepReturnsAnEmptyStructuredResponse(): void { $agent = new class extends StructuredAgent implements HasMiddleware { /** @@ -362,7 +362,8 @@ public function middleware(): array $response = $agent->prompt('Test prompt'); - $this->assertSame(AgentResponse::class, $response::class); + $this->assertInstanceOf(StructuredAgentResponse::class, $response); + $this->assertSame([], $response->structured); $this->assertSame('', $response->text); } diff --git a/tests/Ai/Feature/Providers/Bedrock/ToolCallLoopTest.php b/tests/Ai/Feature/Providers/Bedrock/ToolCallLoopTest.php index 0a276da4da..97c83f1d86 100644 --- a/tests/Ai/Feature/Providers/Bedrock/ToolCallLoopTest.php +++ b/tests/Ai/Feature/Providers/Bedrock/ToolCallLoopTest.php @@ -14,6 +14,7 @@ use Hypervel\Ai\Streaming\Events\ToolResult as ToolResultEvent; use Hypervel\JsonSchema\JsonSchemaTypeFactory; use Hypervel\Tests\Ai\Fixtures\BedrockHelpers; +use Hypervel\Tests\Ai\Fixtures\Tools\ApprovableNumberGenerator; use Hypervel\Tests\Ai\Fixtures\Tools\FixedNumberGenerator; use Hypervel\Tests\Ai\TestCase; use PHPUnit\Framework\Attributes\DataProvider; @@ -243,6 +244,32 @@ public static function jsonAnswerModes(): array ]; } + public function testStructuredRunsPausedForToolApprovalKeepTheStructuredResponseShape(): void + { + $client = $this->fakeBedrockConverseSequence([[ + 'output' => ['message' => ['content' => [ + ['toolUse' => ['toolUseId' => 't1', 'name' => 'ApprovableNumberGenerator', 'input' => []]], + ]]], + 'usage' => ['inputTokens' => 7, 'outputTokens' => 3], + 'stopReason' => 'tool_use', + ]]); + + $response = (new TextGenerationLoop($this->gatewayWithClient($client)))->generate( + $this->bedrockProvider(), + 'anthropic.claude-opus-4-7-v1:0', + null, + messages: [new UserMessage('Generate a number')], + tools: [new ApprovableNumberGenerator], + schema: ['number' => (new JsonSchemaTypeFactory)->integer()], + options: new TextGenerationOptions(maxSteps: 2), + ); + + $this->assertInstanceOf(StructuredTextResponse::class, $response); + $this->assertSame([], $response->structured); + $this->assertCount(1, $response->pendingApprovals); + $this->assertSame('t1', $response->pendingApprovals[0]->id); + } + public function testStreamingToolLoopEmitsSingleStreamEndWithAccumulatedUsage(): void { $client = $this->fakeBedrockStreamSequence([ diff --git a/tests/Ai/Feature/ToolApprovalResumeTest.php b/tests/Ai/Feature/ToolApprovalResumeTest.php index b81598214f..90e38c013b 100644 --- a/tests/Ai/Feature/ToolApprovalResumeTest.php +++ b/tests/Ai/Feature/ToolApprovalResumeTest.php @@ -10,6 +10,7 @@ use Hypervel\Ai\Exceptions\ApprovalMismatchException; use Hypervel\Ai\Exceptions\RateLimitedException; use Hypervel\Ai\Messages\ToolResultMessage; +use Hypervel\Ai\Responses\StructuredAgentResponse; use Hypervel\Ai\Storage\DatabaseConversationStore; use Hypervel\Foundation\Testing\RefreshDatabase; use Hypervel\Http\Client\RequestException; @@ -18,6 +19,7 @@ use Hypervel\Support\Facades\Http; use Hypervel\Tests\Ai\Fixtures\Agents\RememberingApprovableAgent; use Hypervel\Tests\Ai\Fixtures\Agents\RememberingMultiStepApprovableAgent; +use Hypervel\Tests\Ai\Fixtures\Agents\RememberingStructuredApprovableAgent; use Hypervel\Tests\Ai\Fixtures\Agents\RememberingToolUsingAgent; use Hypervel\Tests\Ai\Fixtures\AnthropicHelpers; use Hypervel\Tests\Ai\Fixtures\Tools\ApprovableNumberGenerator; @@ -849,6 +851,39 @@ public function testARejectedResumeStoresAndRehydratesTheToolResultAsDenied(): v $this->assertTrue($toolResultMessage->toolResults->first()->denied); } + public function testARejectedResumeOfAStructuredAgentReturnsAStructuredResponse(): void + { + config(['ai.conversations.generate_title' => false]); + + Http::fake([ + 'api.anthropic.com/*' => Http::response([ + 'id' => 'msg_tool_1', + 'type' => 'message', + 'role' => 'assistant', + 'model' => 'claude-sonnet-4-6', + 'content' => [[ + 'type' => 'tool_use', + 'id' => 'toolu_1', + 'name' => 'ApprovableNumberGenerator', + 'input' => (object) [], + ]], + 'stop_reason' => 'tool_use', + 'usage' => ['input_tokens' => 10, 'output_tokens' => 5], + ]), + ]); + + $user = (object) ['id' => 1]; + + $paused = (new RememberingStructuredApprovableAgent)->forUser($user)->prompt('Generate a number', provider: 'anthropic'); + + $rejected = (new RememberingStructuredApprovableAgent) + ->continue($paused->conversationId, $user) + ->prompt(Decisions::from(['toolu_1' => false]), provider: 'anthropic'); + + $this->assertInstanceOf(StructuredAgentResponse::class, $rejected); + $this->assertSame([], $rejected->structured); + } + public function testAWildcardRejectionResumeRecordsTheDenialOnceOnThePausedRow(): void { config(['ai.conversations.generate_title' => false]); diff --git a/tests/Ai/Fixtures/Agents/RememberingStructuredApprovableAgent.php b/tests/Ai/Fixtures/Agents/RememberingStructuredApprovableAgent.php new file mode 100644 index 0000000000..d386d35030 --- /dev/null +++ b/tests/Ai/Fixtures/Agents/RememberingStructuredApprovableAgent.php @@ -0,0 +1,46 @@ + $schema->integer()->required(), + ]; + } +} diff --git a/types/Ai/Promptable.php b/types/Ai/Promptable.php new file mode 100644 index 0000000000..8ff6576801 --- /dev/null +++ b/types/Ai/Promptable.php @@ -0,0 +1,43 @@ + $schema->integer()->required()]; + } +} + +assertType('Hypervel\Ai\Responses\AgentResponse', (new PromptableTypeAgent)->prompt('Hello')); +assertType('Hypervel\Ai\Responses\StructuredAgentResponse', (new PromptableTypeStructuredAgent)->prompt('Hello')); +assertType('Hypervel\Ai\Responses\StructuredAgentResponse', PromptableTypeStructuredAgent::make()->prompt('Hello')); + +assertType('Hypervel\Ai\Responses\AgentResponse', agent('You are helpful.')->prompt('Hello')); +assertType('Hypervel\Ai\Responses\StructuredAgentResponse', agent(schema: fn ($schema) => [])->prompt('Hello')); From 242a6b53e3bf4ebb7203d1ce056b7b587e781b01 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:31 +0000 Subject: [PATCH 13/16] Fix Gemini file search metadata filters Port laravel/ai#1144 (laravel/ai 1.x at d44ae14d17fb). Gemini filter keys were interpolated unchecked, and conditions were not grouped although AIP-160 gives OR higher precedence than AND. Hypervel already escaped values and supported whereNotIn(); adopt upstream's formatter, which also validates keys and parenthesizes every condition. Also fix empty whereIn() and whereNotIn() lists, which produced an invalid empty group. An empty exclusion list is now omitted, and an empty inclusion list throws before the request, since Gemini has no portable condition that matches nothing. Port upstream's tests, folding Hypervel's duplicate filter tests into them. --- src/ai/src/Providers/GeminiProvider.php | 44 +++-- .../Providers/Gemini/ToolMappingTest.php | 170 ++++++++++++++---- 2 files changed, 162 insertions(+), 52 deletions(-) diff --git a/src/ai/src/Providers/GeminiProvider.php b/src/ai/src/Providers/GeminiProvider.php index c2122ab223..773b986bd4 100644 --- a/src/ai/src/Providers/GeminiProvider.php +++ b/src/ai/src/Providers/GeminiProvider.php @@ -80,26 +80,32 @@ public function fileSearchToolOptions(FileSearch $search): array */ protected function formatMetadataFilter(array $filters): string { - return (new Collection($filters))->map(fn (array $filter): string => match ($filter['type']) { - 'eq' => $this->formatMetadataCondition($filter['key'], '=', $filter['value']), - 'ne' => $this->formatMetadataCondition($filter['key'], '!=', $filter['value']), - 'in' => '(' . (new Collection($filter['value']))->map( - fn (mixed $value): string => $this->formatMetadataCondition($filter['key'], '=', $value) - )->implode(' OR ') . ')', - 'nin' => '(' . (new Collection($filter['value']))->map( - fn (mixed $value): string => $this->formatMetadataCondition($filter['key'], '!=', $value) - )->implode(' AND ') . ')', - })->implode(' AND '); - } - - /** - * Format a condition with an escaped metadata value. - */ - protected function formatMetadataCondition(string $key, string $operator, mixed $value): string - { - return $key . $operator . (is_numeric($value) + $formatValue = fn (mixed $value): string => is_numeric($value) ? (string) $value - : '"' . addcslashes((string) $value, '\"') . '"'); + : '"' . str_replace(['\\', '"'], ['\\\\', '\"'], (string) $value) . '"'; + + return (new Collection($filters))->map(function (array $filter) use ($formatValue): string { + if (! preg_match('/^[A-Za-z_][A-Za-z0-9_.]*$/D', $filter['key'])) { + throw new InvalidArgumentException("Invalid metadata filter key [{$filter['key']}]."); + } + + // An empty group is invalid filter syntax: excluding nothing needs no condition, + // and Gemini has no portable constant-false condition for matching nothing. + if ($filter['type'] === 'nin' && $filter['value'] === []) { + return ''; + } + + if ($filter['type'] === 'in' && $filter['value'] === []) { + throw new InvalidArgumentException("Gemini metadata filters cannot match [{$filter['key']}] against an empty list of values."); + } + + return '(' . match ($filter['type']) { + 'eq' => $filter['key'] . '=' . $formatValue($filter['value']), + 'ne' => $filter['key'] . '!=' . $formatValue($filter['value']), + 'in' => (new Collection($filter['value']))->map(fn (mixed $value): string => $filter['key'] . '=' . $formatValue($value))->implode(' OR '), + 'nin' => (new Collection($filter['value']))->map(fn (mixed $value): string => $filter['key'] . '!=' . $formatValue($value))->implode(' AND '), + } . ')'; + })->filter()->implode(' AND '); } /** diff --git a/tests/Ai/Feature/Providers/Gemini/ToolMappingTest.php b/tests/Ai/Feature/Providers/Gemini/ToolMappingTest.php index 8033122de0..01e89ab955 100644 --- a/tests/Ai/Feature/Providers/Gemini/ToolMappingTest.php +++ b/tests/Ai/Feature/Providers/Gemini/ToolMappingTest.php @@ -4,8 +4,6 @@ namespace Hypervel\Tests\Ai\Feature\Providers\Gemini; -use Hypervel\Ai\Contracts\Gateway\Gateway; -use Hypervel\Ai\Providers\GeminiProvider; use Hypervel\Ai\Providers\Tools\CodeExecution; use Hypervel\Ai\Providers\Tools\FileSearch; use Hypervel\Ai\Providers\Tools\FileSearchQuery; @@ -18,6 +16,8 @@ use Hypervel\Tests\Ai\Fixtures\GeminiHelpers; use Hypervel\Tests\Ai\Fixtures\Tools\FixedNumberGenerator; use Hypervel\Tests\Ai\TestCase; +use InvalidArgumentException; +use PHPUnit\Framework\Attributes\DataProvider; use function Hypervel\Ai\agent; use function Hypervel\Tests\Ai\Fixtures\sentRequest; @@ -149,6 +149,141 @@ public function testProviderToolsAreSentWithoutAToolChoice(): void $this->assertArrayNotHasKey('tool_choice', $body['generation_config'] ?? []); } + #[DataProvider('escapedMetadataFilters')] + public function testFileSearchMetadataFilterEscapesQuotesAndTrailingBackslashes(string $method, string $expected): void + { + Http::fake([ + 'generativelanguage.googleapis.com/*' => $this->fakeTextResponse('ok'), + ]); + + agent(tools: [new FileSearch(['fileSearchStores/store123'], where: function (FileSearchQuery $query) use ($method): void { + if (in_array($method, ['whereIn', 'whereNotIn'], true)) { + $query->{$method}('category', ['x" OR tenant="globex', 'x\\']); + } else { + $query->{$method}('category', 'x" OR tenant="globex')->{$method}('path', 'x\\'); + } + })])->prompt('Question?', provider: 'gemini'); + + $this->assertSame($expected, sentRequest()->data()['tools'][0]['metadata_filter']); + } + + /** + * Provide each filter method with its escaped Gemini filter expression. + */ + public static function escapedMetadataFilters(): array + { + return [ + 'where' => ['where', '(category="x\" OR tenant=\"globex") AND (path="x\\\")'], + 'whereNot' => ['whereNot', '(category!="x\" OR tenant=\"globex") AND (path!="x\\\")'], + 'whereIn' => ['whereIn', '(category="x\" OR tenant=\"globex" OR category="x\\\")'], + 'whereNotIn' => ['whereNotIn', '(category!="x\" OR tenant=\"globex" AND category!="x\\\")'], + ]; + } + + #[DataProvider('invalidMetadataFilterKeys')] + public function testFileSearchMetadataFilterRejectsInvalidKeysBeforeSendingARequest(string $key): void + { + Http::fake(); + + try { + agent(tools: [new FileSearch(['fileSearchStores/store123'], where: [$key => 'value'])]) + ->prompt('Question?', provider: 'gemini'); + $this->fail('The key must be rejected.'); + } catch (InvalidArgumentException $exception) { + $this->assertSame("Invalid metadata filter key [{$key}].", $exception->getMessage()); + } + + Http::assertNothingSent(); + } + + /** + * Provide metadata filter keys that Gemini's filter syntax cannot name. + */ + public static function invalidMetadataFilterKeys(): array + { + return [ + ['tenant="acme" OR category'], + ['tenant name'], + ['tenant-name'], + ['1tenant'], + [''], + ["tenant\n"], + // A numeric array key reaches the provider as a string. + ['2026'], + ]; + } + + public function testFileSearchMetadataFilterGroupsOrdinaryConditionsAndPreservesNumericValues(): void + { + Http::fake([ + 'generativelanguage.googleapis.com/*' => $this->fakeTextResponse('ok'), + ]); + + agent(tools: [new FileSearch(['fileSearchStores/store123'], where: fn (FileSearchQuery $query) => $query + ->where('tenant', 'acme') + ->whereIn('category', ['guide', 'reference']) + ->whereNot('_status', 'draft') + ->where('version.number', 2.5) + ->whereIn('year', [2024, '2025']) + ->whereNot('revision', 0))])->prompt('Question?', provider: 'gemini'); + + $this->assertSame( + '(tenant="acme") AND (category="guide" OR category="reference") AND (_status!="draft") AND (version.number=2.5) AND (year=2024 OR year=2025) AND (revision!=0)', + sentRequest()->data()['tools'][0]['metadata_filter'], + ); + } + + public function testFileSearchMetadataFilterSupportsWhereNotIn(): void + { + Http::fake([ + 'generativelanguage.googleapis.com/*' => $this->fakeTextResponse('ok'), + ]); + + agent(tools: [new FileSearch(['fileSearchStores/store123'], where: fn (FileSearchQuery $query) => $query + ->where('tenant', 'acme') + ->whereNotIn('category', ['draft', 'internal']) + ->whereNotIn('year', [2023, '2024']))])->prompt('Question?', provider: 'gemini'); + + $this->assertSame( + '(tenant="acme") AND (category!="draft" AND category!="internal") AND (year!=2023 AND year!=2024)', + sentRequest()->data()['tools'][0]['metadata_filter'], + ); + } + + public function testFileSearchMetadataFilterOmitsAnEmptyWhereNotIn(): void + { + Http::fake([ + 'generativelanguage.googleapis.com/*' => $this->fakeTextResponse('ok'), + ]); + + agent(tools: [new FileSearch(['fileSearchStores/store123'], where: fn (FileSearchQuery $query) => $query + ->whereNotIn('category', []))])->prompt('Question?', provider: 'gemini'); + + $this->assertArrayNotHasKey('metadata_filter', sentRequest()->data()['tools'][0]); + + agent(tools: [new FileSearch(['fileSearchStores/store123'], where: fn (FileSearchQuery $query) => $query + ->where('tenant', 'acme') + ->whereNotIn('category', []))])->prompt('Question?', provider: 'gemini'); + + $this->assertSame('(tenant="acme")', Http::recorded()[1][0]->data()['tools'][0]['metadata_filter']); + } + + public function testFileSearchMetadataFilterRejectsAnEmptyWhereInBeforeSendingARequest(): void + { + Http::fake(); + + try { + agent(tools: [new FileSearch(['fileSearchStores/store123'], where: fn (FileSearchQuery $query) => $query + ->where('tenant', 'acme') + ->whereIn('category', []))])->prompt('Question?', provider: 'gemini'); + $this->fail('The empty list must be rejected.'); + } catch (InvalidArgumentException $exception) { + $this->assertSame('Gemini metadata filters cannot match [category] against an empty list of values.', $exception->getMessage()); + } + + Http::assertNothingSent(); + } + public function testMixedFunctionAndProviderToolsAreSentSideBySide(): void { Http::fake([ @@ -176,37 +311,6 @@ public function testCodeExecutionToolSendsACodeExecutionDefinition(): void $this->assertSame([['type' => 'code_execution']], sentRequest()->data()['tools']); } - public function testMetadataFiltersEscapeStringLiteralsAndPreserveNumericValues(): void - { - $provider = new GeminiProvider(self::createStub(Gateway::class), [], $this->app->make('events')); - $tool = new FileSearch(['store'], fn (FileSearchQuery $query) => $query - ->where('author', 'A "quoted" author') - ->whereNot('path', 'C:\notes') - ->whereIn('category', ['a"b', 'c\d', 42, '2026']) - ->where('enabled', true)); - - $this->assertSame( - 'author="A \"quoted\" author" AND path!="C:\\\notes" AND (category="a\"b" OR category="c\\\d" OR category=42 OR category=2026) AND enabled="1"', - $provider->fileSearchToolOptions($tool)['metadata_filter'], - ); - } - - public function testMetadataFiltersSupportExclusionListsAndNumericShorthandKeys(): void - { - $provider = new GeminiProvider(self::createStub(Gateway::class), [], $this->app->make('events')); - $tool = new FileSearch(['store'], fn (FileSearchQuery $query) => $query - ->whereNotIn('category', ['a"b', 'c\d', 42])); - - $this->assertSame( - '(category!="a\"b" AND category!="c\\\d" AND category!=42)', - $provider->fileSearchToolOptions($tool)['metadata_filter'], - ); - $this->assertSame( - '2026="x"', - $provider->fileSearchToolOptions(new FileSearch(['store'], ['2026' => 'x']))['metadata_filter'], - ); - } - /** * Find a tool in a Gemini request. */ From 403427f0e3020d3a53a9da3d5ffc8838e97aa23c Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 04:27:31 +0000 Subject: [PATCH 14/16] Document the AI SDK changes from this sync Document laravel/ai#1090, #1134, #1138, #1144 and #1146 in the AI SDK docs, porting laravel/docs 13.x at 12080c72699b where it covers them: - list the new failure events and when they are dispatched - port the Classifying Images section, adapted for the providers that now accept images and for named images, with its table of contents entry and upstream's missing Reranking Collections and Using Stored Files in Conversations entries - note that a structured run without a payload returns an empty structured value - note Gemini's metadata filter key and empty-list rules - show a stored file's raw HTTP response --- src/docs/ai-sdk.md | 53 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 51 insertions(+), 2 deletions(-) diff --git a/src/docs/ai-sdk.md b/src/docs/ai-sdk.md index e0dfdbd113..b74f102654 100644 --- a/src/docs/ai-sdk.md +++ b/src/docs/ai-sdk.md @@ -44,10 +44,13 @@ - [Caching Embeddings](#caching-embeddings) - [Cache Scopes](#embedding-cache-scopes) - [Reranking](#reranking) + - [Reranking Collections](#reranking-collections) - [Classification](#classification) + - [Classifying Images](#classifying-images) - [Yes or No Decisions](#yes-or-no-decisions) - [Choosing From Collections](#choosing-from-collections) - [Files](#files) + - [Using Stored Files in Conversations](#using-stored-files-in-conversations) - [Vector Stores](#vector-stores) - [Adding Files to Stores](#adding-files-to-stores) - [Usage](#usage) @@ -926,6 +929,8 @@ $response = (new SalesCoach)->prompt('Analyze this sales transcript...'); return $response['score']; ``` +When a run ends without structured output, such as when it pauses for [tool approval](#human-tool-approval) or is [stopped](#stopping-a-run) before its first step, the response's `structured` property is an empty array. + #### Nested Objects @@ -1878,7 +1883,7 @@ new FileSearch(stores: ['store_id'], where: [ ]); ``` -Azure OpenAI does not support file search metadata filters. +Azure OpenAI does not support file search metadata filters. When filtering with Gemini, metadata keys must begin with a letter or underscore and may contain only letters, numbers, underscores, and dots. Gemini also does not accept an empty `whereIn` list, while an empty `whereNotIn` list is ignored. For more complex filters, you may pass a closure that receives a `FileSearchQuery` instance: @@ -3196,6 +3201,36 @@ $result->usage; $result->meta->provider; ``` + +### Classifying Images + +When using OpenAI, an OpenAI-compatible provider, TypeSafe, or OpenRouter, images may be classified alongside the given content by passing them as the second argument to the `of` method. Images may be created using the same [file classes used for attachments](#attachments): + +```php +use Hypervel\Ai\Classification; +use Hypervel\Ai\Classification\Boolean; +use Hypervel\Ai\Files\Image; + +$result = Classification::of('Inspect the product in this photo.', [ + Image::fromPath($photo), +]) + ->question('damaged', new Boolean('Does the product have visible damage?')) + ->classify(provider: 'openai'); +``` + +When images are given string keys, each image is labeled with its key, so your questions may refer to the images by name: + +```php +$result = Classification::of('Two product photos.', [ + 'front' => Image::fromPath($front), + 'back' => Image::fromPath($back), +]) + ->question('damaged', new Boolean('Is the product in `front` or `back` damaged?')) + ->classify(provider: 'typesafe'); +``` + +The image content is sent with the request, so images stored with a provider and referenced using `fromId` can't be classified. OpenAI and OpenAI-compatible providers accept JPEG, PNG, GIF, and WebP images, OpenRouter accepts JPEG, PNG, and WebP images, and TypeSafe accepts JPEG and PNG images. + ### Yes or No Decisions @@ -3317,6 +3352,12 @@ $file->id; $file->mimeType(); ``` +Stored and retrieved file responses also expose the provider's [raw HTTP response](#raw-http-responses) via a `raw` property, which you may use to read provider-specific metadata such as the file's size: + +```php +$file->raw?->json('bytes'); +``` + To delete a file from the provider, use the `delete` method: ```php @@ -4170,16 +4211,19 @@ $store->assertAdded(fn (StorableFile $file) => $file->content() === 'Hello, Worl The Hypervel AI SDK dispatches a variety of [events](/docs/{{version}}/events), including: - `AddingFileToStore` +- `AddingFileToStoreFailed` - `AgentFailed` - `AgentFailedOver` - `AgentPrompted` - `AgentStreamAbandoned` - `AgentStreamed` +- `AudioFailed` - `AudioGenerated` - `ClassificationFailed` - `Classified` - `Classifying` - `CreatingStore` +- `CreatingStoreFailed` - `EmbeddingsFailed` - `EmbeddingsGenerated` - `FileAddedToStore` @@ -4190,24 +4234,29 @@ The Hypervel AI SDK dispatches a variety of [events](/docs/{{version}}/events), - `GeneratingEmbeddings` - `GeneratingImage` - `GeneratingTranscription` +- `ImageFailed` - `ImageGenerated` - `InvokingTool` - `PromptingAgent` - `ProviderFailedOver` - `RemovingFileFromStore` +- `RemovingFileFromStoreFailed` - `Reranked` - `Reranking` +- `RerankingFailed` - `StartingStep` - `StepCompleted` - `StepFailed` - `StoreCreated` - `StoreDeleted` - `StoringFile` +- `StoringFileFailed` - `StreamingAgent` - `ToolApprovalRequested` - `ToolApprovalResolved` - `ToolFailed` - `ToolInvoked` +- `TranscriptionFailed` - `TranscriptionGenerated` You can listen to any of these events to log or store AI SDK usage information. @@ -4216,7 +4265,7 @@ Resolving providers and tools and capturing conversation history happen before p `StartingStep` includes the instructions, tools, messages, model, and options actually sent for that step, including changes made by middleware. Listeners may inspect these values without calling the agent's `instructions` or `tools` methods again. Queued listeners receive an empty `tools` list, since live tool instances may contain closures or anonymous classes that cannot be serialized. -`ClassificationFailed` and `EmbeddingsFailed` report each failed provider attempt, including attempts followed by failover. They contain the invocation ID, provider, model, prompt, and original exception. Errors thrown by success-event listeners do not produce failure events. +When a provider call fails, embeddings, image, audio, transcription, reranking, and classification requests dispatch a matching failure event, such as `ImageFailed`, for each failed attempt, including attempts followed by failover. File uploads, store creation, and adding or removing store files dispatch failure events such as `StoringFileFailed`. Each failure event contains the starting event's data and the original exception. Errors thrown by success-event listeners do not produce failure events. `AgentStreamAbandoned` is dispatched when an unfinished stream is closed without an exception, such as when consumption stops early or an HTTP response write fails. A thrown exception instead follows the normal failure or failover path. Stream aggregation and conversation-persistence failures also report `AgentFailed`; errors after `AgentStreamed`, including errors in its listeners, do not report another terminal event. Replaying a completed response does not dispatch these lifecycle events again. From 775d760de05e72b3caa829de47cea6e125a56963 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 05:36:41 +0000 Subject: [PATCH 15/16] Keep integer keys in TypeSafe structured states with images laravel/ai#1134 (laravel/ai 1.x at d44ae14d17fb) merged image references into a structured TypeSafe state with an array spread, which renumbers integer keys. PHP stores numeric string keys such as record IDs as integers, so a state keyed by IDs lost them once images were attached. Merge with a union instead; the existing duplicate-key check already rejects overlaps, so no state value is replaced. Add an integer-key case to the image reference states. Formatting, static analysis and the AI suite pass. --- src/ai/src/Gateway/TypeSafeGateway.php | 3 ++- tests/Ai/Feature/Providers/TypeSafe/ClassificationTest.php | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/src/ai/src/Gateway/TypeSafeGateway.php b/src/ai/src/Gateway/TypeSafeGateway.php index b79947b11f..937ec42bf9 100644 --- a/src/ai/src/Gateway/TypeSafeGateway.php +++ b/src/ai/src/Gateway/TypeSafeGateway.php @@ -91,7 +91,8 @@ protected function stateWithImages(string|array $state, array $references): arra throw new InvalidArgumentException('TypeSafe adds images to a structured state under their attachment names, or [images] when unnamed; [' . implode(', ', array_keys($duplicates)) . '] already in the state.'); } - return [...$state, ...$images]; + // A union keeps integer keys, such as record IDs, which spreading would renumber; the check above rules out overlaps. + return $state + $images; } /** diff --git a/tests/Ai/Feature/Providers/TypeSafe/ClassificationTest.php b/tests/Ai/Feature/Providers/TypeSafe/ClassificationTest.php index 02e2d0a636..06b97d6d51 100644 --- a/tests/Ai/Feature/Providers/TypeSafe/ClassificationTest.php +++ b/tests/Ai/Feature/Providers/TypeSafe/ClassificationTest.php @@ -373,6 +373,11 @@ public static function imageReferenceStates(): array ['front' => Image::fromBase64(base64_encode('a'), 'image/png'), Image::fromBase64(base64_encode('b'), 'image/png')], fn (string $a, string $b): array => ['subject' => 'Broken lamp', 'front' => $a, 'images' => [$b]], ], + 'integer keys in a structured state' => [ + [42 => 'Broken lamp', 'subject' => 'Inspection'], + [Image::fromBase64(base64_encode('a'), 'image/png')], + fn (string $a): array => [42 => 'Broken lamp', 'subject' => 'Inspection', 'images' => [$a]], + ], 'named and unnamed images in a list state' => [ ['first', 'second'], ['front' => Image::fromBase64(base64_encode('a'), 'image/png'), 3 => Image::fromBase64(base64_encode('b'), 'image/png')], From b44851e7a86ea04437534e622506b3cb9740e312 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Tue, 20 Oct 2026 05:36:41 +0000 Subject: [PATCH 16/16] Ignore empty Anthropic text blocks after a server tool laravel/ai#1143 (laravel/ai 1.x at d44ae14d17fb) added a blank line before an empty text block that followed a server tool, so the final text gained a separator the model never wrote. The block also used up the pending separator, so a following line break was doubled. Streaming skips empty deltas, so streamed and final text differed. Skip empty blocks before applying the separator, as streaming does. Cover an empty block at the end, before text and before text that starts a new line. Formatting, static analysis and the AI suite pass. --- .../Concerns/ParsesTextResponses.php | 6 +++ .../Anthropic/ProviderToolCallsTest.php | 39 +++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/src/ai/src/Gateway/Anthropic/Concerns/ParsesTextResponses.php b/src/ai/src/Gateway/Anthropic/Concerns/ParsesTextResponses.php index 103b187d16..4e39170f33 100644 --- a/src/ai/src/Gateway/Anthropic/Concerns/ParsesTextResponses.php +++ b/src/ai/src/Gateway/Anthropic/Concerns/ParsesTextResponses.php @@ -116,6 +116,12 @@ protected function extractText(array $content): string if ($type === 'text') { $blockText = (string) ($block['text'] ?? ''); + + // Streaming skips empty deltas, so an empty block must not use up the pending separator either. + if ($blockText === '') { + continue; + } + $text .= ($afterTool && $text !== '' ? $this->textSeparator($text, $blockText) : '') . $blockText; $afterTool = false; } elseif ($this->isToolBlock($type)) { diff --git a/tests/Ai/Feature/Providers/Anthropic/ProviderToolCallsTest.php b/tests/Ai/Feature/Providers/Anthropic/ProviderToolCallsTest.php index a8651b945f..09e9b6dd55 100644 --- a/tests/Ai/Feature/Providers/Anthropic/ProviderToolCallsTest.php +++ b/tests/Ai/Feature/Providers/Anthropic/ProviderToolCallsTest.php @@ -8,6 +8,7 @@ use Hypervel\Support\Facades\Http; use Hypervel\Tests\Ai\Fixtures\Agents\AssistantAgent; use Hypervel\Tests\Ai\TestCase; +use PHPUnit\Framework\Attributes\DataProvider; class ProviderToolCallsTest extends TestCase { @@ -106,4 +107,42 @@ public function testTextAfterAServerToolIsNotSeparatedWhenTheModelAlreadyEndedTh $this->assertSame("Let me check.\nDone", $response->text); } + + #[DataProvider('textAfterAnEmptyBlock')] + public function testAnEmptyTextBlockAfterAServerToolAddsNoSeparator(array $followingBlocks, string $expected): void + { + Http::fake([ + 'api.anthropic.com/*' => Http::response([ + 'id' => 'msg_1', + 'type' => 'message', + 'role' => 'assistant', + 'model' => 'claude-sonnet-4-6', + 'content' => [ + ['type' => 'text', 'text' => 'Let me check.'], + ['type' => 'server_tool_use', 'id' => 'srvtoolu_1', 'name' => 'web_search', 'input' => ['query' => 'hypervel ai']], + ['type' => 'web_search_tool_result', 'tool_use_id' => 'srvtoolu_1', 'content' => []], + ['type' => 'text', 'text' => ''], + ...$followingBlocks, + ], + 'stop_reason' => 'end_turn', + 'usage' => ['input_tokens' => 10, 'output_tokens' => 5], + ]), + ]); + + $response = (new AssistantAgent)->prompt('Search', provider: 'anthropic'); + + $this->assertSame($expected, $response->text); + } + + /** + * Provide the blocks after an empty text block and the expected response text. + */ + public static function textAfterAnEmptyBlock(): array + { + return [ + 'nothing' => [[], 'Let me check.'], + 'text' => [[['type' => 'text', 'text' => 'Done']], "Let me check.\n\nDone"], + 'text starting a new line' => [[['type' => 'text', 'text' => "\nDone"]], "Let me check.\nDone"], + ]; + } }