Skip to content

Add metadata envelope to queue messages - #197

Open
skie wants to merge 1 commit into
cakephp:3.xfrom
skie:feature/metadata
Open

skie wants to merge 1 commit into
cakephp:3.xfrom
skie:feature/metadata

Conversation

@skie

@skie skie commented Sep 22, 2026

Copy link
Copy Markdown
Member

Add metadata envelope to Queue messages

Allows bookkeeping fields (tags, _uniqueId, batch_id) to travel next to the job payload instead of inside it, so data stays pure and DTO-safe.

Why this is needed:

  • Today there is nowhere else to put envelope fields: Monitoring tools mutates the payload in place ($data['tags'] = ..., $data['_uniqueId'] = ...). That blocks DTO objects (array writes on an object), pollutes monitoring data, and leaks mutator keys into Message::getDto() hydration.
  • Message properties/headers cannot hold it: FailedJobsListener persists only data + requeueOptions to the database, so anything stored in properties is silently lost on failure and requeue.
  • requeueOptions cannot hold it either: that key is routing (config, priority, queue), mapped to FailedJobsTable columns and read by queue requeue. Mixing domain envelope data into routing breaks that contract.

Key Features:

  • QueueManager::push() now accepts a metadata option (array) and records it on the message body as a sibling of data. Omitted when empty or not an array.
  • New Message::getMetadata() returns the envelope array . getArgument() is unchanged and returns the pure payload.
  • Failed jobs keep the envelope: new nullable metadata column on queue_failed_jobs (migration AddMetadataToFailedJobs), FailedJobsListener persists it, RequeueCommand passes it back into push() via FailedJob::decoded_metadata.

Usage:

QueueManager::push(ProcessOrderJob::class, $order, [
    'metadata' => ['tags' => ['finance'], '_uniqueId' => $uuid],
]);
class ProcessOrderJob implements JobInterface
{
    public function execute(Message $message): ?string
    {
        $order = $message->getDto(OrderDto::class);
        $tags = $message->getMetadata()['tags'] ?? [];
        return Processor::ACK;
    }
}
// Failed jobs round-trip without losing the envelope:
$row->decoded_metadata; // ['tags' => [...], '_uniqueId' => ...]
// queue requeue re-pushes data + metadata together

QueueManager::push() accepts a metadata option recorded on the
message body as a sibling of data, Message::getMetadata() exposes
it, and failed jobs persist it through store and
requeue.
@markstory markstory added this to the 3.x milestone Sep 25, 2026
public function change(): void
{
$table = $this->table('queue_failed_jobs');
$table->addColumn('metadata', 'text', [

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you're storing JSON in the column, why not use a json type? That would save you having to manually encode JSON when persisting/accessing entity data.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completely agree, but I was trying to be consistent with current data column in exists migration.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think those columns predated proper JSON column support in the ORM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants