Skip to content

fix(batch): report records that fail model conversion with non-validation errors - #8662

Open
soujasK wants to merge 1 commit into
aws-powertools:developfrom
soujasK:fix/batch-model-conversion-error
Open

soujasK wants to merge 1 commit into
aws-powertools:developfrom
soujasK:fix/batch-model-conversion-error

Conversation

@soujasK

@soujasK soujasK commented Oct 25, 2026

Copy link
Copy Markdown

Issue number: closes #8660

Summary

Changes

When converting a record to the customer's model raised something other than the model's ValidationError (Pydantic v2 only wraps ValueError/AssertionError from validators, so e.g. a KeyError escapes as-is), BatchProcessor._process_record and AsyncBatchProcessor._async_process_record called failure_handler(record=data) with data still None. None ended up in fail_messages, and _clean() then crashed in _collect_*_failures with AttributeError: 'NoneType' object has no attribute 'message_id'.

data is None in the except block means the conversion itself failed, so those records now go through the existing _register_model_validation_error_record path from #2091, which converts the raw record to the event source data class and registers it with the original exception.

Tests: a model whose field_validator raises KeyError for one record of three, for the sync and async processors. Both failed with the AttributeError before the change and pass after it; batch.exceptions keeps the original KeyError. tests/functional/batch passes on Linux (Python 3.12), along with ruff, mypy, ty, bandit and xenon.

User experience

Before: one malformed record fails the whole invocation with an AttributeError that hides the real error, and every record in the batch is retried, including the ones that already succeeded.

After: the malformed record is reported in batchItemFailures (as an event source data class, as documented for early model failures) and the rest of the batch is processed normally.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@soujasK
soujasK requested a review from a team as a code owner October 25, 2026 18:05
@soujasK
soujasK requested a review from hjgraca October 25, 2026 18:05
@boring-cyborg

boring-cyborg Bot commented Oct 25, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@soujasK
soujasK force-pushed the fix/batch-model-conversion-error branch from c3afb78 to e316530 Compare October 26, 2026 10:41
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: batch processor crashes with AttributeError when a model validator raises a non-ValueError

2 participants