Repository navigation
Fixes features speed - #92
Merged
Merged
Conversation
…enever `charset_normalizer` could not detect an encoding (`str(None)`). This made the intended fallback unreachable and silently discarded the content, including any IOCs in it. The fallback is now UTF-8 followed by latin-1, which cannot fail and therefore always preserves the ASCII parts of the input.
…: the check for a header in the first line compared bytes to a list and was always true, and the rebuilt data had all line feeds removed, so for LF-only files the headers and the body were merged into one line (and the body hash of valid messages changed). The garbage in front of the first header field is now skipped by slicing the original data, which also removes the documented performance penalty of this option.
… with `include_attachment_data=True`, as `decode.json_serial()` could not serialize the `bytes` attachment data. It is now exported as (base64) text.
… dates close to `datetime.min`/`datetime.max` (e.g. `Date: Fri, 31 Dec 9999 23:59:59 -0500`) when converting them to UTC, which allowed a crafted `Date` header to break `export_to_json()`. Such dates are now exported with their original offset.
…ed words with an unknown charset (e.g. `=?foo-bar?q?x?=`), which allowed a crafted header to abort the parsing of the whole message. The charset is now detected instead.
… between runs (depending on `PYTHONHASHSEED`) as it was built from a `set`. Headers are now returned in the order of their first occurrence in the message. Additionally, headers whose names only differ in case (e.g. `X-Foo` and `x-foo`) no longer have their values duplicated.
…ounter used by `traverse_multipart()` was passed by value and thus never advanced between parts. It is now shared by the whole traversal, so unnamed attachments are numbered `part-000`, `part-001`, ... (every attachment consumes a number). As a consequence the `counter` argument of `traverse_multipart()` and `prepare_multipart_part_attachment()` is now an iterator instead of an `int`.
…ent ids, as version 1 UUIDs embed the MAC address of the host and a timestamp and so leak information about the machine running the parser. Random version 4 UUIDs are used instead.
…struction time; the e-mail regex is per-instance state passed to `CustomPolicy`, `parserouting()` and `workaround_bug_27257()`, so parsers with different `email_force_tld` settings can coexist (also across threads).
…y missing from the result in the default (hashed) mode. They are now reported as `uri_noscheme_hash` (SHA-256 of the lower-cased value, like `uri_hash`). With `include_raw_body=True` they were already reported as `uri_noscheme`.
…omplete dates (e.g. `March 2024`, `Tue, 5 Mar` or `12:30`) from the current date, which made the result depend on the day of the analysis. Missing parts are now taken from `1970-01-01`.
…f most stdlib defects is empty). The name of the defect class, e.g. `NoBoundaryInMultipartDefect`, is reported now, followed by the message if there is one.
…re'`, which dropped undecodable bytes and glued the neighbouring characters together, creating indicators which are not in the message (`http://good\xff.evil.com` became `http://good.evil.com`). Undecodable bytes in the body are now replaced with a space, so they separate indicators instead of joining them or becoming part of them. An unknown charset falls back to charset detection instead of dropping all non-ASCII characters (new `decode.decode_body()`).
…icates: `uri` and `uri_noscheme` were built from a `set` (order depending on `PYTHONHASHSEED`, which also made the derived `domain` and `ip` lists random) and `uri_hash` held one entry per occurrence of a URL. All are now unique and in order of first appearance.
…parts of large bodies (the rewind logic could move the end of a slice behind its start, so text was skipped and URLs, domains and addresses at slice borders were missed or truncated). Slices now end at the next whitespace and cover the body completely. The unused look-around heuristics are removed, which also makes parsing of large bodies faster.
…t loses bytes (declared charsets are often wrong, e.g. `us-ascii` for UTF-8 text), so non-ASCII indicators are not replaced by U+FFFD. Bodies larger than `decode.MAX_CHARSET_DETECTION_SIZE` are exempt to keep parsing fast.
…nstance (bounded LRU), as bodies contain the same hosts over and over. `domain_force_tld` and `ip_force_routable` must therefore not be changed after construction.
…extracted from inside longer dotted numbers, words or host names (`1.2.3.4.5`, `v1.2.3.4`, `1.2.3.4.nip.io`, `std::vector`, `1:2:3:4:5:6:7:8:9`), names with a numeric top level domain (`999.1.1.1`, `v1.20.30`) are no longer accepted as hosts, and file names (`invoice.pdf`, see the new `eml_parser.constants.FILE_EXTENSIONS`) are not reported as domains, unless the extension is a top level domain according to the public suffix list (e.g. `evil.zip`).
…`http://3232235777/`, `http://0xC0A80101/`, `http://0xC0.0xA8.1.1/`, `http://127.1/`), which browsers resolve like normal addresses and which are a common phishing/filter evasion trick; their address is added to the observed IPs (honouring `ip_force_routable` and `whiteip`). New `EmlParser.get_obfuscated_ip()`, based on `socket.inet_aton()`.
…elding no body and no attachment at all, hiding all URLs, e-mail addresses and attachments in them. Such parts are now split using the first line looking like a delimiter, nested ones too, and a part without any delimiter is scanned as text (`EmlParser.repair_multipart()`). The defects are still reported.
…t` `formaction`, `iframe`/`frame`/`script`/`embed`/`source`/`audio`/`video`/`track`/`input` `src`, `area`/`link`/`base` `href`, `object data`, `body background`, `xlink:href`. These are the usual targets of phishing forms, hidden frames, remote scripts and redirects. Attribute values are normalised like a browser does before use (`decode.normalize_attribute_url()`: character references, tabs and line breaks inside the scheme, backslashes), protocol-relative URLs (`//host/path`) are reported and relative references (`images/logo.png`, `page.html`) are no longer mistaken for hosts.
…line text and `.htm`/`.html` attachments were scanned like the text of the e-mail. Now attachments with a content type (`image/svg+xml`, `application/javascript`, ...) or extension (`.txt`, `.url`, `.svg`, `.js`, `.vbs`, `.ps1`, `.hta`, `.bat`, `.ics`, `.eml`, ...; see `eml_parser.constants`) of scripts, shortcuts, web pages, SVG and invitations are scanned, up to `MAX_TEXT_ATTACHMENT_SIZE` (10 MiB). Data attachments (XML, JSON, CSV, logs) are not scanned as they mostly consist of namespace and schema URLs. Scanned attachments also remain in the attachment list.
…i-control` (right-to-left override U+202E and friends), `filename-control-chars`, `filename-double-extension` (`invoice.pdf.exe`), `filename-trailing-dot-or-space`, `filename-padding` (a run of spaces hiding the real extension) and `extension-content-mismatch` (a PE/ELF executable named like a document or picture or without extension). New module `eml_parser.anomalies`.
…, ...) results found in the `Authentication-Results` and `Received-SPF` headers, e.g. `{"spf": ["pass"], "dkim": ["fail"]}`. These headers can be forged by the sender and are only meaningful if added by a server you trust.
…eing reported as domains.
…`\\host\share\file`, the WebDAV forms `\\host@SSL\DavWWWRoot\x` and `\\host@8080\x`, `\\?\UNC\host\share`) and `file://host/share`, `file:////host/share` and `smb://host/share` URLs. Opening such a path makes the client contact and authenticate to the host (NTLM hash leak) or fetch content from it. They are reported as `unc` (`unc_hash` without `include_raw_body`) and their host is added to the observed domains and IPs (numeric hosts and encoded IPv6 `*.ipv6-literal.net` are resolved; bare NetBIOS names, `localhost`, escaped text such as `C:\\Users\\Bob` and local `file:///C:/` URLs are ignored).
… 40 ms): the public suffix list is parsed once per setting of `domain_force_tld` and shared by all parsers instead of once per parser. This matters most when a new parser is created for every message and makes the test suite about three times faster.
…2` attachments whose text the generator cannot encode (the size of the attachment is now calculated from a lossy rendering instead). - Fix the multipart repair (`EmlParser.repair_multipart()`) looping forever if the stdlib decodes the guessed boundary (e.g. an encoded word like `=?x?q?=ff?=b`), and raising `RecursionError` for thousands of nested multiparts. Such parts are now scanned as text. - Fix `parse_authentication_results()` being quadratic for a header with deeply nested comments (it now uses `decode.noparenthesis()`, which works in one pass). - Fix hostile headers raising `TypeError` (RFC 2047 words in the relaxed header parsing path) or `ValueError` (address parts with CR/LF) in `parse_email()` and `headeremail2list()`, and `decode.robust_string2date()` raising `decimal.InvalidOperation` for absurd numbers in a date. - Move `routing.noparenthesis()` to `decode.noparenthesis()`, as it is no longer specific to the received lines, and add the optional `replacement` argument (inserted in place of every removed pair).
…header parsers of the stdlib are quadratic in the length of a value (a Content-Type with 400 KB of parameters took a minute). Values longer than `constants.MAX_PARSED_HEADER_SIZE` (8 KiB) are not run through them; addresses are still extracted, other values are kept as text and an overlong `Date` is the default date.
…quadratic on bodies with a long tag opening without `>` (1.4 MB of `<form ` took 87 s): the old pattern restarted a lazy scan at every `<tag` position. The scan is now two-stage and linear: it locates the attribute values (`regexes.url_regex_href_attr`) and verifies that the nearest tag opening (up to `regexes.URL_HREF_TAG_WINDOW` characters) carries the attribute (`regexes.findall_href()`). Three minor behaviour differences vs. the old pattern are documented in `eml_parser.regexes.py`.
…ex.md; add changleog to docs; reduce api documentation to what should actually be used, not the whole lib.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.