3.11 doesn't accept tag properties containing html strings anymore

[FredZinelli]

Hello,

I just got some very weird behaviors in a mkdocs project using markdown 3.11: it seems the new version doesn't parse/render correctly tag properties containing html code anymore.

I could confirm the following happens when the md is converted to html in Markdown.convert. Here is a minimal POC :

Markdown content:

<span data-tip-txt="<kbd>Ctrl</kbd>">Anything</span>

Result:

Related html:

<p>&lt;span data-tip-txt="<kbd>Ctrl</kbd>"&gt;Anything</p>

Expected html:

<p><span data-tip-txt="<kbd>Ctrl</kbd>">Anything</span></p>

---

Some observations:

• This doesn't happen with previous versions of markdown (I tested 3.9 and 3.10 successfully).
• This happens with the default configuration of mkdocs (no theme, no plugins, no md extensions at all)

Regards,
Fred

[facelessuser]

Seems like it was introduced by #1621. I need to dig a little and see if we can resolve this, as the change is very useful for performance.

[facelessuser]

I think I understand what is happening. Before, we would not capture the opening <span> because it had < and > in the attribute, but then we capture <kbd> in the attribute as HTML. Then we would rescan the opening <span> and capture it as HTML. This produces invalid HTML.

So the new parsing logic is more efficient. It sees <span> as invalid, then captures <kbd> and never reparses <span>.

Let's be clear though, <span data-tip-txt="<kbd>Ctrl</kbd>">Anything</span> is invalid HTML. Or at least was.

AI seems to say this:

Modern HTML specifications (updated May 2025) now require browsers to always escape < and > in attributes to prevent mutation XSS, but developers should still explicitly escape delimiters to ensure safety across all contexts and browsers. Using framework-specific functions like escapeHtmlAttr() or htmlspecialchars() with ENT_QUOTES is the standard best practice.

Just throwing your input into browsers seems to confirm this as it renders just fine.

So, I guess we have a couple of options.

1. Say the user should just escape < and > in this case. Considering how browsers have evolved, maybe this is the lazy way out.
2. Update things so we capture < and > in attributes and either escape them ourselves or just pass them through, as browsers will protect against them these days anyway. I'm not sure I care which way we go here, but I think maybe this is the proper way moving forward.

[FredZinelli]

I'd say it's your call. Personnally, I'd prefer this way, ofc:

or just pass them through, as browsers will protect against them these days anyway

but I guess I could update my codes to handle escaping/unescaping the <...> where needed, if I have to.

[facelessuser]

I'll let @waylan weigh in.

In short, I don't think what you are experiencing is a bug per se. If anything, I think previous behavior was technically a bug, but accidentally turned out working HTML. We should have never parsed an HTML tag was contained within the start of another HTML tag, it just happened to work out okay. The new parsing logic prevents that.

Now, purposely including it in an HTML attribute with targeted patterns would make more sense if we move in this direction. I guess the outcome is the same, but one is purposeful, and one was just a side effect you were getting by chance.

[facelessuser]

A potential fix:

>>> import markdown
>>> text = '<span data-tip-txt="<kbd>Ctrl</kbd>">Anything</span>'
>>> markdown.markdown(text)
'<p><span data-tip-txt="<kbd>Ctrl</kbd>">Anything</span></p>'

diff --git a/markdown/inlinepatterns.py b/markdown/inlinepatterns.py
index 1908625..58bf082 100644
--- a/markdown/inlinepatterns.py
+++ b/markdown/inlinepatterns.py
@@ -159,7 +159,7 @@ AUTOMAIL_RE = r'<([^<> !]+@[^@<> ]+)>'
 """ Match an automatic email link (`<me@example.com>`). """

 HTML_RE = (
-    r'(<(\/?[a-zA-Z][^<>@ ]*( [^<>]*)?|'          # Tag
+    r'''(<(\/?[a-zA-Z][^<>@ ]*(?:\s+[\w\-:]+(?:\s*=\s*(?:"[^"]*"|'[^']*'|[^\s"'=<>]+))?)*\s*/?|'''  # Tag
     r'!--(?:(?!<!--|-->).)*--|'                   # Comment
     r'[?](?:(?!<[?]|[?]>).)*[?]|'                 # Processing instruction
     r'!\[CDATA\[(?:(?!<!\[CDATA\[|\]\]>).)*\]\]'  # `CDATA`

➜  markdown git:(master) ✗ python3 -m unittest
........................................../Users/facelessuser/Code/github/markdown/markdown/core.py:450: DeprecationWarning: codecs.open() is deprecated. Use open() instead.
  output_file = codecs.open(output, "w",
.......................................................................................s...................................................................................ssssssssssssssssssssssssssssssssssssssssssssss............................................................................ss..........................................................................................................................................................................................................ss..............................................ssssssssssssssssssssssssssssssssssssssss...........................sssssssssssssssssss.........................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 1098 tests in 0.286s

[waylan]

I was surprised to learn that this is now considered valid HTML. My immediate reaction was to say it is invalid HTML and the fact that it worked before would have been a bug. Technically that is still true as it would have been invalid at the time that the code was written.

Given the current spec and browser behavior, my inclination is to consider this a low priority bug (the kind we never get around to fixing). Any other old-school Markdown implementation is going to have the same behavior and I have no inclination to do work to veer from that behavior. But, if someone wants to do the work, then I'll accept a reasonable fix.

[waylan]

One thing that bugs me about this is that it would be impossible to generate the desired output from an etree object. The serializer would escape the brackets in the attribute. Therefore, an extension would not be able to build an etree object to insert in the document. Such an extension would be required to stash its output as raw HTML. I'm not sure if that changes my view at all...

[facelessuser]

You are correct that etree, in HTML mode, will at the very least escape > (< and > in XML mode).

Browsers, while accepting them, force them to be escaped when rendered, and it is still recommended that they are escaped, but browsers accepting and forcing them to be escaped is just an extra security measure.

I don't think we need to guarantee that < and > will remain unchanged, but if they happen to be preserved, I don't have issues with that either. There are already ways in which extensions can get < and > into attributes. As a matter of fact, the HTML parser already accepts that:

>>> import markdown
>>> text = """
... <div attr="<kbd>">
...
... **test**
...
... </div>
... """
>>> markdown.markdown(text)
'<div attr="<kbd>">\n\n**test**\n\n</div>'

I'm personally okay with adjusting inline HTML patterns to accept the input.

[FredZinelli]

I used this pattern without asking myself if it was valid html or not, actually. I just needed something, saw it was working and went for it. 😅

@facelessuser 's suggestion to fix it seems good enough. If you are ok with this update, I'll wait for its publication, otherwise I'll just update my project to account for the new behavior (I won't have time to dig into "yet another project" to find another solution).

---

Edit: btw, I don't understand the "invalid etree" argument. I mean, Reasoning aside of concrete html specs, I'd just expect that a string content of a tag property should just be considered a string content, whatever the content is. I mean, I'd expect there would be nothing to interpret with it, it should be considered just plain text data. Any tool that makes (or tries to make) actual html of it, I would consider it a bug, actually, the parser not taking in consideration the fact the delimiters are inside a string. I'm not saying anything you say is wrong or arguing anything or wanting to go against concrete html specs, here. I'm just stating the first thing that would come to my mind when facing a tool parsing something in way it "obviously" shouldn't (at first sight at least).

[facelessuser]

I've created a PR for this at #1649.