Skip to content

🐛 FIX: parse front matter closed by the YAML ... marker - #158

Open
feiiiiii5 wants to merge 1 commit into
executablebooks:masterfrom
feiiiiii5:fix/front-matter-dots-terminator
Open

feiiiiii5 wants to merge 1 commit into
executablebooks:masterfrom
feiiiiii5:fix/front-matter-dots-terminator

Conversation

@feiiiiii5

Copy link
Copy Markdown

Summary

Front matter closed by the YAML ... document-end marker was not parsed correctly. Two distinct silent failures, both reproduced against main at 519953d:

>>> MarkdownIt().use(front_matter_plugin).render("---\ntitle: a\n...\n")
'<hr />\n<p>title: a\n...</p>\n'
>>> MarkdownIt().use(front_matter_plugin).render("---\ntitle: a\n---\n")
''

The two forms should be equivalent. Instead, when ... closed the block:

  • the block was dropped entirely — no front_matter token was produced at all, and the front matter leaked into the rendered document as visible text (an <hr /> plus a paragraph). This happens when ... is the last line.
  • the marker stayed in the token content — where it did parse, token.content was 'title: a\n...' instead of 'title: a', unlike the dashed form which excludes its terminator.

Root cause

mdit_py_plugins/front_matter/index.py, in the search loop:

while True:
    nextLine += 1
    if nextLine >= endLine:
        return False

    if state.src[start:maximum] == "...":   # tests the PREVIOUS line
        break

    start = state.bMarks[nextLine] + state.tShift[nextLine]   # moves onto the current line
    maximum = state.eMarks[nextLine]

start/maximum are only moved onto nextLine after the ... test, so the test inspects line nextLine - 1. The content slice further down is computed as state.src[state.bMarks[startLine + 1] : state.eMarks[nextLine - 1]], i.e. as if the closer were on nextLine. The check is exactly one line out of phase with the slice, which is why the two symptoms differ: at end-of-document the loop runs off the end and return False discards the block, while with a body after it the marker survives into content.

Fix

Move the test to after the per-line refresh, and set auto_closed so the line is consumed rather than re-parsed as content. auto_closed is read only at state.line = nextLine + (1 if auto_closed else 0).

    start = state.bMarks[nextLine] + state.tShift[nextLine]
    maximum = state.eMarks[nextLine]

    if state.src[start:maximum] == "...":
        auto_closed = True
        break

token.map for the dotted form becomes [0, 3], matching the dashed form.

Tests

tests/test_front_matter.py gains two tests, both red before the fix:

  • test_token_content_excludes_the_closer, parametrized over --- and ..., asserting the token content is a: 1 in both cases.
  • test_dots_closer_on_the_last_line, asserting the ...-at-EOF render equals the --- render (both empty).

The existing test_all cannot see either half of this: the front_matter token is hidden=True, so only md.render() is asserted, and a dropped block renders as visible text rather than failing. The dotted case was otherwise only covered by the should parse until triple dots fixture, which asserts rendered output and so never inspected content.

pytest tests -q    514 passed   (511 before, +3 new)
ruff check / ruff format --check    clean
mypy mdit_py_plugins/front_matter/index.py    Success

Relation to #155

#155 also edits this line, adding marker_chr == "-" to the same condition, but it leaves the check in the same position — so that branch does not have this bug fixed. The two changes compose: applying this fix on top of #155 moves the gated test onto the correct line. I have not rebased onto #155 because the conflict is a single line and the ordering is a maintainer decision; happy to rebase if you would rather land it after #155.

Separate observation, not changed here

The comment two lines above says "unclosed block should be autoclosed by end of document" while the code does return False, discarding the block. That contradiction predates this PR and changing it would alter behaviour for genuinely unclosed input, so I have left it alone.

The `...` document-end marker was tested against the *previous* line,
because `start`/`maximum` are only moved onto the current line after the
test. Two things followed, both silent:

A block whose last line was `...` was dropped entirely, so the front
matter leaked into the document as visible text:

    MarkdownIt().use(front_matter_plugin).render("---\ntitle: a\n...\n")
    before  '<hr />\n<p>title: a\n...</p>\n'
    after   ''

And where it did parse, the marker stayed in the token content, so
`content` was `'title: a\n...'` rather than `'title: a'`, unlike the
dashed form.

Move the test to after the per-line refresh and mark the line consumed.
`auto_closed` is only read at `state.line = nextLine + (1 if
auto_closed else 0)`, so setting it is what stops the marker being
re-parsed as content.

`test_all` could not catch either half: the front_matter token is
hidden, so only the rendered output is asserted, and a dropped block
renders as visible text rather than failing. The new tests assert the
token itself.

    pytest tests -q    514 passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant