From 23a291533a4b3048f7786adc7b953b0a9d788eed Mon Sep 17 00:00:00 2001 From: Anton Wiklund Date: Wed, 11 Nov 2026 23:48:05 +0100 Subject: [PATCH] fix(markdown): show hidden characters in deep source as markers Markdown shows bidi controls, zero-width and other hidden characters in code and link text as markers. What nests past MAX_DEPTH shows as its source instead, and that source kept them as written, so code nested that deep, and the text of a link one level above it, could read other than written with no marker. The Studio check named only tooDeep for such code. Text past MAX_CHARS or MAX_WORK, shown as plain text, kept them as written too, code included. What nests past MAX_DEPTH, raw HTML (in a link's text too) and a table past its caps all show as their source, and that source now shows its hidden characters as markers. So does Markdown past the parser's limits, where the app shows it as plain text; plain text that is not Markdown keeps them as written. A link whose own text would nest past MAX_DEPTH shows as its source rather than as a followed link, so a followed link's own text is never source; markup nested deeper inside it can be, its hidden characters as markers. inspectMarkdown notes code or a link shown as source that holds hidden characters, and markdownProblems names them as hiddenControl, so the check matches what the app shows. Plain text holding one no longer counts as shown as written. Refs: GHSA-9wqg-2c2v-j9v8 --- docs/markdown.md | 21 +++-- packages/markdown/src/parse.test.ts | 105 ++++++++++++++++++++++++- packages/markdown/src/parse.ts | 67 +++++++++++++--- packages/markdown/src/problems.test.ts | 15 ++++ packages/markdown/src/problems.ts | 7 ++ packages/ui/src/ui/DataText.test.tsx | 22 ++++++ packages/ui/src/ui/DataText.tsx | 29 +++++-- 7 files changed, 233 insertions(+), 33 deletions(-) diff --git a/docs/markdown.md b/docs/markdown.md index 911d4db3a..eae3cf6bd 100644 --- a/docs/markdown.md +++ b/docs/markdown.md @@ -77,11 +77,11 @@ package's own tree, folded to what the app shows: | In the text | In the tree | |---|---| | a heading | a paragraph in bold: data never adds to the page's outline | -| raw HTML, block or inline | its source, as text | +| raw HTML, block or inline | its source, as text, its hidden characters as markers | | a picture, inline or by reference | its alt text: never loaded or linked | | a reference link | a link to its definition's URL (the first definition of a label wins); definitions show nothing | | a code block's info string | its first word, lower-cased, when it matches `[a-z0-9+#-]{1,20}`; else none | -| hidden characters in code or link text: every format character (Unicode category Cf, such as the bidi controls U+202A–202E and U+2066–2069, the zero-width characters U+200B–200F, U+2060 and U+FEFF, the soft hyphen U+00AD, the invisible operators U+2061–2064 and the tag characters U+E0000–E007F), and characters that show nothing or look blank yet count (the Hangul fillers U+115F, U+1160, U+3164 and U+FFA0, the combining grapheme joiner U+034F, Khmer's U+17B4–17B5, and the variation selectors U+180B–180F, U+FE00–FE0F and U+E0100–E01EF), a picture's alt text in a link included | a visible marker such as `⟨U+202E⟩` ("Trojan Source": code that reads other than it runs) | +| hidden characters in code or link text: every format character (Unicode category Cf, such as the bidi controls U+202A–202E and U+2066–2069, the zero-width characters U+200B–200F, U+2060 and U+FEFF, the soft hyphen U+00AD, the invisible operators U+2061–2064 and the tag characters U+E0000–E007F), and characters that show nothing or look blank yet count (the Hangul fillers U+115F, U+1160, U+3164 and U+FFA0, the combining grapheme joiner U+034F, Khmer's U+17B4–17B5, and the variation selectors U+180B–180F, U+FE00–FE0F and U+E0100–E01EF), a picture's alt text in a link included, and in anything shown as its source, however deep it nests | a visible marker such as `⟨U+202E⟩` ("Trojan Source": code that reads other than it runs) | | a table body row's cells past the header's | dropped, as GFM has it; a short row isn't padded | | Function | Module | What it gives | @@ -92,7 +92,7 @@ package's own tree, folded to what the app shows: | `plainText(text)` | `plainText` | the text as plain text | | `labelText(text, max?)` | `plainText` | a short name: the plain text of its first block that shows any | | `liveLink(url)` | `links` | whether a link is followed, and its host | -| `inspectMarkdown(text)` | `parse` | whether the text is one paragraph as written (a heading isn't), and notes of what the tree folds away or a check looks at: raw HTML, pictures, links (an autolink told apart), code outside a link, character references, what nests too deep to show as written, a table past its caps, a heading underlined with dashes | +| `inspectMarkdown(text)` | `parse` | whether the text is one paragraph as written (a heading isn't), and notes of what the tree folds away or a check looks at: raw HTML, pictures, links (an autolink told apart), code outside a link, character references, what nests too deep to show as written (and code or a link there with hidden characters), a table past its caps, a heading underlined with dashes | | `markdownProblems(text, rule)` | `problems` | what wouldn't show as its author meant, by the [rules for a release](#rules-for-a-release), in a field held to `SIDE`, `OPTION` or `PROSE` | | `chunksOf(blocks)` | `chunks` | the [chunks](#chunks) blocks are shown in, split at their top-level rules | | `splitAtRules(blocks)` | `chunks` | the pieces between top-level rules, empty ones included | @@ -107,7 +107,8 @@ shapes of Markdown make the parser slow. `MAX_WORK`. - **`MAX_CHARS`, 20,000 characters.** A longer text is plain, its - source still readable. The longest course text is about 2,600. + source still readable, its hidden characters as markers. The longest + course text is about 2,600. - **`MAX_WORK`, 40 units of the parser's work for each character** (of at least 1,000). Each try of one of the parser's rules, each step through a link's label and each token it makes is a unit, counted as @@ -130,7 +131,8 @@ shapes of Markdown make the parser slow. its own. An `href` takes at most three characters for each byte, a percent escape, so the links a text shows hold at most a small multiple of its length. Past the budget the parse stops, and the text - is plain, so no text costs more than time linear in its length. + is plain, as past `MAX_CHARS`, so no text costs more than time linear + in its length. markdown-it reads most text in such time, but some shapes make it read the rest of a text again and again: a block quote after each line of a paragraph, or link openers before an unclosed declaration @@ -141,7 +143,10 @@ shapes of Markdown make the parser slow. source, so every walker of the tree recurses at most 8 deep. Each quote, list item, paragraph, emphasis and link is a level: in seven quotes, a paragraph's text is the eighth level and shows as written, - but markup there shows as its source. + but markup there shows as its source. A link whose own text would be + that deep shows as its source too, so a followed link's own text is + never source; markup nested deeper inside it can be. Source shows its + hidden characters as markers, as code and link text do. - **The parser's own depth, 18 levels of blocks.** A quote is one level and a list two, the list and its item. Past that depth a container's lines are passed over a line at a time, unread, so nested @@ -610,11 +615,11 @@ each card. | A link where none may be (`link`): a card's `front`, `back`, `frontLabel` and `backLabel` and a distractor's `distractorText` (fields `SIDE` and `OPTION`), an autolink `` or `` included | It's in the way of the keys that reveal and grade, and of an option's `