fix(markdown): show hidden characters in deep source as markers

docs/markdown.md

@@ -77,11 +77,11 @@ package's own tree, folded to what the app shows:
 | In the text | In the tree |
 |---|---|
 | a heading | a paragraph in bold: data never adds to the page's outline |
-| raw HTML, block or inline | its source, as text |
+| raw HTML, block or inline | its source, as text, its hidden characters as markers |
 | a picture, inline or by reference | its alt text: never loaded or linked |
 | a reference link | a link to its definition's URL (the first definition of a label wins); definitions show nothing |
 | a code block's info string | its first word, lower-cased, when it matches `[a-z0-9+#-]{1,20}`; else none |
-| hidden characters in code or link text: every format character (Unicode category Cf, such as the bidi controls U+202A–202E and U+2066–2069, the zero-width characters U+200B–200F, U+2060 and U+FEFF, the soft hyphen U+00AD, the invisible operators U+2061–2064 and the tag characters U+E0000–E007F), and characters that show nothing or look blank yet count (the Hangul fillers U+115F, U+1160, U+3164 and U+FFA0, the combining grapheme joiner U+034F, Khmer's U+17B4–17B5, and the variation selectors U+180B–180F, U+FE00–FE0F and U+E0100–E01EF), a picture's alt text in a link included | a visible marker such as `⟨U+202E⟩` ("Trojan Source": code that reads other than it runs) |
+| hidden characters in code or link text: every format character (Unicode category Cf, such as the bidi controls U+202A–202E and U+2066–2069, the zero-width characters U+200B–200F, U+2060 and U+FEFF, the soft hyphen U+00AD, the invisible operators U+2061–2064 and the tag characters U+E0000–E007F), and characters that show nothing or look blank yet count (the Hangul fillers U+115F, U+1160, U+3164 and U+FFA0, the combining grapheme joiner U+034F, Khmer's U+17B4–17B5, and the variation selectors U+180B–180F, U+FE00–FE0F and U+E0100–E01EF), a picture's alt text in a link included, and in anything shown as its source, however deep it nests | a visible marker such as `⟨U+202E⟩` ("Trojan Source": code that reads other than it runs) |
 | a table body row's cells past the header's | dropped, as GFM has it; a short row isn't padded |
 
 | Function | Module | What it gives |
@@ -92,7 +92,7 @@ package's own tree, folded to what the app shows:
 | `plainText(text)` | `plainText` | the text as plain text |
 | `labelText(text, max?)` | `plainText` | a short name: the plain text of its first block that shows any |
 | `liveLink(url)` | `links` | whether a link is followed, and its host |
-| `inspectMarkdown(text)` | `parse` | whether the text is one paragraph as written (a heading isn't), and notes of what the tree folds away or a check looks at: raw HTML, pictures, links (an autolink told apart), code outside a link, character references, what nests too deep to show as written, a table past its caps, a heading underlined with dashes |
+| `inspectMarkdown(text)` | `parse` | whether the text is one paragraph as written (a heading isn't), and notes of what the tree folds away or a check looks at: raw HTML, pictures, links (an autolink told apart), code outside a link, character references, what nests too deep to show as written (and code or a link there with hidden characters), a table past its caps, a heading underlined with dashes |
 | `markdownProblems(text, rule)` | `problems` | what wouldn't show as its author meant, by the [rules for a release](#rules-for-a-release), in a field held to `SIDE`, `OPTION` or `PROSE` |
 | `chunksOf(blocks)` | `chunks` | the [chunks](#chunks) blocks are shown in, split at their top-level rules |
 | `splitAtRules(blocks)` | `chunks` | the pieces between top-level rules, empty ones included |
@@ -107,7 +107,8 @@ shapes of Markdown make the parser slow.
 `MAX_WORK`.
 
 - **`MAX_CHARS`, 20,000 characters.** A longer text is plain, its
-  source still readable. The longest course text is about 2,600.
+  source still readable, its hidden characters as markers. The longest
+  course text is about 2,600.
 - **`MAX_WORK`, 40 units of the parser's work for each character** (of
   at least 1,000). Each try of one of the parser's rules, each step
   through a link's label and each token it makes is a unit, counted as
@@ -130,7 +131,8 @@ shapes of Markdown make the parser slow.
   its own. An `href` takes at most three characters for each byte, a
   percent escape, so the links a text shows hold at most a small
   multiple of its length. Past the budget the parse stops, and the text
-  is plain, so no text costs more than time linear in its length.
+  is plain, as past `MAX_CHARS`, so no text costs more than time linear
+  in its length.
   markdown-it reads most text in such time, but some shapes make it
   read the rest of a text again and again: a block quote after each
   line of a paragraph, or link openers before an unclosed declaration
@@ -141,7 +143,10 @@ shapes of Markdown make the parser slow.
   source, so every walker of the tree recurses at most 8 deep. Each
   quote, list item, paragraph, emphasis and link is a level: in seven
   quotes, a paragraph's text is the eighth level and shows as written,
-  but markup there shows as its source.
+  but markup there shows as its source. A link whose own text would be
+  that deep shows as its source too, so a followed link's own text is
+  never source; markup nested deeper inside it can be. Source shows its
+  hidden characters as markers, as code and link text do.
 - **The parser's own depth, 18 levels of blocks.** A quote is one
   level and a list two, the list and its item. Past that depth a
   container's lines are passed over a line at a time, unread, so nested
@@ -610,11 +615,11 @@ each card.
 | A link where none may be (`link`): a card's `front`, `back`, `frontLabel` and `backLabel` and a distractor's `distractorText` (fields `SIDE` and `OPTION`), an autolink `<ex:title>` or `<http://…>` included | It's in the way of the keys that reveal and grade, and of an option's `<label>`; an autolink loses its angle brackets too. The fix is code. |
 | A link the app doesn't follow (`linkNotFollowed`), in a note, a step's theory or a chapter's description (field `PROSE`) | Only `https:` without a user name or password is followed; any other shows as text. |
 | Link text that reads as an address or a host name other than the link's (`linkHost`) | Misleading text: the named host, as written, must be the link's, port included, give or take a `www.`. A percent escape or a full-width dot in it fails. `[solidproject.org](https://www.solidproject.org/)` passes; `[bank.example](https://evil.example/)` and `[github.io](https://evil.github.io/)` don't. The test is by shape, a dotted word ending in letters: "Node.js" and a file's name such as `package.json`, `v1.ttl` or `README.md` count, even as code. Other wording avoids it ("npm's documentation of package.json"). It checks data only; the defense is the host shown beside a link ([safety](#safety)). |
-| A bidi control, zero-width or other hidden character in code or a link (`hiddenControl`) | Code that reads other than it runs ("Trojan Source"); the app shows them as markers. |
+| A bidi control, zero-width or other hidden character in code or a link (`hiddenControl`), nested past `MAX_DEPTH` too | Code that reads other than it runs ("Trojan Source"); the app shows them as markers. |
 | A character reference outside code (`characterReference`), such as `&aring;` | CommonMark decodes it: it shows as "å". As code, or with its `&` escaped (`\&aring;`), it shows as typed. An unknown name, `&nosuchname;`, is text and passes. |
 | A heading underlined with dashes (`dashHeading`): a line of text with `---` right under it, no blank line between | It looks like text over a thematic break, which a blank line before the `---` makes it (ending a [chunk](#chunks) in a step's theory). That blank line, or a `##` heading, avoids it; `===` underlining passes. |
 | An option that isn't one paragraph (`notOneParagraph`): a distractor's `distractorText`, and the `back` of a card with distractors (field `OPTION`) | Options show on one line, in the inline profile, and must look alike, or the odd one out gives the answer away. A heading is no paragraph: it shows bold on a revealed back. |
-| Past the [limits](#limits): longer than `MAX_CHARS` (`tooLong`), past `MAX_WORK` (`tooComplex`), nested past `MAX_DEPTH` (`tooDeep`, unless it's plain text that shows as written anyway), a table past its caps (`largeTable`) | The text, or that part of it, shows as plain text or as its source. |
+| Past the [limits](#limits): longer than `MAX_CHARS` (`tooLong`), past `MAX_WORK` (`tooComplex`), nested past `MAX_DEPTH` (`tooDeep`, unless it's plain text that shows as written anyway, with no hidden characters), a table past its caps (`largeTable`) | The text, or that part of it, shows as plain text or as its source, its hidden characters as markers. |
 
 A step's theory is held to its chunks too: none of them empty, and as
 many in each language, so a learner who switches language keeps their

packages/markdown/src/parse.test.ts

@@ -429,10 +429,65 @@ describe("parseMarkdown", () => {
     ]);
   });
 
-  it("shows an autolink's text nested past MAX_DEPTH as its URL, without its angle brackets", () => {
-    expect(
-      JSON.stringify(parseMarkdown(`${"> ".repeat(MAX_DEPTH - 2)}<https://x.example>`)),
-    ).toContain('"children":[{"type":"text","value":"https://x.example"}]');
+  /** The blocks of a text nested in `quotes` block quotes, unwrapped. */
+  const inQuotes = (quotes: number, markdown: string): MdBlock[] => {
+    let blocks = parseMarkdown(`${"> ".repeat(quotes)}${markdown}`)!;
+    for (let level = 0; level < quotes; level++)
+      blocks = (blocks[0] as Extract<MdBlock, { type: "quote" }>).children;
+    return blocks;
+  };
+
+  it("shows a link whose text would nest past MAX_DEPTH as its source, not a link", () => {
+    expect(inQuotes(MAX_DEPTH - 2, "<https://x.example>")).toEqual([
+      paragraph(text("<https://x.example>")),
+    ]);
+    expect(inQuotes(MAX_DEPTH - 2, "[pay\u202elapyap.com\u200b](https://x.example/)")).toEqual([
+      paragraph(text("[pay⟨U+202E⟩lapyap.com⟨U+200B⟩](https://x.example/)")),
+    ]);
+    // A level up, it is a link, its text with markers as ever.
+    expect(inQuotes(MAX_DEPTH - 3, "[pay\u202elapyap.com](https://x.example/)")).toEqual([
+      paragraph({
+        type: "link",
+        url: "https://x.example/",
+        children: [text("pay⟨U+202E⟩lapyap.com")],
+      }),
+    ]);
+  });
+
+  it("shows the hidden characters of what shows as its source as markers", () => {
+    // Code nested past MAX_DEPTH, a span and a block.
+    expect(inQuotes(MAX_DEPTH - 1, "`if (a\u202e) b\u200b`")).toEqual([
+      paragraph(text("`if (a⟨U+202E⟩) b⟨U+200B⟩`")),
+    ]);
+    expect(inQuotes(MAX_DEPTH - 2, "`if (a\u202e) b`")).toEqual([
+      paragraph({ type: "inlineCode", value: "if (a⟨U+202E⟩) b" }),
+    ]);
+    expect(JSON.stringify(parseMarkdown(`${"> ".repeat(MAX_DEPTH)}    a\u202e`))).toContain(
+      '"value":"    a⟨U+202E⟩"',
+    );
+    // Text in a followed link's text, past MAX_DEPTH.
+    expect(inQuotes(MAX_DEPTH - 3, "[*a\u202e*](https://x.example/)")).toEqual([
+      paragraph({
+        type: "link",
+        url: "https://x.example/",
+        children: [{ type: "emphasis", children: [text("a⟨U+202E⟩")] }],
+      }),
+    ]);
+    // Raw HTML, in a link's text or not, and a table past its caps.
+    expect(parseMarkdown('[a<i title="\u202e">](https://x.example/) <b title="\u202e">')).toEqual([
+      paragraph(
+        {
+          type: "link",
+          url: "https://x.example/",
+          children: [text("a"), text('<i title="⟨U+202E⟩">')],
+        },
+        text(" "),
+        text('<b title="⟨U+202E⟩">'),
+      ),
+    ]);
+    expect(parseMarkdown("<div>\u202e</div>")).toEqual([paragraph(text("<div>⟨U+202E⟩</div>"))]);
+    const wide = `|${" a\u202e |".repeat(MAX_TABLE_COLUMNS + 1)}\n|${"-|".repeat(MAX_TABLE_COLUMNS + 1)}`;
+    expect(JSON.stringify(parseMarkdown(wide))).not.toContain("\u202e");
   });
 });
 
@@ -688,6 +743,48 @@ describe("inspectMarkdown", () => {
       { type: "tooDeep", source: "&amp;" },
     ]);
   });
+  it("notes the hidden characters of code or a link shown as its source, outside a link", () => {
+    const deep = (markdown: string) => notesOf(`${"> ".repeat(MAX_DEPTH - 1)}${markdown}`);
+    expect(deep("`a\u202e`")).toEqual([
+      { type: "tooDeep", source: "`a\u202e`" },
+      { type: "deepControls", in: "code", source: "`a\u202e`" },
+    ]);
+    const link = "[pay\u202elapyap.com](https://x.example/)";
+    expect(notesOf(`${"> ".repeat(MAX_DEPTH - 2)}${link}`)).toEqual([
+      { type: "tooDeep", source: link },
+      { type: "deepControls", in: "link", source: link },
+    ]);
+    expect(notesOf(`${"> ".repeat(MAX_DEPTH - 2)}[a](https://x.example/)`)).toEqual([
+      { type: "tooDeep", source: "[a](https://x.example/)" },
+    ]);
+    // Text with a hidden character no longer shows as written; raw HTML is neither code nor a link.
+    expect(deep("a\u202e")).toEqual([{ type: "tooDeep", source: "a\u202e" }]);
+    expect(deep('a <b title="\u202e">')).toEqual([
+      { type: "tooDeep", source: '<b title="\u202e">' },
+    ]);
+    // Code in a list or a table nested past MAX_DEPTH.
+    const quotes = "> ".repeat(MAX_DEPTH);
+    expect(notesOf(`${quotes}- \`a\u202e\``)).toEqual([
+      { type: "tooDeep", source: "- `a\u202e`" },
+      { type: "deepControls", in: "code", source: "- `a\u202e`" },
+    ]);
+    const table = `| \`a\u202e\` |\n${quotes}|-|`;
+    expect(notesOf(`${quotes}${table}`)).toEqual([
+      { type: "tooDeep", source: table },
+      { type: "deepControls", in: "code", source: table },
+    ]);
+    // In a followed link's text, the link's note holds them.
+    expect(notesOf(`${"> ".repeat(MAX_DEPTH - 3)}[*a\u202e*](https://x.example/)`)).toEqual([
+      {
+        type: "link",
+        url: "https://x.example/",
+        text: "a\u202e",
+        source: "[*a\u202e*](https://x.example/)",
+        autolink: false,
+      },
+      { type: "tooDeep", source: "a\u202e" },
+    ]);
+  });
 });
 
 describe("hiddenControls", () => {

packages/markdown/src/parse.ts

@@ -205,6 +205,8 @@ export type MdNote =
   | { type: "code"; value: string }
   | { type: "characterReference"; source: string }
   | { type: "tooDeep"; source: string }
+  /** Code or a link shown as its source (tooDeep), outside a link, whose source holds hidden characters: shown as markers. */
+  | { type: "deepControls"; in: "code" | "link"; source: string }
   | { type: "largeTable"; source: string }
   /** A heading underlined with dashes (`Text` over `---`), which reads like a line of text over a thematic break. */
   | { type: "dashHeading"; source: string };
@@ -1036,7 +1038,7 @@ class Fold {
 
   /** One block, or none for a definition, folded to what is shown. */
   private block(node: Node, depth: number): DocBlock[] {
-    if (depth >= MAX_DEPTH) {
+    if (pastDepth(node, depth)) {
       if (this.outline) this.refuse(node, `nested deeper than MAX_DEPTH (${MAX_DEPTH})`);
       return [this.sourceParagraph(node)];
     }
@@ -1116,9 +1118,9 @@ class Fold {
   }
 
   private inline(node: Node, depth: number, inLink: boolean): MdInline | MdInline[] {
-    if (depth >= MAX_DEPTH) {
+    if (pastDepth(node, depth)) {
       if (this.outline) this.refuse(node, `nested deeper than MAX_DEPTH (${MAX_DEPTH})`);
-      return { type: "text", value: this.source(node) };
+      return { type: "text", value: this.shown(node) };
     }
     switch (node.type) {
       case "text":
@@ -1148,7 +1150,7 @@ class Fold {
       // Raw HTML ("html"), and anything else the parser could make, is shown as its source.
       default:
         if (this.outline) this.refuse(node, "raw HTML");
-        return { type: "text", value: this.source(node) };
+        return { type: "text", value: this.shown(node) };
     }
   }
 
@@ -1174,14 +1176,18 @@ class Fold {
     return line + 1;
   }
 
-  /** The Markdown a node was parsed from, as written. */
-  private source(node: Node): string {
-    return this.text.slice(node.start, node.end);
+  /**
+   * The Markdown a node was parsed from, as it is shown: as written, but
+   * for its hidden characters, which show as markers wherever source
+   * shows, as code or a link's text might be in it.
+   */
+  private shown(node: Node): string {
+    return visibleControls(this.text.slice(node.start, node.end));
   }
 
   /** A block shown as its source, its lines kept apart. */
   private sourceParagraph(node: Node): DocBlock {
-    const lines = this.source(node).split("\n");
+    const lines = this.shown(node).split("\n");
     return {
       type: "paragraph",
       children: lines.flatMap((line, index): MdInline[] =>
@@ -1193,6 +1199,34 @@ class Fold {
   }
 }
 
+/**
+ * Whether a node at `depth` shows as its source: one nested past
+ * MAX_DEPTH, or a link whose own text would be, so a followed link's own
+ * text is never its source (markup nested deeper inside it can be).
+ */
+function pastDepth(node: Node, depth: number): boolean {
+  return depth >= MAX_DEPTH || (node.type === "link" && depth + 1 >= MAX_DEPTH);
+}
+
+/**
+ * What a node shown as its source is or holds that a check looks at
+ * for hidden characters: code, else a link, else nothing; walked without
+ * recursion, however deep it nests.
+ */
+function held(node: Node): "code" | "link" | null {
+  let found: "link" | null = null;
+  const stack: Node[] = [node];
+  while (stack.length > 0) {
+    const next = stack.pop()!;
+    if (next.type === "code" || next.type === "inlineCode") return "code";
+    if (next.type === "link") found = "link";
+    if (next.type === "list") stack.push(...next.items.flat());
+    else if (next.type === "table") stack.push(...next.rows.flat(2));
+    else if ("children" in next) stack.push(...next.children);
+  }
+  return found;
+}
+
 /** The text of phrasing as plain text: its text, code and raw HTML as written, a picture's alt text, a hard break a space; walked without recursion, however deep it nests. */
 function phrasingText(nodes: readonly Node[]): string {
   let text = "";
@@ -1212,8 +1246,10 @@ function phrasingText(nodes: readonly Node[]): string {
 /**
  * The walk inspectMarkdown makes, nesting as Fold does, so what Fold
  * shows as its source is noted once, as too deep: unless it would show
- * as it does anyway, plain text with nothing escaped or referenced.
- * Code in a link's text is not noted apart, being in the link's.
+ * as it does anyway, plain text with nothing escaped or referenced and
+ * no hidden character. Code or a link shown so is noted again when its
+ * hidden characters show as markers. Code in a link's text is not noted
+ * apart, being in the link's.
  */
 class Inspect {
   private readonly text: string;
@@ -1229,8 +1265,12 @@ class Inspect {
   }
 
   private visit(node: Node, depth: number, inLink: boolean): void {
-    if (depth >= MAX_DEPTH) {
-      if (!this.asWritten(node)) this.notes.push({ type: "tooDeep", source: this.source(node) });
+    if (pastDepth(node, depth)) {
+      const source = this.source(node);
+      if (!this.asWritten(node)) this.notes.push({ type: "tooDeep", source });
+      // In a link's text, they are in the link's note.
+      const what = inLink || hiddenControls(source).length === 0 ? null : held(node);
+      if (what !== null) this.notes.push({ type: "deepControls", in: what, source });
       return;
     }
     switch (node.type) {
@@ -1282,9 +1322,10 @@ class Inspect {
    * Whether a node Fold shows as its source shows as it would have:
    * text whose source is its value, its line endings as the parser reads
    * them, or a paragraph of one line of it (Fold keeps a source's lines
-   * apart).
+   * apart); with no hidden characters, which Fold shows as markers.
    */
   private asWritten(node: Node): boolean {
+    if (hiddenControls(this.source(node)).length > 0) return false;
     if (node.type === "text") return this.source(node).replace(/\r\n?/g, "\n") === node.value;
     return (
       node.type === "paragraph" &&