[Akiyo974]

Fixes #148
Fixes #166

Highlight markers ({_ _}, {* *}, {~ ~}, {:class: :}) could duplicate text, corrupt the output (broken <span> nesting, invalid UTF-8), or throw a fatal error with withGutter().

Causes and fixes

There were three separate issues:

1. Overlapping children were never grouped (Fatal error when using emphasize tags around a string #148). A marker token became the direct parent of every token it covered, including tokens nested inside another child, for example a string and the SQL keywords inside it. Those children overlapped, so the text was rendered twice. This could also split the multibyte escape tokens (❷, ❸…), which is where the original "Malformed UTF-8" fatal error came from.
   → GroupTokens now groups a token's children the same way when they overlap. A token that runs past the end of its parent is dropped, like any other token that can't be contained.

2. Offsets went stale after an injection rewrote the content (Custom classes tag breaks/errors when code contains function with a parameter (php) #166). Injections like PHP function parameters, CSS in a style attribute or Blade insert nested markup after the marker tokens have been created, so those tokens pointed to the wrong place (($te$test)).
   → IsInjection now reports what it replaced through ParsedInjection::$replacements (a new optional argument, so it stays backwards compatible). Highlighter uses it to move the tokens found earlier.

3. Markers affected pattern matching. {_function didn't match the function keyword, because _ is a word character, and markers inside strings or comments were ignored.
   → Markers are replaced by spaces of the same length while patterns run, so offsets don't change. Pattern tokens never start or end on a marker, and any built-in token can contain a marker.

As a result, wrapping code in {_ … _} now renders exactly like the same code without markers, wrapped in <span class="hl-em">.

Tests

• Added HighlightMarkersTest, covering both issues plus strong, blur, custom class, nested markers, markers inside strings, PHP signatures, HTML with CSS, Blade, and withGutter(). It also checks that the "wrapped equals plain" property holds across languages.
• Added GroupTokens unit tests for children grouping, overflowing tokens and markers inside tokens.
• Updated one expectation in PhpLanguageTest: in {~class~} Foo {}, class is now highlighted as a keyword. The old expectation reflected issue 3, and it contradicted GroupTokensTest::test_exact_overlaps, which expects a blur to contain a keyword.
• All bench fixtures render byte-for-byte the same as on main.
• I also inserted random marker pairs into the bench fixtures (~1,350 runs). On main, 521 runs produced broken output, including 16 exceptions. With this PR there are no exceptions, unbalanced spans or invalid UTF-8.

Known limitation

Markers inside embedded languages are moved onto the rewritten code as long as the resulting markup stays properly nested. In the rare case where a marker would cut through a token highlighted by the embedded language (e.g. a custom class that starts in the middle of a CSS property and ends in its value), that marker is dropped rather than producing broken HTML.

Performance

No measurable difference with main in interleaved local runs (−0.4%, within noise). The extra work only happens when markers or custom classes are present.

Note: this touches Highlighter, GroupTokens and IsInjection, which #236 also changes, so I'm happy to rebase whichever lands first.

[github-actions]

Benchmark Results

Comparison of fix/emphasize-overlapping-tokens against main (3d96482ee77208c8e445d4f68667918c7f440f62).

Open to see the benchmark results

benchmark	subject	set	revs	its	mem_peak	mode	rstdev
HighlighterBench	benchParse	apache	100	5	2.105mb +2.33%	73.804247μs +0.09%	±0.28% -61.36%
HighlighterBench	benchParse	bash	100	5	2.136mb +2.32%	204.112720μs +0.47%	±0.66% -28.95%
HighlighterBench	benchParse	bbcode	100	5	2.092mb +2.34%	74.016027μs -1.25%	±0.64% -41.90%
HighlighterBench	benchParse	blade	100	5	2.775mb +1.88%	1,166.487945μs +5.81%	±0.85% +65.65%
HighlighterBench	benchParse	css	100	5	2.123mb +2.33%	141.734579μs +0.29%	±0.58% +16.55%
HighlighterBench	benchParse	diff	100	5	2.127mb +2.21%	102.552466μs +2.01%	±0.47% -69.45%
HighlighterBench	benchParse	dockerfile	100	5	2.097mb +2.33%	89.620294μs +0.25%	±1.29% +0.99%
HighlighterBench	benchParse	dotenv	100	5	2.108mb +2.32%	77.826751μs -0.16%	±1.77% +128.67%
HighlighterBench	benchParse	gdscript	100	5	2.203mb +2.25%	334.517789μs -0.64%	±0.30% -20.96%
HighlighterBench	benchParse	graphql	100	5	2.123mb +2.33%	154.192720μs +0.04%	±1.34% +302.36%
HighlighterBench	benchParse	html	100	5	2.127mb +2.43%	118.464149μs +0.95%	±0.36% -64.39%
HighlighterBench	benchParse	http	100	5	2.096mb +2.34%	24.658669μs +0.69%	±1.10% +76.78%
HighlighterBench	benchParse	ini	100	5	2.112mb +2.32%	87.277358μs -0.09%	±0.73% +40.26%
HighlighterBench	benchParse	javascript	100	5	2.191mb +2.38%	274.234325μs -0.46%	±0.57% -33.71%
HighlighterBench	benchParse	json	100	5	2.166mb +2.29%	239.942133μs -0.10%	±0.70% -54.55%
HighlighterBench	benchParse	markdown	100	5	2.114mb +2.32%	76.337886μs +0.11%	±1.21% +67.48%
HighlighterBench	benchParse	nginx	100	5	2.178mb +2.27%	269.732661μs +0.38%	±1.05% +6.02%
HighlighterBench	benchParse	php	100	5	2.469mb +2.18%	788.251703μs +3.37%	±0.64% +60.25%
HighlighterBench	benchParse	python	100	5	2.367mb +2.09%	427.289628μs +1.06%	±0.50% -51.87%
HighlighterBench	benchParse	ruby	100	5	2.413mb +2.03%	238.357299μs -0.27%	±0.36% -55.55%
HighlighterBench	benchParse	scss	100	5	2.139mb +2.32%	184.497750μs +0.23%	±1.15% +74.79%
HighlighterBench	benchParse	sema	100	5	2.307mb +2.15%	233.978082μs +2.51%	±2.32% +1134.22%
HighlighterBench	benchParse	sql	100	5	2.138mb +2.29%	159.578141μs +0.04%	±1.13% +70.19%
HighlighterBench	benchParse	svelte	100	5	2.412mb +2.17%	1,131.470000μs +2.92%	±0.33% -67.59%
HighlighterBench	benchParse	swift	100	5	2.185mb +2.27%	255.171272μs -1.38%	±2.56% +103.33%
HighlighterBench	benchParse	terminal	100	5	2.133mb +2.32%	190.855890μs +0.32%	±2.49% +255.17%
HighlighterBench	benchParse	terraform	100	5	2.138mb +2.32%	176.517886μs +1.67%	±0.99% +134.87%
HighlighterBench	benchParse	typescript	100	5	2.563mb +2.04%	254.321800μs +1.04%	±0.75% -12.79%
HighlighterBench	benchParse	txt	100	5	2.067mb +2.36%	8.874795μs +2.23%	±1.97% +105.28%
HighlighterBench	benchParse	twig	100	5	19.245mb +0.30%	1,549.779511μs +2.22%	±1.30% +2.06%
HighlighterBench	benchParse	vue	100	5	2.420mb +2.16%	432.562955μs +2.37%	±1.38% +25.91%
HighlighterBench	benchParse	xml	100	5	2.141mb +2.31%	179.465714μs +0.20%	±1.16% +325.94%
HighlighterBench	benchParse	yaml	100	5	2.199mb +2.25%	322.195010μs -0.54%	±0.78% -17.87%
HighlighterBench	benchParse	jsx	100	5	2.135mb +2.42%	53.762016μs +1.73%	±0.59% -76.09%
HighlighterBench	benchParse	tsx	100	5	2.219mb +2.35%	80.798356μs +0.88%	±0.88% -6.13%

_{Generated by phpbench against commit 9c1e283}

[Copilot]

🟡 Changes recommended

Marker loss during injection processing and false keyword matches from whitespace masking remain unresolved.

3 open issues

• Avoid whitespace-induced keyword matches inside marked identifiers · New
• Preserve custom classes through nested language rewriting · New
• Preserve markers in unchanged context during nested parsing · New

What changed in this PR

This PR addresses #148 and #166 by improving highlight-marker handling around nested tokens and embedded languages.

Changes:

• Groups overlapping child tokens and allows markers inside built-in tokens.
• Tracks injection replacements to adjust existing token offsets.
• Masks markers during pattern matching and adds regression coverage.

File	Description
tests/​Tokens/​GroupTokensTest.php	Tests child grouping, overflow, and embedded markers.
tests/​Languages/​Php/​PhpLanguageTest.php	Updates keyword highlighting expectations.
tests/​Languages/​Base/​HighlightMarkersTest.php	Adds cross-language marker and gutter tests.
src/​Tokens/​Token.php	Allows built-in tokens to contain markers.
src/​Tokens/​GroupTokens.php	Groups overlapping children and drops overflowing tokens.
src/​ParsedInjection.php	Adds replacement-offset metadata.
src/​IsInjection.php	Records injection replacements.
src/​Highlighter.php	Adjusts token offsets and masks marker delimiters.

_{Review effort: Balanced}