Repository navigation
Don't throw when a back-reference captures a lone surrogate - #136
Conversation
…t compiles
A back-reference in an end or while pattern is replaced by the escaped capture. A
lone surrogate in it made an invalid Oniguruma pattern, so tokenizing the line threw
ArgumentException (Invalid regular expression). It now becomes (?:\p{Cs}|\x{FFFD}):
vscode-oniguruma turns it into U+FFFD in both the pattern and the line.
Benchmarks:
|
| Language | master | PR | Change |
|---|---|---|---|
| shellscript | 132.5 μs | 131.9 μs | -0.5% |
| ruby | 582.0 μs | 576.8 μs | -0.9% |
| php | 1,620.4 μs | 1,600.8 μs | -1.2% |
| perl | 419.8 μs | 414.0 μs | -1.4% |
| markdown | 174.3 μs | 173.5 μs | -0.5% |
| lua | 380.3 μs | 376.1 μs | -1.1% |
| cpp | 2,125.0 μs | 2,056.5 μs | -3.2% |
| csharp | 1,579.0 μs | 1,546.7 μs | -2.0% |
BigFileTokenizationBenchmark (the repository's benchmark), run twice, in opposite orders
| Run | master | PR | Change |
|---|---|---|---|
| 1 (master first) | 4.528 s ± 0.035 | 4.614 s ± 0.021 | +1.9% |
| 2 (PR first) | 4.625 s ± 0.015 | 4.718 s ± 0.025 | +2.0% |
The change adds one char.IsSurrogate test per character of a capture pasted into an end/while pattern, which only runs when a begin match resolves a back-reference, and allocates only when it meets a lone surrogate. The samples that exercise that path show no cost. The big C# file is consistently about 2% slower in both runs although it rarely takes that path, so I read it as code layout rather than the change, but I'm reporting it as measured.
|
@sebastienros as far as I understand this PR can be already merged into main, right? |
|
yes, no strings attached |
Tokenizing a line that contains a lone surrogate (invalid UTF-16) can throw
ArgumentException: Invalid regular expressionwhen a grammar's end or while pattern has a back-reference.Cause.
RegExpSource.ResolveBackReferencesreplaces\1with the escaped text the begin pattern captured. When that text holds a lone surrogate, the resulting pattern contains an invalid UTF-16 code unit and Oniguruma refuses to compile it. Examples found by differential fuzzing over the bundled grammars:afollowed by\uD800: the end pattern becomes(\G(?<!\uD800 [^\w-]))|\}|$.ß\uD83Dq\uDE00: the end pattern becomes\uD83D.The grammars are fine. vscode-textmate pastes the capture the same way, but vscode-oniguruma converts both the pattern and the line to UTF-8, which turns each lone surrogate into U+FFFD on both sides, so VS Code never sees an invalid pattern.
Fix. When escaping a capture, a lone surrogate becomes
(?:\p{Cs}|\x{FFFD}), which compiles and matches a lone surrogate in the line or U+FFFD. Surrogate pairs are copied unchanged. A pattern cannot contain a surrogate code point itself: Oniguruma rejects both the raw code unit and\x{D800}.One limit: Oniguruma reads a lone high surrogate followed by another character as one ill-formed unit, so no pattern matches it there. In that case the end pattern does not match at that point, instead of the line throwing.
Tests.
RegExpSourceTestscovers escaping, surrogate pairs, and lone surrogates (the last fails without the fix). All tests pass.