diff --git a/README.md b/README.md index 3ffb2ffe..62f6f65c 100644 --- a/README.md +++ b/README.md @@ -366,7 +366,7 @@ If planned renames clash, the process is halted and no change is applied to any | `-g` | **`--glob-pipe`** Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments. [boolean] | | | **`--null-in`** The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out. [boolean] | | `-R` | **`--replacement-pipe`** Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean] | -| `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. If the javascript source code references to the full match or a captured group the code will run once per match. Otherwise, it will run once per file. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. The numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). A named group is a variable with `_` in front: `_year` for `(?...)`, and `_year_` has a space at the end. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] | +| `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. If the javascript source code references to the full match or a captured group the code will run once per match. Otherwise, it will run once per file. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. The numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). A named group is a variable with `rr_` in front: `rr_year` for `(?...)`, and `rr_year_` has a space at the end. A named group can not end with `_`. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] | | `-x` | **`--exclude-re`** Files with a path that matches this regular expression will be excluded from processing. Can be provided multiple times. [string] | | `-X` | **`--exclude-glob`** Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate. [string] | | `-h` | **`--help`** Display help. [boolean] | diff --git a/src/cli.ts b/src/cli.ts index 406a57a7..8b1b73d2 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -410,7 +410,7 @@ The following values defaults to \`❌\` if haystack does not originate from a f The numbers: \`$n\`: the number of the file, \`$t\`: the number of files, \`$i\`: the number of the file among the files with a match so far, \`$c\`: among the files changed so far, \`$m\`: the number of the match in the file, \`$M\`: across all files. Each has cousins with zeros in front (\`$n0\` has 2 digits, \`$n00\` 3, and so on to 10) and with a space at the end (\`$n_\`). For speed, the code runs once per file unless it names the match, a group, \`$m\` or \`$M\`. -A named group is a variable with \`_\` in front: \`_year\` for \`(?...)\`, and \`_year_\` has a space at the end. +Access a named group via the name prepended with \`rr_\`. \`(?...)\` gives you \`rr_year\`. A named group name can not end with \`_\`. All variables, except from module, date objects, \`nl\` and \`_\`, has a corresponding variable name followed by \`_\` where the content has an extra space at the end (for easy concatenation). ` diff --git a/src/engine.ts b/src/engine.ts index edbb03d5..182dca19 100644 --- a/src/engine.ts +++ b/src/engine.ts @@ -144,6 +144,17 @@ async function run(config) { config.regex = getFinalRegex(config.pattern, config) || ''; + // In --js a named group is a variable like rr_year, and rr_year_ has a space at the end. + // A group that ends with _ would have the same name as another group with a space, so it can not be used. + // With --output-match the code is not run, so there are no variables. + const endsWith_ = + config.replacementJs && !config.outputMatch && config.regex + ? namedGroups(config.regex).find(name => name.endsWith('_')) + : null; + if (endsWith_) { + panic('A named group can not end with _:', endsWith_); + } + // What the regex engine is asked to do. With --replacement-js and --output-match it gets a function. if (config.statusInfo.engine) { config.statusInfo.engine.execution = { @@ -821,15 +832,20 @@ function countFile(config, text, result) { } } -// The code uses a named group of the pattern, like _year for (?…) +// The code uses a named group of the pattern, like rr_year for (?…) function usesNamedGroups(config) { - return [...config.regex.source.matchAll(/\(\?<([A-Za-z_$][\w$]*)>/g)].some(([, name]) => - new RegExp('(^|[^\\w$])_' + name.replace(/\$/g, '\\$') + '(?![\\w$])').test( + return namedGroups(config.regex).some(name => + new RegExp('(^|[^\\w$])rr_' + name.replace(/\$/g, '\\$') + '(?![\\w$])').test( config.replacementOri ) ); } +// The names of the named groups of a regex, like year for (?…) +function namedGroups(regex) { + return [...regex.source.matchAll(/\(\?<([A-Za-z_$][\w$]*)>/g)].map(([, name]) => name); +} + // The numbers of --js and their cousins: $n0 is $n with zeros in front to 2 digits, $n00 to 3 and so on to 10, // and the ones with _ at the end have a space at the end. They are all given to the code as values, so they cost no time to compile. const widths = [2, 3, 4, 5, 6, 7, 8, 9, 10]; @@ -1192,10 +1208,10 @@ function dynamicReplacement( for (var i = 0; i < arguments.length - (hasNamed ? 3 : 2); i++) { capturedGroups += 'var $' + i + '=' + JSON.stringify(arguments[i]) + '; '; } - // A named group is a variable with _ in front, like _year for (?…), and _year_ has a space at the end + // A named group is a variable with rr_ in front, like rr_year for (?…), and rr_year_ has a space at the end if (hasNamed) { for (const [name, value] of Object.entries(named)) { - capturedGroups += `var _${name} = ${JSON.stringify(value)}, _${name}_ = _${name} + ' '; `; + capturedGroups += `var rr_${name} = ${JSON.stringify(value)}, rr_${name}_ = rr_${name} + ' '; `; } } diff --git a/test/fix/799.test.js b/test/fix/799.test.js index f4f4f568..d7c4ddf8 100644 --- a/test/fix/799.test.js +++ b/test/fix/799.test.js @@ -9,16 +9,16 @@ describe('#799 groups in --js', () => { expect((await rr('(?a)(b)', '$1 + $2 + typeof $3', config)).out).toBe('abundefined'); }); - test('a named group is a variable with _ in front, and _ at the end has a space', async () => { + test('a named group is a variable with rr_ in front, and _ at the end has a space', async () => { const config = {pipedData: '2024-05 1999-12', replacementJs: true}; const pattern = '(?\\d{4})-(?\\d\\d)'; - expect((await rr(pattern, '_month + "/" + _year', config)).out).toBe('05/2024 12/1999'); - expect((await rr(pattern, '_month_ + _year', config)).out).toBe('05 2024 12 1999'); + expect((await rr(pattern, 'rr_month + "/" + rr_year', config)).out).toBe('05/2024 12/1999'); + expect((await rr(pattern, 'rr_month_ + rr_year', config)).out).toBe('05 2024 12 1999'); }); test('a named group that took no part in the match is undefined', async () => { const config = {pipedData: 'ab', replacementJs: true}; - expect((await rr('(?a)|(?b)', 'String(_x) + String(_y) + ","', config)).out).toBe( + expect((await rr('(?a)|(?b)', 'String(rr_x) + String(rr_y) + ","', config)).out).toBe( 'aundefined,undefinedb,' ); }); diff --git a/test/fix/815.test.js b/test/fix/815.test.js index 3c882816..9a9028e0 100644 --- a/test/fix/815.test.js +++ b/test/fix/815.test.js @@ -10,7 +10,7 @@ describe('#815 --js code can declare its own variables', () => { ['a function', 'o', 'function $t() { return "F"; } $t()', 'fFF'], ['a number with let', 'o', 'let $t = "T"; $t', 'fTT'], ['a group with const', '(o)', 'const $1 = "X"; $1', 'fXX'], - ['a named group with let', '(?o)', 'let _o = "N"; _o', 'fNN'], + ['a named group with let', '(?o)', 'let rr_o = "N"; rr_o', 'fNN'], ])('with the name of %s', async (name, pattern, code, out) => { expect((await rr(pattern, code, {pipedData: 'foo', replacementJs: true})).out).toBe(out); }); diff --git a/test/fix/825.test.js b/test/fix/825.test.js new file mode 100644 index 00000000..067420b2 --- /dev/null +++ b/test/fix/825.test.js @@ -0,0 +1,66 @@ +// https://github.com/mathiasrw/rexreplace/issues/825 +// A named group was a variable like _year, which could clash: with groups a and a_, _a_ was both a_ and a with a space, +// and a group named _require_ broke r(). +import {describe, test, expect} from 'bun:test'; +import fs from 'fs'; +import {rr} from '../helper.js'; + +const js = {pipedData: 'in 1999 and 2024', replacementJs: true}; + +describe('#825 named groups in --js', () => { + test.each([ + ['a name in lowercase', '(?\\d+)', '"<" + rr_year + ">"', 'in <1999> and <2024>'], + [ + 'the cousin with a space at the end', + '(?\\d+)', + 'rr_year + "," + rr_year_', + 'in 1999,1999 and 2024,2024 ', + ], + ['a name in uppercase', '(?\\d+)', '"Y" + rr_Year', 'in Y1999 and Y2024'], + ['a name with a number', '(?\\d+)', '"Y" + rr_y2k', 'in Y1999 and Y2024'], + ['a name with $', '(?\\d+)', '"Y" + rr_a$', 'in Y1999 and Y2024'], + ['two names that only differ by case', '(?\\d)(?\\d)', 'rr_Y + rr_y', 'in 9199 and 0242'], + ['a name next to its number', '(?\\d+)', '$1 + rr_year', 'in 19991999 and 20242024'], + ])('gives %s', async (name, pattern, code, out) => { + expect(await rr(pattern, code, js)).toEqual({out, err: '', exit: null}); + }); + + test('have rr_ in front, so they can not clash with what RexReplace uses', async () => { + // With _ in front, the group _require with a space was __require_, which r() uses itself. + // $1 makes the code run once per match, where the variables of the groups are made. + const config = {pipedData: 'ab', replacementJs: true}; + expect((await rr('(?<_require>a)', '$1 + r("path").sep', config)).out).toBe('a/b'); + }); + + test('are gone with _ in front', async () => { + expect((await rr('(?\\d+)', '_year', js)).err).toContain('ReferenceError'); + }); +}); + +describe('#825 a named group that ends with _', () => { + test.each([['a_'], ['Year_'], ['_'], ['y2_']])( + 'panics for %s, as that is the name of another group with a space, and nothing is changed', + async name => { + fs.writeFileSync('a.md', 'in 1999'); + const result = await rr(`(?<${name}>\\d+)`, '"x"', { + includeGlob: ['a.md'], + replacementJs: true, + }); + expect(result).toEqual({ + out: '', + err: ` ❌ A named group can not end with _: ${name}\n No further processing.\n\n`, + exit: 3, + }); + expect(fs.readFileSync('a.md', 'utf8')).toBe('in 1999'); + } + ); + + test('is fine with --output-match, as the code is not run', async () => { + const config = {pipedData: 'ab', replacementJs: true, outputMatch: true}; + expect((await rr('(?a)', 'x', config)).exit).toBe(null); + }); + + test('is fine without --js, as there are no variables then', async () => { + expect((await rr('(?o)', 'x', {pipedData: 'foo'})).out).toBe('fxx'); + }); +});