Skip to content

Access named groups from --js code to close #825 #832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mathiasrw merged 16 commits into from
Oct 23, 2026
Merged

Access named groups from --js code to close #825 #832

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
918855d
Give a panic and not a stack trace when a path can not be looked at
mathiasrw Oct 23, 2026
e366dd6
Stop starting new files after a panic with --halt
mathiasrw Oct 23, 2026
ff4c0e5
Read the piped paths of --glob-pipe as UTF-8
mathiasrw Oct 23, 2026
9b6aec0
Remove control characters from the new names of --rename
mathiasrw Oct 23, 2026
a2c62df
Give a message and not a crash when --js code throws something that i…
mathiasrw Oct 23, 2026
8be509b
Name a thrown error whose message can not be read, and cut a big thro…
mathiasrw Oct 23, 2026
eab2c2b
Tell where --js code failed in a module with --verbose
mathiasrw Oct 23, 2026
e225977
Read each line of the stack, skip what is not the code of the user, a…
mathiasrw Oct 23, 2026
2ad9cd2
Only make the --js numbers the code uses, and only count files for $i…
mathiasrw Oct 23, 2026
71b1a51
Tell in the help that only the numbers the code names are made
mathiasrw Oct 23, 2026
1b890fe
Leave the README line of --js to #838, so the stack can be rebased wi…
mathiasrw Oct 23, 2026
07dbd4d
Let --js code declare the names of the numbers and groups with let an…
mathiasrw Oct 23, 2026
f7b634f
Use the prefix rr_ for named groups in --js, and panic on a name that…
mathiasrw Oct 23, 2026
c3fcc36
Do not panic on a group name that ends with _ under --output-match
mathiasrw Oct 23, 2026
ad127c6
Test named groups with uppercase, numbers, $ and the space cousin, an…
mathiasrw Oct 23, 2026
8b406bc
Merge main into fix/825
mathiasrw Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -366,7 +366,7 @@ If planned renames clash, the process is halted and no change is applied to any
| `-g` | **`--glob-pipe`** Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments. [boolean] |
| | **`--null-in`** The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out. [boolean] |
| `-R` | **`--replacement-pipe`** Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean] |
| `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. If the javascript source code references to the full match or a captured group the code will run once per match. Otherwise, it will run once per file. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. The numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). A named group is a variable with `_` in front: `_year` for `(?<year>...)`, and `_year_` has a space at the end. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] |
| `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. If the javascript source code references to the full match or a captured group the code will run once per match. Otherwise, it will run once per file. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. The numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). A named group is a variable with `rr_` in front: `rr_year` for `(?<year>...)`, and `rr_year_` has a space at the end. A named group can not end with `_`. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] |
| `-x` | **`--exclude-re`** Files with a path that matches this regular expression will be excluded from processing. Can be provided multiple times. [string] |
| `-X` | **`--exclude-glob`** Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate. [string] |
| `-h` | **`--help`** Display help. [boolean] |
Expand Down
2 changes: 1 addition & 1 deletion src/cli.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -410,7 +410,7 @@ The following values defaults to \`❌\` if haystack does not originate from a f

The numbers: \`$n\`: the number of the file, \`$t\`: the number of files, \`$i\`: the number of the file among the files with a match so far, \`$c\`: among the files changed so far, \`$m\`: the number of the match in the file, \`$M\`: across all files. Each has cousins with zeros in front (\`$n0\` has 2 digits, \`$n00\` 3, and so on to 10) and with a space at the end (\`$n_\`). For speed, the code runs once per file unless it names the match, a group, \`$m\` or \`$M\`.

A named group is a variable with \`_\` in front: \`_year\` for \`(?<year>...)\`, and \`_year_\` has a space at the end.
Access a named group via the name prepended with \`rr_\`. \`(?<year>...)\` gives you \`rr_year\`. A named group name can not end with \`_\`.

All variables, except from module, date objects, \`nl\` and \`_\`, has a corresponding variable name followed by \`_\` where the content has an extra space at the end (for easy concatenation).
`
Expand Down
26 changes: 21 additions & 5 deletions src/engine.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,17 @@ async function run(config) {

config.regex = getFinalRegex(config.pattern, config) || '';

// In --js a named group is a variable like rr_year, and rr_year_ has a space at the end.
// A group that ends with _ would have the same name as another group with a space, so it can not be used.
// With --output-match the code is not run, so there are no variables.
const endsWith_ =
config.replacementJs && !config.outputMatch && config.regex
? namedGroups(config.regex).find(name => name.endsWith('_'))
: null;
if (endsWith_) {
panic('A named group can not end with _:', endsWith_);
}

// What the regex engine is asked to do. With --replacement-js and --output-match it gets a function.
if (config.statusInfo.engine) {
config.statusInfo.engine.execution = {
Expand Down Expand Up @@ -821,15 +832,20 @@ function countFile(config, text, result) {
}
}

// The code uses a named group of the pattern, like _year for (?<year>…)
// The code uses a named group of the pattern, like rr_year for (?<year>…)
function usesNamedGroups(config) {
return [...config.regex.source.matchAll(/\(\?<([A-Za-z_$][\w$]*)>/g)].some(([, name]) =>
new RegExp('(^|[^\\w$])_' + name.replace(/\$/g, '\\$') + '(?![\\w$])').test(
return namedGroups(config.regex).some(name =>
new RegExp('(^|[^\\w$])rr_' + name.replace(/\$/g, '\\$') + '(?![\\w$])').test(
config.replacementOri
)
);
}

// The names of the named groups of a regex, like year for (?<year>…)
function namedGroups(regex) {
return [...regex.source.matchAll(/\(\?<([A-Za-z_$][\w$]*)>/g)].map(([, name]) => name);
}

// The numbers of --js and their cousins: $n0 is $n with zeros in front to 2 digits, $n00 to 3 and so on to 10,
// and the ones with _ at the end have a space at the end. They are all given to the code as values, so they cost no time to compile.
const widths = [2, 3, 4, 5, 6, 7, 8, 9, 10];
Expand Down Expand Up @@ -1192,10 +1208,10 @@ function dynamicReplacement(
for (var i = 0; i < arguments.length - (hasNamed ? 3 : 2); i++) {
capturedGroups += 'var $' + i + '=' + JSON.stringify(arguments[i]) + '; ';
}
// A named group is a variable with _ in front, like _year for (?<year>…), and _year_ has a space at the end
// A named group is a variable with rr_ in front, like rr_year for (?<year>…), and rr_year_ has a space at the end
if (hasNamed) {
for (const [name, value] of Object.entries(named)) {
capturedGroups += `var _${name} = ${JSON.stringify(value)}, _${name}_ = _${name} + ' '; `;
capturedGroups += `var rr_${name} = ${JSON.stringify(value)}, rr_${name}_ = rr_${name} + ' '; `;
}
}

Expand Down
8 changes: 4 additions & 4 deletions test/fix/799.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@ describe('#799 groups in --js', () => {
expect((await rr('(?<x>a)(b)', '$1 + $2 + typeof $3', config)).out).toBe('abundefined');
});

test('a named group is a variable with _ in front, and _ at the end has a space', async () => {
test('a named group is a variable with rr_ in front, and _ at the end has a space', async () => {
const config = {pipedData: '2024-05 1999-12', replacementJs: true};
const pattern = '(?<year>\\d{4})-(?<month>\\d\\d)';
expect((await rr(pattern, '_month + "/" + _year', config)).out).toBe('05/2024 12/1999');
expect((await rr(pattern, '_month_ + _year', config)).out).toBe('05 2024 12 1999');
expect((await rr(pattern, 'rr_month + "/" + rr_year', config)).out).toBe('05/2024 12/1999');
expect((await rr(pattern, 'rr_month_ + rr_year', config)).out).toBe('05 2024 12 1999');
});

test('a named group that took no part in the match is undefined', async () => {
const config = {pipedData: 'ab', replacementJs: true};
expect((await rr('(?<x>a)|(?<y>b)', 'String(_x) + String(_y) + ","', config)).out).toBe(
expect((await rr('(?<x>a)|(?<y>b)', 'String(rr_x) + String(rr_y) + ","', config)).out).toBe(
'aundefined,undefinedb,'
);
});
Expand Down
2 changes: 1 addition & 1 deletion test/fix/815.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ describe('#815 --js code can declare its own variables', () => {
['a function', 'o', 'function $t() { return "F"; } $t()', 'fFF'],
['a number with let', 'o', 'let $t = "T"; $t', 'fTT'],
['a group with const', '(o)', 'const $1 = "X"; $1', 'fXX'],
['a named group with let', '(?<o>o)', 'let _o = "N"; _o', 'fNN'],
['a named group with let', '(?<o>o)', 'let rr_o = "N"; rr_o', 'fNN'],
])('with the name of %s', async (name, pattern, code, out) => {
expect((await rr(pattern, code, {pipedData: 'foo', replacementJs: true})).out).toBe(out);
});
Expand Down
Loading
Loading