From 15aa52ac2d3df46712719a26e54018f0e4ff2d33 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Fri, 23 Oct 2026 12:30:17 +1100 Subject: [PATCH 1/6] Add --glob-pipe to pipe in the files to work on to close #711 --- README.md | 17 +++++++-- src/cli.ts | 31 ++++++++++----- src/engine.ts | 17 ++++++++- test/cli/run.sh | 30 +++++++++++++++ test/js/fix/703.test.js | 26 +++++++------ test/js/fix/711.test.js | 85 +++++++++++++++++++++++++++++++++++++++++ 6 files changed, 181 insertions(+), 25 deletions(-) create mode 100644 test/js/fix/711.test.js diff --git a/README.md b/README.md index d88508b2..ba4268a5 100644 --- a/README.md +++ b/README.md @@ -168,10 +168,16 @@ The path of each file that changed is printed to stdout, one name on each line. > rexreplace 'foo' 'bar' '**/*.md' | xargs git add ``` -Use `--null` to separate names with `\0` instead of a new line. Can be helpful when chaining data to avoid problems with space and newlins. +Use `--null-out` (or `--null`) to separate names with `\0` instead of a new line. Can be helpful when chaining data to avoid problems with space and newlins. ```bash -> rexreplace 'foo' 'bar' '**/*.md' --null | xargs -0 git add +> rexreplace 'foo' 'bar' '**/*.md' --null-out | xargs -0 git add +``` + +Pipe the names into the next run with `--glob-pipe`, so it only works on the files the first one changed: + +```bash +> rexreplace 'foo' 'bar' '**/*.md' --null-out | rexreplace 'baz' 'qux' --glob-pipe --null-in ``` @@ -233,7 +239,7 @@ Use --js-pipe to access the pipe data in your javascript code. | | **`--name-delimiter`** String printed between the names of the files. Default is a line break, unless the name already ends with one. [string] | | | **`--content-delimiter`** String printed between the content of each file for --output, or between the matches for --output-match. Default is a line break, unless the content already ends with one. [string] | | | **`--delimiter`** Sets --name-delimiter and --content-delimiter to the same value. Supports escape sequences like `\t`, `\n` and the `€` and `§` aliases. [string] | -| | **`--null`** Shorthand for --delimiter '\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null. [boolean] | +| | **`--null-out`**, **`--null`** Shorthand for --delimiter '\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null-out. [boolean] | | | **`--name-delimiter-tail`** Print the name delimiter after the last name. Defaults to true. Use --no-name-delimiter-tail or --name-delimiter-tail=false to not print the delimiter after the last name. [boolean] | | | **`--content-delimiter-tail`** Print the content delimiter after the last content. Defaults to true, but to false when piped data is printed, so it is left as it came in. Matches from --output-match always get it, from a file or a pipe. [boolean] | | | **`--tail`** Shorthand for setting both --name-delimiter-tail and --content-delimiter-tail. Use --no-tail to not print the delimiter after the last name and the last content. The specific tail flags win over --tail. [boolean] | @@ -254,6 +260,8 @@ Use --js-pipe to access the pipe data in your javascript code. | `-T` | **`--trim-pipe`** Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. [boolean] | | `-p` | **`--pipe`** Read the piped data and make it available to --replacement-js as `pipe`. [boolean] | | | **`--js-pipe`** Shorthand for --replacement-js --pipe (-jp). [boolean] | +| `-g` | **`--glob-pipe`** Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments. [boolean] | +| | **`--null-in`** The piped paths and globs of --glob-pipe are separated by NUL bytes, like from `find -print0` or --null-out. [boolean] | | `-R` | **`--replacement-pipe`** Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean] | | `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. If the javascript source code references to the full match or a captured group the code will run once per match. Otherwise, it will run once per file. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] | | `-x` | **`--exclude-re`** Files with a path that matches this regular expression will be excluded from processing. Can be provided multiple times. [string] | @@ -270,6 +278,7 @@ Files identified for potential processing are identified by: - Subtract group C from group A and B (the files are compared by their full path) - Remove any file paths that matches any `-x` provided regex. Use `--full-path` to look for the full path. - All globs are case-insensitive, like the search, unless `--void-ignore-case` is set. +- The lines piped in with `--glob-pipe` are input values too, just like the arguments. ### Features @@ -279,7 +288,7 @@ Files identified for potential processing are identified by: - Supports backreference to matching groups in the replacement - Data can be piped in, both as the text to search and as the replacement - The names of the changed files are printed to stdout, so they can be piped on (to stderr with `--output` and `--output-match`) -- Pick the delimiter between names and between the content of files, like `--null` for `xargs -0` +- Pick the delimiter between names and between the content of files, like `--null-out` for `xargs -0` - See the [release note](https://github.com/mathiasrw/rexreplace/releases) for a log of changes. Descriptions are given in latest patch version. ### Quirks diff --git a/src/cli.ts b/src/cli.ts index b6c4564c..292cbb95 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -155,10 +155,11 @@ const yargs = require('yargs') 'Sets --name-delimiter and --content-delimiter to the same value. Supports escape sequences like `\\t`, `\\n` and the `€` and `§` aliases.' ) - .boolean('null') + .boolean('null-out') + .alias('null-out', 'null') .describe( - 'null', - "Shorthand for --delimiter '\\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null." + 'null-out', + "Shorthand for --delimiter '\\0'. Useful for piping content with spaces or line breaks (like for `xargs -0`). The specific delimiter values win over --delimiter, and --delimiter wins over --null-out." ) .boolean('name-delimiter-tail') @@ -275,6 +276,22 @@ const yargs = require('yargs') .boolean('js-pipe') .describe('js-pipe', `Shorthand for --replacement-js --pipe (-jp).`) + .boolean('g') + .alias('g', 'glob-pipe') + .describe( + 'g', + 'Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments.' + ) + // The piped data can only be one thing + .conflicts('g', ['R', 'p', 'js-pipe']) + + .boolean('null-in') + .describe( + 'null-in', + 'The piped paths and globs of --glob-pipe are separated by NUL bytes, like from `find -print0` or --null-out.' + ) + .implies('null-in', 'glob-pipe') + .boolean('j') .alias('j', 'js') .alias('j', 'replacement-js') @@ -366,11 +383,6 @@ All variables, except from module, date objects, \`nl\` and \`_\`, has a corresp .describe('G', "filename/globas are filename(s) for files containing one filename/globs on each line to be search/replaced") .alias('G', 'globs-file') - .boolean('g') - .describe('g', "filename/globs will be piped in. If any filename/globs are given in command the piped data will be prepened") - .alias('g', 'glob-pipe') - - .boolean('j') .describe('j', "Pattern is javascript source that will return a string giving the pattern to use") .alias('j', 'pattern-js') @@ -563,7 +575,8 @@ const run = () => ); // Like grep, stdin is only read when piped data is expected. An open pipe that nobody writes to would make the run wait forever. -const pipeDataExpected = !config.includeGlob.length || config.replacementPipe || config.pipe; +const pipeDataExpected = + !config.includeGlob.length || config.replacementPipe || config.pipe || config.globPipe; if (!pipeDataExpected || Boolean(process.stdin.isTTY)) { run(); diff --git a/src/engine.ts b/src/engine.ts index d5b3e34f..0bd0fb82 100644 --- a/src/engine.ts +++ b/src/engine.ts @@ -77,6 +77,18 @@ function newStatus(config) { async function run(config) { validateEncoding(config.encoding); + // With --glob-pipe each line of the piped data is a path or a glob, used together with the ones given as arguments + if (config.globPipe) { + if (config.replacementPipe || config.pipe) { + die( + '--glob-pipe can not be used with --replacement-pipe or --pipe, as the piped data can only be one thing' + ); + } + config.pipedDataUsed = true; + const lines = (config.pipedData ?? '').split(config.nullIn ? '\0' : /\r?\n/); + config.includeGlob = config.includeGlob.concat(lines.filter(line => '' !== line)); + } + config.pattern = getFinalPattern(config.pattern, config) || ''; config.replacement = getFinalReplacement(config.replacement, config) || ''; @@ -84,7 +96,7 @@ async function run(config) { config.replacementOri = config.replacement; // The most specific delimiter wins. Without one there is a line break after each item, unless it already ends with one. - const delimiter = config.delimiter ?? (config.null ? '\0' : null); + const delimiter = config.delimiter ?? (config.nullOut ? '\0' : null); config.nameDelimiter = config.nameDelimiter ?? delimiter; config.contentDelimiter = config.contentDelimiter ?? delimiter; @@ -262,6 +274,9 @@ function updated(file, config) { // How the piped data was used function pipeType(config, piped) { + if (config.globPipe) { + return 'GLOBS'; + } if (piped) { return 'CONTENT'; } diff --git a/test/cli/run.sh b/test/cli/run.sh index 200860da..244a421e 100644 --- a/test/cli/run.sh +++ b/test/cli/run.sh @@ -451,6 +451,36 @@ assert "rexreplace 'b' 'x' my.file --content-delimiter '\x4' 2>&1 | grep -c assert "rexreplace 'b' 'x' my.file --delimiter '\x4' 2>&1 | grep -c 'Invalid escape sequence in --delimiter'" '1' assert "cat my.file" 'foobar' +# --glob-pipe: each line of the piped data is a path or a glob to work on (#711) +reset +assert "printf 'my.file\nyour.file\n' | rexreplace 'b' 'x' --glob-pipe -A" 'my.file\nyour.file' +reset +assert "printf '*.file' | rexreplace 'b' 'x' -g -A" 'my.file\nyour.file' +# The names of the changed files go on to the next command, also when a name has a line break in it +reset +assert "rexreplace 'oo' 'x' '*.file' --null-out | rexreplace 'x' 'y' --glob-pipe --null-in" 'my.file' +assert "cat my.file" 'fybar' +# The piped data can only be one thing +reset +assert_raises "printf 'my.file' | rexreplace 'b' 'x' -g -R" 1 +assert_raises "printf 'my.file' | rexreplace 'b' 'x' -g --js-pipe" 1 +assert_raises "printf 'my.file' | rexreplace 'b' 'x' --null-in" 1 +assert "cat my.file" 'foobar' + + +# Data with NUL bytes is binary safe: the NUL bytes come out where they went in +reset +assert "printf 'foo\0boo\0' | rexreplace 'o' 'x' | tr '\0' '|'" 'fxx|bxx|' +assert "printf 'foo\0boo\0' | rexreplace 'o' 'x' --output | tr '\0' '|'" 'fxx|bxx|' +assert "printf 'foo\0boo\0' | rexreplace 'o' '_' --output-match --null-out | tr '\0' '|'" 'o|o|o|o|' +printf 'foo\0boo\0' > nul.file +assert "rexreplace 'o' 'x' nul.file" 'nul.file' +assert "tr '\0' '|' < nul.file" 'fxx|bxx|' +printf 'foo\0boo\0' > nul.file +assert "rexreplace 'o' 'x' nul.file --output 2>/dev/null | tr '\0' '|'" 'fxx|bxx|' +assert "rexreplace 'o' '_' nul.file --output-match --null-out 2>/dev/null | tr '\0' '|'" 'o|o|o|o|' + + # Globs are case-insensitive like the search, unless --void-ignore-case (#716) reset assert "rexreplace 'oo' 'x' '*.FILE' -X 'YOUR*'" 'my.file' diff --git a/test/js/fix/703.test.js b/test/js/fix/703.test.js index 21ab0a07..847f7101 100644 --- a/test/js/fix/703.test.js +++ b/test/js/fix/703.test.js @@ -61,14 +61,16 @@ describe('#703 the names of affected files', () => { expect((await rr('x', 'o', {includeGlob: [file]})).out).toBe(file + '\n'); }); - describe('--null', () => { + describe('--null-out', () => { test('ends each name with a NUL byte', async () => { - expect(await run({null: true})).toEqual({out: 'a.file\0c.file\0', err: '', exit: null}); + expect(await run({nullOut: true})).toEqual({out: 'a.file\0c.file\0', err: '', exit: null}); }); test('keeps a name with a line break in one piece', async () => { await Bun.write('my\n.file', 'foo'); - expect((await rr('o', 'x', {includeGlob: ['*.file'], null: true})).out).toBe('my\n.file\0'); + expect((await rr('o', 'x', {includeGlob: ['*.file'], nullOut: true})).out).toBe( + 'my\n.file\0' + ); }); }); @@ -301,7 +303,7 @@ describe('#703 the names of affected files', () => { }); }); - describe('--delimiter and --null', () => { + describe('--delimiter and --null-out', () => { test('--delimiter sets both delimiters', async () => { expect(await run({output: true, delimiter: ', '})).toEqual({ out: 'fxx, bar, fxx, ', @@ -310,8 +312,8 @@ describe('#703 the names of affected files', () => { }); }); - test('--null is the same as --delimiter with a NUL byte', async () => { - expect(await run({output: true, null: true})).toEqual({ + test('--null-out is the same as --delimiter with a NUL byte', async () => { + expect(await run({output: true, nullOut: true})).toEqual({ out: 'fxx\0bar\0fxx\0', err: 'a.file\0c.file\0', exit: null, @@ -319,7 +321,9 @@ describe('#703 the names of affected files', () => { }); test('the most specific delimiter wins', async () => { - expect((await run({output: true, delimiter: ', ', null: true})).out).toBe('fxx, bar, fxx, '); + expect((await run({output: true, delimiter: ', ', nullOut: true})).out).toBe( + 'fxx, bar, fxx, ' + ); expect( await run({ output: true, @@ -328,7 +332,7 @@ describe('#703 the names of affected files', () => { nameDelimiter: ';', }) ).toEqual({out: 'fxx|bar|fxx|', err: 'a.file;c.file;', exit: null}); - expect(await run({output: true, null: true, nameDelimiter: ', '})).toEqual({ + expect(await run({output: true, nullOut: true, nameDelimiter: ', '})).toEqual({ out: 'fxx\0bar\0fxx\0', err: 'a.file, c.file, ', exit: null, @@ -354,8 +358,8 @@ describe('#703 the names of affected files', () => { expect(await run({[flag]: true, [quiet]: true})).toEqual({out, err: '', exit: null}); }); - test('follow --null, --name-delimiter and --full-path', async () => { - expect((await run({[flag]: true, null: true})).err).toBe('a.file\0c.file\0'); + test('follow --null-out, --name-delimiter and --full-path', async () => { + expect((await run({[flag]: true, nullOut: true})).err).toBe('a.file\0c.file\0'); expect((await run({[flag]: true, nameDelimiter: ', '})).err).toBe('a.file, c.file, '); expect((await run({[flag]: true, fullPath: true})).err).toBe( path.resolve('a.file') + '\n' + path.resolve('c.file') + '\n' @@ -382,7 +386,7 @@ describe('#703 the names of affected files', () => { }); test.each([ - ['null', {null: true}, '\0', '\0'], + ['null', {nullOut: true}, '\0', '\0'], ['name-delimiter', {nameDelimiter: ', ', nameDelimiterTail: false}, ', ', ''], ])('works with --%s', async (name, flags, between, last) => { expect((await run({fullPath: true, ...flags})).out).toBe( diff --git a/test/js/fix/711.test.js b/test/js/fix/711.test.js new file mode 100644 index 00000000..6585e47d --- /dev/null +++ b/test/js/fix/711.test.js @@ -0,0 +1,85 @@ +// https://github.com/mathiasrw/rexreplace/issues/711 +// A list of files from another command could only be used with `xargs` or `$(…)`. +import {describe, test, expect} from 'bun:test'; +import fs from 'fs'; +import {rr, rrStatus} from '../helper.js'; + +const content = name => fs.readFileSync(name, 'utf8'); + +// Make each file with the content `foo` +function makeFiles(...names) { + names.forEach(name => fs.writeFileSync(name, 'foo')); +} + +describe('#711 --glob-pipe', () => { + test('each line of the piped data is a file to work on', async () => { + makeFiles('a.file', 'b.file', 'c.file'); + const config = {pipedData: 'a.file\nb.file\n', globPipe: true}; + expect(await rr('o', 'x', config)).toEqual({out: 'a.file\nb.file\n', err: '', exit: null}); + expect(content('c.file')).toBe('foo'); + }); + + test('a line can be a glob', async () => { + makeFiles('a.file', 'b.file', 'c.txt'); + const result = await rr('o', 'x', {pipedData: '*.file', globPipe: true}); + expect(result.out).toBe('a.file\nb.file\n'); + }); + + test('a line is used together with the paths and globs given as arguments', async () => { + makeFiles('a.file', 'b.file', 'c.txt'); + const config = {pipedData: 'a.file\n', includeGlob: ['*.txt'], globPipe: true}; + expect((await rr('o', 'x', config)).out).toBe('a.file\nc.txt\n'); + }); + + test('a line that is a file is used as it is, like an argument (#718)', async () => { + makeFiles('a[1].file', 'a1.file'); + const result = await rr('o', 'x', {pipedData: 'a[1].file\n', globPipe: true}); + expect(result.out).toBe('a[1].file\n'); + expect(content('a1.file')).toBe('foo'); + }); + + test('a line that starts with ! takes files away from the other globs', async () => { + makeFiles('a.file', 'b.file'); + const result = await rr('o', 'x', {pipedData: '*.file\n!b.file\n', globPipe: true}); + expect(result.out).toBe('a.file\n'); + }); + + test('takes Windows line breaks, leaves out empty lines and keeps spaces in names', async () => { + makeFiles('a.file', 'my file.txt'); + const config = {pipedData: '\r\na.file\r\n\r\nmy file.txt\r\n', globPipe: true}; + expect((await rr('o', 'x', config)).out).toBe('a.file\nmy file.txt\n'); + }); + + test('with --null-in the lines are separated by NUL bytes', async () => { + makeFiles('a.file', 'line\nbreak.file'); + const config = {pipedData: 'a.file\0line\nbreak.file\0', globPipe: true, nullIn: true}; + expect((await rr('o', 'x', config)).out).toBe('a.file\nline\nbreak.file\n'); + expect(content('line\nbreak.file')).toBe('fxx'); + }); + + test('with nothing piped in and no arguments there are no files', async () => { + expect(await rr('o', 'x', {pipedData: '', globPipe: true})).toEqual({ + out: '', + err: '0 files found\n', + exit: 1, + }); + }); + + test('is told about in the status report', async () => { + makeFiles('a.file'); + const status = await rrStatus('o', 'x', {pipedData: 'a.file\n', globPipe: true}); + expect(status.pipe).toEqual({bytes: 7, type: 'GLOBS', updated: false}); + expect(status.files).toEqual({searched: ['a.file'], updated: ['a.file']}); + }); + + test.each([ + ['--replacement-pipe', {replacementPipe: true}], + ['--pipe', {pipe: true}], + ])('can not be used with %s, as the piped data can only be one thing', async (name, flags) => { + makeFiles('a.file'); + const result = await rr('o', 'x', {pipedData: 'a.file', globPipe: true, ...flags}); + expect(result.exit).toBe(1); + expect(result.err).toContain('--glob-pipe can not be used with'); + expect(content('a.file')).toBe('foo'); + }); +}); From 7cfca468d0be637f9e66156305c3d0752fc5096d Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Fri, 23 Oct 2026 12:45:45 +1100 Subject: [PATCH 2/6] Write null byte in place of NUL --- README.md | 2 +- src/cli.ts | 2 +- test/cli/run.sh | 14 +++++++------- test/js/fix/703.test.js | 4 ++-- test/js/fix/711.test.js | 2 +- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index ba4268a5..7a2a4e91 100644 --- a/README.md +++ b/README.md @@ -261,7 +261,7 @@ Use --js-pipe to access the pipe data in your javascript code. | `-p` | **`--pipe`** Read the piped data and make it available to --replacement-js as `pipe`. [boolean] | | | **`--js-pipe`** Shorthand for --replacement-js --pipe (-jp). [boolean] | | `-g` | **`--glob-pipe`** Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments. [boolean] | -| | **`--null-in`** The piped paths and globs of --glob-pipe are separated by NUL bytes, like from `find -print0` or --null-out. [boolean] | +| | **`--null-in`** The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out. [boolean] | | `-R` | **`--replacement-pipe`** Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean] | | `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. If the javascript source code references to the full match or a captured group the code will run once per match. Otherwise, it will run once per file. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] | | `-x` | **`--exclude-re`** Files with a path that matches this regular expression will be excluded from processing. Can be provided multiple times. [string] | diff --git a/src/cli.ts b/src/cli.ts index 292cbb95..0238f445 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -288,7 +288,7 @@ const yargs = require('yargs') .boolean('null-in') .describe( 'null-in', - 'The piped paths and globs of --glob-pipe are separated by NUL bytes, like from `find -print0` or --null-out.' + 'The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out.' ) .implies('null-in', 'glob-pipe') diff --git a/test/cli/run.sh b/test/cli/run.sh index 244a421e..1bda7d45 100644 --- a/test/cli/run.sh +++ b/test/cli/run.sh @@ -468,17 +468,17 @@ assert_raises "printf 'my.file' | rexreplace 'b' 'x' --null-in" 1 assert "cat my.file" 'foobar' -# Data with NUL bytes is binary safe: the NUL bytes come out where they went in +# Data with null bytes is binary safe: the null bytes come out where they went in reset assert "printf 'foo\0boo\0' | rexreplace 'o' 'x' | tr '\0' '|'" 'fxx|bxx|' assert "printf 'foo\0boo\0' | rexreplace 'o' 'x' --output | tr '\0' '|'" 'fxx|bxx|' assert "printf 'foo\0boo\0' | rexreplace 'o' '_' --output-match --null-out | tr '\0' '|'" 'o|o|o|o|' -printf 'foo\0boo\0' > nul.file -assert "rexreplace 'o' 'x' nul.file" 'nul.file' -assert "tr '\0' '|' < nul.file" 'fxx|bxx|' -printf 'foo\0boo\0' > nul.file -assert "rexreplace 'o' 'x' nul.file --output 2>/dev/null | tr '\0' '|'" 'fxx|bxx|' -assert "rexreplace 'o' '_' nul.file --output-match --null-out 2>/dev/null | tr '\0' '|'" 'o|o|o|o|' +printf 'foo\0boo\0' > null.file +assert "rexreplace 'o' 'x' null.file" 'null.file' +assert "tr '\0' '|' < null.file" 'fxx|bxx|' +printf 'foo\0boo\0' > null.file +assert "rexreplace 'o' 'x' null.file --output 2>/dev/null | tr '\0' '|'" 'fxx|bxx|' +assert "rexreplace 'o' '_' null.file --output-match --null-out 2>/dev/null | tr '\0' '|'" 'o|o|o|o|' # Globs are case-insensitive like the search, unless --void-ignore-case (#716) diff --git a/test/js/fix/703.test.js b/test/js/fix/703.test.js index 847f7101..ebbdfc5a 100644 --- a/test/js/fix/703.test.js +++ b/test/js/fix/703.test.js @@ -62,7 +62,7 @@ describe('#703 the names of affected files', () => { }); describe('--null-out', () => { - test('ends each name with a NUL byte', async () => { + test('ends each name with a null byte', async () => { expect(await run({nullOut: true})).toEqual({out: 'a.file\0c.file\0', err: '', exit: null}); }); @@ -312,7 +312,7 @@ describe('#703 the names of affected files', () => { }); }); - test('--null-out is the same as --delimiter with a NUL byte', async () => { + test('--null-out is the same as --delimiter with a null byte', async () => { expect(await run({output: true, nullOut: true})).toEqual({ out: 'fxx\0bar\0fxx\0', err: 'a.file\0c.file\0', diff --git a/test/js/fix/711.test.js b/test/js/fix/711.test.js index 6585e47d..75e2edd5 100644 --- a/test/js/fix/711.test.js +++ b/test/js/fix/711.test.js @@ -50,7 +50,7 @@ describe('#711 --glob-pipe', () => { expect((await rr('o', 'x', config)).out).toBe('a.file\nmy file.txt\n'); }); - test('with --null-in the lines are separated by NUL bytes', async () => { + test('with --null-in the lines are separated by null bytes', async () => { makeFiles('a.file', 'line\nbreak.file'); const config = {pipedData: 'a.file\0line\nbreak.file\0', globPipe: true, nullIn: true}; expect((await rr('o', 'x', config)).out).toBe('a.file\nline\nbreak.file\n'); From 6f82fdd403386c8b9075f33a7b9ad22f6af53376 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Fri, 23 Oct 2026 12:46:21 +1100 Subject: [PATCH 3/6] Say null bytes are kept, not that data is binary safe --- test/cli/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/cli/run.sh b/test/cli/run.sh index 1bda7d45..c72984fb 100644 --- a/test/cli/run.sh +++ b/test/cli/run.sh @@ -468,7 +468,7 @@ assert_raises "printf 'my.file' | rexreplace 'b' 'x' --null-in" 1 assert "cat my.file" 'foobar' -# Data with null bytes is binary safe: the null bytes come out where they went in +# Null bytes are kept: they come out where they went in. (Bytes that are not valid UTF-8 are only kept with -e latin1.) reset assert "printf 'foo\0boo\0' | rexreplace 'o' 'x' | tr '\0' '|'" 'fxx|bxx|' assert "printf 'foo\0boo\0' | rexreplace 'o' 'x' --output | tr '\0' '|'" 'fxx|bxx|' From dda891b6a77b57ad4f43b1d07364dd383f6ac640 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Fri, 23 Oct 2026 12:49:12 +1100 Subject: [PATCH 4/6] Show how to read null-separated data, and how to keep binary data --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 7a2a4e91..86e7538e 100644 --- a/README.md +++ b/README.md @@ -180,6 +180,12 @@ Pipe the names into the next run with `--glob-pipe`, so it only works on the fil > rexreplace 'foo' 'bar' '**/*.md' --null-out | rexreplace 'baz' 'qux' --glob-pipe --null-in ``` +Null bytes are just chars in the search, so null-separated data can be made readable by turning them into line breaks: + +```bash +> find . -name '*.md' -print0 | rexreplace '\x00' '\n' +``` + **Matching pairs** @@ -310,6 +316,7 @@ Files identified for potential processing are identified by: ### Limitations - RexReplace reads each file fully into memory, so working on your 8Gb log files will probably not be ideal. +- Null bytes are kept as they are. Files are read as utf8 by default, so in binary data the bytes that are not valid UTF-8 become `�` when the file is changed. Use `--encoding latin1` to keep every byte. - For versions of Node prior to 6, please use version 2.2.x. For versions of Node prior to 0.12, please use [the legacy version of RexReplace called rreplace](https://www.npmjs.com/package/rreplace) ### Priorities From 5a39f64cf6894079a542b6d0ca18111721d8c5e6 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Fri, 23 Oct 2026 13:29:10 +1100 Subject: [PATCH 5/6] Drop the xargs example from the delimiter feature, and say how to run binary safe --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 86e7538e..477754bb 100644 --- a/README.md +++ b/README.md @@ -294,7 +294,7 @@ Files identified for potential processing are identified by: - Supports backreference to matching groups in the replacement - Data can be piped in, both as the text to search and as the replacement - The names of the changed files are printed to stdout, so they can be piped on (to stderr with `--output` and `--output-match`) -- Pick the delimiter between names and between the content of files, like `--null-out` for `xargs -0` +- Pick the delimiter between names and between the content of files - See the [release note](https://github.com/mathiasrw/rexreplace/releases) for a log of changes. Descriptions are given in latest patch version. ### Quirks @@ -316,7 +316,7 @@ Files identified for potential processing are identified by: ### Limitations - RexReplace reads each file fully into memory, so working on your 8Gb log files will probably not be ideal. -- Null bytes are kept as they are. Files are read as utf8 by default, so in binary data the bytes that are not valid UTF-8 become `�` when the file is changed. Use `--encoding latin1` to keep every byte. +- Files are read as utf8 by default. Null bytes are kept as they are but for binary data in general not all bytes are valid and becomes `�` if changed. Use `--encoding latin1` to run binary safe: every byte is kept, but note that because that means one byte per char search and replace only works with ASCII chars (no utf8 or emojis). - For versions of Node prior to 6, please use version 2.2.x. For versions of Node prior to 0.12, please use [the legacy version of RexReplace called rreplace](https://www.npmjs.com/package/rreplace) ### Priorities From 8ec64ecdbd84c1350705e89a44d3d72c06dcb578 Mon Sep 17 00:00:00 2001 From: Mathias Wulff Date: Fri, 23 Oct 2026 13:32:31 +1100 Subject: [PATCH 6/6] Say set the delimiter, not pick --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 477754bb..5d0d3b2f 100644 --- a/README.md +++ b/README.md @@ -294,7 +294,7 @@ Files identified for potential processing are identified by: - Supports backreference to matching groups in the replacement - Data can be piped in, both as the text to search and as the replacement - The names of the changed files are printed to stdout, so they can be piped on (to stderr with `--output` and `--output-match`) -- Pick the delimiter between names and between the content of files +- Set the delimiter between names and between the content of files - See the [release note](https://github.com/mathiasrw/rexreplace/releases) for a log of changes. Descriptions are given in latest patch version. ### Quirks