Skip to content

Preserve POSIX filesystem bytes in string audio paths - #504

Open
antonsoo wants to merge 2 commits into
bastibe:masterfrom
antonsoo:fix/filesystem-path-encoding
Open

antonsoo wants to merge 2 commits into
bastibe:masterfrom
antonsoo:fix/filesystem-path-encoding

Conversation

@antonsoo

@antonsoo antonsoo commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

A valid POSIX filename can contain bytes that Python represents with surrogateescape. SoundFile opens its bytes path, but opening the str or Path returned by a directory scan raises UnicodeEncodeError because _open() encodes it with strict error handling.

Use os.fsencode() for the POSIX native path. It uses Python's filesystem encoding and error handler, so directory entries roundtrip correctly. The Windows wide-character path stays in its existing branch.

Two regressions create and rediscover a real byte-named WAV, then verify string/Path writes, reads and metadata against Python's independent wave reader. Both fail against unchanged master. Two recorded speech clips also pass all 24 read/info/write path roundtrips; the eight previously failing surrogate paths now work, and ordinary Unicode and byte-path controls stay exact.

Full suite: 373 passed on CPython and PyPy with libsndfile 1.2.2. The same two existing pyright diagnostics reproduce on unchanged master; no new diagnostics. The POSIX-specific regressions are skipped on Windows and first check byte-name support with Python's built-in open; filesystems returning EINVAL or EILSEQ skip that unsupported case. The macOS regression failure at the initial head prompted this test-only capability check; the native conversion is unchanged.

Updated-head macOS Python 3.12/3.13/3.14 jobs also succeed; the inspected 3.12 job has 371 passes and two filesystem-dependent skips. Other matrix jobs are still running.

@antonsoo

antonsoo commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor Author

The initial macOS failures were in the new filename regression. I added an independent byte-filename creation check using Python's file API before testing SoundFile; unsupported filesystems skip only on EINVAL or EILSEQ. The runtime os.fsencode() change is unchanged.

On updated head b8b1490c, the full CPython and PyPy suites each pass 373 tests on Linux, and recorded speech path roundtrips remain exact. The macOS Python 3.12/3.13/3.14 jobs now succeed; the inspected 3.12 job has 371 passes and two filesystem-dependent skips. The rest of the matrix is still running.

@bastibe

bastibe commented Oct 24, 2026

Copy link
Copy Markdown
Owner

there seem to be a lot of cases where we skip the test. In which cases does it actually run, i.e. in which cases is this change useful?

@antonsoo

Copy link
Copy Markdown
Contributor Author

The two cases run on POSIX filesystems that accept the byte filename and where Python represents it with surrogateescape. Both run and pass on Linux here; neither is skipped. They cover str and Path names rediscovered through os.listdir().

For example, on a UTF-8 Linux system, a filename containing the byte 0xe9 can come from an archive or a dataset created with another encoding. Python and SoundFile's bytes path can open it, but SoundFile currently raises UnicodeEncodeError for the str path returned by the directory scan. This is a constructed compatibility reproduction, not a reported production failure.

Windows uses the separate wide-character path. The inspected macOS jobs skip because Python's own byte-name creation fails with EILSEQ; that check runs before SoundFile. The remaining skip is for a filesystem codec that represents the name without surrogateescape, where this particular regression would not exercise the bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants