Skip to content

Read and write files as UTF-8, so Windows does not lose the file - #64

Open
gal-a wants to merge 1 commit into
basnijholt:mainfrom
gal-a:utf8-file-io
Open

gal-a wants to merge 1 commit into
basnijholt:mainfrom
gal-a:utf8-file-io

Conversation

@gal-a

@gal-a gal-a commented Nov 5, 2026 •

Copy link
Copy Markdown

On Windows, in Python's default mode, running markdown-code-runner on a
markdown file crashes and leaves the file empty.

The file is opened for writing with the locale's encoding, which is cp1252
on most Western Windows setups. Opening it for writing empties it, and then writing the
⚠️ This content is auto-generated marker fails, since cp1252 has no ⚠️.
The output file is the input file by default, so the original is gone.

To check, in Git Bash on Windows with Python 3.9 to 3.14 and without
PYTHONUTF8 set:

printf 'Some text\n\n<!-- CODE:START -->\n<!-- print("hi") -->\n<!-- CODE:END -->\n<!-- OUTPUT:START -->\n<!-- OUTPUT:END -->\n' > demo.md
python -m markdown_code_runner demo.md

Before: UnicodeEncodeError, and demo.md is 0 bytes. After: the output is
written and the file keeps its text.

This passes encoding="utf-8" to the four reads and writes in
markdown_code_runner.py. CI runs on Ubuntu, where UTF-8 is the default,
which is why it did not show there.

No test added: on Ubuntu a test would pass with or without the fix.

Found with qikly, an AI-assisted test tool
I maintain, when the project's own tests failed on Windows. I reproduced it
by hand, and reviewed and tested this change myself.

@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] This PR appears safe to merge.

Summary

The PR makes all four file reads and writes in markdown_code_runner.py explicitly use UTF-8.

  • The runner reads and writes its files as UTF-8.

Reviews (1) · Last reviewed commit: "Read and write files as UTF-8" · Reviewed by Greptile

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant