feat: Add file data loading code with reload, retry, and polling

[kinyoklion]

Summary

Adds ldclient.impl.integrations.files.filedata, the file reading, parsing, and merging logic that the file-based override source described by the OVERRIDE specification is built on. The specification requires ordered multi-file merging with duplicate keys handling, a reload that keeps the last good data across a malformed edit and retries, debouncing of change notifications, a polling change detector as well as a watching one, and correct handling of a configured file that does not exist yet. This module provides those pieces; the override source itself is added in a later PR of the stack.

The existing file data sources are not changed. The FDv1 file update processor (Files.new_data_source) and the FDv2 file initializer and synchronizer (Files.new_data_source_v2, datasystem.file_ds_builder) keep their current implementation and behavior in every respect: the flagValues expansion (an on flag with fallthrough variation 0 and version 1, evaluated with a FALLTHROUGH reason), the version fallback that stamps version 1 on entries without one, the failure messages and their levels, the FDv2 synchronizer's OFF state on a failed initial load and its INVALID_DATA error kind, and the polling and watching rules. A new test module, test_file_data_sources_pinned_behavior.py, pins each of those behaviors so the override feature stays purely additive. The baseline test files for the existing sources are unmodified.

What the new module provides for the override source:

• Document parsing. Documents are parsed the way the existing file data sources parse them: with the YAML parser, which reads JSON too, when pyyaml is installed, and with the standard JSON parser otherwise. The content is never inspected to choose a parser. Flag and segment definitions are decoded into the model classes while the file is read, so an invalid definition fails that load. A definition may omit its own key and version, which are filled in from the map key and a version of 1.
• Merging. Files are combined in the configured order. The duplicate keys handling is fail (the load fails) or ignore (the first file's entry is kept). The result records how many entries each file supplied.
• Reloader. Serializes reloads, debounces change signals with a settle window that each signal extends, keeps the last good data by not applying a failed load, retries a failed load after a bounded delay so a file observed mid-write recovers without another notification, reports an identical failure once, skips an application whose file contents are byte-identical to the last applied contents, and applies a success after a failure even when the contents did not change. It can treat a configured file that does not exist as a file with no content. Closing does not wait for an in-flight reload. An exception raised by apply or on_error is reported like any other load failure and retried, and nothing from that load is remembered.
• Poller. Compares modification time and size on an interval, so a same-size rewrite and a same-time size change are both detected, as are files that appear or disappear.
• Watcher. Uses the watchdog package. It watches the directory of each file so an absent file is picked up when it appears, matches the destination of a move event so a file written by rename is detected, retries a directory that does not exist yet, and reacts only to notifications that can change a file's content or presence (the watchdog inotify mask also reports a file being opened or read). A failed observer start is retried every second, and a watched directory that is deleted is watched again once it exists, with one change signaled.

Tests cover parsing, merging, loading, the reloader (initial load, failure retention, debounce coalescing and window extension, retry with and without further signals, identical-failure reporting, skip-unchanged and recovery, serialization of concurrent reloads, close semantics, worker thread lifecycle), the poller, and the watcher, plus the pinned behavior of the existing sources.

SDK-3250

---

Note

Overview
Introduces ldclient.impl.integrations.files.filedata, shared infrastructure for a future file-based flag override source: JSON/YAML parsing (flags, flagValues, segments), ordered multi-file merge with fail vs ignore duplicate keys, and optional skip missing paths.

Adds a Reloader (debounced triggers, bounded retries, last-good data on failure, SHA-256 skip unchanged, recovery apply after errors) plus Poller (mtime + size, appear/disappear) and Watcher (watchdog with directory retry and filtered event types)—behavior that intentionally differs from today’s FDv1/FDv2 pollers/watchers.

No changes to Files.new_data_source or Files.new_data_source_v2; test_file_data_sources_pinned_behavior.py locks their current semantics (messages, flagValues expansion, polling/watching rules). test_filedata.py covers the new module end-to-end.

^{Reviewed by Cursor Bugbot for commit b309a87. Bugbot is set up for automated code reviews on this repo. Configure here.}

[kinyoklion]

bugbot review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b309a87. Configure here.}

[kinyoklion]

The file data poller test that expected exactly one callback per rewrite is replaced by one that checks the poller goes quiet once the file is stable, following launchdarkly/go-server-sdk#473. A plain rewrite is not atomic (the write and the modification-time update are separate steps), so a poll that lands between them legitimately reports an extra change; the poller promises generous detection and the reloader absorbs the excess. The test writes in the ordinary way, drains the callbacks of the write, and then requires silence, which still fails for a poller that does not update its baseline.

Written by Claude Code on behalf of @kinyoklion.

[kinyoklion]

The windows (3.10) failure on 7f80543 was test_reloader_reports_identical_failure_only_once, the same flaw as the poller test, in a test this PR had not touched. The test writes the file with a new failure while the 50 ms retry is running and expects the next retry to report it. The write is ordinary, not atomic, so a retry can read the file while it is empty. parse_document treats an empty document as valid, so that retry succeeds, the retries stop, and the new content is never read; require_errored then times out, which is what the run shows.

Reproduced locally with a deliberately slow write (truncate, sleep 200 ms, write): the old test shape failed 3/3 with the same timeout and the new one passed 3/3. The test now loads the final content with reload_now and inspects the drained reports as a whole (fd206c0, with a drain helper on the fixture); the first half of the test still proves that identical retries are reported once. The test 30/30 runs green, the reloader tests 10/10, test_filedata.py 67, and the file data plus override tests at the top of the stack (8d2c06c) 316. The stack is rebased onto this commit.

Written by Claude Code on behalf of @kinyoklion.