[jwrosewell]

Closes #160.

Builds on #175, whose commits it carries, because the errors of a download read as it arrives go through the message and redaction that pull request adds. Merge #175 first, and this pull request then shows its own change alone.

Problem

• The service read the whole response with Response::bytes (pipeline-engines/src/data_update_service.rs line 645), which reqwest's blocking client limits, as a whole, to its timeout of 30 seconds by default. A data file update that took longer failed, even while it kept arriving.
• It held the download in memory and decompressed it into a second buffer (line 670). The IP Intelligence Enterprise file of 25 October 2026 is 7.19 GB, served gzipped at 2.51 GB, so an update would need about 9.7 GB of memory on top of the engine's data sets, and 670 Mbit/s to arrive within 30 seconds.
• write_atomically (lines 777 to 782) left its temporary file beside the data file whenever the write or the rename failed.
• Dropping the service joins its scheduler thread, so dropping it during a download waited for the download to end.

#160 has the details.

Change

• The response is read through Read, so the client's timeout limits each wait for more of it, and written as it arrives to a file beside the data file, with its MD5 taken as it is read. That file is decompressed into a second one beside it, which replaces the data file with a rename. Both are removed when they go out of scope, so an update that fails at any step leaves nothing behind.
• The new file's modified time is recorded before the rename, which keeps it, so the watcher does not refresh the engine a second time for the same file.
• Reading the response and decompressing stop once the service is dropped, so dropping it no longer waits for a download.
• A memory data source still takes its update as bytes, now read through Read as well.
• The documentation of OnPremiseAspectEngine::temp_data_dir no longer says the service uses it, as it never has.

How it was verified

New tests:

• a_steady_download_longer_than_the_client_timeout_completes (tests/remote_refresh.rs) sends a 51,200 byte file in ten parts over five seconds to a service whose client has a two second timeout.
• downloading_an_update_holds_little_of_the_file_in_memory (tests/download_memory.rs, alone in its file because it counts every allocation through its own global allocator) applies a 32 MiB update, plain and gzipped, and checks the most memory in use rose by under 8 MiB.
• a_failed_update_leaves_no_temporary_file (tests/remote_refresh.rs) puts a folder where the data file should be, so the rename fails, and checks nothing else is left.
• dropping_the_service_stops_a_download_in_progress (tests/remote_refresh.rs) drops the service a second into a scheduled download whose body takes 30 seconds, and checks the drop returns within five seconds and leaves no file behind.

Before the change, on a branch holding only the new tests, all four fail:

• a_steady_download_longer_than_the_client_timeout_completes: Err("data update failed (StreamError): error reading data update response from 'http://127.0.0.1:51991/datafile': error decoding response body").
• downloading_an_update_holds_little_of_the_file_in_memory: plain: a 33554432 byte update added 33968136 bytes at most.
• a_failed_update_leaves_no_temporary_file: ["device.dat", "device.tmp-download"] left in the folder.
• dropping_the_service_stops_a_download_in_progress: dropping the service waited 29.0380915s for the download.

After it, all four pass, and the memory test prints plain: a 33554432 byte update added 473356 bytes at most and gzip: a 33554432 byte update added 211050 bytes at most.

Locally on Windows, with all seven fixes of this sweep merged together (#175, #178, #182, #184, #194, #195 and #196), cargo test --no-fail-fast -p fiftyone-pipeline-engines -p fiftyone-device-detection-onpremise -p fiftyone-ip-intelligence-onpremise -p fiftyone-device-detection -p fiftyone-ip-intelligence -- --nocapture, with 51DEGREES_DD_PATH naming the Lite Hash file, Enterprise-HashV41.hash beside it, and 51DEGREES_IPI_PATH naming the ASN file, 51Degrees-LiteV41.ipi beside it, gives 130 passed, 0 failed and 2 ignored across 24 test binaries. With the fixes merged, cargo doc --workspace --no-deps with RUSTDOCFLAGS="-D warnings" passes, and rustfmt --check passes on every file the seven fixes change. cargo clippy --workspace --all-targets -- -D warnings on the merged fixes has not run yet. The Pull Request workflow, which builds and tests each pull request on every platform, is waiting for approval to run.

Behaviour that changes

• An update is no longer limited to the client's timeout as a whole. A response that stops for longer than the timeout still fails.
• An update needs, beside the data file, free space for the download and for the decompressed file while it is applied, where before it needed memory for both.
• Temporary files are named after the data file with the process id, a counter, and .download or .new, beside the data file, where before one was named with the extension tmp-download.
• Dropping the service ends a download in progress, which is not applied.

What remains

• On Windows the rename still fails while the engine holds the data file open, which every profile but InMemory does. Rust: On Windows a data update cannot replace the data file of an engine that reads it from disk #167 covers that, with an operational copy of the data file. With this change the failed update leaves no file behind.
• The .NET pipeline reads the whole download into memory before writing it, found by reading its code (The data update download is read into memory before it is written to disk pipeline-dotnet#461).