[jwrosewell]

Closes #159.

Problem

FiftyOneUrlFormatter writes the licence keys into the query of the URL each data update poll requests (pipeline-engines/src/data_file.rs line 72). reqwest names the URL it requested in every error, and the service wrote that error into its own message (pipeline-engines/src/data_update_service.rs lines 597 to 602 and 645 to 650). So each key reached the error a caller gets, the warning the scheduler logs (line 737) and the error an engine builder returns when update on start-up fails. #159 has the messages.

Change

• A new request_error_message gives a request error's message and each of its causes, from an error with its URL removed by reqwest's Error::without_url. Both messages that report a request error use it, and go on naming the configured URL, which carries no key.
• Every failing exit of a poll now comes through check_for_update_from_url, which takes the data file's licence keys out of the message by value with fiftyone_pipeline_core::redact::redact_with, as the cloud request engine does for its credentials. That covers a key that reaches a message some other way. The poll itself moves to poll_update_url.

How it was verified

New tests in pipeline-engines/tests/remote_refresh.rs poll through the Distributor formatter with a known licence key:

• a_failed_connection_keeps_the_licence_key_out_of_the_error polls a port nothing listens on, and checks the error names the configured URL and the cause, and not the key.
• a_failed_download_keeps_the_licence_key_out_of_the_error polls a server that promises 1,000 bytes and sends five, and checks the same.

Before the change, on a branch holding only the new tests, both fail, each message carrying the key:

data update failed (HttpsError): error accessing data update service at 'http://127.0.0.1:51989/api/v2/download': error sending request for url (http://127.0.0.1:51989/api/v2/download?LicenseKeys=LICENCE-KEY-NOT-FOR-LOGS&Download=True)
data update failed (StreamError): error reading data update response from 'http://127.0.0.1:51990/api/v2/download': error decoding response body for url (http://127.0.0.1:51990/api/v2/download?LicenseKeys=LICENCE-KEY-NOT-FOR-LOGS&Download=True)

After it, both pass.

Locally on Windows, with all seven fixes of this sweep merged together (#175, #178, #182, #184, #194, #195 and #196), cargo test --no-fail-fast -p fiftyone-pipeline-engines -p fiftyone-device-detection-onpremise -p fiftyone-ip-intelligence-onpremise -p fiftyone-device-detection -p fiftyone-ip-intelligence -- --nocapture, with 51DEGREES_DD_PATH naming the Lite Hash file, Enterprise-HashV41.hash beside it, and 51DEGREES_IPI_PATH naming the ASN file, 51Degrees-LiteV41.ipi beside it, gives 130 passed, 0 failed and 2 ignored across 24 test binaries. With the fixes merged, cargo doc --workspace --no-deps with RUSTDOCFLAGS="-D warnings" passes, and rustfmt --check passes on every file the seven fixes change. cargo clippy --workspace --all-targets -- -D warnings on the merged fixes has not run yet. The Pull Request workflow, which builds and tests each pull request on every platform, is waiting for approval to run.

Behaviour that changes

A failed poll's error, and the warning logged with it, no longer carry the licence keys. They now carry what went wrong, for example error sending request: client error (Connect): tcp connect error: ..., where before they said only error sending request for url (...) with the formatted URL.

What remains

• The download path is rewritten by Write a data file update to disk as it arrives, and remove its temporary files when it fails #178, for Rust: A data file update must arrive within 30 seconds, is held in memory, and leaves a temporary file when it fails #160, which builds on this change and keeps the keys out of its read errors as well.
• The specification change that keeps license keys out of log entries and error messages, Keep license keys out of log entries and error messages specifications#55, and the same change for .NET (The data update service writes the license keys to the log pipeline-dotnet#459) and Java (The data update service writes the license keys to the log pipeline-java#167), which both write the keys to their logs.