[jwrosewell]

Closes #163.

Problem

A refresh swapped in the new data set and kept everything else the engine had taken from the old file.

• The Device Detection engine kept its property metadata, the properties each detection reads and its tier (device-detection-onpremise/src/engine.rs lines 159 to 173, reload_from_path, lines 361 to 379). A Lite engine refreshed onto the Enterprise test file reported the tier Lite and 22 properties and read no value for HardwareModel, where an engine built on that file reports Enterprise, 289 properties and iPhone.
• The IP Intelligence engine kept its tier, the aspect metadata tagged with it, and the publish date the builder recorded (ip-intelligence-onpremise/src/engine.rs lines 218 to 247).

Update on start-up and automatic updates make this common, because a deployment that starts from the free Lite file with a licence key receives the file the key entitles it to. .NET reads the metadata again after every refresh (DeviceDetectionHashEngine.cs lines 489 to 509). #163 has the details.

Change

• Each engine keeps the properties a detection reads with the data set they belong to, swapped with it, as the IP Intelligence engine already did for its readable properties. A detection reads the properties of the data set it ran against, even after a refresh.
• Each engine replaces its property metadata and tier on a refresh that loads a file with others, keeping a tier the builder set. The pipeline traits lend the metadata and the tier by reference for as long as the engine lives, so each one replaced is kept rather than freed, in a small list of the distinct values loaded. A refresh to a file with the same properties and tier adds nothing to it.
• A refresh of the IP Intelligence engine records the new file's publish date.
• Refreshes of one engine run one at a time.

How it was verified

New tests, each needing a second data file beside the one CI provides, and skipping with a note without it:

• a_refresh_reads_and_reports_the_properties_of_the_new_data_file (device-detection-onpremise/tests/refresh.rs) refreshes an engine built on the Lite file after Enterprise-HashV41.hash replaces it, and checks its tier, its properties, its aspect properties and the HardwareModel it reads for an iPhone match an engine built on the Enterprise file.
• a_refresh_reports_the_tier_and_publish_date_of_the_new_data_file (ip-intelligence-onpremise/tests/refresh.rs) refreshes an engine built on the ASN file after 51Degrees-LiteV41.ipi replaces it, and checks its tier, its aspect properties and its publish date match an engine built on the Lite file.
• A unit test of the list of kept values in each crate.

Before the change, on a branch holding only the new tests, both fail. The Device Detection engine refreshed onto the Enterprise file still reports the tier Lite where an engine built on that file reports Enterprise, and the IP Intelligence engine refreshed onto the Lite file still reports IPIV4Asn where an engine built on it reports IPIV4Lite. After it, both pass.

Refresh with work in flight, measured with this change and #195 applied, with a probe that is part of neither: four threads ran 25,866 Device Detection detections while the data file alternated between the Lite and the Enterprise files and 14 refreshes ran two at a time. No detection or refresh failed, a detection made before the first refresh still read its values after the last, and the engine ended reporting Enterprise with 289 properties. The same with IP Intelligence gave 405,765 lookups over 532 refreshes under InMemory, and 43,941 lookups over 124 refreshes under Default, with no failure.

Locally on Windows, with all seven fixes of this sweep merged together (#175, #178, #182, #184, #194, #195 and #196), cargo test --no-fail-fast -p fiftyone-pipeline-engines -p fiftyone-device-detection-onpremise -p fiftyone-ip-intelligence-onpremise -p fiftyone-device-detection -p fiftyone-ip-intelligence -- --nocapture, with 51DEGREES_DD_PATH naming the Lite Hash file, Enterprise-HashV41.hash beside it, and 51DEGREES_IPI_PATH naming the ASN file, 51Degrees-LiteV41.ipi beside it, gives 130 passed, 0 failed and 2 ignored across 24 test binaries. With the fixes merged, cargo doc --workspace --no-deps with RUSTDOCFLAGS="-D warnings" passes, and rustfmt --check passes on every file the seven fixes change. cargo clippy --workspace --all-targets -- -D warnings on the merged fixes has not run yet. The Pull Request workflow, which builds and tests each pull request on every platform, is waiting for approval to run.

Behaviour that changes

After a refresh to a data file with other properties or another tier, the engine lists, reports and reads that file's properties and tier, as an engine built on it does. A tier set on the builder is kept. Each distinct property list an engine has loaded stays in memory until the engine is dropped, one list for each change of the data file's properties or tier.

What remains

• Elements that read an engine's properties once and keep them, such as the set headers element after its first scan, do not see a refresh's new properties until the pipeline is built again.
• The second data file each test needs is not among the files CI provides, so CI runs these tests only where it has one.