[mcarans]

Problem

Updating an existing dataset does not reliably change its locations (groups). Seen on stage with hdx-scraper-ophi:

Dataset	Countries in data	Tagged after update	Missing	Extra
global-mpi	113	112 (identical to prod)	aze, nru, vut	brb, lca
hdx-hapi-poverty-rate	113	99	15 incl. ago…civ, bih, nru	lca

Cause

• _revise_dataset sends the caller's groups as [{"name": ...}, ...] in the update of package_revise.
• CKAN's update_merge_list (dictization_functions.py) merges the update list into the existing list by index, so each existing element keeps its id and only name is overwritten.
• package_membership_list_save (model_save.py) looks a group up by id before name, so every existing position keeps its old group. New groups are only added past the end of the old list.
• _revise_filter only emitted -groups__N when the removed element was the last one, so other removals never reached CKAN.

This matches the stage results exactly: global-mpi had 112 groups, and the only appended element (index 112, zwe) was already present. The HAPI dataset had 98, and of the appended ton…zwe only vut was new.

The bug dates from #60 (6.2.2, Feb 2024).

Fix

For each non-resource list key in both the update and the existing dataset, _merge_list_fields:

• merges each update dict that has an id or name over the HDX dict with the same id (or failing that, name), so fields the caller didn't send are kept (_merge_list). A new group has no id, so CKAN looks it up by name.
• sends dicts with neither id nor name (eg. customviz) as given. Matching them by position would carry fields over from a removed entry.
• drops HDX elements with no match.
• treats a null or non-list HDX value as an empty list.
• returns the keys whose merged list differs from the HDX list.

_revise_filter adds -<key> for each changed key, skipping keys already in keys_to_delete. CKAN applies filter before update, so the list is replaced by the merged list.

Resources are unchanged and still handled by index via resources_to_delete.

keep_crisis_tags now only runs when the update contains tags. Previously it set the update's tags to just the crisis tags (or []), which with the new filter would wipe a dataset's tags when the caller sent none. When the caller sends tags, crisis tags are still kept because keep_crisis_tags adds them to the update's tags.

Effect

• Updates whose groups or tags differ from HDX send -groups/-tags. Pipelines whose location tags have drifted since 2024 will lose stale locations and gain missing ones on their next run.
• Unchanged lists send no filter, so daily runs and the post-create revise don't rewrite memberships.
• Updates that don't send tags leave HDX tags untouched.

Testing

• test_update_logic filters are unchanged from main; their expected updates no longer contain "tags": [].
• Added test_revise_filter_lists (changed vs unchanged lists, merged ids, unsent fields kept, dicts without id/name sent as given, null HDX value, no duplicate filter with keys_to_delete) and test_merge_list_id_before_name.
• Full suite passes except test_ckan, an integration test that needs HDX_PIPELINE_GSHEET_AUTH (unchanged by this PR).
• Verified on stage with a private test dataset (created, updated 5 times, read back after each, deleted):

Check	Result
Groups on create	afg, pse, syr, yem
Groups after removing first and adding ZMB	pse, zmb
Tags / customviz when omitted from update	unchanged
Tags reduced to displacement	displacement
First customviz entry removed	only b, no fields carried over from a
keys_to_delete=["groups"] with changed groups	pse

Not covered on stage: keeping crisis-* tags, and id-before-name matching (unit-tested only).

[github-actions]

Test Results

196 tests  +2   196 ✅ +2   1m 9s ⏱️ -2s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit b28db94. ± Comparison against base commit e8b2af7.

♻️ This comment has been updated with latest results.

[coveralls]

coverage: 97.405% (+0.005%) from 97.4% — fix-revise-list-replace into main