Scope filters should list read-write partitions last #398

❌ This issue is not open for contribution. Visit Contributing guidelines to learn about the contributing process and how to find suitable issues.

Overview

Make the read and write filters of a Scope list the read-write partitions last, so that deserialization in filter order puts shared records before the records that reference them.

Background & Motivation

#392 streams deserialization one partition prefix at a time, in the order of the sync filter. A record whose FK target is in a later partition gets a deserialization error, because the target is not in the app table yet. The legacy deserialization looped model by model across all partitions, so the order of the filter did not matter before 0.9.0.

The contract in #392 is that the caller gives the partitions in a usable order. But callers do not always write the filter. Kolibri passes Scope.read_filter and Scope.write_filter directly as its pull and push filters, and Scope builds them as the read-write partitions, then the read-only or write-only partitions:

In the single-user scope, the logs are in user-rw:U and the FacilityUser is in user-ro:U. Morango deserializes the logs first, and they fail on a device that does not have the user yet. With #392 installed, two of Kolibri's Morango integration tests failed for this reason:

• SingleUserSyncRegressionsTestCase::test_kolibri_issue_8439
• SingleUserSyncRegressionsTestCase::test_issue_fixed_in_morango_pull_146

Both tests passed after Kolibri reordered the read filter itself, with the read-only partitions first. That fix puts a Morango detail in Kolibri, and every other implementer would need the same fix.

User Story / Use Case

As a developer of an app that syncs with Morango,
I want the filters that Scope builds from my scope definition to keep a usable order,
So that I control the deserialization order through my templates, and do not have to reorder Morango's filters.

Description & Expected Outcomes

• Scope.read_filter lists the read-only partitions, then the read-write partitions.
• Scope.write_filter lists the write-only partitions, then the read-write partitions.
• Within each template, the partitions keep the order of the template.
• The order the filter must have for deserialization is documented, so implementers know how to order their templates.

Read-write partitions usually hold the device's own records, such as a user's logs. Those records usually reference shared records in the read-only partitions, such as the user, the facility and the classrooms. The reverse is rare. This order is a convention, not a guarantee. A read-only record that references a read-write record still fails, and Morango cannot detect this case, because it knows the dependencies between models but not between partitions.

Value Add & Impact

• Implementers that pass Scope filters directly get a usable deserialization order without extra code.
• Kolibri does not need to reorder the read filter of the single-user scope.

Deliverables & Contracts

• Scope.read_filter and Scope.write_filter list the read-write partitions last.
• The docstrings of Scope and of the stream source describe the order contract.
• The 0.9.0 section of CHANGELOG.md records the new order.

Acceptance Criteria

• A test builds a Scope from templates with read-only, write-only and read-write partitions, and checks the order of both filters.
• A deserialization test with a filter in the single-user shape deserializes a record before the records that reference it.
• Kolibri's Morango integration tests pass without Kolibri reordering the read filter, including test_kolibri_issue_8439, test_issue_fixed_in_morango_pull_146 and test_single_user_course_syncing.
• make test passes on SQLite and PostgreSQL.

Technical Pointers & Architecture

• Target Components / Context: Scope.__init__ in morango/models/certificates.py. MorangoSource.prefix_conditions in morango/sync/stream/source.py documents the order contract.
• Related Patterns: Filter.__add__ keeps the order of both operands and removes duplicates.
• Data Model & Schema Considerations: No model fields or migrations change. TransferSession.filter stores str(filter), so new transfer sessions store the partitions in the new order.
• Resilience & Failure Modes: The comparisons we found do not depend on order. These are Filter.is_subset_of, which the server uses to validate a requested filter, and Filter.__eq__, which morango/sync/context.py and Kolibri use. Check for any other code that compares filter strings.

Notes & Tradeoffs

• Dependencies: Related to Streaming deserialization part 3: sink #392. Do this before 0.9.0 ships, because 0.9.0 is the first release in which the filter order matters.
• Testing Requirements: tests/testapp/tests/test_api.py takes the first partition of a read_filter with .split()[0]. Its scope has no read-write partitions, so the change does not affect it, but check similar tests.
• Tradeoffs & Alternatives Considered:
  • Kolibri reorders the filter itself. This works, and the Kolibri integration tests pass with it, but each implementer must know how Scope combines the templates.
  • Morango sorts the partition prefixes. Streaming deserialization part 3: sink #392 tried this before release. It broke test_single_user_course_syncing, where the classroom partitions sort before the user partitions they reference.
  • Morango deserializes model by model across all partitions, with one query that combines the prefixes, as the legacy code did. This makes the order irrelevant, but we do not want to combine many prefixes into one query expression.
• Compatibility & Migrations: Devices on older Morango versions do not depend on the filter order, so mixed-version syncs are not affected.

Metadata

Complexity: Low
Target Branch: release-v0.9.x
Labels: TAG: tech update / debt

AI Usage

Drafted with Claude Code while we investigated Kolibri integration test failures against #392. Claude traced the failures to the order of Scope.read_filter, and tested a Kolibri-side reorder that made the tests pass. I rejected a Morango-side fix that combined the prefixes into one query, and chose to track the Scope change in this issue instead of in #392.