Reduce memory overhead of _deserialize_from_store

Observed behavior

When syncing large amounts of data and upon successful transfer, that data is deserialized into the store (_deserialize_from_store), memory usage often spikes.

Expected behavior

The behavior of _deserialize_from_store should reduce its memory usage by loading only the necessary data / models into memory.

User-facing consequences

In some online deployments with strict resource constraints, OOM killer may kill the process that is handling the deserialization.

@ozer550

Automatically unassigning @ozer550 due to no comments here, or updates on the associated pull request for 1 month. @ozer550, if you're still interested in this issue or already have work in progress, please message us here, and we'll assign you again. Thank you!

Closed by #392

  
  ┌──────────────────────────┬─────────┬─────────┬──────────┬──────────┐
  │           Case           │ Records │ Version │ Peak mem │ Duration │
  ├──────────────────────────┼─────────┼─────────┼──────────┼──────────┤
  │ 1: existing 18k users    │ 108,388 │ Before  │ 287.7 MB │ 93.16 s  │
  ├──────────────────────────┼─────────┼─────────┼──────────┼──────────┤
  │                          │         │ After   │ 100.3 MB │ 77.33 s  │
  ├──────────────────────────┼─────────┼─────────┼──────────┼──────────┤
  │ 2: generated 500 users   │ 184,369 │ Before  │ 754.7 MB │ 156.57 s │
  ├──────────────────────────┼─────────┼─────────┼──────────┼──────────┤
  │                          │         │ After   │ 126.0 MB │ 128.03 s │
  ├──────────────────────────┼─────────┼─────────┼──────────┼──────────┤
  │ 3: trimmed to 1000 users │ 33,678  │ Before  │ 178.1 MB │ 28.19 s  │
  ├──────────────────────────┼─────────┼─────────┼──────────┼──────────┤
  │                          │         │ After   │ 77.6 MB  │ 23.78 s  │
  ├──────────────────────────┼─────────┼─────────┼──────────┼──────────┤
  │ 4: trimmed to 5000 users │ 48,804  │ Before  │ 197.5 MB │ 41.22 s  │
  ├──────────────────────────┼─────────┼─────────┼──────────┼──────────┤
  │                          │         │ After   │ 82.9 MB  │ 34.08 s  │
  └──────────────────────────┴─────────┴─────────┴──────────┴──────────┘

Navigation Menu

Repository navigation

Reduce memory overhead of `_deserialize_from_store` #210

Observed behavior

Expected behavior

User-facing consequences

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Reduce memory overhead of _deserialize_from_store #210

Description

Observed behavior

Expected behavior

User-facing consequences

Activity

learning-equality-bot commented on Sep 28, 2026

bjester commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Reduce memory overhead of `_deserialize_from_store` #210