Repository navigation
Add new exercises: RMS Norm, data loading, sliding window bonus + fix BPE - #34
Merged
Merged
Conversation
RMS Norm: - Rewrite rms-norm.ipynb as a proper exercise with stubs - Add rms-norm_SOLN.ipynb with complete solution - Add rms_norm.py grader (6 checks: module type, shape, learnable scale, reference match, gradient flow, numerical stability) - Update problem.toml: grading tier B→A, add entries and hints Byte Pair Encoding: - Fix problem statement in both Q and SOLN notebooks (was describing a Transformer model instead of BPE) - Fix "Byte Pain Encoder" typo in solution - Fix truncated test cell in question notebook - Rewrite merge_vocab to use index-walking instead of fragile string replace - Align end-of-word marker to </w> across Q and SOLN notebooks - Remove debug print statements from solution
Add a follow-up exercise to sliding window attention that uses torch.Tensor.unfold to achieve genuine O(n·w) compute without building the full n×n score matrix. Includes problem statement, step-by-step TODOs in the question notebook, complete solution, and validation tests.
1. Preloaded Dataset (easy) — map-style Dataset with all data in memory, DataLoader batching/shuffling, multi-worker loading 2. Streaming Dataset (medium) — IterableDataset with yield, line-by-line file reading, multi-worker sharding via get_worker_info() 3. Buffered Loading (medium) — shuffle buffer wrapping an IterableDataset, O(buffer_size) memory, approximate shuffling 4. Data Parallel Loading (medium) — DistributedSampler from scratch, rank sharding, deterministic seeded shuffling, padding Each exercise includes question notebook with TODOs, solution notebook, and problem.toml manifest.
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
@mathieu-charbonnel is attempting to deploy a commit to the exorust's projects Team on Vercel. A member of the Team first needs to authorize it. |
num_workers > 0 causes pickle errors in Jupyter notebooks since classes defined in notebook cells can't be serialized for multiprocessing. Replace multi-worker tests with shuffle verification (preloaded) and code inspection (streaming).
mathieu-charbonnel
force-pushed
the
contribution
branch
from
October 20, 2026 12:34
bfec894 to
1e4ffb9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
problem.toml(tier B→A)merge_vocabto use index-walking, aligned end-of-word marker to</w>across Q and SOLNunfold-based follow-up exercise achieving genuine O(n·w) compute without the n×n score matrixv3/data-loading/section):README.mdandPROBLEMS.mdwith new links and graded statusTest plan
python3 scripts/generate.py checkpasses with the new manifests