Skip to content

Add fuzzy rule to rewrite URLs of lesbases.anct.gouv.fr (Fixes #334) - #352

Merged
benoit74 merged 1 commit into
openzim:mainfrom
Sriram-Nambiar:fuzzy-rule-lesbases-anct-gouv
Oct 19, 2026
Merged

benoit74 merged 1 commit into
openzim:mainfrom
Sriram-Nambiar:fuzzy-rule-lesbases-anct-gouv

Conversation

@Sriram-Nambiar

@Sriram-Nambiar Sriram-Nambiar commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #334
Related to openzim/zim-requests#1205

Summary of Changes:

  • Added fuzzy rule for lesbases.anct.gouv.fr in rules/rules.yaml to normalize Next.js _rsc query parameters to a fixed value (_rsc=1).
  • Added test cases covering root path, subpaths, www prefix, and query parameter variations.
  • Added changelog entry in CHANGELOG.md under [Unreleased] -> Added.

@benoit74

Copy link
Copy Markdown
Collaborator

@Sriram-Nambiar can you please share a working ZIM of lesbases.anct.gouv.fr? few pages is enough, no need to ZIM the whole website, just confirm the regex is working fine and they did not changed something which will cause it to fail again

@Sriram-Nambiar

Sriram-Nambiar commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Hi @benoit74 I generated a small sample ZIM and tested it

When clicking on links like /connexion, it unfortunately displays raw React JSON on a black screen instead of the webpage (screenshot attached). Next.js returns text/x-component for requests with ?_rsc=.... When _rsc is stripped by the fuzzy rule, warc2zim saves that component payload over the HTML page.

I've attached
lesbases_test_2026-10.zim.zip
the sample ZIM (.zip, ~6 MB) and the screenshot.
image

@benoit74

Copy link
Copy Markdown
Collaborator

Thank you very much! Do I get it correctly that JS code is inspecting the content-type header returned to take a decision?

@benoit74

Copy link
Copy Markdown
Collaborator

I also see that the main page keeps loading assets ... should we tweak the fuzzy-rule so that _rsc is still set but has a fixed value? Did you achieved to understand why this value is changing (like the ZIM does not produces the same value than the online website) and what it is used for?

@Sriram-Nambiar

Sriram-Nambiar commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Yes, you're correct! Next.js analyzes the Content-Type header of background fetches and determines whether they're of type React Server Components (text/x-component) or HTML (text/html). That's because the _rsc parameter is Next.js's cache-buster hash for those component payloads, and as such varies between builds and router states.

Your suggestion of keeping _rsc with a fixed value (_rsc=1) worked nicely! I tested it on a sample ZIM with 7 pages, and both the HTML and component files are saved separately now without overwriting each other, so /connexion loads cleanly.

I've attached a screenshot and the sample ZIM below. If this looks good to you, should I go ahead and push the updated rule to this PR?
lesbases_test_2026-10.zip

image

@benoit74

Copy link
Copy Markdown
Collaborator

I still see continuous web requests on home page in the new ZIM ... as if Next.js considered some pages were not fine and needed to be downloaded again

@Sriram-Nambiar

Sriram-Nambiar commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

The home page was sending multiple web requests because the prefetch target (/rechercher/tout/ressources) was only an incomplete stub in the 7 page crawl so Next.js kept retrying it. When served with the full payload Nextjs caches it on the first try and stops making any further requests.....so the loop is purely due to the shallow test crawl. The _rsc=1 rule itself is working as expected
image

@benoit74

Copy link
Copy Markdown
Collaborator

Nice, makes sense.

Please go ahead and push the updated rule (please simply amend your commit and rebase on latest main commit)

@Sriram-Nambiar
Sriram-Nambiar force-pushed the fuzzy-rule-lesbases-anct-gouv branch from 49bd5be to 077a542 Compare October 16, 2026 15:53
@codecov

codecov Bot commented Oct 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (936498a) to head (077a542).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #352   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           42        42           
  Lines         2729      2729           
  Branches       406       406           
=========================================
  Hits          2729      2729           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@benoit74 benoit74 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@benoit74 benoit74 assigned benoit74 and Sriram-Nambiar and unassigned benoit74 Oct 19, 2026
@benoit74
benoit74 merged commit cc3723c into openzim:main Oct 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add fuzzy rule to rewrite URLs of lesbases.anct.gouv.fr

2 participants