Skip to content

Resolve relative links against page URLs with consecutive slashes - #358

Open
benoit74 wants to merge 1 commit into
mainfrom
benoit74/relative-links-in-a-page-whose-url-contains-are
Open

benoit74 wants to merge 1 commit into
mainfrom
benoit74/relative-links-in-a-page-whose-url-contains-are

Conversation

@benoit74

Copy link
Copy Markdown
Collaborator

Summary

  • ArticleUrlRewriter resolved relative links (and base_href) with urllib.parse.urljoin, which collapses empty path segments while merging a relative reference onto the base URL. When a page's own URL contains //, this silently collapsed it before normalize() could see it, so the rewritten link pointed to an entry that doesn't exist in the ZIM.
  • Replaced it with a custom join following RFC 3986 section 5.3 (merge + remove-dot-segments) that keeps empty segments, matching how browsers resolve such references.
  • Checked the JS (wombat) side: it already resolves relative URLs correctly via uri-js's URI.resolve, which doesn't have this deviation, so no fix was needed there — added a regression test to document/lock in that behavior instead.

Fix #346

urllib.parse.urljoin collapses empty path segments while merging a
relative reference onto the base URL, so a page whose own URL contains
`//` had its relative links resolved to the wrong absolute URL (slashes
collapsed before normalize() even saw them). Replace it with a custom
join that follows RFC 3986 section 5.3 without that deviation, matching
how browsers resolve such references.

The JS (wombat) side already resolves relative URLs correctly via
uri-js's URI.resolve, so only the Python rewriter needed a fix; added a
regression test on the JS side to document that.

Fix #346
@benoit74

Copy link
Copy Markdown
Collaborator Author

@Sinkleberg / @anshuman83-40 I would be happy to have your feedback on this one. Cannot ask a formal review but would be happy to have an informal one ^^

Looks like there is no easy way besides re-implementing Python function ...

@benoit74 benoit74 self-assigned this Oct 20, 2026
@codecov

codecov Bot commented Oct 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.19048% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.64%. Comparing base (6613ac5) to head (b33d06b).

Files with missing lines Patch % Lines
src/zimscraperlib/rewriting/url_rewriting.py 76.19% 4 Missing and 6 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##              main     #358      +/-   ##
===========================================
- Coverage   100.00%   99.64%   -0.36%     
===========================================
  Files           42       42              
  Lines         2798     2837      +39     
  Branches       417      431      +14     
===========================================
+ Hits          2798     2827      +29     
- Misses           0        4       +4     
- Partials         0        6       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@anshuman83-40

Copy link
Copy Markdown
Contributor

Thanks for picking this up! I had a look and it seems right to me:

  • _urljoin gives the expected result for all 42 examples in RFC 3986 §5.4 (normal and abnormal), and for the // cases (other.html, ../z.html, ?page=2, #top, //cdn…) from a page like https://example.com/x//y/page.html.
  • On 200k random base/reference pairs without // it returns exactly what urllib.parse.urljoin returns, so behaviour shouldn't change elsewhere.

About the coverage: in _remove_dot_segments, the ../, ./ and bare ./.. branches can't be reached since the merged path always starts with / (as the docstring says), so they could be dropped or marked # pragma: no cover. Same for if not base and the non-uses_netloc case. The one reachable gap is a query-only reference like ?page=2 (the if not query branch), which might be worth a test.

Small thing: _resolve_relative_reference in tests/rewriting/test_url_rewriting.py (from #344) now duplicates _remove_dot_segments, so the round-trip test could probably reuse it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Relative links in a page whose URL contains // are resolved with the slashes collapsed

2 participants