[hiteshkava]

Description

Adds an AI_MODERATION_TRUSTED_LINK_DOMAINS setting to the AI moderation from #288. It lets a deployment keep links its learners legitimately share from being classified as spam.

The problem

The classifier only sees the text of a post, never the course it was posted in. On edx.org, learners in a CSS course share their work as CodePen links, and those posts were classified as spam and deleted. To the classifier, a post that is mostly a link to another site looks like link spam. Changing the prompt can't fix this reliably, because the classifier doesn't know which links are on topic in a course and its verdicts vary from run to run.

Removing the trusted link isn't always enough. A short post such as Use this link for registration: <a href="https://zoom.us/meeting/register/…">URL</a> reaches the classifier as Use this link for registration: URL, which it reads as a lure.

What this PR does

When a post is moderated:

1. Links to a trusted domain or its subdomains are removed, whether written as plain URLs (with or without https://) or as <a href> anchors. An anchor is replaced by its text. The host must end exactly at the trusted domain, so codepen.io matches www.codepen.io but not codepen.io.example.com or notcodepen.io.

2. If the post had trusted links and nothing else in it gives readers another way to be contacted or redirected, it is not classified. Any of the following counts as such a way:

   • a link to another domain: with http(s):// or www., a bare domain followed by a path under any TLD (discord.gg/…), or a bare domain under a common TLD (bestessays.com)
   • an anchor or embedded media (<img>, <iframe>, …)
   • an email address, including a spelled-out tutor at gmail dot com
   • a phone number (8 or more digits)
   • a mention of WhatsApp, Telegram, WeChat, Viber, Discord, Skype or Snapchat

   HTML entities, full-width letters and invisible characters are undone before this check, so evil.com and What\u200bsApp (with a zero-width space) are still caught.

3. Otherwise, the post is classified as before, without its trusted links. A scam that adds a trusted link next to a phone number is still caught.

Posts without trusted links are unaffected. Links to untrusted domains, including lookalikes such as codepen.oi, are still judged by the classifier. A false risk match only means the post is classified as it would be today.

Every post that skips classification is logged as AI moderation skipped, only trusted links: <id>, because it gets no audit log entry.

The setting is empty by default, so nothing changes until a deployment lists domains.

Linear-time link handling

Moderation runs inside the request that creates a post, and posts have no size limit, so every pattern stays linear on hostile input:

• Anchors are found by matching opening and closing tags separately, and an opening tag can't run past the next <.
• Every pattern alternative that can match a long run of characters only starts at the beginning of a word.

With Python 3.12, the whole link handling (stripping plus the risk check) takes:

Hostile post (2,000,000 characters)	Time
One long word	0.33 s
<a href="https://codepen.io/x">y repeated, never closed	0.20 s
<a href="x" repeated, no >	0.19 s
a. repeated	1.03 s

Changes

• src/forum/ai_moderation/links.py (new): strip_trusted_links(), is_blank() and has_risk_signals().
• src/forum/ai_moderation/service.py: removes trusted links before classification, skips classification as described above, and logs each skip.
• src/forum/settings/common.py: declares AI_MODERATION_TRUSTED_LINK_DOMAINS, defaulting to [].
• docs/how-tos/configure_ai_moderation.rst: documents the setting.

Supporting information

The same change is running on edx.org in edx/forum#66 and edx/forum#67. The code is identical, apart from one slice rewritten to satisfy this repository's pycodestyle (E203). The tests use StubProviderBackend from #288 instead of edx.org's provider backend.

Testing instructions

1. Configure AI moderation as described in docs/how-tos/configure_ai_moderation.rst, with any backend, and add:

   AI_MODERATION_TRUSTED_LINK_DOMAINS = ["codepen.io"]

2. Enable discussions.enable_ai_moderation and discussions.enable_ai_auto_delete_spam for a test course.
3. Post My solution: https://codepen.io/<user>/pen/<id> why is the footer not sticky?. The backend is not called, the post stays visible with no ModerationAuditLog entry, and the LMS log shows AI moderation skipped, only trusted links: <id>.
4. Post Guaranteed A grades! WhatsApp +1-555-0199 https://codepen.io/<user>/pen/<id>. The backend is called with Guaranteed A grades! WhatsApp +1-555-0199 (the CodePen link removed), and the post is flagged if the backend says spam.
5. Post a link to a domain that isn't listed, such as https://example.com/offer. It is classified as before.

Unit tests:

pytest tests/test_ai_moderation_links.py tests/test_ai_moderation.py tests/test_ai_moderation_backends.py

• 101 new tests:
  • tests/test_ai_moderation_links.py (90 tests) covers:
    • link stripping: plain URLs, anchors, subdomains, lookalike domains
    • has_risk_signals(): bare links on any TLD, spelled-out and entity-encoded contacts, zero-width and full-width characters, plus everyday CSS, JavaScript and file paths that must not count as a risk
    • timing on hostile input
  • TestAIModerationTrustedLinks in tests/test_ai_moderation.py (11 tests) covers the service's behavior.
• Locally, the full suite (446 tests), pylint, pycodestyle, pydocstyle, mypy, doc8 and the Sphinx build pass.

Deadline

None. #288 is merged, and this PR is rebased onto master.

Other information

• Accepted trade-off: a post that only links to a trusted domain is not classified at all. Pages on a trusted domain, such as a shared code snippet or a meeting registration page, are written by whoever posts them, so a lure can sit on the trusted page itself. The docs say this and advise against trusting sites where anyone can publish arbitrary pages, such as github.io.
• A pasted full meeting invite with a meeting ID or dial-in numbers counts as containing a phone number, so it is still classified.
• No migrations and no new dependencies.

Merge checklist:
Check off if complete or not applicable:

• Commit messages (and PR title, if squash merging) use the correct
  Conventional Commits type — they determine the
  release: fix: → patch, feat: → minor, ! / BREAKING CHANGE: → major
• Documentation updated (not only docstrings)
• Fixup commits are squashed away
• Unit tests added/updated
• Manual testing instructions provided
• Noted any: Concerns, dependencies, migration issues, deadlines, tickets

[hiteshkava]

@taimoor-ahmed-1 can you please review this PR.