feat: separate structural and semantic schema validation

docs/decisions/0017-static-authorization-schema.rst

@@ -19,7 +19,7 @@ Decision
 1. Schema format and boundary
 =============================
 
-The authz schema is a versioned YAML format for static permissions, permission categories, roles, and changes to existing roles. Every file declares ``schema_version`` and ``priority``, which apply to all definitions in that file. Priority is used only when role extensions conflict; it does not resolve conflicting base definitions (see `ADR 0031`_). Open edX publishes a :download:`machine-readable JSON Schema <../../src/openedx_authz/schema/authz-schema-v1.json>` for this format so that editors, CI, and the compiler all apply the same field and validation rules.
+The authz schema is a versioned YAML format for static permissions, permission categories, roles, and changes to existing roles. Every file declares ``schema_version`` and ``priority``, which apply to all definitions in that file. Priority is used only when role extensions conflict; it does not resolve conflicting base definitions (see `ADR 0031`_). Open edX publishes a :download:`machine-readable JSON Schema <../../src/openedx_authz/engine/schema/authz-schema-v1.json>` for this format so that editors, CI, and the compiler all apply the same field and validation rules.
 
 The existing static role and permission definitions in Python modules and ``authz.policy`` will move into the schema. Once this migration is complete, the schema becomes the source for static definitions, so developers add a new role or permission there without duplicating it in Python constants or policy files.
 
@@ -155,6 +155,12 @@ Embedded categories on every permission
 
 Repeating category labels makes localization and consistent presentation harder. A normalized category gives permissions one stable grouping reference.
 
+Addendum: 2026-10-20
+********************
+
+The machine-readable JSON Schema now lives with its validation engine at
+:download:`authz-schema-v1.json <../../src/openedx_authz/engine/schema/authz-schema-v1.json>`.
+
 References
 **********
 

docs/decisions/0018-authorization-schema-lifecycle.rst

@@ -107,6 +107,25 @@ Two limitations are accepted:
 * **Single-database assumption.** Serialization holds only when the generated policy and the lock live in the same database. The engine already assumes one database for its policy storage, so this is consistent with the rest of the system.
 * **Blocking wait without observability.** A second apply waits for the first with no timeout and no distinct signal, so a blocked apply is operationally indistinguishable from a slow one. This is acceptable because apply runs at deployment time, not on the request path.
 
+Addendum: Separate structural and semantic validation
+======================================================
+
+**2026-10-20**
+
+Schema processing separates parsing, structural validation, typed loading, and semantic validation into ordered phases.
+
+The lifecycle begins with these phases:
+
+1. **Discover** identifies the contributed schema resources.
+2. **Parse** reads each resource as YAML and preserves the resulting value with its source information.
+3. **Validate structure** checks every parsed value against the configured JSON Schema. The pipeline reports all structural errors and stops when it finds any.
+4. **Load** constructs typed schema definitions from structurally valid mappings.
+5. **Validate semantics** checks relationships across definitions, conflicts between contributions, supported scopes, and values that depend on application data such as the vendored Paragon icon names.
+
+Compile, render, apply, and consume keep the responsibilities defined above. Typed loading and semantic validation do not run when structural validation fails. This ordering lets JSON Schema report malformed collection shapes and value types before typed construction assumes those structures are valid.
+
+The separation adds an intermediate parsed-document representation and another explicit pipeline boundary. Callers that previously treated loading as both YAML parsing and typed construction must use the corresponding phase for the representation they need.
+
 Consequences
 ************
 

docs/references/authorization-schema-json.rst

@@ -3,116 +3,116 @@
 Generated Authorization Schema
 ##############################
 
-This page renders the exact validation rules from the :download:`authz JSON Schema <../../src/openedx_authz/schema/authz-schema-v1.json>`. See the :doc:`Authorization Schema Reference <authorization-schema>` for guidance, examples, and validation performed by the compiler across schema contributions.
+This page renders the exact validation rules from the :download:`authz JSON Schema <../../src/openedx_authz/engine/schema/authz-schema-v1.json>`. See the :doc:`Authorization Schema Reference <authorization-schema>` for guidance, examples, and validation performed by the compiler across schema contributions.
 
 Top-level fields
 ****************
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/properties/schema_version
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/properties/schema_version
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/properties/priority
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/properties/priority
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/properties/permission_categories
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/properties/permission_categories
    :lift_description:
    :hide_key: /items
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/properties/permissions
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/properties/permissions
    :lift_description:
    :hide_key: /items
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/properties/roles
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/properties/roles
    :lift_description:
    :hide_key: /items
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/properties/role_extensions
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/properties/role_extensions
    :lift_description:
    :hide_key: /items
 
 Permission category fields
 **************************
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission_category/properties/id
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission_category/properties/id
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission_category/properties/display_name
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission_category/properties/display_name
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission_category/properties/description
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission_category/properties/description
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission_category/properties/icon
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission_category/properties/icon
    :lift_description:
 
 Permission fields
 *****************
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission/properties/namespace
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission/properties/namespace
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission/properties/name
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission/properties/name
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission/properties/display_name
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission/properties/display_name
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission/properties/description
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission/properties/description
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission/properties/category_id
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission/properties/category_id
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission/properties/scopes
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission/properties/scopes
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/permission/properties/icon
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/permission/properties/icon
    :lift_description:
 
 Role fields
 ***********
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role/properties/id
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role/properties/id
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role/properties/display_name
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role/properties/display_name
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role/properties/description
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role/properties/description
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role/properties/scopes
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role/properties/scopes
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role/properties/permissions
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role/properties/permissions
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role/properties/icon
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role/properties/icon
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role/properties/hidden
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role/properties/hidden
    :lift_description:
 
 Role extension fields
 *********************
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role_extension/properties/role_id
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role_extension/properties/role_id
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role_extension/properties/add_permissions
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role_extension/properties/add_permissions
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role_extension/properties/remove_permissions
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role_extension/properties/remove_permissions
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role_extension/properties/display_name
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role_extension/properties/display_name
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role_extension/properties/description
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role_extension/properties/description
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role_extension/properties/icon
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role_extension/properties/icon
    :lift_description:
 
-.. jsonschema:: ../../src/openedx_authz/schema/authz-schema-v1.json#/$defs/role_extension/properties/hidden
+.. jsonschema:: ../../src/openedx_authz/engine/schema/authz-schema-v1.json#/$defs/role_extension/properties/hidden
    :lift_description:

docs/references/authorization-schema.rst

@@ -19,7 +19,7 @@ Use this reference when creating or reviewing an authz schema file. The examples
 Machine-readable schema
 ***********************
 
-The :download:`downloadable authz JSON Schema <../../src/openedx_authz/schema/authz-schema-v1.json>` describes the fields, types, required values, and identifier formats for schema version ``1.0``. You can also :doc:`browse the generated schema <authorization-schema-json>` or `view the schema source <https://github.com/openedx/openedx-authz/blob/main/src/openedx_authz/schema/authz-schema-v1.json>`_. Editors and validation tools can use it to check YAML files because JSON Schema applies to the data represented by both JSON and YAML.
+The :download:`downloadable authz JSON Schema <../../src/openedx_authz/engine/schema/authz-schema-v1.json>` describes the fields, types, required values, and identifier formats for schema version ``1.0``. You can also :doc:`browse the generated schema <authorization-schema-json>` or `view the schema source <https://github.com/openedx/openedx-authz/blob/main/src/openedx_authz/engine/schema/authz-schema-v1.json>`_. Editors and validation tools can use it to check YAML files because JSON Schema applies to the data represented by both JSON and YAML.
 
 The JSON Schema checks the structure of one file. The compiler also checks references across the combined schema, registered scope namespaces, available Paragon icons, and conflicts between contributions.
 

pyproject.toml

@@ -36,6 +36,7 @@ dependencies = [
     "edx-drf-extensions",
     "edx-opaque-keys",
     "edx-organizations",
+    "jsonschema>=4.0",
     "openedx-atlas",
     "openedx-events",
     "openedx-filters>=3.15.0",

src/openedx_authz/constants/schema.py

@@ -12,6 +12,9 @@
 
 from enum import Enum
 
+AUTHZ_JSON_SCHEMA_SETTING = "OPENEDX_AUTHZ_JSON_SCHEMA"
+DEFAULT_AUTHZ_JSON_SCHEMA = "openedx_authz.engine.schema/authz-schema-v1.json"
+
 
 class SchemaOriginKind(str, Enum):
     """Whether a schema contribution is a base definition or an extension.

src/openedx_authz/schema/authz-schema-v1.json → src/openedx_authz/engine/schema/authz-schema-v1.json

@@ -1,6 +1,6 @@
 {
   "$schema": "https://json-schema.org/draft/2020-12/schema",
-  "$id": "https://raw.githubusercontent.com/openedx/openedx-authz/main/src/openedx_authz/schema/authz-schema-v1.json",
+  "$id": "https://raw.githubusercontent.com/openedx/openedx-authz/main/src/openedx_authz/engine/schema/authz-schema-v1.json",
   "title": "Open edX Authorization schema",
   "description": "A version 1.0 Open edX authorization schema contribution. Applications and deployment configuration use this format to define static permission categories, permissions, roles, and changes to existing roles.",
   "type": "object",
@@ -121,7 +121,7 @@
     "scope_namespace": {
       "description": "A scope namespace registered by a ScopeData type.",
       "type": "string",
-      "minLength": 1,
+      "pattern": "^[a-z][a-z0-9_-]*$",
       "examples": ["course-v1"]
     },
     "display_name": {
@@ -139,7 +139,7 @@
     "icon": {
       "description": "A case-sensitive icon name exported by @openedx/paragon/icons.",
       "type": "string",
-      "minLength": 1,
+      "pattern": "^[A-Z][A-Za-z0-9]*$",
       "examples": ["Visibility"]
     },
     "permission_list": {

src/openedx_authz/engine/schema/issues.py

@@ -0,0 +1,37 @@
+"""Validation finding types shared by structural and semantic checks."""
+
+from dataclasses import dataclass
+from enum import StrEnum
+
+
+class IssueLevel(StrEnum):
+    """Severity of a :class:`ValidationIssue`.
+
+    Members:
+        ERROR: Blocks deployment.
+        WARNING: Reported only, does not block deployment.
+    """
+
+    ERROR = "error"
+    WARNING = "warning"
+
+
+@dataclass(frozen=True)
+class ValidationIssue:
+    """A single validation finding.
+
+    Attributes:
+        level: :attr:`IssueLevel.ERROR` (blocks deployment) or
+            :attr:`IssueLevel.WARNING` (reported only).
+        message: Human-readable description.
+        source_id: The contributing source, when the issue is file-specific.
+    """
+
+    level: IssueLevel
+    message: str
+    source_id: str | None = None
+
+    @property
+    def is_error(self) -> bool:
+        """Return whether this finding blocks the pipeline."""
+        return self.level == IssueLevel.ERROR