Skip to content

feat(v9): add creators for TagAttachment, S3Prefix and FlowControlOperation - #1083

Merged
cmgrote merged 8 commits into
mainfrom
FND-3846
Oct 20, 2026
Merged

cmgrote merged 8 commits into
mainfrom
FND-3846

Conversation

@cmgrote

@cmgrote cmgrote commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

✨ Description

Adds creator() to three shared types that more than one app builds. Each creator reproduces every app's existing qualifiedName, so no app changes identity.

Linear: FND-3846. The issue comments hold the template-reconciliation and required-attribute tables.

Type qualifiedName App
TagAttachment, qn_style="tag_object" (default) {tag_qn}/{object_qn} databricks
TagAttachment, qn_style="tag_attachments_object" {tag_qn}/attachments/{object_qn} dbt
TagAttachment, qn_style="object_tag" {object_qn}/{tag_name} bridge
S3Prefix {bucket_qn}/{prefix}/ (trailing slash) s3, bridge
FlowControlOperation, with scope {conn}/{scope}/{source_id} fabric notebook, sagemaker
FlowControlOperation, with parent_qualified_name {parent_qn}/jobs/{source_id} + flow_controlled_by fabric Spark job

Notes:

  • How the creators are added: as hand-written overlays in pyatlan_v9/model/assets/_overlays/, spliced into the generated class files the same way the generator does it (FND-3843 decision).

  • TagAttachment is passthrough = true in models, so its class is hand-managed. The creator is added directly to tag_attachment.py. The overlay copy only takes effect once passthrough types are generated.

  • S3Prefix also sets the s3_bucket relationship, s3_parent_prefix_qualified_name and s3_prefix_hierarchy, matching atlan-s3-app. The s3_parent_prefix relationship is left to the caller.

  • Models PR atlanhq/models#2093 adds qualifiedNameFormula for S3Prefix and FlowControlOperation. s3_prefix.py is regenerated from that branch, so the S3Prefix pattern is now ^.+/[^/]+/$ and creator output passes validate(for_creation=True). The other two files regenerate byte-identical.

  • TagAttachment is now generated by models master (the passthrough fix landed), and the generated file matches this PR's tag_attachment.py byte for byte. It has no qualifiedNameFormula, because the apps build three different shapes.

  • New connector types: AtlanConnectorType.FABRIC ("fabric", BI) and SAGEMAKER ("sagemaker", SaaS), matching the connector values in atlan-fabric-app and atlan-sagemaker-app. The integration tests need them.

  • S3Object fixes, which make tests_v9/integration/s3_asset_test.py pass in full (17 passed against a tenant):

    • Restores S3Object.create_with_prefix. 76c5b17 added it to the hand-written model, and it was lost when 65ac20e moved the class to overlays.
    • creator and creator_with_prefix now set the bucket relationship, as pyatlan's creators do. Without it bucket.objects came back empty.
    • Adds delete_handler (Atlas deleteHandler) to Entity, ReferenceableNested and the nested conversions. It comes from models#2093's renderer change. The delete tests read "SOFT" from the delete response, so the field really is in the payload.
    • Only the target files are regenerated here: entity, referenceable, s3_object and the three FND-3846 types. The sync bot regenerates the rest after models merges.

Merge order

  1. This PR first.
  2. atlanhq/models#2093 second.

The sync bot regenerates using the overlays on main. Merged in this order, its sync PR only changes the S3Prefix pattern line.


🧩 Type of change

  • 🚀 New feature (non-breaking change that adds functionality)
  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • 🔄 Refactor (code change that neither fixes a bug nor adds a feature)
  • 💥 Breaking change (fix or feature that may break existing functionality)
  • 📦 Dependency upgrade/downgrade
  • 📚 Documentation updates

✅ How has this been tested?

  • Unit: tests_v9/unit/model/{s3_prefix,flow_control_operation,tag_attachment}_test.py have one parity test per app template, plus tests for invalid and blank inputs. uv run pytest tests_v9/unit gives 8336 passed, 5 skipped.
  • Integration, run locally against the test tenant (6 passed):
    • tests_v9/integration/s3_asset_test.py::test_s3_prefix: root and nested prefix, bucket and parent-prefix relationships.
    • tests_v9/integration/flow_control_operation_asset_test.py: Fabric notebook and Spark job, SageMaker job.
    • tests_v9/integration/tag_attachment_asset_test.py: all three qn_style shapes.
  • Generated code: regenerated from models#2093 with this branch's overlays, and the pre-commit hooks leave the generated files unchanged.

CI triage (run 39263757980)

  • Jobs for this PR's files are green:
    • all qa-checks-and-unit-tests and v9-qa-checks-and-unit-tests jobs;
    • flow_control_operation_asset_test.py and tag_attachment_asset_test.py;
    • s3_asset_test.py::test_s3_prefix, which passes inside the s3_asset_test.py job.
  • Failures that were already on main and are unrelated to this PR:
    • s3_asset_test.py now passes in full against the tenant (17 passed). See the S3Object fixes above.
    • The other 40+ integration failures appear on FND-3847's run as well.
  • Six failures appear only on this run, because FND-3847's run cancelled those jobs before they finished. The causes are unrelated to this PR:
    • test_client.py (v9 and aio): add_or_update_classifications is not defined on the v9 assets.
    • custom_metadata_test.py (v9 and aio): the tenant returned ATLAS-423 Failed to get the lock.
    • aio/test_index_search.py: IndexSearchRequest is not JSON serializable.
    • tests/integration/test_client.py: audit response validation.

📋 Checklist

  • My code follows the project’s style guidelines
  • I’ve performed a self-review of my code
  • I’ve added comments in tricky or complex areas
  • I’ve updated the documentation as needed
  • There are no new warnings from my changes
  • I’ve added tests to cover my changes
  • All new and existing tests pass locally

🤖 Generated with Claude Code

https://claude.ai/code/session_01Wc98jPXjP7pdqJoAh3vtfY

…ration

Shared types built by more than one app (FND-3846). Each creator reproduces
every app's existing qualifiedName, so no app changes identity:

- TagAttachment.creator(qn_style=...): tag_object (databricks),
  tag_attachments_object (dbt), object_tag (bridge). Passthrough type, so the
  creator is hand-added to tag_attachment.py; overlay kept for when passthrough
  types are generated.
- S3Prefix.creator: {bucket_qn}/{prefix}/ with trailing slash (s3, bridge);
  sets bucket relationship, parent prefix qualified name and hierarchy.
- FlowControlOperation.creator: {conn}/{scope}/{source_id} (fabric notebook,
  sagemaker) or {parent_qn}/jobs/{source_id} with flow_controlled_by (fabric
  Spark job).

Parity tests cover every app template. A strict xfail records that the
generated S3Prefix _QUALIFIED_NAME_PATTERN rejects the trailing slash until
qualifiedNameFormula is added in atlanhq/models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wc98jPXjP7pdqJoAh3vtfY
@linear

linear Bot commented Oct 14, 2026

Copy link
Copy Markdown

FND-3846

cmgrote and others added 3 commits October 14, 2026 12:46
Resolve s3_prefix.py / flow_control_operation.py by taking main's regenerated
class files and re-injecting the creator overlays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wc98jPXjP7pdqJoAh3vtfY
- s3_prefix.py regenerated from atlanhq/models#2093 with this branch's
  overlays: _QUALIFIED_NAME_PATTERN is now ^.+/[^/]+/$, so creator output
  passes validate(for_creation=True); the strict xfail becomes a real test.
  flow_control_operation.py and tag_attachment.py regenerate byte-identical.
- Integration tests: S3Prefix (root + nested, parent relationship),
  FlowControlOperation (Fabric notebook + Spark job, SageMaker job),
  TagAttachment (all three qn_style shapes).
- AtlanConnectorType.FABRIC ("fabric", bi) and SAGEMAKER ("sagemaker", SaaS),
  matching the connector values in atlan-fabric-app / atlan-sagemaker-app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wc98jPXjP7pdqJoAh3vtfY
@cmgrote cmgrote added the run_pyatlan_v9_integration_tests To trigger pyatlan v9 (msgspec-based) integration tests suite (name the workflow checks) label Oct 19, 2026
cmgrote and others added 2 commits October 19, 2026 15:54
…ay migration

76c5b17 added `create_with_prefix = creator_with_prefix` to the hand-written
s3_object.py; 65ac20e moved the class to Pkl generation + overlays without
carrying the alias into _overlays/s3_object.py, so it disappeared and
tests_v9/integration/s3_asset_test.py (which uses it) has errored since.

Re-add it in the overlay (same alias style as AirflowDag.create), regenerated
from models#2093: the diff in s3_object.py is exactly the alias.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wc98jPXjP7pdqJoAh3vtfY
…andler

- S3Object.creator / creator_with_prefix now set
  bucket=RelatedS3Bucket(qualified_name=s3_bucket_qualified_name), matching
  pyatlan's creators; without it bucket.objects came back empty.
- delete_handler (Atlas deleteHandler) on Entity, ReferenceableNested and the
  nested conversions, regenerated from models#2093's renderer change. Only the
  target files are regenerated here (entity, referenceable, s3_object and the
  three FND-3846 types); the sync bot regenerates the rest after models merges.
- s3_asset_test.py: v9 returns relationships as RelatedS3Object, so assert that
  type. The delete tests now pass and confirm deleteHandler is in the payload.
  The file passes in full against a tenant (17 passed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wc98jPXjP7pdqJoAh3vtfY
@cmgrote cmgrote closed this Oct 20, 2026
@cmgrote cmgrote reopened this Oct 20, 2026
@cmgrote cmgrote closed this Oct 20, 2026
@cmgrote cmgrote reopened this Oct 20, 2026
@cmgrote
cmgrote merged commit b0680d1 into main Oct 20, 2026
90 of 143 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run_pyatlan_v9_integration_tests To trigger pyatlan v9 (msgspec-based) integration tests suite (name the workflow checks)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant