[isc-dchui]

Description

Resolves #986

Adds a second packaging type. Instead of shipping source for the installer to compile, package-database copies the routines database of the namespace it runs in and ships that IRIS.DAT with the module's non-compiled files and metadata. Installing swaps the file in as the target namespace's routines database.

Package contents

<module>-<version>-database.tgz
├── IRIS.DAT              copy of the packaging namespace's routines DB
├── module.xml            manifest + <Packaging>database</Packaging>
│                         + <Checksum Algorithm="SHA-256"> of IRIS.DAT
│                         + <SystemRequirements Version="<major>.<minor>">
├── <root's files>        FileCopy sources, CPFs, wheels, at the paths the
│                         root's processors already read them from
├── deps/<dep>.xml        one IPM metadata export per transitive dependency
└── deps-files/<dep>/     that dependency's files; becomes its Root at install

The pinned SystemRequirements is an exact-match expression, so a package installs only on the IRIS major.minor that built it. Packaging replaces an author's own version range and says so.

Packaging

package-database <module> and publish-database <module> force the %IPM.Lifecycle.Database lifecycle class and run the Package or Publish phase. package-source and package-studio-project are added as explicit names for the existing source and studio-project packaging; the latter is deprecated from the start.

Packaging copies what the namespace holds; it does not build. The module and its dependencies must already be loaded there. The Compile phase runs first, so a root module whose code is missing fails there rather than shipping without it. The only thing %Package loads itself is the root's test resources, under -include-tests.

%Package
   │
   ├─ 1  Validate       IPM is mapped in, not in the routines DB (the swap would replace it)
   │                    routines DB is not mirrored (dismounting breaks the mirror)
   │
   ├─ 2  Warn           unrunnable <Invoke>s, <Global> resources, dependency code missing,
   │                    other modules' code in the DB, test code already compiled in,
   │                    routines DB shared with another namespace, routines DB is also
   │                    the globals DB
   │
   ├─ 3  Load tests     -include-tests only: load + compile the root's Scope="test"/"verify"
   │                    resources and <UnitTest> directories
   │
   ├─ 4  Python deps    on by default: add each requirements.txt wheel as a <PythonWheel>
   │                    to the root's and each dependency's stored manifest
   │
   ├─ 5  Copy the DB    dismount routines DB → copy IRIS.DAT → remount
   │
   └─ 6  Assemble       checksum the staged IRIS.DAT, stage non-compiled files through
                        each processor, write module.xml and deps/, create the .tgz

Step 4 matches what package -export-python-deps does, except that it defaults on here. The staging directory is removed on every path.

Why packaging dismounts

A copy of a mounted IRIS.DAT reads only what is on disk, and recently written blocks sit in the buffer pool until the write daemon flushes them. On IRIS 2026.1, a class compiled seconds before the copy is missing from it. Packaging normally follows a load by seconds, so it would fail nondeterministically. No per-database flush exists. The instance-wide ones are worse than a short outage of a build namespace.

DismountDatabase doesn't refuse a database that is in use; other processes get <PROTECT> until the remount. SYS.Database is only visible from %SYS, and nothing on that path writes output, because output capture can't resolve its handler routine from %SYS.

Staging files: OnStageForDatabasePackage

Each resource processor stages the files it will read at install time through a new %IPM.ResourceProcessor.Abstract hook, OnStageForDatabasePackage(targetDir, .params) As %Status. FileCopy, CPF and PythonWheel override it. The default does nothing. A custom processor that reads files from the module root at install time must override it, or those files won't ship.

Registry

Database packages use the same OCI tag as a source package of that version. The com.intersystems.ipm.packaging=database manifest annotation is what tells them apart; an absent annotation means source. So a version holds either a source or a database package, not both. Base.%Publish takes the type from the lifecycle class's PACKAGING parameter, not the module's stored Packaging, which records how it was loaded.

Installation

install/update from a registry resolve only source packages unless -swap-db is passed, in which case the named module resolves as a database package. Dependencies always resolve as source, since a database package already carries their code. If only a database package exists, the error says to re-run with -swap-db. load <tgz> applies no filter and prompts before the swap unless -swap-db is passed.

A database install is a module with Packaging = database and an IRIS.DAT in its root. LoadNewModule skips LoadDependencies for it. %Reload does all the work, so %Validate, %Compile and %Activate are no-ops.

ValidateBeforeSwap                 failure: nothing changed, module record rolled back
PromptForSwapConfirmation          (skipped with -swap-db)
UnconfigureStaleModules
PerformDatabaseSwap                failure: original IRIS.DAT restored and remounted;
                                   stale modules stay unconfigured (warned)
═══════ commit ═══════             point of no return
RemoveStaleModuleMetadata
RegisterDependencyMetadata         loads deps/*.xml
apply <Mapping>s                   root and dependencies
ApplyDependencyResources           each dependency's phases, in dependency order
ReassertRootInitializeResources
HandleAllUpdateSteps               fresh install: seed all steps as already run

ValidateBeforeSwap orders its checks so the failure reported is the one worth acting on. It checks the namespace first (IPM mapped in, routines database neither mirrored nor shared), then the package's structure and mapping source databases, then that this is the only database package in the namespace, then orphans. The whole-file SHA-256 runs last, since it reads a potentially multi-gigabyte file.

The orphan check compares every other installed module with the outgoing and incoming dependency sets:

• Stale: a dependency the new version drops. Its code leaves with the old IRIS.DAT. It is unconfigured before the swap and its record is removed after it.
• Orphaned: a module in neither set with code in the routines database. Nothing would replace that code, so the install fails and names it. No flag bypasses this.

Why nothing is rolled back after the swap

The swap is a file rename, so it is the last step with a clean undo. LoadNewModule's transaction is committed as soon as it lands. Rolling back later would pair old module records with new code and couldn't undo CPF merges, installed wheels or copied files. A post-swap failure reports what to do and is fixed forward by re-running the same command from the .tgz or registry.

Mappings are applied after that commit because their Config.Map* locks are held until commit, and a dependency's CPF merge (iris merge) can't wait them out. Inside an outer transaction the commit only folds into it, so mappings are skipped with a warning.

Dependency phases

Dependency code arrives compiled, but nothing has run the dependencies' lifecycle, so their CPFs, wheels, copied files, web applications and <Invoke>s would be dropped. ApplyDependencyResources runs each transitive dependency's phases, dependencies before dependents, with its Root pointed at deps-files/<dep>/.

For this tree:

main-module          ← root, the package being installed
└── parent-dep
    ├── leaf-dep-A
    └── leaf-dep-B

Columns are modules, rows are time:

         │ Root                  │ leaf-dep-A     │ leaf-dep-B     │ parent-dep
─────────┼───────────────────────┼────────────────┼────────────────┼────────────────
PRE-SWAP │ Initialize        [1] │                │                │
─────────┴───────────────────────┴────────────────┴────────────────┴────────────────
                          PerformDatabaseSwap, commit
─────────┬───────────────────────┬────────────────┬────────────────┬────────────────
POST-    │ apply <Mapping>s [2]  │                │                │
SWAP     │                       │ Initialize     │                │
         │                       │ Validate       │                │
         │                       │ Activate       │                │
         │                       │ ApplyUpdate[3] │                │
         │                       │                │ Initialize     │
         │                       │                │ Validate       │
         │                       │                │ Activate       │
         │                       │                │ ApplyUpdate[3] │
         │                       │                │                │ Initialize
         │                       │                │                │ Validate
         │                       │                │                │ Activate
         │                       │                │                │ ApplyUpdate[3]
         │ ReassertInitialize[4] │                │                │
         │ HandleAllUpdateSteps  │                │                │

[1] The root's Initialize runs before the swap, against the old database. Dependencies' Initialize runs after, because their code is inside the incoming IRIS.DAT.

[2] For the root and every dependency at once, before any dependency phase runs.

[3] Only on update. A fresh install instead seeds update steps as already run (each dependency's in its Activate, the root's in HandleAllUpdateSteps), so a later update runs only newer steps.

[4] Re-applies the root's Initialize resources, as described below.

These come from the canonical install chain minus Reload (no source ships), Compile (already in IRIS.DAT) and * (the refresh checkpoint for the module being installed). Deriving them means a phase added to the chain later isn't silently skipped for dependencies.

The root's Initialize ran before the swap, so a dependency's Initialize-phase CPF merge or wheel would override the root's. ReassertRootInitializeResources re-applies the root's afterwards, but only for processor types a dependency actually used, since each CPF merge costs a subprocess.

Changes outside the new lifecycle

• Storage.Module.%OnValidateObject derives the lifecycle class from <Packaging> through Base.GetBaseClassForPackaging instead of rejecting a mismatch with the default class.
• Storage.Module.GetDependencyModules/GetDependencyNames return installed transitive dependencies in dependency order from local records only. They reuse ConstructInvertedDependencyGraph and GetFlatDependencyListFromInvertedDependencyGraph. The sort gains a fallback for cycles, which installed records can hold (a Scope="test" dependency back on the root), instead of looping forever.
• Base.%Reload's mapping code is extracted into ApplyModuleMappings so the database install can apply them.
• CPF and FileCopy each get a relative-path helper shared by export, install and staging, so packaging stages the files the install reads.
• %IPM.Utils.Module's database getters use Config.*.Get and throw instead of dereferencing a possibly null Open. GetDatabaseDirectory normalizes, since the directory, not the name, identifies the file being swapped.
• ExportPythonDependencies warns about a missing wheels directory only when the module has a requirements.txt.
• ORAS Pull hashes the downloaded file with ComputeSHA256Hex and throws on a failed LinkToFile.
• Default.Document.OnPhase drops a block that marked Generated resources as handled. No caller reads that flag in the phases where it was set.
• %IPM.Test.Utils.CreateNamespace takes pEnableEnsemble (default 0) instead of enabling interoperability in every test namespace, which shortens the integration suite. When it is set, ^IRIS.Msg is mapped to the namespace's own database so IRIS for Health's read-only HSLIB copy isn't used.

Not handled

• Mirrored routines databases (rejected).
• <Global> data and other content that lives in the globals database (warned).
• A dependency <CPF> that creates the database its own <Mapping> uses: dependency mappings are applied before dependency CPFs run.
• Only ORAS registries understand database packages; other repository types ignore the packaging filter.

Testing

Test.PM.Integration.DatabasePackaging covers:

• packaging and installing, with and without dependencies and test resources
• validation rejections: checksum, structure, bad mapping source, orphaned modules, a second database package
• upgrades from source and between database versions with a reshaped dependency tree, including root and dependency update steps
• Python wheels, <Invoke> timing, dependency resources applied after the swap, and namespace-relative web applications
• mappings plus a post-swap failure and its re-run
• ORAS publish and resolution
• the dismount-copy-remount
• each packaging warning

Two lazily created shared packaging namespaces avoid a create/teardown per test. Tests that mutate their packaging namespace get their own. Test.PM.Unit.LifecycleDatabase covers lifecycle derivation, dependency ordering, the dependency phase exclusions, the staging hook and the SystemRequirements transform.

Checklist

• This branch has the latest changes from the main branch rebased or merged.
• Changelog entry added.
• Unit (zpm test -only) and integration tests (zpm verify -only) pass.
• Style matches the style guide in the contributing guide.
• Documentation has been/will be updated
  • Source controlled docs, e.g. README.md, should be included in this PR and Wiki changes should be made after this PR is merged (add an extra issue for this if needed)
• Pull request correctly renders in the "Preview" tab.

[isc-jili]

Looks great @isc-dchui ! I left some comments!

[isc-kiyer]

@isc-dchui few small notes. I love the description of the MR! Could you add some more information for each of the "Key Details"? In particular, for the following:

• Can you explain why generated resources need special treatment? When packaging, wouldn't they be in the routine db already?
• What is the reason for exporting the module manifests and reimporting them? Could that not be accomplished by setting up a global mapping/copying over the IPM globals from the globals db to the routine db during packaging? Then its one less step at deployment time.

[isc-jlechtne]

What is the metadata value if the tag doesn't exist in the tag list? So that it's not equal to ""

[isc-dchui]

GetAllTags() won't return it so that tag won't be part of the while loop.

  set allTagsString = ..GetAllTags(..Location, name, "", client)
  set allVersionsList = ..AggregatePlatformVersions($listfromstring(allTagsString, ", "), .aggregatedPlatformVersion)
  set pointer = 0
  while $listnext(allVersionsList,pointer,moduleVersion) {
     ...
  }

[isc-dchui]

@isc-dchui few small notes. I love the description of the MR! Could you add some more information for each of the "Key Details"? In particular, for the following:

• Can you explain why generated resources need special treatment? When packaging, wouldn't they be in the routine db already?
• What is the reason for exporting the module manifests and reimporting them? Could that not be accomplished by setting up a global mapping/copying over the IPM globals from the globals db to the routine db during packaging? Then its one less step at deployment time.

@isc-kiyer Updated the description with a little more organization clarity! To explicitly answer these,

1. Standard generated resources like BPL and DTL are fine without special handling, but anything that is special and generated outside of the standard lifecycle phases won't be picked up. So this is just redundancy for that. Maybe it's a case of YAGNI though and overkill.
2. Yeah, but exporting/re-importing turns out to be simpler than setting up and managing the global mappings.

[isc-jili]

Looks good! @isc-dchui One more Q: are all the resources as part of transitive dependencies supposed to be packaged up too? Since ExportGeneratedResources() CopyNonCompiledResources() and ExportPythonDependencies() seem to be invoked on the root module only
A test case along those lines that would be nice also would be having a dependency contain a Generated="1" resource or a requirements.txt

[isc-dchui]

Fixed the issue where, on install, dependencies wouldn't run through their lifecycle phases. Details are in a new section of the description.