[johnpaulalex]

Description of changes

Add FTorch hooks at various places to enable FTorch inference and all necessary backing infra.

Relies on FTorch_Interface PR ESCOMP/FTorch_interface#2

Overview (Execution sequencing)

Listed in execution order, with the deltas from this PR's commits called out:

0. Human user prep:

• Documentation (Commit 3): Read the new Running-with-FTorch.rst documentation in the User's Guide for build and execution instructions.
• Model Export (Commit 2): Call the export library or CLI tool (./tools/ftorch/export_torchscript) to serialize trained PyTorch models to TorchScript (.pt) format.

1a. Source Code Initialization (Pre-build)

• Submodule Population (Commit 1): In CTSM .gitmodules, defines the libraries/FTorch submodule pointing at ESCOMP/FTorch_interface (fxrequired = ToplevelRequired).

1b. Setup & Configuration (create_test / case.setup)

• Parse Build Flags: Reads default USE_FTORCH=FALSE from the CIME environment XMLs (cime/CIME/build.py).
• Test Definition (New test definitions added to CTSM's testlist_clm.xml in Commit 4): Two system tests (SMS_D_Ld1.f10_f10_mg37.I2000Clm60Sp and SMS_D_Ld1_Mmpi-serial.1x1_brazil.I2000Clm60SpRsGs with testmods="clm/default--clm/nofireemis--clm/ftorch_demo") added in cime_config/testdefs/testlist_clm.xml for both intel and gnu on Derecho. They compile with USE_FTORCH=TRUE and execute runtime FTorch inference during CTSM initialization: one tests multi-process MPI on a global 10x15 grid (SMS_D_Ld1.f10_f10_mg37.I2000Clm60Sp), and the second tests single-point mpi-serial execution on 1x1_brazil (SMS_D_Ld1_Mmpi-serial.1x1_brazil.I2000Clm60SpRsGs, representing the LEAP/M2LInES single-point workflow).
• Model Staging (Testmods and shell_commands added in Commit 4): CIME processes clm/ftorch_demo, whose include_user_mods automatically pulls in ../ftorch (clm/ftorch/shell_commands, which sets ./xmlchange USE_FTORCH=TRUE), while clm/ftorch_demo/shell_commands stages the pre-generated TorchScript model (constant_model.pt) into $RUNDIR and clm/ftorch_demo/user_nl_clm enables use_ftorch_demo = .true..

2. Compilation (case.build)

• Build Hook Invocation: cime/CIME/build.py invokes $SRCROOT/libraries/FTorch/buildlib, and cime/CIME/Tools/Makefile passes -DUSE_FTORCH and links -lftorch / -ltorch_cpu when USE_FTORCH=TRUE (or links only the libftorch_wrapper stub when USE_FTORCH=FALSE).
• Library Compilation (companion ESCOMP/FTorch_interface PR):
  1. When USE_FTORCH=FALSE (default): libraries/FTorch/buildlib skips checking TORCH_DIR and compiles only FTorch_cesm_interface.F90 into libftorch_wrapper using the #else stub (which calls shr_abort_abort if invoked at runtime).
  2. When USE_FTORCH=TRUE: libraries/FTorch/buildlib validates that the C++ inference TORCH_DIR exists, compiles the FTorch Fortran-to-C++ bridge and FTorch_cesm_interface.F90 (libftorch_wrapper), and places ftorch.mod in ${CMAKE_BINARY_DIR}/modules to match INCLDIR += -I$(FTORCH_DIR)/modules in cime/CIME/Tools/Makefile.
  3. Cray Dynamic Linking: The FTorch_interface library PR enables shared libraries on Cray, allowing ftorch and ftorch_wrapper to build cleanly as shared libraries on Cray for both USE_FTORCH=TRUE and USE_FTORCH=FALSE. buildlib runs FTMakefile unconditionally so USE_FTORCH changes are re-evaluated by CMake.

3. Runtime Execution (mpiexec cesm.exe)

• Dynamic Linking: At job launch, the OS dynamic loader resolves libtorch_cpu.so and libftorch via the rpath ($(TORCH_DIR)/lib and $(FTORCH_DIR)/src) embedded into cesm.exe at link time.
• Model Initialization (Commit 4): Gated at runtime by if (use_ftorch_demo) on all MPI ranks (with zero #ifdef USE_FTORCH in clm_initializeMod.F90, matching the CESM/CAM convention, and gating only write(iulog,*) log messages on masterproc), clm_initializeMod.F90 calls torch_model_load and maps Fortran arrays to PyTorch tensors via torch_tensor_from_array.
• Inference & Teardown (Commit 4): Gated as above across all MPI ranks, executes the neural network using torch_model_forward and deallocates memory via torch_delete.

Specific notes

Contributors other than yourself, if any:

• (Replace this text and add more list items as needed)

CTSM issues resolved or otherwise addressed, if any:

• Resolves ESCOMP/CTSM#3228
• Resolves ESCOMP/CTSM#3215

If answers are expected to change, describe (delete this line otherwise):

• B4B: The FTorch demo inference logic merely populates temporary variables in the clm_initialize block and logs status output when use_ftorch_demo is .true.. It does not mutate any prognostic physics state.

Any user interface changes (namelist or namelist defaults changes)?

• Added namelist boolean: use_ftorch_demo (default: .false.) in namelist_definition_ctsm.xml.

Testing planned or performed, if any:

• Verified Python unit tests in python/ctsm/test/test_unit_export_torchscript.py pass cleanly both without PyTorch (3/3 on local Mac) and with pytorch-cpu in ctsm_pylib (3/3 on Derecho).
• Verified standard CTSM Fortran pFUnit suite (python3 cime/scripts/fortran_unit_testing/run_tests.py --test-spec-dir src) builds and passes cleanly (59/59 PASS).
• Verified all $2 \times 2$ (USE_FTORCH, use_ftorch_demo) system test permutations on NCAR Derecho (-t ftorch_perm_1):
  • 1a (USE_FTORCH=TRUE, use_ftorch_demo=.true., Intel MPI): SMS_D_Ld1.f10_f10_mg37.I2000Clm60Sp.derecho_intel.clm-default--clm-nofireemis--clm-ftorch_demo (PASS; FTORCH INFERENCE SUCCESSFUL: output = 1.00000000000000 in lnd.log).
  • 1b (USE_FTORCH=TRUE, use_ftorch_demo=.true., GNU mpi-serial): SMS_D_Ld1_Mmpi-serial.1x1_brazil.I2000Clm60SpRsGs.derecho_gnu.clm-default--clm-nofireemis--clm-ftorch_demo (PASS; FTORCH INFERENCE SUCCESSFUL: output = 1.00000000000000 in lnd.log).
  • 2 (USE_FTORCH=TRUE, use_ftorch_demo=.false., Intel MPI): SMS_D_Ld1.f10_f10_mg37.I2000Clm60Sp.derecho_intel.clm-default--clm-nofireemis--clm-ftorch (PASS; clean no-op with zero FTORCH lines in lnd.log).
  • 3 (USE_FTORCH=FALSE, use_ftorch_demo=.false., GNU mpi-serial): SMS_D_Ld1_Mmpi-serial.1x1_brazil.I2000Clm60SpRsGs.derecho_gnu.clm-default--clm-nofireemis (PASS; compiles and links without #ifdef USE_FTORCH against FTorch_cesm_interface stub).
  • 4 (USE_FTORCH=FALSE, use_ftorch_demo=.true., GNU mpi-serial): SMS_D_Ld1_Mmpi-serial.1x1_brazil.I2000Clm60SpRsGs.derecho_gnu.clm-default--clm-nofireemis--clm-ftorch_demo_only (RUN aborts as expected with ERROR: Using FTorch Interface without USE_FTORCH=TRUE).
• Verified negative control on NCAR Derecho (USE_FTORCH=FALSE, default): SMS_D_Ld5.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.interim_restart builds and runs uninterrupted without FTorch (PASS).
• Validated system test with latest PR 1 (fab11fa) and PR 2 (212bda8d0) on NCAR Derecho (Task T15, output root /glade/derecho/scratch/jpalex/tests_ftorch_t15): SMS_D_Ld1.f10_f10_mg37.I2000Clm60Sp.derecho_intel.clm-default--clm-nofireemis--clm-ftorch_demo (PASS; confirmed FTORCH INFERENCE SUCCESSFUL: output = 1.00000000000000).
• Verified full parallel test campaign on NCAR Derecho (scratch root /glade/derecho/scratch/jpalex/tests_ftorch_rerun): all standalone CTSM tests (SMS_D_Ld1...clm-ftorch_demo and generic SMS_D_Ld1...clm-ftorch) and coupled CESM checkout tests (SMS_D_Ld1...clm-ftorch_demo Intel MPI, Intel MPI USE_FTORCH=FALSE, and GNU mpi-serial USE_FTORCH=FALSE) passed.

Requirements before merge:

• I have followed the CTSM contribution guidelines.
• The code in this PR branch builds with no errors.
• The code in this PR branch runs with no errors. Briefly describe tested configuration(s): Derecho Intel/GNU aux_clm system tests and full $2 \times 2$ (USE_FTORCH, use_ftorch_demo) permutation matrix.
• This either (a) does not change answers, (b) it only changes answers at roundoff level, or (c) I have performed a scientific evaluation of the answer changes. Which?: (a)
• I have reviewed relevant parts of the CLM documentation Tech Note or User's Guide to determine if anything needs to be changed or added. If it does, describe: Added Running-with-FTorch.rst to the User's Guide under running-special-cases.
• This PR either (a) does not create a need to update the documentation or (b) includes required documentation updates (see guidelines for contributing documentation). Which?: (b)

Google Gemini Argon 4 was used to write the code and tests, followed by human-guided verification.

[github-actions]

❌ Docs build failed

Build logs

Cleaning documentation build directory...
Done.
Building documentation...
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:3: WARNING: Title overline too short.
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:39: WARNING: Title underline too short.
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:39: WARNING: Title underline too short.
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:61: WARNING: Title underline too short.
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:61: WARNING: Title underline too short.
Documentation build completed, but with problems that must be resolved.
Re-run with --verbose for full output.

[johnpaulalex]

@ekluzek Here's the other half of CTSM FTorch support (paired with ESCOMP/FTorch_interface#2).

[github-actions]

❌ Docs build failed

Build logs

Cleaning documentation build directory...
Done.
Building documentation...
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:3: WARNING: Title overline too short.
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:39: WARNING: Title underline too short.
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:39: WARNING: Title underline too short.
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:61: WARNING: Title underline too short.
/home/user/mounted_home/doc/source/users_guide/running-special-cases/Running-with-FTorch.rst:61: WARNING: Title underline too short.
Documentation build completed, but with problems that must be resolved.
Re-run with --verbose for full output.

[wwieder]

Thanks, @johnpaulalex. It looks like some of the documentation build tests failed. Are you able to correct these?

Who's been handling the ftorch feature request who would be able to review this PR & do testing? Marking as next to discuss at the meeting Thursday.

[johnpaulalex]

hey @wwieder , I pinged a couple people on the Ftorch_interface PR, it's only the second PR ever so I guess it's early days.

Docs build is all fixed.

[wwieder]

This isn't critical for the CLM6 release, so we decided to mark this with a later milestone, but reviewing the code isn't a priority right now.