Content from Introduction

Last updated on 2025-06-24 | Edit this page

Overview

Questions

  • How can I run an HPC workflow reproducibly?
  • What is the easiest way to create a scaling experiment for an HPC application?
  • How do I identify the optimal runtime configuration for an application on an HPC system?

Objectives

  • Enumerate the benefits of workflow automation.
  • Describe potential use cases for HPC workflow automation using JUBE.

When applying for computing time on HPC systems, applicants are often required to provide measurements on different scales of parallelism. Furthermore, preparing performance measurements often involves an application-specific workflow as well as platform-specific configurations. The objective of this lesson is to enable users of HPC systems to run performance measurements with minimal intervention with high reproducibility, using the Jülich Benchemarking Environment (JUBE) [1].

It is important to understand that JUBE, while emerged as a benchmarking system, actually is a workflow management system, where application and system benchmarking is one of its applications. Further use cases may include running multiple HPC workflows as integration tests during application development and defining reproducible execution workflows for data generation for publications.

Reproducibility Note

JUBE is an automation tool. That means, it does not intrinsicly make your workflows reproducible, but it lets you automate all actions that can make your workflow reproducible.

The key benefits of using JUBE are:

  • It enables systematic and reproducible benchmarks.
  • It provides a flexible, script-based framework to set up tasks and control various aspects of the execution.
  • It allows for custom workflows that can adapt to different HPC platforms.
  • It provides a powerful tagging systems to influence execution of a given workflow.
  • It automatically isolates your workflow steps so concurrent tasks don’t overwrite or re-use files in unintentionally shared execution environments.
  • It enables efficient parameter space exploration.
  • It allows to easily retrieve strings from runtime output and create tables and CSV files.

Discussion

Consider your own workflows on an HPC system. What individual steps are involved? Please discuss this in groups.

Steps may include:

  • Loading and preparing the software environment
  • Building a software package
  • Creating a runtime configuration for your application
  • Creating a scheduler job script
  • Submitting a job to the scheduler
  • Extracting strings from application output.

JUBE is written and maintained by the Jülich Supercomputing Centre at Forschungszentrum Jülich in Germany. It is written in Python 3, but is backwards-compatible to Python 2.x. The user can write configurations in XML or YAML syntax. It is freely available for download and also available via HPC software package managers, such as Spack and EasyBuild.

Challenge

Create an empty workspace for all subsequent challenges in this lesson.

It is best to start with an empty directory. You are free to choose any name, but for the sake of this lesson, we will use jube-workspace/.

SH 

$ mkdir jube-workspace/
$ cd jube-workspace/
$ pwd

OUTPUT 

/home/user/jube-workspace

Key Points

  • Workflow automation aids in getting reproducible results.
  • JUBE enables execution and management of complex workflows on HPC systems.
  • JUBE simplifies exploration of a large parameter space of measurements.
  • JUBE automatically isolates individual workpackages of a run in separate directories steps to avoid individual concurrent workpackages to overwrite or unintentional reuse of intermediate data.
  • JUBE does not intrinsically create fully reproducible workflows, but the user has to manually record any parameters that make the workflow reproducible.

Content from An example application

Last updated on 2025-06-24 | Edit this page

Overview

Questions

  • What is a typical HPC application?
  • What is a typical workflow scenario with JUBE?

Objectives

  • Name common steps in an HPC execution workflow.
  • Name an example for each of the potential workflow steps.

HPC systems provide large computational resources to support researchers with their computational projects. Such applications come in many different forms and sizes. Some need to be compiled, before they can be used. Some are pre-installed on the HPC system.

This lesson uses the GROMACS software as an example HPC code. GROMACS is a free and open-source software suite for high-performance molecular dynamics and output analysis.

Preparing the application

Downloading GROMACS

While many HPC systems provide GROMACS installations for different versions, this lesson will use Eventually, downloading GROMACS will be part of the overall HPC workflow, but you can download the 2024.5 version of GROMACS manually from the download page of GROMACS.

SH 

$ mkdir sources
$ cd sources
$ wget https://ftp.gromacs.org/gromacs/gromacs-2024.5.tar.gz

Building GROMACS

GROMACS is a C++ application that uses the CMake build system generator automatic generation of a Makefile, enabling easy configuration and building of the software.

As a C++ application, it needs to be compiled into a binary before it can be executed. This is called building the application.

CMake enables so-called “out-of-source” builds, which means all files generated during the build process stay separate from the source files of GROMACS. For this, we generate a build directory in the

SH 

$ cd sources
$ tar xzf gromacs-2024.5.tar.gz
$ cd ..
$ cmake -S sources/gromacs-2024.5/ -B build_gromacs -DGMX_MPI=ON

OUTPUT 

-- The C compiler identification is IntelLLVM 2024.2.0
-- The CXX compiler identification is IntelLLVM 2024.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /cvmfs/software.hpc.rwth.de/Linux/RH9/x86_64/intel/sapphirerapids/software/intel-compilers/2024.2.0/compiler/2024.2/bin/icx - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /cvmfs/software.hpc.rwth.de/Linux/RH9/x86_64/intel/sapphirerapids/software/intel-compilers/2024.2.0/compiler/2024.2/bin/icpx - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CXX17_COMPILES_SIMPLY
-- Performing Test CXX17_COMPILES_SIMPLY - Success
-- Found Python3: /home/mh269604/.miniforge3/bin/python3.10 (found suitable version "3.10.14", minimum required is "3.7") found components: Interpreter Development Development.Module Development.Embed
...

This first step is called the configuration step. During this step CMake checks different parameters of the build environment and creates build systems files accordingly. This configuration step reveals information that may prove important for reproducibility later on, such as the compiler version used for the build in the output above.

Reproducibility Note

While information about the build process is extremely important for reproducible performance measurements (i.e., benchmarking), it may also prove important for reproducible results when identifying sources for non-bitidentical results. Results are called bitidentical when two different executions of an application produce results that have the exact same bit pattern.

After the configuration step, CMake can be used to build the application binary. One of the advantages of using CMake is its ability to identify build dependencies and enabling the parallel compilation of independent source files, which can significantly reduce the overall build time.

SH 

$ cmake --build build_gromacs --parallel 12

OUTPUT 

[  0%] Generating release version information
[  0%] Building CXX object _deps/muparser-build/CMakeFiles/muparser.dir/src/muParser.cpp.o
[  0%] Building CXX object src/programs/CMakeFiles/mdrun_objlib.dir/mdrun/mdrun.cpp.o
[  0%] Building C object src/gromacs/CMakeFiles/tng_io_zlib.dir/__/external/tng_io/external/zlib/adler32.c.o
[  0%] Building CXX object src/gromacs/options/CMakeFiles/options.dir/abstractoption.cpp.o
[  0%] Building CXX object src/gromacs/energyanalysis/CMakeFiles/energyanalysis.dir/energyterm.cpp.o
[  0%] Building CXX object src/gromacs/CMakeFiles/colvars_objlib.dir/__/external/colvars/colvar.cpp.o
[  0%] Building CXX object src/gromacs/linearalgebra/CMakeFiles/linearalgebra.dir/eigensolver.cpp.o
[  0%] Building CXX object src/gromacs/CMakeFiles/lmfit_objlib.dir/__/external/lmfit/lmmin.cpp.o
[  0%] Building C object src/gromacs/CMakeFiles/tng_io_obj.dir/__/external/tng_io/src/compression/fixpoint.c.o
[  1%] Built target release-version-info
[  4%] Building C object src/gromacs/CMakeFiles/tng_io_obj.dir/__/external/tng_io/src/compression/huffman.c.o
[  4%] Built target thread_mpi
[  4%] Building C object src/gromacs/CMakeFiles/tng_io_obj.dir/__/external/tng_io/src/compression/huffmem.c.o
[  4%] Building C object src/gromacs/CMakeFiles/tng_io_zlib.dir/__/external/tng_io/external/zlib/compress.c.o
[  4%] Building CXX object src/programs/CMakeFiles/gmx_objlib.dir/gmx.cpp.o
...
[ 98%] Building CXX object api/gmxapi/CMakeFiles/gmxapi.dir/cpp/mdmodule.cpp.o
[ 98%] Building CXX object api/gmxapi/CMakeFiles/gmxapi.dir/cpp/mdsignals.cpp.o
[ 98%] Built target gmx
[ 98%] Building CXX object api/gmxapi/CMakeFiles/gmxapi.dir/cpp/session.cpp.o
[ 98%] Building CXX object api/gmxapi/CMakeFiles/gmxapi.dir/cpp/status.cpp.o
[ 98%] Building CXX object api/gmxapi/CMakeFiles/gmxapi.dir/cpp/system.cpp.o
[ 98%] Building CXX object api/gmxapi/CMakeFiles/gmxapi.dir/cpp/version.cpp.o
[ 98%] Building CXX object api/gmxapi/CMakeFiles/gmxapi.dir/cpp/workflow.cpp.o
[ 98%] Building CXX object api/gmxapi/CMakeFiles/gmxapi.dir/cpp/tpr.cpp.o
[ 98%] Building CXX object api/nblib/CMakeFiles/nblib.dir/particlesequencer.cpp.o
[ 98%] Building CXX object api/nblib/CMakeFiles/nblib.dir/particletype.cpp.o
[ 98%] Building CXX object api/nblib/CMakeFiles/nblib.dir/simulationstate.cpp.o
[ 98%] Building CXX object api/nblib/CMakeFiles/nblib.dir/topologyhelpers.cpp.o
[ 98%] Building CXX object api/nblib/CMakeFiles/nblib.dir/topology.cpp.o
[ 98%] Building CXX object api/nblib/CMakeFiles/nblib.dir/tpr.cpp.o
[100%] Building CXX object api/nblib/CMakeFiles/nblib.dir/virials.cpp.o
[100%] Building CXX object api/nblib/CMakeFiles/nblib.dir/listed_forces/calculator.cpp.o
[100%] Building CXX object api/nblib/CMakeFiles/nblib.dir/listed_forces/transformations.cpp.o
[100%] Building CXX object api/nblib/CMakeFiles/nblib.dir/listed_forces/conversions.cpp.o
[100%] Building CXX object api/nblib/CMakeFiles/nblib.dir/listed_forces/convertGmxToNblib.cpp.o
[100%] Building CXX object api/nblib/CMakeFiles/nblib.dir/util/setup.cpp.o
[100%] Linking CXX shared library ../../lib/libgmxapi.so
[100%] Built target gmxapi
[100%] Linking CXX shared library ../../lib/libnblib_gmx.so
[100%] Built target nblib
[100%] Building CXX object api/nblib/samples/CMakeFiles/argon-forces-integration.dir/argon-forces-integration.cpp.o
[100%] Building CXX object api/nblib/samples/CMakeFiles/methane-water-integration.dir/methane-water-integration.cpp.o
[100%] Linking CXX executable ../../../bin/argon-forces-integration
[100%] Built target argon-forces-integration
[100%] Linking CXX executable ../../../bin/methane-water-integration
[100%] Built target methane-water-integration

The output provides information about which parts of GROMACS are currently compiled and the overall progress of the build process.

Installing GROMACS

CMake can also help with the proper installation of the software into a specifc path. You can specify the target path for the installation via the --prefix commandline option.

SH 

$ cmake --install build_gromacs --prefix install/gromacs-2024.5

OUTPUT 

-- Install configuration: "Release"
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/COPYING
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/README.tutor
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/README_FreeEnergyModifications.txt
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/table6-8.xvg
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/ffoplsaa.itp
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/forcefield.doc
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/aminoacids.rtp
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/tip5p.itp
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/rna.r2b
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/ffbonded.itp
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/aminoacids.r2b
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/aminoacids.c.tdb
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/tip3p.itp
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/atomtypes.atp
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/aminoacids.hdb
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/dna.r2b
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/tip4pew.itp
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/gromacs/top/amber99sb.ff/spce.itp
...
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-spatial.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-nmtraj.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-sorient.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-rotmat.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-rotacf.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-cluster.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-genion.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-lie.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-tune_pme.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-hbond.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-dielectric.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-tcaf.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-nmr.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-gyrate-legacy.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-genconf.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-velacc.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-pairdist.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-trjorder.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-insert-molecules.1
-- Installing: /home/mh269604/jube-workspace/install/gromacs-2024.5/share/man/man1/gmx-mdrun.1

After adding the bin/ subdirectory of the install path to your PATH environment variable, GROMACS is ready to be used.

SH 

$ export PATH=install/gromacs-2024.5/bin:$PATH
$ which gmx

OUTPUT 

install/gromacs-2024.5/bin/gmx

Preparing the input

Some application may need specific preparation of its input. For example, matrices may need to be pre-conditioned, or the application domain needs to be partitioned prior to the application execution.

The provided input package for this lesson has three different inputs.

System Sidelength of simulation box No. of atoms Simulated time
MD_5NM_WATER.tpr 5 nm 12165 1 ns
MD_10NM_WATER.tpr 10 nm 98319 1 ns
MD_15NM_WATER.tpr 15 nm 325995 1 ns

The simulation domain is a three-dimensional box with a given side length. This means its size grows cubicly with its side length. To accomodate for a similar amount of work across the three inputs the number of atoms in each input roughly corresponds to the overall volume of the simulation box. This means, the 10 nm input contains roughly 8 (2x2x2) times as many atoms as the 5 nm input, and the 15 nm input contains roughly 27 (3x3x3) times as many atoms as the 5nm input.

The size of the simulation box of the differnet inputs
The size of the simulation box of the different inputs,

For this lesson the inputs do not need further pre-processing.

Discussion

What other types of preparation can you think of? Which prior step before execution does your application need?

  • Copying an input to a specific directory
  • (Re-)naming input files to specific names
  • …

Key Points

  • HPC applications may need to be downloaded and built before use
  • HPC applications may have additional dependencies.
  • HPC applications often do not have an automatic dependency management system (like pip or conda).
  • HPC applications may need serial preparation of inputs
  • steps of individual HPC workflows may occur at different levels of parallelism

Content from Working with JUBE

Last updated on 2025-06-24 | Edit this page

Overview

Questions

  • Which actions need to be taken when running JUBE?
  • Which commands correspond to which state in a JUBE workflow?

Objectives

  • List the basic commands for running a JUBE workflow.
  • Specify the purpose of each top-level command.
  • Name the different states a JUBE workflow can have.

JUBE workflows are started with the command jube run <jube-spec>. The configuration file can be either an XML or a YAML file.

A minimal configuration in XML contains the tags <jube> and <benchmark>.

This minimal configuration does not include any definition of workflow steps and the output will look similar to the following:

OUTPUT 

######################################################################
# benchmark: hello_world
# id: 0
#
# A simple workflow example
######################################################################

Running workpackages (#=done, 0=wait, E=error):
............................................................ (  0/  0)

  | stepname | all | open | wait | error | done |
  |----------|-----|------|------|-------|------|

>>>> Benchmark information and further useful commands:
>>>>       id: 0
>>>>   handle: episodes/files/bench_run
>>>>      dir: episodes/files/bench_run/000000
>>>>  analyse: jube analyse episodes/files/bench_run --id 0
>>>>   result: jube result episodes/files/bench_run --id 0
>>>>     info: jube info episodes/files/bench_run --id 0
>>>>      log: jube log episodes/files/bench_run --id 0
######################################################################

Query the status of a workflow

To check whether a run with a given id if finished or not, you can use the jube status command. Oviously, the workflow you just executed is finished.

SH 

$ jube status --id 0

OUTPUT 

FINISHED

Defining a workflow

With no steps defined, the workflow will immediatley complete without specific actions executed, other than file JUBE-internal handling of the run and creation of corresponding directories and files.

Looking into the benchmark output directory created, we will see somethin that will look similar to the following stucture:

SH 

$ tree bench_run        # the tree command provides a nice tree for the directory structure

OUTPUT 

bench_run               # the given outpath
|
+- 000000               # the benchmark id
   |
   +- analyse.log       # accumulated log information for analyse command
   +- analyse.xml       # the retrieved analyse information
   +- configuration.xml # the stored benchmark configuration
   +- parse.log         # log information
   +- result.log        # accumulated log information for result command
   +- run.log           # accumulated log information for run command
   +- timestamps        # accumulated of change timestamps
   +- workpackages.xml  # workpackage information

Log Files

The files ending in .log collect details on the execution of the corresponding commands. Users will only be interested in evaluating these files when debugging a workflow.

Instead of opening the file directly (which would be possible), you can also use the jube log command to show the log of a given command.

SH 

$ jube log --command run --id <id>

OUTPUT 

tbd.

Configuration Files

The files ending in .xml save configuration information of the workflow. The file configuration.xml holds a copy of the overall configuration used to start the workflow. The file workpackages.xml holds information about individual instances of parameters in a specific step of the workflow.

Before running a workflow, at least a single step needs to be defined for the workflow. A step is a collection of tasks (defined by individual do directives) that share a specific instance of parameter values. By allowing parameters to have different values in different instances, JUBE will automatically save a snapshot of the collection of all parameter values across all instances.

Defining parameters

Next to defining the steps of a workflow, parameters can both represent values picked up during the workflow and influence how a workflow is executed. To make it easier to manage multiple parameters, you need to organize them in parameter sets. Let’s define a minimal set of parameters for our initial Hello World example.

Naming conventions

Several entities defined in the specification are later used by its name in the step definitions. To make it easier to understand more complex configurations, it is good practice to encode the type of the entity into its name, for example as a suffix, such as _pset, _files, _sub, _pat. As steps themselves are the highest level entity and never used by other steps, you donot need to name them with a special suffix.

Here are some suggestions for suffixes according to their type:

Entity Suffix
Fileset _files
Parameterset _pset
Patterset _pat
Substituteset _sub

Defining workflow steps

The only reason to define parameters in the first place is to use them in the execution of actions in your workflow steps. To do this, the keyword use, referencing the name of a parameter set, is needed in the step definintion. Actions are defined using the do keyword and should be standard shell commands (including arguments). Parameters can be referenced using the $ prefix to its name and are replaced by their respective value prior to the execution

Defining dependencies

A workflow only starts to make sense when you have more than one step and there are dependencies between steps. In JUBE you can define these dependencies with the depend keyword.

Steps in a workflow that depend on prior steps in the workflow are only executed once the corresponding dependent step has executed successfully.

Running a workflow

A JUBE workflow is started by the run command, putting this workflow into the running state. For each defined step, individual workpackages are created with a specific instance of parameter values. Any defined tasks When no asynchronous tasks are defined (See Running an application) the steps will be processes until all workpackages are completed.

The JUBE workflow with `run`, `continue`, `analysis`, and `result` commands.
The JUBE workflow with run, continue, analysis, and result commands.

A JUBE workflow is initiated with the run command. At that time, the first step will create workpackages based on the use parametersets.

Key Points

  • The different states of a JUBE workflow can be running and completed.
  • A workflow is completed if all steps of the workflow are completed (even if some steps may have failed).
  • A JUBE workflow is started with jube run.
  • JUBE executes the workflow until the end, or until an asynchronous task is discovered.
  • If JUBE exits with completion of asynchronous tasks still pending, the user needs to call jube continue to check for their completion and further running of dependent workflow steps until overall completion is achieved.
  • The user can analyze output and print results at any time.
  • Comma-separated values in parameter definition are tokenized and create individual workpackages. Using multiple comma-separated values for parameters enables easy creation of parameter spaces.

Content from An initial workflow specification

Last updated on 2025-06-24 | Edit this page

Overview

Questions

  • How to use JUBE to automate the build process of a given HPC application?

Objectives

  • Define an initial workflow.
  • Create the benchmarks systems description automatically.

Writing an initial workflow

Mapping our experience with JUBE so far to the steps involved in compiling GROMACS, we can define two different steps:

  • download and unpack GROMACS sources
  • building the application

Discussion

Why don’t we have more or less steps defined?

If we would define all actions taken in the episode manually building GROMACS in one step, we would always download the sources of GROMACS for each build.

If we would separate the configure and the make phases, those will seem independent, yet there is always a single configure phase with each make phase. Therefore, these two actions should be part of the same step.

After executing this workflow we can make several observations:

  1. Some of the strings now hardcoded are part of a pattern that could be represented as a parameter instead.

  2. Downloading and unpacking is done as part of the workflow into the workflow directory tree. Additional runs will download and unpack the sources again.

So let’s address these issues one at a time.

Reducing code-copy in the specification

In several locations, the version of GROMACS is referenced: in the source archive, the unpacked source directory, and the installation path. If we’d change the version in the future, we would have to change several locations in the workflow configuration. We can reduce this by creating an appropriate parameterset.

Edit your initial GROMACS workflow configuration and use parameters for key variables in your workflow, to make this more flexible.

Breaking out of the workpackage sandbox

By default, JUBE will create a specific directory for each workpackage to ensure that independent workpackages do not interfere with each other. However, sometime it is helpful to break out of this safety net.

Caution

The sandboxing of individual workflow runs and their workpackages is done for good reason: to limit potential interactions among independent workpackages and ensure consistency and reproducibility.

Only deviate from this when you have a very strong argument to do so.

Next to workflow-specific parameters defined by the workflow itself, JUBE also defines variables containing information about the current workflow run. These variables can be referenced just as any parameter defined as part of a parameterset. One of these variables is $jube_benchmark_home, and it contains the absolute path to the location of the workflow specification.

Callout

Find a full list of internal variables set by JUBE in the glossary of JUBE’s documentation.

Using this variable, we can now define the the installation path outside of the directory structure referenced by outpath. However, as any paths outside of JUBE’s run directory tree will be accessed (and potentially written to) by multiple workflow runs. Therefore, you will need to take precautions not to overwrite installations accidentially.

Now we installed GROMACS externally but yet have to automate the decision whether to build or use the installed version. This can be handled with the active attribute. Steps and do tags (and a few others) can contain an attribute active that can be either true or false or any parsable boolean Python expression. When evaluated to false, the respective entity is disabled. When evaluated to true, the respective entity remains enabled (just as if no active attribute had been given).

To only build GROMACS, when no complete install is available, we need

  • an indicator that a previous install was successfull,
  • the evaluation of that indicator,
  • an expression to use as the value for the active attribute, and
  • add an action to remove any preexisting installation (that may be incomplete).

For this purpose we add a final do action to the build step that creates a file indicating that this step was complete and, because of transitivity, all prior do actions completed successfully. Furthermore, we then create a parameter as part of the gromacs_pset that indicates the existence of the file in the target directory. We use a parameter here, because of it ease of use when checking for the existence of a file as part of a shell expresseion. This parameter can then be referenced in the appropriate do actions in the build step.

Key Points

  • Group actions that belong together and have a 1:1 relation ship in a single step.
  • Basing parameter values on other parameter values can help code copy and increase flexibility and maintainability of your workflow.
  • You can generate build files from templates using dynamic values from parameter sets.

Content from Influencing parameters with tags

Last updated on 2025-05-09 | Edit this page

Overview

Questions

  • How can a workflow be influenced from the command line without changing the workflow specification files?

Objectives

  • Use tags to influence configuration values.
  • Run different workflows with the same configuration.

Now that we transferred our initial workflow to a JUBE configuration, we would like to extend this to work with other versions of GROMACS as well.

Fortunately, all names in the URL and archive follow the same convension, and we have already separated the values involved into the proper parameters. So now, we can modify our GROMACS workflow to use a different version of the software.

SH 

$ nano gromacs.xml

When we run our workflow again, it will download and build the 2024.1 version of the GROMACS software.

SH 

$ jube run gromacs.xml

Now, we might not want to modify our configuration every time directly if we want to compare runs of two different versions, or comment in or our specific lines in our parametersets.

Uniqueness of parameters

Parameters in JUBE have to be globally unique. That means now two different parametersets are allowed to define a parameter of the same name.

However, if several parameter definitions for the same name exist, the last definition will determine the parameter’s value during the workflow run.

The following specification will result in the parameter var being assigned the value two.

Visibility of configuration entities

JUBE allows the attribute tag for any part of the configuration. Tagged parts of the configuration are either retained in or removed from the workflow specification during a run, depending on tags for that run specified on the command line.

In the presence of tags,

  1. Any entity without a tag attribute will be retained in the workflow.
  2. Any entity with a tag attribute and the value of the attribute defined, will be retained in the workflow.
  3. Any entity with a tag attribute and the value of the attribute not defined, will be removed in the workflow.

Tag attributes are also allowed to be comma-separated lists of tag names. A specification of ! (exclamation mark) in front of a tag name will invert rules 2 and 3 above (not specification).

The not (!) specification has higher precedence than the normal tag specification. Consider the following parameter definitions:

When specified with --tag a b, the parameter will removed for that run, as the !b takes higher precedence over the a.

Challenge

Define the gromacs_version parameter such that per default the version 2024.5 is chosen, but in the presense of the tag v2024.1 the version 2024.1.

Using tags to enable a forced rebuild

Using the tag functionality, we can add a second definintion of the build_gromacs parameter with a rebuild tag to the workflow specification to force JUBE to rebuild of GROMACS just by adding the rebuild tag on the command line when running a workflow.

Key Points

  • Use tags to include or exclude specific parts of a workflow.

Content from Spanning parameter spaces

Last updated on 2025-05-21 | Edit this page

Overview

Questions

  • How to test multiple parameters in a single workflow?
  • How to control multi-dimensional parameter spaces?

Objectives

  • Create multiple workpackages with comma-separated lists.
  • Control parameter spaces with dependent parameters.

By default, JUBE will split any parameter that contains the , (comma) character into multiple tokens, using the comma character as the separator. For each token, JUBE will create a separate workpackage. This means a value of hello,world, will result in two distinct workpackages: one with the corresponding parameter set to hello, the other one set to world.

Callout

JUBE allows for values containing the , (comma) character by specifying the separator manually for specific parameters.

If multiple parameters can be tokenized, each parameter will become a dimenson in the parameter space. Consider the following example with two parameters containing comma-separated values.

This will result in workpackages with the following parameter combinations:

processes threads cores
1 1 1
1 2 2
1 4 4
2 1 2
2 2 4
2 4 8

This make it easy to create larger parameter spaces along specific dimensions. We can combine this to create a scaling experiment spanning multiple compute nodes, with different process to thread ratios.

Challenge

Assume a node has 96 cores on two sockets (48 cores each). Create a configuration that runs a scalability study from 1 up to 4 nodes and tests different process to thread ratios but always uses all cores on the node. Apart from a configuration for a single process per node, the remaining configurations should have the same number of processes per socket (i.e., an even number of processes per node).

With the constraints above, a configuration has

  1. a single process, or
  2. an even number of processes that is also a divisor of 96.

For this, we can set up multiple parameters that help us compute some of the parameters based on others.

Spanning parameter spaces can be used for different purposes. As we see above, we can use this to create scalability studies where we scale the number of logical and/or physical computational entities of our HPC run. However, this can be just as well used to span multiple different inputs or different combinations of arguments to the simulation.

Key Points

  • Comma-separated values in parameter definition are tokenized and create individual workpackages.
  • Using multiple comma-separated values for parameters enables easy creation of parameter spaces.

Content from Using templates

Last updated on 2025-05-23 | Edit this page

Overview

Questions

  • How do I generate dynamic text-based input files for my workflow.
  • How do I create batch scripts for by workflow.

Objectives

  • Create simple text-based templates generate files based on parameter settings.

In HPC workflows, oftentimes input files or batch scripts need to be created. While some parts of these files are specific to the current workflow running, the files either have a static structure and/or even largely static content. To handle this, JUBE provides a mechanism of generating text files based on search and replace strategy.

Substitutions

At the heart of this process are substitute sets, which are a collection of individual source to destination replacements. The replacements can either be based on static text or variables of parameters from any parameterset in use at the time of substitution.

Such substitutions need to be part of a substitute set that defines the input template as well as the output file. These are defined by iofile clauses. Each substitute set needs one of more such clauses. Providing multiple iofile clauses will apply the same text replacements to all files listed.

File sets

To make the substitutions more independent, JUBE provides so called file sets. These are a collection of clauses describing that either a file or directory should be copied or linked to the workpackage of a given step.

Caution

Linking files may reduce redundancy, especially if large read-only files and directories are involved in the workflow. However, keep in mind that a link potentially creates a connection to a file path outside of the run directory, which may impact concurrent steps and even separate runs.

To ensure reproducibility and consistency of workflow runs, the user should therefore check for potential side effects and employ a combination of copy and link commands that suits the workflow best.

Callout

Copy and link clauses can also specify additional attributes, such as source_dir and target_dir to specify a path to prefix the given filename with, rel_path_ref to indicate externalreference (relative to the XML/YAML file) for the file (the default), or internal referencing the current working directory (to link to files of another step). See the full description of the copy_tag for more details.

Key Points

  • Filesets let you easily copy templates into the workpackage space.
  • Substitution sets perform simple text-based substitutions in templates.
  • Using parameter values in substitution sets enables easy value manipulation during template generation.

Content from Including external data

Last updated on 2025-05-22 | Edit this page

Overview

Questions

  • How to better structure your workflow files to allow for reuse?
  • How to reduce redundancy of definitions?

Objectives

  • List different mechanisms to include data from multiple configuration files.
  • Select an appropriate inclusion mechanism in different scenarios.

When workflows become complex it often helps to organize parts of those workflows in different files. JUBE provides three different ways to include configuration data from different files.

Reusing external sets

The simplest form of reusing external configuration data is to define a set in an external file and use it as is in a different configuration file as part of a step definition.

Initializing local sets with external data

Sometimes values defined in external sets work almost for a different workflow scenario, yet small changes would be necessary. Copying the set and modifying the small number of values to be changed would create a lot of redundancy. For such situations JUBE allows users to define sets and initialize them with the values from a different set. The new set contains all values of the external set, and only needs to redefine the values that need changing.

Callout

Next to tagging, this is another way to change the value of a configuration entity in JUBE.

Including arbitrary configuration data

So far, we only reused set defined in external files. Howerver, JUBE also allows to include arbitrary parts of an external configuration file. The external configuration file is even allowed to contain non-JUBE tags (e.g., to make it easier to identify a block of entities), as long as the included portion of the external configuration does not retain any unknown tags.

The specifics of this is out of the scope for this tutorial at this stage, so please reffer to the JUBE tutorial pages of the Jülich Supercomputing Centre for more details on that.

Key Points

  • from inside a use clause allows to include an external set as is.
  • init_with allows to initialize child sets with the values of a parent set and modify and extend the set.
  • include allows to include arbitrary parts of external configuration files.

Content from Running an application

Last updated on 2025-05-22 | Edit this page

Overview

Questions

  • How can you use JUBE to submit batch jobs on an HPC System?

Objectives

  • Use the done_file attribute to deal with asynchronous commands.
  • Automatically generate job scripts from templates.
  • Automatically submit jobs on an HPC system.

Dealing with asynchronous commands

The commands in the do clauses so far, were all synchronous. This means that the effect of the specified command is observable after the command completed. For example a mkdir -p my_dir will return after the directory my_dir is created. JUBE will continue execution of the next do clause defined in the step of complete the step.

HPC systems however are used differently than personal computers. To ensure the resources are used fairly and efficiently, work is handed to a scheduler to allocate resources for the job. However, those resources may not be available right away. The scheduler therefore does not block the command, but rather return to the user and handles the job without further interaction of the user.

This means, however, the effect of the job script (a completed application run) will most likely not have materialized when the command return and JUBE shoud wait for it before continuing execution of the next do clause. JUBE handles such asynchronous behavior via the done_file attribute of the do clause. If specified, JUBE will not assume the successful execution of the specified command in the do clause for completion, but the existence of the the file specified in done_file as a side effect of the do command.

This way, JUBE does not need to understand the details of the asynchronous command specified, as long as the command eventually generates the file specified in done_file. For HPC systems and batch jobs, this can be exploited by generating a file as part of the batch job, if the application ran successfully as part of the batch job.

SH 

$ nano batchscript.sh

SH 

....
srun ...
JUBE_ERR_CODE=$?
if [ $JUBE_ERR_CODE -ne 0 ]; then
    touch error
    exit $JUBE_ERR_CODE
fi
...
touch ready

In case of the SLURM scheduler, the corresponding do clause would then look like so.

OUTPUT 

######################################################################
# benchmark: GROMACS
# id: 24
#
# MD Simulation Workflow
######################################################################

Running workpackages (#=done, 0=wait, E=error):
########################################00000000000000000000 (  2/  3)

  |        stepname | all | open | wait | error | done |
  |-----------------|-----|------|------|-------|------|
  | prepare_sources |   1 |    0 |    0 |     0 |    1 |
  |           build |   1 |    0 |    0 |     0 |    1 |
  |             run |   1 |    0 |    1 |     0 |    0 |

>>>> Benchmark information and further useful commands:
>>>>       id: 24
>>>>   handle: jube_run
>>>>      dir: jube_run/000024
>>>> continue: jube continue jube_run --id 24
>>>>  analyse: jube analyse jube_run --id 24
>>>>   result: jube result jube_run --id 24
>>>>     info: jube info jube_run --id 24
>>>>      log: jube log jube_run --id 24
######################################################################

Any workpackage with a pending done_file will be listed under wait in the table of open tasks. To try to advance any waiting workpackage the user needs to needs the continue command, as listed in the output given by JUBE.

SH 

$ jube continue jube_run --id 24

OUTPUT 

```output
######################################################################
# benchmark: GROMACS
# id: 24
#
# MD Simulation Workflow
######################################################################

Running workpackages (#=done, 0=wait, E=error):
############################################################ (  3/  3)

  |        stepname | all | open | wait | error | done |
  |-----------------|-----|------|------|-------|------|
  | prepare_sources |   1 |    0 |    0 |     0 |    1 |
  |           build |   1 |    0 |    0 |     0 |    1 |
  |             run |   1 |    0 |    0 |     0 |    1 |

>>>> Benchmark information and further useful commands:
>>>>       id: 24
>>>>   handle: jube_run
>>>>      dir: jube_run/000024
>>>>  analyse: jube analyse jube_run --id 24
>>>>   result: jube result jube_run --id 24
>>>>     info: jube info jube_run --id 24
>>>>      log: jube log jube_run --id 24
######################################################################

Platform-independent workflows

JUBE offers batch job templates compatible with a variety of batch systems that utilize this mechanism. These files are available in the JUBE installation directory unter share/jube/platform, with subdirectories per scheduler. Provided with version 2.7.1 of JUBE are files prepared for LSF, Moab, PBS, and SLURM.

In each of these directories JUBE provides at least the two files platform.xml and submit.job.in. The former is a collection of parameter sets, file sets, and substitution sets. The latter is a template for the corresponding scheduler.

Callout

By specifying the path corresponding to your scheduler as part of the environment variable JUBE_INCLUDE_PATH, you can make both files readily available inside your workflow scripts.

SH 

$ export JUBE_INCLUDE_PATH=/path/to/your/jube/base/share/jube/platform/slurm:$JUBE_INCLUDE_PATH

Taking a look into the batch script template reveils the semingly automatic handling of generating either an error or a ready file as part of the batch job.

SH 

...
JUBE_ERR_CODE=$?
if [ $JUBE_ERR_CODE -ne 0 ]; then
    #FLAG_ERROR#
    exit $JUBE_ERR_CODE
fi
...
#FLAG#

If the return code is not 0 (zero), the action specified by the substitution parameter #FLAG_ERROR# is executed, and the script exits with the corresponding non-zero return code. At the end of the batch script, the substitution parameter #FLAG# will generate the specified file.

If we take a look into the corresponding substitution set in platform.xml, we can see te following configuration.

XML 

<sub source="#FLAG#" dest="touch $done_file" />
<sub source="#FLAG_ERROR#" dest="touch $error_file" />

as well as the corresponding the parameterset executeset in platform.xml contains the definition of done_file and error_file.

XML 

<parameter name="done_file">ready</parameter>
<parameter name="error_file">error</parameter>

Using these pre-defined sets in platform.xml in combination with the batch script template, we can easily create the run step for GROMACS.

Using the indirecttion of the corresponding sets in the platform.xml we now have a workflow specification that does not in itself reference any specific scheduler, as that is handled via the JUBE_INCLUDE_PATH from the shell JUBE is executed in.

Callout

Checkout further substitution patterns available in the batch script template to identify parameters you can use to add commands to your batch script.

Note that those scripts and definitions are only for your convenience. If they don’t fit your need, you can adapt them to your needs or fully rely on self-provided configurations.

Key Points

  • JUBE provides a default job script template for different schedulers.
  • JUBE allows for asynchronous execution of workflows using the done_file attribute.
  • Asynchronous JUBE executions are dependent on the generation of files by the asynchronous task to continue.

Content from Output analysis and presentation

Last updated on 2025-06-27 | Edit this page

Overview

Questions

  • How can you retrieve output from your HPC application and workflow?
  • How to generate tables and CSV output from JUBE workflow parameters?

Objectives

  • Generate regular expression patterns to analyse the output of a step.
  • Generate a basic table from benchmark parameters and output.

Prerequisite

If you cannot execute GROMACS via the generated batch script to generate an output, you can down load an example error log job.err and place it in the work directory of your run stage (jube_run/<id>/000002_run/work/). Furthermore you need to let JUBE know that the execution was successful with the following command executed in the same directory.

SH 

jube_run/<id>/000002_run/work/$ touch ready

Then continue the workflow to complete it.

Analyzing workflow output

Next to parameters, which are evaluated at use time in the steps of a workflow, another way to automatically obtain and store information about a workflow execution is to analyze the output generated during execution. JUBE provides an easy way for regular expressions to parse workflow output and store values of matched patterns in variables to be included in later result tables.

Patterns are defined as part of pattern sets that can be applied to specific files during analysis. As with parameters, the names of patterns need to be unique for the whole workflow. When included in result tables, their names can be shortened or changed completely to better fit the generated table.

Challenge

For the workflow defined, we have two steps with potentially valuable information: build and run. Check the outputs in the corresponding workpackages and identify interesting output files to parse for meaningful data.

  • The cmake_configure.log in the build workpackage contains output from CMake that provides some insight on which libraries were found and used for the build.
  • The stdout in the run workpackage contains the output of the job submission command
  • The job.err in the run workpackage contains the GROMACS output.

Patterns are defines as regular expressions as part of a pattern set. When we take the following snippets from cmake_configure.log

OUTPUT 

-- The C compiler identification is IntelLLVM 2024.2.0
-- The CXX compiler identification is IntelLLVM 2024.2.0
...
-- Detected best SIMD instructions for this CPU - AVX_512
-- Enabling 512-bit AVX-512 SIMD instructions using CXX flags:  -march=skylake-avx512
...
-- Using external FFT library - Intel MKL
...

we can already identify some interesting information that are worth extracting.

Each pattern can contain an arbitrary amount of wildcards, but must contain exactly one matching operator (), which defines the value of the pattern. JUBE also defines several patterns for common elementar types such as numbers and individual words:

  • $jube_pat_int: integer number w/ matching operator
  • $jube_pat_nint: integer number w/o matchin operator
  • $jube_pat_fp: floating point number w/ matching operator
  • $jube_pat_nfp: floating point number w/o matching operator
  • $jube_pat_wrd: word w/ matching operator
  • $jube_pat_nwrd: word w/o matching operator
  • $jube_pat_bl: blank space (variable length) w/o matching operator

Callout

While all steps of a JUBE workflow need to be defined before the workflow is started and cannot be altered after that, patterns and result tables can be updated while the workflow is active and even after a workflow has completed. This enables an iterative approach for defining patterns and result tables for previously unknown output.

To use the defined patterns, JUBE needs an analyser specifying which patterns to use for which file or which patterns to use globally.

We can then start and test the analyser with the analyse command. However, as we modified the workflow configuration by adding new patterns and analyser definitions, we need to tell JUBE to update its information about the workflow. This can be done with the -u (for update) argument followed by the updated workflow specification.

OUTPUT 

######################################################################
# Analyse benchmark "GROMACS" id: 35
######################################################################
>>> Start analyse
>>> Analyse finished
>>> Analyse data storage: jube_run/000035/analyse.xml
######################################################################

Without the definition of a result table, we cannot visualise this directly, but we can investigate the data storage given in the output and check which patterns were matched for which workpackage.

XML 

<?xml version="1.0" encoding="UTF-8"?>
<analyse>
  <analyser name="gromacs_analyser">
    <step name="build">
      <workpackage id="1">
        <pattern name="cmake_c_compiler_id" type="string">IntelLLVM</pattern>
        <pattern name="cmake_c_compiler_id_first" type="string">IntelLLVM</pattern>
        <pattern name="cmake_c_compiler_id_cnt" type="int">1</pattern>
        <pattern name="cmake_c_compiler_id_last" type="string">IntelLLVM</pattern>
        <pattern name="cmake_c_compiler_version" type="string">2024.2.0</pattern>
        <pattern name="cmake_c_compiler_version_first" type="string">2024.2.0</pattern>
        <pattern name="cmake_c_compiler_version_cnt" type="int">1</pattern>
        <pattern name="cmake_c_compiler_version_last" type="string">2024.2.0</pattern>
        <pattern name="cmake_cxx_compiler_id" type="string">IntelLLVM</pattern>
        <pattern name="cmake_cxx_compiler_id_first" type="string">IntelLLVM</pattern>
        <pattern name="cmake_cxx_compiler_id_cnt" type="int">1</pattern>
        <pattern name="cmake_cxx_compiler_id_last" type="string">IntelLLVM</pattern>
        <pattern name="cmake_cxx_compiler_version" type="string">2024.2.0</pattern>
        <pattern name="cmake_cxx_compiler_version_first" type="string">2024.2.0</pattern>
        <pattern name="cmake_cxx_compiler_version_cnt" type="int">1</pattern>
        <pattern name="cmake_cxx_compiler_version_last" type="string">2024.2.0</pattern>
        <pattern name="SIMD_detected" type="string">AVX_512</pattern>
        <pattern name="SIMD_detected_first" type="string">AVX_512</pattern>
        <pattern name="SIMD_detected_cnt" type="int">1</pattern>
        <pattern name="SIMD_detected_last" type="string">AVX_512</pattern>
        <pattern name="SIMD_flags" type="string">-march=skylake-avx512</pattern>
        <pattern name="SIMD_flags_first" type="string">-march=skylake-avx512</pattern>
        <pattern name="SIMD_flags_cnt" type="int">1</pattern>
        <pattern name="SIMD_flags_last" type="string">-march=skylake-avx512</pattern>
        <pattern name="FFT_detected" type="string">Intel MKL</pattern>
        <pattern name="FFT_detected_first" type="string">Intel MKL</pattern>
        <pattern name="FFT_detected_cnt" type="int">1</pattern>
        <pattern name="FFT_detected_last" type="string">Intel MKL</pattern>
      </workpackage>
    </step>
  </analyser>
</analyse>

JUBE also tracks multiple matches per pattern and tracks it in “shadow” patterns with additional suffixes. You can find a more details description in the JUBE Glossary under ‘statistical values’. For numerical statistics (e.g., min, max, avg, std) the pattern needs to be of type int or float. Also, a unit specified as a string can be stored with a pattern.

Generating a result table

With our first successful matches, we can now define our first result table for build-related information in a result specification.

OUTPUT 

gromacs_build:
|  compiler | compiler version |    SIMD |       FFT |
|-----------|------------------|---------|-----------|
| IntelLLVM |         2024.2.0 | AVX_512 | Intel MKL |

Challenge

Add an additional patternset, analyser and result definition to generate an additional table similar to the following:

OUTPUT 

gromacs_run:
| wp | gromacs_core_time[s] | gromacs_wall_time[s] | gromacs_core_perf[ns/day] | gromacs_wall_perf[hours/ns] |
|----|----------------------|----------------------|---------------------------|-----------------------------|
|  2 |               89.664 |                3.736 |                    89.664 |                       3.736 |

The wp column references a JUBE variable identifying the workpackage of the row. This makes it easier to identify the right directory of the workpackage in case multiple run steps were executed.

Callout

Each table has a separate file in the result/ directory with its name as is name and the extension .dat.

Tables can be either pretty printed or in CSV (comma-separated values) format. The latter being the default type.

Key Points

  • JUBE allows for definition of patterns to retrieve values from a workflow step.
  • Patterns can be defined as regular expressions or Python expressions.
  • JUBE allows for the generate of pretty-printed tables and CSV format.

Content from Thoughts on reproducibility

Last updated on 2025-05-23 | Edit this page

Overview

Questions

  • How can I improve reproducibility of my JUBE workflow?

Objectives

  • Identify key aspects of the workflow and its parameters.
  • Define ways to record values of these aspects as part of a run.

JUBE in itself does not make every workflow definition inherently reproducible. However, all parts of a workflow definition and parameter evaluation are recorded and retained during workflow execution. To increase reproducibility of a workflow, it’s the user’s repsonsibility to identify all aspects of a workflow and define ways to record them during execution of the workflow.

Discussion

Which aspects of the build and measurement steps should be recorded to increase reproducibility of the workflow?

Potential aspects that merit recording:

  • For general workflow execution
  • Build flags used to generate binaries
  • Versions of software and libraries used
  • Input files and configuration
  • Output files
  • Execution scale and domain decomposition
  • …
  • For performance measurements (additionally to the ones above)
    • Networking libraries involved
    • Configuration of networking library
    • Processor model and microcode version
    • OS and kernel version
    • Node names used for the execution
    • Execution configuration (pinning, distribution, …)
    • …

Challenge

Select one or more of the aspects identified in the prior discussion and specify a configuration to record it as part of a workflow.

Here are some examples on how to record some reproducibility-relevant aspacts of your workflow

  • Build flags can either be specified directly or tried to be extracted from generated build files (e.g., Makefiles)
  • Versions of software and dependent librariess should be a mix of specified and extracted
    • Note: It can be difficult to identify the corresponding versions of linked libraries if they didn’t record this information
  • Output files are a natural part of the run directory
  • Scale and specifics of execution can easily be defined as a parameter and used to generate input and run files
  • Network libraries are commonly part of the system’s software stack and notoriously hard to identify on the low level
    • Note: It is easy to record the version of a selected MPI library, while it may be harder to identify the built-time configuration of a pre-installed system library
    • Note: Generate software environments (e.g., docker or apptainer images) to help conserve the execution environments
  • OS and kernel version can be recorded from commands like uname or files in /proc
    • Note: Make sure to record this information from the correct location (login nodes vs. batch nodes)
  • Node names can be gathered after execution with scheduler tools and the JOB ID (extracted from submission output)
  • Execution confguration such as pinning should be explicitly set and checked for validity

Key Points

  • Identify aspects of the workflow that are important for reproducibility.
  • Obtain values for these either through parameters or patterns.
  • Add actions to the workflow that help in recording information through patterns.

Content from Wrap-up and outlook

Last updated on 2025-05-22 | Edit this page

Overview

Questions

  • Which features of JUBE were not covered in this course?

Objectives

  • Identify advanced features of JUBE.

Summary

In this course we created a minimal workflow to build and run the GROMACS application with a given input. You can download the corresponding configurations gromacs.xml and gromacs.yaml for further exploration.

Outlook to other topics

In this workshop we have only scratched the surface of handling workflows with JUBE. As with every software, inspiration for creative uses of specific features come with experience with those features.

Here is a list of some of the topics that could not be covered in this course

  • Running multiple iterations of a step
  • Re-evaluating parameter values at different
  • More advanced combinations of parameter modes
  • Advanced tagging with duplicate attribute for parameters
  • Multiple benchmark definitions per file
  • Shared steps
  • Step cycles
  • Using SQlite databases for results
  • Debugging workflows

Challenges with JUBE workflows

The main challenge with JUBE workflows is the separation of analysis and result tables from workflow steps. This makes it hard to define additional steps that analyse output and create result tables.

The easiest way to cope with this is to create additional scripts outside of the workflow that act on the created tables (e.g. creating plots, etc.). However, this is unsatisfying as the means part of our overall workflow (i.e., the generation of plots) is outside of the actual workflow specification and has to be done manually.

Another option is to have a second layer of workflow specifications where the execution of a JUBE workflow is a step in a parent workflow, also defined in JUBE. This way, the full workflow can be defined in JUBE, yet the additional layer add complexity overall.

Finally, as analyses can be performed and result tables created and updated while the workflow is not yet finished, another viable approach can be to add a do action with a done_file at the location where the workflow should pause until the result tables are complete, and then create the done_file manually once all workpackages have completed up to that point.

Key Points

  • JUBE is a very flexible workflow management system.
  • JUBE’s advanced features can be further customized with Python and Perl expressions.