Welcome to Maestro Workflow Conductor Documentation
Maestro is an open-source HPC software tool that defines a YAML-based study specification for defining multi-step workflows and automates execution of software flows on HPC resources. The core design tenants of Maestro focus on encouraging clear workflow communication and documentation, while making consistent execution easier to allow users to focus on science.
Maestro’s study specification helps users think about complex workflows in a step-wise, intent-oriented, manner that encourages modularity and tool reuse. Maestro’s development centers around a user-centric design approach and makes use of software design practices such as abstract interfacing and utilizing design patterns, forming the foundation of a vision for enabling a layered architecture to workflow tool design.
These principles are becoming increasingly important as computational science is continuously more present in scientific fields and users continue to perform increasingly complex workflow processes across platforms.
Getting Started is Quick and Easy
Maestro can be installed easily with
pip install maestrowf
YAML file named
study.yaml and paste the following content
into the file:
description: name: hello_world description: A simple 'Hello World' study. study: - name: say-hello description: Say hello to the world! run: cmd: | echo "Hello, World!" > hello_world.txt
PHILOSOPHY: Maestro believes in the principle of a clearly defined process, specified as a list of tasks, that are self-documenting and clear in their intent.
hello_world study is as simple as:
maestro run study.yaml
Creating a Parameter Study is just as Easy
With the addition of the
global.parameters block, and a few simple tweaks
to your study block, the complete specification should look like this:
description: name: hello_planet description: A simple study to say hello to planets (and Pluto) study: - name: say-hello description: Say hello to a planet! run: cmd: | echo "Hello, $(PLANET)!" > hello_$(PLANET).txt global.parameters: PLANET: values: [Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto] label: PLANET.%%
PHILOSOPHY: Maestro believes that a workflow should be easily parameterized with minimal modifications to the core process.
Maestro will automatically expand each parameter into its own isolated workspace, generate a script for each parameter, and automatically monitor execution of each task.
And, running the study is still as simple as:
maestro run study.yaml
- Installing Maestro
- Quick Start Guide
- Basics of Study Construction
- LULESH Specification Breakdown
- Scheduling Studies (a.k.a. the Batch Block)
- Specifying Study Parameters
- Maestro Core Concepts
- Maestro Workflow Conductor
- maestrowf package
- setup module
- tests package