Creating and configuring a Spack-Manager Project

Spack-Manager projects are the method for organizing configuration files and package repositories associated with a given software application.

If a pre-configred project exists then it simply has to be add to the Spack-Manager configuration file. This section will demonstrate the process of setting up a Spack-Manager project from scratch using the ExaWind application.
This is just an example that is not going to be kept up-to-date for Exawind since it is a living, independent project. The actual ExaWind configuration can be found here.

Creating a Project from Scratch

A Project is really just a collection of directories and couple of optional python files. In the spack-manager.yaml configuration file paths are provided to the projects that this instance of Spack-Manager is supporting.

$ cat spack-manager/spack-manager.yaml

spack-manager:
    projects:
    - $SCRATCH/exawind-demo

$ ls -lh $SCRATCH/exawind-demo
ls: /Users/psakiev/scratch/exawind-demo: No such file or directory

If a non-existent path is added to the Projects list in the spack-manager/spack-manager.yaml file then these initial directories will automatically be created the first time a spack manager command is run. It should also be noted that the autocompletion of variables inside these paths (such as the $SCRATCH environment variable in this example, follow the same variable conventions utilized by Spack. This is because Spack-Manager is importing and using the Spack code for these operations.

$ spack manager find-machine --list
Project:         Machine:        Detected: (+/-)
------------------------------------------------------------

$ tree $SCRATCH/exawind-demo
/Users/psakiev/scratch/exawind-demo
├── configs
└── repos

3 directories, 0 files

Once the first spack manager command is run the project configs and repos directories get populated. The configs directory is where machine specific spack configuration files are stored. Since this directory is currently empty no machines show up when spack manager find-machine --list is run. The next step is to add some machines and configuration files. Note that machines is a loose term, and it is really a bifurcation of configurations. The term machine was selected since application projects typically have to tweak their spack configurations on each new machine/platform.

Populating Machine Specific Configurations

Directories within the project’s configs directory will be delineated by their names. Let’s add a directory named darwin inside the configs directory and re-run spack manager find-machine --list.

$ mkdir $SCRATCH/exawind-demo/configs/darwin
$ spack manager find-machine --list
Project:         Machine:        Detected: (+/-)
------------------------------------------------------------
exawind-demo     darwin          -

darwin is automatically picked up as an available machine by the find-machine command, and any configuration files that are added inside this directory will be added to an environment created with a darwin machine specified. It should also be noted that the project name exawind-demo comes from the name of the parent directory. Now configurations can be added. The following two configurations are used by ExaWind.

# config.yaml
config:
  mirrors:
    e4s: https://cache.e4s.io

# packages.yaml
packages:
  hypre:
    variants: +shared~fortran
  all:
    compiler: [apple-clang, gcc, clang]
    providers:
      mpi: [mpich, openmpi]

Anonymous Machines

There are two anonymous machines that are reserved for project and users to utilize: base and user. The configs/base directory is designed to hold project wide configurations that are the defaults utilized by every environment. For example, the software project has non-default package variants they wish to use uniformly across all platforms these could be set in the configs/base/packages.yaml file.

The configs/users directory is one that allows users to set their own personal preferences or make tweaks before pushing them to the whole team. An example here is if one user really prefers a specific flavor of MPI and the software project is not constrained then they could configure that in the configs/user/packages.yaml file.

The hierarchy of precedent for these configs are:

user
machine
base where the smaller number means higher precedent.

So any configurations in base are not the law and can be overridden on each machine as necessary. An example of this would be if the project prefers to build with +shared but on specific platform can only support ~shared. As a reminder, this hierarchy is only for creating the default environment configuration on each platform when it is created. These are added to the spack.yaml via entries in the includes list. Each environment can still be customized by modifying the spack.yaml file and using additional configuration techniques.

For completeness, the optional base and user directories will now be added to the exawind-demo project.

$ mkdir $SCRATCH/exawind-demo/configs/base
$ mkdir $SCRATCH/exawind-demo/configs/user
$ spack manager find-machine --list
Project:         Machine:        Detected: (+/-)
------------------------------------------------------------
exawind-demo     darwin          -

The output above shows how these config directories are ignored by the find-maachine command. Next we’ll add some of the base configs used by ExaWind to the base directory.

# config.yaml
config:
  source_cache: ~/.spack_downloads
  misc_cache: $spack/../.cache
  build_stage:
    - $spack/../stage
  concretizer: clingo

# concretizer.yaml
concretizer:
  unify: false
  reuse: false

Now if we create an environment using spack manager create-env --machine darwin we will see all the configs that have been added in the include.yaml file.

$ spack manager create-env --machine darwin -d $SCRATCH/example-env
making /Users/psakiev/scratch/example-env

$ cat $SCRATCH/example-env/spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs: []
  view: false
  concretizer:
    unify: true
  include:
  - include.yaml

$ cat $SCRATCH/example-env/include.yaml
concretizer:
  unify: false
  reuse: false
packages:
  hypre:
    variants: +shared~fortran
  all:
    compiler: [apple-clang, gcc, clang]
    providers:
      mpi: [mpich, openmpi]
config:
  mirrors:
    e4s: https://cache.e4s.io
  source_cache: ~/.spack_downloads
  misc_cache: $spack/../.cache
  build_stage:
  - $spack/../stage
  concretizer: clingo

Configuring Machine Auto-detection

The last column from the output of spack manager find-machine --list indicates if the configuration was detected for the current machine. The default behavior is to detect nothing and require users to specify the machine they want to use. However, setting up automatic detection is simple and highly configurable for each project and each machine.

To add detection a project must have a python file named find-[project].py in the top-level directory of the project (exawind-demo in this example). find-[project].py needs to have a method named detector that takes a string with the machine name and returns True or False depending on if the current machine meets the criteria for that name.

Here is an example find-exawind-demo.py script

import sys

def detector(name):
    """
    A function that will check if the supplied name/machine
    matches a known machine configuration
    """
    # dictionary that is easily extensible where key is the name we want to match
    # and the value is function that can be evaluated to test the actual system we
    # are one
    known_machines = {
        "darwin": lambda: sys.platform == "darwin",
    }

    if name in known_machines:
        return known_machines[name]()
    else:
        return False
        

Now when the find-machine command is run the darwin machine will be detected.

$ spack manager find-machine --list
Project:         Machine:        Detected: (+/-)
------------------------------------------------------------
exawind-demo     darwin          +

Users are free to implement and sort of detection script they want.
The only requirements are that the method detector have a positional argument for the name of the machine to check for, and returns a boolean to indicate if that supplied name was detected.

Setting up Spack Package Repositories

The [Project]/repos directory is a place holder for package repositories that are paired with the softwware applications development/deployment. The simplest way to ensure the appropriate repos are included is to add a reference to them in [Project]/configs/base/repos.yaml file. This is a way to ensure that any environment created with Spack-Manager for the desired project will include the repos.

An example for ExaWind is as follows:

# repos.yaml
repos:
  - $spack/../repos/exawind

Where the $spack is a supported configuration variable that will be expanded by spack. This works for ExaWind because ExaWind creates a fixed mirror of spack that is submoduled into their Project repository.

Utilizing a configuration variable, environment variable, or manually updating the repo paths are currently the only way to point to repos in arbitrary locations on the filesystem. Users may also add the copy_repos: true flag to their projects inside the spack-manager.yaml configuration file if they wish to just automatically copy the repo files locally to an environment when it is created.

spack-manager:
  projects:
  - $SCRATCH/exawind-demo
    copy_repos: true

In this case the repo specification would be properly resolved with the following repo.yaml file in the base configs.

# repos.yaml
repos:
  - $env/repos/exawind

Now when an environment is created the [Projects]/repos directory will be copied completely to the environment, and the environment will look for the copy relative to its own location.

Please note, that these are mainly small tricks to utilize spack’s builtin path resolution strategies. Additional work in the future is anticipated to make this a more seamless setup and transition.

Adding Version Control

At this point the exawind-demo project is populated with an initial set of configurations. It is highly suggested that it be placed under some form of version control. git is by far the most popular tool for version control at the moment. A suggested .gitignore file would look something like the following:

# basic python files
__pycache__
*pyc
# some operations in spack-manager can currently create this directory
.tmp