Advanced Topics for Developers

This section is designed to go over some of the tips and tricks for developers that will help you improve your workflow and handle some more nuanced scenarios.

Handling Multiple Environments

Spack-Manager is designed to support multiple environments from a single Spack instance. This allows common libraries to be reused between installations. Where you place the environments is entirely up to you. There is a default location $SPACK_MANAGER/environments where the environments will be created if you pass the --name or -n flag into one of the environment creation commands. Using the --directory or -d argument will create the environment at the local path you specify. Omitting either of these arguments will lead to the environment being created in the current working directory.

In terms of when to create an environment, our recommendation is that an environment be created for each unique concept you are working on, and once that concept is finalized you uninstall the software and/or delete the environment.

Cleaning Up Old Environments

If an environment is stale and hasn’t been used in a while, our recommendation is to create a new environment using it’s spack.yaml and then just delete the old environment.

An example of how to do this is:

quick-create -d [location of new env] -y [old environment]/spack.yaml
spack -e [old environment] uninstall --all
rm -rf [old environment]

TIP: Using spack -e [env location] allows you to run commands in that environment without activating it in your shell.

If you truly wish to revise the environment then it is a good idea to try and reconcretize before making changes to the environment. spack concretize -f.

Modifying Specs in an Environment

Once an environment is active in your shell it can still be modified. The definition of the environment is in the spack.yaml file and modifying this will modify the environment. So to add a variant, such as making the build a debug build, simply open the spack.yaml in your environment’s directory and add build_type=Debug to the spec for the package you want to be built in debug mode.

TIP: Spack has a built in command for automatically opening the spack.yaml file in a text editor for you. It is spack config edit. The default editor is Vim, but you can set it to be anything you want simply by setting the EDITOR environment variable i.e. export EDITOR=emacs.

A more archaic way to do this from the command line would be to remove the spec and then add it back with the edits that you want.

For example say you have amr-wind@master+openfast+hypre as the spec in your environment. You could run

spack rm amr-wind@master
spack add amr-wind@master+openfast+hypre build_type=Debug

if you really don’t want to open the yaml file. This is discouraged though because it involves more typing and is prone to more errors. All users are encouraged to become comfortable reading and editing the spack.yaml file. The ones generated by Spack-Manager are quite small and only contain information pertinent to you as an end user.

Debugging Spack Errors

It is inevitable that you will encounter some error from Spack while using Spack-Manager. Spack-Manager is a dynamic tool that tries to stay up-to-date with Spack’s new features. Sometimes we catch bugs and sometimes there are incompatibilities or inconsistencies in the state of your Spack installation.

We suggest the following steps for how to handle an error from Spack:

If you just updated Spack-Manager make sure the Spack submodule is up-to-date: git submodule update
Read the error message again and see if it makes sense. Many of the error messages are quite good.
Run sm-clean. This is a heavy clean up command for Spack-Manager that we’ve created based on the errors we’ve seen in the past.
Run your Spack command with the -d flag (debug) i.e. spack -d install and see if that gives you any more insight
Reach out for help on slack, github issues or email.

When and How to Use Multiple Instances of Spack-Manager

Sometimes it is useful to have multiple instances of Spack-Manager. Some examples of when it would be a good idea to do this:

You have multiple machines on a shared filesystem and they are conflicting, or one is problematic while the other is stable.
You want to archive production environments/builds and separate them from your software development work.
You are developing features for Spack or Spack-Manager and want to keep it separate from the rest of your work.

The key concept here is that there is something stable and something more dynamic. It is useful to blow away everything in dynamic software development and start from scratch periodically.

Spack-Manager has been designed to be self-contained and to not create or use dependencies from other Spack instances as much as possible. The key thing is you don’t want to mix Spack instances in the same shell. So to use different Spack-Manager instances you just need to make sure that the SPACK_MANAGER environment variable is set to the right instance and that you don’t switch things up in the same shell.

An example of how this can be done in a bashrc is:

# Load dynamic spack-manager for software development
function development-spack-manager(){
  export SPACK_MANAGER=/path/to/my/dynamic/spack-manager
  source ${SPACK_MANAGER}/start.sh
  spack-start
}

# Load stable builds
function production-spack-manager(){
  export SPACK_MANAGER=/path/to/my/production/spack-manager
  source ${SPACK_MANAGER}/start.sh
  spack-start
}

In this example the different Spack-Manager instances can be loaded in a new shell by calling separate functions.

Strategies for Conserving Disk Space

If Spack is not periodically pruned it can become a storage hog. When doing software development Spack will create and preserve the build directory for develop specs, and then it will install the binaries in the common Spack install tree.

To prune your local build directories you can simply run rm -rf spack* inside a build directory. This will remove all the files and directories Spack created but not modify your source code. It is essentially just removing the CMake build directory and all of Spack’s logs. If you are frequently modifying specs for your environment (switch from Release to Debug) then you will notice several build directories spack-build-[hash]. Preserving these allows for incremental builds, and deleting them will cause you to rebuild the entire software product. So there is a trade off between waiting on builds and storage space.

To handle the issue of install trees we’ve added the --local-source or -l flag to the environment creation commands. This will make it so all the binaries are installed local to the environment in a directory named opt/. Now when you delete the environment you will also delete all the install binaries and you don’t need to worry about them piling up in the main Spack install tree.

However, if you only use this option you will end up rebuilding all the binaries in every environment. Using --local-source means that the environment can’t doesn’t see any of the other environments and you will need to create a link between them. To link to your main Spack database create the $SPACK_MANAGER/configs/user/upstreams.yaml file and put the following inside of it:

upstreams:
 root_spack:
   install_tree: $spack/opt/spack

Any files added to the $SPACK_MANAGER/configs/user directory will automatically be added to the environments you create going forward. This allows users to set configurations like upstreams or personal preferences without needing to have them get pushed to the main repo. The addition above makes it so any --local-source environment you create can reuse binaries from your main install tree.

So to setup a very space conservative Spack-Manager repo you should do the following:

Create a base environment and install the software you want to use without using --local-source
Create the upstreams.yaml file as specified above.
Create all development environments with the --local-source or -l flag.

Now whenever you want to clean things up just delete your development environment directory and all the space from that specific build will be removed. Doing this is a bit tedious and the new base environments will need to be periodically created for new compilers and flags, or the existing ones will need to be updated.

Constructing and Utilizing Multiple builds

One of the nice features about using Spack is the build environments you setup can have arbitrary complexity. A nice feature for this is to setup your development environment to perform multiple builds that you may need from the same source such as a +cuda and ~cuda build, or Release and Debug builds.

To do this you need to edit the spack.yaml file so that the concretizer:unify parameter is set to false. Unify tells the concretizer that all the software has to concretize into a single graph. We turn on unification by default in Spack-Manager to serve as a guard-rail for users and to simply syntax for setting up environments.

spack:
  # having the two specs
  specs: 
  - nalu-wind@master+hypre+cuda
  - nalu-wind@master+hypre~cuda
  view: false
  concretizer:
    unify: false # this is defaulted to true
  include:
  - include.yaml
  develop:
    nalu-wind:
      # this develop spec matches both the root specs and so the source code will be used for both
      spec: nalu-wind@master

You may not want to rebuild both instances of the code every time you make a change. However, you can do build-env-dive nalu-wind~cuda if you want to develop the code on a non-cuda environment for the faster link times. When using build-env-dive you can simply call make, make clean, run the tests, etc from inside the build directory. When you want to test the cuda build you can exit the build-env and rebuild the Cuda case. An example of this workflow is illustrated below.

# Setup the environment
quick-create -d two-build-one-env -s nalu-wind+cuda nalu-wind~cuda
spack manager develop nalu-wind@master
spack config add conncretizer:unify:false # a way to edit the spack.yaml configs from the command line
spack install 
# dive into the non-cuda build environment
build-env-dive nalu-wind~cuda
# make some code changes
[...]
# run the unit-tests and interate
make -j8 && ./unittestX
# jump out of the build0-env 
exit
# now rebuild everything with the changes
spack install
# check the cuda tests
build-env-dive nalu-wind+cuda
ctest -R unit -VV