FIREWHEEL Repositories

In order for FIREWHEEL to identify available model components, model components must be stored in a “repository”. A repository is simply a directory that holds one or more Model Components and is present on the head node of the FIREWHEEL cluster. That is, its primary job is to inform FIREWHEEL where a set of Model Components can be located. Since version control is a software best practice, they are generally going to be either git repositories and/or Python packages, but it is not required. While installing a repository as a Python package is the easiest way to use repositories, users can also create and initialize a top-level folder (e.g., /opt/firewheel/model_components) and clone/move any repositories into that folder. Using this technique any model components that are located within this folder and any new Model Components added to it will accessible by FIREWHEEL. Because installing Python packages and cloning various Model Component git repositories into /opt/firewheel/model_components is trivial, we expect that this feature will only be utilized by FIREWHEEL developers and administrators.

How FIREWHEEL Interacts with Repositories

When FIREWHEEL launches an experiment, it begins by traversing available repositories and searching for model components. Figure 4, shows some of the contents and structure of the model_components repository that’s included with FIREWHEEL. Model components can be the direct children of a repository or nested within container parent folders (like the tests.router_tree model component). However you organize the model components in a repository, FIREWHEEL will locate them when needed at run time.

Model components found within any of the installed repositories can be included as a dependency by other model components. FIREWHEEL handles this dependency management automatically, thus enabling code reuse similar to how libraries or modules can be included in software programs written in C++ or Python. In this way, model components within and across repositories can be linked together to create more powerful model components, and to build up full experiment topologies with scheduled activities. For more information on model component dependencies, see Model Components.

Since users can (and are encouraged to) create repositories containing one or more of their own experiment models, this eliminates the need for project specific code to be present in FIREWHEEL’s main codebase. If there are model components that are releasable or might be useful to others, we encourage users to publish them in a separate git repository and/or as a Python package. The flexibility of a repository also provides a way to keep all your experiment specific model components in a single location (i.e. experiment topology definition, scheduled action executables, and VM images not already included with FIREWHEEL, etc.).

Fig. 4 FIREWHEEL’s model_components repository (partial) contents/structure.

Installing a Repository into FIREWHEEL

The easiest way to install a repository is via a Python package manager (e.g., Pip).

If the repository is not a Python package, once the repository is on the FIREWHEEL Control Node, then the repository can be installed with the following command:

$ firewheel repository install <path to repository directory>

Once installed (whether via a package manager or manually), all model components contained within a repository will be available for use.

Some Model Components may require additional Python packages to be installed or data to be downloaded. In this case, the Model Component can have an INSTALL file, which can be any executable script (as denoted by a shebang line). There are two ways in which these files can be executed:

1. Lazily when the model component is used in an experiment and FIREWHEEL determines that it hasn’t been installed.

2. Proactively by calling The repository install Helper with the -s flag (see repository install for more details). This will iterate through all repositories and identify uninstalled model components and install them.

Warning

The execution of Model Component INSTALL scripts can be a dangerous operation. Please ensure that you trust the repository developer prior to executing these scripts.

Converting a Repository to a Python Package

Similar to other frameworks (e.g., Flake8) FIREWHEEL uses Entry Points which are provided by setuptools. This allows any package to register a new repository with FIREWHEEL.

We currently use the group firewheel.mc_repo to register a new repository. In the example below (using a pyproject.toml file), we tell setuptools to register our entry point repo_path inside the group firewheel.mc_repo. While is does not matter the name of the variable repo_path, the value does matter and must be <package_name>.__path__. This provides FIREWHEEL access to the actual location of the model components so that it can access non-Python related data such as the MANIFEST file or VM resources.

Note

We also recommend that users use a tool like setuptools-scm to ensure all git-committed data is captured within the package.

[build-system]
requires = ["setuptools>=61.0", "setuptools_scm[toml]>=6.2"]
build-backend = "setuptools.build_meta"

[project]
dynamic = ["version"]
name = "firewheel_repository_name"
authors = [
{ name="Example Author", email="author@example.com" },
]
description = "A small example package"
readme = "README.rst"
requires-python = ">=3.9"
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]

[tool.setuptools_scm]

[project.entry-points."firewheel.mc_repo"]
repo_path = "firewheel_repository_name:__path__"

With the previous toml file, the package can be built by running the following command in the top level Model Component directory:

python -m build

This will create a .whl file in the dist directory, which can then be installed by running:

cd dist
python -m pip install <whl-file>

Alternatively, the package can be installed all at once by running the following command in the top level Model Component directory:

python -m pip install .