Model Components

Model components (MCs) are the building blocks of a FIREWHEEL experiment. Essentially they are folders within repositories, which may contain code, VM images, and other resources. Running a FIREWHEEL experiment is done by telling FIREWHEEL which model components define the network topology and any actions to be taken, and the launcher for the virtualization system you want FIREWHEEL to use for instantiating your experiment. A model component’s folder must contain metadata that identifies it to FIREWHEEL as a model component, by declaring its name, contents, dependencies, and what it provides that other MCs could then depend on being able to use.

Model components provide a place to colocate all the new files required to accomplish a specific objective. A model component’s function, or purpose (as defined by it’s developer), can be anything that an experiment might need. For instance, they can provide:

• Functions for constructing an experiment network’s topology

• Definitions of vertex and edge types that can then be included in an experiment

• Configuration scripts designed for individual VMs or whole classes of VMs

• Executable code for performing experiment related actions on the network at a given time

• Scheduling instructions for when to execute actions on designated VMs in an experiment

• A launcher for a FIREWHEEL experiment composed of other model components

• Any combination of these and many other possibilities

Previously (in FIREWHEEL v1.x) these files were scattered throughout a handful of separate folders, but without a standard way of identifying what each did or which other related collections of components each depended on (other than via python import statements). Now, a model component is a single place where code and data related to the model component’s purpose can be found, along with metadata that allows FIREWHEEL to locate its dependencies, match what it provides to other model components’ needs, and enforce certain constraints.

What’s in a Model Component?

A model component is pretty flexible, and it’s folder can include several types of files. The different kinds of files that can be present, and that provide the model component’s functionality, include plugins, VM resources, model component objects, and images. Another file, called the MANIFEST file, contains the metadata that describes a model component to FIREWHEEL, and this is the only file that must be present in a model component’s folder. Each of these file types will be explained in sections that follow. In addition to the files, it’s highly recommended and encouraged (though optional), that a model component’s folder contain either a README.rst or a README.md. This file should contain RST (or Markdown) formatted documentation about the model component (i.e. what it is and how to use it). Additionally, this file will automatically be generated if the mc generate CLI Helper is used to create the model component skeleton.

Lastly, Model Components may sometimes require installing additional Python packages or downloading data (e.g. VM Resources) from the Internet. To facilitate this, users can add an INSTALL file, which can be any executable script (as denoted by a shebang line). More information about this file can be found in Model Component INSTALL file.

Generating a Model Component

It should be noted that FIREWHEEL provides a CLI command to generate a skeleton of a model component called mc generate. Therefore, in general most users will start their model component by running this Helper (i.e. firewheel mc generate). To see the usage of the command run the following:

$ firewheel "mc generate --help"
Usage: firewheel mc generate [-h] [--name NAME]
                [--attribute_depends ATTRIBUTE_DEPENDS [ATTRIBUTE_DEPENDS ...]]
                [--attribute_provides ATTRIBUTE_PROVIDES [ATTRIBUTE_PROVIDES ...]]
                [--attribute_precedes ATTRIBUTE_PRECEDES [ATTRIBUTE_PRECEDES ...]]
                [--model_component_depends MODEL_COMPONENT_DEPENDS [MODEL_COMPONENT_DEPENDS ...]]
                [--model_component_precedes MODEL_COMPONENT_PRECEDES [MODEL_COMPONENT_PRECEDES ...]]
                [--plugin PLUGIN]
                [--model_component_objects MODEL_COMPONENT_OBJECTS]
                [--location LOCATION] [--plugin_class PLUGIN_CLASS]
                [--author AUTHOR] [--version VERSION]
                [--language LANGUAGE]
                [--vm_resource VM_RESOURCES [VM_RESOURCES ...]]
                [--image IMAGE] [--arch ARCH] [--non-interactive]
                [--skip_docs] [--template_dir TEMPLATE_DIR]
                [--no_templates]

Generate a new ModelComponent

optional arguments:
    -h, --help            show this help message and exit

MANIFEST:
    --name NAME           ModelComponent name
    --attribute_depends ATTRIBUTE_DEPENDS [ATTRIBUTE_DEPENDS ...]
                            (space-separated-strings) Graph Attribute(s) depended
                            on by the new ModelComponent
    --attribute_provides ATTRIBUTE_PROVIDES [ATTRIBUTE_PROVIDES ...]
                            (space-separated-strings) Graph Attribute(s) provided
                            by the new ModelComponent
    --attribute_precedes ATTRIBUTE_PRECEDES [ATTRIBUTE_PRECEDES ...]
                            (space-separated-strings) Graph Attribute(s) preceded
                            by the new ModelComponent
    --model_component_depends MODEL_COMPONENT_DEPENDS [MODEL_COMPONENT_DEPENDS ...]
                            (space-separated-strings) ModelComponent(s) required
                            by name
    --model_component_precedes MODEL_COMPONENT_PRECEDES [MODEL_COMPONENT_PRECEDES ...]
                            (space-separated-strings) ModelComponent(s) that will
                            be preceded by name
    --plugin PLUGIN       File for a plugin
    --model_component_objects MODEL_COMPONENT_OBJECTS
                            File for Model Component Objects
    --location LOCATION   Location for the new ModelComponent
    --plugin_class PLUGIN_CLASS
                            Name for the new plugin class
    --author AUTHOR       Author for the model component
    --version VERSION     Initial version number for the model component
    --language LANGUAGE   Documentation language for the model component
    --vm_resource VM_RESOURCES [VM_RESOURCES ...]
                            (space-separated-strings) File(s) to be used as a
                            vm_resource
    --image IMAGE         File to be used as a VM disk
    --arch ARCH           Architecture for specified image

    Configuration:
    --non-interactive     Require minimum parameters as arguments and do not
                            prompt for any values
    --skip_docs           Do not generate any documentation files
    --template_dir TEMPLATE_DIR
                            Override the configured templates directory
    --no_templates        Do not generate files from templates. Only generate a
                            MANIFEST

This command will be used throughout the various tutorials and will be modified to fit the specific need of the tutorial. Each piece of the various model component settings is explained below.

The MANIFEST File

Every model component has a MANIFEST that describes it to FIREWHEEL, and a model component’s MANIFEST is contained in a file, appropriately named MANIFEST, that’s located in the model component’s folder. A model component’s MANIFEST consists of YAML formatted data (the model component’s metadata) that specifies to FIREWHEEL:

• The unique name of the model component

• The types of attributes the model component depends, provides, and precedes

• The names of other model components it explicitly depends on and precedes

• The types and names of files (either images or VM resources) contained within it.

A MANIFEST file must be included in a model component’s folder in order for FIREWHEEL to recognize the folder as a model component.

Each model component’s MANIFEST contains at least three elements: a name field, an attributes object, and a model_components object. Other, optional elements (fields and objects) that may be included in the MANIFEST are: plugin, vm_resources, model_component_objects, and images.

A Note About YAML

While YAML (https://yaml.org/) is a human-readable data-serialization language, FIREWHEEL only uses it to make MANIFEST files more human-readable. YAML is a superset of JSON, making any [1] JSON MANIFEST files valid. There are several benefits for using YAML over JSON:

1. It is difficult to write valid JSON manually. For users that will be developing new model components without using the MC generation tool, JSON has a more complex syntax. Editing JSON MANIFEST files has a similar issue. These challenges can be alleviated using the more human-readable (and human writable) YAML.

2. YAML works with version control systems such as Git better than JSON does because it does not require commas to separate items in arrays and lists (https://en.wikipedia.org/wiki/YAML#Comparison_with_JSON).

3. YAML is less verbose than JSON. It does not require strings to be in quotation marks and does not require brackets or curly braces around objects. This saves some development time with manual MANIFEST editing.

Here are a few YAML examples:

Creating a list

vm_resources: [vmr1.sh, vmr2.py]
"vm_resources": ["vmr1.sh", vmr2.py]
vm_resources:
    - vmr1.sh
    - "vmr2.py"

Creating a dictionary

attributes: {depends: [], provides: [], precedes: []}
"attributes": {
    depends: [],
    provides: [],
    precedes: []
}
attributes:
    depends: []
    provides: []
    precedes: []

Creating a string

name: "test.mc"
name: test.mc

[1]

Tabs are NOT valid in YAML, but are valid JSON. See https://yaml.org/faq.html for more details. You should avoid using tabs when creating MANIFEST files.

The Name field

The name field’s value can be any valid string, but needs to be unique among all model components contained in all repositories collectively available on a Control Node. The name of a model component does not need to be the same as the name of it’s folder. A model component’s name can be used by model component developers to refer to it when they need to explicitly depend on it in their model component’s MANIFEST (see the Model Components object for more details).

It takes the following form:

name: <model_component_name>

A common naming convention is <repository name>.<purpose name>, e.g.:

name: acme.topology

In this example, the model component is named acme.topology because it is part of the acme repository and it does the primary construction of the acme network topology. While this naming convention isn’t enforced by FIREWHEEL, it’s a good idea to use it to avoid name collisions amongst model components across repositories.

The Attributes Object

Model component attributes can be thought of as commodities which can be provided by the model component and depended on or precede other MCs. The attributes object in a MANIFEST specifies any attributes a model component provides to other model components, any attributes it depends on other model components to provide to it, and any attributes which it precedes and must be resolved after completion. Declaring that a model component depends on or precedes an attribute will generate a dependency between it and a model component that provides that attribute. However, unlike depending on another model component explicitly (see the Model Components object for details), declaring that a model component depends on a attribute is laissez-faire with regard to which specific model component will provide the attribute at run time.

The attributes object contains three fields, depends, provides, and precedes, each expecting an array of strings for their values. Each field’s value array can contain zero or more attribute labels. In the depends field’s value array, attribute labels specify which attributes (if any) that a model component expects will be provided to it by some other model component at run time, without caring in any way which model component provides the attribute to it. In the provides field’s value array, attribute labels declare the types of attributes (if any) that a model component produces for others to consume. In the precedes field’s value array, attribute labels specify which attributes (if any) that are required to be provided by a model component that runs after the current MC completes execution.

Functionally, attribute labels are used by FIREWHEEL to create relationships amongst model components in an experiment. This information is used at run time, along with the values in the Model Components object and the ordering of model components listed on the command line, to create a dependency tree representing the interdependencies amongst all of the model components required to run any given FIREWHEEL experiment, across all available repositories. See the section on dependency management for more information on how and when to use attribute labels.

The attributes object is required to be present in a MANIFEST, but its depends, provides and/or precedes fields’ value arrays can be empty.

It takes the following form:

attributes:
    depends: []
    provides: []
    precedes: []

For example, the first model component in an experiment to begin creating the experiment’s network topology will need to depend on the graph attribute (since it will need to add vertices and edges to it) and possibly it provides a topology attribute to other model components in the experiment, e.g.:

attributes:
    depends:
        - graph
    provides:
        - topology
    precedes: []

Now, if there is another model component that has the purpose of modifying vertices in a topology in some way (e.g. by assigning hostnames to each one), then its attributes object would declare that that it depends on having a topology provided to it by another model component, and also that it provides something (e.g. hostnames) that other model components can depend on having been provided by it. Therefore, a model component that sets the hostnames for each vertex in a topology, thus providing a modified topology where each vertex has a unique hostname assigned to it, would have an attributes object that resembles the following:

attributes:
    depends:
        - topology
    provides:
        - hostnames
    precedes: []

Note

Attribute labels in provides fields’ value arrays are made up by the developers of the various model components provide attributes. Therefore, if you create a model component that provides something, it’s up to you to determine the attribute labels that get declared in the MANIFEST, as being provided by your model component. We strongly suggest that you choose a name that is clearly connected to whatever the model component achieves.

The attributes object’s depends value array only allows users to specify the roots of a model component’s attribute-based dependency tree, and then FIREWHEEL adds any further attribute-based dependencies in the tree for you. This facilitates sharing repositories with others without burdening them with knowing the exact required dependencies within and across them. See Dependency Management for more information on how FIREWHEEL manages dependencies between model components.

The Model Components object

The model_components object in a MANIFEST explicitly specifies the specific, uniquely named MCs that it depends on and precedes. The model_components object contains two fields, depends and precedes, and they expects an array of strings as the value. The value array can contain zero or more names of other model components, as declared using The Name field in their MANIFEST files.

This information is used at run time, along with the values in the attributes object and the ordering of MCs listed on the command line, to create a dependency tree representing the interdependencies amongst all of the model components required to run any given FIREWHEEL experiment, across all available repositories. See Model Component Dependencies for more information on when and why to declare dependencies using the model_components object.

The model_components object is required to be present in a MANIFEST, but the depends and precedes field’s value arrays can be empty.

It takes the following form:

model_components:
    depends: []
    precedes: []

For example, the acme.topology model component adds vertices and edges to the graph, and needs some of those vertices to be of type Ubuntu1604Server. The Ubuntu1604Server class is defined in the linux.ubuntu1604 MC, therefore, acme.topology will need to attempt to import the Ubuntu1604Server class into its plugin python module e.g. using from linux.ubuntu1604 import Ubuntu1604Server) before it can assign this object to the appropriate vertices.

The linux.ubuntu1604 model component, and any python objects within it, will be made explicitly available to the acme.topology model component if, and only if, the acme.topology MANIFEST includes it in the model_components object, e.g.:

model_components:
    depends:
        - linux.ubuntu1604
    precedes: []

Similarly, since acme.topology needs to be able to create a Helium118 (VyOS router) then it will need to depend on the model component that defines the Helium118 model component objects class, vyos.helium118, e.g.:

model_components:
    depends:
        - linux.ubuntu1604
        - vyos.helium118
    precedes: []

Finally, acme.topology also needs to be able to create a Switch which is defined in the base_graph_objects MC, so it is also required, e.g.:

model_components:
    depends:
        - linux.ubuntu1604
        - vyos.helium118
        - base_graph_objects
    precedes: []

The Plugin field

The plugin field specifies the name of the file, located within the model component’s folder, that contains the Python Class that defines the functionality of a model component’s plugin module (if it has one). The plugin field expects a string as its value. The role of a model component’s plugin is to operate on the graph and a model component can contain at most one plugin. The plugin field’s value is used by FIREWHEEL at run time to locate and execute a model component’s plugin module at the appropriate time.

The plugin field is only required to be present in a MANIFEST if the model component contains a plugin module file.

It takes the following form:

"plugin": "<plugin_file_name>"

For example, in The Attributes Object section above we considered a model component called acme.set_hostname. In this model component there would be a file that contains a SetHostname class that walks the topology graph, provided by its topology attribute provider, setting the hostnames for all the vertices. To identify the plugin file containing the SetHostname class, the acme.set_hostname model component’s MANIFEST file would need to include the plugin parameter, e.g.:

plugin: plugin.py

This states that the SetHostname class that needs to be run is located in the plugin.py file, which itself is located in the acme.set_hostname model component’s folder. [2]

[2]

The convention is to name the plugin’s python module plugin.py. This is not strictly enforced by FIREWHEEL, but has been found to make file management a bit easier.

Within the plugin module, a single class determines the execution behavior of the plugin. This plugin class is defined by the user as a subclass of FIREWHEEL’s AbstractPlugin, and it requires a run method be defined to describe the plugin’s actions.

This plugin class is automatically loaded by the model component, so only one plugin class may be defined per plugin module (more than one plugin class would result in ambiguity and cause FIREWHEEL to raise an error).

The VM Resources field

The vm_resources field specifies the names of the files, located within the model component’s folder, which are to be scheduled for execution/use using Schedule Entries. The vm_resources field expects an array of strings as its value. The role of VM resources is to either perform some action(s) on a VM, or provide some other resource used for accomplishing an action, and a model component can contain zero or more VM resources.

VM resources can be scripts, executables, data files, or blobs, etc., that are added to vertices and scheduled for use on the VM images assigned to vertices in a network topology, once FIREWHEEL has instantiated them while launching an experiment. VM resources can be used to execute any function you want to have operate on a VM. They are used to configure VMs, install and configure applications on them, and to carry out other actions needed to conduct an experiment, such as generating network traffic and collecting data for analysis. The vm_resources field’s values are used by FIREWHEEL at run time to locate a model component’s VM resource files at the appropriate time during experiment launch/execution.

Including the vm_resources field in a MANIFEST is only required if the model component contains one or more VM resource files.

It takes the following form:

vm_resources: []

To continue the acme.set_hostname example, its plugin schedules a VM resource called set_hostname.py on every vertex in the experiment topology graph. Since this VM resource is contained within the acme.set_hostname model component’s folder, its MANIFEST must include the following vm_resources parameter:

vm_resources:
    - set_hostname.py

If the model component provides many vm_resources we recommend putting them in a folder (called vm_resources). Then you can set the parameter:

"vm_resources":["vm_resources/**"]

Here are the following permutations which are valid paths for vm_resources:

• Non-recursively provide access to a directories files: path_to_dir OR path_to_dir/

• Non-recursively, provide access to directory files matching a pattern: path_to_dir/*.ext

• Recursively provide access to all files: path_to_dir/** OR path_to_dir/**/

• Recursively provide access to all files matching a pattern: path_to_dir/**/*.ext

The Model Component Objects field

The model_component_objects field specifies the name of the file, located within the model component’s folder, that contains one or more Python Classes that define new objects you want/need to have for use in your experiments. The model_component_objects field expects a string as its value. The types of python classes you might include in a model component objects file include:

• Classes that define vertex and edge types that can then be added to the experiment graph when defining a topology

• Classes that add schedule entries to vertices in an experiment topology, to schedule actions performed by VM resources

• Classes that expose helper functions for performing any number of tasks needed by an experiment

The model_component_objects field’s value is used by FIREWHEEL at run time to locate and execute a model component’s model component object classes when needed.

Every vertex in a graph that will instantiate a VM has a model component object class associated with it, to define it’s properties, and every VM resource that will be used in an experiment is scheduled for use by adding a model component object class to a vertex, to identify the resource and define its execution time. When model component contains a model component objects file, which includes one or more python classes that define vertex types, schedule entries, or helper functions, FIREWHEEL ensures those python classes are available to the experiment through the model_component_objects parameter.

Including the model_component_objects field in a MANIFEST is only necessary if the model component contains a files that defines model component objects.

It takes the following form:

"model_component_objects": "<model_component_objects_file_name>"

For example, if a model component contains a model component object to specify a Win7 vertex, and the Win7 python class is defined in model_component_objects.py [3], then the model component’s MANIFEST would need to include:

model_component_objects: model_component_objects.py

It’s important to note that model component objects don’t only have to apply to vertices within the graph. Model components can have model component objects that apply to edges as well.

[3]

The convention is to name the python module that contains model component object class definitions as model_component_objects.py. This is not strictly enforced by FIREWHEEL, but has been found to make file management a bit easier.

The Images field

The images field in a MANIFEST specifies information about all VM image files, located within the model component’s folder, that a model component contains and may provide to other model components. The images field expects an array of objects as its value. Each object in the images field’s value array contains two fields, called paths and architecture, with paths expecting an array of strings for its value, and architecture expecting a string for its value.

The images field’s value array can contain zero or more objects. Each object in an images field’s value array can specify the names of one or more files containing VM images, but the image files listed in any single object must all share the same architecture specification. The images field’s values are used by FIREWHEEL at run time to locate the VM image files needed for instantiating their corresponding graph objects when launching an experiment topology that includes them.

Many times it makes sense to have the images compressed for storage purposes (especially when using version control). Therefore, FIREWHEEL will automatically detect and decompress images that are using tar or LZMA compression. That is, if your file uses LZMA compression (e.g. the xz utility) or tar compression (including tar with gzip), then it will automatically be decompressed by FIREWHEEL. Specifically, we support the following extensions: .xz, .tar, .tar.gz, and .tgz.

The images field only needs to be included in a MANIFEST if the model component contains one or more VM image files.

It takes the following form:

"images": [
    {
        "paths": [],
        "architecture": ""
    }
]

For example, the Win7 graph object specified in The Model Component Objects field specifies the windows-7-enterprise.qc2.xz image file, and since this image file is also contained in the same model component as the graph object, then that model component would also need the following images field in its MANIFEST file:

"images": [
    {
        "paths": ["windows-7-enterprise.qc2.xz"],
        "architecture": "x86_64"
    }
]

# A YAML version would look like:
images:
    - paths:
        - windows-7-enterprise.qc2.xz
      architecture: "x86_64"

It’s worth pointing out here that the file containing a VM image, used for instantiating a graph object, and the file defining the graph object’s class don’t both necessarily need to reside in the same model component, though they often will. There are cases where, for instance, you might want to develop a new graph object that uses the same VM image as another graph object does, and that VM image already exists in that other graph object’s model component.

In this case, your new graph object’s model component would simply need to depend on the model component containing the VM image file i.e. in the Model Components object in its MANIFEST. Then you’d reference it in your new graph component class and FIREWHEEL would be able to locate the VM image for instantiating your new graph object at run time.

Example MANIFEST Files

The following MANIFEST file is for the ubuntu1604 model component.

name: linux.ubuntu1604
attributes:
    depends: []
    provides: []
    precedes: []
model_components:
    depends:
        - linux.ubuntu
    precedes: []
images:
    - paths:
        - "images/ubuntu-16.04.4-server-amd64.qcow2.xz"
    architecture: x86_64
    - paths:
        - "images/ubuntu-16.04.4-desktop-amd64.qcow2.xz"
    architecture: x86_64
model_component_objects: model_component_objects.py

The following MANIFEST file is for a model component that creates the ACME topology.

name: acme.topology
attributes:
    depends:
        - graph
    provides:
        - topology
        - acme_topology
    precedes: []
model_components:
    depends:
        - base_objects
        - vyos.helium118
        - linux.ubuntu1604
    precedes: []

plugin: plugin.py

The following MANIFEST file is for a model component that provides the SetHostname plugin.

name: acme.set_hostname
attributes:
    depends:
        - acme_topology
    provides:
        - hostnames
model_components:
    depends:
        - linux.ubuntu1604
    precedes: []
plugin: plugin.py
vm_resources:
    - set_hostname.py