Control

experiment_graph.py

This module contains numerous classes related to the FIREWHEEL experiment graph.

firewheel.control.experiment_graph.CACHED_DECORATOR_OBJECTS

This module-level dictionary enables the caching of each MC Object used and the methods/attributes (e.g., dir()) of those objects. By caching this, the performance is greatly increased due to fewer calls to dir().

Type:

dict

class firewheel.control.experiment_graph.AbstractPlugin(graph, log)

This class will be inherited by all model component plugin files.

__init__(graph, log)

Initialize the plugin.

Parameters:

• graph (ExperimentGraph) – The experiment graph in which the plugin will execute.

• log (logging.Logger) – A log the plugin can use.

get_experiment_graph()

Returns:

The experiment graph.

Return type:

ExperimentGraph

exception firewheel.control.experiment_graph.DecorationError

This exception is used if there is an error decorating a Vertex.

exception firewheel.control.experiment_graph.DecoratorConflictError

If there are two decorators which interfere with each other this exception is used.

class firewheel.control.experiment_graph.Edge(source_vertex, destination_vertex)

This class represents a FIREWHEEL-specific Edge. It inherits from ExperimentGraphDecorable and implements all the expected methods for a networkx Edge.

__contains__(key)

Check if the Edge has a particular attribute.

Parameters:

key (str) – The attribute to query.

Returns:

Tue, if the key exists, False otherwise.

Return type:

bool

Raises:

RuntimeError – If the Edge is not Edge.valid.

__delitem__(key)

Remove a Edge attribute based on a key.

Parameters:

key (str) – An attribute/property to delete.

Raises:

RuntimeError – If the Edge is not Edge.valid.

__eq__(other)

Determine if two Edges are the same. Equality is based on having the same source and the same destination. Edges are not directed, so we are equal even if the direction is reversed. This function also verifies that itself and another are of type Edge and are Edge.valid.

Parameters:

other (Edge) – The other Edge.

Returns:

True if they are the same, False otherwise.

Return type:

bool

Raises:

RuntimeError – If either Edge is not Edge.valid.

__getitem__(key)

Dictionary-style access to read Edge attributes.

Parameters:

key (str) – A key to query.

Raises:

RuntimeError – If the Edge is not Edge.valid.

Returns:

The Edge matching the key.

Return type:

Edge

__hash__()

Get the hash of a tuple containing the hash of the destination and a hash of the source.

Returns:

The hash of a tuple containing the hash of the destination and a hash of the source.

Return type:

int

__init__(source_vertex, destination_vertex)

Initialize the Edge.

source

The source Vertex for this Edge.

Type:

Vertex

destination

The destination Vertex for this Edge.

Type:

Vertex

valid

If the Edge should exist in the graph (i.e., has not yet been deleted).

Type:

bool

log

The logger to use for this Edge.

Type:

firewheel.lib.log.Log

Parameters:

• source_vertex (Vertex) – The source Vertex for this Edge.

• destination_vertex (Vertex) – The destination Vertex for this Edge.

Raises:

• TypeError – If either the source or destination are not a Vertex.

• ValueError – If the Vertices are not in the same graph or they are not Edge.valid.

__iter__()

Iterate over the Edge.

Returns:

An iterator that iterates over the Edge properties.

__ne__(other)

Determine if two Edges are not equal. Inequality is based on the opposite of the __eq__() method. This function also verifies that itself and another are of type Edge and are Edge.valid.

Parameters:

other (Edge) – The other Edge.

Returns:

True if they are not the same, False otherwise.

Return type:

bool

Raises:

RuntimeError – If either Edge is not Edge.valid.

__setitem__(key, value)

Dictionary-style access to set a Edge attributes.

Parameters:

• key (Any) – An attribute key.

• value (Any) – The value of the attribute.

Raises:

RuntimeError – If the Edge is not Edge.valid.

delete()

Remove this Edge from the ExperimentGraph. This sets the Edge.valid property to False.

edge_log = <Logger ExperimentGraphEdge (DEBUG)>

get_object()

Get the Edge object attribute (i.e. self).

Returns:

This Edge.

Return type:

Edge

Raises:

RuntimeError – If the Edge is not Edge.valid.

has(key)

Dictionary-style query for the presence of a key.

Parameters:

key (str) – A key to query.

Returns:

If the key exists.

Return type:

bool

Raises:

RuntimeError – If the Edge is not Edge.valid.

class firewheel.control.experiment_graph.EdgeIterator(graph, source_id_list)

A custom Edge iterator which parses the graph and returns the next Edge.

__init__(graph, source_id_list)

Set up the Edge iterator.

Parameters:

• graph (ExperimentGraph) – The experiment graph to iterate over.

• source_id_list (list) – A list of Vertex IDs that are the source of an Edge.

Raises:

NoSuchVertexError – If the Vertex ID was not found in the graph.

__iter__()

Return this EdgeIterator.

Returns:

Returns this EdgeIterator.

Return type:

EdgeIterator

__next__()

Get the next Edge in the graph.

Returns:

The next Edge in the graph.

Return type:

Edge

Raises:

Exception – If the Edge object could not be found. This is most likely caused because the graph has not been constructed with the correct APIs. Users can double check that all additions/deletions are going though the graph API.

class firewheel.control.experiment_graph.ExperimentGraph

The graph describing a FIREWHEEL experiment.

This is a FIREWHEEL specific graph that leverages networkx but makes some modifications to all Vertex instances/Edges as they are added to the graph.

It conducts some specific error checking and also provides methods for getting Vertices by different properties (i.e. name and ID) as well as an efficient method for getting the all-pairs shortest path for a graph.

__init__()

Initialize the networkx.Graph, set up a counter and the logger.

g

The NetworkX graph that underlies this FIREWHEEL graph.

Type:

networkx.Graph

last_node_id

The ID of the last node in the graph.

Type:

int

_add_edge(source_id, dest_id)

Add an Edge object to the graph.

Parameters:

• source_id (int) – The source Vertex’s ID.

• dest_id (int) – The destination Vertex’s ID.

Raises:

NoSuchVertexError – If either the source or destination Vertex cannot be found.

_add_vertex(new_id=None)

Add a Vertex object to the graph.

Parameters:

new_id (int) – A possible ID for the new node. If None, then one will be created.

Returns:

The ID of the new Vertex.

Return type:

int

Raises:

RuntimeError – If there is a conflict with the provided Vertex ID and an existing Vertex in the graph.

_setup_logging()

Set up the logger for the experiment graph class.

_single_process_all_pairs_shortest_path(vertex_filter, path_action)

All pairs shortest path with a single thread of execution. Computes shortest path among all pairs where each Vertex matches vertex_filter, and calls path_action with the path between each pair.

Parameters:

• vertex_filter (func) – Callable taking a vertex (object) and returning True or :py:data`False`.

• path_action (func) – Callable path_action(path_source, path_dest, current_path) Where path_source is a Vertex, path_dest is a Vertex, and current_path is a list of Vertex on the path. The return value for path_action is ignored.

filtered_all_pairs_shortest_path(vertex_filter=None, path_action=None, num_workers=0, sample_pct=0)

All pairs shortest path with multiple threads of execution. Computes shortest path among all pairs where each Vertex matches vertex_filter, and calls path_action with the path between each pair.

Parameters:

• vertex_filter (func) – Callable taking a Vertex and returning True or False.

• path_action (func) – Callable path_action(path_source, path_dest, current_path) Where path_source is a Vertex, path_dest is a Vertex, and current_path is a list of Vertex on the path. The return value for path_action is ignored.

• num_workers (int) – Number of threads which will calculate the all-pairs shortest path.

• sample_pct (int) – The percentage of nodes for which the all pairs will be preformed. This speeds up the time it takes for the calculation to occur.

Returns:

If there are no other workers, the method _single_process_all_pairs_shortest_path() is returned.

Return type:

None

find_edge(ep1, ep2)

Try to find an Edge based on the passed in Vertex instances.

Note

This is a networkx specific method for finding the Edge.

Parameters:

• ep1 (Vertex) – The first of the Vertices that is part of the Edge being found.

• ep2 (Vertex) – The second of the Vertices that is part of the Edge being found.

Returns:

The found Edge, or None if an error has occurred.

Return type:

Edge

find_vertex(name)

Try to find a Vertex based on the passed in Vertex name.

Note

This is a networkx specific method for finding the Vertex.

Parameters:

name (str) – The name of the Vertex we are trying to locate.

Returns:

The found Vertex, or None if the Vertex cannot be found.

Return type:

Vertex

find_vertex_by_id(vert_id)

Try to find a Vertex based on the passed in Vertex ID.

Note

This is a networkx specific method for finding the Vertex.

Parameters:

vert_id (int) – The ID of the Vertex we are trying to locate.

Returns:

The found Vertex, or None if the Vertex cannot be found.

Return type:

Vertex

get_edges()

Get an iterator of the graph Edges.

Returns:

The EdgeIterator for the experiment graph.

Return type:

EdgeIterator

get_vertices()

Get an iterator of the graph Vertex instances.

Returns:

The Vertex iterator for the experiment graph.

Return type:

VertexIterator

class firewheel.control.experiment_graph.ExperimentGraphDecorable

A per-instance decorable object. This permits the attributes and interface to change for instances of this object, with the changes limited to the particular instance that is “decorated”.

Because this affects the instance level, there are some limitations:

• Class methods and attributes in the decorator become instance methods and attributes (respectively) in the decorated instance.

• Method descriptors are supported, with the caveat that they are reduced to basic methods. That is, the method descriptor’s __get__ function is called with the merged instance, and the result becomes an instance method.

• Descriptors (and properties) are handled as class attributes. This breaks the “descriptor-ness” of the attributes–lookups simply return the actual descriptor object.

• Using a descriptor with any method in the skip_set may not function as expected.

Note

Some common methods such as __eq__ are actually class methods. This results in decorators being unable to affect operators such as ==.

Names in the decorator and decoratee instance may conflict. If these occur at the class-level in the decorator, these conflicts may be resolved by providing the decorate function with a conflict_handler. This is a callable taking 3 positional arguments: entry_name, decorator_value, current_instance_value, where the values are the values of the attribute at entry_name. The callable must return the value for the merged attribute if the conflict handler found an appropriate way to deconflict, otherwise it should raise an IncorrectConflictHandlerError, signaling that another conflict handler was likely intended to be used for the given attribute.

The merge process also ignores some built-in attributes by name to avoid always causing merge conflicts. Specifically, these are defined in skip_set.

Note

Notably, the only built-in methods which we expect users to modify are __str__ and __repr__. Any modification of other built-in types (e.g., __hash__ or __eq__) may not function as expected.

Finally, once the methods/attributes have been merged into the instance, the decorator’s __init__ method is called. This method is given the merged instance, so any values the decorator expects to initialize should behave normally from the perspective of the decorator’s code. Arguments for the decorator’s __init__ may be passed to the decorate function, with positional arguments as a list to the init_args keyword and keyword arguments as a dictionary to the init_kwargs keyword.

The behaviors outlined here are expected to combine to allow decorators to be implemented as a normal class.

Examples

Merging Example

Current instance defines common_method as (this could be in some already-applied decorator):

class Foo:
    def common_method(self):
        return 42

and the decorator being applied defines common_method as:

class Bar:
    def common_method(self):
        return 50

If we want the merge to use Bar’s definition of common_method over any other definition (as a way to simulate inheritance) we could provide the following function as the value of conflict_handler:

def handle_conflict(entry_name, decorator_value, instance_value):
    if entry_name == 'common_method':
        return Bar.common_method
    raise IncorrectConflictHandlerError

__annotations__ = {}

__init__()

Initialize an empty list of decorators and also a list of methods to skip.

decorators

A set of decorators (initially empty).

Type:

set

skip_set

A set of methods to skip/ignore. Entries are in this list for various reasons. All are “built-ins” in Python classes. None are of type types.BuiltinMethodType, since we can skip those more generically. Some are dictionaries, others are type type, and some are method_descriptor’s. A method_descriptor defines some object that is not implemented in Python source (e.g., it is in C). We do not want to skip all of these, because we may actually want to merge some objects that are implemented in C (so we put them in the skip list by name instead). Reference: https://stackoverflow.com/q/15512183. There are also wrapper_descriptor’s, with similar reasoning to method_descriptors. Primarily, this list includes most built-in methods that FIREWHEEL users are unlikely to use and, therefore, can be ignored as they will always be left alone. By ignoring these built-in methods we can greatly improve performance of decorating a Vertex. Notably, the only built-in methods which we expect users to modify are __str__ and __repr__. Perhaps a starting document is: https://docs.python.org/3/howto/descriptor.html

Type:

set

cached_self_dir

A set of methods/attributes for the current object. by initializing this set in the __init__ function and then adding to it with each decoration, we can reduce the number of calls to dir(), which is an expensive operation.

Type:

set

decorate(decorator_class, init_args=None, init_kwargs=None, conflict_handler=None)

Each graph Vertex and Edge are a layered stack of Python objects where each layer is a python decorator. Class functions can then be called on all the objects. Using this methodology, model component objects can be mixed and matched to create the desired Vertex/Edge. This creates an easy way to build complicated experiments with modular code. Decoration can be used to:

• Provide specific Vertex/Edge functions.

• Specify images or VM Resources

• Leverage software/configuration available in other Model Components.

Here is an example of a potential model component object stack:

Vertex("host.acme.com")
|
-> VMEndpoint()
|   - run_executable()
|   - drop_file()
|   - connect()
|
--> LinuxHost()
|    - configure_ips()
|    - set_hostname()
|
---> UbuntuHost()
|     - install_debs()
|
----> Ubuntu1604Server()

The Vertex starts with only a name, but as it is decorated by various other objects, it gains new properties and methods which can be leveraged.

There are two ways to build up this model component object stack. You can use explicit decoration, in which all components are added explicitly, as shown below:

server = Vertex("host.acme.com")
server.decorate(VMEndpoint)
server.decorate(LinuxHost)
server.decorate(UbuntuHost)
server.decorate(Ubuntu1604Server)

Alternatively, you can use dependency decoration and only call decorate on the last object in the stack (assuming it uses @require_class on the previous layers):

server = Vertex("host.acme.com")
server.decorate(Ubuntu1604Server)

The decorate method is necessary to decorate graph Vertices or Edges with additional model component objects. It is important to note that we make class attributes into instance attributes (as the general case for data and functions). This could impact behavior, especially for assumptions with respect to class data attributes.

Additionally, we expect that users will NOT modify most built-in methods which were defined in the skip_set. We only assume that users have potentially modified __str__ and __repr__. If either of these methods has been modified (e.g. not equal to the default object implementation, then we will need a conflict handler.

Note

We are using getattr() rather than inspect.getattr_static(). If a MC Object sets either __str__ or __repr__ using Python descriptors, that code will be executed and may differ. This decision was made because 1) The chances of using descriptor to set one of these methods is rare when creating a FIREWHEEL Model Component Object and 2) getattr() is significantly faster than inspect.getattr_static(). The performance hit from using inspect is dramatic when you consider each Vertex may be decorated numerous times and there may be thousands of Vertices/Edges in the graph.

Here is a timeit comparison:

>>> import timeit
>>> timeit.timeit("getattr(object, '__str__')")
0.1993947122246027
>>> timeit.timeit("inspect.getattr_static(object, '__str__')", setup="import inspect")
3.008564186282456

This method also uses a module-level dictionary of all the MC objects and their methods/attributes (CACHED_DECORATOR_OBJECTS). This is a performance improvement as dir() is an expensive operation and will no longer need to be called for each instance of the same object. This will only have minor memory impacts as most experiments use only a handful of different MC Objects types.

Parameters:

• decorator_class (object) – The model component object which will decorate the Vertex/Edge.

• init_args (list) – Any initial arguments required by the decorator_class.

• init_kwargs (dict) – Any initial keyword arguments required by the decorator_class.

• conflict_handler (func) – A conflict handler callable. This is described in more detail in the class documentation.

Raises:

• TypeError – If decorator_class is not a class.

• DecoratorConflictError – If there is a conflict between two objects in the stack.

is_decorated_by(decorator_class)

Check if a Vertex/Edge is decorated by a particular class.

Parameters:

decorator_class (Object) – The model component object to check against.

Returns:

True if the Vertex/Edge was decorated by the passed in class, False otherwise.

Return type:

bool

exception firewheel.control.experiment_graph.IncorrectConflictHandlerError

This exception is intended to be thrown by decorator conflict handlers when the entry name is not one that the handler is intended to handle.

exception firewheel.control.experiment_graph.NoSuchVertexError

Occurs when the Vertex does not exist.

class firewheel.control.experiment_graph.Vertex(graph, name=None, graph_id=None)

This class represents a FIREWHEEL-specific Vertex. It inherits from ExperimentGraphDecorable and implements all the expected methods for a networkx Node.

__annotations__ = {}

__contains__(key)

Check if the Vertex has a particular attribute.

Parameters:

key (str) – The attribute to query.

Returns:

Tue, if the key exists, False otherwise.

Return type:

bool

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

__delitem__(key)

Remove a Vertex attribute based on a key.

Parameters:

key (str) – An attribute/property to delete.

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

__eq__(other)

Determine if two Vertex instances are the same. Equality is based on having the same graph and the same graph_id. This function also verifies that itself and another are of type Vertex and are Vertex.valid.

Parameters:

other (Vertex) – The other Vertex.

Returns:

True if they are the same, False otherwise.

Return type:

bool

Raises:

RuntimeError – If either Vertex is not Vertex.valid.

__ge__(other)

Determine if a Vertex is greater than or equal to another Vertex. The comparison is based on the Vertex.graph_id. This function also verifies that itself and another are of type Vertex and are Vertex.valid.

Parameters:

other (Vertex) – The other Vertex.

Returns:

True if self is greater than or equal to other, False otherwise.

Return type:

bool

Raises:

RuntimeError – If either Vertex is not Vertex.valid.

__getitem__(key)

Dictionary-style access to read Vertex attributes.

Parameters:

key (str) – A key to query.

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

Returns:

The value of the requested attribute.

__gt__(other)

Determine if a Vertex is greater than another Vertex. The comparison is based on the Vertex.graph_id. This function also verifies that itself and another are of type Vertex and are Vertex.valid.

Parameters:

other (Vertex) – The other Vertex.

Returns:

True if self is greater than other, False otherwise.

Return type:

bool

Raises:

RuntimeError – If either Vertex is not Vertex.valid.

__hash__()

Get the hash of a tuple containing the ExperimentGraph and the Vertex.graph_id.

Returns:

The hash of a tuple containing the ExperimentGraph and the Vertex.graph_id.

Return type:

int

__init__(graph, name=None, graph_id=None)

Initialize the Vertex.

g

The graph in which the Vertex should exist.

Type:

ExperimentGraph

graph_id

The integer representation of the Vertex.

Type:

int

valid

If the Vertex should exist in the graph (i.e., has not yet been deleted).

Type:

bool

log

The logger to use for this Vertex.

Type:

firewheel.lib.log.Log

name

The name of the Vertex.

Type:

str

Parameters:

• graph (ExperimentGraph) – The in which the Vertex should exist.

• name (str) – The name of the Vertex.

• graph_id (int) – The integer representation of the Vertex.

__iter__()

Iterate over the Vertex.

Returns:

An iterator that iterates over the Vertex properties.

__le__(other)

Determine if a Vertex is less than or equal to another Vertex. The comparison is based on the Vertex.graph_id. This function also verifies that itself and another are of type Vertex and are Vertex.valid.

Parameters:

other (Vertex) – The other Vertex.

Returns:

True if self is less than or equal to other, False otherwise.

Return type:

bool

Raises:

RuntimeError – If either Vertex is not Vertex.valid.

__lt__(other)

Determine if a Vertex is less than another Vertex. The comparison is based on the Vertex.graph_id. This function also verifies that itself and another are of type Vertex and are Vertex.valid.

Parameters:

other (Vertex) – The other Vertex.

Returns:

True if self is less than other, False otherwise.

Return type:

bool

Raises:

RuntimeError – If either Vertex is not Vertex.valid.

__ne__(other)

Determine if two Vertex instances are not equal. Inequality is based on the opposite of the Vertex.__eq__() method. This function also verifies that itself and another are of type Vertex and are Vertex.valid.

Parameters:

other (Vertex) – The other Vertex.

Returns:

True if they are not the same, False otherwise.

Return type:

bool

Raises:

RuntimeError – If either Vertex is not Vertex.valid.

__setitem__(key, value)

Dictionary-style access to set a Vertex attributes.

Parameters:

• key (str) – An attribute key.

• value (Any) – The value of the attribute.

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

__str__()

Provide a nicely formatted string describing the Vertex.

Returns:

A nicely formatted string describing the Vertex.

Return type:

str

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

delete()

Remove this Vertex from the ExperimentGraph. This sets the Vertex.valid property to False.

get_degree()

Get the degree of the Vertex. That is, the number of Edges that are incident to the Vertex.

Returns:

The degree of the Vertex.

Return type:

int

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

get_neighbors()

Get an iterator of all of this Vertex’s neighbors.

Returns:

All the Vertex instances that this Vertex is connected to.

Return type:

VertexIterator

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

get_object()

Get the Vertex object attribute (i.e. self).

Returns:

This Vertex.

Return type:

Vertex

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

has(key)

Dictionary-style query for the presence of a key.

Parameters:

key (str) – A key to query.

Returns:

If the key exists.

Return type:

bool

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

keys()

Dictionary-style access to get a list of Vertex attribute keys.

Returns:

A list of Vertex attributes.

Return type:

list

Raises:

RuntimeError – If the Vertex is not Vertex.valid.

vertex_log = <Logger ExperimentGraphVertex (DEBUG)>

class firewheel.control.experiment_graph.VertexIterator(graph, vertex_iterable)

A custom Vertex iterator which parses the graph and returns the next Vertex.

__init__(graph, vertex_iterable)

Set up the Vertex iterator.

Parameters:

• graph (ExperimentGraph) – The experiment graph to iterate over.

• vertex_iterable – An iterable that provides Vertex instances.

g

The experiment graph to iterate over.

Type:

ExperimentGraph

vertex_iter

The iterator over the Vertex instances.

Type:

iterator

__iter__()

Return this VertexIterator.

Returns:

Returns this VertexIterator.

Return type:

VertexIterator

__next__()

Get the next Vertex in the graph.

Returns:

The next Vertex in the graph.

Return type:

Vertex

Raises:

NoSuchVertexError – If the Vertex ID was not found in the graph.

class firewheel.control.experiment_graph.require_class(required_decorator, conflict_handler=None)

A Python decorator to express a dependency from one decorator to another.

Example:

class VMEndpoint(object):
    ...

# A GenericRouter is always a VMEndpoint
@require_class(VMEndpoint)
class GenericRouter(object):
    ...

__call__(graph_object)

Call self as a function. This takes in the graph object to be decorated and ensures that the decoration happens.

Parameters:

graph_object (Object) – The graph object to be decorated.

Returns:

The newly decorated graph object.

Return type:

Object

__init__(required_decorator, conflict_handler=None)

Initialize the required decorator.

Parameters:

• required_decorator (Object) – The class which must decorate the new model component object.

• conflict_handler – A conflict handler callable. This is described in more detail in the class documentation.

model_component.py

class firewheel.control.model_component.ModelComponent(name=None, path=None, repository_db=None, arguments=None, vm_resource_store=None, image_store=None, install=None)

This class defines a Model Component which is the building block for FIREWHEEL experiments.

__eq__(other)

Determine if two model components are the same. Equality is based on having the same name and the same path. This function also verifies that itself and another are not None as that would cause issues.

Parameters:

other (ModelComponent) – The other model component.

Returns:

True if they are the same, False otherwise.

Return type:

bool

__init__(name=None, path=None, repository_db=None, arguments=None, vm_resource_store=None, image_store=None, install=None)

Constructor. Allows specification of various database objects, typically used for testing. Must specify either name or path. If both are specified, name must match the MANIFEST at path.

Parameters:

• name (str) – The name of this ModelComponent. Corresponds to the “name” property of the MANIFEST.

• path (str) – The path to this ModelComponent, specifically the directory containing the MANIFEST file.

• repository_db (RepositoryDb) – A RepositoryDb object. If not given, will use the default RepositoryDb constructor.

• arguments (dict) – A dictionary with a ‘plugin’ key. The value of this key is itself a dictionary, with a format specified by ModelComponentManager. Keyword arguments use key/value pairs in the dict. Positional arguments use the empty string ('') as a key, and may be a single value or a list of values.

• vm_resource_store (firewheel.vm_resource_manager.vm_resource_store.VmResourceStore) – A VmResourceStore object. If not given, will use the default VmResourceStore constructor.

• image_store (firewheel.control.image_store.ImageStore) – An ImageStore object. If not given, will use the default ImageStore constructor.

• install (bool) – Whether or not to install the model component. If True, the MC will be installed automatically, and if False, the MC will not be installed. If left as None, the user will be prompted about whether or not the MC should be installed via the INSTALL script.

Raises:

• ValueError – Caused if a user didn’t specify name or path.

• ValueError – Caused if the name and manifest name do not match.

• ValueError – Caused if the arguments dictionary is malformed.

__ne__(other)

Determine if two model components not equal. In this case we are using the inverse of __eq__.

Parameters:

other (ModelComponent) – The other model component.

Returns:

True if they are not the same, False otherwise.

Return type:

bool

__str__()

Provide a nicely formatted string describing the ModelComponent. The string provides a pretty-printed MANIFEST, the path, and the Dependency Graph ID.

Returns:

A nicely formatted string describing the ModelComponent

Return type:

str

_load_manifest(path)

Try to get the path to the ModelComponents MANIFEST file.

Parameters:

path (str) – The path from where to load the MANIFEST file.

Returns:

The full path to the MANIFEST file.

Return type:

string

Raises:

RuntimeError – If the MANIFEST file does not exist or if the MANIFEST is malformed (i.e. not valid JSON).

_resolve_path()

Try to find the path for the current model component by iterating through all model components searching for the one whose name matches. Once a match is found the manifest and path attributes are set.

Raises:

ValueError – If it cannot find the model component.

_upload_images()

Upload all image files from the manifest.

Raises:

MissingImageError – If the image is not found in the model component.

Returns:

Actions for each specified file. Order is sequential, proceeding through images, for each image proceed through each specified file before moving to next image. Possible actions are:

• no_date – There was no upload date for the given file in the

  ImageStore. It was uploaded.

• new_hash – The modified time of the file on disk differs from the

  last upload time in the ImageStore and the hashes did not match. File was uploaded.

• same_hash – The file on disk was modified after the upload time in

  the ImageStore but the hashes are the same. File was not uploaded.

• False – None of the other conditions occurred. For example, the file

  on disk was modified before the ImageStore upload time

Return type:

list

_upload_vm_resource(resource)

Upload a file to the VmResourceStore. It interrupts the path of the VM resources in the following way:

• Non-recursive all dir’s files: path_to_dir, path_to_dir/, or path_to_dir/*

• Non-recursive all dir’s files matching pattern: path_to_dir/*.ext

• Recursive all files: path_to_dir/**, path_to_dir/**/, or path_to_dir/**/*

• Recursive all files matching pattern: path_to_dir/**/*.ext

Raises:

MissingVmResourceError – if the given file path does not exist, or its modification time cannot be determined.

Parameters:

resource (str) – Path relative to this component’s root to the file being uploaded.

Returns:

Indication of what happened. This may be one of:

no_date – There was no upload date for the given file in the

VmResourceStore. It was uploaded.

new_hash – The modified time of the file on disk differs from the

last upload time in the VmResourceStore and the hashes did not match. File was uploaded.

same_hash – The file on disk was modified after the upload time in

the VmResourceStore but the hashes are the same. File was not uploaded.

False – None of the other conditions occurred. For example, the file

on disk was modified before the VmResourceStore upload time

Return type:

str

_upload_vm_resources()

Upload all VM resources from the manifest. It interrupts the path of the VM resources in the following way:

• Non-recursive all dir’s files: path_to_dir, path_to_dir/, or path_to_dir/*

• Non-recursive all dir’s files matching pattern: path_to_dir/*.ext

• Recursive all files: path_to_dir/**, path_to_dir/**/, or path_to_dir/**/*

• Recursive all files matching pattern: path_to_dir/**/*.ext

Returns:

True if any resource was uploaded, False otherwise.

Return type:

bool

Raises:

RuntimeError – If the vm_resources field in the MANIFEST is not a list.

get_attribute_depends()

Get the attributes depends block from the manifest.

Returns:

Contains the attributes depends list or an empty list if there are no depends attributes.

Return type:

list

get_attribute_precedes()

Get the attributes precedes block from the manifest.

Returns:

Contains the attributes precedes list or an empty list if there are no preceded attributes.

Return type:

list

get_attribute_provides()

Get the attributes provides block from the manifest.

Returns:

Contains the attributes provides list or an empty list if there are no provides attributes.

Return type:

list

get_attributes()

Get the attributes block from the manifest.

Returns:

Contains both the attributes depends, provides and precedes lists.

Return type:

tuple

get_dependency_graph_id()

Get the dependency graph ID.

Returns:

The dependency graph ID.

Return type:

int

get_model_component_depends()

Get the model component’s dependency list.

Returns:

The model components dependencies.

Return type:

list

get_model_component_objects_path()

Try to get the path to the ModelComponents model_components_objects file.

Returns:

The full path to the model_component_objects file or an empty string if an error occurred.

Return type:

string

Raises:

RuntimeError – If the model_component_objects file does not exist or is a valid path but not a file.

get_model_component_precedes()

Get the model component’s precedes list.

Returns:

The model components preceded model components.

Return type:

list

get_plugin_path()

Try to get the path to the ModelComponents plugin file.

Returns:

The full path to the plugin file or an empty string if an error occurred.

Return type:

string

Raises:

RuntimeError – If the plugin does not exist or is a valid path but not a file.

set_dependency_graph_id(new_id)

Set the dependency graph ID.

Parameters:

new_id (int) – The ID that will become the dependency graph ID.

upload_files()

Upload any VM Resources and Images needed for the experiment to the cache.

model_component_install.py

class firewheel.control.model_component_install.InstallPrompt(prompt: str | Text = '', *, console: Console | None = None, password: bool = False, choices: List[str] | None = None, case_sensitive: bool = True, show_default: bool = True, show_choices: bool = True)

A prompt that has a custom error message.

__annotations__ = {}

__orig_bases__ = (rich.prompt.PromptBase[str],)

__parameters__ = ()

illegal_choice_message = "[prompt.invalid.choice]Please select one of the available options:\ny - yes, install execute this script\nn - no, do not execute this script\nv - view, see the script text\nvc - view color, see the script text with color support, must usea system pager which supports this behavior (e.g. PAGER='less -R')\nq - quit, exit immediately\n"

msg = "[prompt.invalid.choice]Please select one of the available options:\ny - yes, install execute this script\nn - no, do not execute this script\nv - view, see the script text\nvc - view color, see the script text with color support, must usea system pager which supports this behavior (e.g. PAGER='less -R')\nq - quit, exit immediately\n"

response_type

alias of str

validate_error_message = "[prompt.invalid.choice]Please select one of the available options:\ny - yes, install execute this script\nn - no, do not execute this script\nv - view, see the script text\nvc - view color, see the script text with color support, must usea system pager which supports this behavior (e.g. PAGER='less -R')\nq - quit, exit immediately\n"

class firewheel.control.model_component_install.ModelComponentInstall(mc=None)

Some Model Components may provide an additional install script called INSTALL which can be executed to perform other setup steps (e.g. installing an extra python package or downloading an external VM resource). This class helps execute that file and install a Model Component. INSTALL scripts can be can be any executable file type as defined by a shebang line.

Warning

The execution of Model Component INSTALL scripts can be a DANGEROUS operation. Please ensure that you fully trust the repository developer prior to executing these scripts.

See also

See Model Component INSTALL file for more information on INSTALL scripts.

When installing a Model Component, users will have a variety of choices to select:

• y - yes, install execute this script

• n - no, do not execute this script

• v - view, see the script text

• vc - view color, see the script text with color support, must use a system pager which supports this behavior (e.g. PAGER='less -R')

• q - quit, exit immediately

__init__(mc=None)

Initialize the object.

Parameters:

mc (ModelComponent) – The ModelComponent to install.

Raises:

ValueError – Caused if a user didn’t specify mc.

install_mc(name, install_path)

Execute a given Model Component’s INSTALL script.

Parameters:

• name (str) – The name of the Model Component.

• install_path (pathlib.Path) – The path of the INSTALL file.

Returns:

True if the MC was installed successfully (or it was already installed), False otherwise

Return type:

bool

run_install_script(insecure=False)

Ask the user to run the install script for the given Model Component.

Parameters:

insecure (bool) – Whether to automatically install the Model Component without asking. Default is False.

Returns:

True if all MCs were installed successfully, False otherwise.

Return type:

bool

model_component_manager.py

exception firewheel.control.model_component_manager.InvalidDefaultProviderError

This exception is caused if a given default provider is not valid.

exception firewheel.control.model_component_manager.InvalidStateError

This is caused if the dependency graph is in an invalid state.

class firewheel.control.model_component_manager.ModelComponentManager(repository_db=None, attribute_defaults_config=None)

This class manages creating the dependency graph for a given experiment. It ensures that all the constraints (dependencies) are met by model components.

__init__(repository_db=None, attribute_defaults_config=None)

Initialize class variables.

Parameters:

• repository_db (RepositoryDb) – Users can provide a different repository database.

• attribute_defaults_config (dict) – A set of default attributes to use when selecting model components.

_dependency_cycle_handler()

The dependency graph had cycles so we should retrieve those cycles and alert the user.

Raises:

UnsatisfiableDependenciesError – Output the cycles in the graph.

_import_model_component_objects(path, mc_name)

This method imports a model components objects file so that it can be added to the sys.modules path of the experiment.

Parameters:

• path (str) – The path of the model_component_objects file.

• mc_name (str) – The name of the model component.

Raises:

• ImportError – If the file cannot be found or if the model component has already been imported.

• ModelComponentImportError – If an ImportError occurs when executing the Model Component.

_import_plugin(path, mc_name)

This method imports a model components plugin file so that it can be executed.

Parameters:

• path (str) – The path of the Plugin file.

• mc_name (str) – The name of the model component.

Returns:

The plugin class which will be run.

Return type:

AbstractPlugin

Raises:

• ImportError – If the file cannot be found or if there are multiple plugins in the module.

• ModelComponentImportError – If the Model Component has an import error.

_print_plugin_initialization_help(mc_name, plugin_instance, call_args, call_kwargs)

Print helpful information when plugin initialization fails.

Parameters:

• mc_name (str) – The model component name.

• plugin_instance (AbstractPlugin) – The instance of the MC’s Plugin.

• call_args (list) – A list of the arguments passed to the Plugin.

• call_kwargs (dict) – A dictionary of the key word arguments passed to the Plugin.

build_dependency_graph(initial_component_list, install_mcs=None)

This is the primary method which generates the dependency graph.

Parameters:

• initial_component_list (list) – An initial list of model components which need to be added to the graph.

• install_mcs (bool) – A flag indicating whether to install model components automatically. By default, this method will defer to the default defined by the model component object’s constructor. If set to False, model components will not be installed.

Raises:

RuntimeError – If there is an infinite loop building the graph.

build_experiment_graph()

Builds the experiment graph by processing all the model components

Returns:

A list of errors that were reported when trying to execute.

Return type:

list

Raises:

InvalidStateError – If the dependency graph does not exist.

check_list_ordering(cur_mc_list, parent, component)

This method verifies that a given parent is before a given component in the dependency graph.

Parameters:

• cur_mc_list (list) – The list of model components in which the parent and component are located.

• parent (str) – The name of the parent model component.

• component (str) – The name of the child model component.

Raises:

ValueError – If either parent or component are not found in the cur_mc_list.

Returns:

True if the ordering is correct, False if it is not.

Return type:

bool

get_default_component_for_attribute(attribute, install_mcs=None)

Get the default model component which provides a given attribute. We first, check for a single model component installed that provides the attribute. If more than one is found, attribute_defaults is checked.

Parameters:

• attribute (str) – The attribute which a model component needs to provide.

• install_mcs (bool) – A flag indicating whether to install model components automatically. By default, this method will defer to the default defined by the model component object’s constructor. If set to False, model components will not be installed.

Returns:

The model component which provides the attribute.

Return type:

ModelComponent

Raises:

• InvalidDefaultProviderError – If there is an error with the model component which should provide the attribute or the defaults table.

• NoDefaultProviderError – If no default provider was found but it is necessary for there to be one.

get_ordered_model_component_list()

Get an ordered list of model components from the dependency graph.

Returns:

The list of model components.

Return type:

list

Raises:

InvalidStateError – If the dependency graph does not exist.

process_model_component(mc, experiment_graph)

This method helps process model components for execution. It: * Uploads/prepares any necessary files (images/vm_resources). * Loads any model component objects, if they exist. * Loads and runs the plugin, if it exists.

Parameters:

• mc (ModelComponent) – The model component to process.

• experiment_graph (ExperimentGraph) – The experiment graph. This is passed to the plugin and also returned by this method.

Returns:

Tuple containing a bool of whether errors occurred and the experiment graph.

Return type:

tuple

Raises:

TypeError – If the Plugin doesn’t run due to issues with passed-in arguments.

exception firewheel.control.model_component_manager.NoDefaultProviderError

An exception that is thrown if there is more than one provider for a given model component attribute but no default provider is defined in attribute_default_config.

dependency_graph.py

class firewheel.control.dependency_graph.DependencyGraph

A dependency graph. The graph consists of two vertex types: entities and constraints. Each entity may depend on a set of constraints and provide a set of constraints. Entities may additionally be “associated” or ordered among other entities. The dependencies represented may be satisfied as long as there are no cycles in the graph.

The end goal of the data structure is to produce a list of entities in a canonicalized order which satisfies all constraints. This is done with the get_ordered_entity_list() method.

Entities are represented using an arbitrary identifier which is returned to the caller when they are created. Constraints are represented using strings (names).

__init__()

Initialize the dependency graph by creating a new networkx DiGraph.

associate_entities(source, dest)

Associate two entities with a directional relationship.

Parameters:

• source (int) – Identifier of the entity at the source of the relationship.

• dest (int) – Identifier of the entity at the destination of the relationship.

Raises:

InvalidNodeError – If the node is not a valid type or not found.

get_graph_json()

Return a JSON formatted string representation of the graph. The representation is based on the D3 nodes-links format with links referencing node ID rather than list position.

Returns:

Has the following format:

{
    "nodes": [
        {"id": <id>, "type": <"entity" or "constraint">},
        ...
    ],
    "links": [
        {"source": <id>, "target": <id>},
        ...
    ],
    "graph": {},
    "directed": True,
    "multigraph": False
}

Return type:

str

get_in_degree_zero_constraints()

Retrieve a list of all constraints that have an in-degree of zero.

Returns:

Constraint IDs for constraint vertices with in-degree zero.

Return type:

list

get_ordered_entity_list()

Return a list of entities in dependency-valid canonical order. For a given graph, the order returned is always the same–lexicographical order is used.

Returns:

Entity IDs in canonical, dependency-satisfying order.

Return type:

list

Raises:

UnsatisfiableDependenciesError – Occurs if there are cycles.

has_cycles()

Determine if cycles exist in the graph.

Returns:

True if cycles exist.

Return type:

bool

insert_entity(depends, provides, grouping)

Add an entity to the graph with associated constraints.

Parameters:

• depends (list) – Iterable of constraint names the entity depends on.

• provides (list) – Iterable of constraint names the entity provides.

• grouping (int) – The grouping of model components’ dependencies.

Returns:

Identifier for the created entity.

Return type:

int

topological_compare(node)

Comparison function for the ‘key’ parameter of networkx’s lexicographical_topological_sort function.

The node IDs are a mix of strings and integers and the value for grouping is not unique, therefore simply comparing the grouping value is not enough information to make a unique sorting decision. Therefore, prepend the grouping ID to the node ID so that when the grouping ID is the same, the comparison returns to the default of comparing the node IDs as strings.

Parameters:

node (int) – A node ID from the graph

Returns:

A unique key that can be used for sorting nodes

Return type:

str

exception firewheel.control.dependency_graph.InvalidNodeError

Exception thrown if the Node is not found

class firewheel.control.dependency_graph.TopologicalCompare(grouping, node)

A class to enable custom sorting when using networkx’s lexicographical_topological_sort. Nodes are first sorted based on their model component group number (integer comparison). If those are equal, we sort based on their node ID (str comparison).

Note

Only the __lt__ function is needed for sorting, not __gt__.

__init__(grouping, node)

Initialize the comparison object.

Parameters:

• grouping (int) – The grouping of model components’ dependencies.

• node (str|int) – The node ID

__lt__(other)

Function to perform custom comparison for < operator. Nodes are first sorted based on their model component group number (integer comparison). If those are equal, we sort based on their node ID (str comparison).

Parameters:

other (TopologicalCompare) – The other node to compare to.

Returns:

True if self is less than other.

Return type:

bool

exception firewheel.control.dependency_graph.UnsatisfiableDependenciesError

Exception thrown if there are dependencies that cannot be met.

model_component_dependency_graph.py

class firewheel.control.model_component_dependency_graph.ModelComponentDependencyGraph(*args, **kwargs)

This class provides a specific implementation of DependencyGraph() which is made for tracking Model Component dependencies.

__annotations__ = {}

__init__(*args, **kwargs)

Initialize DependencyGraph() and our logger.

Parameters:

• *args – Positional arguments that will be passed into DependencyGraph().

• **kwargs – Keyword arguments that will be passed into DependencyGraph().

associate_model_components(prev_component, component)

Create a relationship between two model components in the graph. A directional edge will be created like: component —> prev_component. A component may not have actually been added to the dependency graph (for example it was already present). In this case, make sure to fill in some missing info we need.

Parameters:

• prev_component (ModelComponent) – The predecessor in the graph.

• component (ModelComponent) – The current node.

Raises:

InvalidNodeError – If the dependency graph ID for one of the inputs is not found.

count_model_component_occurrences(model_component)

Count the number of times a model component occurs in the component_map.

Parameters:

model_component (ModelComponent) – The model component to count.

Returns:

The number of ModelComponent’s of the same name are present in this dependency graph.

Return type:

int

get_cycles()

Try to identify all the cycles in the DiGraph that could be created by a user. These errors are then logged.

Returns:

A list of cycles created by a user.

Return type:

list

get_first(model_component)

Get the first time a model component is found in the ordered list of dependencies.

Parameters:

model_component (ModelComponent) – The model component to find.

Returns:

The instance of the ModelComponent which matches model_component.name. This is None if none are found.

Return type:

ModelComponent

get_ordered_entity_list()

Return a list of entities in dependency-valid canonical order. For a given graph, the order returned is always the same–lexicographical order is used.

Returns:

ModelComponents in canonical, dependency-satisfying order.

Return type:

list

get_ordered_entity_list_with_grouping()

Return a list of tuples (entity, grouping) in dependency-valid canonical order. For a given graph, the order returned is always the same–lexicographical order is used.

Returns:

(ModelComponent, grouping) in canonical, dependency-satisfying order.

Return type:

list

insert(model_component, grouping, duplicate=False)

Insert a ModelComponent into the graph.

Parameters:

• model_component (ModelComponent) – The model component to insert.

• grouping (int) – The model component’s grouping.

• duplicate (bool) – If the model component is a duplicate.

Returns:

True if insertion was successful, False otherwise.

Return type:

bool

Raises:

ValueError – If the passed in model_component is not an instance of ModelComponent().

image_store.py

class firewheel.control.image_store.ImageStore(store: str = 'images', decompress: bool = True)

A repository for VM images that uses the minimega FileStore for easy access on all hosts in a Firewheel cluster.

__init__(store: str = 'images', decompress: bool = True) → None

Initialize the ImageStore.

Parameters:

• store (str) – The relative path from the minimega files directory for this FileStore. Defaults to “images”.

• decompress (bool) – Whether to decompress files by default when using this FileStore. Defaults to True.

repository_db.py

class firewheel.control.repository_db.RepositoryDb(host='', port=50051, db='prod')

Provides an interface to the Repository database on the gRPC service.

Repositories in the database are uniquely identified by their path.

A repository, as stored in the database, is expected to have the form:

{
    "path": ""
}

__init__(host='', port=50051, db='prod')

Set up the connection to the gRPC server.

Parameters:

• host (str) – The GRPC server IP/hostname.

• port (int) – The GRPC server port.

• db (str) – The GRPC database.

_validate_repository(repository)

Validate a given repository has the correct format

Parameters:

repository (dict) – A repository dictionary to add. See format for the database.

Raises:

• KeyError – if there is a missing key/val pair in the repository.

• FileNotFoundError – If the repository does not exist.

• PermissionError – If the user is unable to access that directory.

add_repository(repository)

Add a new repository entry to the database.

Parameters:

repository (dict) – A repository dictionary to add. See format for the database.

close()

Close the database connection.

delete_repository(repository)

Delete a repository entry from the database.

Parameters:

repository (dict) – A repository dictionary to delete.

Returns:

Number of entries removed (expect 0 or 1), or None if an error occurred.

Return type:

int

list_repositories()

List the repositories in the database.

This method will list all repositories in the database and check for any installed Python packages that provide the firewheel.mc_repo entry point.

Returns:

An iterator, where each dictionary is a repository.

(See repository format for the database).

Return type:

list

model_component_iterator.py

class firewheel.control.model_component_iterator.ModelComponentIterator(repositories)

This class iterates over the various repositories looking for model components.

__init__(repositories)

Initialize the path iterator.

Parameters:

repositories (list_iterator) – The list of repositories.

model_component_path_iterator.py

class firewheel.control.model_component_path_iterator.ModelComponentPathIterator(repositories)

This class takes in a repository iterator and enables searching over all the repositories for model components and their specific fully qualified paths.

__init__(repositories)

Initialize the class variables and the current position of the iterator.

Parameters:

repositories (list_iterator) – The list of repositories.

__next__()

A custom __next__ method to get the next ModelComponentPathIterator. It resets all the class variables and returns a new instance.

Returns:

The next MCPI with a different repository.

Return type:

ModelComponentPathIterator

Raises:

StopIteration – If there are no more repositories, this is desired behavior.

_is_path_model_component(path)

Check to see if the passed-in directory is a model component. The condition for being a model component in this context is if a MANIFEST file exists.

Parameters:

path (str) – Path of the potential model component.

Returns:

True if a MANIFEST file exists within the directory, False otherwise.

Return type:

bool

_walk_dir(path)

This is a helper method for walk_repository_for_model_component_paths(). It will walk the path and create a recursive list of directories with the repository which are model components.

Parameters:

path (str) – Path of the repository.

Returns:

A list of model component paths contained within the repository.

Return type:

list

walk_repository_for_model_component_paths(path)

Search each repository for model component paths.

Parameters:

path (str) – Path of the repository.

Returns:

A list of components contained within the repository.

Return type:

list

model_component_exceptions.py

This file holds various specific exceptions.

exception firewheel.control.model_component_exceptions.MissingImageError

This exception occurs when required image is not uploaded.

exception firewheel.control.model_component_exceptions.MissingRequiredVMResourcesError(vm_resources)

This exception occurs when required vm_resource(s) are not uploaded.

__init__(vm_resources)

Initialize exception.

Parameters:

vm_resources (list) – A list of vm_resources that are not uploaded.

__str__()

Return str(self).

exception firewheel.control.model_component_exceptions.MissingVmResourceError(path)

This exception occurs when a vm resource is not included in a model component.

__init__(path)

Initialize exception.

Parameters:

path (str) – The path is the path of the model component

__str__()

Return str(self).

exception firewheel.control.model_component_exceptions.ModelComponentImportError(mc_name, error)

This is caused if a plugin or model component objects file fail to import an object correctly. This is typically caused by importing a model component but neglecting to depend on the that model component in the MANIFEST file.

__init__(mc_name, error)

Initialize exception.

Parameters:

• mc_name (str) – The name of the model component which had the error.

• error (list) – The bottom few lines of the stack trace which caused this problem.

__str__()

Return str(self).

utils/new_model_component.py

This module implements automatic generation of ModelComponents in either interactive or argument-only modes. The generation includes the ModelComponent MANIFEST, documentation skeleton, and templates for Python modules.

This module may be run as a script in either mode. Also included are 2 related classes: PythonModule and ModelComponentGenerator.

firewheel.control.utils.new_model_component.ARGUMENT_PROMPT_SECTIONS = ['MANIFEST']

Defined which sections of arguments will be prompted for in interactive mode. If an argument is both required and outside of one of the sections in this list then it must be specified on the command line regardless of interactive or non-interactive mode being used.

Type:

list

firewheel.control.utils.new_model_component.ARG_DESCRIPTION = {'Configuration': {'--no_templates': {'action': 'store_true', 'default': False, 'dest': 'no_templates', 'help': 'Do not generate files from templates. Only generate a MANIFEST', 'nargs': 1, 'required': False, 'type': <class 'bool'>}, '--non-interactive': {'action': 'store_true', 'default': False, 'dest': 'non_interactive', 'help': 'Require minimum parameters as arguments and do not prompt for any values', 'nargs': 1, 'required': False, 'type': <class 'bool'>}, '--template_dir': {'action': 'store', 'default': 'templates', 'dest': 'template_dir', 'help': 'Override the configured templates directory', 'nargs': 1, 'required': False, 'type': <class 'str'>}}, 'MANIFEST': {'--arch': {'action': 'store', 'dest': 'arch', 'help': 'Architecture for specified image', 'nargs': 1, 'required': False, 'type': <class 'str'>}, '--attribute_depends': {'action': 'store', 'dest': 'attribute_depends', 'help': '(space-separated-strings) Graph Attribute(s) depended on by the new ModelComponent', 'nargs': '+', 'required': False, 'type': <class 'str'>}, '--attribute_precedes': {'action': 'store', 'dest': 'attribute_precedes', 'help': '(space-separated-strings) Graph Attribute(s) preceded by the new ModelComponent', 'nargs': '+', 'required': False, 'type': <class 'str'>}, '--attribute_provides': {'action': 'store', 'dest': 'attribute_provides', 'help': '(space-separated-strings) Graph Attribute(s) provided by the new ModelComponent', 'nargs': '+', 'required': False, 'type': <class 'str'>}, '--image': {'action': 'store', 'dest': 'image', 'help': 'File to be used as a VM disk', 'nargs': 1, 'required': False, 'type': <class 'str'>}, '--location': {'action': 'store', 'dest': 'location', 'help': 'Location for the new ModelComponent', 'nargs': 1, 'required': True, 'type': <class 'str'>}, '--model_component_depends': {'action': 'store', 'dest': 'model_component_depends', 'help': '(space-separated-strings) ModelComponent(s) required by name', 'nargs': '+', 'required': False, 'type': <class 'str'>}, '--model_component_objects': {'action': 'store', 'dest': 'model_component_objects', 'help': 'File for Model Component Objects', 'nargs': 1, 'required': False, 'type': <function python_file>}, '--model_component_precedes': {'action': 'store', 'dest': 'model_component_precedes', 'help': '(space-separated-strings) ModelComponent(s) that will be preceded by name', 'nargs': '+', 'required': False, 'type': <class 'str'>}, '--name': {'action': 'store', 'dest': 'name', 'help': 'ModelComponent name', 'nargs': 1, 'required': True, 'type': <class 'str'>}, '--plugin': {'action': 'store', 'dest': 'plugin', 'help': 'File for a plugin', 'nargs': 1, 'required': False, 'type': <function python_file>}, '--plugin_class': {'action': 'store', 'default': 'Plugin', 'dest': 'plugin_class', 'help': 'Name for the new plugin class', 'nargs': 1, 'required': False, 'type': <class 'str'>}, '--vm_resource': {'action': 'store', 'dest': 'vm_resources', 'help': '(space-separated-strings) File(s) to be used as a vm_resource', 'nargs': '+', 'required': False, 'type': <class 'str'>}}}

Declaration of the arguments and their sections. Structured here instead of pure ArgParse to enable the interactive/prompting functionality. All arguments are optional to ArgParse, but custom functionality enforces their use (so interactive mode works correctly). Type specification is also enforced in interactive mode using values specified here.

Each section of arguments is a top-level key in this dictionary:

{
    'SECTION': {
    }
}

Each section dictionary is made up of keys representing arguments:

'SECTION': {
    '--arg': {
    }
}

Finally, each argument dictionary defines some attributes:

'--arg': {
    'dest': 'template_dir',
    'type': str,
    'required': False,
    'nargs': 1,
    'help': 'Help text',
    'action': 'store',
    'default': 'DefaultValue'
}

Each of these keys is a keyword passed to the add_argument() method, with some notable exceptions:

• required is used by custom code, and the required keyword in add_argument has the value False.

• nargs is omitted from the call to add_argument if the value is 1.

• default is optional, and omitted from the call to add_argument is omitted from this dictionary.

Type:

dict

class firewheel.control.utils.new_model_component.ModelComponentGenerator(root_path, name, template_path='templates', strict_template_vars=True)

Represents the state and actions required to produce a new ModelComponent. Set values in properties first, then call creation method.

Variables:

• jinja_env (jinja2.Environment) – Jinja2 Environment capable of loading the templates used while generating this ModelComponent.

• readme_filename (str) – Name of the README file for this ModelComponent (default: “README.rst”)

• plugin_module (PythonModule) – Python module and associated API documentation for the plugin, if one exists in this ModelComponent.

• model_component_module (PythonModule) – Python module and associated API documentation for the model_component objects, if used in this ModelComponent.

__init__(root_path, name, template_path='templates', strict_template_vars=True)

Initialize several class variables.

Parameters:

• root_path (str) – The root path for the new model component.

• name (str) – The name of the model component.

• template_path (str) – The path to the Jinja templates.

• strict_template_vars (bool) – If we want to set self.jinja_env.undefined to StrictUndefined.

property arch

Architecture for a specified image This value can be x86_64, x86, etc.

Returns:

Architecture for a specified image (typically x86_64).

Return type:

str

Raises:

TypeError – Value is not a string

property attribute_depends

Provides list of strings which are graph attributes on which this ModelComponent depends.

Returns:

List of strings which are graph attributes on which this ModelComponent depends.

Return type:

list

Raises:

TypeError – Not all list members are strings (write), value is not a list of string (which is converted to a 1-element list) (write). # noqa: DAR402

property attribute_precedes

Provides list of strings which are graph attributes on which this ModelComponent precedes.

Returns:

List of strings which are graph attributes on which this ModelComponent precedes.

Return type:

list

Raises:

TypeError – Not all list members are strings (write), value is not a list of string (which is converted to a 1-element list) (write). # noqa: DAR402

property attribute_provides

Provides list of strings which are graph attributes which this ModelComponent provides.

Returns:

List of strings which are graph attributes which this ModelComponent provides.

Return type:

list

Raises:

TypeError – Not all list members are strings (write), value is not a list of string (which is converted to a 1-element list) (write). # noqa: DAR402

create_component(manifest_only=False)

Create this ModelComponent, based on the configuration of this object.

Parameters:

manifest_only (bool) – Only generate a MANIFEST file (default False).

create_install()

Create a INSTALL file. This is based on a template.

create_model_component_objects_module()

Create a Python module for the ModelComponent’s model_component objects.

Raises:

ValueError – If the Model Component objects extension is not ‘.py’.

create_plugin_module()

Create a Python module for the ModelComponent’s plugin.

Raises:

ValueError – If the plugin extension is not ‘.py’

create_readme()

Create a README file. This is based on a template if one is specified.

property image

VM disk file in this ModelComponent. Each entry is the relative path from the MANIFEST to the disk file.

Returns:

VM disk file in this ModelComponent.

Return type:

str

Raises:

TypeError – Value is not a string (automatically converted to a 1-element list of strings) or a list of strings (write).

property install_path

Relative path (including filename) to the INSTALL for this Model Component.

Returns:

Relative path (including filename) to the INSTALL for this ModelComponent. the default is derived from self.root_path and self.install_filename.

Return type:

str

property model_component_depends

List of ModelComponents this ModelComponent depends on, by name.

Returns:

List of strings containing ModelComponent names on which this ModelComponent depends.

Return type:

list

Raises:

TypeError – Not all list members are strings (write), value is not a list of string (which is converted to a 1-element list) (write). # noqa: DAR402

property model_component_objects

Relative path to the model_component objects file (including full file name).

Returns:

Relative path to the model_component objects file (including full file name).

Return type:

str

Raises:

• TypeError – Specified path is not a string (write). # noqa: DAR402

• ValueError – Specified path is not relative (write). # noqa: DAR402

property model_component_precedes

List of ModelComponents this ModelComponent precedes, by name.

Returns:

List of strings containing ModelComponent names on which this ModelComponent precedes.

Return type:

list

Raises:

TypeError – Not all list members are strings (write), value is not a list of string (which is converted to a 1-element list) (write). # noqa: DAR402

property name

Name of the ModelComponent, appears in the MANIFEST.

Returns:

Name of the ModelComponent, appears in the MANIFEST.

Return type:

str

Raises:

• AttributeError – No name defined (read).

• TypeError – Tried to set a non-string name (write).

property plugin

Relative path to the plugin file (including full file name).

Returns:

Relative path to the plugin file (including full file name).

Return type:

str

Raises:

• TypeError – Specified path is not a string (write). # noqa: DAR402

• ValueError – Specified path is not relative (write). # noqa: DAR402

property plugin_class

Name of the plugin class for this ModelComponent. This plugin class is defined in the plugin file (self.plugin), and inherits from firewheel.control.experiment_graph.AbstractPlugin.

Returns:

Name of the plugin class for this ModelComponent.

Return type:

str

Raises:

ValueError – Specified value is not a valid Python identifier (write). # noqa: DAR402

property readme_path

Relative path (including filename) to the README for this ModelComponent.

Returns:

Relative path (including filename) to the README for this ModelComponent. the default is derived from self.root_path and self.readme_filename.

Return type:

str

property vm_resources

List of vm_resources contained in this ModelComponent.

Each vm_resource is a relative path from the MANIFEST to the executable (vm_resource) file.

Returns:

List of strings of vm_resources contained in this ModelComponent.

Return type:

list

Raises:

TypeError – Value is not a string (automatically converted to a 1-element list of strings) or a list of strings (write). # noqa: DAR402

write_manifest()

Write a MANIFEST file based on this instance’s properties, including the creation of any necessary directory structure.

class firewheel.control.utils.new_model_component.PythonModule(jinja_env)

Template-based generation of a Python module and its associated documentation file. Use properties to set up the desired state, then call creation methods to realize the state on disk. Templates use Jinja2.

__init__(jinja_env)

Initialize several class variables.

Variables:

module_extension (str) – File extension for the Python module (default: “py”).

Parameters:

jinja_env (jinja2.Environment) – Jinja2 Environment instance capable of loading templates needed when creating this module.

Raises:

ValueError – If a jinja2.Environment is not provided.

property base_path

Provides the directory that serves as a base for all other relative paths.

Returns:

The base path.

Return type:

str

Raises:

ValueError – No path specified (read), or specified path is invalid (write), or unable to write to specified path (write).

create_module(other_vars=None)

Create the Python module, along with the required directory structure to support it. Use the specified template, or a generic TODO comment if a template has not been specified.

Parameters:

other_vars (dict) – Dictionary of additional template variables.

module_exists()

Determine if this instance’s specified module path exists on disk.

Returns:

True if self.module_path is a file on disk, False otherwise.

Return type:

bool

property module_name

Name of the Python module. No file extension (e.g “.py”) is used.

Returns:

The name of the Python module without a file extension.

Return type:

str

Raises:

ValueError – No module name specified (read), or specified name is not a valid Python module name (write).

property module_path

The full path (including file extension) to the Python module file.

Returns:

The full path (including file extension) to the Python module file.

Return type:

str

property module_relpath

Path to the Python module, relative to self.base_path.

Returns:

The path to the Python module, relative to the base path, or an empty string.

Return type:

str

Raises:

ValueError – Path specified is not relative (write).

property module_template

Jinja2 template name for the Python module template.

Returns:

Jinja2 template name for the Python module template.

Return type:

str

Raises:

ValueError – Jinja cannot load specified template (write).

firewheel.control.utils.new_model_component.check_and_prompt_args(args)

Used by interactive mode to fill in values for all prompt-able arguments.

Sets values in the args argument.

Parameters:

args (Namespace) – argparse.ArgumentParser’s argument parsing results.

firewheel.control.utils.new_model_component.check_required_args(args)

Determine which required arguments are missing.

Used by non-interactive mode to enforce required arguments.

Parameters:

args (Namespace) – argparse.ArgumentParser’s argument parsing results.

Returns:

A list of strings which are the required arguments that have not been specified.

Return type:

list

firewheel.control.utils.new_model_component.get_arg_parser()

Construct an argparse.ArgumentParser based on the ARG_DESCRIPTION dictionary.

Returns:

Argument parser instance.

Return type:

argparse.ArgumentParser

firewheel.control.utils.new_model_component.main()

Determine ModelComponent configuration and generate a new ModelComponent.

firewheel.control.utils.new_model_component.prompt_arg(arg_name, arg_dict)

Used by interactive mode to prompt for arguments that have no value specified.

Parameters:

• arg_name (str) – Name of the argument to prompt for.

• arg_dict (dict) – Dictionary specifying parameters for the argument. Uses ‘type’, ‘default’, ‘required’, ‘help’.

Returns:

Value(s) given by the user.

Return type:

list

firewheel.control.utils.new_model_component.python_file(value)

Function that can perform ArgParse type checking for python files. Enforces the value given ends with the string “.py”.

Parameters:

value (str) – The value to check.

Returns:

The value converted into a string.

Return type:

str

Raises:

argparse.ArgumentTypeError – Given value does not end in “.py”.

utils/vm_builder.py

class firewheel.control.utils.vm_builder.VMBuilder(vm_image, memory, vcpus)

Wrap our functionality in a class so we can keep track of the state we change in the OS (e.g. network interfaces) and clean up after ourselves.

Each instance of VMBuilder works with a single VM Image and instantiation of that image.

The functionality of this class is designed to be used within a “with” block, which allows us to track and free the resources used. The class is constructed at the opening of the with block, and when the with block terminates, all resources used (network tap, control socket, temp directory) have been destroyed and QEMU is no longer running. The code will not, however, protect you from yourself if you try to circumvent the with block. Don’t do this.

__enter__()

Begin “with” block. Initialize our resources.

Returns:

The current instance.

Return type:

VMBuilder

__exit__(exception_type, exception_value, exception_traceback)

End “with” block. Destroy our resources.

Parameters:

• exception_type (str) – Indicates class of exception.

• exception_value (str) – Indicates type of exception.

• exception_traceback (str) – Report all of the information needed to solve the exception.

__init__(vm_image, memory, vcpus)

Sets up for working with a VM image.

Parameters:

• vm_image (str) – Path to VM image (i.e. the QCOW2 file).

• memory (int) – The amount of memory for the VM.

• vcpus (int) – The number of VCPUs for the VM.

build_vm_network()

Create the tap device for the VM’s NIC.

get_vnc_port(vm_name)

Get the VNC port for a (presumed) running QEMU instance.

Parameters:

vm_name (str) – The name of the VM.

Returns:

The VNC port of the launched VM.

Return type:

int

Raises:

RuntimeError – When an error occurred when connecting to the VM.

launch_vm(network=False, snapshot=False, cdrom=None)

Launch a VM image. Used for modification or snapshot modes. Waits for the VM to terminate before returning.

Parameters:

• network (bool) – If True, give the VM a NIC. Defaults to False.

• snapshot (bool) – Determines whether to start the VM in snapshot mode. Defaults to False.

• cdrom (list) – List of file names of a CD image to attach to the VM.

minimega_start_vm(network=False, cdrom=None, virtio=False, snapshot=True)

Start a disk-image based VM using minimega. May start with various hardware configurations commonly encountered when preparing FIREWHEEL VMs.

Parameters:

• network (bool) – If True, give the VM a NIC. Defaults to False.

• cdrom (list) – List of file names of a CD image to attach to the VM.

• virtio (bool) – Determines whether or not to use VirtIO on the VM. Defaults to True.

• snapshot (bool) – Determines whether to start the VM in snapshot mode. Defaults to False.

Returns:

The name of the VM.

Return type:

str

firewheel.control.utils.vm_builder.main()

Accept all the user arguments and launch/modify a given image.

firewheel.control.utils.vm_builder.resolve_cdroms(cd_list)

Resolves the argument list of CD-ROMs to absolute paths. We need to do this before we change working directories.

Parameters:

cd_list (list) – A list of CD disk image paths to insert into the VM.

Returns:

A new list of CDs with the full path to the file.

Return type:

list

utils/paths.py

This module enables us to check whether a path is valid in Python or if it is able to be created. This is also useful for checking windows paths:

These functions were taken from: https://stackoverflow.com/a/34102855 on March 26, 2018

Documentation was improved on August 3, 2019.

firewheel.control.utils.paths.ERROR_INVALID_NAME = 123

Windows-specific error code indicating an invalid pathname.

See Also: System Error Codes (0-499) which contains official listing of all such codes.

firewheel.control.utils.paths.is_path_creatable(pathname: str) → bool

Checks to see if a path is able to be created.

Parameters:

pathname (str) – The path to check.

Returns:

True if the current user has sufficient permissions to create the passed pathname; False otherwise.

Return type:

bool

firewheel.control.utils.paths.is_path_exists_or_creatable(pathname: str) → bool

Checks to see if a path exists or is able to be created. This function is guaranteed to _never_ raise exceptions.

Parameters:

pathname (str) – The path to check.

Returns:

True if the passed pathname is a valid pathname for the current OS _and_ either currently exists or is hypothetically able to be created; False otherwise.

Return type:

bool

firewheel.control.utils.paths.is_path_exists_or_creatable_portable(pathname: str) → bool

Checks if a path exists or can be created. This function is guaranteed to _never_ raise exceptions.

Parameters:

pathname (str) – The pathname to check.

Returns:

True if the passed pathname is a valid pathname on the current OS _and_ either currently exists or is hypothetically is able to be created in a cross-platform manner optimized for POSIX-unfriendly filesystems; False otherwise.

Return type:

bool

firewheel.control.utils.paths.is_path_sibling_creatable(pathname: str) → bool

Checks if a path sibling is able to be created. We adjust the meaning of sibling to mean lowest-level path that exists.

Parameters:

pathname (str) – The path to check.

Returns:

True if the current user has sufficient permissions to create siblings (i.e., arbitrary files in the parent directory) of the passed pathname; False otherwise.

Return type:

bool

Raises:

RuntimeError – If the directory recursion failed.

firewheel.control.utils.paths.is_pathname_valid(pathname: str) → bool

Checks to see if a pathname is valid.

Parameters:

pathname (str) – The path to check.

Returns:

True if the passed pathname is a valid pathname for the current OS; False otherwise.

Return type:

bool

New Model Component Templates

Listing 11 The template for a new model component INSTALL file.

#!/bin/bash

#######################################################
# This is a sample install file for {{mc_name}}.
# This file can be used to perform one-time actions
# which help prepare the model component for use.
#
# Common uses of INSTALL files include downloading
# VM Resources from the Internet and installing new
# Python packages into FIREWHEEL's virtual environment.
#
# NOTE: When you are creating these files, it is
# imperative that specific versions of software are
# used. Without being as specific as possible,
# experimental results will **NOT** be repeatable.
# We strongly recommend that any changes to software
# versions are accompanied by a warning and new model
# component version.
#######################################################

# Create a flag for verifying installation
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
INSTALL_FLAG=$SCRIPT_DIR/.{{mc_name}}.installed

#######################################################
# Checking if there this script has already been complete.
#######################################################
function check_flag() {
    if [[ -f "$INSTALL_FLAG" ]]; then
        echo >&2 "{{mc_name}} is already installed!"
        exit 117;  # Structure needs cleaning
    fi
}


#######################################################
# Install python packages into the virtual environment
# used by FIREWHEEL. This takes in an array of packages.
#######################################################
function install_python_package() {
    pkgs=("$@")
    for i in "${pkgs[@]}";
    do
        python -m pip install "$i"
    done
}


#######################################################
# Download using wget and then checksum the downloaded files.
#
# It is important to verify that the downloaded files
# are the files are the same ones as expected.
# This function provides an outline of how to checksum files,
# but will need to be updated with the specific hashes/file names
# that have been downloaded.
#
# This function assumes that the passed in hashes are SHA-256
#######################################################
function wget_and_checksum() {
    downloads=("$@")
    # Uses 2D arrays in bash: https://stackoverflow.com/a/44831174
    declare -n d
    for d in "${downloads[@]}";
    do
        wget "${d[0]}"
        echo "${d[1]}  ${d[2]}" | shasum -a 256 --check || return 1
    done
}


#######################################################
# A function to help users clean up a partial installation
# in the event of an error.
#######################################################
function cleanup() {
    echo "Cleaning up {{mc_name}} install"
    # TODO: Cleanup any downloaded files
    # rm -rf file.tar
    rm -rf $INSTALL_FLAG
    exit 1
}
trap cleanup ERR

# Start to run the script

# Ensure we only complete the script once
check_flag

#######################################################
# Uncomment if there are Pip packages to install
# `pip_packages` should be space separated strings of
# the packages to install
#######################################################
# pip_packages=("requests" "pandas")
# install_python_package "${pip_packages[@]}"


#######################################################
# Uncomment if there is data/VM resources/images to download.
# `file1`, `file2`, etc. should be space separated strings of
# (URL SHASUM-256 FILENAME).
#
# We recommend that explicit versions are used for all Images/VMRs to prevent
# possible differences between instances of a given Model Component.
# Please be mindful of the software versions as it can have unintended
# consequences on your Emulytics experiment.
#
# We require checksums of the files to assist users in verifying
# that they have downloaded the same version.
#######################################################
# Be sure to use SHA-256 hashes for the checksums (e.g. shasum -a 256 <file>)
# file1=("url1" "e0287e6339a4e77232a32725bacc7846216a1638faba62618a524a6613823df5" "file1")
# file2=("url2" "53669e1ee7d8666f24f82cb4eb561352a228b1136a956386cd315c9291e59d59" "file2")
# files=(file1 file2)
# wget_and_checksum "${files[@]}"
# echo "Downloaded and checksummed all files!"


#######################################################
# Add any other desired configuration/packaging here
#######################################################
echo "The {{mc_name}} INSTALL file currently doesn't do anything!"

# Set the flag to notify of successful completion
touch $INSTALL_FLAG

Listing 12 The template for a new model component model_component_objects.py file.

"""This module contains all necessary Model Component Objects for {{mc_name}}."""

from firewheel.control.experiment_graph import require_class

class MyObject:
    """MyObject Class documentation."""

    def __init__(self):
        # TODO: Implement Class constructor here
        pass

Listing 13 The template for a new model component plugin.py file.

from firewheel.control.experiment_graph import AbstractPlugin, Vertex

class {{class_name}}(AbstractPlugin):
    """{{mc_name}} plugin documentation."""

    def run(self):
        """Run method documentation."""
        # TODO: Implement plugin actions here
        pass

Listing 14 The template for a new model component README.rst file.

.. {{mc_name}} documentation file, created by FIREWHEELs Model Component generation.
   You can adapt this file completely to your liking.

.. _{{mc_name}}_mc:

{{name_heading}}
{{mc_name}}
{{name_heading}}

TODO: Describe {{mc_name}} here

{% if attr_provides is defined %}
**Attribute Provides:**
{{attr_provides}}
{% endif %}
{% if attr_depends is defined %}
**Attribute Depends:**
{{attr_depends}}
{% endif %}
{% if attr_precedes is defined %}
**Attribute Precedes:**
{{attr_precedes}}
{% endif %}
{% if mc_depends is defined %}
**Model Component Dependencies:**
{{mc_depends}}
{% endif %}
{% if mc_precedes is defined %}
**Model Component Precedes:**
{{mc_precedes}}
{% endif %}
{% if plugin is defined %}
******
Plugin
******

.. automodule:: {{mc_name}}_plugin
    :members:
    :undoc-members:
    :special-members:
    :private-members:
    :show-inheritance:
    :exclude-members: __dict__,__weakref__,__module__
{% endif %}
{% if model_component_objects is defined %}
*****************
Available Objects
*****************

.. automodule:: {{mc_name}}
    :members:
    :undoc-members:
    :special-members:
    :private-members:
    :show-inheritance:
    :exclude-members: __dict__,__weakref__,__module__
{% endif %}