Lib

log.py

class firewheel.lib.log.Log(name, log_file=None, log_format=None, log_level=None)[source]

Contains a set of helper methods which create logs/loggers for FIREWHEEL.

__init__(name, log_file=None, log_format=None, log_level=None)[source]

Create logger for the given name.

This constructor develops a consistent way of logging for FIREWHEEL. If a logging handler cannot be created, a warning will be printed to the screen and a logging.NullHandler will be used.

Additionally, each log file will attempt to update the log files to use the FIREWHEEL default group (if any).

log

The logger created by the class.

Type:: logging.Logger

log_file

The log file name.

Type:: str

log_file_path

The full path to the log file.

Type:: str

log_level

The specified log level.

Type:: str

Parameters:

name (str) – What logging name should be used. Typically this is either a class name or a FIREWHEEL sub-system name (e.g. CLI).
log_file (str) – Which file should the logger log to.
log_format (str) – A specific logging format.
log_level (str) – The specified log level.

_define_formatter(log_format)[source]

class firewheel.lib.log.UTCLog(name, log_file=None, log_format=None, log_level=None)[source]

A subclass of Log that logs messages in UTC.

__annotations__ = {}

_define_formatter(log_format)[source]

utilities.py

firewheel.lib.utilities.badlink(info, base)[source]

Checks to see if the provided link is underneath the given base path.

Parameters:

info (tarfile.TarInfo) – The file member that is going to be extracted.
base (pathlib.Path) – The path of the current working directory.

Returns:

True if the path is not under the proposed base path otherwise False.

Return type:

bool

firewheel.lib.utilities.badpath(path, base)[source]

Checks to see if the provided file path is underneath the given base path.

Parameters:

path (str) – The proposed extraction path of the tar file member to check.
base (pathlib.Path) – The path of the current working directory.

Returns:

True if the path is not under the proposed base path otherwise False.

Return type:

bool

firewheel.lib.utilities.get_safe_tarfile_members(tarfile)[source]

Identify and return the members of a tarfile.TarFile that are considered safe. See the documentation for tarfile.TarFile.extractall() for more information. This function, as well as badlink() and badpath() were based on https://stackoverflow.com/a/10077309.

Parameters:: tarfile (tarfile.TarFile) – The tar file to extract.
Returns:: A list of “safe” members to extract.
Return type:: list

firewheel.lib.utilities.hash_file(fname: str) → str[source]

A relatively efficient way of hashing a file https://stackoverflow.com/a/3431838. Through various performance tests, we found that SHA1 is currently the fastest hashlib function. We also found that SHA-1 performance improved by using a chunk size of 1048576.

Parameters:: fname (str) – The name of the file to hash.
Returns:: The hash of the file.
Return type:: str

firewheel.lib.utilities.render_rich_string(text)[source]

Convert a string with rich markup to standard string with ANSI sequences. This is useful for printing without the rich.console.Console.

Note

Adapted from: https://github.com/Textualize/rich/issues/3152#issuecomment-1759770089

Parameters:: text (str) – The text to convert.
Returns:: A string with ANSI markup as defined by rich.
Return type:: str

firewheel.lib.utilities.retry(num_tries, exceptions=None, base_delay=10, exp_factor=2)[source]

This function provides a decorator which enables automatic retrying of functions which make connections to the FileStore and fail due to timeout errors. This code was adapted from: https://dzone.com/articles/pymongo-pointers-how-to-make-robust-and-highly-ava-1

Parameters:

num_tries (int) – The number of times to try the function.
exceptions (tuple) – A tuple of exceptions to except.
base_delay (int) – The constant amount of time to sleep between attempts.
exp_factor (int) – The exponential amount of time to sleep between attempts. When set to 1, there will be no exponential increase in sleep times.

Returns:

The decorator function.

firewheel.lib.utilities.strtobool(val)[source]

Convert a string representation of truth to true (1) or false (0). True values are ‘y’, ‘yes’, ‘t’, ‘true’, ‘on’, and ‘1’; false values are ‘n’, ‘no’, ‘f’, ‘false’, ‘off’, and ‘0’. Raises ValueError if ‘val’ is anything else.

Copied from distutils which has since been deprecated.

Parameters:: val (str) – A string to check if it is a boolean value.
Returns:: 1 if the value is True and 0 if the value is False.
Return type:: int
Raises:: ValueError – If an invalid input is provided.

grpc/firewheel_grpc_client.py

class firewheel.lib.grpc.firewheel_grpc_client.FirewheelGrpcClient(hostname='', port=50051, db='prod', log=None, require_connection=True)[source]

A client to interact with the FirewheelGrpcServicer.

__init__(hostname='', port=50051, db='prod', log=None, require_connection=True)[source]

Initializes the client.

Parameters:

hostname (str) – The gRPC server’s hostname.
port (str) – The gRPC server’s port.
db (str) – The database that this client will be using. (e.g. “prod” or “test”).
log (logging.Logger) – A log the client can use.
require_connection (bool) – Check whether or not to raise an error on a missing connection.

Raises:

RuntimeError – If the gRPC port is out of range.

check_connection(error=True)[source]

Check on whether a connection exists to the gRPC server.

Parameters:: error (bool) – Whether to raise an exception on a missing connection.
Returns:: True if a connection exists, False otherwise.
Return type:: bool
Raises:: ConnectionError – If the connection cannot be started (and error=True). # noqa: DAR401,DAR402

clear_db()[source]

Clear the gRPC database.

This method essentially calls initialize_experiment_start_time() and destroy_all_vm_mappings().

Returns:: Contains the response codes (initialize_experiment_start_time(), destroy_all_vm_mappings())
Return type:: Tuple

close()[source]: Closes the gRPC channel.

count_vm_mappings_not_ready()[source]

Requests the count of VMs that are not ready.

Returns:: Dictionary representation of firewheel_grpc_pb2.CountVMMappingsNotReadyResponse.
Return type:: dict

destroy_all_vm_mappings()[source]

Requests to destroy all vm_mappings.

Returns:: Dictionary representation of firewheel_grpc_pb2.DestroyAllVMMappingsReponse. Empty on success.
Return type:: dict

destroy_vm_mapping_by_uuid(vm_uuid)[source]

Requests to destroy the vm_mapping corresponding to a given uuid.

Parameters:: vm_uuid (str) – vm uuid to search on.
Returns:: Dictionary representation of firewheel_grpc_pb2.DestroyVMMappingResponse. Empty on success.
Return type:: dict

get_experiment_launch_time()[source]

Requests the experiment_launch_time.

Returns:: The datetime that the experiment was launched.
Return type:: datetime.datetime

get_experiment_start_time()[source]

Requests the experiment_start_time.

Returns:: The datetime that the experiment was started.
Return type:: datetime.datetime

get_info()[source]

Requests general gRPC server info.

Returns:: Dictionary representation of firewheel_grpc_pb2.GetInfoResponse.
Return type:: dict

get_repositories_as_dict()[source]

Requests all of the repositories.

Returns:: Dictionary representations of firewheel_grpc_pb2.Repository.
Return type:: (iterable) dict

get_vm_mapping_by_uuid(vm_uuid)[source]

Requests to get the vm_mapping corresponding to a given uuid.

Parameters:: vm_uuid (str) – vm uuid to search on.
Returns:: (dict) Dictionary representation firewheel_grpc_pb2.VMMapping.

initialize_experiment_start_time()[source]

Requests to initialize the experiment_start_time.

Returns:: Dictionary representation of firewheel_grpc_pb2.InitializeExperimentStartTimeResponse. Empty on success.
Return type:: dict

list_repositories()[source]

Requests all of the repositories.

Yields:: dict – Dictionary representations of firewheel_grpc_pb2.Repository.

list_vm_mappings()[source]

Requests to list all vm_mappings.

Returns:: (list) Dictionary representations of firewheel_grpc_pb2.VMMapping.

remove_all_repositories()[source]

Requests to remove all repositories.

Returns:: Dictionary representation of firewheel_grpc_pb2.RemoveAllRepositoriesResponse
Return type:: dict

remove_repository(repository)[source]

Requests that the given repository is removed.

Parameters:: repository (dict) – The repository as a dictionary.
Returns:: Dictionary representation of firewheel_grpc_pb2.RemoveRepositoryResponse.
Return type:: dict

set_experiment_launch_time(launch_time_dt)[source]

Requests to set the experiment_launch_time.

Parameters:: launch_time_dt (datetime.datetime) – The datetime that the experiment was launched.
Returns:: The datetime that the experiment was launched.
Return type:: datetime.datetime

set_experiment_start_time(start_time_dt)[source]

Requests to set the experiment_start_time.

Parameters:: start_time_dt (datetime.datetime) – The datetime that the experiment was started.
Returns:: The datetime that the experiment was started.
Return type:: datetime.datetime

set_repository(repository)[source]

Requests that the given repository is set.

Parameters:: repository (dict) – The repository as a dictionary.
Returns:: Dictionary representation of firewheel_grpc_pb2.SetRepositoryResponse.
Return type:: dict

set_vm_mapping(vmm)[source]

Requests to set a given vm_mapping.

Parameters:: vmm (dict) – Dictionary representation of vm_mapping.
Returns:: (dict) Dictionary representation firewheel_grpc_pb2.VMMapping.

set_vm_state_by_uuid(vmm)[source]

Requests to set the state of the vm_mapping corresponding to a given uuid.

Parameters:: vmm (dict) – Dictionary containing server_uuid and state.
Returns:: (dict) Dictionary representation of the updated firewheel_grpc_pb2.VMMapping.

set_vm_time_by_uuid(vmm)[source]

Requests to set the time of the vm_mapping corresponding to a given uuid.

Parameters:: vmm (dict) – Dictionary containing server_uuid and current_time.
Returns:: (dict) Dictionary representation of the updated firewheel_grpc_pb2.VMMapping.

grpc/firewheel_grpc_server.py

class firewheel.lib.grpc.firewheel_grpc_server.FirewheelServicer[source]

The Servicer for the Firewheel GRPC Service.

CountVMMappingsNotReady(request, context)[source]

Returns the count of VMs that are not ready.

Parameters:

request (firewheel_grpc_pb2.CountVMMappingsNotReadyRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

Message containing the count of the not ready VMs.

Return type:

firewheel_grpc_pb2.CountVMMappingsNotReadyResponse

DestroyAllVMMappings(request, context)[source]

Destroys all vm_mappings.

Parameters:

request (firewheel_grpc_pb2.DestroyAllVMMappingsRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

Empty message on success.

Return type:

firewheel_grpc_pb2.DestroyAllVMMappingsReponse

DestroyVMMappingByUUID(request, context)[source]

Destroys the vm_mapping associated with the given uuid.

Parameters:

request (firewheel_grpc_pb2.VMMappingUUID) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

Empty message on success.

Return type:

firewheel_grpc_pb2.DestroyVMMappingResponse

GetExperimentLaunchTime(request, context)[source]

Gets the experiment launch time.

Parameters:

request (firewheel_grpc_pb2.GetExperimentLaunchTimeRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

The set experiment launch time or None on failure.

Return type:

firewheel_grpc_pb2.ExperimentLaunchTime

GetExperimentStartTime(request, context)[source]

Gets the experiment start time.

Parameters:

request (firewheel_grpc_pb2.GetExperimentStartTimeRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

The set experiment start time.

Return type:

firewheel_grpc_pb2.ExperimentStartTime

GetInfo(request, context)[source]

Returns general server info on version, uptime, and whether there is a currently running experiment.

Parameters:

request (firewheel_grpc_pb2.GetInfoRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

The server info.

Return type:

firewheel_grpc_pb2.GetInfoResponse

GetVMMappingByUUID(request, context)[source]

Gets the vm_mapping associated with the given uuid.

Parameters:

request (firewheel_grpc_pb2.VMMappingUUID) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

The found vm_mapping or None on failure.

Return type:

firewheel_grpc_pb2.VMMapping

InitializeExperimentStartTime(request, context)[source]

Initializes the experiment launch time.

Parameters:

request (firewheel_grpc_pb2.InitializeExperimentStartTimeRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

Empty message on success.

Return type:

firewheel_grpc_pb2.InitializeExperimentStartTimeResponse

ListRepositories(request, context) → Iterable[Repository][source]

Iterates through all requested repositories.

Parameters:

request (firewheel_grpc_pb2.ListRepositoriesRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Yields:

firewheel_grpc_pb2.Repository – The iterated repositories.

ListVMMappings(request, context) → Iterable[VMMapping][source]

Iterates through all requested vm_mappings.

Parameters:

request (firewheel_grpc_pb2.ListVMMappingsRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Yields:

firewheel_grpc_pb2.VMMapping – The iterated vm_mappings.

RemoveAllRepositories(request, context)[source]

Removes all repositories.

Parameters:

request (firewheel_grpc_pb2.RemoveAllpositories) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

firewheel_grpc_pb2.RemoveAllRepositoriesResponse.

RemoveRepository(request, context)[source]

Removes a repository.

Parameters:

request (firewheel_grpc_pb2.Repository) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

firewheel_grpc_pb2.RemoveRepositoryResponse.

SetExperimentLaunchTime(request, context)[source]

Sets the experiment launch time.

Parameters:

request (firewheel_grpc_pb2.ExperimentLaunchTime) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

The set experiment launch time.

Return type:

firewheel_grpc_pb2.ExperimentLaunchTime

SetExperimentStartTime(request, context)[source]

Sets the experiment start time.

Parameters:

request (firewheel_grpc_pb2.ExperimentStartTime) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

The set experiment start time.

Return type:

firewheel_grpc_pb2.ExperimentStartTime

SetRepository(request, context)[source]

Sets a repository.

Parameters:

request (firewheel_grpc_pb2.Repository) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

Empty message on success.

Return type:

firewheel_grpc_pb2.SetRepositoryResponse

SetVMMapping(request, context)[source]

Sets the given vm_mapping.

Parameters:

request (firewheel_grpc_pb2.VMMapping) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

The set vm_mapping.

Return type:

firewheel_grpc_pb2.VMMapping

SetVMStateByUUID(request, context)[source]

Sets the state for the vm_mapping associated with the given uuid.

Parameters:

request (firewheel_grpc_pb2.SetVMStateByUUIDRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

The updated vm_mapping.

Return type:

firewheel_grpc_pb2.VMMapping

SetVMTimeByUUID(request, context)[source]

Sets the time for the vm_mapping associated with the given uuid.

Parameters:

request (firewheel_grpc_pb2.SetVMTimeByUUIDRequest) – The gRPC request.
context (grpc._server._Context) – The gRPC context.

Returns:

The updated vm_mapping.

Return type:

firewheel_grpc_pb2.VMMapping

__init__()[source]: Initialize the Servicer.

_init_db(db_name)[source]

Initializes the database with name db_name.

Parameters:: db_name (str) – The name of the database to initialize.

_read_repository_db_from_file(db)[source]

Utility function for reading the RepositoryDb from a file.

Parameters:: db (RepositoryDb) – The database to read.
Returns:: True on success, False otherwise.
Return type:: bool
Raises:: RuntimeError – If the repository path does not exist.

_update_not_ready_vmms(vmm, db)[source]

Updates the number of VMs that are ready and not ready.

Parameters:

vmm (firewheel_grpc_pb2.SetVMStateByUUIDRequest) – The VMM request object.
db (str) – The database that this client will be using. (e.g. “prod” or “test”)

_write_repository_db_to_file(db)[source]

Utility function for writing the RepositoryDb to a file.

Parameters:: db (RepositoryDb) – The database to write to a file.

get_vm_mapping(db, uuid)[source]

Returns the vm_mapping object associated with the given db and uuid.

Parameters:

db (dict) – The database to search.
uuid (str) – The key to search on.

Returns:

The found vm_mapping, or None on KeyError.

Return type:

firewheel_grpc_pb2.VMMapping

firewheel.lib.grpc.firewheel_grpc_server.serve()[source]: Initializes the gRPC server and servicer. Starts the server.

grpc/firewheel_grpc_resources

firewheel.lib.grpc.firewheel_grpc_resources.msg_to_dict(msg)[source]

Parses a protobuf message into a python dictionary. Replaces the literal string ‘None’ with the python NoneType.

Parameters:: msg (google.protobuf.message.Message) – The message to convert.
Returns:: Dictionary representation of protobuf message.
Return type:: dict

grpc/firewheel_grpc_pb2_grpc

Client and server classes corresponding to protobuf-defined services.

class firewheel.lib.grpc.firewheel_grpc_pb2_grpc.Firewheel[source]

Missing associated documentation comment in .proto file.

static CountVMMappingsNotReady(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static DestroyAllVMMappings(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static DestroyVMMappingByUUID(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static GetExperimentLaunchTime(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static GetExperimentStartTime(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static GetInfo(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static GetVMMappingByUUID(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static InitializeExperimentStartTime(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static ListRepositories(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static ListVMMappings(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static RemoveAllRepositories(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static RemoveRepository(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static SetExperimentLaunchTime(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static SetExperimentStartTime(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static SetRepository(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static SetVMMapping(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static SetVMStateByUUID(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

static SetVMTimeByUUID(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None)[source]

class firewheel.lib.grpc.firewheel_grpc_pb2_grpc.FirewheelServicer[source]

Missing associated documentation comment in .proto file.

CountVMMappingsNotReady(request, context)[source]: Returns the count of VMs that are not ready.

DestroyAllVMMappings(request, context)[source]: Destroys all vm_mappings.

DestroyVMMappingByUUID(request, context)[source]: Destroys the vm_mapping associated with the given uuid.

GetExperimentLaunchTime(request, context)[source]: Gets the experiment launch time.

GetExperimentStartTime(request, context)[source]: Gets the experiment start time.

GetInfo(request, context)[source]: Gets the gRPC server info.

GetVMMappingByUUID(request, context)[source]: Gets the vm_mapping associated with the given uuid.

InitializeExperimentStartTime(request, context)[source]: Initializes the experiment launch time.

ListRepositories(request, context)[source]: List all repositories.

ListVMMappings(request, context)[source]: Iterates through all requested vm_mappings.

RemoveAllRepositories(request, context)[source]: Remove all repositories.

RemoveRepository(request, context)[source]: Remove a repository.

SetExperimentLaunchTime(request, context)[source]: Sets the experiment start time.

SetExperimentStartTime(request, context)[source]: Sets the experiment start time.

SetRepository(request, context)[source]: Set a repository.

SetVMMapping(request, context)[source]: Sets the given vm_mapping.

SetVMStateByUUID(request, context)[source]: Sets the state for the vm_mapping associated with the given uuid.

SetVMTimeByUUID(request, context)[source]: Sets the time for the vm_mapping associated with the given uuid.

__annotations__ = {}

class firewheel.lib.grpc.firewheel_grpc_pb2_grpc.FirewheelStub(channel)[source]

Missing associated documentation comment in .proto file.

__init__(channel)[source]

Constructor.

Parameters:: channel – A grpc.Channel.

firewheel.lib.grpc.firewheel_grpc_pb2_grpc.add_FirewheelServicer_to_server(servicer, server)[source]

grpc/firewheel_grpc_pb2

Generated protocol buffer code.

class firewheel.lib.grpc.firewheel_grpc_pb2.CountVMMappingsNotReadyRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.CountVMMappingsNotReadyResponse

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.DestroyAllVMMappingsRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.DestroyAllVMMappingsResponse

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.DestroyVMMappingResponse

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.ExperimentLaunchTime

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.ExperimentStartTime

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.GetExperimentLaunchTimeRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.GetExperimentStartTimeRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.GetInfoRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.GetInfoResponse

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.InitializeExperimentStartTimeRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.InitializeExperimentStartTimeResponse

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.ListRepositoriesRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.ListVMMappingsRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.RemoveAllRepositoriesRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.RemoveAllRepositoriesResponse

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.RemoveRepositoryResponse

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.Repository

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.SetRepositoryResponse

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.SetVMStateByUUIDRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.SetVMTimeByUUIDRequest

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.VMMapping

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

class firewheel.lib.grpc.firewheel_grpc_pb2.VMMappingUUID

DESCRIPTOR = <google._upb._message.Descriptor object>

__annotations__ = {}

__slots__ = ()

FIREWHEEL gRPC Protocol

FIREWHEEL gRPC Protocol Documentation

This documentation was compiled using protoc and protoc-gen-doc with:

protoc --doc_out=./doc/_static/ --doc_opt=html,proto_doc.html ./grpc/firewheel_grpc.proto

minimega/api.py

class firewheel.lib.minimega.api.minimegaAPI(mm_base=None, timeout=120, skip_retry=False)[source]

This class implements an API to minimega to run common commands and parse outputs into python objects.

__init__(mm_base=None, timeout=120, skip_retry=False)[source]

Initializes the object with a minimega connection.

Parameters:

mm_base (str) – The root directory for minimega. The default is None, in which case the directory is pulled from the current configuration.
timeout (int) – Number of seconds to wait for minimega socket when initializing before raising a TimeoutError.
skip_retry (bool) – Do not attempt to retry connecting to minimega if there is an error.

Raises:

RuntimeError – If the minimega socket does not exist.
TimeoutError – If the connection to minimega timed out.

_check_version(timeout, skip_retry=False)[source]

Checks if the version of the minimega python bindings matches the versions of all running instances of minimega in the namespace.

To enable timeouts, a process is spawned to call the necessary minimega functions.

Parameters:

timeout (int) – Number of seconds to wait for minimega socket before raising a TimeoutError.
skip_retry (bool) – Do not attempt to retry connecting to minimega if there is an error.

Returns:

True if versions match, False otherwise.

Return type:

bool

Raises:

TimeoutError – If a timeout occurs when connecting to minimega.
RuntimeError – If a timeout occurs when connecting to minimega but skip_retry is true.

_cmd_to_dict(cmd)[source]

Debug function. Used to return a dictionary representation of the output of a minimega shell command.

Parameters:: cmd (list) – The minimega command to run.
Returns:: A parsed table representation of the minimega output.
Return type:: list

_parse_host(host_item)[source]

Parses a host response item from minimega.

Parameters:: host_item (tuple) – Tuple of hostname, host_values from minimega host output.
Returns:: The parsed host.
Return type:: dict

static _parse_output(output)[source]

Debug function. Used to parse the output of a minimega shell command.

Parameters:: output (str) – The minimega command output.
Returns:: A list representation of the minimega shell command output.
Return type:: list

static _parse_table(table_output)[source]

Debug function. Used to parse the tabular output of a minimega shell command.

Parameters:: table_output (list) – The minimega command output.
Returns:: A parsed table representation of the minimega output.
Return type:: list

static _run_cmd(args)[source]

Debug function. Used to return the output of a minimega shell command.

Note

In general, this provides direct access to minimega and can have some serious security implications. Please review Security for more details.

Parameters:: args (list) – The minimega command to run.
Returns:: The output of the minimega shell command.
Return type:: str

static check_host_filter(filter_dict, elem)[source]

Checks an element against a provided dictionary of filter keys and expected values.

The provided dictionary is expected to be of the form {filter_key : filter_obj}, where the filter_obj is a tuple of (filter_relation, filter_value).

Filtering an element proceeds as follows:

For each filter_key, ensure that the provided element’s value for that key satisfies the filter_relation for the given filter_value.
Return True only if each filter check is satisfied.

Currently, the following filter_relation values are supported: {"=", "!=", "~", "!~"}.

Parameters:

filter_dict (dict) – A dictionary of filters of the form: {filter_key : filter_obj}, where filter_obj is of the form: (filter_relation, filter_value).
elem (dict) – The element to check against the filter.

Returns:

Whether the element passes all of the filters checks.

Return type:

bool

Raises:

RuntimeError – If the filter_relation is unsupported.

static get_am_head_node()[source]

Provide method for determining if the current node is the head node.

Returns:: True if the current node is the head node. False otherwise.
Return type:: bool

get_cpu_commit_ratio()[source]

Returns the ratio of committed CPUs to logical CPUs on the current physical host. This is used to intelligently throttle based on load.

Returns:: (CPU commit / logical CPUs).
Return type:: float

static get_head_node()[source]

Get the head node from the FIREWHEEL configuration.

Returns:: The head node from the cluster. This is assumed to be the first node of the control node list.
Return type:: str
Raises:: RuntimeError – If no cluster control node exists.

get_hosts(host_key=None)[source]

Get the hosts in the minimega namespace, and return the parsed hosts as a dict keyed on hostname.

Parameters:: host_key (str) – Optional, when provided return only the parsed host with hostname equal to host_key. Otherwise, return all parsed hosts as a dict.
Returns:: A dict of parsed hosts.
Return type:: dict

get_mesh_size()[source]

Gets the size of the minimega mesh.

Returns:: The size of the minimega mesh.
Return type:: int

mm_vms(filter_dict=None)[source]

List the VMs in current experiment. Optionally filtered by supplied filter_dict.

Parameters:: filter_dict (dict) – A dictionary of filters of the form: {filter_key : filter_obj}, where filter_obj is of the form: (filter_relation, filter_value)
Returns:: A dictionary representation of the filtered VMs.
Return type:: dict

static mmr_map(raw_response, first_value_only=False)[source]

Attempts to map a raw minimega output into a python dictionary

Parameters:

raw_response (str) – raw output from a minimega command.
first_value_only (bool) – If True, return only the first value in the response. Defaults to False.

Returns:

Dictionary representation of minimega output.

Return type:

dict

ns_kill_processes(path)[source]

Kill a specified process using pkill -f

Parameters:: path (str) – The path to kill. (This path will be prefixed with sys.executable.)
Returns:: True on success, False otherwise.
Return type:: bool
Raises:: minimega.Error – If minimega has an issue running the command.

set_group_perms(path)[source]

Recursively sets the group permissions on a path to be equal to the user permissions.

Parameters:: path (str) – The path to set group permissions.
Returns:: True on success, False otherwise.
Return type:: bool

minimega/file_store.py

class firewheel.lib.minimega.file_store.FileStore(store: str, mm_base: str = '/tmp/minimega', decompress: bool = False, log: Logger | None = None)[source]

A repository for files uses a distributed file system for easy access on all hosts in a FIREWHEEL cluster. Currently uses minimega to store files. Ideally, this should be modifiable without affecting the interface.

__annotations__ = {}

__init__(store: str, mm_base: str = '/tmp/minimega', decompress: bool = False, log: Logger | None = None) → None[source]

Initializes the object with a minimegaAPI connection.

Parameters:

store (str) – The relative path from the minimega files directory for this FileStore.
mm_base (str) – The root directory for minimega.
decompress (bool) – Whether to decompress files by default when using this FileStore.
log (firewheel.lib.log.Log.log) – Override the default FIREWHEEL log.

Raises:

PermissionError – If the FileStore was unable to create the cache due to bad permissions.

_check_mesh_file_consistency(mm_file_path: str) → Dict[str, List[Dict[str, str | List[str] | None]] | bool][source]

Checks whether there is a consistent version of the file on all hosts in the mesh.

Parameters:: mm_file_path (str) – The path of the file being transferred.
Returns:: Dictionary containing information such as consistent (if all versions of the file are the same), exists (whether any host has a version of this a file).
Return type:: dict

_check_mesh_transfer(mm_file_path: str) → bool[source]

Blocks until a final transfer is complete.

Parameters:: mm_file_path (str) – The path of the file being transferred.
Returns:: Whether each host in the mesh has a consistent version of the file in their cache.
Return type:: bool

_decompress_error(exp: OSError, tmp_local_path: str, host_file_path: str) → None[source]

Output specific warnings/errors if an issue happened while trying to decompress a file. Additionally, try to remove the file which caused the issue.

Parameters:

exp (OSError) – The exception being raised.
tmp_local_path (str) – The temporary path of the file name used for decompression.
host_file_path (str) – The location to cache the file locally.

_get_lock(cache_location: str) → bool[source]

Get a lock (FIREWHEEL-specific, not OS lock) for a specific location in the local cache.

Parameters:: cache_location (str) – The name of the file to lock.
Returns:: True on lock released.
Return type:: bool

_minimega_get_data(host_file_path: str, filename: str, decompress: bool = False) → Tuple[str, str][source]

Get the requested file from minimega and return the path to the locally cached version of the file. Detect if the file is already cached and just return the path.

Parameters:

host_file_path (str) – The location to cache the file locally.
filename (str) – The name of the file to get from minimega.
decompress (bool) – Whether to decompress the file.

Returns:

Contains the path to the locally cached file and an empty string. If there is an error then it returns empty string and the second argument is a string description of the error.

Return type:

tuple

_minimega_get_file(cache_location: str, filename: str) → bool[source]

Perform the mechanics of a read operation from minimega. This function assumes that you have the file’s lock already.

Parameters:

cache_location (str) – The local file to write to.
filename (str) – The minimega file to read.

Returns:

True on success, False otherwise (or an Exception is raised).

Return type:

bool

Raises:

FileNotFoundError – If the file is not found.
MinimegaError – If there is an error running minimega.

_release_lock(cache_location: str) → bool[source]

Release the (FIREWHEEL-specific) lock on a specific location in the local cache.

Parameters:: cache_location (str) – The name of the file to release the lock on.
Returns:: True on lock released.
Return type:: bool

_strip_extension(filename: str) → str[source]

Check to see if a filename has a compression extension and remove it if it does. Currently checking for {".xz". ".tar.gz", ".tar", ".tgz"}.

Parameters:: filename (str) – The name of the file to update (if needed).
Returns:: The updated filename without the extension
Return type:: str

_wait_for_lock(cache_location: str) → None[source]

Wait for the (FIREWHEEL-specific) lock on a specific location in the local cache.

Parameters:: cache_location (str) – The name of the file to wait for the lock on.

add_file(path: str, force: bool = True) → None[source]

Add a file to the FileStore.

Parameters:

path (str) – The path of the file to be uploaded into the FileStore.
force (bool) – If True, then remove the existing file before adding the new one.

Raises:

OSError – If there is an issue adding the file.

add_file_from_content(content: str, filename: str, force: bool = True, broadcast: bool = True) → None[source]

Creates file and add it to the FileStore with the given filename.

Parameters:

content (str) – Content of the file to add.
filename (str) – The name of the file to be uploaded into the FileStore.
force (bool) – Whether to attempt to remove the file from the FileStore first.
broadcast (bool) – Whether to have all mesh nodes attempt to put this file in their cache.

add_image_file(path: str, force: bool = True) → bool[source]

Adds an image file to FileStore.

Parameters:

path (str) – The path of the file being transferred.
force (bool) – Whether to force adding the new image.

Returns:

Whether the broadcast was successful; i.e. Whether each host in the mesh has a consistent version of the file in their cache.

Return type:

bool

broadcast_get_file(mm_file_path: str) → bool[source]

Add a file to the FileStore and ensure that all hosts download it into their cache.

Parameters:: mm_file_path (str) – The path of the file to be uploaded into the FileStore.
Returns:: Whether the broadcast was successful; i.e. Whether each host in the mesh has a consistent version of the file in their cache.
Return type:: bool
Raises:: Exception – If an error occurs interacting with minimega.

check_path(filename: str) → bool[source]

Check to see if a file’s path exists.

Parameters:: filename (str) – The name of the file to cache.
Returns:: True if it exists, False otherwise.
Return type:: bool

file_lock(location: str) → Generator[bool | None, None, None][source]

Context Manager for acquiring locks. Enables using the with context to get a lock using an optional timeout. Release lock at the end of the with block.

Parameters:: location (str) – Location of the file to lock.
Yields:: Optional[bool] – True on lock acquired.

get_file(filename: str) → FileStoreFile[source]

Get an FileStoreFile instance representing a specific file.

Importantly, this method returns successfully even if the specified file does not exist. In this case, a FileNotFoundError is raised upon using the returned FileStoreFile.

Use example:

with file_store.get_file('my_vm_resource.py') as vm_resource_file:
    chunk = vm_resource_file.read()

Invalid file example:
# Raises FileNotFoundError
with file_store.get_file('invalid') as vm_resource_file:
    pass

Parameters:: filename (str) – The name of the file to get a representation for.
Returns:: An instance of FileStoreFile for the requested file.
Return type:: FileStoreFile

get_file_hash(filename: str) → str[source]

Returns the hash of a file in the MM files directory

Parameters:: filename (str) – Name of the file.
Returns:: Hash of the file.
Return type:: str

get_file_path(filename: str) → str[source]

Get the proper path to the given file.

Parameters:: filename (str) – The name of the file for which to find the path.
Returns:: The fully qualified path of the file.
Return type:: str

get_file_size(filename: str) → int[source]

Returns the length (in bytes) of a file in the FileStore. This method ignores the local cache.

Parameters:

filename (str) – Name of the file.

Returns:

Size of the file, measured in bytes.

Return type:

int

Raises:

FileNotFoundError – If the file is not found in the database.
RuntimeError – If the number of files does not equal one.

get_file_upload_date(filename: str) → datetime | None[source]

Returns the upload date of a file in minimega

Parameters:: filename (str) – Name of the file.
Returns:: Datetime date that the file was uploaded into local the minimega files directory.
Return type:: datetime.datetime

get_path(filename: str) → str[source]

Ensure a specified file is cached locally.

Parameters:

filename (str) – The name of the file to cache.

Returns:

The path to the locally cached file.

Return type:

str

Raises:

FileNotFoundError – When the local path for the image cannot be found.
RuntimeError – When the file could not be decompressed.

list_contents(pattern: str = '') → List[Tuple[str, str, str]][source]

List the contents of the FileStore.

Parameters:

pattern (str) – pattern to match filenames against

Returns:

A list of dictionaries, where each dictionary is an entry from the FileStore.

Return type:

list

Raises:

RuntimeError – If there is an error getting the minimega file list.
Exception – If an error occurred during this method.

list_distinct_contents(pattern: str = '') → List[str][source]

List the contents of the FileStore.

Parameters:: pattern (str) – The pattern to match filenames against.
Returns:: A list of dictionaries, where each dictionary is an entry from the FileStore.
Return type:: list
Raises:: Exception – If there is an error getting the file list.

remove_file(filename: str) → None[source]

Do the removal of a file name from minimega.

Parameters:: filename (str) – minimega file name to remove.
Raises:: OSError – If an error occurs interacting with minimega.

class firewheel.lib.minimega.file_store.FileStoreFile(filename: str, database: FileStore)[source]

A pseudo-File-like object representing a file stored in the FileStore.

It is pseudo-File-like because it is a Context Manager (it is used with the “with” statement) and it provides a read() method. It does not implement other portions of the File interface.

__enter__() → FileStoreFile[source]

Open (for reading) a specific file in the FileStore.

Returns:: Reference to this instance.
Return type:: FileStoreFile

__exit__(exc_type: BaseException | None = None, exc_val: BaseException | None = None, exc_tb: TracebackType | None = None) → bool[source]

Close the file.

Parameters:

exc_type – Any exception type. Ignored.
exc_val – Any exception value. Ignored.
exc_tb – The exception trace back. Ignored.

Returns:

Whether the file was successfully closed. Will be False if it was never opened in the first place.

Return type:

bool

__init__(filename: str, database: FileStore)[source]

Create a new instance of FileStoreFile.

Parameters:

filename (str) – Name of the file in the database to access
database (FileStore) – Instance of FileStore where the file is stored

read(size: int = 16777216) → bytes[source]

Read the next chunk of data from a file handle. Assumes __enter__ has been called (call this from within a “with” block.

Parameters:: size (int) – The size of the chunk to read from the file. Default is 16 MiB.
Returns:: A chunk of data of the specified size (default 16 MiB) from the file referenced by handle.
Raises:: RuntimeError – If the file handle is not found.

discovery/api.py

class firewheel.lib.discovery.api.discoveryAPI(hostname=None, port=None)[source]

This class implements an API to discovery.

__init__(hostname=None, port=None)[source]

Initializes the object.

Parameters:

hostname (str) – discovery hostname
port (int) – discovery port number

connect_endpoint(node_id, network_id)[source]

Connect node with node_id to network with network_id.

Parameters:

node_id (str) – Node identifier.
network_id (str) – Network identifier.

Returns:

Dictionary of updated endpoint.

Return type:

dictionary

delete_all()[source]

Deletes all endpoints and networks.

Returns:: True on success.
Return type:: bool
Raises:: RuntimeError – If not all the endpoints or networks were successfully deleted.

delete_all_endpoints()[source]

Deletes all endpoints.

Returns:: List of dictionary representations of deleted endpoints.
Return type:: list

delete_all_networks()[source]

Deletes all networks.

Returns:: List of dictionary representations of deleted networks.
Return type:: list

delete_endpoints(key='', value='')[source]

Delete endpoints. If key and value are provided, then all endpoints with matching: key:value are deleted. If only value is provided, then all endpoints that have any key with a matching value are deleted.

Parameters:

key (str) – endpoint key
value (str) – endpoint value

Returns:

List of dictionary representations of deleted endpoints.

Return type:

list

delete_networks(key='', value='')[source]

Delete networks. If key and value are provided, then all networks with matching: key:value are deleted. If only value is provided, then all networks that have any key with a matching value are deleted.

Parameters:

key (str) – network key
value (str) – network value

Returns:

List of dictionary representations of deleted networks.

Return type:

list

get_config(config_key='')[source]

Get the config value for key, config_key. If no key is provided, returns entire config.

Parameters:: config_key (str) – config key.
Returns:: Config as a dictionary.
Return type:: dict

get_endpoints()[source]

Get all endpoints.

Returns:: A list of all endpoints on success.
Return type:: list

get_networks()[source]

Get all networks.

Returns:: A list of all networks on success.
Return type:: list

insert_endpoint(mm_node_properties=None)[source]

Insert endpoint with the provided properties.

Parameters:: mm_node_properties (dict) – Dictionary of node properties.
Returns:: List of dictionary representations of inserted endpoints on success.
Return type:: list

insert_network()[source]

Insert a network.

Returns:

Dictionary representation of response. On success, this is: the inserted network.

Return type:

dict

set_config(config_key, config_value)[source]

Set the config value for key, config_key, to config_value.

Parameters:

config_key (str) – config key.
config_value (dict) – The new value for the given config_key.

Returns:

True on success.

Return type:

bool

start_discovery()[source]

Start the discovery service with minimega if it is not already started.

Returns:: True if it was started successfully, False otherwise.
Return type:: bool

test_connection()[source]

Check to see if the discovery API is active.

Returns:: True if the discovery server is up, False otherwise.
Return type:: bool

update_endpoint(mm_node_properties=None)[source]

Updates endpoint with corresponding mm_node_properties. This matches on NID in mm_node_properties.

Parameters:: mm_node_properties (dict) – Dictionary of node properties.
Returns:: List of dictionary representations of inserted endpoints on success.
Return type:: list