CLI
firewheel_cli.py
- class firewheel.cli.firewheel_cli.FirewheelCLI(interactive=False)[source]
Entry class for the CLI.
Uses the cmd module to handle commands. Most functionality is based on “Helpers”, the names of which are determined at run-time. Commands are designed to support Helpers. If you want to add functionality, it should probably be a Helper.
- __init__(interactive=False)[source]
Constructor. Initialize our Helper system and our session history.
- ruler
The character used to draw separator lines under the help-message headers. If empty, no ruler line is drawn.
- Type:
- Parameters:
interactive (bool) – Should this be in interactive mode or not.
- _list_helpers(dict_to_list, group_filter=None)[source]
Recursively produce a flat list of available Helpers.
- Parameters:
- Returns:
The list of available Helpers.
- Return type:
- _prepare_helpers()[source]
Determine the available Helpers and load them into memory.
Assumed to be called once at start-up (object creation).
- _print_help_doc(cmds=None, helper_list=None, misc_list=None, undoc_list=None)[source]
Print the “help” documentation using rich.
- _setup_logging()[source]
Create custom CLI logger.
The CLI logger should have the output include the time, level, session ID and sequence number. All CLI logging should use the same logger. If the existing log file is inaccessible this method tries to create a new file by appending the current users username. In the event which the original log file was created by a different user (and therefore has different permissions) this will enable logging to continue. If a log file is still unable to be created (likely due to a directory which cannot be accessed) the CLI will continue without a log file.
- base_do_help(arg)[source]
List available commands with “help” or detailed help with “help cmd”.
This is a slightly modified version of do_help from the base class. Here, if the help method isn’t found, it raises the AttributeError to the calling method.
- Parameters:
arg (str) – the command/Helper from which we need to get the help docs.
- Raises:
AttributeError – Caused if the help method isn’t found.
- complete_author(text, line, _begidx, _endidx)[source]
Tab completion for the author command.
We expect a Helper name, so we need to give the list of Helper names as potential completions.
- complete_help(text, line, _begidx, _endidx)[source]
Tab completion for the help command.
We expect a Helper or command name, so we need to give the list of Helper names as potential completions.
- complete_run(text, line, _begidx, _endidx)[source]
Tab completion for the run command.
We expect a Helper name, so we need to give the list of Helper names as potential completions.
- default(line)[source]
Defined by cmd for the action taken when an unrecognized command is given.
In our system this could be the name of Helper, so we’ll search for it.
- do_author(args)[source]
Print the AUTHOR section of the specified Helper.
- Parameters:
args (str) – The Helper name for which to print the author.
Examples
$ firewheel author experiment FIREWHEEL Team
- do_config(args)[source]
Enables command-line access to get and set the FIREWHEEL configuration.
This command essentially calls
firewheel.cli.configure_firewheel.ConfigureFirewheel
.- Parameters:
args (str) – This argument is passed to
firewheel.cli.configure_firewheel.ConfigureFirewheel
to handle.
- do_docs(args)[source]
Generate documentation file for all available Helpers and commands.
This command generates an RST file (helper_docs.rst) which contains the DESCRIPTION section for all available Helpers. Additionally, the docstring for all available commands is compiled into a single RST file (commands.rst). These files are then written to the input location or, if no argument is passed in, to
../../../../docs/source/cli
which is where FIREWHEEL’s CLI documentation is located if the repository has been cloned.- Parameters:
args (str) – Optional directory to write docs to. If not provided, this path will be
../../../../docs/source/cli
.
Examples
$ firewheel docs FIREWHEEL Helper documentation placed in: /opt/firewheel/docs/source/cli/helper_docs.rst FIREWHEEL Command documentation placed in: /opt/firewheel/docs/source/cli/commands.rst
- do_help(arg)[source]
Print the help text for Helpers and commands.
For Helpers, the DESCRIPTION section is printed. For Commands, the docstring is printed. In interactive mode all commands/Helpers can be tab completed.
- Parameters:
arg (str) – the command/Helper from which we need to get the help docs.
Example
$ firewheel help history Print the history of commands/Helpers. Shows full command line as entered and includes the associated sequence number ...
$ firewheel help vm mix Generates a table showing the VM Images for a running experiment. The table also includes the power state of the VMs and the vm_resource state. Images that are the same and have the same power/vm_resource state are grouped. The count of the various VMs are provided. Additionally, the total number of scheduled VMs is shown at the bottom of the table. ...
- do_history(args)[source]
Print the history of commands/Helpers.
Shows full command line as entered and includes the associated sequence number and session ID. History is preserved between sessions and until the logs are cleared (typically during a
firewheel restart hard
. The output is shown in the form of<Count>: <ID>:<Sequence Number> -- <command>
.- Parameters:
args (str) – This argument is ignored.
Example
$ firewheel history <Count>: <ID>:<Sequence Number> -- <command> 0: 1ff79073-5e4a-4279-9d4c-8d81168736b1:0 -- vm mix 1: 1fcb30cb-00fb-4179-b99c-b2f4ae6f7577:0 -- list 2: a7af6f9c-6eb3-46b4-b6d8-9c0f9604808d:0 -- version ...
- do_init(args)[source]
Initializes FIREWHEEL on this node.
- Parameters:
args (str) – This argument is passed to
firewheel.cli.init_firewheel.InitFirewheel
for command interpretation.
Examples
$ firewheel init Checking /usr/bin/firewheel: OK Checking write access to default_output_dir: /tmp/firewheel Successfully able to write to default_output_dir. ...
- do_list(args)[source]
List the available Helpers by name.
This enables users to identify all the available FIREWHEEL Helpers. Users can optionally filter the list by partially completing a Helper name.
- Parameters:
args (str) – Optionally specify a group to list.
Examples
$ firewheel list FIREWHEEL Helper commands: example_helpers pytest example_helpers subgroup index ...
$ firewheel list FIREWHEEL Helper commands containing 'vm:' vm list vm mix
- do_run(args)[source]
Runs the scripts found in the specified Helper file.
This command is functionally equivalent to running the same Helper without the keyword run in front of it. It is largely useful when using interactive mode.
- Parameters:
args (str) – Name of the Helper to execute.
- Returns:
The result of
firewheel.cli.firewheel_cli.FirewheelCLI.handle_run()
which is the number of executable sections in the Helper that encountered errors. 0 on success. Negative (e.g. -1) on other errors.- Return type:
Examples
$ firewheel run start_time Experiment start time: 03-25-2020 16:19:38 UTC
- do_sync(_args)[source]
Update the Helper cache on all hosts controlled by the CLI.
This command essentially calls
firewheel.cli.host_accessor.sync()
. All Helpers are executed from this cache. Therefore, this command should be run on the creation of a new FIREWHEEL cluster and after updating a Helper.- Parameters:
_args (str) – This argument is ignored.
Example
$ firewheel sync $
- do_version(arg)[source]
Print FIREWHEEL’s version.
- Parameters:
arg (str) – This argument is ignored.
Example
$ firewheel version 2.6.0
- emptyline()[source]
Defined by cmd as the action taken when a blank line is entered.
We want this to be a no-op.
- handle_run(args)[source]
Brains of running a Helper that are common to both ways of invoking a Helper.
Error handling may want different messages, so we don’t handle much here, just (rudely) throw things.
- Parameters:
args (str) – The Helper invocation string (arguments to the “run” command or the whole command-line if “run” wasn’t used).
- Returns:
The number of executable sections in the Helper that encountered errors. 0 on success. Negative (e.g. -1) on other errors.
- Return type:
- Raises:
HelperNotFoundError – If the named Helper cannot be located.
Exception – If and thing there is a terrible error.
- help_author()[source]
Help message for the author command.
- Returns:
The docstring for
do_author()
.- Return type:
- help_config()[source]
Help message for the config command.
- Returns:
The help message for
firewheel.cli.configure_firewheel.ConfigureFirewheel
.- Return type:
- help_history()[source]
Help message for the history command.
- Returns:
The docstring for
do_history()
.- Return type:
- help_init()[source]
Help message for the init command.
- Returns:
The help message for
firewheel.cli.init_firewheel.InitFirewheel
.- Return type:
- help_version()[source]
Help message for the version command.
- Returns:
The docstring for
do_version()
.- Return type:
- invalid_helper(arg)[source]
Identify what the user was asking help on and give them a starting point.
- Parameters:
arg (str) – Specify a group to list.
- postcmd(stop, line)[source]
Defined by cmd as the action taken after each command is completed.
We use this method to track our history.
- Parameters:
stop (func) – Used internally by cmd, needs to be returned.
line (str) – The line that has been executed.
- Returns:
The previously passed in parameter
- Return type:
stop
configure_firewheel.py
- class firewheel.cli.configure_firewheel.ConfigureFirewheel[source]
Enables command-line access to get and set the FIREWHEEL configuration.
Users can interact with the config command (i.e.
firewheel config
) series of sub-commands which enable easily getting/setting various configuration options.- __init__() None [source]
Initialize the
cmd.Cmd
and the class logger.- log
The log to use for class.
- Type:
- _help_get() str [source]
Help message for the
do_get()
sub-command.- Returns:
The help message.
- Return type:
- _help_help() str [source]
Help message for the help sub-command.
- Returns:
The help message.
- Return type:
- _help_reset() str [source]
Help message for the
do_reset()
sub-command.- Returns:
The help message.
- Return type:
- _help_set() str [source]
Help message for the
do_set()
sub-command.- Returns:
The help message.
- Return type:
- define_get_parser() ArgumentParser [source]
Create an
argparse.ArgumentParser
for config get.- Returns:
The parser needed for config get.
- Return type:
- define_reset_parser() ArgumentParser [source]
Create an
argparse.ArgumentParser
for config reset.- Returns:
The parser needed for config reset.
- Return type:
- define_set_parser() ArgumentParser [source]
Create an
argparse.ArgumentParser
for config set.- Returns:
The parser needed for config set.
- Return type:
- do_get(args: str) None [source]
Enable a user to get a particular FIREWHEEL configuration option.
Users can either get a single configuration value or get the entire configuration.
- Parameters:
args (str) – A string of arguments which are passed in by the user.
- Raises:
SystemExit – If incorrect arguments are given.
- do_reset(args: str = '') None [source]
Reset the FIREWHEEL configuration to the defaults.
Users can reset the current FIREWHEEL configuration to the defaults provided in the template.
- Parameters:
args (str) – A string of arguments passed in by the user.
- do_set(args: str) None [source]
Enable a user to set a particular FIREWHEEL configuration option.
Users can either set a single configuration value or pass in a file to replace the entire configuration.
- Parameters:
args (str) – A string of arguments which are passed in by the user.
- Raises:
SystemExit – If an incorrect arguments are given.
- emptyline() None [source]
Print help when a blank line is entered.
Required by
cmd
as the action taken when a blank line is entered. We want this to print help.
- get_docs() str [source]
Get the documentation for the set of commands that relate to config.
- Returns:
The full documentation for all hostgroup commands.
- Return type:
- help_reset() None [source]
Print help for the
do_reset()
sub-command.
helper_group.py
- class firewheel.cli.helper_group.HelperGroup(helper_path)[source]
Represent a group of Helpers.
This extends a basic dictionary by implementing the build_cache() method.
- __init__(helper_path)[source]
Initialize some class variables.
- Parameters:
helper_path (str) – The name of the Helper group.
- __iter__()[source]
Return iterator over the self.helper keys.
- Returns:
The iterable of self.helper keys.
- Return type:
iterable
- __setitem__(key, item)[source]
Dictionary-style write access.
- Parameters:
key (str) – The string key (i.e. Helper name)
item (Any) – The value.
- build_cache(path=None)[source]
Make sure the cache structure is correctly in place.
This will then call
build_cache()
for each Helper we have.- Parameters:
path (str) – The path to the cache directory. By default it uses the FIREWHEEL CLI cache_dir.
helper.py
- class firewheel.cli.helper.Helper(helper_path, helpers_root)[source]
Class representing a single Helper.
Allows access to all sections as a dictionary, keyed on section name. Because multiples are allowed for RUN section, the entry for “RUN” is actually a list. This entry is not expected to be accessed directly, but rather through methods like run.
- REQUIRED_SECTIONS = ['RUN', 'AUTHOR', 'DESCRIPTION']
- SECTION_TERMINATOR = 'DONE'
- __init__(helper_path, helpers_root)[source]
Initialize this Helper, reading our sections from the file defining us.
- _read_sections(helper_path)[source]
Read the sections from definition file and store them in a dictionary.
- Parameters:
helper_path (str) – The absolute path to the Helper definition file.
- Raises:
MalformedSectionError – Caused by syntax errors.
- build_cache(path=None)[source]
Build the cache of this Helper’s RUN sections on the local system.
- Parameters:
path (str) – The path to the cache directory. By default it uses the FIREWHEEL CLI cache_dir.
- run(session, arguments)[source]
Run this Helper (all RUN sections).
Executes the list found in the “RUN” key, not necessarily all ExecutableSections (although these 2 groups should always be the same).
- Parameters:
- Returns:
The summation of the error codes from the executable sections that were run. In the case of the Helpers executor, this would be the number of Helpers that failed. In the case of the LocalPython executor this would be the error code from the Helper itself. For the Python and Shell executors, the number of nodes which produced errors. This will return 0 on success.
- Return type:
- Raises:
MalformedSectionError – Caused if there are no sections in the “RUN” key or if an object in the “RUN” key returns false from its is_executable method.
host_accessor.py
Enable interaction with the FIREWHEEL cluster.
This module is responsible for containing classes and functions which communicate
to the rest of the FIREWHEEL cluster. This module uses ClusterShell to send
files and commands to the necessary nodes. This module also includes a custom
ClusterShell ClusterShell.Event.EventHandler
.
- class firewheel.cli.host_accessor.ClusterHandler(cmd=None)[source]
Define a custom
ClusterShell.Event.EventHandler
for FIREWHEEL.This helps process the results from ClusterShell tasks. Most of the processing requires and handling/logging errors and printing output.
- __init__(cmd=None)[source]
Initialize the event handler.
- log
Provides the logger to the various methods.
- Type:
- Parameters:
cmd (str) – The command which is being executed.
- ev_close(worker, timedout)[source]
Indicate that a worker has just finished.
- Parameters:
worker (ClusterShell.Worker.Ssh.WorkerSsh) – Worker derived object.
timedout (bool) – Indicates if the worker has timed out.
- ev_hup(_worker, node, rc)[source]
Indicate that a worker command for a specific node has just finished.
This method also helps count errors that occur with the various workers.
- Parameters:
_worker (ClusterShell.Worker.Ssh.WorkerSsh) – Worker derived object.
node (str) – The node which completed the task.
rc (int) – The command return code (or None if the worker doesn’t support command return codes).
- ev_read(_worker, node, sname, msg)[source]
Indicate that a worker has data to read from a specific node (or key).
- Parameters:
_worker (ClusterShell.Worker.Ssh.WorkerSsh) – Worker derived object.
node (str) – The node which completed the task.
sname (str) – The stream name (stdout or stderr).
msg (bytes) – The resulting output/message from the task.
- class firewheel.cli.host_accessor.HostAccessor(host_group_name)[source]
Enable copying files and running commands with a FIREWHEEL cluster.
Loads the host group from the config information and permits access to hosts. Allows file transfer and single-command execution leveraging ClusterShell.
- __init__(host_group_name)[source]
Initialize the HostAccessor attributes and ensure the host group is valid.
- log
A class-accessible log.
- Type:
- ssh_options
Necessary SSH options for ClusterShell. Currently the default options used include: *
-C
- Enable compression *-oStrictHostKeyChecking=no
- Disable host key checking because many clusters use the same hosts but have new keys. *-oUserKnownHostsFile=/dev/null
- Blackholing the known hosts file due to auto-generated keys. *-oNumberOfPasswordPrompts=1
- Only get a single password prompt (if any). *-oLogLevel=ERROR
- Remove any debug/warnings from SSH connections.- Type:
- Parameters:
host_group_name (str) – The name of the host group we’re representing.
- Raises:
RuntimeError – If the hostgroup is not valid.
- copy_file(local_file_path, remote_file_path)[source]
Copy a file from the local system to all remote systems in the host group.
File permissions (such as execute) are preserved.
- run_command(command_string, session, arguments)[source]
Run a single command on all remote systems in the host group.
This method attempts to run the command on the remote nodes. If the command fails with status code 2 or 127, this may indicate that the command/Helper is not found on the remote system. Therefore, we call
sync()
and then retry. If the retry fails or if there are other non-zero status codes the number of errors is returned.Before each command, this method appends a change directory command which changes directory to the users current directory.
- Parameters:
- Returns:
Non-zero if an unresolvable error is encountered in execution. Zero otherwise.
- Return type:
init_firewheel.py
- class firewheel.cli.init_firewheel.InitFirewheel(cli_path: None = None)[source]
Enables easy ability for a user to “initialize” a FIREWHEEL node.
Initialization includes checking various FIREWHEEL config path and verifying that non-standard dependencies (minimega and discovery) are installed and working.
- __annotations__ = {}
- __init__(cli_path: None = None) None [source]
This is the constructor which initializes several class variables.
- Parameters:
cli_path (str) – The path to the CLI binary.
- _check_minimega_socket()[source]
Check the status of minimega’s socket.
- Returns:
False if minimega is not running, True otherwise.
- Return type:
- _help_help()[source]
Help message for the help sub-command.
- Returns:
The help message.
- Return type:
- _help_static() str [source]
Help message for the
do_static()
sub-command.- Returns:
The help message.
- Return type:
- default(_line)[source]
Defined by cmd for the action taken when an unrecognized command is given.
In the case of
InitFirewheel
there are no sub-commands, so any incorrect commands will run this method.- Parameters:
_line (str) – The command-line as entered by the user.
- do_static(_args)[source]
Do not check if any services are running any only check if they exist.
- Parameters:
_args (str) – This is unused in this method.
- doc_header = 'Initialize the FIREWHEEL cluster.'
- emptyline()[source]
Print help when a blank line is entered.
Required by
cmd
as the action taken when a blank line is entered. We want this to print help.
- get_docs() str [source]
Get the documentation for the set of commands that relate to init.
- Returns:
The full documentation for init command.
- Return type:
- help_static()[source]
Help message for the static command.
- Returns:
The docstring for
do_static()
.- Return type:
section.py
- exception firewheel.cli.section.MalformedSectionError[source]
Error for when a section declaration doesn’t meet our syntax expectations.
- class firewheel.cli.section.Section(content, _arguments)[source]
A basic section, the building-block of a Helper.
This is a “text” section. It isn’t runnable, but stores the section contents. These contents may be requested by the user and we’ll print them.
- __init__(content, _arguments)[source]
Initialize ourselves.
We don’t do any operations besides setting instance variables.
- Parameters:
content (list) – The content of this section.
_arguments – This argument is ignored.
- Raises:
MalformedSectionError – If the content is not a list.
- get_file_extension()[source]
Needed by ExecutableSections.
- Raises:
NotImplementedError – This method is not used.
- has_content()[source]
Checks if the item has any content.
- Returns:
True if content exists and False if content does not exists
- Return type:
executable_section.py
- exception firewheel.cli.executable_section.BadExecutorError[source]
Error for when a specified Executor doesn’t exist.
- class firewheel.cli.executable_section.ExecutableSection(content: int | List[str], arguments: List[str] | str | List[str | List[str]] | None)[source]
Represent a RUN section from a Helper.
RUN sections are like other sections, except they have an execute method.
- __annotations__ = {}
- __init__(content: int | List[str], arguments: List[str] | str | List[str | List[str]] | None) None [source]
Constructor, invokes Section (superclass) constructor.
Store our section content and process our arguments to make sure we have an Executor name and at least one host group to run on.
- Parameters:
content (str) – The content of the RUN section we’re representing, as one string.
arguments (list) – The arguments from the declaration of this section, as a list. We expect at least 2 items in this list. The first item is taken to be an executor name. The second item is taken to be a list of host groups.
- Raises:
MalformedSectionError – If the arguments are not a list.
IllegalListError – If there are not exactly two elements in the
arguments
list.
- _load_executor(name: str) Type[Helpers] | Type[LocalPython] | Type[Python] | Type[Shell] [source]
Initialize and return the given executor.
- Parameters:
name (str) – Name of the executor to load, should be in (upper) camel case
- Returns:
The loaded executor class
- Return type:
- Raises:
BadExecutorError – If there are non-ascii characters in the executor name.
BadExecutorError – If the executor is not found.
Exception – If an unknown error occurs while loading the executor.
- execute(cache_file: str, session: Dict[str, Decimal | int | UUID], arguments: List[str] | None) int [source]
Try to create the correct Executor and have it run our content.
- Parameters:
- Returns:
A summation of the return codes from the executed Helpers. Generally this will be the number of hosts that failed. However, some Helpers may give specific error codes which are then added. This will be 0 on success.
- Return type:
- get_file_extension() str [source]
Gets the file extension expected by the Executor for this section.
- Returns:
The file extension expected by the Executor for this section.
- Return type:
- Raises:
BadExecutorError – If there is an error with the executor.
utils.py
- exception firewheel.cli.utils.HelperNotFoundError[source]
Error thrown when we cannot locate a referenced Helper.
- exception firewheel.cli.utils.InvalidHelperTypeError[source]
Error thrown when we find a Helper group instead of a Helper.
- class firewheel.cli.utils.ParallelSSHManager(max_call_attempts=10, capture_output=False, test_connections=False)[source]
Class managing Parallel SSH calls made from a helper.
- __abstractmethods__ = frozenset({})
- __call__(command, options=())[source]
Execute a parallel SSH call.
- Parameters:
- Returns:
The result of the SSH-based subprocess call.
- Return type:
CompletedProcess
- Raises:
ValueError – Raised when neither the
'h'
nor'-H'
option is provided to define the destination hosts.
- __init__(max_call_attempts=10, capture_output=False, test_connections=False)[source]
Initialize the call manager, setting general parameters.
Establishes an object that executes parallel-ssh commands for use by the FIREWHEEL parallel-ssh helper. It uses the minimega API to determine the current host, then uses that information to make subsequent parallel-ssh calls to remote VMs.
- Parameters:
max_call_attempts (int) – The number of times to attempt making calls before throwing an error back to the user. By default the ” caller attempts 10 times.
capture_output (bool) – A flag indicating whether subprocess calls will capture output. The default is
False
.test_connections (bool) – A flag indicating whether to test connections before attempting to execute an SSH-based command. The default is
False
. Note that when set, connections will be checked individually and performance may degrade.
- _abc_impl = <_abc._abc_data object>
- _connection_test_command = 'ping -w 1 -c 1'
- _connection_test_interval = 2
- _form_parallel_ssh_instruction(command, options)[source]
Add necessary arguments to the user-provided parallel-ssh command.
Create an parallel-ssh-specific instruction. Combine any options specified by the user with options that are required by the experimental setup.
- Parameters:
- Returns:
The complete parallel-ssh instruction.
- Return type:
- _prepare_options(options)[source]
Prepare options for inclusion in the protocol instruction.
Join options specified on the command line with the required default options. Joins options lists (tuples of options and arguments) as a string ready to be included in the protocol instruction.
- argument_options = 'hHlpoetOxX'
- property default_options
Default options for the parallel-ssh protocol (set by FIREWHEEL).
- Returns:
The list of default options to include in the protocol instruction.
- Return type:
- flag_options = 'vAiIP'
- classmethod parse_cli_input(argv)[source]
Given FIREWHEEL SSH input from the command line, separate the options.
Takes inputs from the command line for executing SSH to a FIREWHEEL VM and separates it into a list of options and a list of positional arguments.
- Parameters:
argv (list) – Command line input components (e.g., in the format of
sys.argv
).- Returns:
A tuple of
(args, optlist)
. Whereargs
is a list of arguments left over after options have been stripped. The first is a string giving the target address, the second is a string giving the command to be executed after establishing the SSH connection.optlist
is a list of tuples giving(option, value)
pairs.- Return type:
- protocol_name = 'PSSH'
- class firewheel.cli.utils.RichDefaultTable(*args, **kwargs)[source]
A the default table format for FIREWHEEL outputs.
This is the specification for a default FIREWHEEL table displayed via the CLI. It is a subclass of the
rich.table.Table
with specific attributes set to default values for FIREWHEEL.- __annotations__ = {}
- __init__(*args, **kwargs)[source]
Build a default table for FIREWHEEL CLI output.
- Parameters:
*args – Arguments passed to the
rich.table.Table
constructor.**kwargs – Keyword arguments passed to the
rich.table.Table
constructor.
- class firewheel.cli.utils.SCPManager(max_call_attempts=10, capture_output=False, test_connections=True)[source]
Class managing SCP calls made from a helper.
- __abstractmethods__ = frozenset({})
- __annotations__ = {}
- __call__(target, *sources, options=())[source]
Execute an SCP call.
- Parameters:
target (str) – The target file to be produced, provided as a string consisting of a username, hostname, and file path. (See the SCP manual entry for formatting guidance.)
*sources – The source files to be copied, each provided as a string consisting of a username, hostname, and file path. (See the SCP manual entry for formatting guidance.)
options (list) – A list of options to pass to the SCP instruction given as tuples of
(option, value)
pairs. The second element (thevalue
) is the empty string for options that take no arguments.
- Returns:
The result of the SSH-based subprocess call.
- Return type:
CompletedProcess
- Raises:
ValueError – Raised when no source files are provided.
- _abc_impl = <_abc._abc_data object>
- _connection_test_command = 'ping -c 1'
- _connection_test_interval = 4
- property _control_network_err_msg
- _form_scp_instruction(target, *sources, options)[source]
Add necessary arguments to the user-provided SCP command.
Create an SCP-specific instruction. Combine any arguments specified by the user with their username and the remote VM IP address, along with options that are required by the experimental setup.
- Parameters:
target (str) – The target file to be produced, provided as a string consisting of a username, VM IP address, and file path.
*sources – The source files to be copied, each provided as a string consisting of a username, hostname, and file path.
options (list) – A list of options to pass to the SCP instruction given as tuples of
(option, value)
pairs. The second element (thevalue
) is the empty string for options that take no arguments.
- Returns:
The complete SSH instruction.
- Return type:
- argument_options = 'cFiloPS'
- property default_options
Default options for the SCP protocol (set by FIREWHEEL).
- Returns:
The list of default options to include in the protocol instruction.
- Return type:
- flag_options = '12346BCpqrv'
- classmethod parse_cli_input(argv)[source]
Given FIREWHEEL SCP input from the command line, separate the options.
Takes inputs from the command line for executing SCP with FIREWHEEL VMs and separates the inputs into a list of positional arguments and a list of options.
- Parameters:
argv (list) – Command line input components (e.g., in the format of sys.argv).
- Returns:
A tuple of
(args, optlist)
. Whereargs
is a list of arguments left over after options have been stripped. The first is the target destination, while all other arguments are the source files to copy to that target.optlist
is a list of tuples giving(option, value)
pairs.- Return type:
- protocol_name = 'SCP'
- class firewheel.cli.utils.SSHManager(max_call_attempts=10, capture_output=False, test_connections=True)[source]
Class managing SSH calls made from a helper.
- __abstractmethods__ = frozenset({})
- __annotations__ = {}
- __call__(destination, command='', options=())[source]
Execute an SSH call.
- Parameters:
destination (str) – The machine to be accessed, provided as a string consisting of a username and hostname (separated by an ‘@’ symbol, per convention). If the username is not provided it is inferred to be the current user’s username.
command (str) – A command to execute after making the SSH connection.
options (list) – A list of options to pass to the SSH instruction given as tuples of
(option, value)
pairs. The second element (thevalue
) is the empty string for options that take no arguments.
- Returns:
The result of the SSH-based subprocess call.
- Return type:
CompletedProcess
- __init__(max_call_attempts=10, capture_output=False, test_connections=True)[source]
Initialize the call manager, setting general parameters.
Establishes an object that executes SSH commands for use by the FIREWHEEL SSH helper. It uses the minimega API to determine the current host, then uses that information to make subsequent SSH calls to remote VMs.
- Parameters:
max_call_attempts (int) – The number of times to attempt making calls before throwing an error back to the user. By default the ” caller attempts 10 times.
capture_output (bool) – A flag indicating whether subprocess calls will capture output. The default is
False
.test_connections (bool) – A flag indicating whether to test connections before attempting to execute an SSH-based command. The default is
True
.
- _abc_impl = <_abc._abc_data object>
- _connection_test_command = 'ping -w 1 -c 1'
- _connection_test_interval = 2
- _form_ssh_instruction(destination, command, options)[source]
Add necessary arguments to the user-provided SSH command.
Create an SSH-specific instruction. Combine any options specified by the user with their username and the remote VM IP address, along with options that are required by the experimental setup.
- Parameters:
destination (str) – The machine to be accessed, provided as a string consisting of a username and VM IP address.
command (str) – A command to execute after making the SSH connection.
options (list) – A list of options to pass to the SSH instruction given as tuples of
(option, value)
pairs. The second element (thevalue
) is the empty string for options that take no arguments.
- Returns:
The complete SSH instruction.
- Return type:
- argument_options = 'bcDEeFIiJLlmOopQRSWw'
- flag_options = '1246AaCfGgKkMNnqsTtVvXxYy'
- classmethod parse_cli_input(argv)[source]
Given FIREWHEEL SSH input from the command line, separate the options.
Takes inputs from the command line for executing SSH to a FIREWHEEL VM and separates it into a list of options and a list of positional arguments.
- Parameters:
argv (list) – Command line input components (e.g., in the format of
sys.argv
).- Returns:
A tuple of
(args, optlist)
. Whereargs
is a list of arguments left over after options have been stripped. The first is a string giving the target address, the second is a string giving the command to be executed after establishing the SSH connection.optlist
is a list of tuples giving(option, value)
pairs.- Return type:
- protocol_name = 'SSH'
- class firewheel.cli.utils._SSHProtocolManager(max_call_attempts=10, capture_output=False, test_connections=True)[source]
Class managing SSH-based protocol calls made from a helper.;
- __abstractmethods__ = frozenset({'argument_options', 'flag_options', 'protocol_name'})
- __annotations__ = {}
- _abc_impl = <_abc._abc_data object>
- _call(*args, options)[source]
Generalized call method.
This is a call method that is generalized for SSH-based commands. All of those commands require positional arguments and options.
- Parameters:
*args – Additional positional arguments that are specific to the command being executed.
options (list) – A list of options to pass to the SSH/SCP instruction given as tuples of
(option, value)
pairs. The second element (thevalue
) is the empty string for options that take no arguments.
- Returns:
The result of the SSH-based subprocess call.
- Return type:
CompletedProcess
- property _connection_test_command
- property _connection_test_interval
- _console = <console width=80 None>
- property _control_network_err_msg
- classmethod _exit(*messages, display_usage=False)[source]
Exit the method, printing any messages to the console.
- Parameters:
*messages – A sequence of objects that can be passed to the console object for this manager.
display_usage (bool) – A flag indicating whether the protocol usage should be displayed after exiting.
- _get_remote_vm_ip(remote_hostname)[source]
Look up the specified VM from among the available minimega VMs.
Given the hostname for a remote VM, look up the IP address for that VM and test that a connection can be made. If not, or the IP address does not exist, fail gracefully.
- _prepare_options(options)[source]
Prepare options for inclusion in the protocol instruction.
Join options specified on the command line with the required default options. Joins options lists (tuples of options and arguments) as a string ready to be included in the protocol instruction.
- _test_vm_connection(vm_ip)[source]
Test the connection to the VM.
- Parameters:
vm_ip (str) – The IP address of the VM whose connection is being tested.
- abstract property argument_options
- property default_options
Default options for the protocol (set by FIREWHEEL).
- Returns:
The list of default options to include in the protocol instruction.
- Return type:
- abstract property flag_options
- classmethod parse_cli_input(argv)[source]
Given FIREWHEEL input from the command line, separate the options.
Takes inputs from the command line and separates it into a list of options and a list of positional arguments.
- abstract property protocol_name
- firewheel.cli.utils.build_helper_dict()[source]
Builds a dictionary of Helper objects.
This method uses
process_helper_group()
and passes in the directory of the CLI Helpers.- Returns:
A dictionary of Helper and HelperGroup objects.
- Return type:
dictionary
- firewheel.cli.utils.load_helper(filename, helper_dict)[source]
Identify the Helper path, create Helper object, and add it to the working_dict.
- firewheel.cli.utils.parse_destination(remote_name)[source]
Split a machine name destination into its user and host components.
Given the name of a machine destination, separate the name into its user and host components. The components are split by the ‘@’ symbol.
- firewheel.cli.utils.parse_location(location)[source]
Split a file location into its user/hostname and path components.
Given the location of a file, separate the string into its user/hostname and path components. The components are split by a colon.
- Parameters:
location (str) – The location of a file (including user/hostname and the filename).
- Returns:
A tuple of
(user_host, filename)
. Whereuser_host
is a string of the user/hostname component andfilename
is a string of the name of the file on the machine described by the first component.- Return type:
Note
This method assumes that all filenames do not contain any colons. Colons are permitted in the user/hostname pair, in addition to their use as the standard separator between the machine name and the filename.
- firewheel.cli.utils.parse_to_helper(args, helpers_dict)[source]
Retrieve a Helper object for the given command-line args.
- Parameters:
- Returns:
Containing the Helper object that was found and the list of arguments for the Helper.
- Return type:
- Raises:
HelperNotFoundError – If the Helper cannot be located.
InvalidHelperTypeError – If a Helper group was specified rather than a Helper.
- firewheel.cli.utils.process_helper_group(directory_name, helper_dict)[source]
Identify the path of a Helper group recursively.
This method then calls load_helper to load the Helper with the correct path. The Helpers are added to the helper_dict.
executors/abstract_executor.py
- class firewheel.cli.executors.abstract_executor.AbstractExecutor(host_list_path, content)[source]
Abstract base class to define the Executor interface.
As it currently stands, the division of work between __init__ and execute isn’t all that important because they’re always called in immediate succession.
- __abstractmethods__ = frozenset({'execute', 'get_file_extension'})
- __annotations__ = {}
- __init__(host_list_path, content)[source]
Create a new Executor.
Executors are intended to run a specific type of content. A new Executor is created for each RUN section to be executed. The content is always run on a group of hosts (there may only be 1 host in this group, however).
- Parameters:
host_list_path (list) – A list of hostnames (or IP addresses as strings). If hostnames are used, the system must be able to resolve them (when ClusterShell is ultimately called).
content (list) – The section’s content that we want to execute. We’re expected to dump it into a file for transfer, compile it, etc.
- _abc_impl = <_abc._abc_data object>
- abstract execute(cache_file, session, arguments)[source]
Run our remote command(s).
This method works best when it prints helpful error messages instead of throwing Exceptions. The CLI won’t really handle decent error messages for us (it’s too generic), but it shouldn’t crash if we do end up throwing something.
- Parameters:
cache_file (str) – The file (cached copy of the section content) to try and run from.
session (dict) – The overall CLI’s view of the current session. This includes the current sequence number and session ID.
arguments (list) – Command-line arguments for the remote command. We are expected to pass this along to the HostAccessor’s command execution method as well.
- Raises:
NotImplementedError – This should be implemented by a subclass.
- abstract get_file_extension()[source]
Return the file extension expected for input files used with this Executor.
This information is expected to be used to cache Helpers’ RUN sections remotely.
For example, a shell executor would expect a .sh extension.
- Raises:
NotImplementedError – This should be implements by a subclass.
executors/helpers.py
- class firewheel.cli.executors.helpers.Helpers(host_list_path: List[str], content: List[str], helper_dict: Dict[str, Helper] | None = None)[source]
An Executor to handle invoking a list of Helpers (newline separated).
This class tries to avoid throwing any exceptions.
- __abstractmethods__ = frozenset({})
- __annotations__ = {}
- __init__(host_list_path: List[str], content: List[str], helper_dict: Dict[str, Helper] | None = None) None [source]
Initialize.
- Parameters:
host_list_path (list) – A list of hostnames (or IP addresses as strings). If hostnames are used, the system must be able to resolve them (when ClusterShell is ultimately called).
content (list) – The section’s content that we want to execute. We’re expected to dump it into a file for transfer, compile it, etc.
helper_dict (dict) – Optional dictionary of FIREWHEEL helpers with the key=str(helper_name) and the value is the Helper object (corresponding to the name).
- _abc_impl = <_abc._abc_data object>
- _prepare_helpers() None [source]
Determine the available Helpers and load them into memory.
Assumed to be called once at start-up (object creation).
- execute(_cache_file: str, session: Dict[str, Decimal], arguments: List[str] | None) int [source]
Execute Helpers specified in the content list.
All arguments are passed directly to the Helpers. This method tries to avoid throwing any Exceptions, instead printing helpful error messages.
- Parameters:
_cache_file (str) – The file (cached copy of the section content) to try and run from.
session (dict) – The overall CLI’s view of the current session. This includes the current sequence number and session ID.
arguments (list) – Command-line arguments for the remote command. We are expected to pass this along to the HostAccessor’s command execution method as well.
- Returns:
Zero on success, non-zero otherwise. In general, return values are the number of Helpers that encountered at least 1 error.
- Return type:
executors/local_python.py
- class firewheel.cli.executors.local_python.LocalPython(host_list_path, content)[source]
Executor to handle Python scripts on the machine where the CLI is run.
This class tries to avoid throwing any Exceptions.
- __abstractmethods__ = frozenset({})
- __annotations__ = {}
- _abc_impl = <_abc._abc_data object>
- execute(_cache_file: str, _session: Dict[str, Decimal], arguments: List[str] | None) int [source]
Execute Python scripts via the current Python environment.
Dump our section content into a temporary file and invoke it using the Python interpreter.
This method tries to avoid throwing any Exceptions, instead printing helpful error messages.
- Parameters:
_cache_file (str) – The file (cached copy of the section content) to try and run from.
_session (dict) – The overall CLI’s view of the current session. This is an unused argument.
arguments (list) – Command-line arguments for the local execution. This is the file of the Helper and any arguments that should be passed to the Helper.
- Returns:
Zero on success, non-zero otherwise.
- Return type:
executors/python.py
- class firewheel.cli.executors.python.Python(host_list_path, content)[source]
Executor to handle Python scripts.
This class tries to avoid throwing any Exceptions.
- __abstractmethods__ = frozenset({})
- __annotations__ = {}
- _abc_impl = <_abc._abc_data object>
- execute(cache_file: str, session: Dict[str, Decimal], arguments: List[str] | None) int [source]
Execute Python scripts via the
firewheel.cli.host_accessor.HostAccessor
.Dump our section content into a temporary file, copy it to the remote hosts, and invoke it using the Python interpreter.
This method tries to avoid throwing any Exceptions, instead printing helpful error messages.
- Parameters:
cache_file (str) – The file (cached copy of the section content) to try and run from.
session (dict) – The overall CLI’s view of the current session. This includes the current sequence number and session ID.
arguments (list) – Command-line arguments for the remote command. We are expected to pass this along to the HostAccessor’s command execution method as well.
- Returns:
Zero on success, non-zero otherwise. Expects HostAccessor to handle the details of specific return code meanings.
- Return type:
Note
FIREWHEEL assumes that the path to python is the same across the entire cluster. This means that if FIREWHEEL was installed in a virtual environment then the virtual environment should have the same path for each node in the cluster.
executors/shell.py
- class firewheel.cli.executors.shell.Shell(host_list_path, content)[source]
An Executor to handle bash shell scripts.
This class tries to avoid throwing any Exceptions.
- __abstractmethods__ = frozenset({})
- __annotations__ = {}
- _abc_impl = <_abc._abc_data object>
- execute(cache_file: str, session: Dict[str, Decimal], arguments: List[str] | None) int [source]
Execute Shell scripts via the
firewheel.cli.host_accessor.HostAccessor
.Dump our content to a temporary file, upload it to the remote hosts we’ll run on, then run that file (a shell script).
This method tries to avoid throwing any Exceptions, instead printing helpful error messages.
- Parameters:
cache_file (str) – The file (cached copy of the section content) to try and run from.
session (dict) – The overall CLI’s view of the current session. This includes the current sequence number and session ID.
arguments (list) – Command-line arguments for the remote command. We are expected to pass this along to the HostAccessor’s command execution method as well.
- Returns:
Zero on success, non-zero otherwise. Expects HostAccessor to handle the details of specific return code meanings.
- Return type:
completion/actions.py
Provide FIREWHEEL-dependent tools to be used by the Bash completion script.
- firewheel.cli.completion.actions.get_available_cli_commands()[source]
Get the set of available CLI commands.
Uses the
firewheel.cli.firewheel_cli.FirewheelCLI
object to look up the complete set of available CLI commands so that they may be used for autocompletion.
completion/get_available_cli_commands.py
Callable script wrapper for the
firewheel.cli.completion.get_available_cli_commands()
function.
completion/get_model_component_names.py
Callable script wrapper for the
firewheel.cli.completion.get_model_component_names()
function.
completion/get_total_model_components_size.py
Callable script wrapper for the
firewheel.cli.completion.get_total_model_components_size()
function.
completion/prepare_completion_script.py
Prepare the tab completion script using the template and configuration values.
This script uses values from the current FIREWHEEEL configuration to populate the tab completion template script. This generator script is always run on installation to prepare a new completion script for the current configuration, and it may be run again at any point to generate a new completion script that is is up-to-date with the current FIREWHEEL configuration. Tab completion for Bash and Zsh is explicitly supported, though the script may work with other shells permitting tab completion.
Examples
$ python -m firewheel.cli.completion.prepare_completion_script
- firewheel.cli.completion.prepare_completion_script.display_instructions(script_path: Path) None [source]
Display instructions for using the script to set autocompletion.
- Parameters:
script_path (Path) – The location where the completion script should be written.
- firewheel.cli.completion.prepare_completion_script.main() None [source]
Prepare the completion script, provide setup instructions, and print its path.