analytics

The Analytics MC provides FIREWHEEL experiments with tools for monitoring and collecting data on each VM during an experiment. By decorating a Vertex with the Analytics MC, that Vertex gains access to methods that will schedule different analytic capabilities to be executed at the start of an experiment (time == 1). After decorating your Vertex, you will need to call the newly available methods depending on the kind of data which should be collected.

Current capabilities

All the methods are documented in analytics.Analytics, but for ease of use we list a few of the more useful methods here. Some methods are Linux dependent (e.g. strace) while others depend on psutil.

Linux dependent

  • analytics.Analytics.run_tcpdump()

    Runs tcpdump on the VM to capture any network traffic during the positive time of the experiment.

    Default command: tcpdump -U -w /opt/analytics/pcaps/tmp.pcap -Z root

  • analytics.Analytics.add_port_tracking()

    Runs netstat -tulpn every <refresh_interval_sec> seconds and tracks the changes between each call. One of three possible headers will be outputted with each log: INITIAL, ADDED, or DELETED. These headers are used to track the state of each line outputted by each netstat execution.

  • analytics.Analytics.tailf_dir()

    Given a positive ( >= 1 ) time, a target directory, and a regex to match against, this method schedules a script that will execute a tail -f on any file created in the target directory that matches on the regex. This script was designed for use by the strace method so that each new .strace file will be reported back to the VM Resource Management logs on the Compute Nodes (see Extracting VMR data more details). Each outputted line from the tail -f is wrapped in the format used by the JSONLogger (see the utilities.python), so that it is also placed into the VM Resource .json logs.

  • analytics.Analytics.strace()

    This script utilizes pgrep and a provided regular expression to determine the processes to attach strace. The programmer could, instead of using this tool, schedule the strace with the original execution of the process they want to track, which would avoid the need to find and attach to the process after the fact. However, this tool was designed in order to avoid that exact scenario.

    Default command: strace -ff -tt -s 1024 -o /opt/analytics/traces/(no_tailf_dirs|tailf_dirs)/<time>/ i.e. strace -ff -tt -s 1024 -o /opt/analytics/traces/tailf_dirs/10/

psutil dependent

Future Capabilities

We would like to be able to track the files that are dropped during the execution of an experiment. This would be great as we could see if any artifacts are stored or left behind after execution. A difficulty with this is knowing where to track for new file creation, as tracking the entire drive doesn’t seem like the smartest idea.

FIREWHEEL currently uses minimega for launching VMs. minimega has a suite of tools used to collect data similar to what this model component provides, but has additional features including VNC capture/replay. Users should visit https://minimega.org/articles/api.article to learn more about the possible ways to collect experiment data.

As an example for how to use these tools in your experiment, the below snippets are provided.

Example

model_component_objects.py

from analytics import Analytics

@require_class(Analytics)
class MyClass:

    def __init__(self):
        print("This is my test class!")

    def add_analytics(self):
        self.run_tcpdump()
        self.add_system_memory_tracking(refresh_interval_sec=10)

plugin.py

from firewheel.control.experiment_graph import AbstractPlugin, Vertex
from my_model_component import MyClass

class MyModelComponent(AbstractPlugin):

    def run(self):
        my_vm = Vertex(self.g, "myclass_vm")
        my_vm.decorate(MyClass)
        my_vm.add_analytics()

You will also have to include “analytics” in your MANIFEST file’s list of model_component dependencies. Below is not a complete MANIFEST file, but just a snippet of one to demonstrate the necessary inclusion.

MANIFEST

name: my_model_component
attributes:
    depends: []
    precedes: []
    provides: []
model_components:
    depends:
    - analytics
    precedes: []
model_component_objects: model_component_objects.py
plugin: plugin.py
Model Component Dependencies:

Available Objects

class analytics.Analytics(*args, **kwargs)[source]

Bases: object

This class provides numerous methods available for any decorated Vertex. By default, this class performs the following operations:

  • Installs the JSON logger python package.

  • Creates an /opt/analytics directory on the VM.

  • Adds the kill_analytics.py VM resource to the VM.

All other analytic functions must be called individually.

__annotate_func__ = None
__annotations_cache__ = {}
__firstlineno__ = 119
__init__(*args, **kwargs)
__static_attributes__ = ('python_version',)
add_cpu_tracking(**kwargs)[source]

Uses psutil to track the systems CPU frequencies on each CPU every <interval> seconds. Writes the output to /opt/analytics/cpu_tracking.log on the VM.

Note

This method is decorated with the analytics.run_once() decorator which ensures that, even if the method is called multiple times, the code will only be executed once.

Parameters:

refresh_interval_sec (int) – Interval to track the statistics. Defaults to 1.

add_disk_io_tracking(**kwargs)[source]

Uses psutil to track the systems disk IO statistics on each partition every <interval> seconds. Writes to /opt/analytics/disk_io_tracking.log on the VM.

Note

This method is decorated with the analytics.run_once() decorator which ensures that, even if the method is called multiple times, the code will only be executed once.

Parameters:

refresh_interval_sec (int) – Interval to track the statistics. Defaults to 5.

add_disk_usage_tracking(**kwargs)[source]

Uses psutil to track the systems disk usage on each partition every <interval> seconds. Writes to /opt/analytics/disk_usage_tracking.log on the VM.

Note

This method is decorated with the analytics.run_once() decorator which ensures that, even if the method is called multiple times, the code will only be executed once.

Parameters:

refresh_interval_sec (int) – Interval to track the statistics. Defaults to 5.

add_network_io_tracking(**kwargs)[source]

Uses psutil to track the systems network IO on each NIC every <interval> seconds. Writes to /opt/analytics/network_io_tracking.log on the VM.

Note

This method is decorated with the analytics.run_once() decorator which ensures that, even if the method is called multiple times, the code will only be executed once.

Parameters:

refresh_interval_sec (int) – Interval to track the statistics. Defaults to 1.

add_port_tracking(**kwargs)[source]

Adds a VM resource to repeatedly call netstat and track any changes.

Note

This method is decorated with the analytics.run_once() decorator which ensures that, even if the method is called multiple times, the code will only be executed once.

Parameters:

refresh_interval_sec (int) – Interval to track the statistics (e.g. time between netstat calls). Defaults to 1.

add_system_memory_tracking(**kwargs)[source]

Uses psutil to track the systems memory statistics every <interval> seconds. Writes to /opt/analytics/system_memory_tracking.log on the VM.

Note

This method is decorated with the analytics.run_once() decorator which ensures that, even if the method is called multiple times, the code will only be executed once.

Parameters:

refresh_interval_sec (int) – Interval to track the statistics. Defaults to 5.

install_inotify(**kwargs)[source]

Installs the inotify-tools package which helps with file system monitoring. For more information about this package see the following article: https://medium.com/100-days-of-linux/an-introduction-to-file-system-monitoring-tools-afd99164ce66

Note

This method is decorated with the analytics.run_once() decorator which ensures that, even if the method is called multiple times, the code will only be executed once.

install_psutil(**kwargs)[source]

Installs psutil on the VM.

Raises:

NotImplementedError – If the python version is not python3.7 or python3.10

Note

This method is decorated with the analytics.run_once() decorator which ensures that, even if the method is called multiple times, the code will only be executed once.

run_tcpdump(**kwargs)[source]

Schedule tcpdump to be called and the output saved to /opt/analytics/pcaps/tmp.pcap on the VM.

Note

This method is decorated with the analytics.run_once() decorator which ensures that, even if the method is called multiple times, the code will only be executed once.

Parameters:

  • options (str) – A string of options that can be used with the tcpdump command. We recommend avoiding setting this unless you are familiar with tcpdump. For more information about the options see: https://www.tcpdump.org/manpages/tcpdump.1.html.

  • install_tcpdump (bool) – A boolean value indicating whether or not tcpdump should be installed as part of this function call.

strace(time, process_regex, options=None, first_match_only=True, tailf_traces=True)[source]

Adds a VM resource to call pgrep for the passed in regular expression until a match, then use strace on the matched PIDs.

Parameters:

  • time (int) – The time to run the strace VM resource

  • process_regex (str) – The regex to match on to find the PIDs to strace

  • options (str) – Optional arguments with which to override the call to strace. Default options are -ff -tt -s 1024

  • first_match_only (bool) – Whether pgrep should keep finding matches or stop after the first match. Defaults to True.

  • tailf_traces (bool) – Whether each outputted trace file should use analytics.Analytics.tailf_dir() which causes it to be added to logs. Defaults to True.

tailf_dir(**kwargs)[source]

Will run tail -f on any NEW file in the specified directory with filename matching the supplied regex.

Note

This method is decorated with the analytics.run_once_with_unique() decorator which ensures that, even if the method is called multiple times, the code will only be executed once per unique directory provided. That is, users won’t accidentally be able to schedule multiple instances of tail -f on the same directory (as that would provide redundant data).

Parameters:

  • time (int) – Positive integer of when tracking will be scheduled.

  • directory (str) – Directory to track. It will be created if it doesn’t exist

  • matching_regex (str) – Very simple regex to match against filenames, double escaped. e.g. \\.\\*\\\\.log will match all files that end with .log.

analytics.run_once(func)[source]

Each function decorated with this can only be ran once in each object calling it.

Parameters:

func (function) – The passed in function to check.

Returns:

The wrapped function or None if it was previously executed.

Return type:

function

analytics.run_once_with_unique(unique_args, unique_kwargs)[source]

Runs the function only if the combined specified args and kwargs are unique to the execution.

Parameters:

  • unique_args (list) – List of integers of positions

  • unique_kwargs (list) – List of strings of kwargs

Returns:

The decorated function.

Return type:

function