Preparing for VM resources
Now that the topology has been created and works, we can start automating the experiment actions.
In FIREWHEEL, we use the term VM Resources as a catch-all term for any automated actions which modify the experiment after the VMs start.
This could include VM configuration changes, installing new software, or automating some action.
In this experiment, we will want the Server to have the following VM resources:
A Python web server running from a specific directory.
A file located in that specific directory.
The Client will need the following VM resource:
A script to download a file from the server.
See also
For more information, please refer to the VM Resource Manager documentation.
To prepare for adding these new VM resources, we need to add the vm_resources field to our MANIFEST file (see The VM Resources field).
To make it easy, we will create a new folder in the simple_server Model Component folder called “vm_resources”:
$ mkdir vm_resources
Now open the MANIFEST file and add the lines:
vm_resources:
- vm_resources/*
Now all files located in the “vm_resources” folder will automatically be available to use as a VM resource.
Note
For general information on using VM resources in FIREWHEEL see Using VMRs in an Experiment.
Using model_component_objects.py
We can easily schedule our VM resources in plugin.py, however, as the complexity of the experiment grows, code readability decreases.
Therefore, we will be using the model_component_objects.py file to create new Objects for our Server and Client.
Open the model_component_objects.py file.
It should look like:
"""This module contains all necessary Model Component Objects for tutorials.simple_server."""
from firewheel.control.experiment_graph import require_class
class MyObject:
"""MyObject Class documentation."""
def __init__(self):
# TODO: Implement Class constructor here
pass
Creating the Server Object
We will modify that first object to create the Server. First rename “MyObject” to “SimpleServer”.
class SimpleServer:
Next, we want the Server to be a Ubuntu1604Server.
In the Plugin, we simply decorated the server with the correct object.
In the Model Component Objects file, there is an alternative approach.
You can use FIREWHEEL’s require_class decorator to decorate
the object with the correct class.
In this case, we will first need to import Ubuntu1604Server then decorate our SimpleServer object.
from linux.ubuntu1604 import Ubuntu1604Server
@require_class(Ubuntu1604Server)
class SimpleServer:
Creating the Client Object
Initially, client object will be the same as the SimpleServer object and will also be added to the model_component_objects.py file.
It should look like:
@require_class(Ubuntu1604Server)
class SimpleClient:
"""SimpleClient Class documentation."""
def __init__(self):
# TODO: Implement Class constructor here
pass
Updating the Plugin
Now we can update the Plugin to use the newly created objects.
Open plugin.py.
First we will import our new object and remove the import statement for Ubuntu1604Server, which is no longer needed.
Then we can decorate our client/server with the newly created objects.
The new plugin will look like:
from firewheel.control.experiment_graph import AbstractPlugin, Vertex
from base_objects import Switch
from tutorials.simple_server import SimpleServer, SimpleClient
class Plugin(AbstractPlugin):
"""tutorials.simple_server plugin documentation."""
def run(self):
"""Run method documentation."""
# Create the Server
server = Vertex(self.g, name="Server")
server.decorate(SimpleServer)
# Create the switch
switch = Vertex(self.g, name="Switch")
switch.decorate(Switch)
# Connect the server and the switch
server.connect(
switch, # The Switch Vertex
"1.0.0.1", # The IP address for the server
"255.255.255.0" # The subnet mask for the IP address network
)
# Create the client
client = Vertex(self.g, name="Client")
client.decorate(SimpleClient)
# Connect the client and the switch
client.connect(
switch, # The Switch Vertex
"1.0.0.2", # The IP address for the client
"255.255.255.0" # The subnet mask for the IP address network
)
Adding the VM Resources
Now that we have a new structure, we can schedule all of our VM resources in model_component_objects.py and no longer have to update the Plugin.
Adding the Server’s VMRs
Creating a File
First, we need to create a file for the client to pull.
To do this, we will just generate a new 50MB file, and put it in our vm_resources folder.
Open up model_component_objects.py.
It’s best to create a new method in the SimpleServer object for this functionality.
To create this file we will use the os.urandom() method to generate random data which can be written to a file.
We will name the file file.txt.
The initial method will look like:
def configure_files_to_serve(self, file_size=52428800):
"""
Generate a file that is of size ``file_size`` (e.g. default of 50MB) and drop it on the VM.
Args:
file_size (int): The size of the file to create. By default it is 50MB.
"""
# Get the current executing directory
current_module_path = os.path.abspath(os.path.dirname(__file__))
# Create a path to the soon-to-be-created file
filename = "file.txt"
path = os.path.join(current_module_path, "vm_resources", filename)
# Generate the random data which will fill the file
random_bytes = os.urandom(file_size)
# Write the file to disk
with open(path, "wb") as fout:
fout.write(random_bytes)
Now that we can generate a file, we need to add it to the VM.
We will use the drop_file method to do so.
In this case, we will place file.txt in /opt/file.txt on the Server VM.
Additionally, we will want this file placed in the VM during Negative Time as this part of the experiment is used primarily for configuration of the VMs.
Add the following to the newly created method.
# Drop the new file onto the VM.
self.drop_file(
-5, # The experiment time to add the content. (i.e. during configuration)
f"/opt/{filename}", # The location on the VM of the file to drop
filename # The filename of the newly created file on the physical host.
)
Lastly, because we always want this method to run, we will add it to the Object’s init() method.
@require_class(Ubuntu1604Server)
class SimpleServer:
"""SimpleServer Class documentation."""
def __init__(self):
self.configure_files_to_serve()
Running The Server
For our web server, we will use the http.server module.
Additionally, we will want to start the server in the /opt directory on the VM so that it will have access to the newly created file.txt.
To make all of those actions happen, we can use the following line in bash.
bash -c 'pushd /opt; python3 -m http.server; popd'
Warning
When running a VM Resource, be mindful of what implicit environment variables are needed. Ignoring these assumptions can cause issues. See VMRs In-Experiment Environment for more information.
To schedule this action, we will use the run_executable method.
We want the Start Time for the VMR to be as soon as the experiment is configured (e.g. 1 second after the experiment is configured).
In the init() method, add the following lines:
# Start the web server at time=1
# The server needs to run in the ``/opt`` directory because that is where the
# file will be located.
self.run_executable(
1, # The experiment time to run this program (e.g. 1 second after start).
"bash", # The name of the executable program to run.
arguments="-c 'pushd /opt; python3 -m http.server; popd'" # The arguments for the program.
)
The full SimpleServer Object should now look like:
@require_class(Ubuntu1604Server)
class SimpleServer:
"""SimpleServer Class documentation."""
def __init__(self):
self.configure_files_to_serve()
# Start the web server at time=1
# The server needs to run in the ``/opt`` directory because that is where the
# file will be located.
self.run_executable(
1, # The experiment time to run this program (e.g. 1 second after start).
"bash", # The name of the executable program to run.
arguments="-c 'pushd /opt; python3 -m http.server; popd'" # The arguments for the program.
)
def configure_files_to_serve(self, file_size=52428800):
"""Generate a file that is of size ``file_size`` (e.g. 50MB) and drop it on the VM.
Args:
file_size (int): The size of the file to create. By default it is 50MB.
"""
# Get the current executing directory
current_module_path = os.path.abspath(os.path.dirname(__file__))
# Create a path to the soon-to-be-created file
filename = "file.txt"
path = os.path.join(current_module_path, "vm_resources", filename)
# Generate the random data which will fill the file
random_bytes = os.urandom(file_size)
# Write the file to disk
with open(path, "wb") as fout:
fout.write(random_bytes)
# Drop the new file onto the VM.
self.drop_file(
-5, # The experiment time to add the content. (i.e. during configuration)
f"/opt/{filename}", # The location on the VM of the file to drop
filename # The filename of the newly created file on the physical host.
)
Adding the Client’s VMRs
The client is responsible for requesting the file from the web server and calculating how much time it took.
Because we need to measure the download speed, we need to be thoughtful about how to present the resulting data.
There are several methods for getting data out of an experiment.
We can pull specific files out of the experiment using either the pull file CLI command, we can schedule file extraction using the file_transfer method, or we can use the automatically extracted VM resource stdout as detailed in Extracting VMR data.
In this case, we will opt to use the output from our VM resources (see Data Transfer/Interaction for more details on the other methods).
Additionally, because we want to visualize the data, we will ensure that it is in JSON format.
As it turns out, the cURL command has the ability to write output in a specific format.
Therefore, we will use this feature (the -w flag) to help grab the download time.
To make use of this flag we will first drop our format string into a file on the VM, then we will run cURL on the VM using the -w flag.
Lastly, before we can cURL, we will need to know the IP address of the server.
Grabbing the File
Open model_component_objects.py to begin editing the SimpleClient.
We will begin by adding a new method to this object which takes in the Server’s IP address.
This method will run cURL and output our desired results.
@require_class(Ubuntu1604Server)
class SimpleClient:
"""SimpleClient Class documentation."""
def __init__(self):
pass
def grab_file(self, server_ip):
# Drop the cURL format string
pass
Remember, that we want the total download time in JSON format.
That is, our output should look like: {"time": "0.206"}.
We will be dropping this content into a file on the VM using the drop_content method.
def grab_file(self, server_ip):
# Drop the cURL format string
self.drop_content(
-5, # The experiment time to add the content. (i.e. during configuration)
"/opt/curl_format.txt", # The location on the VM of the file
'{"time":"%{time_total}"}\\n' # The content to add to the file.
)
Now we can call cURL to use this format and grab the file from the server.
The Start Time for the cURL command will be 10 (e.g. ten seconds after the entire experiment has been configured) because we want to allow some buffer time for the Server to start.
def grab_file(self, server_ip):
# Drop the cURL format string
self.drop_content(
-5, # The experiment time to add the content. (i.e. during configuration)
"/opt/curl_format.txt", # The location on the VM of the file
'{"time":"%{time_total}"}\\n' # The content to add to the file.
)
# Run cURL command
self.run_executable(
10, # The experiment time to run this program (e.g. 10 seconds after start).
"/usr/bin/curl", # The name of the executable program to run.
arguments=f'-w "@/opt/curl_format.txt" -O {server_ip}:8000/file.txt',
)
Note
It is best practice to use the full path of the binary being executed.
Now that our VM resources are in place, we need to update our plugin to call this method and pass in the Server’s IP address.
Updating the Plugin
Open plugin.py.
First, we will want to replace the hard-coded IP address of the server with a variable.
It should now look like this:
# Connect the server and the switch
server_ip = "1.0.0.1"
server.connect(
switch, # The Switch Vertex
server_ip, # The IP address for the server
"255.255.255.0" # The subnet mask for the IP address network
)
Lastly, at the end of the Plugin, have the client call the newly created grab_file method and pass in server_ip.
client.grab_file(server_ip)
Ensuring it works
Once all the updates have been made, restart your experiment to identify any syntax errors which might have been made. In the next section we will learn how to analyze the output from our VMRs.
Completed model_component_objects.py
"""This module contains all necessary Model Component Objects for tutorials.simple_server."""
import os
from firewheel.control.experiment_graph import require_class
from linux.ubuntu1604 import Ubuntu1604Server
@require_class(Ubuntu1604Server)
class SimpleServer:
"""SimpleServer Class documentation."""
def __init__(self):
self.configure_files_to_serve()
# Start the web server at time=1
# The server needs to run in the ``/opt`` directory because that is where the
# file will be located.
self.run_executable(
1, "bash", arguments="-c 'pushd /opt; python3 -m http.server; popd'"
)
def configure_files_to_serve(self, file_size=52428800):
"""
Generate a file that is of size ``file_size`` (e.g. default of 50MB) and drop it on the VM.
Args:
file_size (int): The size of the file to create. By default it is 50MB.
"""
# Get the current executing directory
current_module_path = os.path.abspath(os.path.dirname(__file__))
# Create a path to the soon-to-be-created file
filename = "file.txt"
path = os.path.join(current_module_path, "vm_resources", filename)
# Generate the random data which will fill the file
random_bytes = os.urandom(file_size)
# Write the file to disk
with open(path, "wb") as fout:
fout.write(random_bytes)
# Drop the new file onto the VM.
self.drop_file(
-5, # The experiment time to add the content. (i.e. during configuration)
f"/opt/{filename}", # The location on the VM of the file to drop
filename # The filename of the newly created file on the physical host.
)
@require_class(Ubuntu1604Server)
class SimpleClient:
"""SimpleClient Class documentation."""
def __init__(self):
pass
def grab_file(self, server_ip):
"""
Add a curl format to the VM and then run our ``fetch_file.sh`` VM resource
which will attempt to curl the file from the server and record the time.
"""
# Drop the cURL format string
self.drop_content(
-5, # The experiment time to add the content. (i.e. during configuration)
"/opt/curl_format.txt", # The location on the VM of the file
'{"time":"%{time_total}"}\\n' # The content to add to the file.
)
# Run cURL command
self.run_executable(
10, # The experiment time to run this program (e.g. 10 seconds after start).
"/usr/bin/curl", # The name of the executable program to run.
arguments=f'-w "@/opt/curl_format.txt" -O {server_ip}:8000/file.txt',
)
Completed plugin.py
from firewheel.control.experiment_graph import AbstractPlugin, Vertex
from base_objects import Switch
from tutorials.simple_server import SimpleServer, SimpleClient
class Plugin(AbstractPlugin):
"""tutorials.simple_server plugin documentation."""
def run(self):
"""Run method documentation."""
# Create the Server
server = Vertex(self.g, name="Server")
server.decorate(SimpleServer)
# Create the switch
switch = Vertex(self.g, name="Switch")
switch.decorate(Switch)
# Connect the server and the switch
server_ip = "1.0.0.1"
server.connect(
switch, # The Switch Vertex
server_ip, # The IP address for the server
"255.255.255.0" # The subnet mask for the IP address network
)
# Create the client
client = Vertex(self.g, name="Client")
client.decorate(SimpleClient)
# Connect the client and the switch
client.connect(
switch, # The Switch Vertex
"1.0.0.2", # The IP address for the client
"255.255.255.0" # The subnet mask for the IP address network
)
client.grab_file(server_ip)