Security

FIREWHEEL is a complex cluster-based software and there are a number of potential security concerns. This section explores each known issue/assumption and addresses ways to reduce risk. While several of these issues are a function of FIREWHEEL dependencies (e.g. minimega), we included them to provide complete transparency to users. Additionally, we are working with the developers of these tools to address potential concerns.

Known Issues

Passwordless SSH

FIREWHEEL’s CLI depends on having SSH access to the entire FIREWHEEL Cluster. This remains true even when running on a single node. This access enables FIREWHEEL to be able to sync the Helper cache and to execute CLI Helpers. For convenience, users likely will not set a password on the SSH private key file. For more information about this topic, people can review this article.

Risks

Generally, using SSH keys is more secure than passwords. However, key reuse across multiple services could enable attackers broader access in the event of a key compromise. Additionally, while not entirely security-related, administrators should be aware that FIREWHEEL’s SSH usage may also hinder auditing system logs. That is, some login events attributed to the user could have been the result of the user executing a FIREWHEEL command.

Mitigations

  1. We recommend generating a new key each time a FIREWHEEL cluster is initialized. This will eliminate reuse concerns.

  2. Use proper network segmentation techniques to separate each FIREWHEEL clusters from each other and from the remainder of the network.

  3. For auditing purposes, administrators should save FIREWHEEL’s cli.log, firewheel.log, and cli_history.log. This will enable correlation between access events and automated action taken by FIREWHEEL’s CLI.

Passwordless sudo

Currently, we recommend that FIREWHEEL users have passwordless sudo access to facilitate easy use of privileged commands. FIREWHEEL uses sudo for the following actions:

  • Restarting the minimega service. Found in start, stop hard, and mm make_bridge.

  • Removing logs created by services started by minimega (they have root permissions). Found in stop hard.

  • Setting correct group permissions on minimega’s base directory (typically /tmp/minimega). Found in stop hard.

  • Ensuring all VM Resource Handler’s have been terminated. Found in stop hard and stop.

  • Creating an experiment bridge and setting up correct routing for minimega using Open vSwitch. Found in mm make_bridge.

Risks

It is considered best practice for root access to be restricted due to the permission level. Passwordless sudo enables programs to automatically escalate privileges posing obvious risks.

Mitigations

  1. We strongly recommend that only a single user (or small set of trusted users) should have access to the FIREWHEEL Cluster. To easily facilitate this, we recommend using bare metal provisioning system like igor or MAAS to create a fresh installation for each user.

  2. We recommend only installing trusted software on your FIREWHEEL cluster.

  3. For situations where the previous suggestions cannot be implemented, administrator can provide passwordless sudo for specific commands (for example see this post).

  4. For single-node clusters, administrators can require users to always enter a password for sudo. Users who are required to enter a password should minimize the use of stop hard to improve work-flow.

minimega Permissions

minimega currently requires root permissions to run. This level of access is necessary for running system-level tasks like launching VMs and using Open vSwitch.

Risks

These permissions extend to all minimega commands. This means that each of the commands can be executed with root permissions.

Mitigations

  1. The minimega team is currently working on addressing this issue. Once this has been fixed, this will no longer be an issue.

  2. A recent pull request enables users part of the minimega group to have permissions. This prevents all users from needing root permissions. We recommend using this group and only adding those users to the minimega group which are trusted to have root permissions on the physical host.

Arbitrary Command Execution With minimega

minimega enables users to execute shell commands through the use of the shell and background commands. FIREWHEEL leverages these commands to launch the gRPC Server, miniweb, Discovery, and the VM Resource Handler's needed during an experiment.

Risks

When combining this risk with running minimega as root enables users to execute shell commands with those privileges. The shell and background minimega commands are available via the minimega CLI, the minimega Python bindings, and FIREWHEEL’s minimegaAPI, which improves ease of use of the minimega Python bindings. Therefore, any model components (or any Python code operating in the same environment as FIREWHEEL or minimega) can easily leverage this potentially dangerous functionality.

Mitigations

While the minimega team is working to address running as root, this is a fundamental property of minimega and will likely not be changed. To reduce risk, we recommend:

  1. Only allowing trusted users on the system running FIREWHEEL. That is, do not share hardware among untrusted participants.

  2. Use proper network segmentation techniques to separate each FIREWHEEL clusters from each other and from the remainder of the network.

  3. Do not load untrusted Python packages into your environment.

  4. Do not run untrusted model components within your experiment. You can potentially search through unknown model components and verify if they are calling minimega.

Using Python Pickle for Schedule Entries

FIREWHEEL uses pickle to send VM resource schedules to the VM Resource Handler. Using pickle enables generating and transferring complex ScheduleEntry objects containing binary or Python object data which is not possible using a safer alternative (like JSON).

Risks

There is inherent risk to using pickle because:

It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.

In the case of FIREWHEEL, malicious model components can add harmful ScheduleEntries and perform arbitrary code execution across the cluster.

Mitigations

  1. The FIREWHEEL developers are investigating using JSON to transfer ScheduleEntries. We will be doing a full assessment of what (if any) functionality will be lost using JSON and the potential impacts.

  2. Until a safer alternative is used, we recommend users to avoid using untrusted Model Components as they could contain malicious ScheduleEntries.

  3. As we have previously mentioned, we strongly recommend that only a single user (or small set of trusted users) should have access to the FIREWHEEL cluster. This mitigates the possibility of an untrusted user creating a malicious model component or accessing the picked data.

Executing Helpers

In FIREWHEEL users have the ability to extend the CLI using Helpers. These Helpers can be shell or Python scripts. New Helpers can easily be created which will then be executed on the physical nodes via our firewheel.cli.host_accessor.

Risks

If a malicious Helper is installed, this could pose a risk to your cluster.

Mitigations

  1. Review the list of Available CLI Helpers to identify possible security issues. Please report any issues to the FIREWHEEL development team.

  2. We recommend only installing Helpers from trusted parties.

  3. You can validate which Helpers are installed using the list command. Additionally, you can view the Helper cache (typically located in /scratch/fw-cli) and ensure only expected files are present.

Installing Model Component Repositories

Because the Model Component framework is flexible, users have the ability to share groups of Model Components via FIREWHEEL Repositories. When these repositories are installed using the repository install Helper, users have the option to run a Model Component-specific script to download/install other data (i.e. a Model Component INSTALL file). These INSTALL files are executed and can contain any arbitrary executable code. Therefore, it is critical that users fully trust the repository/Model Component developers prior to installing the repository.

Risks

If a malicious Model Component INSTALL file is executed, it has the ability to conduct arbitrary code execution.

Mitigations

  1. We recommend only installing Model Components from trusted parties.

  2. By default, users have to use the repository install -s option when installing repositories to prevent accidentally running these files.

  3. When using the repository install -s option, a prompt helps users view the INSTALL files and manually confirm that they can be installed.

  4. We do NOT recommend using the repository install --insecure option, which can bypass individual confirmation.