utilities.python

Provides a MC object representation of Python within the experiment. This method calls by VM objects such as installing newer versions of Python and adding pip packages. Additionally, this MC contains copies of several common Python packages which can be installed/used by VMs.

Model Component Dependencies:

Warning

Some required VM resource files are not available on the Internet, though they are unlikely to be used in recent experiments. If you do NOT need the following files [python-pip-1404.tgz, python-pip-1604.tgz, xenial_python_3.7.tgz, xenial_python3.7_bin.tgz, xenial_python3.7_include.tgz, and xenial_python3.7_lib.tgz] you should continue to run the MC INSTALL scripts and then manually create .utilities.python.installed. Please contact the FIREWHEEL developers if you need these files or have additional questions.

VM Resources

  • python-pip-1404.tgz – Pip (as a debian package) for Python 2 on Ubuntu 14.04.

  • python-pip-1604.tgz – Pip (as a debian package) for Python 2 on Ubuntu 16.04.

  • xenial_python3.7_bin.tgz – Python 3.7 binaries compiled for Xenial. Should be placed in /usr/local/bin/ if needed.

  • xenial_python3.7_lib.tgz – Python 3.7 libraries compiled for Xenial. Should be placed in /usr/local/lib/ if needed.

  • xenial_python_3.7.tgz – Python 3.7 debian packages for Xenial.

  • get-pip.py – A bootstrapping script that enables users to install pip, setuptools, and wheel in Python environments that don’t already have them. This will install pip version 18.1. See https://github.com/pypa/get-pip for more details.

  • install_pip_package.py – A legacy VRM that can install a tarball of Python packages on a Linux system (used by utilities.python.PythonVM.install_pip_package()).

Pip Packages

  • certifi-2019.3.9-py2.py3-none-any.whl - Certifi provides Mozilla’s carefully curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts.

  • chardet-3.0.4-py2.py3-none-any.whl - Chardet is a universal encoding detector for Python 2 and 3.

  • idna-2.8-py2.py3-none-any.whl - Internationalized Domain Names in Applications (IDNA).

  • pip-18.1-py2.py3-none-any.whl - The PyPA recommended tool for installing Python packages.

  • requests-2.21.0-py2.py3-none-any.whl - Requests is a simple, yet elegant, HTTP library.

  • setuptools-40.4.3-py2.py3-none-any.whl - Setuptools enables users to easily download, build, install, upgrade, and uninstall Python packages.

  • urllib3-1.24.2-py2.py3-none-any.whl - urllib3 is a powerful, user-friendly HTTP client for Python.

  • wheel-0.32.2-py2.py3-none-any.whl - wheel is the reference implementation of the Python wheel packaging standard, as defined in PEP 427.

  • ipython - The iPython package and all its dependencies.

Upgrading Python

To install a newer version of Python in a VM, there are two primary methods:

  1. Getting the compiled packages provided by some service (e.g. the deadsnakes PPA).

  2. Compiling Python on the same image that will be used in the experiments.

This MC supports installing debian packages for:

  • Ubuntu 16.04 (i.e. Xenial)

    • Python 3.7

This MC supports installing compiled Python for:

  • Ubuntu 16.04 (i.e. Xenial)

    • Python 3.7

VMs can use these available upgraded versions of Python by decorating their VMs with utilities.python.PythonVM and then calling the utilities.python.PythonVM.install_python() method.

Creating Compiled Python

To create a new compiled version of Python, users should launch the exact image of their desired OS, then follow the steps below:

  1. Download the Python source code.

    wget https://www.python.org/ftp/python/3.7.12/Python-3.7.12.tgz

  2. Install necessary dependencies. We recommend following either the official Python documentation or the Real Python guide.

  3. Build Python.

    tar xzvf Python-3.7.12.tgz
cd Python-3.7.12
./configure --enable-optimizations --enable-shared --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

    Note

    You may want other and/or different flags for your version of Python.

  4. Install it

    make altinstall

  5. Package up the installed version of Python

    cd /usr/local/include
tar -czvf xenial_python3.7_include.tgz python3.7m/
mv xenial_python3.7_include.tgz /home/ubuntu
cd /usr/local/bin
tar -czvf xenial_python3.7_bin.tgz *
mv xenial_python3.7_bin.tgz /home/ubuntu
cd /usr/local/lib
tar -czvf xenial_python3.7_lib.tgz python3.7 pkgconfig libpython3.so libpython3.7m.so.1.0 libpython3.7m.so
mv xenial_python3.7_lib.tgz /home/ubuntu

    Note

    These files may differ depending on your installed version or system configuration.

See also

For other guides on installing Python see:

Available Objects

class utilities.python.PythonVM(*args, **kwargs)[source]

Bases: object

This decorator enables a host of functionality to install various Python packages onto a VM.

__annotate_func__ = None
__annotations_cache__ = {}
__firstlineno__ = 9
__init__(*args, **kwargs)
__static_attributes__ = ('python_version_installed',)
_install_linux(rel_time, package_bundle_name, pip_name, python_version='python')[source]
_install_pip(python_version='python')[source]

Install a version of pip at time -1000.

Parameters:

python_version (str) – Python executable to use e.g. python for default, python3, python3.5 etc..

Raises:

NotImplementedError – If the OS is not supported by this MC (e.g. not linux)

_install_windows(rel_time, package_bundle_name, pip_name, python_version='python')[source]
add_ipython(rel_time, python_version='python3')[source]

Install the iPython package.

Parameters:

  • rel_time (int) – Relative time to run the pip installation at. Must be greater than -1000.

  • python_version (str) – Python executable to use. Default is python3.

install_from_source(rel_time, paths, python_version='python')[source]

Install a list python packages from source. It expects to find a path to the source code for the various packages. That is, if you want to install the requests package from source. Your input arguments would look similar to:

>>> self.install_from_source(-90, "/home/ubuntu/requests")

Then on the system, the folder /home/ubuntu/requests should contain the requests source code.

Parameters:

  • rel_time (int) – Relative time at which to run the installation. Must be greater than -1000.

  • paths (list) – List of folder names for the package being installed.

  • python_version (str) – python executable to use e.g. python for default, python3, etc.

install_pip_package(rel_time, package_bundle_name, pip_name, python_version='python')[source]

Install a pip package (.tgz, .whl, etc.), referenced by name. This installer uses the legacy install_pip_package.py vm_resource from firewheel 1 and expects a tarball of pip_packages to install. The install_pip_package_list installer uses the FIREWHEEL 2 schedule entry and should be preferred over this.

Parameters:

  • rel_time (int) – Relative time to run the pip installation at. Must be greater than -1000.

  • package_bundle_name (str) – Tar file containing 1 or more pip packages (.tgz, .whl, etc.)

  • pip_name (list) – List of the pip package(s) to install.

  • python_version (str) – Python executable to use e.g. python for default, python3, python3.5 etc..

install_pip_package_list(rel_time, package_names, pip_args=None, python_version='python')[source]

Install a list of pip package vm_resources. This installer uses the OS independent run_executable schedule entry and should be preferred over the install_pip_package.py VMR for portability and facilitate reuse of VMRs across model components.

Parameters:

  • rel_time (int) – Relative time to run the pip installation at. Must be greater than -1000.

  • package_names (list) – List of pip package VM resources (e.g. .tgz, .whl, etc.).

  • pip_args (list) – List of arguments that will be passed to pip when installing the given packages.

  • python_version (str) – Python executable to use e.g. python for default, python3, python3.5 etc..

install_python(python_version='3.7', compiled=False)[source]

Install a newer version of Python on a VM at time -1001.

Parameters:

  • python_version (str) – Python minor version to use e.g. 3.7 for default.

  • compiled (bool) – Whether to use the compiled version or pre-built packages.

Raises:

NotImplementedError – If the OS or Python version is not supported by this MC.