DEPRECATED Setting Up ELK

Error

This module has been deprecated in favor of using Jupyter notebooks. The remainder of this module is dedicated to installing ELK and ingesting VM resource logs. However, this will be removed in a future release.

In this module, you will install Elasticsearch and Kibana and learn how to ingest experiment data into them. While FIREWHEEL does not officially support/automate the installation of these tools, they will be useful for your experiment analysis.

The following script, which assumes unobstructed access to the Internet, will install Elasticsearch and Kibana on Ubuntu.

Note

Please update the ELK_VERSION to use the latest supported version found here: https://www.elastic.co/downloads/elasticsearch/!!!

Listing 7 setup_es.sh

 #!/bin/bash

 ELK_VERSION="8.13.4"

 pushd /tmp

 # Install elasticsearch
 wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-$ELK_VERSION-amd64.deb
 wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-$ELK_VERSION-amd64.deb.sha512
 shasum -a 512 -c elasticsearch-$ELK_VERSION-amd64.deb.sha512
 sudo dpkg -i elasticsearch-$ELK_VERSION-amd64.deb

 # Install kibana
 wget https://artifacts.elastic.co/downloads/kibana/kibana-$ELK_VERSION-amd64.deb
 wget https://artifacts.elastic.co/downloads/kibana/kibana-$ELK_VERSION-amd64.deb.sha512
 shasum -a 512 kibana-$ELK_VERSION-amd64.deb.sha512
 sudo dpkg -i kibana-$ELK_VERSION-amd64.deb

 # Start the system services
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
 sudo systemctl enable kibana.service

 sudo systemctl start elasticsearch
 sudo systemctl start kibana

 # Storing the HTTP Certificates and the basic authentication password
 # NOTE: This is NOT production safe code!
 sudo cp /etc/elasticsearch/certs/http_ca.crt .
 sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -a -b -u elastic -s > es_pass.txt

 popd

Note

If there are issues running the script and installing Elasticsearch or Kibana, be sure to check your network settings for any potential issues (e.g., proxies, firewalls, etc.).

Ingesting Data into Elasticsearch

Once Elasticsearch is installed, we can use the elasticsearch-py, the official Python package, to help ingest our data. First install this package into our virtual environment.

$ source /opt/firewheel/fwpy/bin/activate
$ python -m pip install elasticsearch

Then we can use the following script to ingest our data:

Listing 8 ingest_json_file.py

#!/usr/bin/env python3

import sys
import json
from elasticsearch import Elasticsearch, helpers, exceptions

def generate_data(paths):
    """
    Reads all VM Resource Logs and properly formats them to
    yield a single document. This function is passed into the bulk()
    helper to create many documents in sequence.
    """
    for fn in paths:
        with open(fn) as f:
            for line in f:
                formatted = json.loads(line.rstrip())
                timestamp = formatted['timestamp']
                formatted['timestamp'] = timestamp.replace(' ', 'T')
                yield formatted

paths = sys.argv[1:]
index = 'vm_resource_logs'

es_pass = ""
with open("/tmp/es_pass.txt") as f:
    es_pass = f.read()

es_client = Elasticsearch(
    'https://localhost:9200',
    ca_certs="/tmp/http_ca.crt",
    basic_auth=("elastic", es_pass)
)

try:
    es_resp = helpers.bulk(client=es_client, actions=generate_data(paths), index=index)
except exceptions.ConnectionError as e:
    print("ERROR: sending data to Elasticsearch, it may not be up yet")
    raise

Save the script as ingest_json_file.py. Now we can call ingest_json_file.py with our JSON logs as the argument.

$ python ingest_json_file.py /scratch/vm_resource_logs/*.json

Note

If your default shell does not expand the * you will need to modify ingest_json_file.py to ingest the desired files.

Depending on how your cluster is configured, Kibana will likely be running on http://localhost:5601. If you had to port-forward your miniweb dashboard, you will need to do the same for Kibana (e.g. ssh -Llocalhost:9001:localhost:9001 -Llocalhost:5601:localhost:5601 <node>.

Once you open Kibana, you can create a new visualization to explore the data. We defer to external tutorials/resources for information on how to use Kibana.