8.7. HEAT API

8.7.1. High-level API

class HeatArtifact(packets=<factory>, blocks=<factory>, block_ids=<factory>, device_ip='', device_mac='', device_oui='', station_ip='', station_mac='', station_oui='', direction='', start_time=None, end_time=None, duration=0.0, expected_blocks=0, reconstructed_artifact=b'', file_name='', file_path=None)

Data and metadata that make up an artifact extracted by HEAT.

packets: list[dict] = <dataclasses._MISSING_TYPE object>

Raw packets associated with the artifact.

blocks: list[dict] = <dataclasses._MISSING_TYPE object>

block_id, data.

Type:

List of dicts with the following keys

block_ids: set[int] = <dataclasses._MISSING_TYPE object>

Unique block IDs.

device_ip: str = ''

device_mac: str = ''

device_oui: str = ''

station_ip: str = ''

station_mac: str = ''

station_oui: str = ''

direction: str = ''

DOWNLOAD or UPLOAD.

start_time: datetime = None

end_time: datetime = None

duration: float = 0.0

expected_blocks: int = 0

blocks expected based on END_* packet.

reconstructed_artifact: bytes = b''

file_name: str = ''

file_path: Path = None

property id: str

class FTPHeatArtifact(packets=<factory>, blocks=<factory>, block_ids=<factory>, device_ip='', device_mac='', device_oui='', station_ip='', station_mac='', station_oui='', direction='', start_time=None, end_time=None, duration=0.0, expected_blocks=0, reconstructed_artifact='', file_name='', file_path=None, artifact_name='', zeek_name='')

Data and metadata that make up an artifact extracted by HEAT.

artifact_name: str = ''

reconstructed_artifact: str = ''

zeek_name: str = ''

property id: str

class TelnetHeatArtifact(packets=<factory>, blocks=<factory>, block_ids=<factory>, device_ip='', device_mac='', device_oui='', station_ip='', station_mac='', station_oui='', direction='', start_time=None, end_time=None, duration=0.0, expected_blocks=0, reconstructed_artifact=b'', file_name='', file_path=None, source_ip='', source_mac='', source_oui='', dest_ip='', dest_mac='', dest_oui='', artifact_file_name='', command='', bytestream=b'', start=0, stop=-1)

packets: list[dict] = <dataclasses._MISSING_TYPE object>

Raw packets associated with the artifact.

source_ip: str = ''

source_mac: str = ''

source_oui: str = ''

dest_ip: str = ''

dest_mac: str = ''

dest_oui: str = ''

direction: str = ''

DOWNLOAD or UPLOAD.

start_time: datetime = None

end_time: datetime = None

duration: float = 0.0

reconstructed_artifact: bytes = b''

file_name: str = ''

file_path: Path = None

artifact_file_name: str = ''

command: str = ''

bytestream: bytes = b''

start: int = 0

stop: int = -1

property id: str

class S7commHeatArtifact(packets=<factory>, blocks=<factory>, block_ids=<factory>, device_ip='', device_mac='', device_oui='', station_ip='', station_mac='', station_oui='', direction='', start_time=None, end_time=None, duration=0.0, expected_blocks=0, reconstructed_artifact='', file_name='', file_path=None, raw_data='', filename='')

raw_data: str = ''

file_path: Path = None

filename: str = ''

reconstructed_artifact: str = ''

class HeatProtocol(es_obj)

Base class defining a protocol extractor for HEAT.

Authors

• Christopher Goes

• Ryan Adams

run()

Return type:

bool

abstract get_data()

Retrieve Eratosthenes Packetbeat data from the Elasticsearch server.

Return type:

None

abstract extract_blocks()

Parses and sorts Elasticsearch data, then extracts protocol blocks.

Return type:

None

abstract assemble_artifacts()

Assembles blocks into artifacts to pass to PEAT for parsing.

Return type:

None

abstract export_artifacts()

Export artifacts to files and/or Elasticsearch.

Return type:

None

abstract parse_artifacts()

Parse constructed artifacts using PEAT.

Return type:

None

8.7.2. UMAS extractor

HEAT protocol extractor for the Schneider UMAS protocol.

UMAS packets in Elasticsearch will have the ‘type’ field set to ‘umas’.

UMAS protocol fields

• modbus.transaction_identifier

• umas.connection_id

• umas.function_code

• umas.function_name

• umas.function_description

• umas.direction

• umas.data

• umas.block_id (only if umas.function_name is UPLOAD_BLOCK or DOWNLOAD_BLOCK)

• umas.block_len (only if umas.function_name is UPLOAD_BLOCK)

• umas.max_packet_size (only if umas.function_name is INITIALIZE_DOWNLOAD)

• umas.blocks_transferred (only if umas.function_name is END_UPLOAD or END_DOWNLOAD)

Authors

• Christopher Goes

• John Jacobellis

• Ryan Adams

class UmasExtractor(es_obj)

HEAT protocol extractor for the Schneider UMAS protocol.

START_FUNCS = {'INITIALIZE_DOWNLOAD', 'INITIALIZE_UPLOAD'}

BLOCK_FUNCS = {'DOWNLOAD_BLOCK', 'UPLOAD_BLOCK'}

END_FUNCS = {'END_DOWNLOAD', 'END_UPLOAD'}

RESP_FUNCS = {'RESPONSE_ERROR', 'RESPONSE_OK'}

FUNCS = ['INITIALIZE_DOWNLOAD', 'INITIALIZE_UPLOAD', 'UPLOAD_BLOCK', 'DOWNLOAD_BLOCK', 'END_DOWNLOAD', 'END_UPLOAD', 'RESPONSE_ERROR', 'RESPONSE_OK']

get_data()

Retrieve Eratosthenes Packetbeat data from the Elasticsearch server.

Return type:

None

extract_blocks()

Parses and sorts Elasticsearch data, then extracts protocol blocks.

Return type:

None

assemble_artifacts()

Assembles blocks into artifacts to pass to PEAT for parsing.

Return type:

None

export_artifacts()

Export artifacts to files and/or Elasticsearch.

Return type:

None

parse_artifacts()

Parse constructed artifacts using PEAT.

Return type:

None

8.7.3. Telnet extractor

HEAT protocol extractor for SEL relay Telnet protocol.

Telnet packets in Elasticsearch will have the ‘type’ field set to ‘telnet’. TCP packets in Elasticsearch will have the ‘type’ field set to ‘TCP’.

Authors

• Walter Weiffenbach

class TelnetExtractor(es_obj)

HEAT protocol extractor for SEL relay Telnet protocol.

• Step 1: get packet data for each telnet stream

• Step 2: reconstruct each TCP data stream for each file

• Step 3: identify commands to indicate the start of a file transfer

• Step 4: reconstruct each artifact from those byte streams

• Step 5: parse the artifact

crctab = [0, 4129, 8258, 12387, 16516, 20645, 24774, 28903, 33032, 37161, 41290, 45419, 49548, 53677, 57806, 61935, 4657, 528, 12915, 8786, 21173, 17044, 29431, 25302, 37689, 33560, 45947, 41818, 54205, 50076, 62463, 58334, 9314, 13379, 1056, 5121, 25830, 29895, 17572, 21637, 42346, 46411, 34088, 38153, 58862, 62927, 50604, 54669, 13907, 9842, 5649, 1584, 30423, 26358, 22165, 18100, 46939, 42874, 38681, 34616, 63455, 59390, 55197, 51132, 18628, 22757, 26758, 30887, 2112, 6241, 10242, 14371, 51660, 55789, 59790, 63919, 35144, 39273, 43274, 47403, 23285, 19156, 31415, 27286, 6769, 2640, 14899, 10770, 56317, 52188, 64447, 60318, 39801, 35672, 47931, 43802, 27814, 31879, 19684, 23749, 11298, 15363, 3168, 7233, 60846, 64911, 52716, 56781, 44330, 48395, 36200, 40265, 32407, 28342, 24277, 20212, 15891, 11826, 7761, 3696, 65439, 61374, 57309, 53244, 48923, 44858, 40793, 36728, 37256, 33193, 45514, 41451, 53516, 49453, 61774, 57711, 4224, 161, 12482, 8419, 20484, 16421, 28742, 24679, 33721, 37784, 41979, 46042, 49981, 54044, 58239, 62302, 689, 4752, 8947, 13010, 16949, 21012, 25207, 29270, 46570, 42443, 38312, 34185, 62830, 58703, 54572, 50445, 13538, 9411, 5280, 1153, 29798, 25671, 21540, 17413, 42971, 47098, 34713, 38840, 59231, 63358, 50973, 55100, 9939, 14066, 1681, 5808, 26199, 30326, 17941, 22068, 55628, 51565, 63758, 59695, 39368, 35305, 47498, 43435, 22596, 18533, 30726, 26663, 6336, 2273, 14466, 10403, 52093, 56156, 60223, 64286, 35833, 39896, 43963, 48026, 19061, 23124, 27191, 31254, 2801, 6864, 10931, 14994, 64814, 60687, 56684, 52557, 48554, 44427, 40424, 36297, 31782, 27655, 23652, 19525, 15522, 11395, 7392, 3265, 61215, 65342, 53085, 57212, 44955, 49082, 36825, 40952, 28183, 32310, 20053, 24180, 11923, 16050, 3793, 7920]

get_data()

Retrieve Eratosthenes Packetbeat data from the Elasticsearch server.

Return type:

None

get_list_text(data)

Return type:

bytearray

ymodem_crc(data)

CRC calculation derived from prior work by Stephen Satchell, Satchell Evaluations and Chuck Forsberg, Omen Technology especially Forsberg’s 1988 article XMODEM/YMODEM PROTOCOL REFERENCE: A compendium of documents describing the XMODEM and YMODEM File Transfer Protocols and Satchell’s 1986 article regarding updcrc.

Effectively calculates the CRC for a data block iteratively using the macro #define updcrc(cp, crc) ( crctab[((crc >> 8) & 255)] ^ (crc << 8) ^ cp) from “rbsb.c”

Return type:

int

parseStream(stream, streamid)

Return type:

None

extract_blocks()

extract bytes into self.artifacts.

Return type:

None

assemble_artifacts()

Assembles blocks into artifacts to pass to PEAT for parsing.

Return type:

None

export_artifacts()

Export artifacts to files and/or Elasticsearch.

Return type:

None

parse_artifacts()

Parse constructed artifacts using PEAT.

Return type:

None

8.7.4. FTP extractor

HEAT protocol extractor for the FTP protocol for SEL devices.

Authors

• Walter Weiffenbach

• Christopher Goes

class FTPExtractor(es_obj)

HEAT protocol extractor for the FTP protocol for SEL devices.

get_log_dir(f)

get directory path for a pcap’s zeek logs.

Return type:

str

get_data()

Retrieve Eratosthenes Packetbeat data from the Elasticsearch server.

Return type:

None

nth_index(haystack, needle, n)

Return type:

int

make_json(file)

Return type:

list | None

extract_blocks()

Parses and sorts Elasticsearch data, then extracts protocol blocks.

Return type:

None

assemble_artifacts()

Assembles blocks into artifacts to pass to PEAT for parsing.

Return type:

None

export_artifacts()

Export artifacts to files and/or Elasticsearch.

Return type:

None

parse_artifacts()

Parse constructed artifacts using PEAT.

Return type:

None