Parallel file output publisher_api implementation. This plugin generates writes each message to the configured directory tree with <adct-json></adct-json> delimiters surrounding it. The output directory is "." by default, but may be overriden with a full path defined in env("ADC_MULTIFILE_PLUGIN_DIRECTORY"). The resulting mass of files can be reduced independently later by concatenating all files in the tree or more selectively with a single call to the adc::consolidate_multifile_logs() function. More...

Inheritance diagram for adc::multifile_plugin:

[legend]

Collaboration diagram for adc::multifile_plugin:

[legend]

Public Member Functions
	multifile_plugin ()

int	publish (std::shared_ptr< builder_api > b)
	Publish the content of the builder.

int	config (const std::map< std::string, std::string > &m)
	Configure the plugin with the options given.

int	config (const std::map< std::string, std::string > &m, std::string_view env_prefix)
	Configure the plugin with the options given and the corresponding environment variables.

const std::map< const std::string, const std::string > &	get_option_defaults ()
	Look up the settable options and their defaults.

int	initialize ()
	Ready the plugin to publish following the configuration options set or defaulted.

void	finalize ()
	Stop publishing and release any resources held for managing publication.

void	pause ()
	Pause publishing until a call to resume. Duplicate calls are allowed.

void	resume ()
	Resume publishing Duplicate calls are allowed.

string_view	name () const

string_view	version () const

	~multifile_plugin ()

Static Public Member Functions
static std::vector< std::string >	consolidate_multifile_logs (const string &pattern, std::vector< std::string > &old_paths, int debug=0)

static std::string	get_multifile_log_path (string_view dir, string_view wfid)

static int	glob_sendfile_join (const char dest, const char pattern, int perm, std::vector< std::string > &merged, std::vector< std::string > &unmerged)
	use sendfile to join all files matching pattern in new dest file. No checking of input format correctness is performed. Ignores directory names.

static std::vector< size_t >	validate_multifile_log (string_view, bool, size_t &record_count)

Detailed Description

Parallel file output publisher_api implementation. This plugin generates writes each message to the configured directory tree with <adct-json></adct-json> delimiters surrounding it. The output directory is "." by default, but may be overriden with a full path defined in env("ADC_MULTIFILE_PLUGIN_DIRECTORY"). The resulting mass of files can be reduced independently later by concatenating all files in the tree or more selectively with a single call to the adc::consolidate_multifile_logs() function.

Multiple independent multifile publishers may be created; exact filenames are not user controlled, to avoid collisions. Likewise there is no append mode.

DIR/$user/[$adc_wfid.]H_$host.P$pid.T$start.$publisherptr/$application.$rank.XXXXXX

Files opened will remain opened until the publisher is finalized.

Definition at line 47 of file multifile.ipp.

Constructor & Destructor Documentation

◆ multifile_plugin()

adc::multifile_plugin::multifile_plugin ( )

inline

Definition at line 280 of file multifile.ipp.

◆ ~multifile_plugin()

adc::multifile_plugin::~multifile_plugin ( )

inline

Definition at line 417 of file multifile.ipp.

Member Function Documentation

◆ publish()

int adc::multifile_plugin::publish ( std::shared_ptr< builder_api > b )

inlinevirtual

Publish the content of the builder.

Parameters

b	converted to a single json object and published

Implements adc::publisher_api.

Definition at line 282 of file multifile.ipp.

◆ config() [1/2]

int adc::multifile_plugin::config ( const std::map< std::string, std::string > & m )

inlinevirtual

Configure the plugin with the options given.

Parameters

m	a map with keys documented in the plugin-specific header.

For plugin QQQ, Environment variables ADC_QQQ_PLUGIN_* will override the source code default for any key not defined in m. Here QQQ is the uppercase version of the plugin name.

Implements adc::publisher_api.

Definition at line 327 of file multifile.ipp.

◆ config() [2/2]

int adc::multifile_plugin::config	(	const std::map< std::string, std::string > &	m,
		std::string_view	env_prefix
	)

inlinevirtual

Configure the plugin with the options given and the corresponding environment variables.

Parameters

m	a map with keys documented in the plugin-specific header.
env_prefix	is prepended to the expected keys for the plugin and values found with getenv that match are used, overriding elements of m. Typically, env_prefix will be PPP_ADC_QQQ_PLUGIN_ if application PPP wants to override the defaults of plugin QQQ. Here QQQ is the uppercase version of the plugin name.

Implements adc::publisher_api.

Definition at line 331 of file multifile.ipp.

◆ get_option_defaults()

const std::map< const std::string, const std::string > & adc::multifile_plugin::get_option_defaults ( )

inlinevirtual

Look up the settable options and their defaults.

Some plugins without options will return an empty map.

Implements adc::publisher_api.

Definition at line 338 of file multifile.ipp.

◆ initialize()

int adc::multifile_plugin::initialize ( )

inlinevirtual

Ready the plugin to publish following the configuration options set or defaulted.

Implements adc::publisher_api.

Definition at line 342 of file multifile.ipp.

◆ finalize()

void adc::multifile_plugin::finalize ( )

inlinevirtual

Stop publishing and release any resources held for managing publication.

Implements adc::publisher_api.

Definition at line 388 of file multifile.ipp.

◆ pause()

void adc::multifile_plugin::pause ( )

inlinevirtual

Pause publishing until a call to resume. Duplicate calls are allowed.

Implements adc::publisher_api.

Definition at line 401 of file multifile.ipp.

◆ resume()

void adc::multifile_plugin::resume ( )

inlinevirtual

Resume publishing Duplicate calls are allowed.

Implements adc::publisher_api.

Definition at line 405 of file multifile.ipp.

◆ name()

string_view adc::multifile_plugin::name ( ) const

inlinevirtual

Returns: the name of the plugin

Implements adc::publisher_api.

Definition at line 409 of file multifile.ipp.

◆ version()

string_view adc::multifile_plugin::version ( ) const

inlinevirtual

Returns: the version of the plugin (should follow semantic versioning practices)

Implements adc::publisher_api.

Definition at line 413 of file multifile.ipp.

◆ consolidate_multifile_logs()

static std::vector< std::string > adc::multifile_plugin::consolidate_multifile_logs	(	const string &	pattern,
		std::vector< std::string > &	old_paths,
		int	debug = `0`
	)

inlinestatic

Definition at line 423 of file multifile.ipp.

◆ get_multifile_log_path()

static std::string adc::multifile_plugin::get_multifile_log_path	(	string_view	dir,
		string_view	wfid
	)

inlinestatic

Utility to get output directory glob pattern from the multifile_publisher.

Definition at line 482 of file multifile.ipp.

◆ glob_sendfile_join()

static int adc::multifile_plugin::glob_sendfile_join	(	const char *	dest,
		const char *	pattern,
		int	perm,
		std::vector< std::string > &	merged,
		std::vector< std::string > &	unmerged
	)

inlinestatic

use sendfile to join all files matching pattern in new dest file. No checking of input format correctness is performed. Ignores directory names.

Parameters

dest	file to write merge result.
pattern	glob pattern to match possible input files.
merged	list of files added
unmerged	list of files found but not added due to some error. If this list is not empty, the output file should be abandoned, as it may contain a partially copied file.

Returns: 0 and updated merged list or errno and file and merged vector content undefined.

Definition at line 513 of file multifile.ipp.

◆ validate_multifile_log()

static std::vector< size_t > adc::multifile_plugin::validate_multifile_log	(	string_view	,
		bool	,
		size_t &	record_count
	)

inlinestatic

Definition at line 579 of file multifile.ipp.

The documentation for this class was generated from the following file:

adc/publisher/impl/multifile.ipp

Public Member Functions

Static Public Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ multifile_plugin()

◆ ~multifile_plugin()

Member Function Documentation

◆ publish()

◆ config() [1/2]

◆ config() [2/2]

◆ get_option_defaults()

◆ initialize()

◆ finalize()

◆ pause()

◆ resume()

◆ name()

◆ version()

◆ consolidate_multifile_logs()

◆ get_multifile_log_path()

◆ glob_sendfile_join()

◆ validate_multifile_log()