Zoltan User's Guide  |  Next  |  Previous

Unstructured Communication Utilities

The unstructured communication package provides a simple interface for doing complicated patterns of point-to-point communication, such as those associated with data remapping. This package consists of a few simple functions which create or modify communication plans, perform communication, and destroy communication plans upon completion. The package is descended from software first developed by Steve Plimpton and Bruce Hendrickson, and has proved useful in a variety of different applications. For this reason, it is maintained as a separate library and can be used independently from Zoltan.

In a prototypical usage of the unstructured communication package each processor has some objects to send to other processors, but no processor knows what messages it will receive. A call to Zoltan_Comm_Create produces a data structure called a communication plan which encapsulates the basic information about the communication operation. The plan does not know anything about the types of objects being transferred, only the number of them. So the same plan can be used repeatedly to transfer different types of data as long as the number of objects in the transfers remains the same. The actual size of objects isn't specified until the call to Zoltan_Comm_Do which performs the data transfer.

The plan which is produced by Zoltan_Comm_Create assumes that all the objects are of the same size. If this is not true, then a call to Zoltan_Comm_Resize can specify the actual size of each object, and the plan is augmented appropriately. Zoltan_Comm_Resize can be invoked repeatedly on the same plan to specify varying sizes for different data transfer operations.

Although a friendlier interface may be added in the future, for now the data to be sent must be passed to Zoltan_Comm_Do as a packed buffer in which the objects are stored consecutively. This probably requires the application to pull the data out of native data structures and place in into the buffer. The destination of each object is specified by the proclist argument to Zoltan_Comm_Create. Some flexibility is supported by allowing proclist to contain negative values, indicating that the corresponding objects are not to be sent. The communication operations allow for any object to be sent to any destination processor. However, if the objects are grouped in such a way that all those being sent to a particular processor are consecutive, the time and memory of an additional copy is avoided.

Function Zoltan_Comm_Do_Reverse reverses the communication plan to send back messages to the originators.

To allow overlap between communication and processing, POST and WAIT variants of Zoltan_Comm_Do and Zoltan_Comm_Do_Reverse are provided. Communication is initiated by the POST function (Zoltan_Comm_Do_Post or Zoltan_Comm_Do_Reverse_Post); incoming messages are posted and outgoing messages are sent. Then the user can continue processing. After the processing is complete, the corresponding WAIT function (Zoltan_Comm_Do_Wait or Zoltan_Comm_Do_Reverse_Wait) is called to wait for all incoming messages to be received. For convenience, these functions use the same calling arguments as Zoltan_Comm_Do and Zoltan_Comm_Do_Reverse.

All the functions in the unstructured communication library return integer error codes identical to those used by Zoltan.

The C++ interface to the unstructured communication utility is found in the zoltan_comm_cpp.h header file which defines the Zoltan_Comm class.

A Fortran90 interface is not yet available.

Source code location: Utilities/Communication
C Function prototypes file: Utilities/Communication/zoltan_comm.h
C++ class definition: Utilities/Communication/zoltan_comm_cpp.h
Library name: libzoltan_comm.a
Other libraries used by this library: libmpi.a, libzoltan_mem.a.
High Level Routines:
Zoltan_Comm_Create:  computes a communication plan for sending objects to destination processors.
Zoltan_Comm_Do:  uses a communication plan to send data objects to destination processors.  The POST and WAIT variants are
    Zoltan_Comm_Do_Post and
    Zoltan_Comm_Do_Wait.
Zoltan_Comm_Do_Reverse:  performs the reverse (opposite) communication of Zoltan_Comm_Do.  The POST and WAIT variants are
    Zoltan_Comm_Do_Reverse_Post and
    Zoltan_Comm_Do_Reverse_Wait.
Zoltan_Comm_Resize:  augments the plan to allow objects to be of variable sizes. 
Zoltan_Comm_Copy:  create a new communication plan and copy an existing one to it. 
Zoltan_Comm_Copy_To:  copy one existing communication plan to another. 
Zoltan_Comm_Destroy:  free memory associated with a communication plan. 
Low Level Routines:
Zoltan_Comm_Exchange_Sizes: updates the sizes of the messages each processor will receive.
Zoltan_Comm_Invert_Map:  given a set of messages each processor wants to send, determines the set of messages each processor needs to receive.
Zoltan_Comm_Sort_Ints:  sorts an array of integer values. 
Zoltan_Comm_Info:  returns information about a communication plan.
Zoltan_Comm_Invert_Plan:  given a communication plan, converts the plan into a plan for the reverse communication.
Use in Zoltan:
The Zoltan library uses the unstructured communication package in its migration tools and in some of the load-balancing algorithms. For example, in Zoltan_Migrate, Zoltan_Comm_Create is used to develop a communication map for sending objects to be exported to their new destination processors. The sizes of the exported objects are obtained and the communication map is augmented with a call to Zoltan_Comm_Resize. The data for the objects is packed into a communication buffer and sent to the other processors through a call to Zoltan_Comm_Do. After the received objects are unpacked, the communication plan is no longer needed, and it is deallocated by a call to Zoltan_Comm_Destroy. Zoltan developers use the package whenever possible, as improvements made to the package (for example, support for heterogeneous architectures) automatically propagate to the algorithms. 
C:
int Zoltan_Comm_Create( struct Zoltan_Comm_Obj **plan, int nsend, int *proclist, MPI_Comm comm, int tag, int *nreturn);
C++:
Zoltan_Comm( const int & nsend, int *proclist, const MPI_Comm & comm, const int & tag, int *nreturn);
   or
Zoltan_Comm();
Zoltan_Comm::Create( const int & nsend, int *proclist, const MPI_Comm & comm, const int & tag, int *nreturn);
The Zoltan_Comm_Create function sets up the communication plan in the unstructured communication package. Its input is a count of objects to be sent to other processors, a list of the processors to which the objects should be sent (repetitions are allowed), and an MPI communicator and tag. It allocates and builds a communication plan that describes to which processors data will be sent and from which processors data will be received. It also computes the amount of data to be sent to and received from each processor. It returns the number of objects to be received by the processor and a pointer to the communication plan it created. The communication plan is then used by calls to Zoltan_Comm_Do to perform the actual communication.
 
Arguments:
    plan A pointer to the communication plan created by Zoltan_Comm_Create.
    nsend The number of objects to be sent to other processors.
    proclist An array of size nsend of destination processor numbers for each of the objects to be sent.
    comm The MPI communicator for the unstructured communication.
    tag A tag for MPI communication.
    nreturn Upon return, the number of objects to be received by the processor.
Returned Value:
   int Error code.

In the C++ interface to the communication utility, the communication plan is represented by a Zoltan_Comm object. It is created when the Zoltan_Comm constructor executes. There are two constructors. The first one listed above uses parameters to initialize the plan. The second constructor does not, but the plan can subsequently be initialized with a call to Zoltan_Comm::Create().

C:
int Zoltan_Comm_Do(struct Zoltan_Comm_Obj *plan, int tag, char *send_data, int nbytes, char *recvbuf); 
int Zoltan_Comm_Do_Post(struct Zoltan_Comm_Obj *plan, int tag, char *send_data, int nbytes, char *recvbuf); 
int Zoltan_Comm_Do_Wait(struct Zoltan_Comm_Obj *plan, int tag, char *send_data, int nbytes, char *recvbuf); 
C++:
int Zoltan_Comm::Do(const int & tag, char *send_data, const int & nbytes, char *recvbuf); 
int Zoltan_Comm::Do_Post(const int & tag, char *send_data, const int & nbytes, char *recvbuf); 
int Zoltan_Comm::Do_Wait(const int & tag, char *send_data, const int & nbytes, char *recvbuf); 
The Zoltan_Comm_Do function performs the communication described in a communication plan built by Zoltan_Comm_Create. Using the plan, it takes a buffer of object data to be sent and the size (in bytes) of each object's data in that buffer and sends the data to other processors. Zoltan_Comm_Do also receives object data from other processors and stores it in a receive buffer. The receive buffer must be allocated by the code calling Zoltan_Comm_Do using the number of received objects returned by Zoltan_Comm_Create or Zoltan_Comm_Resize. If the objects have variable sizes, then Zoltan_Comm_Resize must be called before Zoltan_Comm_Do.
 
Arguments:
    plan A pointer to a communication plan built by Zoltan_Comm_Create.
    tag An MPI message tag.
    send_data A buffer filled with object data to be sent to other processors.
    nbytes The size (in bytes) of the data for one object, or the scale factor if the objects have variable sizes. (See Zoltan_Comm_Resize for more details.)
    recvbuf Upon return, a buffer filled with object data received from other processors.
Returned Value:
    int Error code.

C:
int Zoltan_Comm_Do_Reverse( struct Zoltan_Comm_Obj *plan, int tag, char *send_data, int nbytes, int *sizes, char *recvbuf); 
int Zoltan_Comm_Do_Reverse_Post( struct Zoltan_Comm_Obj *plan, int tag, char *send_data, int nbytes, int *sizes, char *recvbuf); 
int Zoltan_Comm_Do_Reverse_Wait( struct Zoltan_Comm_Obj *plan, int tag, char *send_data, int nbytes, int *sizes, char *recvbuf); 
C++:
int Zoltan_Comm::Do_Reverse( const int & tag, char *send_data, const int & nbytes, int *sizes, char *recvbuf); 
int Zoltan_Comm::Do_Reverse_Post( const int & tag, char *send_data, const int & nbytes, int *sizes, char *recvbuf); 
int Zoltan_Comm::Do_Reverse_Wait( const int & tag, char *send_data, const int & nbytes, int *sizes, char *recvbuf); 
The Zoltan_Comm_Do_Reverse function performs communication based on a communication plan built by Zoltan_Comm_Create. But unlike Zoltan_Comm_Do, this routine performs the reverse of the communication pattern. Specifically, all sends in the plan are treated as receives and vice versa. Zoltan_Comm_Do_Reverse is particularly well suited to return updated data objects to their originating processors when the objects were initially transferred via Zoltan_Comm_Do.
 
Arguments:
    plan A pointer to a communication plan built by Zoltan_Comm_Create.
    tag An MPI message tag to be used by this routine.
    send_data A buffer filled with object data to be sent to other processors.
    nbytes The size (in bytes) of the data associated with an object, or the scale factor if the objects have variable sizes.
    sizes If not NULL, this input array specifies the size of all the data objects being transferred. This argument is passed directly to Zoltan_Comm_Resize. This array has length equal to the nsend value passed to Zoltan_Comm_Create. But note that for Zoltan_Comm_Do_Reverse this array describes the sizes of the values being received, not sent.
    recvbuf Upon return, a buffer filled with object data received from other processors.
Returned Value:
    int Error code.

C:
int Zoltan_Comm_Resize( struct Zoltan_Comm_Obj *plan, int *sizes, int tag , int *total_recv_size); 
C++:
int Zoltan_Comm::Resize( int *sizes, const int & tag , int *total_recv_size); 
If the objects being communicated are of variable sizes, then the plan produced by Zoltan_Comm_Create is incomplete. This routine allows the plan to be augmented to allow for variable sizes. Zoltan_Comm_Resize can be invoked repeatedly on the same plan to specify different object sizes associated with different data transfers.
 
Arguments:
    plan A communication plan built by Zoltan_Comm_Create.
    sizes An input array of length equal to the nsend argument in the call to Zoltan_Comm_Create which generated the plan. Each entry in the array is the size of the corresponding object to be sent. If sizes is NULL (on all processors), the objects are considered to be the same size. Note that the true size of a message will be scaled by the nbytes argument to Zoltan_Comm_Do. 
    tag A message tag to be used for communication within this routine, based upon the communicator in plan.
    total_recv_size Sum of the sizes of the incoming messages. To get the actual size (in bytes), you need to scale by the nbytes argument to Zoltan_Comm_Do. 
Returned Value:
    int Error code.

C:  struct Zoltan_Comm_Obj *Zoltan_Comm_Copy( struct Zoltan_Comm_Obj *plan); 
C++:  Zoltan_Comm(const Zoltan_Comm &plan);
Zoltan_Comm_Copy creates a new Zoltan_Comm_Obj structure and copies the existing plan to it. The corresponding C++ method is the Zoltan_Comm copy constructor.
 
Arguments:
    plan A pointer to the communication plan to be copied to the new Zoltan_Comm_Obj structure.  
Returned Value:
    struct Zoltan_Comm_Obj * the newly created plan, or NULL on error.

C:  int Zoltan_Comm_Copy_To( struct Zoltan_Comm_Obj **to,  struct Zoltan_Comm_Obj *from); 
C++:  Zoltan_Comm &  operator= (const Zoltan_Comm &plan);
Zoltan_Comm_Copy_To copies one existing communication plan to another. The corresponding C++ method is the Zoltan_Comm copy operator.
 
Arguments:
    to A pointer to a pointer to the communication plan that will be copied to. We destroy the plan first, and set the pointer to the plan to NULL, before proceeding with the copy.  
    from A pointer the communication plan that we will make a copy of.  
Returned Value:
    int Error code

C:  int Zoltan_Comm_Destroy( struct Zoltan_Comm_Obj **plan); 
C++:  ~Zoltan_Comm();
The Zoltan_Comm_Destroy function frees all memory associated with a communication plan created by Zoltan_Comm_Create. The C++ Zoltan_Comm object does not have an explicit Destroy method. It is deallocated when its destructor is called.
 
Arguments:
    plan A pointer to a communication plan built by Zoltan_Comm_Create.  Upon return, plan is set to NULL. 
Returned Value:
    int Error code.

C:
int Zoltan_Comm_Exchange_Sizes( int *sizes_to, int *procs_to, int nsends, int self_msg, int *sizes_from, int *procs_from, int nrecvs, int *total_recv_size, int my_proc, int tag, MPI_Comm comm ); 
C++:
static int Zoltan_Comm::Exchange_Sizes( int *sizes_to, int *procs_to, const int & nsends, const int & self_msg, int *sizes_from, int *procs_from, const int & nrecvs, int *total_recv_size, const int & my_proc, const int & tag, const MPI_Comm & comm ); 
This routine is used by Zoltan_Comm_Resize to update the sizes of the messages each processor is expecting to receive. The processors already know who will send them messages, but if variable sized objects are being communicated, then the sizes of the messages are recomputed and exchanged via this routine.
 
Arguments:
    sizes_to Input array with the size of each message to be sent. Note that the actual number of bytes in the message is the product of this value and the nbytes argument to Zoltan_Comm_Do.
    procs_to Input array with the destination processor for each of the messages to be sent.
    nsends Input argument with the number of messages to be sent. (Length of the procs_to array.)
    self_msg Input argument indicating whether a processor has data for itself (=1) or not (=0) within the procs_to and lengths_to arrays.
    sizes_from Returned array with the size of each message that will be received. Note that the actual number of bytes in the message is the product of this value and the nbytes argument to Zoltan_Comm_Do.
    procs_from Returned array of processors from which data will be received.
    nrecvs Returned value with number of messages to be received. (length of procs_from array.)
    total_recv_size The total size of all the messages to be received. As above, the actual number of bytes will be scaled by the nbytes argument to Zoltan_Comm_Do.
    my_proc The processor's ID in the comm communicator. 
    tag A message tag which can be used by this routine.
    comm MPI Communicator for the processor numbering in the procs arrays.
Returned Value:
    int Error code.

C:
int Zoltan_Comm_Invert_Map( int *lengths_to, int * procs_to, int nsends, int self_msg, int ** lengths_from, int ** procs_from, int * nrecvs, int my_proc, int nprocs, int out_of_mem, int tag, MPI_Comm comm ); 
C++:
static int Zoltan_Comm::Invert_Map( int *lengths_to, int * procs_to, const int & nsends, const int & self_msg, int * & lengths_from, int * & procs_from, int & nrecvs, const int & my_proc, const int & nprocs, const int & out_of_mem, const int & tag, const MPI_Comm & comm ); 
The Zoltan_Comm_Invert_Map function is a low level communication routine. It is useful when a processor knows to whom it needs to send data, but not from whom it needs to receive data. Each processor provides to this routine a set of lengths and destinations for the messages it wants to send. The routine then returns the set of lengths and origins for the messages a processor will receive. Note that by inverting the interpretation of to and from in these arguments, the routine can be used to do the opposite: knowing how much data to receive and from which processors, it can compute how much data to send and to which processors.
 
Arguments:
    lengths_to Input array with the number of values in each of the messages to be sent. Note that the actual size of each value is not specified until the Zoltan_Comm_Do routine is invoked.
    procs_to Input array with the destination processor for each of the messages to be sent.
    nsends Input argument with the number of messages to be sent. (Length of the lengths_to and procs_to arrays.)
    self_msg Input argument indicating whether a processor has data for itself (=1) or not (=0) within the procs_to and lengths_to arrays.
    lengths_from Returned array with lengths of messages to be received.
    procs_from Returned array of processors from which data will be received.
    nrecvs Returned value with number of messages to be received (lengths of lengths_from and procs_from arrays).
    my_proc The processor's ID in the comm communicator. 
    nprocs Number of processors in the comm communicator. 
    out_of_mem Since it has a barrier operation, this routine is a convenient time to tell all the processors that one of them is out of memory. This input argument is 0 if the processor is OK, and 1 if the processor has failed in a malloc call. All the processors will return with a code of COMM_MEMERR if any of them is out of memory.
    tag A message tag which can be used by this routine.
    comm MPI Communicator for the processor numbering in the procs arrays.
Returned Value:
    int Error code.

int Zoltan_Comm_Sort_Ints( int *vals_sort, int *vals_other, int nvals); 
As its name suggests, the Zoltan_Comm_Sort_Ints function sorts a set of integers via the quicksort algorithm. The integers are reordered from lowest to highest, and a second array of integers is reordered in the same fashion. This second array can be used to return the permutation associated with the sort operation. There is no C++ interface to this function. You can use the C function instead.
 
Arguments:
    vals_sort The array of integers to be sorted. This array is permuted into sorted order. 
    vals_other Another array of integers which is permuted identically to vals_sort. 
    nvals The number of values in the two integer arrays. 
Returned Value:
    int Error code.
C:
int Zoltan_Comm_Info( struct Zoltan_Comm_Obj *plan, int *nsends, int *send_procs, int *send_lengths, int *send_nvals, int *send_max_size, int *send_list, int *nrecvs, int *recv_procs, int *recv_lengths, int *recv_nvals, int *recv_total_size, int *recv_list, int *self_msg);
C++:
int Zoltan_Comm::Info( int *nsends, int *send_procs, int *send_lengths, int *send_nvals, int *send_max_size, int *send_list, int *nrecvs, int *recv_procs, int *recv_lengths, int *recv_nvals, int *recv_total_size, int *recv_list, int *self_msg) const;
Zoltan_Comm_Info returns information about a communication plan. All arguments, except the plan itself, may be NULL; values are returned only for non-NULL arguments.
 
Arguments:
    plan Communication data structure created by Zoltan_Comm_Create.
    nsends Upon return, the number of processors to which messages are sent; does not include self-messages.
    send_procs Upon return, a list of processors to which messages are sent; self-messages are included.
    send_lengths Upon return, the number of values to be sent to each processor in send_procs.
    send_nvals Upon return, the total number of values to send.
    send_max_size Upon return, the maximum size of a message to be sent; does not include self-messages.
    send_list Upon return, the processor assignment of each value to be sent.
    nrecvs Upon return, the number of processors from which to receive messages; does not include self-messages.
    recv_procs Upon return, a list of processors from which messages are received; includes self-messages.
    recv_lengths Upon return, the number of values to be received from each processor in recv_procs.
    recv_nvals Upon return, the total number of values to receive.
    recv_total_size Upon return, the total size of items to be received.
    recv_list Upon return, the processor assignments of each value to be received.
    self_msg Upon return, the number of self-messages.
Returned Value:
    int Error code.
C:  int Zoltan_Comm_Invert_Plan( struct Zoltan_Comm_Obj **plan );
C++:  int Zoltan_Comm::Invert_Plan();
Given a communication plan, Zoltan_Comm_Invert_Plan alters the plan to make it the plan for the reverse communication. Information in the input plan is replaced by information for the reverse-communication plan. All receives in the reverse-communication plan are blocked; thus, using the inverted plan does not produce the same results as Zoltan_Comm_Do_Reverse. If an error occurs within Zoltan_Comm_Invert_Plan, the original plan is returned unaltered.
 
Arguments:
    plan Communication data structure created by Zoltan_Comm_Create; the contents of this plan are irretrievably modified by Zoltan_Comm_Invert_Plan.
Returned Value:
    int Error code.

[Table of Contents  |  Next:  Distributed Directory Utility  |  Previous:  Memory Management Utilities  |  Privacy and Security]