Zoltan User's Guide  |  Next  |  Previous

Query-Function Examples

Examples of query functions provided by a simple application are included below.  The general-interface examples include a simple implementation of ZOLTAN_GEOM_FN and ZOLTAN_OBJ_LIST_FN query functions and variants of the simple implementation that exploit local identifiers and data pointers.  Migration examples for packing and unpacking objects are also included.   Robust error checking is not included in the routines; application developers should include more explicit error checking in their query functions. All the examples use a mesh data structure consisting of nodes in the mesh.  these nodes are the objects passed to Zoltan.  A node is described by its 3D coordinates and a global ID number that is unique across all processors.   The type definitions for the mesh and node data structures used in the examples are included below.
 
/* Node data structure. */
/* A node consists of its 3D coordinates and */
/* an ID number that is unique across all processors. */
struct Node_Type { 
  double Coordinates[3]; 
  int Global_ID_Num; 
}; 

/* Mesh data structure. */ 
/* Mesh consists of an array of nodes and */
/* the number of nodes owned by the processor. */
struct Mesh_Type {
  struct Node_Type Nodes[MAX_NODES]; 
  int Number_Owned; 
};
Data types for the query-function examples.

 
! Node data structure.
! A node consists of its 3D coordinates and
! an ID number that is unique across all processors.
type Node_Type  
  real(Zoltan_DOUBLE) :: Coordinates(3) 
  integer(Zoltan_INT) :: Global_ID_Num 
end type Node_Type
 

! Mesh data structure. 
! Mesh consists of an array of nodes and
! the number of nodes owned by the processor.
type Mesh_Type 
  type(Node_Type) :: Nodes(MAX_NODES) 
  integer(Zoltan_INT) :: Number_Owned 
end type Mesh_Type
 
Data types for the Fortran query-function examples.

General Interface Query Function Examples

In the following examples, ZOLTAN_OBJ_LIST_FN and ZOLTAN_GEOM_FN query functions are implemented for an application using the mesh and node data structures described above.  The nodes are the objects passed to Zoltan.

Through a call to Zoltan_Set_Fn, the function user_return_owned_nodes is registered as the ZOLTAN_OBJ_LIST_FN query function.  It returns  global and local identifiers for each node owned by a processor.

The function user_return_coords is registered as a  ZOLTAN_GEOM_FN query function.  Given the global and local identifiers for a node, this function returns the node's coordinates.  All the examples exploit the local identifier to quickly locate nodal data.  If such an identifier is not available in an application, a search using the global identifier can be performed.

The Basic Example includes the simplest implementation of the query routines.  In the query routines, it uses global application data structures and a local numbering scheme for the local identifiers.  The User-Defined Data Pointer Example uses only local application data structures; this model is useful if the application does not have global data structures or if objects from more than one data structure are to be passed to Zoltan.  Differences between the latter example and the Basic Example are shown in red.

Basic Example

In the simplest example, the query functions access the application data through a global data structure (Mesh) representing the mesh.  In the calls to Zoltan_Set_Fn, no pointers to application data are registered with the query function (i.e., the data pointer is not used).   A node's local identifier is an integer representing the index in the Mesh.Nodes array of the node.  The local identifier is set to the index's value in user_return_owned_nodes.  It is used to access the global Mesh.Nodes array in user_return_coords.
 
/* in application's program file */ 
#include "zoltan.h"

/* Declare a global Mesh data structure. */
struct Mesh_Type Mesh;

main() 
{ 
... 
    /* Indicate that local and global IDs are one integer each. */
    Zoltan_Set_Param(zz, "NUM_GID_ENTRIES", "1");
    Zoltan_Set_Param(zz, "NUM_LID_ENTRIES", "1");

    /* Register query functions. */
    /* Do not register a data pointer with the functions; */
    /* the global Mesh data structure will be used. */
    Zoltan_Set_Fn(zz, ZOLTAN_GEOM_FN_TYPE, 
              (void (*)()) user_return_coords, NULL);  
    Zoltan_Set_Fn(zz, ZOLTAN_OBJ_LIST_FN_TYPE, 
              (void (*)()) user_return_owned_nodes, NULL);  
... 
} 

void user_return_owned_nodes(void *data,  
     int num_gid_entries, int num_lid_entries,
     ZOLTAN_ID_PTR global_ids, ZOLTAN_ID_PTR local_ids,
     int wgt_dim, float *obj_wgts,
     int *ierr) 
{ 
int i; 
    /* return global node numbers as global_ids. */ 
    /* return index into Nodes array for local_ids. */ 
    for (i = 0; i < Mesh.Number_Owned; i++){ 
        global_ids[i*num_gid_entries] = Mesh.Nodes[i].Global_ID_Num; 
        local_ids[i*num_lid_entries] = i; 
    } 
    *ierr = ZOLTAN_OK; 
} 

void user_return_coords(void *data, 
     int num_gid_entries, int num_lid_entries,
     ZOLTAN_ID_PTR global_id, ZOLTAN_ID_PTR local_id, 
     double *geom_vec, int *ierr) 
{ 
    /* use local_id to index into the Nodes array. */ 
    geom_vec[0] = Mesh.Nodes[local_id[0]].Coordinates[0]; 
    geom_vec[1] = Mesh.Nodes[local_id[0]].Coordinates[1]; 
    geom_vec[2] = Mesh.Nodes[local_id[0]].Coordinates[2]; 
    *ierr = ZOLTAN_OK; 
}
Example of general interface query functions (simplest implementation).

 
! in application's program file 

module Global_Mesh_Data
! Declare a global Mesh data structure.
   type(Mesh_Type) :: Mesh
end module

program query_example_1 
use zoltan
... 
    ! Indicate that local and global IDs are one integer each. 
    ierr = Zoltan_Set_Param(zz, "NUM_GID_ENTRIES", "1");
    ierr = Zoltan_Set_Param(zz, "NUM_LID_ENTRIES", "1");

    ! Register query functions.
    ! Do not register a data pointer with the functions;
    ! the global Mesh data structure will be used.
    ierr = Zoltan_Set_Fn(zz, ZOLTAN_GEOM_FN_TYPE, user_return_coords)  
    ierr = Zoltan_Set_Fn(zz, ZOLTAN_OBJ_LIST_FN_TYPE, user_return_owned_nodes)  
... 
end program 

subroutine user_return_owned_nodes(data, &
    num_gid_entries, num_lid_entries, &
    global_ids, local_ids, wgt_dim, obj_wgts, ierr) 
use zoltan 
use Global_Mesh_Data 
integer(Zoltan_INT) :: data(1) ! dummy declaration, do not use
integer(Zoltan_INT), intent(in) :: num_gid_entries, num_lid_entries
integer(Zoltan_INT), intent(out) :: global_ids(*), local_ids(*)
integer(Zoltan_INT), intent(in) :: wgt_dim
real(Zoltan_FLOAT), intent(out) :: obj_wgts(*)
integer(Zoltan_INT), intent(out) :: ierr
integer i 
    ! return global node numbers as global_ids.  
    ! return index into Nodes array for local_ids.  
    do i = 1, Mesh%Number_Owned 
        global_ids(1+(i-1)*num_gid_entries) = &
            Mesh%Nodes(i)%Global_ID_Num 
        local_ids(1+(i-1)*num_lid_entries) = i 
    end do 
    ierr = ZOLTAN_OK 
end subroutine 

subroutine user_return_coords(data, num_gid_entries, num_lid_entries, &
    global_id, local_id, geom_vec, ierr) 
use zoltan 
use Global_Mesh_Data 
integer(Zoltan_INT) :: data(1) ! dummy declaration, do not use
integer(Zoltan_INT), intent(in) :: num_gid_entries, num_lid_entries
integer(Zoltan_INT), intent(in) :: global_id(*), local_id(*)
real(Zoltan_DOUBLE), intent(out) :: geom_vec(*)
integer(Zoltan_INT), intent(out) :: ierr
    ! use local_id to index into the Nodes array.  
    geom_vec(1:3) = Mesh%Nodes(local_id(1))%Coordinates 
    ierr = ZOLTAN_OK 
end subroutine
Fortran example of general interface query functions (simplest implementation).

User-Defined Data Pointer Example

In this example, the address of a local mesh data structure is registered with the query functions for use by those functions.  This change eliminates the need for a global mesh data structure in the application.  The address of the local data structure is included as an argument in calls to Zoltan_Set_Fn.  This address is then used in user_return_owned_nodes and user_return_coords to provide data for these routines.  It is cast to the Mesh_Type data type and accessed with local identifiers as in the Basic Example.  Differences between this example and the Basic Example are shown in red.

This model is useful when the application does not have a global data structure that can be accessed by the query functions.  It can also be used for operations on different data structures.  For example, if an application had more than one mesh, load balancing could be performed separately on each mesh without having different query routines for each mesh.  Calls to Zoltan_Set_Fn would define which mesh should be balanced, and the query routines would access the mesh currently designated by the Zoltan_Set_Fn calls.
 

/* in application's program file */
#include "zoltan.h"

main()
{
/* declare a local mesh data structure. */
struct Mesh_Type mesh;
...
    /* Indicate that local and global IDs are one integer each. */
    Zoltan_Set_Param(zz, "NUM_GID_ENTRIES", "1");
    Zoltan_Set_Param(zz, "NUM_LID_ENTRIES", "1");

    /* Register query functions. */
    /* Register the address of mesh as the data pointer. */
    Zoltan_Set_Fn(zz, ZOLTAN_GEOM_FN_TYPE,
              (void (*)()) user_return_coords, &mesh); 
    Zoltan_Set_Fn(zz, ZOLTAN_OBJ_LIST_FN_TYPE,
              (void (*)()) user_return_owned_nodes, &mesh); 
...
}

void user_return_owned_nodes(void *data,
     int num_gid_entries, int num_lid_entries,
     ZOLTAN_ID_PTR global_ids, ZOLTAN_ID_PTR local_ids,
     int wgt_dim, float *obj_wgts,
     int *ierr) 
{
int i; 
/* cast data pointer to type Mesh_Type. */
struct Mesh_Type *ptr = (struct Mesh_Type *) data;

    /* return global node numbers as global_ids. */  
    /* return index into Nodes array for local_ids. */
    for (i = 0; i < ptr->Number_Owned; i++) {
        global_ids[i*num_gid_entries] = ptr->Nodes[i].Global_ID_Num;
        local_ids[i*num_lid_entries] = i;
    }
    *ierr = ZOLTAN_OK;
}

void user_return_coords(void *data, 
     int num_gid_entries, int num_lid_entries,
     ZOLTAN_ID_PTR global_id, ZOLTAN_ID_PTR local_id, 
     double *geom_vec, int *ierr)
{

/* cast data pointer to type Mesh_Type. */
struct Mesh_Type *ptr = (struct Mesh_Type *) data;

    /* use local_id to address the requested node. */
    geom_vec[0] = ptr->Nodes[local_id[0]].Coordinates[0];
    geom_vec[1] = ptr->Nodes[local_id[0]].Coordinates[1];
    geom_vec[2] = ptr->Nodes[local_id[0]].Coordinates[2];
    *ierr = ZOLTAN_OK;
}
Example of general interface query functions using the application-defined data pointer.

 
/* included in file zoltan_user_data.f90 */
! User defined data type as wrapper for Mesh
type Zoltan_User_Data_1
   type(Mesh_type), pointer :: ptr
end type Zoltan_User_Data_1

! in application's program file 

program query_example_3 
use zoltan
! declare a local mesh data structure and a User_Data to point to it.
type(Mesh_Type), target :: mesh
type(Zoltan_User_Data_1) data
... 
    ! Indicate that local and global IDs are one integer each. 
    ierr = Zoltan_Set_Param(zz, "NUM_GID_ENTRIES", "1");
    ierr = Zoltan_Set_Param(zz, "NUM_LID_ENTRIES", "1");

    ! Register query functions.
    ! Use the User_Data variable to pass the mesh data
    data%ptr => mesh
    ierr = Zoltan_Set_Fn(zz, ZOLTAN_GEOM_FN_TYPE, user_return_coords, data)  
    ierr = Zoltan_Set_Fn(zz, ZOLTAN_OBJ_LIST_FN_TYPE,
                     user_return_owned_nodes, data)
... 
end program 

subroutine user_return_owned_nodes(data, &
    num_gid_entries, num_lid_entries, &
    global_ids, local_ids, wgt_dim, obj_wgts, ierr) 
use zoltan 
type(Zoltan_User_Data_1) :: data
integer(Zoltan_INT), intent(in) :: num_gid_entries, num_lid_entries
integer(Zoltan_INT), intent(out) :: global_ids(*), local_ids(*)
integer(Zoltan_INT), intent(in) :: wgt_dim
real(Zoltan_FLOAT), intent(out) :: obj_wgts(*)
integer(Zoltan_INT), intent(out) :: ierr
integer i 
type(Mesh_Type), pointer :: Mesh

   ! extract the mesh from the User_Data argument
    Mesh => data%ptr

    ! return global node numbers as global_ids.  
    ! return index into Nodes array for local_ids.  
    do i = 1, Mesh%Number_Owned 
        global_ids(1+(i-1)*num_gid_entries) = &
            Mesh%Nodes(i)%Global_ID_Num 
        local_ids(1+(i-1)*num_lid_entries) = i 
    end do 
    ierr = ZOLTAN_OK 
end subroutine 

subroutine user_return_coords(data, global_id, local_id, &
     geom_vec, ierr) 
use zoltan 
type(Zoltan_User_Data_1) :: data
integer(Zoltan_INT), intent(in) :: num_gid_entries, num_lid_entries
integer(Zoltan_INT), intent(in) :: global_id(*), local_id(*)
real(Zoltan_DOUBLE), intent(out) :: geom_vec(*)
integer(Zoltan_INT), intent(out) :: ierr
type(Mesh_Type), pointer :: Mesh

   ! extract the mesh from the User_Data argument
    Mesh => data%ptr

    ! use local_id to index into the Nodes array.  
    geom_vec(1:3) = Mesh%Nodes(local_id(1))%Coordinates 
    ierr = ZOLTAN_OK 
end subroutine
Fortran example of general interface query functions using the application-defined data pointer.

Migration Examples

Packing and Unpacking Data

Simple migration query functions for the Basic Example are included below.  These functions are used by the migration tools to move nodes among the processors.  The functions user_size_node, user_pack_node, and user_unpack_node are registered through calls to Zoltan_Set_Fn.  Query function user_size_node returns the size (in bytes) of data representing a single node.  Query function user_pack_node copies a given node's data into the communication buffer buf.  Query function user_unpack_node copies a data for one node from the communication buffer buf into the Mesh.Nodes array on its new processor.

These query routines are simple because the application does not dynamically allocate memory for each node.  Such dynamic allocation would have to be accounted for in the ZOLTAN_OBJ_SIZE_FN, ZOLTAN_PACK_OBJ_FN, and ZOLTAN_UNPACK_OBJ_FN routines. 
 

main() 
{ 
... 
    /* Register migration query functions. */
    /* Do not register a data pointer with the functions; */
    /* the global Mesh data structure will be used. */
    Zoltan_Set_Fn(zz, ZOLTAN_OBJ_SIZE_FN_TYPE,
              (void (*)()) user_size_node, NULL);  
    Zoltan_Set_Fn(zz, ZOLTAN_PACK_OBJ_FN_TYPE,
              (void (*)()) user_pack_node, NULL); 
    Zoltan_Set_Fn(zz, ZOLTAN_UNPACK_OBJ_FN_TYPE,
              (void (*)()) user_unpack_node, NULL);  
... 
} 

int user_size_node(void *data,
    int num_gid_entries, int num_lid_entries,
    ZOLTAN_ID_PTR global_id, ZOLTAN_ID_PTR local_id, int *ierr)
{
/* Return the size of data associated with one node. */
/* This case is simple because all nodes have the same size. */
    *ierr = ZOLTAN_OK;
    return(sizeof(struct Node_Type));
}

void user_pack_node(void *data,
     int num_gid_entries, int num_lid_entries,
     ZOLTAN_ID_PTR global_id, ZOLTAN_ID_PTR local_id, 
     int dest_proc, int size, char *buf, int *ierr) 
{
/* Copy the specified node's data into buffer buf. */
struct Node_Type *node_buf = (struct Node_Type *) buf;

    *ierr = ZOLTAN_OK;
    node_buf->Coordinates[0] = Mesh.Nodes[local_id[0]].Coordinates[0];
    node_buf->Coordinates[1] = Mesh.Nodes[local_id[0]].Coordinates[1];
    node_buf->Coordinates[2] = Mesh.Nodes[local_id[0]].Coordinates[2];
    node_buf->Global_ID_Num = Mesh.Nodes[local_id[0]].Global_ID_Num;
}

void user_unpack_node(void *data, int num_gid_entries,
     ZOLTAN_ID_PTR global_id, int size, 
     char *buf, int *ierr)
{
/* Copy the node data in buf into the Mesh data structure. */
int i;
struct Node_Type *node_buf = (struct Node_Type *) buf;

    *ierr = ZOLTAN_OK;
    i = Mesh.Number_Owned;
    Mesh.Number_Owned = Mesh.Number_Owned + 1;
    Mesh.Nodes[i].Coordinates[0] = node_buf->Coordinates[0];
    Mesh.Nodes[i].Coordinates[1] = node_buf->Coordinates[1];
    Mesh.Nodes[i].Coordinates[2] = node_buf->Coordinates[2];
    Mesh.Nodes[i].Global_ID_Num = node_buf->Global_ID_Num;
}
Example of migration query functions for the Basic Example.
[Table of Contents  | Next:  Release Notes  |  Previous:  Migration Examples  |  Privacy and Security]