Zoltan User's Guide  |  Next  |  Previous

Using the Zoltan library

This section contains information needed to use the Zoltan library with applications:
System requirements
Building the Zoltan library
Testing the Zoltan library
Reporting bugs in Zoltan
Incorporating Zoltan into Applications
Building applications that use Zoltan
Data types for global and local IDs
C++ interface
F90 interface

System Requirements

Zoltan was designed to run on parallel computers and clusters of workstations. The most common builds and installations of Zoltan use the following: Zoltan has been tested on a variety of platforms, including Linux, Mac OS X, a variety of clusters, and Sandia's ASC RedStorm machine. Builds for Windows platforms are available as part of the Trilinos CMake build system.

Building the Zoltan Library

The Zoltan library is implemented in ANSI C and can be compiled with any ANSI C compiler. In Zoltan, there are two build environments currently supported: an Autotools build environment and a CMake build environment used with the Trilinos framework. The Autotools build environment can be used to build Zoltan in a stand-alone installation; the CMake build environment must be used within Trilinos.

Using Autotools to Build Zoltan

Users should not run autotools directly in the main Zoltan directory; rather they should create a build-directory (e.g., a subdirectory of the main Zoltan directory) in which they configure and build Zoltan. Say, for example, a user creates a directory called BUILD_DIR in the Zoltan directory. Then, to configure and build zoltan, the user would
cd zoltan/BUILD_DIR
../configure {options described below}
make everything
make install
Options to the configure command allow paths to third-party libraries such as ParMETIS, PT-Scotch and PaToH to be specified. Building with MPI compilers (e.g., mpicc) is the default for Autotools builds of Zoltan; many options allow specification of particular MPI paths and compilers.

Users desiring a Fortran90 interface to Zoltan must specify the "--enable-f90interface" option.

All options can be seen with the following command issued in the zoltan/BUILD_DIR directory:

../configure --help

The following script is an example of configuration and build commands using Autotools. It specifies that Zoltan should be built with both the ParMETIS and PT-Scotch interfaces. Paths to both ParMETIS and PT-Scotch are given. The prefix option states where Zoltan should be installed; in this example, Zoltan's include files will be installed in /homes/username/zoltan/BUILD_DIR/include, and the libraries will be installed in /homes/username/zoltan/BUILD_DIR/lib. This examples assumes the script is run from /homes/username/zoltan/BUILD_DIR.

#
../configure \
--prefix=/homes/username/zoltan/BUILD_DIR \
--with-gnumake \
--with-scotch \
--with-scotch-incdir="/Net/local/proj/zoltan/arch/all/src/Scotch5" \
--with-scotch-libdir="/Net/local/proj/zoltan/arch/linux64/lib/openmpi/Scotch5" \
--with-parmetis \
--with-parmetis-incdir="/Net/local/proj/zoltan/arch/all/src/ParMETIS3" \
--with-parmetis-libdir="/Net/local/proj/zoltan/arch/linux64/lib/openmpi/ParMETIS3"
make everything
make install

The configure script also allows you to specify the data type for a global identifier. The choices are unsigned int, unsigned long, and unsigned long long. The default data type is unsigned int. If your space of global identifiers requires more than 32 bits, you can specify a 64-bit data type.

--with-id-type=uint
--with-id-type=ulong
--with-id-type=ullong

Support of 64-bit identifiers is new as of Zoltan version 3.5. At this point in time all methods except for refinement tree partitioning have been modified to work with 64-bit identifiers. Zoltan's Fortran90 interface does not yet support 64-bit identifiers.

More examples are in the directory zoltan/SampleConfigurationScripts.

After the configuration is done in the build directory, object files and executables can be removed with make clean; the same configuration will be used for subsequent builds. Configuration information is removed with make distclean.

Using CMake to Build Zoltan

Zoltan can be built as part of the Trilinos framework using the CMake build system. CMake builds will succeed only when Zoltan is in the Trilinos directory structure (as when downloaded with Trilinos). Users should not run CMake directly in the main Zoltan directory; rather they should create a build-directory (e.g., a subdirectory of the main Trilinos directory) in which they configure and build Zoltan. Say, for example, a user creates a directory called BUILD_DIR in the Trilinos directory. Then, to configure and build zoltan, the user would
cd Trilinos/BUILD_DIR
cmake \
-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_Zoltan:BOOL=ON \
{options described below} \
..
make
make install

CMake also allows you to specify the data type for a global identifier. The choices are unsigned int, unsigned long, and unsigned long long. The default data type for a global identifier is unsigned int. The options to set the global identifier data type are shown below.

-D Zoltan_ENABLE_UINT_IDS:Bool=ON
-D Zoltan_ENABLE_ULONG_IDS:Bool=ON
-D Zoltan_ENABLE_ULLONG_IDS:Bool=ON

Support of 64-bit identifiers is new as of Zoltan version 3.5. At this point in time all methods except for refinement tree partitioning have been modified to work with 64-bit identifiers. Zoltan's Fortran90 interface does not yet support 64-bit identifiers.

Serial builds are the default in Trilinos; for serial builds, Zoltan builds and links with the siMPI library provided by Pat Miller in the Zoltan distribution. More commonly, Zoltan users desire parallel builds with MPI libraries such as OpenMPI or MPICH. For such builds, users must specify CMake option

-D TPL_ENABLE_MPI:BOOL=ON
Trilinos also defaults to using a Fortran compiler, but Fortran is not required to build Zoltan; the option to disable this check is
-D Trilinos_ENABLE_Fortran:BOOL=OFF

Other options to the cmake command allow paths to third-party libraries such as ParMETIS, PT-Scotch and PaToH to be specified.

Users desiring a Fortran90 interface to Zoltan must specify the option
-D Zoltan_ENABLE_F90INTERFACE:BOOL=ON

All options can be seen with the following command issued in the Trilinos/BUILD_DIR directory:

rm CMakeCache.txt
cmake -LAH -D Trilinos_ENABLE_Zoltan:BOOL=ON ..

The following script is an example of configuration and build commands using CMake. It specifies that Zoltan should be built with both the ParMETIS and PT-Scotch interfaces. Paths to both ParMETIS and PT-Scotch are given. The prefix option states where Zoltan should be installed; in this example, Zoltan's include files will be installed in /homes/username/Trilinos/BUILD_DIR/include, and the libraries will be installed in /homes/username/Trilinos/BUILD_DIR/lib. This examples assumes the script is run from /homes/username/Trilinos/BUILD_DIR.

#
cmake \
-D CMAKE_INSTALL_PREFIX:FILEPATH="/home/username/Trilinos/BUILD_DIR" \
-D TPL_ENABLE_MPI:BOOL=ON \
-D CMAKE_C_FLAGS:STRING="-m64 -g" \
-D CMAKE_CXX_FLAGS:STRING="-m64 -g" \
-D CMAKE_Fortran_FLAGS:STRING="-m64 -g" \
-D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_Zoltan:BOOL=ON \
-D Zoltan_ENABLE_EXAMPLES:BOOL=ON \
-D Zoltan_ENABLE_TESTS:BOOL=ON \
-D Zoltan_ENABLE_ParMETIS:BOOL=ON \
-D ParMETIS_INCLUDE_DIRS:FILEPATH="/home/username/code/ParMETIS3_1" \
-D ParMETIS_LIBRARY_DIRS:FILEPATH="/home/username/code/ParMETIS3_1" \
-D Zoltan_ENABLE_Scotch:BOOL=ON \
-D Scotch_INCLUDE_DIRS:FILEPATH="/home/username/code/scotch_5.1/include" \
-D Scotch_LIBRARY_DIRS:FILEPATH="/home/username/code/scotch_5.1/lib" \
..
make
make install

More examples are in the directory zoltan/SampleCmakeScripts. More details of CMake use in Trilinos are in
Trilinos/cmake/TrilinosCMakeQuickstart.txt.

Testing the Zoltan Library

The examples directory contains simple C and C++ examples which use the Zoltan library. The makefile in this directory has three targets: These examples are built automatically when the Autotools build environment or CMake build environment is used.

The "right" answer for these tests depends on the number of processes with which you run the tests. In general, if they compile successfully, run quickly (in seconds), and produce reasonable looking output, then Zoltan is built successfully.

Reporting Bugs in Zoltan

Zoltan uses Bugzilla to collect bug reports. Please read the instructions for reporting bugs through the Zoltan Bugzilla database.

Incorporating Zoltan into Applications

Incorporating Zoltan into applications requires three basic steps: The set of query functions needed by an application depends on the particular tools (e.g., partitioning, ordering) used and on the algorithms selected within the tools. Not all query functions are needed by every application. See documentation on tools and algorithms to determine which query functions are needed.

Building Applications that use Zoltan

The C library interface is described in the include file include/zoltan.h; this file should be included in all C application source files that call Zoltan library routines.

The C++ interface to Zoltan is implemented in header files which define classes that wrap the Zoltan C library. The file include/zoltan_cpp.h defines the Zoltan class which encapsulates a load balancing data structure and the Zoltan load balancing functions which operate upon it. Include this header file instead in your C++ application. Note that C++ applications should call the C function Zoltan_Initialize before creating a Zoltan object.

Fortran applications must USE module zoltan and specify the Zoltan installation's include directory as a directory to be searched for module information files.

The C, C++ or Fortran application should then be linked with the Zoltan library (built with Fortran support in the Fortran case) by including

-lzoltan
in the linking command for the application. Communication within Zoltan is performed through MPI, so appropriate MPI libraries must be linked with the application. Third-party libraries, such as ParMETIS, PT-Scotch and PaToH, must be also be linked with the application if they were included in compilation of the Zoltan library. 

The installed files include/Makefile.export.zoltan* contain macros that can specify Zoltan paths and libraries in an application's Makefiles. Using these files, applications can be assured they are using the same build options that were used when Zoltan was built.

Data Types for Object IDs

Application query functions and application callable library functions use global and local identifiers (IDs) for objects. All objects to be used in load balancing must have unique global IDs. Zoltan stores an ID as an array of ZOLTAN_ID_TYPE. The default for ZOLTAN_ID_TYPE is unsigned int, but configuration parameters can select unsigned long or unsigned long long as the type; see above for the appropriate configuration flags for Autotools and CMake. The number of entries in these arrays can be set using the NUM_GID_ENTRIES and NUM_LID_ENTRIES parameters; by default, one ZOLTAN_ID_TYPE represents an ID. Applications may use whatever format is most convenient to store their IDs; the IDs can then be converted to and from Zoltan's ID format in the application-registered query functions.

Definitions of ZOLTAN_ID_TYPE and ZOLTAN_ID_MPI_TYPE are in include/zoltan_types.h; they can be used by an application for memory allocation, MPI communication, and as arguments to load-balancing interface functions and application-registered query functions. In the Fortran interface, IDs are passed as arrays of integers since unsigned integers are not supported in Fortran. See the description of the Fortran interface for more details.

The local IDs passed to Zoltan are not used by the library; they are provided for the convenience of the application and can contain any information desired by the application. For instance, local array indices for objects may be passed as local IDs, enabling direct access to object data in the query function routines. See the application-registered query functions for more details. The source code distribution contains an example application zdrive in which global IDs are integers and local IDs are local array indices. One may choose not to use local ids at all, in which case NUM_LID_ENTRIES may be set to zero.

Some Zoltan routines (e.g., Zoltan_LB_Partition and Zoltan_Invert_Lists) allocate arrays of type ZOLTAN_ID_PTR and return them to the application. Others (e.g., Zoltan_Order and Zoltan_DD_Find) require the application to allocate memory for IDs. Memory for IDs can be allocated as follows:

ZOLTAN_ID_PTR gids;
int num_gids, int num_gid_entries;
gids = (ZOLTAN_ID_PTR) ZOLTAN_MALLOC(num_gids * num_gid_entries * sizeof(ZOLTAN_ID_TYPE);
The system call malloc may be used instead of ZOLTAN_MALLOC.
[Table of Contents  | Next:  C++ Interface  |  Previous: Introduction  |  Privacy and Security]