The CCSE Applications Suite consists of two application codes for adaptively solving time-dependent partial differential equations (a solver for a Hyperbolic system of Conservation Laws (HyperCLaw) and an incompressible flow solver (IAMR)), C++ class libraries (BoxLib,AmrLib, and others), a 2- and 3-dimensional visualization system (AmrVis), and a package for post-processing data generated by the above AMR applications (AmrDerive). Also provided is a single grid, i.e. non-adaptive, application code for solving incompressible flows (VarDen in serial, pVarDen in parallel). The most basic components for building AMR applications are contained in BoxLib and AmrLib; the most complete applications are the two solvers. The capability to solve elliptic equations on an adaptive hierarchy of grids with either cell-centered or node-centered data is contained in two of the multigrid libraries (MGLib and HGProj). Because the applications depend on some or all of the more fundamental software libraries, users must download the entire package, but need only build the executables for the application they want.
AMR uses block-structured refinement, so that the solution to an AMR calculation is composed of data in multiple non-intersecting rectangular grids at multiple levels with different resolution. The visualization and post-processing tools are specially constructed to efficiently handle this type of data set.
All components of the CCSE Applications Suite except VarDen are designed to execute in parallel using MPI. However, the user has the option of building executables without MPI to run on serial platforms.
IAMR is a an application code for solving the incompressible Navier-Stokes equations in an AMR framework. The algorithm in this code uses a fractional step approach, with upwind Godunov methodology for the advection step, Crank-Nicolson discretization of the viscous and diffusive terms, and a variable density approximate projection to impose the divergence constraint. The algorithm is more fully described in A Conservative Adaptive Projection Method for the Variable Density Incompressible Navier-Stokes equations.
HyperCLaw is an application code for solving hyperbolic systems of conservation laws. A sample application for solving the Euler equations of gas dynamics using shock-capturing in an AMR framework is provided. HyperCLaw is based on a higher-order Godunov integrator; a description of the algorithm is contained in Parallelization of structured, hierarchical adaptive mesh refinement algorithms and in the references therein.
VarDen is an application code for simulating the Variable Density incompressible Navier-Stokes equations on a single rectangular grid. It relies on C++ data structures defined in BoxLib and the visualization capability of AmrVis. The algorithm used in VarDen is the single grid version of the variable density projection method implemented in IAMR. Both IAMR and VarDen use Godunov upwinding methodology and an approximate projection. Selected results from calculations using VarDen are shown here. The code and documentation for 2-D and 3-D versions of VarDen are available.
BoxLib is a foundation library of C++ classes that facilitate the development of block-structured finite difference algorithms such as arise in the solution of systems of partial differential equations. BoxLib is particularly useful in designing and building adaptive algorithms. BoxLib contains rich data structures for describing operations which take place on data defined in regions of index space that are unions of non-intersecting rectangles. Because we are interested in unsteady problems where the regions of interest may change in response to an evolving solution, support is provided in BoxLib for efficiently building and destroying the data structures. All constructs in BoxLib are designed to run efficiently in parallel.
AmrLib is a framework library of C++ classes that extends the capabilities provided in BoxLib to support adaptive data sets, i.e., data on multiple levels at different resolution. In addition, AmrLib provides the basic facilities for constructing end-to-end AMR applications. These facilities include the memory management and processor distribution of adaptively defined data, time-step control for sub-cycled-in-time adaptive simulations, output of plot files in a form suitable for post-processing using AmrVis or AmrDerive, and checkpoint files used to restart calculations.
The CCSE Applications Suite also includes the following:
AmrVis is a visualization and data analysis tool for examining data files generated by AMR algorithms. The user can view color planar images, color volume renderings of the data, or spreadsheets of the actual data values. AmrVis can visualize single rectangular arrays of data, or it can visualize data defined on unions of rectangles upon multiple levels of refinement. With AMR algorithms, it is common to have more detail in the calculation than can be shown on a display screen. Therefore AmrVis allows the user to subregion the data and zoom in on a region of particular interest. AmrVis can animate time sequences of AMR data. Color images can be output in several formats, including postscript. AmrVis works on 2D and 3D data and requires at least an XWindow display with at least 256 colors (i.e., no specialized graphics hardware.)
The plot files generated by the applications above are specifically designed to be read by AmrVis. General purpose visualization tools do not, in general, understand AMR data formats. While AMR PDE solvers can output data interpolated to a single fine grid as required for most visualization tools, this greatly increases the data volume.
When configured for use with the Stanford volume rendering library, VolPack, AmrVis can be used to construct interactive volume renderings of 3D adaptive datasets.
AmrDerive is a data extraction and analysis tool for the type of AMR data sets created by this software. It creates plot files from other plot files, typically creating new data sets from state data. AmrDerive works for 2-D and 3-D data sets and can run in parallel on both SMP and distributed memory parallel machines using BoxLib.
AmrDerive requires the user to write code to perform specific derivation functions. The version released here contains a framework for reading and writing plot files and accessing data in parallel, and examples are provided for the user to use and modify.
We provide here a brief description of the organization of the software contained in the CCSE Applications Suite together with some discussion of how to build and execute the software. This information is by no means intended to be complete; below we provide contact information to help in obtaining additional instructions.
After unpacking, the software will be arranged in several directories
rooted at CCSEApps. The directory names within
CCSEApps are somewhat related to the names used above in
describing the contents of the CCSE Applications Suite. For example
the BoxLib library is contained in the
CCSEApps/BoxLib. The two main applications are
contained in the directories CCSEApps/iamrlib for
IAMR, and CCSEApps/HCAll for HyperCLaw.
Each of the directories contains a GNUmakefile, a
make database that controls the compilation of the files in
that directory.
Executables are built using the makefiles in the application
directories. For example, CCSEApps/iamrlib/run2d contains a
GNUmakefile that builds an adaptive solver for the
incompressible Navier-Stokes equations in two dimensions.
The GNUmakefiles in the library directories can be used
to build a Unix library archive that can be linked using the makefiles
in the application directories. This is not recommended; it is
preferred to build the CCSE applications in place using the makefiles
in the application directories.
Note that the makefiles in the CCSE Applications Suite can only be
processed by the GNU version of make. On some systems
this may be installed under the name gmake.
The directory CCSEApps/mk contains makefile fragments
that are included in the application makefiles.
The makefile fragments are responsible for declaring the rules for
building executables and object files, for declaring the command line
parameters for the C++ and Fortran compilers, and for providing the
location of the MPI parallel libraries, among other things.
As an example of building an application we build the 2-D adaptive
incompressible Navier-Stokes solver. Change to the 2-D IAMR directory,
CCSEApps/iamrlib/run2d and examine the
GNUmakefile. The makefile contains a series of variable
declarations that control the compilation. The most important of
these are:
PRECISION, defines the floating precision of the
application. This is usually DOUBLE for
double-precisionDEBUG, determines whether the application is
being compiled for debugging (TRUE) or is fully optimized
(FALSE).DIM, determines the spatial dimensionality of the
application.COMP, specifies the compilation system.
COMP=KCC and COMP=g++ are known to
work.USE_MPI, specifies the use of MPI if
TRUE.Other variables might be present, depending on which application is being built.
After executing
make -f GNUmakefile
an executable will
be produced, with a name, in this case, something like
amr2d.Linux.g++.DEBUG.MPI.ex. This unwieldy name encodes
some of the options used in the makefile allowing executables with
different options to coexist within the same directory.
The code built in the previous example is executed by providing an
inputs file to set the problem parameters. BoxLib
provides a parameter parsing facility, ParmParse that
reads the inputs file; parameters in the inputs file may be
overridden on the command-line. For example, to
execute the IAMR for 5 time steps, with MGLib
debugging turned on, one would type:
./amr2d.Linux.g++.DEBUG.ex inputs.2d.bubble max_step=5
mg.v=2
How one invokes an MPI linked, parallel executable depends on the
particular MPI implementation in use. For example, in the
MPICH version of MPI one might run the above example
using two processors by typing
mpirun -np 2 ./amr2d.Linux.g++.DEBUG.MPI.ex inputs.2d.bubble
max_step=5 mg.v=2
Depending on the application being run and the contents of the
inputs file used on the command-line, several output
files will be created. Files with names like plt0001 are
plot files; they contain data that can be viewed graphically
using AmrVis. Files with names like chk0001 are
checkpoint/restart files; they can be used to restart a job.
For example, to continue the previous example, the following
command-line might be used to advance the solution from time step 5 to
time step 10:
mpirun -np 2 ./amr2d.Linux.g++.DEBUG.MPI.ex inputs.2d.bubble
max_step=10 mg.v=2 amr.restart=chk0005
Further information about the CCSE Applications Suite can be found by contacting Charles A. Rendleman of CCSE.
If you plan to become a user of this software, and would like to be notified with any updates or bug fixes, please be sure to register by sending mail to AMR@lbl.gov You can also check this CCSE web site regularly for news, updates and fixes.