Code structuring - Communicator

[Amit.Itagi at seagate.com]

Barry,

I got a part of what I was trying to do (sub-communicator etc.), working.
Now suppose I want to repeat a calculation with a different input, I have
two ways of doing it (based on what I have coded).

1)

MPI_Initialize
Create a group using MPI_Comm_group
Create several sub-groups and sub-communicators using MPI_Group_Incl and
MPI_Comm_create
Assign the sub-communicator to PETSC_COMM_WORLD
// Calculation 1
{
Do PetscInitialize
Perform the calculation
Do PetscFinalize
}
// Calculation 2
{
Do PetscInitialize
Perform the calculation
Do PetscFinalize
}
Do MPI_finalize

2)

MPI_Initialize
Create a group using MPI_Comm_group
Create several sub-groups and sub-communicators using MPI_Group_Incl and
MPI_Comm_create
Assign the sub-communicator to PETSC_COMM_WORLD
Do PetscInitialize
// Calculation 1
{
Perform the calculation
}
// Calculation 2
{
Perform the calculation
}
Do PetscFinalize
Do MPI_finalize


The first method crashes. I am trying to understand why.  The documentation
says that PetscFinalize calls MPI_finalize only if MPI_Initialize is not
called before PetscInitialize. In my case, is PetscFinalize destroying the
sub-communicators ?

Thanks

Rgds,
Amit




                                                                           
             Barry Smith                                                   
             <bsmith at mcs.anl.g                                             
             ov>                                                        To 
             Sent by:                  petsc-users at mcs.anl.gov             
             owner-petsc-users                                          cc 
             @mcs.anl.gov                                                  
             No Phone Info                                         Subject 
             Available                 Re: Code structuring - Communicator 
                                                                           
                                                                           
             05/09/2008 03:07                                              
             PM                                                            
                                                                           
                                                                           
             Please respond to                                             
             petsc-users at mcs.a                                             
                  nl.gov                                                   
                                                                           
                                                                           





    There are many ways to do this, most of them involve using MPI to
construct subcommunicators
for the various sub parallel tasks. You very likely want to keep
PetscInitialize() at
the very beginning of the program; you would not write the calls in
terms of
PETSC_COMM_WORLD or MPI_COMM_WORLD, rather you would use the
subcommunicators to create the objects.

    An alternative approach is to look at the manual page for
PetscOpenMPMerge(), PetscOpenMPRun(),
PetscOpenMPNew() in petsc-dev. These allow a simple master-worker
model of parallelism
with PETSc with a bunch of masters that can work together (instead of
just one master) and each
master controls a bunch of workers. The code in src/ksp/pc/impls/
openmp uses this code.

Note that OpenMP has NOTHING to do with OpenMP the standard. Also I
don't really have
any support for Fortran, I hope you use C/C++. Comments welcome. It
sounds like this matches
what you need. It's pretty cool,  but underdeveloped.

    Barry



On May 9, 2008, at 12:46 PM, Amit.Itagi at seagate.com wrote:

>
> Hi,
>
> I have a question about the Petsc communicator. I have a petsc program
> "foo" which essentially runs in parallel and gives me
> y=f(x1,x2,...), where
> y is an output parameter and xi's are input parameters. Suppose, I
> want to
> run a parallel optimizer for the input parameters. I am looking at the
> following functionality. I submit the optimizer job on 16 processors
> (using
> "mpiexec -np 16 progName"). The optimizer should then submit 4 runs of
> "foo", each running parallely on 4 processors. "foo" will be written
> as a
> function and not as a main program in this case. How can I get this
> functionality using Petsc ? Should PetscInitialize be called in the
> optimizer, or in each foo run ? If PetscInitialize is called in the
> optimizer, is there a way to make the foo function run only on a
> subset of
> the 16 processors ?
>
> May be, I haven't done a good job of explaining my problem. Let me
> know if
> you need any clarifications.
>
> Thanks
>
> Rgds,
> Amit
>