[petsc-users] pctype hmpi

[Barry Smith]

On Aug 30, 2012, at 10:02 PM, George Pau <gpau at lbl.gov> wrote:

> Hi Barry,
> 
> I tried with the addition of 
> 
> -hmpi_spawn_size 3
> 
> but I am still getting the same error though.

    The EXACT same error? Or some other error?

     What happens if you run with the -hmpi_merge_size <size> option instead?

   Barry

1) I am getting a crash with the spawn version I suspect is due to bugs in the MPICH version I am using related to spawn.

2) I am getting errors with the merge version due to Apple's ASLR which they make hard to turn off.


>  I am using mpich2.  Any other options to try?
> 
> George
> 
> 
> On Aug 30, 2012, at 7:28 PM, Barry Smith wrote:
> 
>> 
>> On Aug 30, 2012, at 7:24 PM, George Pau <gpau at lbl.gov> wrote:
>> 
>>> Hi,
>>> 
>>> I have some issues using the -pctype hmpi.  I used the same setting found at 
>>> 
>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html
>>> 
>>> i.e. 
>>> -pc_type hmpi 
>>> -ksp_type preonly
>>> -hmpi_ksp_type cg 
>>> -hmpi_pc_type hypre 
>>> -hmpi_pc_hypre_type boomeramg
>>> 
>>> My command  is
>>> 
>>> mpiexec -n 1 myprogram
>> 
>>  Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has
>> 
>>  See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner
>> 
>>  This will tell you want option to start PETSc up with.
>> 
>>   I will fix the PC so that it prints a far more useful error message.
>> 
>> 
>> 
>>  Barry
>> 
>> 
>>> 
>>> But, I get 
>>> 
>>> [gilbert:4041] *** An error occurred in MPI_Bcast
>>> [gilbert:4041] *** on communicator MPI_COMM_WORLD
>>> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator
>>> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
>>> 
>>> with openmpi.  I get similar error with mpich2
>>> 
>>> Fatal error in PMPI_Bcast: Invalid communicator, error stack:
>>> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed
>>> PMPI_Bcast(1418): Invalid communicator
>>> 
>>> I couldn't figure out what is wrong.    My petsc is  version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine.
>>> 
>>> Note that with the default pc_type and ksp_type, everything is fine.  It was also tested with multiple processors.  I wondering whether there are some options that I am not specifying correctly?
>>> 
>>> -- 
>>> George Pau
>>> Earth Sciences Division
>>> Lawrence Berkeley National Laboratory
>>> One Cyclotron, MS 74-120
>>> Berkeley, CA 94720
>>> 
>>> (510) 486-7196
>>> gpau at lbl.gov
>>> http://esd.lbl.gov/about/staff/georgepau/
>>> 
>> 
>