[petsc-users] pctype hmpi

[Barry Smith]

On Aug 30, 2012, at 7:24 PM, George Pau <gpau at lbl.gov> wrote:

> Hi,
> 
> I have some issues using the -pctype hmpi.  I used the same setting found at 
> 
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHMPI.html
> 
> i.e. 
> -pc_type hmpi 
> -ksp_type preonly
> -hmpi_ksp_type cg 
> -hmpi_pc_type hypre 
> -hmpi_pc_hypre_type boomeramg
> 
> My command  is
> 
> mpiexec -n 1 myprogram

   Sorry the documentation doesn't make this clearer. You need to start PETSc with special options to get the "worker" processes initialized. From the manual page for PCHMPI it has

   See PetscHMPIMerge() and PetscHMPISpawn() for two ways to start up MPI for use with this preconditioner

   This will tell you want option to start PETSc up with.

    I will fix the PC so that it prints a far more useful error message.



   Barry


> 
> But, I get 
> 
> [gilbert:4041] *** An error occurred in MPI_Bcast
> [gilbert:4041] *** on communicator MPI_COMM_WORLD
> [gilbert:4041] *** MPI_ERR_COMM: invalid communicator
> [gilbert:4041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
> 
> with openmpi.  I get similar error with mpich2
> 
> Fatal error in PMPI_Bcast: Invalid communicator, error stack:
> PMPI_Bcast(1478): MPI_Bcast(buf=0x7fffb683479c, count=1, MPI_INT, root=0, comm=0x0) failed
> PMPI_Bcast(1418): Invalid communicator
> 
> I couldn't figure out what is wrong.    My petsc is  version 3.3.3 and the configuration is -with-debugging=0 --with-mpi-dir=/usr/lib/openmpi --download-hypre=1 and I am on a Ubuntu machine.
> 
> Note that with the default pc_type and ksp_type, everything is fine.  It was also tested with multiple processors.  I wondering whether there are some options that I am not specifying correctly?
> 
> -- 
> George Pau
> Earth Sciences Division
> Lawrence Berkeley National Laboratory
> One Cyclotron, MS 74-120
> Berkeley, CA 94720
> 
> (510) 486-7196
> gpau at lbl.gov
> http://esd.lbl.gov/about/staff/georgepau/
>