[petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job:

[Matthew Knepley]

On Sat, Aug 6, 2011 at 5:05 AM, Dominik Szczerba <dominik at itis.ethz.ch>wrote:

> On Sat, Aug 6, 2011 at 4:12 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >  Does the PETSc example src/vec/vec/examples/tutorials/ex1.c run
> correctly on 8+ processes?
>
> yes, as per:
>
> dsz at nexo:~/pack/petsc-3.1-p8/src/vec/vec/examples/tutorials$
> ~/pack/petsc-3.1-p8/externalpackages/mpich2-1.0.8/bin/mpiexec -np 12
> ./ex1
> Vector length 20
> Vector length 20 40 60
> All other values should be near zero
> VecScale 0
> VecCopy  0
> VecAXPY 0
> VecAYPX 0
> VecSwap  0
> VecSwap  0
> VecWAXPY 0
> VecPointwiseMult 0
> VecPointwiseDivide 0
> VecMAXPY 0 0 0
>

Okay, this tells us you have a problem in your code.


> >  Are you sure the MPI shared libraries are the same on both systems?
>
> I was not precise, I have only one system consisting of two 6core
> Intels. 12 cores in total.
> I have openmpi installed alongside, but was explicitly calling mpiexec
> from petsc external packages.
>
> >   You can try the option -on_error_attach_debugger
>
> When run with np 12 It only opens 6 windows, saying:
>
> [9]PETSC ERROR: MPI error 14
>

This is the error for message truncation on receive. That usually means you
used the
wrong type in an MPI call.

Also, run with -start_in_debugger and get a stack trace when it fails.

   Matt


> [1]PETSC ERROR: MPI error 14
> [7]PETSC ERROR: MPI error 14
> [9]PETSC ERROR: PETSC: Attaching gdb to
> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on
> display localhost:11.0 on machine nexo
> [1]PETSC ERROR: PETSC: Attaching gdb to
> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on
> display localhost:11.0 on machine nexo
> [7]PETSC ERROR: PETSC: Attaching gdb to
> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on
> display localhost:11.0 on machine nexo
> [9]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in
> src/sys/utils/mpimesg.c
> [1]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in
> src/sys/utils/mpimesg.c
> [7]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in
> src/sys/utils/mpimesg.c
> [1]PETSC ERROR: PETSC: Attaching gdb to
> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on
> display localhost:11.0 on machine nexo
> [9]PETSC ERROR: PETSC: Attaching gdb to
> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on
> display localhost:11.0 on machine nexo
> [7]PETSC ERROR: PETSC: Attaching gdb to
> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on
> display localhost:11.0 on machine nexo
>
> When now starting the program in the 6 windows with its expected args
> results in:
>
> [cli_9]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in
> cmd
> [cli_9]: parse_kevals failed -1
>
> I will not be able to do proper valgrinding/puryfying before next
> week. In the meantime I will still appreciate any hints.
>
> Regards,
> Dominik
>
> --
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110806/f9753029/attachment.htm>