[petsc-users] Fatal error in MPI_Allreduce: Error message texts are not available[cli_9]: aborting job:

[Dominik Szczerba]

>> [9]PETSC ERROR: MPI error 14
>
> This is the error for message truncation on receive. That usually means you
> used the
> wrong type in an MPI call.

Thanks, that's already something to start with.

> Also, run with -start_in_debugger and get a stack trace when it fails.
>    Matt

I get:

[0]PETSC ERROR: PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12238 on
display localhost:11.0 on machine nexo
[1]PETSC ERROR: PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12239 on
display localhost:11.0 on machine nexo
[2]PETSC ERROR: PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12240 on
display localhost:11.0 on machine nexo
[3]PETSC ERROR: PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12241 on
display localhost:11.0 on machine nexo
[4]PETSC ERROR: [5]PETSC ERROR: PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12243 on
display localhost:11.0 on machine nexo
PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12242 on
display localhost:11.0 on machine nexo
[6]PETSC ERROR: PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12244 on
display localhost:11.0 on machine nexo
[7]PETSC ERROR: [8]PETSC ERROR: PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12246 on
display localhost:11.0 on machine nexo
PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12245 on
display localhost:11.0 on machine nexo
[11]PETSC ERROR: [10]PETSC ERROR: PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12249 on
display localhost:11.0 on machine nexo
PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12248 on
display localhost:11.0 on machine nexo
[9]PETSC ERROR: PETSC: Attaching gdb to
/home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 12247 on
display localhost:11.0 on machine nexo

I am indeed remotely logged in, but can open terminal windows etc.
Also -on_error_attach_debugger opens the gdb consoles properly.

Thanks,
Dominik

>
>>
>> [1]PETSC ERROR: MPI error 14
>> [7]PETSC ERROR: MPI error 14
>> [9]PETSC ERROR: PETSC: Attaching gdb to
>> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on
>> display localhost:11.0 on machine nexo
>> [1]PETSC ERROR: PETSC: Attaching gdb to
>> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on
>> display localhost:11.0 on machine nexo
>> [7]PETSC ERROR: PETSC: Attaching gdb to
>> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on
>> display localhost:11.0 on machine nexo
>> [9]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in
>> src/sys/utils/mpimesg.c
>> [1]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in
>> src/sys/utils/mpimesg.c
>> [7]PETSC ERROR: PetscGatherNumberOfMessages() line 62 in
>> src/sys/utils/mpimesg.c
>> [1]PETSC ERROR: PETSC: Attaching gdb to
>> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11790 on
>> display localhost:11.0 on machine nexo
>> [9]PETSC ERROR: PETSC: Attaching gdb to
>> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11798 on
>> display localhost:11.0 on machine nexo
>> [7]PETSC ERROR: PETSC: Attaching gdb to
>> /home/dsz/build/framework-debug/trunk/bin/sm3t4mpi of pid 11796 on
>> display localhost:11.0 on machine nexo
>>
>> When now starting the program in the 6 windows with its expected args
>> results in:
>>
>> [cli_9]: PMIU_parse_keyvals: unexpected key delimiter at character 54 in
>> cmd
>> [cli_9]: parse_kevals failed -1
>>
>> I will not be able to do proper valgrinding/puryfying before next
>> week. In the meantime I will still appreciate any hints.
>>
>> Regards,
>> Dominik
>>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener
>