[petsc-users] error: Petsc has generated inconsistent data, MPI_Allreduce() called in different locations (code lines) on different processors

Sun Apr 7 08:48:24 CDT 2019

You have MatSetOption inside of a loop for the number of rows. This is
apparently a collective operation. Move it to line 156.

On Sun, Apr 7, 2019 at 5:44 AM Eda Oktay via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear Barry,
>
> Thank you for answering. I am sending my code, makefile and 2 PETSc Binary
> File where airfoil1_binary has odd numbered size (the one I got error) and
> gr_30_30_binary has even numbered size (I got correct result). I got the
> error for lines 164 and 169.
>
> By the way, I think I have done something wrong at line 320 (I tried to
> allocate an array twice) but I just need to reduce the error in 164 and 169
> then I will get to reduce the error in line 320.
>
> Thanks,
>
> Eda
>
>
> Smith, Barry F. <bsmith at mcs.anl.gov>, 6 Nis 2019 Cmt, 09:29 tarihinde
> şunu yazdı:
>
>>
>>   Eda,
>>
>>    Can you send us your code (and any needed data files)? We certainly
>> expect PETSc to perform correctly if the size of the matrix cannot be
>> divide by the number of processors. It is possible the problem is due to
>> bugs either in MatStashScatterBegin_BTS() or in your code.
>>
>>    Thanks
>>
>>     Barry
>>
>>
>> > On Apr 5, 2019, at 7:20 AM, Matthew Knepley via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> >
>> > On Fri, Apr 5, 2019 at 3:20 AM Eda Oktay via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> > Hello,
>> >
>> > I am trying to calculate unweighted Laplacian of a matrix by using 2
>> cores. If the size of matrix is in even number then my program works.
>> However, when I try to use a matrix having odd number for size, I guess
>> since size of the matrix cannot be divided into processors correctly, I get
>> the following error:
>> >
>> > [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > [1]PETSC ERROR: Petsc has generated inconsistent data
>> > [1]PETSC ERROR: MPI_Allreduce() called in different locations (code
>> lines) on different processors
>> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> > [1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> > [0]PETSC ERROR: Petsc has generated inconsistent data
>> > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code
>> lines) on different processors
>> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
>> 09:50:54 2019
>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> > [1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
>> 09:50:54 2019
>> > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> > [1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > #1 MatStashScatterBegin_BTS() line 843 in
>> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
>> > [1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > #2 MatStashScatterBegin_BTS() line 843 in
>> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
>> > [1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in
>> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
>> > [0]PETSC ERROR: #3 main() line 164 in
>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>> > [1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in
>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>> > [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
>> > [0]PETSC ERROR: -f
>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>> > #5 MatAssemblyBegin() line 5251 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > [1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
>> > [0]PETSC ERROR: -unweighted
>> > #6 main() line 169 in
>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>> > [1]PETSC ERROR: PETSc Option Table entries:
>> > [1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message
>> -------send entire error message to petsc-maint at mcs.anl.gov----------
>> > -f
>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>> > [1]PETSC ERROR: -mat_partitioning_type parmetis
>> > [1]PETSC ERROR: -unweighted
>> > [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 1) -
>> process 0
>> > ----------------End of Error Message -------send entire error message
>> to petsc-maint at mcs.anl.gov----------
>> > application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
>> >
>> >
>> >  where line 164 in my main program is MatSetOption and line 169 is
>> MatAssemblyBegin. I am new to MPI usage so I did not understand why
>> MPI_Allreduce() causes a problem and how can I fix the problem.
>> >
>> > You have to call collective methods on all processes in the same order.
>> This is not happening in your code. Beyond that,
>> > there is no way for us to tell how this happened.
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> > Thank you,
>> >
>> > Eda
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190407/686c5952/attachment.html>