[petsc-users] Building MatIS with dense local matrix ?
Stefano Zampini
stefano.zampini at gmail.com
Tue Jun 20 10:08:29 CDT 2017
It should be fixed right now, both in master and in maint.
Again, sorry for this ping-pong of fixes, my brain it's not fully
functional these days....
https://bitbucket.org/petsc/petsc/commits/c6f20c4fa7817632f09219574920bd3bd922f6f1
2017-06-20 16:30 GMT+02:00 Franck Houssen <franck.houssen at inria.fr>:
> OK. I moved from petsc-3.7.6 to development version (git clone bitbucket).
> The very first version of the dummy example (=
> matISCSRDenseSquare/Rect.cpp) works with this fix:
> https://bitbucket.org/petsc/petsc/commits/4c8dd594d1988a0cbe282f8a37d991
> 6f61e0c445.
> The second version of the dummy example works too with the fix if one
> moves to petsc bitbucket (master).
>
> But, the code still breaks in "my" initial "real" case (using now master
> from petsc bitbucket)... With another error "SEGV under
> MatISSetMPIXAIJPreallocation_Private" (note: this is not a "new non zero"
> message, this seems to be another problem).
>
> Here is a third version of the dummy example that breaks with "SEGV under
> MatISSetMPIXAIJPreallocation_Private" (using master from petsc bitbucket)
> : the idea is the same but with N procs (not only 2) and a rectangular
> matrix of size N*(N+1).
> With 2 procs, it works (all cases).
> With 4 procs, new problems occur:
> >> mpirun -n 4 ./matISCSRDenseSquare.exe csr; mpirun -n 4
> ./matISCSRDenseSquare.exe dense => OK
> >> mpirun -n 4 ./matISCSRDenseRect.exe csr => OK
> but
> >> mpirun -n 4 ./matISCSRDenseRect.exe dense;
> dense
> dense
> dense
> dense
> [3]PETSC ERROR: ------------------------------
> ------------------------------------------
> [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [3]PETSC ERROR: likely location of problem given in stack below
> [3]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [3]PETSC ERROR: INSTEAD the line number of the start of the function
> [3]PETSC ERROR: is given.
> [3]PETSC ERROR: [3] MatISSetMPIXAIJPreallocation_Private line 1055
> /home/fghoussen/Documents/INRIA/petsc/src/mat/impls/is/matis.c
> [3]PETSC ERROR: [3] MatISGetMPIXAIJ_IS line 1230 /home/fghoussen/Documents/
> INRIA/petsc/src/mat/impls/is/matis.c
> [3]PETSC ERROR: [3] MatISGetMPIXAIJ line 1384 /home/fghoussen/Documents/
> INRIA/petsc/src/mat/impls/is/matis.c
> [3]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
>
> I tried to go through this with caution... But I have to say I feel messy.
> Can you reproduce this problem at your side ?
>
> Franck
>
> >> git diff .
> --- a/matISCSRDenseRect.cpp
> +++ b/matISCSRDenseRect.cpp
> @@ -14,19 +14,17 @@ int main(int argc,char **argv) {
> if (matType != "csr" && matType != "dense") {cout << "error: need arg =
> csr or dense" << endl; return 1;}
>
> PetscInitialize(&argc, &argv, NULL, NULL);
> - int size = 0; MPI_Comm_size(MPI_COMM_WORLD, &size); if (size != 2)
> {cout << "error: mpi != 2" << endl; return 1;}
> + int size = 0; MPI_Comm_size(MPI_COMM_WORLD, &size);
> int rank = 0; MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>
> - PetscInt localIdx[2] = {0, 0};
> - if (rank == 0) {localIdx[0] = 0; localIdx[1] = 1;}
> - else {localIdx[0] = 1; localIdx[1] = 2;}
> + PetscInt localIdx[2] = {rank, rank+1};
> ISLocalToGlobalMapping rmap;
> ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 1, 1, &rank,
> PETSC_COPY_VALUES, &rmap);
> ISLocalToGlobalMapping cmap;
> ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 1, 2, localIdx,
> PETSC_COPY_VALUES, &cmap);
>
> Mat A;
> - MatCreateIS(PETSC_COMM_WORLD, 1, PETSC_DECIDE, PETSC_DECIDE, 2, 3,
> rmap, cmap, &A);
> + MatCreateIS(PETSC_COMM_WORLD, 1, PETSC_DECIDE, PETSC_DECIDE, size,
> size+1, rmap, cmap, &A);
>
> Mat Aloc;
> if (matType == "csr") {cout << matType << endl;
> MatCreateSeqAIJ(PETSC_COMM_SELF, 1, 2, 2, NULL, &Aloc);}
>
> --- a/matISCSRDenseSquare.cpp
> +++ b/matISCSRDenseSquare.cpp
> @@ -14,19 +14,17 @@ int main(int argc,char **argv) {
> if (matType != "csr" && matType != "dense") {cout << "error: need arg =
> csr or dense" << endl; return 1;}
>
> PetscInitialize(&argc, &argv, NULL, NULL);
> - int size = 0; MPI_Comm_size(MPI_COMM_WORLD, &size); if (size != 2)
> {cout << "error: mpi != 2" << endl; return 1;}
> + int size = 0; MPI_Comm_size(MPI_COMM_WORLD, &size);
> int rank = 0; MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>
> - PetscInt localIdx[2] = {0, 0};
> - if (rank == 0) {localIdx[0] = 0; localIdx[1] = 1;}
> - else {localIdx[0] = 1; localIdx[1] = 2;}
> + PetscInt localIdx[2] = {rank, rank+1};
> ISLocalToGlobalMapping rmap;
> ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 1, 2, localIdx,
> PETSC_COPY_VALUES, &rmap);
> ISLocalToGlobalMapping cmap;
> ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 1, 2, localIdx,
> PETSC_COPY_VALUES, &cmap);
>
> Mat A;
> - MatCreateIS(PETSC_COMM_WORLD, 1, PETSC_DECIDE, PETSC_DECIDE, 3, 3,
> rmap, cmap, &A);
> + MatCreateIS(PETSC_COMM_WORLD, 1, PETSC_DECIDE, PETSC_DECIDE, size+1,
> size+1, rmap, cmap, &A);
>
> Mat Aloc;
> if (matType == "csr") {cout << matType << endl;
> MatCreateSeqAIJ(PETSC_COMM_SELF, 2, 2, 2, NULL, &Aloc);}
>
> ------------------------------
>
> *De: *"Stefano Zampini" <stefano.zampini at gmail.com>
> *À: *"Franck Houssen" <franck.houssen at inria.fr>
> *Cc: *"PETSc users list" <petsc-users at mcs.anl.gov>
> *Envoyé: *Mardi 20 Juin 2017 13:23:27
>
> *Objet: *Re: [petsc-users] Building MatIS with dense local matrix ?
>
> Franck
>
> I tested your new example with master and it works. However, It doesn't
> work with maint. I fixed the rectangular case a while ago in master and
> forgot to add the change to maint too. Sorry for that.
>
> This should fix the problem with maint: https://bitbucket.org/petsc/
> petsc/commits/0ea065fb06d751599c4157d36bfe1a1b41348e0b
>
> Test your real case and let me know.
> If you could, it would be good to test against master too.
>
> Thanks,
> Stefano
>
>
> 2017-06-20 12:58 GMT+02:00 Franck Houssen <franck.houssen at inria.fr>:
>
>> As I said, it is often difficult to reduce the "real" problem: it turns
>> out that your fix solves the "matISCSRDenseSquare/Rect.cpp" dummy example I
>> sent, but, it's still not working in "my real" situation.
>>
>> I changed a bit the "matISCSRDenseSquare/Rect.cpp" dummy example (see git
>> diff below - I just changed the point that overlaps) : the dummy example is
>> still failing
>>
>> "mpirun -n 2 ./matISCSRDenseSquare.exe csr" and "mpirun -n 2
>> ./matISCSRDenseSquare.exe dense" : OK
>> but
>> "mpirun -n 2 ./matISCSRDenseRect.exe csr" and "mpirun -n 2
>> ./matISCSRDenseRect.exe dense": KO with error "Argument out of range - New
>> nonzero at (0,2) caused a malloc"
>>
>> I would say, the problem (I am concerned with the "real" case) is around
>> lines 360-380 of /src/mat/impls/is/matis.c (not around 181 : this fixes a
>> valid problem, but, this problem is another one)
>>
>> Franck
>>
>> --- a/matISCSRDenseRect.cpp
>> +++ b/matISCSRDenseRect.cpp
>> @@ -18,7 +18,7 @@ int main(int argc,char **argv) {
>> int rank = 0; MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>
>> PetscInt localIdx[2] = {0, 0};
>> - if (rank == 0) {localIdx[0] = 0; localIdx[1] = 1;}
>> + if (rank == 0) {localIdx[0] = 0; localIdx[1] = 2;}
>> else {localIdx[0] = 1; localIdx[1] = 2;}
>> ISLocalToGlobalMapping rmap;
>> ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 1, 1, &rank,
>> PETSC_COPY_VALUES, &rmap);
>> diff --git a/Graphs/Franck/03.petscDDM/02.petscMailList/matISCSRDenseSquare.cpp
>> b/Graphs/Franck/03.petscDDM/02.petscMailList/matISCSRDenseSquare.cpp
>> index 4bc6190..4a6ea41 100644
>> --- a/matISCSRDenseSquare.cpp
>> +++ b/matISCSRDenseSquare.cpp
>> @@ -18,7 +18,7 @@ int main(int argc,char **argv) {
>> int rank = 0; MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>
>> PetscInt localIdx[2] = {0, 0};
>> - if (rank == 0) {localIdx[0] = 0; localIdx[1] = 1;}
>> + if (rank == 0) {localIdx[0] = 0; localIdx[1] = 2;}
>> else {localIdx[0] = 1; localIdx[1] = 2;}
>> ISLocalToGlobalMapping rmap;
>> ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 1, 2, localIdx,
>> PETSC_COPY_VALUES, &rmap);
>>
>> ------------------------------
>>
>> *De: *"Stefano Zampini" <stefano.zampini at gmail.com>
>> *À: *"Franck Houssen" <franck.houssen at inria.fr>
>> *Cc: *"PETSc users list" <petsc-users at mcs.anl.gov>
>> *Envoyé: *Mardi 20 Juin 2017 00:23:24
>>
>> *Objet: *Re: [petsc-users] Building MatIS with dense local matrix ?
>>
>> It should be fixed now in maint and master
>>
>> https://bitbucket.org/petsc/petsc/commits/4c8dd594d1988a0cbe282f8a37d991
>> 6f61e0c445
>>
>> Thanks for reporting the problem,
>> Stefano
>>
>> On Jun 19, 2017, at 10:46 PM, Stefano Zampini <stefano.zampini at gmail.com>
>> wrote:
>>
>> Franck,
>>
>> Thanks. I'll get back soon with a fix.
>>
>> Stefano
>>
>>
>> Il 19 Giu 2017 18:17, "Franck Houssen" <franck.houssen at inria.fr> ha
>> scritto:
>>
>>> The problem was difficult to reduce as reducing make things disappear...
>>> Luckily, I believe I got it (or at least, it looks "like" the one I
>>> "really" have...).
>>>
>>> Seems that for square matrix, it works fine for csr and dense matrix.
>>> But, If I am not mistaken, it does not for dense rectangular matrix (still
>>> OK for csr).
>>>
>>> matISCSRDenseSquare.cpp: 2 procs, global 3x3 matrix, each proc adds a
>>> 2x2 local matrix in the global matrix.
>>> matISCSRDenseRect.cpp: 2 procs, global *2*x3 matrix, each proc adds a
>>> *1*x2 local *vector* in the global matrix.
>>>
>>> reminder: running debian/testing with gcc-6.3 + petsc-3.7.6
>>>
>>> Franck
>>>
>>> >> mpirun -n 2 ./matISCSRDenseSquare.exe csr; mpirun -n 2
>>> ./matISCSRDenseSquare.exe dense
>>> csr
>>> csr
>>> Mat Object: 2 MPI processes
>>> type: is
>>> Mat Object: 1 MPI processes
>>> type: seqaij
>>> row 0: (0, 1.) (1, 0.)
>>> row 1: (0, 0.) (1, 1.)
>>> Mat Object: 1 MPI processes
>>> type: seqaij
>>> row 0: (0, 1.) (1, 0.)
>>> row 1: (0, 0.) (1, 1.)
>>> dense
>>> dense
>>> Mat Object: 2 MPI processes
>>> type: is
>>> Mat Object: 1 MPI processes
>>> type: seqdense
>>> 1.0000000000000000e+00 0.0000000000000000e+00
>>> 0.0000000000000000e+00 1.0000000000000000e+00
>>> Mat Object: 1 MPI processes
>>> type: seqdense
>>> 1.0000000000000000e+00 0.0000000000000000e+00
>>> 0.0000000000000000e+00 1.0000000000000000e+00
>>>
>>> >> mpirun -n 2 ./matISCSRDenseRect.exe csr; mpirun -n 2
>>> ./matISCSRDenseRect.exe dense
>>> csr
>>> csr
>>> Mat Object: 2 MPI processes
>>> type: is
>>> Mat Object: 1 MPI processes
>>> type: seqaij
>>> row 0: (0, 1.) (1, 0.)
>>> Mat Object: 1 MPI processes
>>> type: seqaij
>>> row 0: (0, 1.) (1, 0.)
>>> dense
>>> dense
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Argument out of range
>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [1]PETSC ERROR: Argument out of range
>>> [1]PETSC ERROR: New nonzero at (0,1) caused a malloc
>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>> turn off this check
>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [1]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017
>>> [1]PETSC ERROR: ./matISCSRDenseRect.exe on a arch-linux2-c-debug named
>>> yoda by fghoussen Mon Jun 19 18:08:58 2017
>>> New nonzero at (0,1) caused a malloc
>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to
>>> turn off this check
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.7.6, Apr, 24, 2017
>>> [1]PETSC ERROR: Configure options --prefix=/home/fghoussen/
>>> Documents/INRIA/petsc-3.7.6/local --with-mpi=1 --with-pthread=1
>>> --download-f2cblaslapack=yes --download-mumps=yes --download-scalapack=yes
>>> --download-superlu=yes --download-suitesparse=yes
>>> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 616 in
>>> /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/
>>> impls/aij/mpi/mpiaij.c
>>> [1]PETSC ERROR: #2 MatSetValues() line 1190 in /home/fghoussen/Documents/
>>> INRIA/petsc-3.7.6/src/mat/interface/matrix.c
>>> [1]PETSC ERROR: #3 MatSetValuesLocal() line 2053 in
>>> /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/interface/matrix.c
>>> [1]PETSC ERROR: #4 MatISGetMPIXAIJ_IS() line 365 in
>>> /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/impls/is/matis.c
>>> [1]PETSC ERROR: #5 MatISGetMPIXAIJ() line 437 in
>>> /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/impls/is/matis.c
>>> [0]PETSC ERROR: ./matISCSRDenseRect.exe on a arch-linux2-c-debug named
>>> yoda by fghoussen Mon Jun 19 18:08:58 2017
>>> [0]PETSC ERROR: Configure options --prefix=/home/fghoussen/
>>> Documents/INRIA/petsc-3.7.6/local --with-mpi=1 --with-pthread=1
>>> --download-f2cblaslapack=yes --download-mumps=yes --download-scalapack=yes
>>> --download-superlu=yes --download-suitesparse=yes
>>> [0]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 582 in
>>> /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/
>>> impls/aij/mpi/mpiaij.c
>>> [0]PETSC ERROR: #2 MatSetValues() line 1190 in /home/fghoussen/Documents/
>>> INRIA/petsc-3.7.6/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #3 MatSetValuesLocal() line 2053 in
>>> /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #4 MatISGetMPIXAIJ_IS() line 365 in
>>> /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/impls/is/matis.c
>>> [0]PETSC ERROR: #5 MatISGetMPIXAIJ() line 437 in
>>> /home/fghoussen/Documents/INRIA/petsc-3.7.6/src/mat/impls/is/matis.c
>>> Mat Object: 2 MPI processes
>>> type: is
>>> Mat Object: 1 MPI processes
>>> type: seqdense
>>> 1.0000000000000000e+00 0.0000000000000000e+00
>>> Mat Object: 1 MPI processes
>>> type: seqdense
>>> 1.0000000000000000e+00 0.0000000000000000e+00
>>>
>>> >> diff matISCSRDenseSquare.cpp matISCSRDenseRect.cpp
>>> 3c3
>>> < // ~> g++ -o matISCSRDenseSquare.exe matISCSRDenseSquare.cpp -lpetsc
>>> -lm; mpirun -n 2 matISCSRDenseSquare.exe
>>> ---
>>> > // ~> g++ -o matISCSRDenseRect.exe matISCSRDenseRect.cpp -lpetsc -lm;
>>> mpirun -n 2 matISCSRDenseRect.exe
>>> 24c24
>>> < ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 1, 2, localIdx,
>>> PETSC_COPY_VALUES, &rmap);
>>> ---
>>> > ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, 1, 1, &rank,
>>> PETSC_COPY_VALUES, &rmap);
>>> 29c29
>>> < MatCreateIS(PETSC_COMM_WORLD, 1, PETSC_DECIDE, PETSC_DECIDE, 3, 3,
>>> rmap, cmap, &A);
>>> ---
>>> > MatCreateIS(PETSC_COMM_WORLD, 1, PETSC_DECIDE, PETSC_DECIDE, 2, 3,
>>> rmap, cmap, &A);
>>> 32,33c32,33
>>> < if (matType == "csr") {cout << matType << endl;
>>> MatCreateSeqAIJ(PETSC_COMM_SELF, 2, 2, 2, NULL, &Aloc);}
>>> < else {cout << matType << endl; MatCreateSeqDense(PETSC_COMM_SELF,
>>> 2, 2, NULL, &Aloc);}
>>> ---
>>> > if (matType == "csr") {cout << matType << endl;
>>> MatCreateSeqAIJ(PETSC_COMM_SELF, 1, 2, 2, NULL, &Aloc);}
>>> > else {cout << matType << endl; MatCreateSeqDense(PETSC_COMM_SELF,
>>> 1, 2, NULL, &Aloc);}
>>> 35,36c35,36
>>> < PetscScalar localVal[4] = {1., 0., 0., 1.};
>>> < MatSetValues(Aloc, 2, localIdx, 2, localIdx, localVal, ADD_VALUES);
>>> // Add local 2x2 matrix
>>> ---
>>> > PetscScalar localVal[2] = {1., 0.}; PetscInt oneLocalRow = 0;
>>> > MatSetValues(Aloc, 1, &oneLocalRow, 2, localIdx, localVal,
>>> ADD_VALUES); // Add local row
>>>
>>> ------------------------------
>>>
>>> *De: *"Stefano Zampini" <stefano.zampini at gmail.com>
>>> *À: *"Franck Houssen" <franck.houssen at inria.fr>
>>> *Cc: *"PETSc users list" <petsc-users at mcs.anl.gov>
>>> *Envoyé: *Lundi 19 Juin 2017 15:25:35
>>> *Objet: *Re: [petsc-users] Building MatIS with dense local matrix ?
>>>
>>> Can you send a minimal working example so that I can fix the code?
>>>
>>> Thanks
>>> Stefano
>>>
>>> Il 19 Giu 2017 15:20, "Franck Houssen" <franck.houssen at inria.fr> ha
>>> scritto:
>>>
>>>> Hi,
>>>>
>>>> I try to call MatISGetMPIXAIJ on a MatIS (A) that has been feed locally
>>>> by sequential (Aloc) dense matrix.
>>>> Seems this ends up with this error: [0]PETSC ERROR: New nonzero at
>>>> (0,1) caused a malloc. Is this a known error / limitation ? (not supposed
>>>> to work with dense matrix ?)
>>>>
>>>> This (pseudo code) works fine:
>>>> MatCreateIS(..., A)
>>>> MatCreateSeqAIJ(..., Aloc)
>>>> MatISSetLocalMat(pcA, pcALoc)
>>>> MatISGetMPIXAIJ(A, ...) // OK !
>>>>
>>>> When I try to replace MatCreateSeqAIJ(..., Aloc) with
>>>> MatCreateSeqDense(..., Aloc), it does no more work.
>>>>
>>>> Franck
>>>>
>>>> PS: running debian/testing with gcc-6.3 + petsc-3.7.6
>>>>
>>>
>>>
>>
>>
>
>
> --
> Stefano
>
>
>
--
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20170620/7402f8ae/attachment-0001.html>
More information about the petsc-users
mailing list