<div dir="ltr"><div dir="ltr">On Thu, Oct 24, 2019 at 6:09 AM Thibaut Appel <<a href="mailto:t.appel17@imperial.ac.uk">t.appel17@imperial.ac.uk</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matthew,</p>
<p>Thanks for having a look, your example runs just like mine in
Fortran.</p>
<p>In serial, the value (0.0,0.0) was inserted whereas it shouldn't
have.</p></div></blockquote><div>I do not see that in serial. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>In parallel, you'll see that an error "Inserting a new nonzero at
global row/column" is triggered.</p></div></blockquote><div>Yes, I see the error running in parallel. Will check Barry's fix.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>In both cases, MatSetValues tries to insert a zero value whereas
IGNORE_ZERO_ENTRIES was set. That's what Barry is looking into, if
I'm not mistaken.</p>
<p><br>
</p>
<p>Thibaut<br>
</p>
<p><br>
</p>
<div>On 24/10/2019 02:31, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Tue, Oct 22, 2019 at 1:37 PM Thibaut Appel
<<a href="mailto:t.appel17@imperial.ac.uk" target="_blank">t.appel17@imperial.ac.uk</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi both,</p>
<p>Please find attached a tiny example (in Fortran, sorry
Matthew) that - I think - reproduces the problem we
mentioned.</p>
<p>Let me know.</p>
</div>
</blockquote>
<div>Okay, I converted to C so I could understand, and it runs
fine for me:</div>
<div><br>
</div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">master
*:~/Downloads/tmp$ PETSC_ARCH=arch-master-complex-debug make
main</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">/PETSc3/petsc/bin/mpicc
-o main.o -c -Wall -Wwrite-strings -Wno-strict-aliasing
-Wno-unknown-pragmas -fstack-protector -Qunused-arguments
-fvisibility=hidden -g3 <span> </span>-I/PETSc3/petsc/petsc-dev/include
-I/PETSc3/petsc/petsc-dev/arch-master-complex-debug/include
-I/PETSc3/petsc/include -I/opt/X11/include<span> </span>`pwd`/main.c</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">/PETSc3/petsc/bin/mpicc
-Wl,-multiply_defined,suppress -Wl,-multiply_defined
-Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first
-Wl,-no_compact_unwind<span> </span>-Wall
-Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
-fstack-protector -Qunused-arguments -fvisibility=hidden -g3<span> </span>-o main
main.o
-Wl,-rpath,/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib
-L/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib
-Wl,-rpath,/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib
-L/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib
-Wl,-rpath,/PETSc3/petsc/lib -L/PETSc3/petsc/lib
-Wl,-rpath,/opt/X11/lib -L/opt/X11/lib
-Wl,-rpath,/usr/local/lib/gcc/x86_64-apple-darwin10.4.0/4.6.0
-L/usr/local/lib/gcc/x86_64-apple-darwin10.4.0/4.6.0
-Wl,-rpath,/usr/local/lib -L/usr/local/lib -lpetsc
-lfftw3_mpi -lfftw3 -llapack -lblas -lhdf5hl_fortran
-lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis -lmetis
-ltriangle -lz -lX11 -lctetgen -lstdc++ -ldl -lmpichf90
-lpmpich -lmpich -lopa -lmpl -lpthread -lgfortran
-lgcc_s.10.5 -lstdc++ -ldl</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">master
*:~/Downloads/tmp$ ./main</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">After
first assembly:</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">Mat
Object: 1 MPI processes</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span> </span>type: seqaij</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
0: (0, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
1: (1, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
2: (2, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
3: (3, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
4: (4, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
5: (5, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
6: (6, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
7: (7, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
8: (8, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
9: (9, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">After
second assembly:</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">Mat
Object: 1 MPI processes</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span> </span>type: seqaij</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
0: (0, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
1: (1, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
2: (2, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
3: (3, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
4: (4, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
5: (5, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
6: (6, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
7: (7, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
8: (8, 1. + 1. i)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
9: (9, 1. + 1. i)</p>
<div><span style="font-family:Menlo;font-size:11px">row 0 col:
9 val: 0. + 0. i</span></div>
<div><br>
</div>
<div>I attach my code. So then I ran your Fortran:</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">/PETSc3/petsc/bin/mpif90
-c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument
-Wno-unused-variable -g<span> </span>-I/PETSc3/petsc/petsc-dev/include
-I/PETSc3/petsc/petsc-dev/arch-master-complex-debug/include
-I/PETSc3/petsc/include -I/opt/X11/include<span> </span>-o
main2.o main2.F90</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">/PETSc3/petsc/bin/mpif90
-Wl,-multiply_defined,suppress -Wl,-multiply_defined
-Wl,suppress -Wl,-commons,use_dylibs
-Wl,-search_paths_first -Wl,-no_compact_unwind<span> </span>-Wall
-ffree-line-length-0 -Wno-unused-dummy-argument
-Wno-unused-variable -g <span> </span>-o main2
main2.o
-Wl,-rpath,/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib
-L/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib
-Wl,-rpath,/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib
-L/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib
-Wl,-rpath,/PETSc3/petsc/lib -L/PETSc3/petsc/lib
-Wl,-rpath,/opt/X11/lib -L/opt/X11/lib
-Wl,-rpath,/usr/local/lib/gcc/x86_64-apple-darwin10.4.0/4.6.0
-L/usr/local/lib/gcc/x86_64-apple-darwin10.4.0/4.6.0
-Wl,-rpath,/usr/local/lib -L/usr/local/lib -lpetsc
-lfftw3_mpi -lfftw3 -llapack -lblas -lhdf5hl_fortran
-lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis -lmetis
-ltriangle -lz -lX11 -lctetgen -lstdc++ -ldl -lmpichf90
-lpmpich -lmpich -lopa -lmpl -lpthread -lgfortran
-lgcc_s.10.5 -lstdc++ -ldl</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">master
*:~/Downloads/tmp$ ./main2</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span> </span>After first
assembly:</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">Mat
Object: 1 MPI processes</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span> </span>type:
seqaij</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
0: (0, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
1: (1, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
2: (2, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
3: (3, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
4: (4, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
5: (5, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
6: (6, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
7: (7, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
8: (8, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
9: (9, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span> </span>After second
assembly:</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">Mat
Object: 1 MPI processes</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span> </span>type:
seqaij</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
0: (0, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
1: (1, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
2: (2, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
3: (3, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
4: (4, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
5: (5, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
6: (6, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
7: (7, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
8: (8, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row
9: (9, 1.)<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span> </span>row:<span> </span>0 col:<span> </span>9 val:<span> </span>0.000000000000000000E+00<span> </span>0.000000000000000000E+00</p>
</div>
<div><br>
</div>
<div>I am not seeing an error. Am I not running it correctly?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> MAtt </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Thibaut<br>
</p>
<p><br>
</p>
<div>On 22/10/2019 17:48, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Tue, Oct 22, 2019 at 12:43 PM
Thibaut Appel via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Hong,</p>
<p>Thank you for having a look, I copied/pasted
your code snippet into ex28.c and the error
indeed appears if you change that col[0].
That's because you did not allow a new
non-zero location in the matrix with the
option MAT_NEW_NONZERO_LOCATION_ERR.<br>
</p>
<p>I spent the day debugging the code and
already checked my calls to MatSetValues:</p>
<p>For all MatSetValues calls corresponding to
the row/col location in the error messages in
the subsequent assembly, the numerical value
associated with that row/col was exactly
(0.0,0.0) (complex arithmetic) so it shouldn't
be inserted w.r.t. the option
MAT_IGNORE_ZERO_ENTRIES. It seems MatSetValues
still did it anyway.<br>
</p>
</div>
</blockquote>
<div>Okay, lets solve this problem first. You say
that</div>
<div><br>
</div>
<div> - You called
MatSetOption(A, MAT_IGNORE_ZERO_ENTRIES,
PETSC_TRUE)<br>
</div>
<div> - You called MatSetValues(A, ,,,, ADD_VALUES,
..., val) and val had a complex 0 in it</div>
<div> - PETSc tried to insert the complex 0</div>
<div><br>
</div>
<div>This should be really easy to test in a tiny
example. Do you mind making it? If its broken, I
will fix it.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I was able to solve the problem by adding
MatSetOption(L,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE)
after my first assembly. However I don't know
why it fixed it as the manual seems to say
this is just for efficiency purposes</p>
<p>It "should be specified after the first
matrix has been fully assembled. This option
ensures that certain data structures and
communication information will be reused
(instead of regenerated during successive
steps, thereby increasing efficiency"</p>
<p>So I'm still puzzled by why I got that error
in the first place. Unless "regenerated"
implies resetting some attributes of the
preallocated non-zero structure / first
assembly?</p>
<p><br>
</p>
<p>Thibaut</p>
<p><br>
</p>
<div>On 22/10/2019 17:06, Zhang, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Thibaut:<br>
</div>
<div>Check your code on MatSetValues(),
likely you set a value "to a new nonzero
at global row/column (200, 160) into
matrix" L.</div>
<div>I
tested petsc/src/mat/examples/tests/ex28.c
by adding </div>
<div>@@ -95,6 +95,26 @@ int main(int
argc,char **args)<br>
/* Compute numeric factors using same
F, then solve */<br>
for (k = 0; k < num_numfac; k++) {<br>
/* Get numeric factor of A[k] */<br>
+ if (k==0) {<br>
+ ierr =
MatZeroEntries(A[0]);CHKERRQ(ierr);<br>
+ for (i=rstart; i<rend; i++) {<br>
+ col[0] = i-1; col[1] = i; col[2]
= i+1;<br>
+ if (i == 0) {<br>
+ ierr =
MatSetValues(A[k],1,&i,2,col+1,value+1,INSERT_VALUES);CHKERRQ(ierr);<br>
+ } else if (i == N-1) {<br>
+ ierr =
MatSetValues(A[k],1,&i,2,col,value,INSERT_VALUES);CHKERRQ(ierr);<br>
+ } else {<br>
+ ierr =
MatSetValues(A[k],1,&i,3,col,value,INSERT_VALUES);CHKERRQ(ierr);<br>
+ }<br>
+ }<br>
+ if (!rank) {<br>
+ i = N - 1; col[0] = N - 1;<br>
+ ierr =
MatSetValues(A[k],1,&i,1,col,value,INSERT_VALUES);CHKERRQ(ierr);<br>
+ }<br>
+ ierr =
MatAssemblyBegin(A[k],MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);<br>
+ ierr =
MatAssemblyEnd(A[k],MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);<br>
+ }<br>
+<br>
</div>
<div><br>
</div>
<div>It works in both sequential and
parallel. If I set col[0] = 0, then I get
the same error as yours.</div>
<div>Hong</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Dear PETSc developers,</p>
<p>I'm extending a validated matrix
preallocation/assembly part of my
code to solve multiple linear
systems with MUMPS at each iteration
of a main loop, following the
example
src/mat/examples/tests/ex28.c that
Hong Zhang added a few weeks ago.
The difference is that I'm using
just 1 matrix to solve different
systems.<br>
</p>
<p>I'm trying to investigate a nasty
bug arising when I try to assemble
"for a second time" that MPIAIJ
matrix. The issue arises only in
parallel, serial works fine.</p>
<p>Creating 1 MPIAIJ matrix,
preallocating it "perfectly" with
the case where I have the fewest
zero entries in the non-zero
structure, before getting its
symbolic factorization.</p>
<p>Further in the main loop, I'm
solely changing its entries <b>retaining
the non-zero structure</b>.<br>
</p>
<p>Here is the simplified Fortran code
I'm using:<br>
</p>
<p><tt>! Fill (M,N) case to ensure all
non-zero entries are preallocated</tt><tt><br>
</tt><tt>CALL set_equations(M,N)</tt><tt><br>
</tt><tt><br>
</tt><tt>CALL alloc_matrix(L)</tt><tt><br>
</tt><tt> ! --> Call
MatSeqAIJSetPreallocation/MatMPIAIJSetPreallocation</tt><tt><br>
</tt><tt> ! --> Sets
MAT_IGNORE_ZERO_ENTRIES,
MAT_NEW_NONZERO_ALLOCATION_ERR,
MAT_NO_OFF_PROC_ENTRIES to true</tt><tt><br>
</tt><tt><br>
</tt><tt>CALL assemble_matrix(L)</tt><tt><br>
</tt><tt> ! --> Calls
MatSetValues with ADD_VALUES</tt><tt><br>
</tt><tt> ! --> Call
MatAssemblyBegin/MatAssemblyEnd</tt><tt><br>
</tt><tt><br>
</tt><tt>! Tell PETSc that new
non-zero insertions in matrix are
forbidden</tt><tt><br>
</tt><tt>CALL
MatSetOption(L,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE,ierr)</tt><tt><br>
</tt><tt><br>
</tt><tt>CALL set_mumps_parameters()</tt><tt><br>
</tt><tt><br>
</tt><tt>! Get symbolic LU
factorization using MUMPS</tt><tt><br>
</tt><tt>CALL
MatGetFactor(L,MATSOLVERMUMPS,MAT_FACTOR_LU,F,ierr)</tt><tt><br>
</tt><tt>CALL
MatGetOrdering(L,MATORDERINGNATURAL,rperm,cperm,ierr)</tt><tt><br>
</tt><tt>CALL
MatLUFactorSymbolic(F,L,rperm,cperm,info,ierr)</tt><tt><br>
</tt><tt><br>
</tt><tt>CALL
initialize_right_hand_sides()</tt><tt><br>
</tt><tt><br>
</tt><tt>! Zero matrix entries</tt><tt><br>
</tt><tt>CALL MatZeroEntries(L,ierr)</tt><tt><br>
</tt><tt><br>
</tt><tt>! Main loop</tt><tt><br>
</tt><tt>DO itr=1, maxitr</tt><tt><br>
</tt><tt><br>
</tt><tt> DO m = 1, M</tt><tt><br>
</tt><tt> DO n = 1, N</tt><tt><br>
</tt><tt><br>
</tt><tt> CALL set_equations(m,n)</tt><tt><br>
</tt><tt> CALL assemble_matrix(L)
! ERROR HERE when m=1, n=1, CRASH
IN MatSetValues call</tt><tt><br>
</tt><tt><br>
</tt><tt> ! Solving the linear
system associated with (m,n)</tt><tt><br>
</tt><tt> CALL
MatLUFactorNumeric(F,L,info,ierr)</tt><tt><br>
</tt><tt> CALL
MatSolve(F,v_rhs(m,n),v_sol(m,n),ierr)
</tt><tt><br>
</tt><tt><br>
</tt><tt> ! Process v_rhs's from
v_sol's for next iteration</tt><tt><br>
</tt><tt><br>
</tt><tt> CALL
MatZeroEntries(L,ierr)</tt><tt><br>
</tt><tt><br>
</tt><tt> END DO</tt><tt><br>
</tt><tt> END DO</tt><tt><br>
</tt><tt><br>
</tt><tt>END DO</tt></p>
<p><tt><br>
</tt></p>
<p>Testing on a small case, the error
I get is</p>
<tt>[1]PETSC ERROR:
--------------------- Error Message
--------------------------------------------------------------</tt><tt><br>
</tt><tt>[1]PETSC ERROR: Argument out
of range</tt><tt><br>
</tt><tt>[1]PETSC ERROR: Inserting a
new nonzero at global row/column
(200, 160) into matrix</tt><tt><br>
</tt><tt>[1]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">
https://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.</tt><tt><br>
</tt><tt>[1]PETSC ERROR: Petsc Release
Version 3.12.0, unknown </tt><tt><br>
</tt><tt>[1]PETSC ERROR: Configure
options --PETSC_ARCH=cplx_gcc_debug
--with-scalar-type=complex
--with-precision=double
--with-debugging=1 --with-valgrind=1
--with-debugger=gdb
--with-fortran-kernels=1
--download-mpich
--download-fblaslapack
--download-scalapack
--download-metis --download-parmetis
--download-ptscotch --download-mumps
--download-slepc --COPTFLAGS="-O0
-g" --CXXOPTFLAGS="-O0 -g"
--FOPTFLAGS="-O0 -g -fbacktrace"</tt><tt><br>
</tt><tt>[1]PETSC ERROR: #1
MatSetValues_MPIAIJ() line 634 in
/home/thibaut/Packages/petsc/src/mat/impls/aij/mpi/mpiaij.c</tt><tt><br>
</tt><tt>[1]PETSC ERROR: #2
MatSetValues() line 1375 in
/home/thibaut/Packages/petsc/src/mat/interface/matrix.c</tt><tt><br>
</tt><tt>[1]PETSC ERROR: #3 User
provided function() line 0 in User
file</tt><tt><br>
</tt><tt>application called
MPI_Abort(MPI_COMM_SELF, 63) -
process 0</tt><tt><br>
</tt>
<p> <br>
</p>
<p>which I don't understand. That
element was not in the non-zero
structure and wasn't preallocated. I
printed the value to be inserted at
this location (200,160) and it is
exactly
(0.0000000000000000,0.0000000000000000)
so this entry should not be inserted
due to MAT_IGNORE_ZERO_ENTRIES,
however it seems it is. I'm using
ADD_VALUES in MatSetValues but it is
the only call where (200,160) is
inserted.<br>
</p>
<p><br>
</p>
<p> - I zero the matrix entries
with MatZeroEntries which retains
the non-zero structure (checked when
I print the matrix) but tried to
comment the corresponding calls.<br>
</p>
<p> - I tried to set
MAT_NEW_NONZERO_LOCATION_ERR AND
MAT_NEW_NONZERO_ALLOCATION_ERR to
PETSC_FALSE without effect.</p>
<p><br>
</p>
<p>Perhaps there's something
fundamentally wrong in my approach,
in any case would you have any
suggestions to identify the exact
problem? <br>
</p>
<p>Using PETSc 3.12.0. Thanks for your
support,</p>
<p><br>
</p>
<p>Thibaut<br>
</p>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to which
their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>