<div dir="ltr"><div dir="ltr">On Tue, Oct 22, 2019 at 1:37 PM Thibaut Appel <<a href="mailto:t.appel17@imperial.ac.uk">t.appel17@imperial.ac.uk</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi both,</p>
<p>Please find attached a tiny example (in Fortran, sorry Matthew)
that - I think - reproduces the problem we mentioned.</p>
<p>Let me know.</p>
<p></p></div></blockquote><div>Okay, I converted to C so I could understand, and it runs fine for me:</div><div><br></div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">master *:~/Downloads/tmp$ PETSC_ARCH=arch-master-complex-debug make main</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">/PETSc3/petsc/bin/mpicc -o main.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g3 <span class="gmail-Apple-converted-space"> </span>-I/PETSc3/petsc/petsc-dev/include -I/PETSc3/petsc/petsc-dev/arch-master-complex-debug/include -I/PETSc3/petsc/include -I/opt/X11/include<span class="gmail-Apple-converted-space"> </span>`pwd`/main.c</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">/PETSc3/petsc/bin/mpicc -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind<span class="gmail-Apple-converted-space"> </span>-Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g3<span class="gmail-Apple-converted-space"> </span>-o main main.o -Wl,-rpath,/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib -L/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib -Wl,-rpath,/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib -L/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib -Wl,-rpath,/PETSc3/petsc/lib -L/PETSc3/petsc/lib -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib -Wl,-rpath,/usr/local/lib/gcc/x86_64-apple-darwin10.4.0/4.6.0 -L/usr/local/lib/gcc/x86_64-apple-darwin10.4.0/4.6.0 -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lpetsc -lfftw3_mpi -lfftw3 -llapack -lblas -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis -lmetis -ltriangle -lz -lX11 -lctetgen -lstdc++ -ldl -lmpichf90 -lpmpich -lmpich -lopa -lmpl -lpthread -lgfortran -lgcc_s.10.5 -lstdc++ -ldl</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">master *:~/Downloads/tmp$ ./main</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">After first assembly:</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">Mat Object: 1 MPI processes</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>type: seqaij</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 0: (0, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 1: (1, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 2: (2, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 3: (3, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 4: (4, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 5: (5, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 6: (6, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 7: (7, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 8: (8, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 9: (9, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">After second assembly:</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">Mat Object: 1 MPI processes</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>type: seqaij</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 0: (0, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 1: (1, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 2: (2, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 3: (3, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 4: (4, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 5: (5, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 6: (6, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 7: (7, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 8: (8, 1. + 1. i)</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 9: (9, 1. + 1. i)</p>
<div><span style="font-family:Menlo;font-size:11px">row 0 col: 9 val: 0. + 0. i</span></div><div><br></div><div>I attach my code. So then I ran your Fortran:</div><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">/PETSc3/petsc/bin/mpif90 -c -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -Wno-unused-variable -g<span class="gmail-Apple-converted-space"> </span>-I/PETSc3/petsc/petsc-dev/include -I/PETSc3/petsc/petsc-dev/arch-master-complex-debug/include -I/PETSc3/petsc/include -I/opt/X11/include<span class="gmail-Apple-converted-space"> </span>-o main2.o main2.F90</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">/PETSc3/petsc/bin/mpif90 -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind<span class="gmail-Apple-converted-space"> </span>-Wall -ffree-line-length-0 -Wno-unused-dummy-argument -Wno-unused-variable -g <span class="gmail-Apple-converted-space"> </span>-o main2 main2.o -Wl,-rpath,/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib -L/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib -Wl,-rpath,/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib -L/PETSc3/petsc/petsc-dev/arch-master-complex-debug/lib -Wl,-rpath,/PETSc3/petsc/lib -L/PETSc3/petsc/lib -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib -Wl,-rpath,/usr/local/lib/gcc/x86_64-apple-darwin10.4.0/4.6.0 -L/usr/local/lib/gcc/x86_64-apple-darwin10.4.0/4.6.0 -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lpetsc -lfftw3_mpi -lfftw3 -llapack -lblas -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis -lmetis -ltriangle -lz -lX11 -lctetgen -lstdc++ -ldl -lmpichf90 -lpmpich -lmpich -lopa -lmpl -lpthread -lgfortran -lgcc_s.10.5 -lstdc++ -ldl</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">master *:~/Downloads/tmp$ ./main2</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>After first assembly:</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">Mat Object: 1 MPI processes</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>type: seqaij</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 0: (0, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 1: (1, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 2: (2, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 3: (3, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 4: (4, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 5: (5, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 6: (6, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 7: (7, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 8: (8, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 9: (9, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>After second assembly:</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">Mat Object: 1 MPI processes</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>type: seqaij</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 0: (0, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 1: (1, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 2: (2, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 3: (3, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 4: (4, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 5: (5, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 6: (6, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 7: (7, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 8: (8, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo">row 9: (9, 1.)<span class="gmail-Apple-converted-space"> </span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span class="gmail-Apple-converted-space"> </span>row:<span class="gmail-Apple-converted-space"> </span>0 col:<span class="gmail-Apple-converted-space"> </span>9 val:<span class="gmail-Apple-converted-space"> </span>0.000000000000000000E+00<span class="gmail-Apple-converted-space"> </span>0.000000000000000000E+00</p></div><div><br></div><div>I am not seeing an error. Am I not running it correctly?</div><div><br></div><div> Thanks,</div><div><br></div><div> MAtt </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>Thibaut<br>
</p>
<p><br>
</p>
<div>On 22/10/2019 17:48, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Tue, Oct 22, 2019 at 12:43 PM Thibaut Appel
via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Hong,</p>
<p>Thank you for having a look, I copied/pasted your code
snippet into ex28.c and the error indeed appears if you
change that col[0]. That's because you did not allow a
new non-zero location in the matrix with the option
MAT_NEW_NONZERO_LOCATION_ERR.<br>
</p>
<p>I spent the day debugging the code and already checked
my calls to MatSetValues:</p>
<p>For all MatSetValues calls corresponding to the row/col
location in the error messages in the subsequent
assembly, the numerical value associated with that
row/col was exactly (0.0,0.0) (complex arithmetic) so it
shouldn't be inserted w.r.t. the option
MAT_IGNORE_ZERO_ENTRIES. It seems MatSetValues still did
it anyway.<br>
</p>
</div>
</blockquote>
<div>Okay, lets solve this problem first. You say that</div>
<div><br>
</div>
<div> - You called MatSetOption(A, MAT_IGNORE_ZERO_ENTRIES,
PETSC_TRUE)<br>
</div>
<div> - You called MatSetValues(A, ,,,, ADD_VALUES, ..., val)
and val had a complex 0 in it</div>
<div> - PETSc tried to insert the complex 0</div>
<div><br>
</div>
<div>This should be really easy to test in a tiny example. Do
you mind making it? If its broken, I will fix it.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I was able to solve the problem by adding
MatSetOption(L,MAT_NEW_NONZERO_LOCATIONS,PETSC_FALSE)
after my first assembly. However I don't know why it
fixed it as the manual seems to say this is just for
efficiency purposes</p>
<p>It "should be specified after the first matrix has been
fully assembled. This option ensures that certain data
structures and communication information will be reused
(instead of regenerated during successive steps, thereby
increasing efficiency"</p>
<p>So I'm still puzzled by why I got that error in the
first place. Unless "regenerated" implies resetting some
attributes of the preallocated non-zero structure /
first assembly?</p>
<p><br>
</p>
<p>Thibaut</p>
<p><br>
</p>
<div>On 22/10/2019 17:06, Zhang, Hong wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Thibaut:<br>
</div>
<div>Check your code on MatSetValues(), likely you set
a value "to a new nonzero at global row/column (200,
160) into matrix" L.</div>
<div>I tested petsc/src/mat/examples/tests/ex28.c by
adding </div>
<div>@@ -95,6 +95,26 @@ int main(int argc,char **args)<br>
/* Compute numeric factors using same F, then
solve */<br>
for (k = 0; k < num_numfac; k++) {<br>
/* Get numeric factor of A[k] */<br>
+ if (k==0) {<br>
+ ierr = MatZeroEntries(A[0]);CHKERRQ(ierr);<br>
+ for (i=rstart; i<rend; i++) {<br>
+ col[0] = i-1; col[1] = i; col[2] = i+1;<br>
+ if (i == 0) {<br>
+ ierr =
MatSetValues(A[k],1,&i,2,col+1,value+1,INSERT_VALUES);CHKERRQ(ierr);<br>
+ } else if (i == N-1) {<br>
+ ierr =
MatSetValues(A[k],1,&i,2,col,value,INSERT_VALUES);CHKERRQ(ierr);<br>
+ } else {<br>
+ ierr =
MatSetValues(A[k],1,&i,3,col,value,INSERT_VALUES);CHKERRQ(ierr);<br>
+ }<br>
+ }<br>
+ if (!rank) {<br>
+ i = N - 1; col[0] = N - 1;<br>
+ ierr =
MatSetValues(A[k],1,&i,1,col,value,INSERT_VALUES);CHKERRQ(ierr);<br>
+ }<br>
+ ierr =
MatAssemblyBegin(A[k],MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);<br>
+ ierr =
MatAssemblyEnd(A[k],MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);<br>
+ }<br>
+<br>
</div>
<div><br>
</div>
<div>It works in both sequential and parallel. If I
set col[0] = 0, then I get the same error as yours.</div>
<div>Hong</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Dear PETSc developers,</p>
<p>I'm extending a validated matrix
preallocation/assembly part of my code to
solve multiple linear systems with MUMPS at
each iteration of a main loop, following the
example src/mat/examples/tests/ex28.c that
Hong Zhang added a few weeks ago. The
difference is that I'm using just 1 matrix to
solve different systems.<br>
</p>
<p>I'm trying to investigate a nasty bug arising
when I try to assemble "for a second time"
that MPIAIJ matrix. The issue arises only in
parallel, serial works fine.</p>
<p>Creating 1 MPIAIJ matrix, preallocating it
"perfectly" with the case where I have the
fewest zero entries in the non-zero structure,
before getting its symbolic factorization.</p>
<p>Further in the main loop, I'm solely changing
its entries <b>retaining the non-zero
structure</b>.<br>
</p>
<p>Here is the simplified Fortran code I'm
using:<br>
</p>
<p><tt>! Fill (M,N) case to ensure all non-zero
entries are preallocated</tt><tt><br>
</tt><tt>CALL set_equations(M,N)</tt><tt><br>
</tt><tt><br>
</tt><tt>CALL alloc_matrix(L)</tt><tt><br>
</tt><tt> ! --> Call
MatSeqAIJSetPreallocation/MatMPIAIJSetPreallocation</tt><tt><br>
</tt><tt> ! --> Sets
MAT_IGNORE_ZERO_ENTRIES,
MAT_NEW_NONZERO_ALLOCATION_ERR,
MAT_NO_OFF_PROC_ENTRIES to true</tt><tt><br>
</tt><tt><br>
</tt><tt>CALL assemble_matrix(L)</tt><tt><br>
</tt><tt> ! --> Calls MatSetValues with
ADD_VALUES</tt><tt><br>
</tt><tt> ! --> Call
MatAssemblyBegin/MatAssemblyEnd</tt><tt><br>
</tt><tt><br>
</tt><tt>! Tell PETSc that new non-zero
insertions in matrix are forbidden</tt><tt><br>
</tt><tt>CALL
MatSetOption(L,MAT_NEW_NONZERO_LOCATION_ERR,PETSC_TRUE,ierr)</tt><tt><br>
</tt><tt><br>
</tt><tt>CALL set_mumps_parameters()</tt><tt><br>
</tt><tt><br>
</tt><tt>! Get symbolic LU factorization using
MUMPS</tt><tt><br>
</tt><tt>CALL
MatGetFactor(L,MATSOLVERMUMPS,MAT_FACTOR_LU,F,ierr)</tt><tt><br>
</tt><tt>CALL
MatGetOrdering(L,MATORDERINGNATURAL,rperm,cperm,ierr)</tt><tt><br>
</tt><tt>CALL
MatLUFactorSymbolic(F,L,rperm,cperm,info,ierr)</tt><tt><br>
</tt><tt><br>
</tt><tt>CALL initialize_right_hand_sides()</tt><tt><br>
</tt><tt><br>
</tt><tt>! Zero matrix entries</tt><tt><br>
</tt><tt>CALL MatZeroEntries(L,ierr)</tt><tt><br>
</tt><tt><br>
</tt><tt>! Main loop</tt><tt><br>
</tt><tt>DO itr=1, maxitr</tt><tt><br>
</tt><tt><br>
</tt><tt> DO m = 1, M</tt><tt><br>
</tt><tt> DO n = 1, N</tt><tt><br>
</tt><tt><br>
</tt><tt> CALL set_equations(m,n)</tt><tt><br>
</tt><tt> CALL assemble_matrix(L) ! ERROR
HERE when m=1, n=1, CRASH IN MatSetValues
call</tt><tt><br>
</tt><tt><br>
</tt><tt> ! Solving the linear system
associated with (m,n)</tt><tt><br>
</tt><tt> CALL
MatLUFactorNumeric(F,L,info,ierr)</tt><tt><br>
</tt><tt> CALL
MatSolve(F,v_rhs(m,n),v_sol(m,n),ierr) </tt><tt><br>
</tt><tt><br>
</tt><tt> ! Process v_rhs's from v_sol's
for next iteration</tt><tt><br>
</tt><tt><br>
</tt><tt> CALL MatZeroEntries(L,ierr)</tt><tt><br>
</tt><tt><br>
</tt><tt> END DO</tt><tt><br>
</tt><tt> END DO</tt><tt><br>
</tt><tt><br>
</tt><tt>END DO</tt></p>
<p><tt><br>
</tt></p>
<p>Testing on a small case, the error I get is</p>
<tt>[1]PETSC ERROR: --------------------- Error
Message
--------------------------------------------------------------</tt><tt><br>
</tt><tt>[1]PETSC ERROR: Argument out of range</tt><tt><br>
</tt><tt>[1]PETSC ERROR: Inserting a new nonzero
at global row/column (200, 160) into matrix</tt><tt><br>
</tt><tt>[1]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">
https://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.</tt><tt><br>
</tt><tt>[1]PETSC ERROR: Petsc Release Version
3.12.0, unknown </tt><tt><br>
</tt><tt>[1]PETSC ERROR: Configure options
--PETSC_ARCH=cplx_gcc_debug
--with-scalar-type=complex
--with-precision=double --with-debugging=1
--with-valgrind=1 --with-debugger=gdb
--with-fortran-kernels=1 --download-mpich
--download-fblaslapack --download-scalapack
--download-metis --download-parmetis
--download-ptscotch --download-mumps
--download-slepc --COPTFLAGS="-O0 -g"
--CXXOPTFLAGS="-O0 -g" --FOPTFLAGS="-O0 -g
-fbacktrace"</tt><tt><br>
</tt><tt>[1]PETSC ERROR: #1
MatSetValues_MPIAIJ() line 634 in
/home/thibaut/Packages/petsc/src/mat/impls/aij/mpi/mpiaij.c</tt><tt><br>
</tt><tt>[1]PETSC ERROR: #2 MatSetValues() line
1375 in
/home/thibaut/Packages/petsc/src/mat/interface/matrix.c</tt><tt><br>
</tt><tt>[1]PETSC ERROR: #3 User provided
function() line 0 in User file</tt><tt><br>
</tt><tt>application called
MPI_Abort(MPI_COMM_SELF, 63) - process 0</tt><tt><br>
</tt>
<p> <br>
</p>
<p>which I don't understand. That element was
not in the non-zero structure and wasn't
preallocated. I printed the value to be
inserted at this location (200,160) and it is
exactly
(0.0000000000000000,0.0000000000000000) so
this entry should not be inserted due to
MAT_IGNORE_ZERO_ENTRIES, however it seems it
is. I'm using ADD_VALUES in MatSetValues but
it is the only call where (200,160) is
inserted.<br>
</p>
<p><br>
</p>
<p> - I zero the matrix entries with
MatZeroEntries which retains the non-zero
structure (checked when I print the matrix)
but tried to comment the corresponding calls.<br>
</p>
<p> - I tried to set
MAT_NEW_NONZERO_LOCATION_ERR AND
MAT_NEW_NONZERO_ALLOCATION_ERR to PETSC_FALSE
without effect.</p>
<p><br>
</p>
<p>Perhaps there's something fundamentally wrong
in my approach, in any case would you have any
suggestions to identify the exact problem? <br>
</p>
<p>Using PETSc 3.12.0. Thanks for your support,</p>
<p><br>
</p>
<p>Thibaut<br>
</p>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>