Barry,<div><br></div><div>Indeed. </div><div><br></div><div>As a side remark, the number of unknowns for my least-squares problem is well within the maximum 32-bit integer limit. That's why I did not immediately think that 32-bit ints may cause a problem. It's only the matrix nonzero count that goes over that bound. Quick overview of my "A":</div>
<div><br></div><div> mglb (rows) =131857963, nglb (cols) =18752388, nnz_glb (nonzeros) = 5812947924</div><div><br></div><div>Going to 64-bit integers is not really an option. Long story short, I am working in single-precision mode, and the PETSc code is called from a Fortran kernel where we have imposed an EQUIVALENCE between single precision floats and ints (legacy design...)</div>
<div><br></div><div>Best,</div><div>Mihai</div><div><br><div class="gmail_quote">On Fri, Dec 14, 2012 at 2:28 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Mihai,<br>
<br>
Thanks for tracking down the problem. As a side note, you are getting close to using all of the space in int in your matrix row/column sizes, when you matrix sizes are great than 2^{31}-1 you will need to configure PETSc with --with-64-bit-indices to have PETSc use long long int for PetscInt.<br>
<br>
Satish,<br>
<br>
Could you please patch 3.3 and replace the use of %D with %lld and replace the (PetscInt) casts with (long long int) casts in the two lines<br>
<div class="im"> ierr = PetscViewerASCIIPrintf(viewer,"total: nonzeros=%D, allocated nonzeros=%D\n",(PetscInt)info.nz_used,(PetscInt)info.nz_allocated);CHKERRQ(ierr);<br>
ierr = PetscViewerASCIIPrintf(viewer,"total number of mallocs used during MatSetValues calls =%D\n",(PetscInt)info.mallocs);CHKERRQ(ierr);<br>
<br>
</div> Thanks<br>
<span class="HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Dec 14, 2012, at 4:59 AM, Mihai Alexe <<a href="mailto:malexe@vt.edu">malexe@vt.edu</a>> wrote:<br>
<br>
> Barry,<br>
><br>
> I've tracked down the problem.<br>
><br>
> I ran with -info -mat_view_info, and fpe's enabled and got a SIGFPE after entering MatCreateMPIAIJWithSplitArrays (Petsc did not produce a stacktrace unfortunately). This was due to a floating point exception in a typecast inside mat/interface/matrix.c:<br>
><br>
> if (mat->ops->getinfo) {<br>
> MatInfo info;<br>
> ierr = MatGetInfo(mat,MAT_GLOBAL_SUM,&info);CHKERRQ(ierr);<br>
> ierr = PetscViewerASCIIPrintf(viewer,"total: nonzeros=%D, allocated nonzeros=%D\n",(PetscInt)info.nz_used,(PetscInt)info.nz_allocated);CHKERRQ(ierr);<br>
> ierr = PetscViewerASCIIPrintf(viewer,"total number of mallocs used during MatSetValues calls =%D\n",(PetscInt)info.mallocs);CHKERRQ(ierr);<br>
> }<br>
><br>
> My sparse matrix has about 6 billion nonzeros. When I disable FPEs, i get a silent overflow when converting MatInfo.nz_used from PetscLogDouble to (32-bit) PetscInt:<br>
><br>
> Matrix Object: 96 MPI processes<br>
> type: mpiaij<br>
> rows=131857963, cols=18752388<br>
> total: nonzeros=-<a href="tel:2147483648" value="+12147483648">2147483648</a>, allocated nonzeros=0<br>
><br>
> and the code runs just fine. Maybe PETSc should cast nz_used to a long int?<br>
><br>
><br>
> Mihai<br>
> On Thu, Nov 29, 2012 at 6:25 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> On Nov 29, 2012, at 9:48 AM, Mihai Alexe <<a href="mailto:malexe@vt.edu">malexe@vt.edu</a>> wrote:<br>
><br>
> > Hello all,<br>
> ><br>
> > I am creating a large rectangular MPIAIJ matrix, then a shell NormalMatrix that eventually gets passed to a KSP object (all part of a constrained least-squares solver).<br>
> > Code looks as follows:<br>
> ><br>
> > //user.A_mat and user.Hess are PETSc Mat<br>
> ><br>
> > info = MatCreateMPIAIJWithSplitArrays( PETSC_COMM_WORLD, *locrow, *loccol, nrow,<br>
> > *ncol, onrowidx, oncolidx,<br>
> > (PetscScalar*) onvals, offrowidx, offcolidx,<br>
> > (PetscScalar*) values, &user.A_mat ); CHKERRQ(info);<br>
> ><br>
> > info = MatCreateNormal( user.A_mat, &user.Hess ); CHKERRQ(info);<br>
> > info = MatSetUp( user.Hess );<br>
> ><br>
> > Is MatSetUp() required for A or Hess to be initialized correctly? Or some call to MatSetPreallocation?<br>
> '<br>
> No you shouldn't need them. Try with valgrind <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>
><br>
> Barry<br>
><br>
> ><br>
> > My code crashes after displaying (with -info -mat_view_info):<br>
> ><br>
> > [0] PetscCommDuplicate(): Duplicating a communicator 47534399113024 67425648 max tags = <a href="tel:2147483647" value="+12147483647">2147483647</a><br>
> > [0] PetscCommDuplicate(): Duplicating a communicator 47534399112000 67760592 max tags = <a href="tel:2147483647" value="+12147483647">2147483647</a><br>
> > [0] MatCreate_SeqAIJ_Inode(): Not using Inode routines due to -mat_no_inode<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 8920860 X 1508490; storage space: 0 unneeded,34572269 used<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 615<br>
> > Matrix Object: 1 MPI processes<br>
> > type: seqaij<br>
> > rows=8920860, cols=1508490<br>
> > total: nonzeros=34572269, allocated nonzeros=0<br>
> > total number of mallocs used during MatSetValues calls =0<br>
> > not using I-node routines<br>
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 47534399112000 67760592<br>
> > [0] MatCreate_SeqAIJ_Inode(): Not using Inode routines due to -mat_no_inode<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 8920860 X 18752388; storage space: 0 unneeded,1762711 used<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 349<br>
> > Matrix Object: 1 MPI processes<br>
> > type: seqaij<br>
> > rows=8920860, cols=18752388<br>
> > total: nonzeros=1762711, allocated nonzeros=0<br>
> > total number of mallocs used during MatSetValues calls =0<br>
> > not using I-node routines<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 8920860 X 1508490; storage space: 0 unneeded,34572269 used<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 615<br>
> > Matrix Object: 1 MPI processes<br>
> > type: seqaij<br>
> > rows=8920860, cols=1508490<br>
> > total: nonzeros=34572269, allocated nonzeros=0<br>
> > total number of mallocs used during MatSetValues calls =0<br>
> > not using I-node routines<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 8920860 X 18752388; storage space: 0 unneeded,1762711 used<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 349<br>
> > Matrix Object: 1 MPI processes<br>
> > type: seqaij<br>
> > rows=8920860, cols=18752388<br>
> > total: nonzeros=1762711, allocated nonzeros=0<br>
> > total number of mallocs used during MatSetValues calls =0<br>
> > not using I-node routines<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 8920860 X 1508490; storage space: 0 unneeded,34572269 used<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 615<br>
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 47534399112000 67760592<br>
> > [0] PetscCommDuplicate(): Using internal PETSc communicator 47534399112000 67760592<br>
> > [0] VecScatterCreateCommon_PtoS(): Using blocksize 1 scatter<br>
> > [0] VecScatterCreate(): General case: MPI to Seq<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 8920860 X 38109; storage space: 0 unneeded,1762711 used<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 349<br>
> > Matrix Object: 160 MPI processes<br>
> > type: mpiaij<br>
> > rows=131858910, cols=18752388<br>
> ><br>
> > The code ran just fine on a smaller (pruned) input dataset.<br>
> > I don't get a stacktrace unfortunately... (running in production mode, trying to switch to debug mode now).<br>
> ><br>
> ><br>
> > Regards,<br>
> > Mihai<br>
> ><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div>