<div dir="ltr">Oh, I See Fande's thread now. So you extended this.</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 11, 2017 at 2:52 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_679072094062580012gmail-"><br>
</span>Must be the same place Fande encountered this error.<br>
<br></blockquote><div><br></div></span><div>Satish, here is our stack trace. Do you have any thoughts vis-a-vis Fande's experience what the problem is here?  My best guess is that the local matrix size is huge, but this code can have some very large stencils.</div><div><br></div><div>Treb: about how many processors have this error? Is is it just one, a few, or tons?</div><div><br></div><div><span class=""><span class="m_679072094062580012gmail-im" style="font-size:12.8px">[5]PETSC ERROR: Argument out of range<br></span></span><span style="font-size:12.8px">[5]PETSC ERROR: A really huge hash is being requested.. cannot process: 67108864</span><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: See </span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" style="font-size:12.8px" target="_blank">http://www.mcs.anl.gov/<wbr>petsc/documentation/faq.html</a><span style="font-size:12.8px"> <wbr>for trouble shooting.</span><span class=""><span class="m_679072094062580012gmail-im" style="font-size:12.8px"><br>[5]PETSC ERROR: Petsc Development GIT revision: v3.7.4-2584-g7175242 GIT Date: 2016-12-21 13:19:16 -0600<br></span></span><span class=""><span style="font-size:12.8px">[5]PETSC ERROR: /tmp/targetExec on a arch-xc30-dbg64-intel named nid00013 by u6338 Sat Jan  7 18:35:34 2017</span><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: Configure options --COPTFLAGS="-O0 -g -no-ipo" --CXXOPTFLAGS="-O0 -g -no-ipo" --FOPTFLAGS="-O0 -g -ipo" --download-parmetis --download-metis --download-hypre --with-ssl=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 --with-x=0 --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices --prefix=/global/homes/m/madam</span><span style="font-size:12.8px"><wbr>s/petsc_install/petscv3.7-xc3[</span><span style="font-size:12.8px"><wbr>1983]PETSC ERROR: --------------------- Error Message ------------------------------</span><span style="font-size:12.8px"><wbr>------------------------------</span><span style="font-size:12.8px"><wbr>--</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: [522]PETSC ERROR: --------------------- Error Message ----------</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: #2 PetscTableCreate() line 52 in /global/u2/m/madams/petsc_inst</span><span style="font-size:12.8px"><wbr>all/petsc/src/sys/utils/ctable</span><span style="font-size:12.8px"><wbr>.c</span><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: #3 MatTransposeMatMultSymbolic_MP</span><span style="font-size:12.8px"><wbr>IAIJ_MPIAIJ() line 1691 in /global/u2/m/madams/petsc_inst</span><span style="font-size:12.8px"><wbr>all/petsc/src/mat/impls/aij/</span><span style="font-size:12.8px">mp<wbr>i/mpimatmatmult.c</span><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: #4 MatTransposeMatMult_MPIAIJ_MPI</span><span style="font-size:12.8px"><wbr>AIJ() line 917 in /global/u2/m/madams/petsc_inst</span><span style="font-size:12.8px"><wbr>all/petsc/src/mat/impls/aij/</span><span style="font-size:12.8px">mp<wbr>i/mpimatmatmult.c</span><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: [608]PETSC ERROR: --------------------- Error Message ----------</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: #6 PCGAMGCoarsen_AGG() line 955 in /global/u2/m/madams/petsc_inst</span><span style="font-size:12.8px"><wbr>all/petsc/src/ksp/pc/impls/</span><span style="font-size:12.8px">gam<wbr>g/agg.c</span><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: #7 PCSetUp_GAMG() line 524 in /global/u2/m/madams/petsc_inst</span><span style="font-size:12.8px"><wbr>all/petsc/src/ksp/pc/impls/</span><span style="font-size:12.8px">gam<wbr>g/gamg.c</span><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: #8 PCSetUp() line 968 in /global/u2/m/madams/petsc_inst</span><span style="font-size:12.8px"><wbr>all/petsc/src/ksp/pc/interface</span><span style="font-size:12.8px"><wbr>/precon.c</span><br style="font-size:12.8px"><span style="font-size:12.8px">[5]PETSC ERROR: #9 KSPSetUp() line 393 in /global/u2/m/madams/petsc_inst</span><span style="font-size:12.8px"><wbr>all/petsc/src/ksp/ksp/interfac</span><span style="font-size:12.8px"><wbr>e/itfunc.c</span><br style="font-size:12.8px"></span><span style="font-size:12.8px">[5]PETSC ERROR: [1640]PETSC ERROR: --------------------- Error Message -----</span><br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
MatSetUpMultiply_MPIAIJ() does:<br>
<br>
  ierr = PetscTableCreate(aij->B->rmap-<wbr>>n,mat->cmap->N+1,&gid1_lid1);<wbr>CHKERRQ(ierr);<br>
<br>
So the local matrix size (n) must be: 4294967296<br>
<br>
[assuming atleast one nonzero per row - the local matrix itself must use<br>
more memory than the hash-table..]<br>
<span class="m_679072094062580012gmail-HOEnZb"><font color="#888888"><br>
Satish<br>
</font></span></blockquote></span></div><br></div></div>
</blockquote></div><br></div>