[petsc-users] matcreate and assembly issue

[Karl Lin]

After recompiling with 64bit option, the program ran successfully. Thank
you very much for the insight.

On Thu, Jun 11, 2020 at 12:00 PM Satish Balay <balay at mcs.anl.gov> wrote:

> On Thu, 11 Jun 2020, Karl Lin wrote:
>
> > Hi, Matthew
> >
> > Thanks for the suggestion, just did another run and here are some
> detailed
> > stack traces, maybe will provide some more insight:
> >  *** Process received signal ***
> > Signal: Aborted (6)
> > Signal code:  (-6)
> > /lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0]
> >  [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337]
> >  [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28]
> >  [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4]
> >  [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65]
> >  [ 5]
> /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0]
> >  [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17]
> >  [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164]
> >  [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6]
> >  [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda]
> >
> > By reconfiguring, you mean recompiling petsc with that option, correct?
>
> yes. you can use a different PETSC_ARCH for this build - so that both
> builds are usable [by just switching PETSC_ARCH from your appliation
> makefile]
>
> Satish
>
> >
> > Thank you.
> >
> > Karl
> >
> > On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley <knepley at gmail.com>
> wrote:
> >
> > > On Thu, Jun 11, 2020 at 11:51 AM Karl Lin <karl.linkui at gmail.com>
> wrote:
> > >
> > >> Hi, there
> > >>
> > >> We have written a program using Petsc to solve large sparse matrix
> > >> system. It has been working fine for a while. Recently we encountered
> a
> > >> problem when the size of the sparse matrix is larger than 10TB. We
> used
> > >> several hundred nodes and 2200 processes. The program always crashes
> during
> > >> MatAssemblyBegin.Upon a closer look, there seems to be something
> unusual.
> > >> We have a little memory check during loading the matrix to keep track
> of
> > >> rss. The printout of rss in the log shows normal increase up to rank
> 2160,
> > >> i.e., if we load in a portion of matrix that is 1GB, after
> MatSetValues for
> > >> that portion, rss will increase roughly about that number. From rank
> 2161
> > >> onwards, the rss in every rank doesn't increase after matrix loaded.
> Then
> > >> comes MatAssemblyBegin, the program crashed on rank 2160.
> > >>
> > >> Is there a upper limit on the number of processes Petsc can handle?
> or is
> > >> there a upper limit in terms of the size of the matrix petsc can
> handle?
> > >> Thank you very much for any info.
> > >>
> > >
> > > It sounds like you overflowed int somewhere. We try and check for this,
> > > but catching every place is hard. Try reconfiguring with
> > >
> > >   --with-64-bit-indices
> > >
> > >   Thanks,
> > >
> > >      Matt
> > >
> > >
> > >> Regards,
> > >>
> > >> Karl
> > >>
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to which
> their
> > > experiments lead.
> > > -- Norbert Wiener
> > >
> > > https://www.cse.buffalo.edu/~knepley/
> > > <http://www.cse.buffalo.edu/~knepley/>
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200611/7ffa884d/attachment.html>