[petsc-users] Slow MatAssemblyBegin MAT_FINAL_ASSEMBLY

[Jose A. Abell M.]

Thank you all! I will check with a smaller example to see how my allocation
is doing.

--

José Abell
*PhD Candidate*
Computational Geomechanics Group
Dept. of Civil and Environmental Engineering
UC Davis
www.joseabell.com


On Thu, Dec 17, 2015 at 10:59 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Dec 17, 2015, at 11:12 AM, Dave May <dave.mayhem23 at gmail.com> wrote:
> >
> >
> >
> > On Thursday, 17 December 2015, Jose A. Abell M. <jaabell at ucdavis.edu>
> wrote:
> > Thank you Dave!
> >
> > Do you have a rough idea of how long a matrix like that should take to
> assemble?
> > Not hours. Right?
> >
> > If the preallocation is correct, and most of the entries to be inserted
> live locally (and don't need to be scattered to another rank), it should
> definitely not take hours.
>
>    If the preallocation is correct then the time to do the first assembly
> may take maybe twice as long as subsequent assemblies but it should not be
> much more than that.
>
>   Barry
>
> >
> >
> >
> > Regards,
> > Jose
> >
> > --
> >
> > José Abell
> > PhD Candidate
> > Computational Geomechanics Group
> > Dept. of Civil and Environmental Engineering
> > UC Davis
> > www.joseabell.com
> >
> >
> > On Thu, Dec 17, 2015 at 12:58 AM, Dave May <dave.mayhem23 at gmail.com>
> wrote:
> >
> >
> > On 17 December 2015 at 08:06, Jose A. Abell M. <jaabell at ucdavis.edu>
> wrote:
> > Hello dear PETSc users,
> >
> > This is a problem that pops up often, from what I see, in the mailing
> list. My program takes a long time assembling the matrix.
> >
> > What I know:
> >
> >       • Matrix Size is (MatMPIAIJ) 2670402
> >       • Number of processes running PETSc: 95
> >       • Not going to virtual memory (no swapping, used mem well withing
> each node's capacity)
> >       • System is partitioned with ParMETIS for load balancing
> >       • I see memory moving around in each node (total used memory
> changes a bit, grows and then frees)
> >       • Matrix is filled in blocks of size 81x81 (FEM code, so this ends
> up being a sparse matrix)
> >       • I don't do flushes at all. Only MAT_FINAL_ASSEMBLY when all the
> MatSetValues are done.
> > Should I do MAT_FLUSH_ASSEMBLY even though I have enough memory to store
> the buffers? If so, how often? Every 100 blocks?
> >
> > What else could it be?
> >
> > Its taking several hours to asseble this matrix. I re-use the sparsity
> pattern, so subsequent assemblies are fast. Does this mean that my
> preallocation is wrong?
> >
> > The preallocation could be wrong. That is the usual cause of very slow
> matrix assembly. To confirm this hypothesis, run your code with the command
> line option -info. You will get an enormous amount of information in
> stdout. You might consider using -info with a smallish problem size / core
> count.
> >
> > Inspect the output generated by -info and look for lines like this:
> >
> > [1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
> >
> > If the number of mallocs during MatSetValues() is not zero, then your
> preallocation is not exactly correct. A small number of mallocs, say less
> than 10, might be accepted (performance wise). However if the number of
> mallocs is > 100, then assembly time will be terribly slow.
> >
> > Thanks,
> >   Dave
> >
> >
> >
> >
> > Regards,
> >
> >
> >
> >
> > --
> >
> > José Abell
> > PhD Candidate
> > Computational Geomechanics Group
> > Dept. of Civil and Environmental Engineering
> > UC Davis
> > www.joseabell.com
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151217/2b7b4385/attachment.html>