[petsc-users] MatAssemblyEnd taking too long

[Matthew Knepley]

On Wed, Aug 19, 2020 at 8:55 PM Jed Brown <jed at jedbrown.org> wrote:

> Manav Bhatia <bhatiamanav at gmail.com> writes:
>
> > Thanks for the followup, Jed.
> >
> >> On Aug 19, 2020, at 7:42 PM, Jed Brown <jed at jedbrown.org> wrote:
> >>
> >> Can you share a couple example stack traces from that debugging?
> >
> > Do you mean a similar screenshot at different system sizes? Or a
> different format?
>
> Sorry, I missed the screenshots (they were tucked away in the text/html
> and I was reading the text/plain version of your message).
>
> >> About how many nonzeros per row?
> >
> > This is a 3D elasticity run with Hex8 elements. So, each row has 81
> non-zero entries, although I have not verified that (I will do so now). Is
> there a command line argument that will print this for the matrix?
> Although, on second thought that will not be printed unless the Assembly
> routine has finished.
>
> You could run a smaller problem size with -snes_view, which would show
> matrix stats.
>
> Can you try running with -matstash_legacy?
>
> What version of Open MPI is this?
>

Jed is more knowledgeable about the communication, but I have a simple
question about the FEM method. Normally, the way
we divide unknowns is that the only unknowns which might have entries
computed off-process are those on the partition boundary.
However, it sounds like you have a huge number of communicated values. Is
it possible that the division of rows in your matrix does
not match the division of the cells you compute element matrices for?

  Thanks,

     Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/ae6ce60c/attachment.html>