Reuse matrix and vector

[Matthew Knepley]

On Tue, Nov 10, 2009 at 4:51 AM, Jed Brown <jed at 59a2.org> wrote:

> jarunan at ascomp.ch wrote:
> > Total number of cells is 744872, divided into 40 blocks. In one
> > processor, MatCreateMPIAIJWithArrays() takes 0.097 sec but 280 sec with
> > 4 processors. Usually, this routine has no problem with small test case.
> > It works the same for one or more than one processors.
>
> This sounds like incorrect preallocation.  Is your PETSc built with
> debugging?  Debug does some extra integrity checks that don't add
> significantly to the time (although other Debug checks do), but it would
> be useful to know that they pass.  In particular, it checks that your
> rows are sorted.  If they are not sorted then PETSc's preallocation
> would be wrong.  (I actually don't think this requirement enables
> significantly faster implementation, so I'm tempted to change it to work
> correctly with unsorted rows.)
>

I do not think its preallocation per se, since 1 proc is fast. I think that
your
partition of rows fed to the MatCreate() call does not match what you
provide
to MatSetValues() and thus you do a lot of communication in
MatAssemblyEnd().
There are 2 ways to debug this:

  1)  -log_summary to see where the time is spent

  2) MatSetOption(A, *MAT_NEW_NONZERO_LOCATION_ERR)*

  Matt

You can also run with -info |grep malloc, there should be no mallocs in
> MatSetValues().
>
> > in the first iteration.
> >     Mat Ap
> >
> >     call MatCreateMPIAIJWithArrays(PETSC_COMM_WORLD, istorf_no_ovcell,
>       &
> >       istorf_no_ovcell, PETSC_DETERMINE, PETSC_DETERMINE, rowind,
> columnind,   &
> >       A, Ap, ierr)
> >
> >     call MatAssemblyBegin(Ap,MAT_FINAL_ASSEMBLY,ierr)
> >     call MatAssemblyEnd(Ap,MAT_FINAL_ASSEMBLY,ierr)
>
> This assembly is superfluous (but harmless).
>
> > Does the communication of MatCreateMPIAIJWithArrays() in parallel
> > computation cost a lot? What could be the cause that
> > MatCreateMPIAIJWithArrays() so slow in the first iteration?
>
> There is no significant communication, it has to be preallocation.
>
> Jed
>
>


-- 
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20091110/35a51c31/attachment-0001.htm>