assembly

[Thomas Geenen]

i call 
ierr = MatStashSetInitialSize(A[*seqsolve],stash_size, 
stash_size);CHKERRQ(ierr);
with 100 000 000 for the stash size to make sure that's not the bottleneck

the assemble time remains unchanged however.

nstash in MatAssemblyBegin_MPIAIJ (CPU=0) = 109485 
reallocs in MatAssemblyBegin_MPIAIJ =  0

cheers
Thomas

On Saturday 02 February 2008 23:19, Barry Smith wrote:
>     The matstash has a concept of preallocation also. During the first
> setvalues
> it is allocating more and more memory for the stash. In the second
> setvalues
> the stash is large enough so does not require any addition allocation.
>
>     You can use the option -matstash_initial_size <size> to allocate
> enough space
> initially so that the first setvalues is also fast. It does not look
> like there is a way
> coded to get the <size> that you should use. It should be set to the
> maximum nonzeros
> any process has that belongs to other processes. The stash handling
> code is
> in src/mat/utils/matstash.c, perhaps you can figure out how to
> printout with PetscInfo()
> the sizes needed?
>
>
>     Barry
>
> On Feb 2, 2008, at 12:30 PM, Thomas Geenen wrote:
> > On Saturday 02 February 2008 18:33, Hong Zhang wrote:
> >> On Sat, 2 Feb 2008, Thomas Geenen wrote:
> >>> Dear Petsc users,
> >>>
> >>> I would like to understand what is slowing down the assembly phase
> >>> of my
> >>> matrix. I create a matrix with MatCreateMPIAIJ i make a rough
> >>> guess of
> >>> the number of off diagonal entries and then use a conservative
> >>> value to
> >>> make sure I do not need extra mallocs. (the number of diagonal
> >>> entries is
> >>> exact)
> >>> next i call MatSetValues and MatAssemblyBegin, MatAssemblyEnd.
> >>> The first time i call MatSetValues and MatAssemblyBegin,
> >>> MatAssemblyEnd it takes about 170 seconds
> >>> the second time 0.3 seconds.
> >>> I run it on 6 cpu's and I do fill quit a number of row-entries on
> >>> the
> >>> "wrong" cpu. However thats also the case the second run. I checked
> >>> that there are no additional mallocs
> >>> MatGetInfo info.mallocs=0 both after MatSetValues and after
> >>> MatAssemblyBegin, MatAssemblyEnd.
> >>
> >> Run your code with the option '-log_summary' and check which function
> >> call dominates the execution time.
> >
> > the time is spend in MatStashScatterGetMesg_Private
> >
> >>> I run it on 6 cpu's and I do fill quit a number of row-entries on
> >>> the
> >>> "wrong" cpu.
> >>
> >> Likely, the communication that sending the entries to the
> >> corrected cpu consume the time. Can you fill the entries in the
> >> correct cpu?
> >
> > the second time the entries are filled on the wrong CPU as well.
> > i am curious about the difference in time between run 1 and 2.
> >
> >> Hong
> >>
> >>> cheers
> >>> Thomas