Slow assembly

[John R. Wicks]

What happened was that I used MatDiagonalSet and then used MatSetValues and
MatAssemblyEnd, but the log shows that MatAssemblyEnd was called twice;
MatDiagonal must force an implicit MatAssemblyBegin/End.  By avoiding the
call to MatDiagonalSet, everything worked fine (in particular, that shows
that my preallocations were correct, as was).

> -----Original Message-----
> From: owner-petsc-users at mcs.anl.gov 
> [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Matthew Knepley
> Sent: Sunday, October 28, 2007 4:29 PM
> To: petsc-users at mcs.anl.gov
> Subject: Re: Slow assembly
> 
> 
> Perhaps this is a miscalculation for parallel allocation, 
> meaning you allocate  the correct number of values, but do 
> not divide them correctly between the diagonal and offdiagonal parts.
> 
>   Matt
> 
> On 10/28/07, John R. Wicks <jwicks at cs.brown.edu> wrote:
> > I ran on a smaller example with the -info switch set.  I 
> preallocated 
> > all my matrices (there are two of them, one sequential and one 
> > distributed) so that there should be no allocs needed, but 
> I notice it 
> > reports: something like: [3] MatAssemblyEnd_SeqAIJ(): 
> Matrix size: 11 
> > X 11; storage space: 10 unneeded,11 used [3] 
> MatAssemblyEnd_SeqAIJ(): 
> > Number of mallocs during MatSetValues() is 0 [3] 
> > MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1 [1] 
> > MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 13 
> > unneeded,10 used [1] MatAssemblyEnd_SeqAIJ(): Number of 
> mallocs during 
> > MatSetValues() is 0 [1] MatAssemblyEnd_SeqAIJ(): Maximum 
> nonzeros in 
> > any row is 1 [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; 
> > storage space: 22 unneeded,11 used
> > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during 
> MatSetValues() is 0
> > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
> > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 9
> > unneeded,10 used
> > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during 
> MatSetValues() is 0
> > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
> > [4] MatAssemblyEnd_SeqAIJ(): Matrix size: 9 X 9; storage space: 13
> > unneeded,9 used
> > [4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during 
> MatSetValues() is 0
> > [4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage space: 19
> > unneeded,11 used
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during 
> MatSetValues() is 0
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
> > [6] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 17
> > unneeded,10 used
> > [6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during 
> MatSetValues() is 0
> > [6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
> > [2] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage 
> space: 115
> > unneeded,31 used
> > [2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during 
> MatSetValues() is 9
> > [2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 5
> > [5] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 67
> > unneeded,18 used
> > [5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during 
> MatSetValues() is 5
> > [5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3
> > [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage space: 88
> > unneeded,28 used
> > [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during 
> MatSetValues() is 7
> > [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 5
> >
> > I can't figure why it should allocate anything, b/c I've 
> precomputed 
> > the number of entries in each row.
> >
> > > -----Original Message-----
> > > From: owner-petsc-users at mcs.anl.gov 
> > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith
> > > Sent: Friday, October 26, 2007 7:54 PM
> > > To: petsc-users at mcs.anl.gov
> > > Subject: RE: Slow assembly
> > >
> > >
> > >
> > >   The sorting does not matter.
> > >
> > >   Under normal conditions the MatAssembly should take a 
> fraction of 
> > > a second. The only cause that we know that slows it down to the 
> > > extreme you have is that it is sending a huge amount of 
> data across 
> > > processes (the -info option Satish suggested will tell us 
> if that is 
> > > true).
> > >
> > >   Are you only call MatAssemblyBegin/End() once? You 
> should, don't 
> > > call it multiple times.
> > >
> > >   The sorting is not important (in fact it takes advantage of it 
> > > automatically and does not need to be set).
> > >
> > >    Barry
> > >
> > >
> > > On Fri, 26 Oct 2007, John R. Wicks wrote:
> > >
> > > > I have confirmed that I am calling MatSetValues() for local
> > > rows only
> > > > and am only setting each value exactly once.
> > > >
> > > > Because of how the matrix was partitioned for another non-Petsc 
> > > > program, each partition is partitioned (by columns) 
> into 32 blocks 
> > > > (corresponding to the row partitions).  I enter the 
> data for each 
> > > > block one row at a time, i.e., for any one SetValues call,
> > > the entries
> > > > are sorted by increasing column index.  Does that mean 
> I can use 
> > > > MatrixSetOption(A,MAT_COLUMNS_SORTED).  Should that help?
> > > >
> > > > P.S.: I tried it, and it still seems to be taking quite a long 
> > > > time.
> > > >
> > > > > -----Original Message-----
> > > > > From: owner-petsc-users at mcs.anl.gov 
> > > > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of 
> Satish Balay
> > > > > Sent: Friday, October 26, 2007 3:04 PM
> > > > > To: petsc-users at mcs.anl.gov
> > > > > Subject: Re: Slow assembly
> > > > >
> > > > >
> > > > > On Fri, 26 Oct 2007, John R. Wicks wrote:
> > > > >
> > > > > > I am working on computing PageRank for a web scale graph
> > > > > which uses a
> > > > > > square matrix which is 1.2x10^8 dimensional with about 10^9
> > > > > entries.
> > > > > > I have partitioned the matrix for 32 processors myself
> > > into my own
> > > > > > ascii format, and I know the memory allocation, so I:
> > > > > >
> > > > > > 1) create the matrix with "A = MatCreateMPIAIJ(*n, *n, *N,
> > > > > *N, 0, nnz,
> > > > > > 0, onnz)",
> > > > > > 2) load the entries by repeatedly calling 
> > > > > > "MatSetValues(A,1,&row,links,cols,vals,INSERT_VALUES)", and
> > > > > >
> > > > > > 3) call MatAssemblyBegin/End.
> > > > > >
> > > > > > Steps 1 and 2 complete in a couple minutes, but step 3 is 
> > > > > > taking several hours.  What is going on?  Is there a way to 
> > > > > > speed
> > > > > up matrix
> > > > > > assembly?
> > > > >
> > > > > Are you makeing sure that you call 
> MatGetOwnershipRange() - and 
> > > > > calling MatSetValues() for mostly local rows only?
> > > > >
> > > > > Also can you confirm that multiple processes [for eg: 
> proc-0 and 
> > > > > proc-1 etc..]  are not setting the same value [i.e 
> both of them 
> > > > > calling MatSetValues(row=0,col=0)]
> > > > >
> > > > > Satish
> > > > >
> > > >
> > > >
> > >
> >
> >
> 
> 
> -- 
> What most experimenters take for granted before they begin 
> their experiments is infinitely more interesting than any 
> results to which their experiments lead.
> -- Norbert Wiener
>