Slow assembly

[John R. Wicks]

I ran on a smaller example with the -info switch set.  I preallocated all my
matrices (there are two of them, one sequential and one distributed) so that
there should be no allocs needed, but I notice it reports: something like:
[3] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage space: 10
unneeded,11 used
[3] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[3] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[1] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 13
unneeded,10 used
[1] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[1] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[2] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage space: 22
unneeded,11 used
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[5] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 9
unneeded,10 used
[5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[4] MatAssemblyEnd_SeqAIJ(): Matrix size: 9 X 9; storage space: 13
unneeded,9 used
[4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage space: 19
unneeded,11 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[6] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 17
unneeded,10 used
[6] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[6] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1
[2] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage space: 115
unneeded,31 used
[2] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 9
[2] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 5
[5] MatAssemblyEnd_SeqAIJ(): Matrix size: 10 X 10; storage space: 67
unneeded,18 used
[5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 5
[5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 3
[0] MatAssemblyEnd_SeqAIJ(): Matrix size: 11 X 11; storage space: 88
unneeded,28 used
[0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 7
[0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 5

I can't figure why it should allocate anything, b/c I've precomputed the
number of entries in each row.

> -----Original Message-----
> From: owner-petsc-users at mcs.anl.gov 
> [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Barry Smith
> Sent: Friday, October 26, 2007 7:54 PM
> To: petsc-users at mcs.anl.gov
> Subject: RE: Slow assembly
> 
> 
> 
>   The sorting does not matter.
> 
>   Under normal conditions the MatAssembly should take a 
> fraction of a second. The only cause that we know that slows 
> it down to the extreme you have is that it is sending a huge 
> amount of data across processes (the -info option Satish 
> suggested will tell us if that is true). 
> 
>   Are you only call MatAssemblyBegin/End() once? You should, 
> don't call it 
> multiple times. 
> 
>   The sorting is not important (in fact it takes advantage of 
> it automatically and does not need to be set).
> 
>    Barry
> 
> 
> On Fri, 26 Oct 2007, John R. Wicks wrote:
> 
> > I have confirmed that I am calling MatSetValues() for local 
> rows only 
> > and am only setting each value exactly once.
> > 
> > Because of how the matrix was partitioned for another non-Petsc 
> > program, each partition is partitioned (by columns) into 32 blocks 
> > (corresponding to the row partitions).  I enter the data for each 
> > block one row at a time, i.e., for any one SetValues call, 
> the entries 
> > are sorted by increasing column index.  Does that mean I can use 
> > MatrixSetOption(A,MAT_COLUMNS_SORTED).  Should that help?
> > 
> > P.S.: I tried it, and it still seems to be taking quite a long time.
> > 
> > > -----Original Message-----
> > > From: owner-petsc-users at mcs.anl.gov
> > > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay
> > > Sent: Friday, October 26, 2007 3:04 PM
> > > To: petsc-users at mcs.anl.gov
> > > Subject: Re: Slow assembly
> > > 
> > > 
> > > On Fri, 26 Oct 2007, John R. Wicks wrote:
> > > 
> > > > I am working on computing PageRank for a web scale graph
> > > which uses a
> > > > square matrix which is 1.2x10^8 dimensional with about 10^9
> > > entries.
> > > > I have partitioned the matrix for 32 processors myself 
> into my own
> > > > ascii format, and I know the memory allocation, so I:
> > > > 
> > > > 1) create the matrix with "A = MatCreateMPIAIJ(*n, *n, *N,
> > > *N, 0, nnz,
> > > > 0, onnz)",
> > > > 2) load the entries by repeatedly calling
> > > > "MatSetValues(A,1,&row,links,cols,vals,INSERT_VALUES)", and
> > > > 
> > > > 3) call MatAssemblyBegin/End.
> > > > 
> > > > Steps 1 and 2 complete in a couple minutes, but step 3 is taking
> > > > several hours.  What is going on?  Is there a way to speed 
> > > up matrix
> > > > assembly?
> > > 
> > > Are you makeing sure that you call MatGetOwnershipRange() -
> > > and calling MatSetValues() for mostly local rows only?
> > > 
> > > Also can you confirm that multiple processes [for eg: proc-0
> > > and proc-1 etc..]  are not setting the same value [i.e both 
> > > of them calling MatSetValues(row=0,col=0)]
> > > 
> > > Satish
> > > 
> > 
> > 
>