General matrix interative solver

[Satish Balay]

- calling MAT_FINAL_ASSEMBLY before inserting the values is of no use.

- if the code get slow after 200 calls => your preallocation ran out
  at that time. i.e incorrect preallocation.

- You can call MatSetOption(mat,MAT_NEW_NONZERO_ALLOCATION_ERR) to
  verify [If an insertion causes a malloc - the above will flag it as
  error]. You can then run in a debugger [By placing a breakpint in
  PetscError() - and then checking - why the specified preallocation
  info is not sufficient]

- You might want to rethink your MatSetValues code. Eventhough its ok
  right now to have that many calls to MAT_FLUSH_ASSEMBLY, the code
  could probably be organized to avoid this.

- This thread is getting pretty long on the petsc-users mailing list.
  If you still have issues - please continue this discussion at
  petsc-maint at mcs.anl.gov

Satish


On Thu, 12 Oct 2006, Julian wrote:

> Ok, so this is what I am doing now... And I still get the same result. It goes very slowly till the 200th MAT_FLUSH_ASSEMBLY call and then just stays there... The output that I gave u earlier is the complete output its been giving for the following code:
> 
> 	Mat		mat; 
> 	PetscInitialize(0,0,"petscOptions.txt",0);
> 	int i;
> 	int *nnz;
> 	nnz=new int[L];
> 	for(i=0;i<L;i++){
> 		nnz[i]=row[i].getNum();
> 	}
> 
> 	MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, &mat);
> 	delete [] nnz;
> 
> Maybe it is not giving the output you are requiring because the program never gets to the point where MAT_FINAL_ASSEMBLY is called.
> 
> Now, when I do a MAT_FINAL_ASSEMBLY call right after the preallocation, then I get a different output
> 
> 	Mat		mat; 
> 	PetscInitialize(0,0,"petscOptions.txt",0);
> 	int i;
> 	int *nnz;
> 	nnz=new int[L];
> 	for(i=0;i<L;i++){
> 		nnz[i]=row[i].getNum();
> 	}
> 
> 	MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, &mat);
> 
> 	MatAssemblyBegin(mat, MAT_FINAL_ASSEMBLY);	// <-----------------
> 	MatAssemblyEnd(mat, MAT_FINAL_ASSEMBLY);		// <-----------------
> 
> 	delete [] nnz;
> 
> This is what I get :
> [0] PetscInitializePETSc successfully started: number of processors = 1
> [0] PetscGetHostNameRejecting domainname, likely is NIS ALPHA2.ÿÿ0ÿÿ
> [0] PetscInitializeRunning on machine: ALPHA2
> [0] PetscCommDuplicateDuplicating a communicator 1 1 max tags = 100000000
> [0] MatAssemblyEnd_SeqSBAIJMatrix size: 14883 X 14883, block size 1; storage spa
> ce: 1040055 unneeded, 0 used
> [0] MatAssemblyEnd_SeqSBAIJNumber of mallocs during MatSetValues is 0
> [0] MatAssemblyEnd_SeqSBAIJMost nonzeros blocks in any row is 0
> 
> In this case, the assembly goes very fast TILL the 200th MAT_FLUSH_ASSEMBLY call and THEN gets stuck!
> 
> 
> 
> > -----Original Message-----
> > From: owner-petsc-users at mcs.anl.gov 
> > [mailto:owner-petsc-users at mcs.anl.gov] On Behalf Of Satish Balay
> > Sent: Thursday, October 12, 2006 6:46 PM
> > To: petsc-users at mcs.anl.gov
> > Subject: RE: General matrix interative solver
> > 
> > On Thu, 12 Oct 2006, Julian wrote:
> > 
> > > I am now using MatCreateSeqSBAIJ() - I am assuming the second 
> > > parameter bs=1 in my case because I am using one processor. Is that 
> > > correct ?
> > 
> > No - it refers to a blocksize. The matrix could have a block 
> > structure.  In this case - the correct blocksize can be 
> > specified. The matrix operations will exploit this block 
> > structure for better performance.
> > 
> > bs=1 is a fine [This is the equivalent of AIJ type in the 
> > block notation]
> > 
> > > The method of creating a matrix I used was taken from Pavel Solin's 
> > > book 'Partial Differential Equations and the finite Element 
> > Method'. 
> > > Is there any particular reason why is should follow this
> > > method:
> > > mat = (double*)malloc(sizeof(Mat))
> > > MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, nnz, 
> > > (Mat*)mat); In the examples that I have seen in the petsc website I 
> > > don't see this statement where memory is allocated.
> > 
> > No - you should just use PETSc example codes a guidelines. i.e
> > 
> >    Mat mat;
> >    MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, PETSC_DEFAULT, 
> > nnz, &mat);
> > 
> > Mat is just a pointer - and all the memory allocation 
> > required is done inside XXXCreate() routines.
> > 
> > > Anyway, I changed my code to the following:
> > > 	PetscInitialize(0,0,"petscOptions.txt",0);
> > > 
> > > 	int i;
> > > 	int *nnz;
> > > 	nnz=new int[L];
> > > 	for(i=0;i<L;i++){
> > > 		nnz[i]=row[i].getNum();
> > > 	}
> > > 
> > > 	Assert( mat = (double*)malloc(sizeof(Mat)) );
> > > 	MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, 
> > PETSC_DEFAULT, nnz, (Mat*)mat);
> > > 	delete [] nnz;
> > 
> > > And I'm still getting stuck at the same place... There is a 
> > difference 
> > > though.. Now the assembly process is even slower than before (when 
> > > there was no preallocation) I noticed that it is getting 
> > stuck around 
> > > the 200th MAT_FLUSH_ASSEMBLY call.. I don't know if that is 
> > > significant.Is there some limit on how many times 
> > MAT_FLUSH_ASSEMBLY 
> > > can be called ?
> > 
> > MAT_FLUSH_ASSEMBLY is cheap - in sequential mode - but 
> > expensive in parallel mode [becasue of parallel 
> > syncronization]. However ...
> > 
> > > This is the petsc output I get:
> > > [0] PetscInitializePETSc successfully started: number of 
> > processors = 
> > > 1 [0] PetscGetHostNameRejecting domainname, likely is NIS 
> > ALPHA2.ÿÿ0ÿÿ 
> > > [0] PetscInitializeRunning on machine: ALPHA2 [0] 
> > > PetscCommDuplicateDuplicating a communicator 1 1 max tags = 
> > 100000000
> > 
> > The above log is incomplete. You should be looking for lines 
> > with the string 'malloc'
> > 
> > Satish
> > 
> > > 
> > > I also tried this :
> > > 	mat = new Mat;
> > > 	MatCreateSeqSBAIJ(PETSC_COMM_SELF, 1, L, L, 
> > PETSC_DEFAULT, nnz, mat);
> > > 
> > > And got the same performance.
> > 
> 
>