[petsc-users] Very poor speed up performance

[Yongjun Chen]

Hi everyone,


I use PETSC (version 3.1-p5) to solve a linear problem Ax=b. The matrix A
and right hand vector b are read from files. The dimension of A is
1.2Million*1.2Million. I am pretty sure the matrix A and vector b have been
read correctly.

I compiled the program with optimized version (--with-debugging=0), tested
the speed up performance on two servers, and I have found that the
performance is very poor.

For the two servers, one is 4 cpus * 4 cores per cpu, i.e., with a total 16
cores. And the other one is 4 cpus * 12 cores per cpu, with a total 48
cores.

On each of them, with the increasing of computing cores k from 1 to 8
(mpiexec –n  k ./Solver_MPI -pc_type jacobi -ksp-type gmres), the speed up
will increase from 1 to 6, but when the computing cores k increase from 9 to
16(for the first server) or 48 (for the second server), the speed up
decrease firstly and then remains a constant value 5.0 (for the first
server) or 4.5(for the second server).

Actually, the program LAMMPS speed up excellently on these two servers.

Any comments are very appreciated! Thanks!



--------------------------------------------------------------------------------------------------------------------------

PS: the related codes are as following,


//firstly read A and b from files

...

//then



              ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);

              ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr);

              ierr = VecAssemblyBegin(b); CHKERRQ(ierr);

              ierr = VecAssemblyEnd(b); CHKERRQ(ierr);



              ierr = MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE);
CHKERRQ(ierr);

              ierr = MatGetRowUpperTriangular(A); CHKERRQ(ierr);

              ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);



              ierr =
KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);

              ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);

              ierr =
KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr);

              ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);



              ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr);



              ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr);



              ierr = KSPGetSolution(ksp, &x);CHKERRQ(ierr);



              ierr = VecAssemblyBegin(x);CHKERRQ(ierr);

              ierr = VecAssemblyEnd(x);CHKERRQ(ierr);

...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20101220/f4679d9f/attachment.htm>