[petsc-users] MatMult scalability problem

[Javier Fresno]

I have a very simple Petsc program that multiplies a matrix and a vector 
several times. It works fine but it has some scalability issues. I 
execute it in a shared memory machine with 16 processors and it only 
runs 5 or 6 times faster (only taking into account the MatMult call). I 
have programmed the same algorithm with C and MPI and it shows a proper 
speedup (around 14 or 15). The matrices I use have millions of non zero 
elements, so I think they are big enough.

What can I do to get the same speedup that in the manual C version?

I enclose an except of the code. Thank you in advance.

Javier



/**
  * Main function
  */
int main(int argc, char ** argv){

     // Initialize Petsc
     PetscInitialize(&argc, &argv, (char *) 0, NULL);

     // Timers
     PetscLogDouble t_start, t_end;

     // File Viewer
     PetscViewer fd;
    
  PetscViewerBinaryOpen(PETSC_COMM_WORLD,"matrix_file",FILE_MODE_READ,&fd);

     // M matrix
     Mat M;
     MatCreate(PETSC_COMM_WORLD,&M);
     MatSetFromOptions(M);
     MatLoad(M,fd);
     PetscViewerDestroy(&fd);
     MatAssemblyBegin(M,MAT_FINAL_ASSEMBLY);
     MatAssemblyEnd(M,MAT_FINAL_ASSEMBLY);

     PetscInt n, m, local_n, local_m;
     MatGetSize(M,&n,&m);
     MatGetLocalSize(M,&local_n,&local_m);

     // b and c vectors
     Vec b,c;
     VecCreate(PETSC_COMM_WORLD,&b);
     VecSetFromOptions(b);
     VecSetSizes(b,local_n,n);

     VecCreate(PETSC_COMM_WORLD,&c);
     VecSetFromOptions(c);
     VecSetSizes(c,local_n,n);

     init_vector_values(b);

     VecAssemblyBegin(b);
     VecAssemblyEnd(b);


     // Main computation
     PetscGetTime(&t_start);
     int i;
     for(i=0; i<iter/2; i++){
         MatMult(M,b,c);
         MatMult(M,c,b);
     }
     PetscGetTime(&t_end);

     PetscPrintf(PETSC_COMM_WORLD,"Comp time: %lf\n",t_end-t_start);

     PetscFinalize();

     return 0;
}