[petsc-users] MatMult scalability problem

[Mark F. Adams]

THings to check are:

1) run on a dedicated machine

2) are the matrices partitioned in the same way for both tests.

3) MatVec is memory bandwidth limited.  14 or 15 speedup on 16 shared memory cores is great.  5 or 6 is bad.  My experience with old IBS SPs was getting a speed of 12 and SPs have a great (and expensive) memory system.  I don't know what state-of-the-art is now but memory bandwidth has been going down generally wrt processor speed so I'd take a look at the number from your code again.

Mark

On May 2, 2012, at 12:01 PM, Javier Fresno wrote:

> 
> 
> I have a very simple Petsc program that multiplies a matrix and a vector several times. It works fine but it has some scalability issues. I execute it in a shared memory machine with 16 processors and it only runs 5 or 6 times faster (only taking into account the MatMult call). I have programmed the same algorithm with C and MPI and it shows a proper speedup (around 14 or 15). The matrices I use have millions of non zero elements, so I think they are big enough.
> 
> What can I do to get the same speedup that in the manual C version?
> 
> I enclose an except of the code. Thank you in advance.
> 
> Javier
> 
> 
> 
> /**
> * Main function
> */
> int main(int argc, char ** argv){
> 
>    // Initialize Petsc
>    PetscInitialize(&argc, &argv, (char *) 0, NULL);
> 
>    // Timers
>    PetscLogDouble t_start, t_end;
> 
>    // File Viewer
>    PetscViewer fd;
>    PetscViewerBinaryOpen(PETSC_COMM_WORLD,"matrix_file",FILE_MODE_READ,&fd);
> 
>    // M matrix
>    Mat M;
>    MatCreate(PETSC_COMM_WORLD,&M);
>    MatSetFromOptions(M);
>    MatLoad(M,fd);
>    PetscViewerDestroy(&fd);
>    MatAssemblyBegin(M,MAT_FINAL_ASSEMBLY);
>    MatAssemblyEnd(M,MAT_FINAL_ASSEMBLY);
> 
>    PetscInt n, m, local_n, local_m;
>    MatGetSize(M,&n,&m);
>    MatGetLocalSize(M,&local_n,&local_m);
> 
>    // b and c vectors
>    Vec b,c;
>    VecCreate(PETSC_COMM_WORLD,&b);
>    VecSetFromOptions(b);
>    VecSetSizes(b,local_n,n);
> 
>    VecCreate(PETSC_COMM_WORLD,&c);
>    VecSetFromOptions(c);
>    VecSetSizes(c,local_n,n);
> 
>    init_vector_values(b);
> 
>    VecAssemblyBegin(b);
>    VecAssemblyEnd(b);
> 
> 
>    // Main computation
>    PetscGetTime(&t_start);
>    int i;
>    for(i=0; i<iter/2; i++){
>        MatMult(M,b,c);
>        MatMult(M,c,b);
>    }
>    PetscGetTime(&t_end);
> 
>    PetscPrintf(PETSC_COMM_WORLD,"Comp time: %lf\n",t_end-t_start);
> 
>    PetscFinalize();
> 
>    return 0;
> }
> 
> 
>