[petsc-users] (no subject)

[Salko, Robert K.]

I'm attempting to implement PETSc for the parallel solution of a large sparse matrix in a computer program I work with. The program is written in Fortran. I currently have been able to get the program working using PETSc and giving correct results, but the issue is its performance. It is much slower than solving the matrix in serial. I seem to have narrowed the major PETSc time sink down to actually constructing the matrix (getting the matrix terms into the PETSc matrix). Specifically, it is hanging for a very long time to do this step:

  do values_counter = 1, total_values
     call MatSetValues(A,1,coeff_row(values_counter),1,coeff_col(values_counter),coeff(values_counter),INSERT_VALUES,petsc_err)
  end do

Building the right-hand-side vector for this matrix is much faster:

  call VecSetValues(y,nrows,index_of_values,rhs_values,INSERT_VALUES,petsc_err)

The difference, as I see it, is that I'm sending the matrix coefficients to the PETSc matrix one element at a time whereas with the RHS vector, I'm sending all the values in one shot. There are 7 times more elements in the matrix than the vector, so I get that it will take longer. So I did some timing studies using MPI_Wtime and putting the matrix values into the PETSc matrix is taking 4,800 times longer than putting the values into the RHS vector.

Then there is the actual assembly of the matrix that you have to do using MatAssemblyBegin and MatAssemblyEnd. That takes even longer than assigning values to the matrix. More than twice as long. Actually solving the matrix and then distributing the solution back to the domains in the program takes about as much time as the RHS vector assignment step, so that is nearly 5,000 times faster than putting data into PETSc. Surely there must be something wrong that I'm doing with the assignment of data into PETSc.

So my question is, is there a more efficient way to build the matrix? I already have my coefficients stored in AIJ form in the program. I understand that PETSc uses AIJ. Can't I somehow just send the three AIJ vectors to PETSc in one shot?

Thanks,
Bob