[petsc-users] MatAssembly Cost Jump

[Ali Berk Kahraman]

Dear All,

I have a sequential algorithm to determine the nonzero structure of my 
Jacobian on my unstructured grid. The overall code goes like following,

1.Do other MPI stuff regarding the problem, irrelevant of this mail
2. If you are rank 0, set all of the nonzeros of the MPI Jacobian Matrix
3. MatAssemblyBegin/End, send the nonzero locations to other processes 
from rank 0

I have been testing the algorithm to see how long it takes on a 
structured grid for which I know the Jacobian non-zero structure. Here 
is where I have a question. For a square grid, 129x129 nodes making 
16641x16641 Jacobian Matrix, having 16641x13 nonzeros in it, step 2 
takes 1 second and step 3 takes 2 seconds. For a square grid, 257x257 
nodes making a 66049x66049 Jacobian matrix, having 66049x13 nonzeros in 
it, step 2 takes 9 seconds, step 3 takes around 50 seconds.

My question is, why does the time get multiplied by a factor of 25 while 
the data that should be transferred only quadruples in step 3. Is this 
expected behavior?

Best Regards,

Ali Berk Kahraman
M.Sc. Student, Mechanical Engineering
Boğaziçi Uni, Istanbul, Turkey