Slow speed after changing from serial to parallel

[Satish Balay]

On Fri, 18 Apr 2008, Ben Tay wrote:

> Hi,
> 
> I've email my school super computing staff and they told me that the queue which I'm using is one meant for testing, hence, it's
> handling of work load is not good. I've sent my job to another queue and it's run on 4 processors. It's my own code because there seems
> to be something wrong with the server displaying the summary when using -log_summary with ex2f.F. I'm trying it again.

Thats wierd. We should first make sure ex2f [or ex2] are running
properly before looking at your code.

> 
> Anyway comparing just kspsolve between the two, the speedup is about 2.7. However, I noticed that for the 4 processors one, its
> MatAssemblyBegin is  1.5158e+02, which is more than KSPSolve's 4.7041e+00. So is MatAssemblyBegin's time included in KSPSolve? If not,
> does it mean that there's something wrong about my MatAssemblyBegin?

MatAssemblyBegin is not included in KSPSolve(). Something wierd is
going here. There are 2 possibilities.

- whatever code you have before matrix assembly is unbalanced, so
  MatAssemblyBegin() acts as a barrier .

- MPI communication is not optimal within the node.

Its best to first make sure ex2 or ex2f runs fine. As recommended
earlier - you should try latest mpich2 with --with-device=ch3:nemesis:newtcp
and compare ex2/ex2f performance with your current MPI.

Satish