Slow speed after changing from serial to parallel (with ex2f.F)

[Matthew Knepley]

On Wed, Apr 16, 2008 at 8:44 AM, Ben Tay <zonexo at gmail.com> wrote:
> Hi,
>
>  Am I right to say that despite all the hype about multi-core processors,
> they can't speed up solving of linear eqns? It's not possible to get a 2x
> speedup when using 2 cores. And is this true for all types of linear
> equation solver besides PETSc? What about parallel direct solvers (e.g.
> MUMPS) or those which uses openmp instead of mpich? Well, I just can't help
> feeling disappointed if that's the case...

Notice that Satish got much much better scaling than you did on our box here.
I think something is really wrong either with the installation of MPI
on that box
or something hardware-wise.

  Matt

>  Also, with a smart enough LSF scheduler, I will be assured of getting
> separate processors ie 1 core from each different processor instead of 2-4
> cores from just 1 processor. In that case, if I use 1 core from processor A
> and 1 core from processor B, I should be able to get a decent speedup of
> more than 1, is that so? This option is also better than using 2 or even 4
> cores from the same processor.
>
>  Thank you very much.
>
>  Satish Balay wrote:
>
> > On Wed, 16 Apr 2008, Ben Tay wrote:
> >
> >
> >
> > > Hi Satish, thank you very much for helping me run the ex2f.F code.
> > >
> > > I think I've a clearer picture now. I believe I'm running on Dual-Core
> Intel
> > > Xeon 5160. The quad core is only on atlas3-01 to 04 and there's only 4
> of
> > > them. I guess that the lower peak is because I'm using Xeon 5160, while
> you
> > > are using Xeon X5355.
> > >
> > >
> >
> > I'm still a bit puzzled. I just ran the same binary on a 2 dualcore
> > xeon 5130 machine [which should be similar to your 5160 machine] and
> > get the following:
> >
> > [balay at n001 ~]$ grep MatMult log*
> > log.1:MatMult             1192 1.0 1.0591e+01 1.0 3.86e+09 1.0 0.0e+00
> 0.0e+00 0.0e+00 14 11  0  0  0  14 11  0  0  0   364
> > log.2:MatMult             1217 1.0 6.3982e+00 1.0 1.97e+09 1.0 2.4e+03
> 4.8e+03 0.0e+00 14 11100100  0  14 11100100  0   615
> > log.4:MatMult              969 1.0 4.7780e+00 1.0 7.84e+08 1.0 5.8e+03
> 4.8e+03 0.0e+00 14 11100100  0  14 11100100  0   656
> > [balay at n001 ~]$
> >
> >
> > > You mention about the speedups for MatMult and compare between KSPSolve.
> Are
> > > these the only things we have to look at? Because I see that some other
> event
> > > such as VecMAXPY also takes up a sizable % of the time. To get an
> accurate
> > > speedup, do I just compare the time taken by KSPSolve between different
> no. of
> > > processors or do I have to look at other events such as MatMult as well?
> > >
> > >
> >
> > Sometimes we look at individual components like MatMult() VecMAXPY()
> > to understand whats hapenning in each stage - and at KSPSolve() to
> > look at the agregate performance for the whole solve [which includes
> > MatMult VecMAXPY etc..]. Perhaps I should have also looked at
> > VecMDot() aswell - at 48% of runtime - its the biggest contributor to
> > KSPSolve() for your run.
> >
> > Its easy to get lost in the details of log_summary. Looking for
> > anamolies is one thing. Plotting scalability charts for the solver is
> > something else..
> >
> >
> >
> > > In summary, due to load imbalance, my speedup is quite bad. So maybe
> I'll just
> > > send your results to my school's engineer and see if they could do
> anything.
> > > For my part, I guess I'll just 've to wait?
> > >
> > >
> >
> > Yes - load imbalance at MatMult level is bad. On 4 proc run you have
> > ratio = 3.6 . This implies - there is one of the mpi-tasks is 3.6
> > times slower than the other task [so all speedup is lost here]
> >
> > You could try the latest mpich2 [1.0.7] - just for this SMP
> > experiment, and see if it makes a difference. I've built mpich2 with
> > [default gcc/gfortran and]:
> >
> > ./configure --with-device=ch3:nemesis:newtcp -with-pm=gforker
> >
> > There could be something else going on on this machine thats messing
> > up load-balance for basic petsc example..
> >
> > Satish
> >
> >
> >
> >
>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener