[petsc-users] Very poor speed up performance

[Yongjun Chen]

On Wed, Dec 22, 2010 at 5:40 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> On Dec 22, 2010, at 9:55 AM, Yongjun Chen wrote:
>
> >
> > Satish,
> >
> > I have reconfigured the PETSC with –download-mpich=1 and
> –with-device=ch3:sock. The results show that the speed up can now remain
> increasing when computing cores increase from 1 to 16. However, the maximum
> speed up is still only around 6.0 with 16 cores. The new log files can be
> found in the attachment.
> >
> >
> > (1)
> >
> > I checked the configuration of the first server again. This server is a
> shared-memory computer, with
> >
> > Processors: 4 CPUS * 4Cores/CPU, with each core 2500MHz
> >
> > Memories: 16 *2 GB DDR2 333 MHz, dual channel, data width 64 bit, so the
> memory Bandwidth for 2 memories is 64/8*166*2*2=5.4GB/s.
>
>    Wait a minute. You have 16 cores that share 5.4 GB/s???? This is not
> enough for iterative solvers, in fact this is absolutely terrible for
> iterative solvers. You really want 5.4 GB/s PER core! This machine is
> absolutely inappropriate for iterative solvers. No package can give you good
> speedups on this machine.
>
>   Barry
>


Barry, there are 16 memories, every 2 memories make up one dual channel,
thus in this machine there are 8 dual channel, each dual channel has the
memory bandwidth 5.4GB/s.

Yongjun




>
> >
> > It seems that each core can get 2.7GB/s memory bandwidth which can
> fulfill the basic requirement for sparse iterative solvers.
> >
> > Is this correct? Does the shared-memory type of computer have no benefit
> for PETSC when the memory bandwidth is limited?
> >
> >
> > (2)
> >
> > Beside, we would like to continue our work by employing a matrix
> partitioning / reordering algorithm, such as Metis or ParMetis, to improve
> the speed up performance of the program. (The current program works without
> any matrix decomposition.)
> >
> >
> > Matt, as you said in
> http://lists.mcs.anl.gov/pipermail/petsc-users/2007-January/001017.html,“Reordering a matrix can result in fewer iterations for an iterative
> solver“.
> >
> > Do you think the matrix partitioning/reordering will work for this
> program? Or any further suggestions?
> >
> >
> > Any comments are very welcome! Thank you!
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Dec 20, 2010 at 11:04 PM, Satish Balay <balay at mcs.anl.gov>
> wrote:
> > On Mon, 20 Dec 2010, Yongjun Chen wrote:
> >
> > > Matt, Barry, thanks a lot for your reply! I will try mpich hydra
> firstly and
> > > see what I can get.
> >
> > hydra is just the process manager.
> >
> > Also --download-mpich uses a slightly older version - with
> > device=ch3:sock for portability and valgrind reasons [development]
> >
> > You might want to install latest mpich manually with the defaut
> > device=ch3:nemsis and recheck..
> >
> > satish
> >
> >
> >
> >
> <log_ch3sock_jacobi_bicg_4cpus.txt><log_ch3sock_jacobi_bicg_8cpus.txt><log_ch3sock_jacobi_bicg_12cpus.txt><log_ch3sock_jacobi_bicg_16cpus.txt>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20101222/e62fe60b/attachment.htm>