[petsc-users] Scalability of PETSc on vesta.alcf

[Roc Wang]

Hello, 

   I am testing a petsc program on vesta.alcf.acl.gov. The scalability was fine when the number of ranks is less then 1024. However, when the 2048 ranks were used, the computation time increased. ( see below)

 No. of cores     No. of iter     timing (s)  
                               
    1       &   out of memory  &        
    2       &   out of memory  &       
    4       &   300                   &  3208.4     
    8       &   280                   &  1521.52      
  16       &   335                   &    901.54      
  32       &   340                   &    456.06       
  64       &   356                   &    253.41       
 128      &   371                   &    132.43      
 256      &   374                   &      63.8       
 512      &   396                   &      32.13     
1024     &   434                   &      17.64      
2048     &   455                   &      30.76   

 The number of grid nodes in each dimension is 320*320*320. The number of ranks in each dimension for 1024 ranks is 8*8*16 and for 2048 ranks is 16*8*16.    The Mat and Vec in the program were managed by DMDA. I believe PETSc has good scalability, however, it's just strange for me the computation time for 2048 ranks increased very much.  I am wondering is it because of the memories shared by the cores of one node?  Since the cases with  less than 1024 ranks used 1024 node and one core per node while 2048 case used 32 by 64 cores. If it is because the amount of memories available for one core, how can I get better scalability for more cores (say 4096 cores)?

I attached the runtime log files for 1024 by 1 and 2048(32by64).


 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140120/61bce956/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: p1024_p1_153436.output
Type: application/octet-stream
Size: 138709 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140120/61bce956/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: p2048_32c64_148588.output
Type: application/octet-stream
Size: 249371 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140120/61bce956/attachment-0003.obj>