[petsc-dev] MatGetSubMatrices taking a long time

[Barry Smith]

  Jungho,

   Thanks

   The first thing you need to do is edit $PETSC_ARCH/include/petscconf.h for the optimized version on Fusion and remove the lines

#ifndef PETSC_USE_CTABLE
#define PETSC_USE_CTABLE 1
#endif

  Then run make again to recompile the PETSc library (make sure it gets recompiled). Then recompile the example and  run the 256 process case again and send me the new -log_summary file.

  I think a change to make another case of MatGetSubMatrices() more scalable has made this case much less scalable. 

  Thanks



   Barry


On Sep 18, 2011, at 7:11 PM, Jungho Lee wrote:

> Here they are.
> 
> Jungho
> 
> On Sun, Sep 18, 2011 at 6:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> On Sep 18, 2011, at 5:28 PM, Jungho Lee wrote:
>> 
>>> Barry,
>>> 
>>> I ran ex65 on fusion, with
>>> 
>>> 1     proc: -da_refine 5 -da_grid_x 16 -da_grid_x 16
>>> 4     proc: -da_refine 6-da_grid_x 16 -da_grid_x 16
>>> 16   proc: -da_refine 7 -da_grid_x 16 -da_grid_x 16
>>> 64   proc: -da_refine 8 -da_grid_x 16 -da_grid_x 16
>>> 256 proc: -da_refine 9 -da_grid_x 16 -da_grid_x 16
>>> 
>>> and looked at 3 operations that were taking a huge chunk of time
>>> (PCApply, PCSetUp, MatGetSubMatrcies). I attach a pdf file with the
>>> table. As you can see, the percentage of time spent in
>>> MatGetSubMatrices increases up to a whopping 51% with 256 processors.
>>> Isn't this ridiculous?
>> 
>> 
>>   Yes it is, and I guess we have to do something about it. Now for this code the MatGetSubMatrices() is called once by MatGetSubMatrix() to get the part of the matrix associated with the variables associated with inactive constraints.  Thus each process ends up with a subset of the rows and columns it started with and doesn't need rows from other processes. This really shouldn't be a hugely time consuming operation, there must be something non-scalable for this particular case that we need to determine and eliminate. First send me the -log_summary files.
>> 
>>   Barry
>> 
>>> 
>>> Jungho
>>> <table.pdf>
>> 
> <ex65p256_grid16_ref9.out.694579.fmgt2.lcrc.anl.gov><ex65p64_grid16_ref8.out.694578.fmgt2.lcrc.anl.gov><ex65p16_grid16_ref7.out.694577.fmgt2.lcrc.anl.gov><ex65p4_grid16_ref6.out.694576.fmgt2.lcrc.anl.gov><ex65p1_grid16_ref5.out.694575.fmgt2.lcrc.anl.gov>