[petsc-dev] MatGetSubMatrices taking a long time

[Barry Smith]

On Sep 18, 2011, at 5:28 PM, Jungho Lee wrote:

> Barry,
> 
> I ran ex65 on fusion, with
> 
> 1     proc: -da_refine 5 -da_grid_x 16 -da_grid_x 16
> 4     proc: -da_refine 6-da_grid_x 16 -da_grid_x 16
> 16   proc: -da_refine 7 -da_grid_x 16 -da_grid_x 16
> 64   proc: -da_refine 8 -da_grid_x 16 -da_grid_x 16
> 256 proc: -da_refine 9 -da_grid_x 16 -da_grid_x 16
> 
> and looked at 3 operations that were taking a huge chunk of time
> (PCApply, PCSetUp, MatGetSubMatrcies). I attach a pdf file with the
> table. As you can see, the percentage of time spent in
> MatGetSubMatrices increases up to a whopping 51% with 256 processors.
> Isn't this ridiculous?


   Yes it is, and I guess we have to do something about it. Now for this code the MatGetSubMatrices() is called once by MatGetSubMatrix() to get the part of the matrix associated with the variables associated with inactive constraints.  Thus each process ends up with a subset of the rows and columns it started with and doesn't need rows from other processes. This really shouldn't be a hugely time consuming operation, there must be something non-scalable for this particular case that we need to determine and eliminate. First send me the -log_summary files.

   Barry

> 
> Jungho
> <table.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: table.pdf
Type: application/pdf
Size: 17222 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20110918/110e9880/attachment.pdf>