[petsc-users] MatAssemblyEnd taking too long

[Manav Bhatia]

Hi, 

   I have an application that uses the KSP solver and I am looking at the performance with increasing system size. I am currently running on MacBook Pro with 32 GB memory and Petsc obtained from GitHub (commit df0e43005dbe6ff47eff22a32b336a6c37d02c3a). 

   The application runs fine till about 2e6 DoFs using gamg without any problems. 

   However, when I try a larger system size, in this case with 5.4e6 DoFs, the application hangs for an hour and I have to kill the MPI processes. 

   I tried to use Xcode Instruments to profile the 8 MPI processes and I have attached a screenshot of the recorded results from each process. All processes are stuck inside MatAssemblyEnd, but at different function calls. 

   I am not sure how to debug this issue, and would greatly appreciate any guidance. 

   For reference, I am calling PETSc with the following options:
  -ksp_type gmres -pc_type gamg -mat_block_size 3 -mg_levels_ksp_max_it 4 -ksp_monitor -ksp_converged_reason

Regards,
Manav

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/3ece724d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2020-08-19 at 7.16.08 PM.png
Type: image/png
Size: 211493 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/3ece724d/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2020-08-19 at 7.16.06 PM.png
Type: image/png
Size: 202337 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/3ece724d/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2020-08-19 at 7.16.04 PM.png
Type: image/png
Size: 184314 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/3ece724d/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2020-08-19 at 7.16.02 PM.png
Type: image/png
Size: 185896 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/3ece724d/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2020-08-19 at 7.15.59 PM.png
Type: image/png
Size: 804446 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/3ece724d/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2020-08-19 at 7.15.57 PM.png
Type: image/png
Size: 851646 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/3ece724d/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2020-08-19 at 7.15.55 PM.png
Type: image/png
Size: 787467 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/3ece724d/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2020-08-19 at 7.15.52 PM.png
Type: image/png
Size: 966045 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200819/3ece724d/attachment-0015.png>