[petsc-users] Why my petsc program get stuck and hang there when assembling the matrix?

[Fangbo Wang]

Hi,

I construct a big matrix which is 1.3million by 1.3million which is using
approximatly 100GB memory. I have a computer with 500GB memory.

I run the Petsc program and it get stuck when finally assembling the
matrix. The program is using around 200GB memory only. However, the program
just get stuck there. Here is the output message when it gets stuck.
.
.
*previous outputs not shown here*
.
[12] MatStashScatterBegin_Ref(): No of messages: 15
[12] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 328581552 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 163649328 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 95512224 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 317711616 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 170971776 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 254000064 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 163146720 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 345150048 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 163411584 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 739711296 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 13: size: 435247344 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 435136752 bytes
[12] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 346167552 bytes
[14] MatAssemblyBegin_MPIAIJ(): Stash has 263158893 entries, uses 14
mallocs.
[8] MatAssemblyBegin_MPIAIJ(): Stash has 286768572 entries, uses 14 mallocs.
[12] MatAssemblyBegin_MPIAIJ(): Stash has 291181818 entries, uses 14
mallocs.
[13] MatStashScatterBegin_Ref(): No of messages: 15
[13] MatStashScatterBegin_Ref(): Mesg_to: 0: size: 271636416 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 1: size: 271636416 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 2: size: 220594464 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 3: size: 51041952 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 4: size: 276201408 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 5: size: 256952256 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 6: size: 198489024 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 7: size: 218657760 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 8: size: 219686880 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 9: size: 288874752 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 10: size: 428874816 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 11: size: 172579968 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 12: size: 639835680 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 14: size: 270060144 bytes
[13] MatStashScatterBegin_Ref(): Mesg_to: 15: size: 511244160 bytes
[13] MatAssemblyBegin_MPIAIJ(): Stash has 268522881 entries, uses 14
mallocs.
[5] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space:
89786788 unneeded,7025212 used
[5] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[5] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81
[5] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines.
[5] MatSeqAIJCheckInode(): Found 32271 nodes of 96812. Limit used: 5. Using
Inode routines
[4] MatAssemblyEnd_SeqAIJ(): Matrix size: 96812 X 96812; storage space:
89841924 unneeded,6970076 used
[4] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0
[4] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 81
[4] MatCheckCompressedRow(): Found the ratio (num_zerorows
0)/(num_localrows 96812) < 0.6. Do not use CompressedRow routines.
[4] MatSeqAIJCheckInode(): Found 32272 nodes of 96812. Limit used: 5. Using
Inode routines

*stuck here!!!!*


Any one have ideas on this? Thank you very much!



Fangbo Wang



-- 
Fangbo Wang, PhD student
Stochastic Geomechanics Research Group
Department of Civil, Structural and Environmental Engineering
University at Buffalo
Email: *fangbowa at buffalo.edu <fangbowa at buffalo.edu>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20170226/483bab9c/attachment.html>