[petsc-users] LU factorization and solution of independent matrices does not scale, why?

[Thomas Witkowski]

In my multilevel FETI-DP code, I have localized course matrices, which  
are defined on only a subset of all MPI tasks, typically between 4 and  
64 tasks. The MatAIJ and the KSP objects are both defined on a MPI  
communicator, which is a subset of MPI::COMM_WORLD. The LU  
factorization of the matrices is computed with either MUMPS or  
superlu_dist, but both show some scaling property I really wonder of:  
When the overall problem size is increased, the solve with the LU  
factorization of the local matrices does not scale! But why not? I  
just increase the number of local matrices, but all of them are  
independent of each other. Some example: I use 64 cores, each coarse  
matrix is spanned by 4 cores so there are 16 MPI communicators with 16  
coarse space matrices. The problem need to solve 192 times with the  
coarse space systems, and this takes together 0.09 seconds. Now I  
increase the number of cores to 256, but let the local coarse space be  
defined again on only 4 cores. Again, 192 solutions with these coarse  
spaces are required, but now this takes 0.24 seconds. The same for  
1024 cores, and we are at 1.7 seconds for the local coarse space solver!

For me, this is a total mystery! Any idea how to explain, debug and  
eventually how to resolve this problem?

Thomas