[petsc-users] solving saddle point problem in distributed memory

Wed Jul 29 05:30:34 CDT 2020

Thanks for your suggestions. Sure. I will check my codes of MatZeroRows from a small problem. The program does not terminate during solving in distributed memory, but repeating this error in run time. In fact, this is the entire error message repeating itself during runtime.

From: Matthew Knepley [mailto:knepley at gmail.com]
Sent: Wednesday, 29 July 2020 5:21 PM
To: Bin Liu <lbllm2018 at hotmail.com>
Cc: Barry Smith <bsmith at petsc.dev>; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] solving saddle point problem in distributed memory

On Tue, Jul 28, 2020 at 11:33 PM Bin Liu <lbllm2018 at hotmail.com<mailto:lbllm2018 at hotmail.com>> wrote:
Dear Barry,

Thanks for your suggestions. I just tested it. I still get the same error. I attached it below:

[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Object is in wrong state
[1]PETSC ERROR: Matrix is missing diagonal entry in the zeroed row 24214
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.9.1, Apr, 29, 2018
[1]PETSC ERROR: ./re100 on a  named kdp-MS-7850 by kdp Wed Jul 29 11:26:57 2020
[1]PETSC ERROR: Configure options --prefix=/home/kdp/petsc --debugging=0 --with-mpi-dir=/home/kdp/mpich --download-fblaslapack --download-metis --download-parmetis --download-hypre=/home/kdp/hypre-2.14.0.tar.gz --download-superlu_dist
[1]PETSC ERROR: #1 MatZeroRows_SeqAIJ() line 1819 in /home/kdp/petsc-3.9.1/src/mat/impls/aij/seq/aij.c
[1]PETSC ERROR: #2 MatZeroRows() line 5710 in /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c
[1]PETSC ERROR: #3 MatZeroRows_MPIAIJ() line 797 in /home/kdp/petsc-3.9.1/src/mat/impls/aij/mpi/mpiaij.c
[1]PETSC ERROR: #4 MatZeroRows() line 5710 in /home/kdp/petsc-3.9.1/src/mat/interface/matrix.c

It still report missing diagonal entry. However I have epxlicitly assign 0.0 to the diagonal element of the system matrix and solve it in sequential mode without error. Where could possibly my wrong set up be?

1) Please send the complete error message. The stack trace here is cut off.

2) It does not look like it is failing in the solver, but in the call to MatZeroRows(), which as it says, requires diagonal elements

3) It looks like you are not assembling the matrix correctly in parallel

I recommend making as small a problem as possible, running on 1 and 2 processes, and printing out the matrix using -mat_view to compare them entry-by-entry.

  Thanks,

     Matt

Best

________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Wednesday, July 29, 2020 10:49 AM
To: Bin Liu <lbllm2018 at hotmail.com<mailto:lbllm2018 at hotmail.com>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] solving saddle point problem in distributed memory

   No, PETSc direct solvers do not work in parallel. Only SuperLU_DIST, MUMPS, and PasTix.

    -pc_type lu (or cholesky depending on the package) -pc_factor_mat_solver_type superlu_dist or mumps or pastix

   to run the direct solver in parallel.

On Jul 28, 2020, at 9:01 PM, Bin Liu <lbllm2018 at hotmail.com<mailto:lbllm2018 at hotmail.com>> wrote:

Dear Barry,

Thanks for your explanation. Does it mean the default director solver LU in PETSc can also run in the distributed memory mode? I only used iterative solvers with preconditioners in distributed memory before.  My experience in using director solver in parallel is indeed limited, especially the saddle point problem in parallel. I have done some search online, but I did not find a working set up at this moment. Could you give a sample setup of the direct solver for parallel run? It is really appreciated.

From: Barry Smith [mailto:bsmith at petsc.dev]
Sent: Wednesday, 29 July 2020 9:48 AM
To: Bin Liu <lbllm2018 at hotmail.com<mailto:lbllm2018 at hotmail.com>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] solving saddle point problem in distributed memory

   SuperLU_DIST won't "magically" run on saddle point problems. It only does limited pivoting, realistically a parallel LU cannot always do complete pivoting or it becomes a sequential algorithm. For parallel you need to use PCFIELDSPLIT, for sequential you can use SuperLU (not SuperLU_DIST) since it can do more pivoting, being a sequential algorithm.

  Barry

On Jul 28, 2020, at 7:59 PM, Bin Liu <lbllm2018 at hotmail.com<mailto:lbllm2018 at hotmail.com>> wrote:

Thanks for your tutorials. Yes. I tried PCFIELDSPLIT. However, it only works for sequential runs. When I run the code in distributed memory, it reports errors. In fact, the essence of my wonder is (a) how to set up superlu_dist in petsc for solving saddle point problem in distributed memory? (b) does the direct solvers in petsc can run in distributed memory for solving saddle point problem?

From: Stefano Zampini [mailto:stefano.zampini at gmail.com]
Sent: Tuesday, 28 July 2020 6:55 PM
To: Bin Liu <lbllm2018 at hotmail.com<mailto:lbllm2018 at hotmail.com>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] solving saddle point problem in distributed memory

If you want advice you should post the error trace PETSc reports.

Anyway, solving Stokes is not so trivial (without direct solvers, you may need mesh dependent information), but we have examples for it

https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex42.c

https://gitlab.com/petsc/petsc/-/blob/master/src/ksp/ksp/tutorials/ex43.c

https://gitlab.com/petsc/petsc/-/blob/master/src/snes/tutorials/ex69.c

If you scroll to the end of those files, you see a bunch of possible options either using PCFIELDSPLIT,  PCBDDC or KSPFETIDP.

On Jul 28, 2020, at 12:37 PM, Bin Liu <lbllm2018 at hotmail.com<mailto:lbllm2018 at hotmail.com>> wrote:

I would like to solve a saddle point problem arising from the stokes equation. I got successful to use the direct solvers in sequential runs. However, I would like to extend it for distributed memory computation. I tried to use superlu_dist, but the program returns errors. Is it possible to solve a saddle point problem in distributed memory using superlu_dist? Could anyone give a simple sample code to set up the parameters of the solver?

Thanks

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200729/e6d1abd2/attachment-0001.html>