[petsc-users] PCHPDDM and matrix type

Mon Apr 17 23:23:43 CDT 2023

1) PCHPDDM handles AIJ, BAIJ, SBAIJ, IS, NORMAL, NORMALHERMITIAN, SCHURCOMPLEMENT, HTOOL
2) This PC is based on domain decomposition, with no support yet for “over decomposition”. If you run with a single process, it’s like PCASM or PCBJACOBI, you’ll get the same behavior as if you were just using the sub PC (in this case, an exact factorization)
3) The error you are seeing is likely due to a failure while coarsening, I will ask you for some info
4) Unrelated, but you should probably not use --with-cxx-dialect=C++11 and instead stick to --with-cxx-dialect=11 (unless you have a good reason to)

Thanks,
Pierre

> On 18 Apr 2023, at 1:26 AM, Alexander Lindsay <alexlindsay239 at gmail.com> wrote:
> 
> I don't really get much more of a stack trace out:
> 
> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>  
> [0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
> [1]PETSC ERROR: Invalid argument
> [1]PETSC ERROR:  
> [0]PETSC ERROR:   Option left: name:-i value: full_upwinding_2D.i source: command line
> [0]PETSC ERROR: [1]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
>   Option left: name:-ksp_converged_reason value: ::failed source: code
> [0]PETSC ERROR:   Option left: name:-pc_hpddm_coarse_mat_type value: baij source: command line
> [0]PETSC ERROR:   Option left: name:-pc_hpddm_coarse_pc_type value: lu source: command line
> [1]PETSC ERROR:   Option left: name:-i value: full_upwinding_2D.i source: command line
> [1]PETSC ERROR:   Option left: name:-ksp_converged_reason value: ::failed source: code
> [0]PETSC ERROR:   Option left: name:-snes_converged_reason value: ::failed source: code
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.17.4-3368-g5a48edb989d  GIT Date: 2023-04-16 17:35:24 +0000
> [1]PETSC ERROR:   Option left: name:-pc_hpddm_coarse_mat_type value: baij source: command line
> [1]PETSC ERROR:   Option left: name:-pc_hpddm_coarse_pc_type value: lu source: command line
> [0]PETSC ERROR: ../../../moose_test-opt on a arch-moose named rod.hpc.inl.gov <http://rod.hpc.inl.gov/> by lindad Mon Apr 17 16:11:09 2023
> [0]PETSC ERROR: Configure options --download-hypre=1 --with-shared-libraries=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0   --with-debugging=no --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-strumpack=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-openmp=1 --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices  --with-make-np=256 --download-hpddm
> [1]PETSC ERROR:   Option left: name:-snes_converged_reason value: ::failed source: code
> [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [1]PETSC ERROR: [0]PETSC ERROR: #1 buildTwo() at /raid/lindad/moose/petsc/arch-moose/include/HPDDM_schwarz.hpp:1012
> Petsc Development GIT revision: v3.17.4-3368-g5a48edb989d  GIT Date: 2023-04-16 17:35:24 +0000
> [1]PETSC ERROR: ../../../moose_test-opt on a arch-moose named rod.hpc.inl.gov <http://rod.hpc.inl.gov/> by lindad Mon Apr 17 16:11:09 2023
> [1]PETSC ERROR: Configure options --download-hypre=1 --with-shared-libraries=1 --download-hdf5=1 --with-hdf5-fortran-bindings=0   --with-debugging=no --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-strumpack=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-openmp=1 --with-cxx-dialect=C++11 --with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices  --with-make-np=256 --download-hpddm
> [1]PETSC ERROR: #1 buildTwo() at /raid/lindad/moose/petsc/arch-moose/include/HPDDM_schwarz.hpp:1012
> 
> On Mon, Apr 17, 2023 at 4:55 PM Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>> I don't think so. Can you show the whole stack?
>> 
>>   THanks,
>> 
>>     Matt
>> 
>> On Mon, Apr 17, 2023 at 6:24 PM Alexander Lindsay <alexlindsay239 at gmail.com <mailto:alexlindsay239 at gmail.com>> wrote:
>>> If it helps: if I use those exact same options in serial, then no errors and the linear solve is beautiful :-) 
>>> 
>>> On Mon, Apr 17, 2023 at 4:22 PM Alexander Lindsay <alexlindsay239 at gmail.com <mailto:alexlindsay239 at gmail.com>> wrote:
>>>> I'm likely revealing a lot of ignorance, but in order to use HPDDM as a preconditioner does my system matrix (I am using the same matrix for A and P) need to be block type, e.g. baij or sbaij ? In MOOSE our default is aij and I am currently getting
>>>> 
>>>> [1]PETSC ERROR: #1 buildTwo() at /raid/lindad/moose/petsc/arch-moose/include/HPDDM_schwarz.hpp:1012
>>>> 
>>>> with options:
>>>> 
>>>> -pc_type hpddm -pc_hpddm_block_splitting -pc_hpddm_coarse_mat_type baij -pc_hpddm_coarse_pc_type lu -pc_hpddm_define_subdomains -pc_hpddm_levels_1_eps_gen_non_hermitian -pc_hpddm_levels_1_eps_nev 50 -pc_hpddm_levels_1_eps_threshold 0.1 -pc_hpddm_levels_1_st_matstructure SAME -pc_hpddm_levels_1_st_share_sub_ksp -pc_hpddm_levels_1_sub_pc_factor_mat_solver_type mumps -pc_hpddm_levels_1_sub_pc_type lu
>>>> 
>>>> Alex
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230418/60752d93/attachment.html>