[petsc-users] snes matrix-free jacobian fails with preconditioner shape mismatch

Mon Mar 11 03:48:24 CDT 2024

Dear Barry,

Thanks for your response. Now I am doing simple debugging for my customized mat_mult_op of my shell jacobian matrix. As far as I understand, because the input and output of shell jacobian are all global vectors, I need to do the global to local mapping of my input vector (dF) by myself. Before starting debugging the mapping, I first try to verify the size match of input and output.

Because the input and output of a mat_mult_op of my shell matrix should have the same size. So I tried just equating them in my customized mat_mult_op. Basically like this 

subroutine GK_op(Jac,dF_global,output_global,err_PETSc)

  DM                                   :: dm_local ! Yi: later for is,ie
  Vec                                  :: dF_global
  Vec                                  :: output_global
  PetscErrorCode                       :: err_PETSc

  real(pREAL), pointer,dimension(:,:,:,:) :: dF_scal, output_scal

  call SNESGetDM(SNES_mech,dm_local,err_PETSc)
  CHKERRQ(err_PETSc)

  output_global = dF_global 

end subroutine GK_op

When I run with mpirun -np 3, it gives me similar error like previous, “Preconditioner number of local rows 27 does not equal input vector size 81”, (I changed my problem size so the numbers are different). 

Maybe simple equating input and output is not valid (due to ownership of different portion of a global vector). Then it may give me different error message. In fact my global dF has size 9*3*3*3, when running on 3 processors, the local dF has size 9*3*3*1 (I split my domain in z direction). The error message seems to suggest I am using a local dF rather than a global dF. And the output and input vector sizes seems to be different. Do I miss something here? 

Best regards,
Yi

From: Barry Smith <bsmith at petsc.dev> 
Sent: Sunday, March 10, 2024 6:45 PM
To: Yi Hu <y.hu at mpie.de>
Cc: Mark Adams <mfadams at lbl.gov>; petsc-users <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] snes matrix-free jacobian fails with preconditioner shape mismatch

On Mar 10, 2024, at 10:16 AM, Yi Hu <y.hu at mpie.de> wrote:

This Message Is From an External Sender
This message came from outside your organization.
Dear Mark,
Thanks for your reply. I see this mismatch. In fact my global DoF is 324. It seems like I always get the local size = global Dof / np^2, np is my processor number. By the way, I used DMDASNESsetFunctionLocal() to set my form function. Is it eligible to mix DMDASNESsetFunctionLocal() and a native SNESSetJacobian()?

Yes

Best,
Yi
On 3/10/24 13:55, Mark Adams wrote:
It looks like your input vector is the global vector, size 162, and the local matrix size is 81.Mark[1]PETSC ERROR: Preconditioner number of local rows 81 does not equal input vector size 162

On Sun, Mar 10, 2024 at 7:21 AM Yi Hu <y.hu at mpie.de> wrote:
This Message Is From an External Sender 
This message came from outside your organization. 

Dear petsc team, I implemented a matrix-free jacobian, and it can run sequentially. But running parallel I got the pc error like this (running with mpirun -np 2, only error from rank1 is presented here) [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------[1]PETSC ERROR: Nonconforming object sizes[1]PETSC ERROR: Preconditioner number of local rows 81 does not equal input vector size 162[1]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0PlyUrtdfCA$ for trouble shooting.[1]PETSC ERROR: Petsc Release Version 3.17.3, Jun 29, 2022[1]PETSC ERROR: /home/yi/workspace/DAMASK_yi/bin/DAMASK_grid on a arch-linux-c-opt named carbon-x1 by yi Sun Mar 10 12:01:46 2024[1]PETSC ERROR: Configure options --download-fftw --download-hdf5 --with-hdf5-fortran-bindings --download-fblaslapack --download-chaco --download-hypre --download-metis --download-mumps --download-parmetis --download-scalapack --download-suitesparse --download-superlu --download-superlu_dist --download-triangle --download-zlib --download-cmake --with-cxx-dialect=C++11 --with-c2html=0 --with-debugging=0 --with-ssl=0 --with-x=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3[1]PETSC ERROR: #1 PCApply() at /home/yi/App/petsc-3.17.3/src/ksp/pc/interface/precon.c:424[1]PETSC ERROR: #2 KSP_PCApply() at /home/yi/App/petsc-3.17.3/include/petsc/private/kspimpl.h:376[1]PETSC ERROR: #3 KSPInitialResidual() at /home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itres.c:64[1]PETSC ERROR: #4 KSPSolve_GMRES() at /home/yi/App/petsc-3.17.3/src/ksp/ksp/impls/gmres/gmres.c:242[1]PETSC ERROR: #5 KSPSolve_Private() at /home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itfunc.c:902[1]PETSC ERROR: #6 KSPSolve() at /home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itfunc.c:1078[1]PETSC ERROR: #7 SNESSolve_NEWTONLS() at /home/yi/App/petsc-3.17.3/src/snes/impls/ls/ls.c:222[1]PETSC ERROR: #8 SNESSolve() at /home/yi/App/petsc-3.17.3/src/snes/interface/snes.c:4756[1]PETSC ERROR: #9 User provided function() at User file:0 However, from snes matrix-free documentation (https://urldefense.us/v3/__https://petsc.org/release/manual/snes/*matrix-free-methods__;Iw!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0Ply6ZbywOw$), it is said matrix-free is used with pcnone. So I assume it would not apply preconditioner, but it did use preconditioning probably the same as my matrix-free shell matrix. Here is how i initialize my shell matrix and the corresponding customized multiplication.    call MatCreateShell(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,&int(9*product(cells(1:2))*cells3,pPETSCINT),&int(9*product(cells(1:2))*cells3,pPETSCINT),&                       F_PETSc,Jac_PETSc,err_PETSc)   call MatShellSetOperation(Jac_PETSc,MATOP_MULT,GK_op,err_PETSc)   call SNESSetDM(SNES_mech,DM_mech,err_PETSc)   call SNESSetJacobian(SNES_mech,Jac_PETSc,Jac_PETSc,PETSC_NULL_FUNCTION,0,err_PETSc)   call SNESGetKSP(SNES_mech,ksp,err_PETSc)   call PCSetType(pc,PCNONE,err_PETSc) And my GK_op is like subroutine GK_op(Jac,dF_global,output_local,err_PETSc)    DM                                   :: dm_local   Vec                                  :: dF_global, dF_local, output_local   Mat                                  :: Jac   PetscErrorCode                       :: err_PETSc    real(pREAL), pointer,dimension(:,:,:,:) :: dF_scal, output_scal    real(pREAL), dimension(3,3,cells(1),cells(2),cells3) :: &     dF   real(pREAL), dimension(3,3,cells(1),cells(2),cells3) :: &     output    call SNESGetDM(SNES_mech,dm_local,err_PETSc)    call DMGetLocalVector(dm_local,dF_local,err_PETSc)   call DMGlobalToLocalBegin(dm_local,dF_global,INSERT_VALUES,dF_local,err_PETSc)   call DMGlobalToLocalEnd(dm_local,dF_global,INSERT_VALUES,dF_local,err_PETSc)    call DMDAVecGetArrayReadF90(dm_local,dF_local,dF_scal,err_PETSc)   dF = reshape(dF_scal, [3,3,cells(1),cells(2),cells3]) .......     call DMDAVecRestoreArrayF90(dm_local,output_local,output_scal,err_PETSc)   CHKERRQ(err_PETSc)    call DMDAVecRestoreArrayF90(dm_local,dF_local,dF_scal,err_PETSc)   CHKERRQ(err_PETSc) end subroutine GK_op I checked my cells3, it corresponds to my local size, and it seems the local size of dF_local is ok. I am a bit lost here to find the reason for the preconditioner bug. Could you help me on this? Thanks. Best regards, Yi    -------------------------------------------------Stay up to date and follow us on LinkedIn, Twitter and YouTube. Max-Planck-Institut für Eisenforschung GmbHMax-Planck-Straße 1D-40237 Düsseldorf Handelsregister B 2533 Amtsgericht Düsseldorf GeschäftsführungProf. Dr. Gerhard DehmProf. Dr. Jörg NeugebauerProf. Dr. Dierk RaabeDr. Kai de Weldige Ust.-Id.-Nr.: DE 11 93 58 514 Steuernummer: 105 5891 1000  Please consider that invitations and e-mails of our institute are only valid if they end with …@mpie.de. If you are not sure of the validity please contact rco at mpie.de Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mailsaus unserem Haus nur mit der Endung …@mpie.de gültig sind. In Zweifelsfällen wenden Sie sich bitte an rco at mpie.de------------------------------------------------- 

-------------------------------------------------
Stay up to date and follow us on LinkedIn, Twitter and YouTube.

Max-Planck-Institut für Eisenforschung GmbH
Max-Planck-Straße 1
D-40237 Düsseldorf

Handelsregister B 2533 
Amtsgericht Düsseldorf

Geschäftsführung
Prof. Dr. Gerhard Dehm
Prof. Dr. Jörg Neugebauer
Prof. Dr. Dierk Raabe
Dr. Kai de Weldige

Ust.-Id.-Nr.: DE 11 93 58 514 
Steuernummer: 105 5891 1000

Please consider that invitations and e-mails of our institute are 
only valid if they end with …@mpie.de. 
If you are not sure of the validity please contact rco at mpie.de

Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
In Zweifelsfällen wenden Sie sich bitte an rco at mpie.de
-------------------------------------------------

-------------------------------------------------
Stay up to date and follow us on LinkedIn, Twitter and YouTube.

Max-Planck-Institut für Eisenforschung GmbH
Max-Planck-Straße 1
D-40237 Düsseldorf

Handelsregister B 2533 
Amtsgericht Düsseldorf

Geschäftsführung
Prof. Dr. Gerhard Dehm
Prof. Dr. Jörg Neugebauer
Prof. Dr. Dierk Raabe
Dr. Kai de Weldige

Ust.-Id.-Nr.: DE 11 93 58 514 
Steuernummer: 105 5891 1000

Please consider that invitations and e-mails of our institute are 
only valid if they end with …@mpie.de. 
If you are not sure of the validity please contact rco at mpie.de

Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
In Zweifelsfällen wenden Sie sich bitte an rco at mpie.de
-------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240311/ce31e799/attachment-0001.html>