<!DOCTYPE html>
<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Dear Mark, Thanks for your reply. I see this mismatch. In fact my global DoF is 324. It seems like I always get the local size = global Dof / np^2, np is my processor number. By the way, I used DMDASNESsetFunctionLocal() to set my form function.
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannercw8qp5q" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannercw8qp5q" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannercw8qp5q" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannercw8qp5q" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html>
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannercw8qp5q { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtoncw8qp5q:hover, .pfptPrimaryButtoncw8qp5q:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtoncw8qp5q:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear Mark,</p>
<p>Thanks for your reply. I see this mismatch. In fact my global DoF
is 324. It seems like I always get the local size = global Dof /
np^2, np is my processor number. By the way, I used
DMDASNESsetFunctionLocal() to set my form function. Is it eligible
to mix DMDASNESsetFunctionLocal() and a native SNESSetJacobian()?</p>
<p>Best,</p>
<p>Yi<br>
</p>
<div class="moz-cite-prefix">On 3/10/24 13:55, Mark Adams wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CADOhEh4G4PU5WwW+uqBnJ0EyD+PgaiY7CrF2SNkyZQ0wNx1LyQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<pre style="font-size:17.8px;font-family:sans-serif">It looks like your input vector is the global vector, size 162, and the local matrix size is 81.</pre>
<pre style="font-size:17.8px;font-family:sans-serif">Mark</pre>
<pre style="font-size:17.8px;font-family:sans-serif">[1]PETSC ERROR: Preconditioner number of local rows 81 does not equal
input vector size 162</pre>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Mar 10, 2024 at
7:21 AM Yi Hu <<a href="mailto:y.hu@mpie.de"
moz-do-not-send="true" class="moz-txt-link-freetext">y.hu@mpie.de</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="msg-4788138792073467353">
<div
style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
Dear petsc team, I implemented a matrix-free jacobian,
and it can run sequentially. But running parallel I got
the pc error like this (running with mpirun -np 2, only
error from rank1 is presented here) [1]PETSC ERROR:
--------------------- </div>
<div
style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr"
id="m_-4788138792073467353pfptBanner2satfg2"
style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)">
<div id="m_-4788138792073467353pfptBanner2satfg2"
style="float:left;display:block;margin:0px 0px 1px;max-width:600px">
<div id="m_-4788138792073467353pfptBanner2satfg2"
style="display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">
This Message Is From an External Sender </div>
<div id="m_-4788138792073467353pfptBanner2satfg2"
style="font-weight:normal;display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">
This message came from outside your organization. </div>
</div>
<div
style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px"> </div>
</div>
<div
style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">ZjQcmQRYFpfptBannerEnd</div>
<div>
<pre
style="font-family:sans-serif;font-size:100%;white-space:pre-wrap">Dear petsc team,
I implemented a matrix-free jacobian, and it can run sequentially. But
running parallel I got the pc error like this (running with mpirun -np
2, only error from rank1 is presented here)
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: Nonconforming object sizes
[1]PETSC ERROR: Preconditioner number of local rows 81 does not equal
input vector size 162
[1]PETSC ERROR: See <a
href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0PlyUrtdfCA$"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0PlyUrtdfCA$</a> for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.17.3, Jun 29, 2022
[1]PETSC ERROR: /home/yi/workspace/DAMASK_yi/bin/DAMASK_grid on a
arch-linux-c-opt named carbon-x1 by yi Sun Mar 10 12:01:46 2024
[1]PETSC ERROR: Configure options --download-fftw --download-hdf5
--with-hdf5-fortran-bindings --download-fblaslapack --download-chaco
--download-hypre --download-metis --download-mumps --download-parmetis
--download-scalapack --download-suitesparse --download-superlu
--download-superlu_dist --download-triangle --download-zlib
--download-cmake --with-cxx-dialect=C++11 --with-c2html=0
--with-debugging=0 --with-ssl=0 --with-x=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3
FOPTFLAGS=-O3
[1]PETSC ERROR: #1 PCApply() at
/home/yi/App/petsc-3.17.3/src/ksp/pc/interface/precon.c:424
[1]PETSC ERROR: #2 KSP_PCApply() at
/home/yi/App/petsc-3.17.3/include/petsc/private/kspimpl.h:376
[1]PETSC ERROR: #3 KSPInitialResidual() at
/home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itres.c:64
[1]PETSC ERROR: #4 KSPSolve_GMRES() at
/home/yi/App/petsc-3.17.3/src/ksp/ksp/impls/gmres/gmres.c:242
[1]PETSC ERROR: #5 KSPSolve_Private() at
/home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itfunc.c:902
[1]PETSC ERROR: #6 KSPSolve() at
/home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itfunc.c:1078
[1]PETSC ERROR: #7 SNESSolve_NEWTONLS() at
/home/yi/App/petsc-3.17.3/src/snes/impls/ls/ls.c:222
[1]PETSC ERROR: #8 SNESSolve() at
/home/yi/App/petsc-3.17.3/src/snes/interface/snes.c:4756
[1]PETSC ERROR: #9 User provided function() at User <a
class="moz-txt-link-freetext" href="file:0">file:0</a>
However, from snes matrix-free documentation
(<a
href="https://urldefense.us/v3/__https://petsc.org/release/manual/snes/*matrix-free-methods__;Iw!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0Ply6ZbywOw$"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://urldefense.us/v3/__https://petsc.org/release/manual/snes/*matrix-free-methods__;Iw!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0Ply6ZbywOw$</a>), it is said
matrix-free is used with pcnone. So I assume it would not apply
preconditioner, but it did use preconditioning probably the same as my
matrix-free shell matrix. Here is how i initialize my shell matrix and
the corresponding customized multiplication.
call MatCreateShell(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,&
int(9*product(cells(1:2))*cells3,pPETSCINT),&
int(9*product(cells(1:2))*cells3,pPETSCINT),&
F_PETSc,Jac_PETSc,err_PETSc)
call MatShellSetOperation(Jac_PETSc,MATOP_MULT,GK_op,err_PETSc)
call SNESSetDM(SNES_mech,DM_mech,err_PETSc)
call
SNESSetJacobian(SNES_mech,Jac_PETSc,Jac_PETSc,PETSC_NULL_FUNCTION,0,err_PETSc)
call SNESGetKSP(SNES_mech,ksp,err_PETSc)
call PCSetType(pc,PCNONE,err_PETSc)
And my GK_op is like
subroutine GK_op(Jac,dF_global,output_local,err_PETSc)
DM :: dm_local
Vec :: dF_global, dF_local, output_local
Mat :: Jac
PetscErrorCode :: err_PETSc
real(pREAL), pointer,dimension(:,:,:,:) :: dF_scal, output_scal
real(pREAL), dimension(3,3,cells(1),cells(2),cells3) :: &
dF
real(pREAL), dimension(3,3,cells(1),cells(2),cells3) :: &
output
call SNESGetDM(SNES_mech,dm_local,err_PETSc)
call DMGetLocalVector(dm_local,dF_local,err_PETSc)
call
DMGlobalToLocalBegin(dm_local,dF_global,INSERT_VALUES,dF_local,err_PETSc)
call
DMGlobalToLocalEnd(dm_local,dF_global,INSERT_VALUES,dF_local,err_PETSc)
call DMDAVecGetArrayReadF90(dm_local,dF_local,dF_scal,err_PETSc)
dF = reshape(dF_scal, [3,3,cells(1),cells(2),cells3])
.......
call DMDAVecRestoreArrayF90(dm_local,output_local,output_scal,err_PETSc)
CHKERRQ(err_PETSc)
call DMDAVecRestoreArrayF90(dm_local,dF_local,dF_scal,err_PETSc)
CHKERRQ(err_PETSc)
end subroutine GK_op
I checked my cells3, it corresponds to my local size, and it seems the
local size of dF_local is ok.
I am a bit lost here to find the reason for the preconditioner bug.
Could you help me on this? Thanks.
Best regards,
Yi
-------------------------------------------------
Stay up to date and follow us on LinkedIn, Twitter and YouTube.
Max-Planck-Institut für Eisenforschung GmbH
Max-Planck-Straße 1
D-40237 Düsseldorf
Handelsregister B 2533
Amtsgericht Düsseldorf
Geschäftsführung
Prof. Dr. Gerhard Dehm
Prof. Dr. Jörg Neugebauer
Prof. Dr. Dierk Raabe
Dr. Kai de Weldige
Ust.-Id.-Nr.: DE 11 93 58 514
Steuernummer: 105 5891 1000
Please consider that invitations and e-mails of our institute are
only valid if they end with …@<a href="https://urldefense.us/v3/__http://mpie.de__;!!G_uCfscf7eWS!eslPb6ZEnby8M7NKRe_U_RT-95w2J8O2ngsc8rmrQVlMpEcB6ZTKlN2g65crXCWv7D2F9ubkzYI5nDpoeQ$" target="_blank" moz-do-not-send="true">mpie.de</a>.
If you are not sure of the validity please contact <a
href="mailto:rco@mpie.de" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">rco@mpie.de</a>
Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
aus unserem Haus nur mit der Endung …@<a href="https://urldefense.us/v3/__http://mpie.de__;!!G_uCfscf7eWS!eslPb6ZEnby8M7NKRe_U_RT-95w2J8O2ngsc8rmrQVlMpEcB6ZTKlN2g65crXCWv7D2F9ubkzYI5nDpoeQ$" target="_blank" moz-do-not-send="true">mpie.de</a> gültig sind.
In Zweifelsfällen wenden Sie sich bitte an <a href="mailto:rco@mpie.de"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">rco@mpie.de</a>
-------------------------------------------------
</pre>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<BR />
<BR />
<HR />
-------------------------------------------------<BR />
Stay up to date and follow us on LinkedIn, Twitter and YouTube.<BR />
<BR />
Max-Planck-Institut für Eisenforschung GmbH<BR />
Max-Planck-Straße 1<BR />
D-40237 Düsseldorf<BR />
<BR />
Handelsregister B 2533 <BR />
Amtsgericht Düsseldorf<BR />
<BR />
Geschäftsführung<BR />
Prof. Dr. Gerhard Dehm<BR />
Prof. Dr. Jörg Neugebauer<BR />
Prof. Dr. Dierk Raabe<BR />
Dr. Kai de Weldige<BR />
<BR />
Ust.-Id.-Nr.: DE 11 93 58 514 <BR />
Steuernummer: 105 5891 1000<BR />
<BR />
<BR />
Please consider that invitations and e-mails of our institute are <BR />
only valid if they end with …@mpie.de. <BR />
If you are not sure of the validity please contact rco@mpie.de<BR />
<BR />
Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails<BR />
aus unserem Haus nur mit der Endung …@mpie.de gültig sind. <BR />
In Zweifelsfällen wenden Sie sich bitte an rco@mpie.de<BR />
-------------------------------------------------<BR />
</body>
</html>