<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannertjdp0yo { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtontjdp0yo:hover, .pfptPrimaryButtontjdp0yo:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtontjdp0yo:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
I think you misunderstand "global" vector. The global solution is never on one processor. A global vector in this case seems to have 81 global values and 27 local values. It looks like you create a "global" vector that has
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannertjdp0yo" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannertjdp0yo" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannertjdp0yo" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannertjdp0yo" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">I think you misunderstand "global" vector. The global solution is never on one processor. A global vector in this case seems to have 81 global values and 27 local values.<div>It looks like you create a "global" vector that has 81 local values that should never be created other than for debigging.</div><div>GlobalToLocal refers to a "local" vector with ghost cells, so it would have > 27 values in this case and you use it for local operations only (its communicator is PETSC_COMM_SELF).</div><div>Hope this helps,</div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 11, 2024 at 4:48 AM Yi Hu <<a href="mailto:y.hu@mpie.de">y.hu@mpie.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg6768418666828824524"><div lang="en-DE"><div class="m_6768418666828824524WordSection1"><p class="MsoNormal"><span lang="EN-US">Dear Barry,<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US">Thanks for your response. Now I am doing simple debugging for my customized mat_mult_op of my shell jacobian matrix. As far as I understand, because the input and output of shell jacobian are all global vectors, I need to do the global to local mapping of my input vector (dF) by myself. Before starting debugging the mapping, I first try to verify the size match of input and output.<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US">Because the input and output of a mat_mult_op of my shell matrix should have the same size. So I tried just equating them in my customized mat_mult_op. Basically like this <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(86,156,214)">subroutine</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> </span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(220,220,170)">GK_op</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)">(</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(156,220,254)">Jac</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)">,</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(156,220,254)">dF_global</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)">,</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(156,220,254)">output_global</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)">,</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(156,220,254)">err_PETSc</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)">)<u></u><u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"><u></u> <u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> DM :: dm_local </span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(106,153,85)">! Yi: later for is,ie</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"><u></u><u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> Vec :: dF_global<u></u><u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> Vec :: output_global<u></u><u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> PetscErrorCode :: err_PETSc<u></u><u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"><u></u> <u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> </span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(86,156,214)">real</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)">(pREAL), </span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(86,156,214)">pointer</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)">,</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(86,156,214)">dimension</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)">(:,:,:,:) </span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(212,212,212)">::</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> dF_scal, output_scal<u></u><u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"><u></u> <u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> </span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(197,134,192)">call</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> </span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(220,220,170)">SNESGetDM</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)">(SNES_mech,dm_local,err_PETSc)<u></u><u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> CHKERRQ(err_PETSc)<u></u><u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"><u></u> <u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> output_global </span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(212,212,212)">=</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> dF_global</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(86,156,214)"> <u></u><u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(86,156,214)"><u></u> <u></u></span></p><p class="MsoNormal" style="line-height:14.25pt;background:rgb(31,31,31)"><span style="font-size:10.5pt;font-family:Consolas;color:rgb(86,156,214)">end subroutine</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"> </span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(220,220,170)">GK_op</span><span style="font-size:10.5pt;font-family:Consolas;color:rgb(204,204,204)"><u></u><u></u></span></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span lang="EN-US">When I run with mpirun -np 3, it gives me similar error like previous, “Preconditioner number of local rows 27 does not equal input vector size 81”, (I changed my problem size so the numbers are different). <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US">Maybe simple equating input and output is not valid (due to ownership of different portion of a global vector). Then it may give me different error message. In fact my global dF has size 9*3*3*3, when running on 3 processors, the local dF has size 9*3*3*1 (I split my domain in z direction). The error message seems to suggest I am using a local dF rather than a global dF. And the output and input vector sizes seems to be different. Do I miss something here? <u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US">Best regards,<u></u><u></u></span></p><p class="MsoNormal"><span lang="EN-US">Yi<u></u><u></u></span></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><u></u> <u></u></p><div><div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"><p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> <br><b>Sent:</b> Sunday, March 10, 2024 6:45 PM<br><b>To:</b> Yi Hu <<a href="mailto:y.hu@mpie.de" target="_blank">y.hu@mpie.de</a>><br><b>Cc:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>; petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b> Re: [petsc-users] snes matrix-free jacobian fails with preconditioner shape mismatch<u></u><u></u></span></p></div></div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><u></u> <u></u></p><div><p class="MsoNormal"><br><br><u></u><u></u></p><blockquote style="margin-top:5pt;margin-bottom:5pt"><div><p class="MsoNormal">On Mar 10, 2024, at 10:16 AM, Yi Hu <<a href="mailto:y.hu@mpie.de" target="_blank">y.hu@mpie.de</a>> wrote:<u></u><u></u></p></div><p class="MsoNormal"><u></u> <u></u></p><div><div id="m_6768418666828824524pfptBannercw8qp5q"><div id="m_6768418666828824524pfptBannercw8qp5q"><div id="m_6768418666828824524pfptBannercw8qp5q"><p class="MsoNormal">This Message Is From an External Sender<u></u><u></u></p></div><div id="m_6768418666828824524pfptBannercw8qp5q"><p class="MsoNormal">This message came from outside your organization.<u></u><u></u></p></div></div></div><p style="font-variant-caps:normal;text-align:start;word-spacing:0px"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif">Dear Mark,<u></u><u></u></span></p><p style="font-variant-caps:normal;text-align:start;word-spacing:0px"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif">Thanks for your reply. I see this mismatch. In fact my global DoF is 324. It seems like I always get the local size = global Dof / np^2, np is my processor number. By the way, I used DMDASNESsetFunctionLocal() to set my form function. Is it eligible to mix DMDASNESsetFunctionLocal() and a native SNESSetJacobian()?<u></u><u></u></span></p></div></blockquote><div><p class="MsoNormal"><u></u> <u></u></p></div><p class="MsoNormal">Yes<u></u><u></u></p></div><div><p class="MsoNormal"><br><br><u></u><u></u></p><blockquote style="margin-top:5pt;margin-bottom:5pt"><div><p style="font-variant-caps:normal;text-align:start;word-spacing:0px"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif">Best,<u></u><u></u></span></p><p style="font-variant-caps:normal;text-align:start;word-spacing:0px"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif">Yi<u></u><u></u></span></p><div><p class="MsoNormal"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif">On 3/10/24 13:55, Mark Adams wrote:<u></u><u></u></span></p></div><blockquote style="margin-top:5pt;margin-bottom:5pt;font-variant-caps:normal;text-align:start;word-spacing:0px"><div><div><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">It looks like your input vector is the global vector, size 162, and the local matrix size is 81.<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Mark<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: Preconditioner number of local rows 81 does not equal <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">input vector size 162<u></u><u></u></span></pre></div><p class="MsoNormal"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif"><u></u> <u></u></span></p><div><div><p class="MsoNormal"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif">On Sun, Mar 10, 2024 at 7:21 AM Yi Hu <<a href="mailto:y.hu@mpie.de" target="_blank">y.hu@mpie.de</a>> wrote:<u></u><u></u></span></p></div><blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"><div><div style="border-right:none;border-bottom:none;border-left:none;border-top:3pt solid rgb(144,164,174);padding:6pt 0in 0in;margin-top:12pt;margin-bottom:12pt;border-radius:4px;min-width:200px" id="m_6768418666828824524m_-4788138792073467353pfptBanner2satfg2"><div style="margin-bottom:0.75pt;float:left" id="m_6768418666828824524m_-4788138792073467353pfptBanner2satfg2"><div id="m_6768418666828824524m_-4788138792073467353pfptBanner2satfg2"><p class="MsoNormal" style="line-height:13.5pt;background:rgb(208,216,220)"><b><span style="font-size:10.5pt;font-family:Arial,sans-serif">This Message Is From an External Sender<span class="m_6768418666828824524apple-converted-space"> </span><u></u><u></u></span></b></p></div><div style="margin-top:1.5pt" id="m_6768418666828824524m_-4788138792073467353pfptBanner2satfg2"><p class="MsoNormal" style="line-height:13.5pt;background:rgb(208,216,220)"><span style="font-size:9pt;font-family:Arial,sans-serif">This message came from outside your organization.<span class="m_6768418666828824524apple-converted-space"> </span><u></u><u></u></span></p></div></div><div><p class="MsoNormal" style="line-height:0%;background:rgb(208,216,220)"><span style="font-size:1pt;font-family:Helvetica,sans-serif"> <u></u><u></u></span></p></div></div><div><pre style="white-space:pre-wrap"><span style="font-size:13.5pt;font-family:Arial,sans-serif">Dear petsc team,<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">I implemented a matrix-free jacobian, and it can run sequentially. But <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">running parallel I got the pc error like this (running with mpirun -np <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">2, only error from rank1 is presented here)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: --------------------- Error Message <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">--------------------------------------------------------------<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: Nonconforming object sizes<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: Preconditioner number of local rows 81 does not equal <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">input vector size 162<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: See <a href="https://urldefense.us/v3/__https:/petsc.org/release/faq/__;!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0PlyUrtdfCA$" target="_blank">https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0PlyUrtdfCA$</a> for trouble shooting.<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: Petsc Release Version 3.17.3, Jun 29, 2022<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: /home/yi/workspace/DAMASK_yi/bin/DAMASK_grid on a <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">arch-linux-c-opt named carbon-x1 by yi Sun Mar 10 12:01:46 2024<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: Configure options --download-fftw --download-hdf5 <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">--with-hdf5-fortran-bindings --download-fblaslapack --download-chaco <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">--download-hypre --download-metis --download-mumps --download-parmetis <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">--download-scalapack --download-suitesparse --download-superlu <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">--download-superlu_dist --download-triangle --download-zlib <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">--download-cmake --with-cxx-dialect=C++11 --with-c2html=0 <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">--with-debugging=0 --with-ssl=0 --with-x=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">FOPTFLAGS=-O3<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: #1 PCApply() at <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">/home/yi/App/petsc-3.17.3/src/ksp/pc/interface/precon.c:424<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: #2 KSP_PCApply() at <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">/home/yi/App/petsc-3.17.3/include/petsc/private/kspimpl.h:376<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: #3 KSPInitialResidual() at <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">/home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itres.c:64<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: #4 KSPSolve_GMRES() at <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">/home/yi/App/petsc-3.17.3/src/ksp/ksp/impls/gmres/gmres.c:242<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: #5 KSPSolve_Private() at <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">/home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itfunc.c:902<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: #6 KSPSolve() at <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">/home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itfunc.c:1078<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: #7 SNESSolve_NEWTONLS() at <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">/home/yi/App/petsc-3.17.3/src/snes/impls/ls/ls.c:222<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: #8 SNESSolve() at <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">/home/yi/App/petsc-3.17.3/src/snes/interface/snes.c:4756<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">[1]PETSC ERROR: #9 User provided function() at User <a>file:0</a><u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">However, from snes matrix-free documentation <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">(<a href="https://urldefense.us/v3/__https:/petsc.org/release/manual/snes/*matrix-free-methods__;Iw!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0Ply6ZbywOw$" target="_blank">https://urldefense.us/v3/__https://petsc.org/release/manual/snes/*matrix-free-methods__;Iw!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0Ply6ZbywOw$</a>), it is said <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">matrix-free is used with pcnone. So I assume it would not apply <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">preconditioner, but it did use preconditioning probably the same as my <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">matrix-free shell matrix. Here is how i initialize my shell matrix and <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">the corresponding customized multiplication.<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call MatCreateShell(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,&<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">int(9*product(cells(1:2))*cells3,pPETSCINT),&<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">int(9*product(cells(1:2))*cells3,pPETSCINT),&<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> F_PETSc,Jac_PETSc,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call MatShellSetOperation(Jac_PETSc,MATOP_MULT,GK_op,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call SNESSetDM(SNES_mech,DM_mech,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">SNESSetJacobian(SNES_mech,Jac_PETSc,Jac_PETSc,PETSC_NULL_FUNCTION,0,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call SNESGetKSP(SNES_mech,ksp,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call PCSetType(pc,PCNONE,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">And my GK_op is like<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">subroutine GK_op(Jac,dF_global,output_local,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> DM :: dm_local<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> Vec :: dF_global, dF_local, output_local<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> Mat :: Jac<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> PetscErrorCode :: err_PETSc<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> real(pREAL), pointer,dimension(:,:,:,:) :: dF_scal, output_scal<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> real(pREAL), dimension(3,3,cells(1),cells(2),cells3) :: &<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> dF<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> real(pREAL), dimension(3,3,cells(1),cells(2),cells3) :: &<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> output<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call SNESGetDM(SNES_mech,dm_local,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call DMGetLocalVector(dm_local,dF_local,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">DMGlobalToLocalBegin(dm_local,dF_global,INSERT_VALUES,dF_local,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">DMGlobalToLocalEnd(dm_local,dF_global,INSERT_VALUES,dF_local,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call DMDAVecGetArrayReadF90(dm_local,dF_local,dF_scal,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> dF = reshape(dF_scal, [3,3,cells(1),cells(2),cells3])<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">.......<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call DMDAVecRestoreArrayF90(dm_local,output_local,output_scal,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> CHKERRQ(err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> call DMDAVecRestoreArrayF90(dm_local,dF_local,dF_scal,err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> CHKERRQ(err_PETSc)<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">end subroutine GK_op<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">I checked my cells3, it corresponds to my local size, and it seems the <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">local size of dF_local is ok.<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">I am a bit lost here to find the reason for the preconditioner bug. <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Could you help me on this? Thanks.<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Best regards,<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Yi<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">-------------------------------------------------<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Stay up to date and follow us on LinkedIn, Twitter and YouTube.<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Max-Planck-Institut für Eisenforschung GmbH<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Max-Planck-Straße 1<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">D-40237 Düsseldorf<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Handelsregister B 2533 <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Amtsgericht Düsseldorf<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Geschäftsführung<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Prof. Dr. Gerhard Dehm<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Prof. Dr. Jörg Neugebauer<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Prof. Dr. Dierk Raabe<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Dr. Kai de Weldige<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"> <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Ust.-Id.-Nr.: DE 11 93 58 514 <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Steuernummer: 105 5891 1000<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Please consider that invitations and e-mails of our institute are <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">only valid if they end with …@<a href="https://urldefense.us/v3/__http:/mpie.de__;!!G_uCfscf7eWS!eslPb6ZEnby8M7NKRe_U_RT-95w2J8O2ngsc8rmrQVlMpEcB6ZTKlN2g65crXCWv7D2F9ubkzYI5nDpoeQ$" target="_blank">mpie.de</a>. <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">If you are not sure of the validity please contact <a href="mailto:rco@mpie.de" target="_blank">rco@mpie.de</a><u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">aus unserem Haus nur mit der Endung …@<a href="https://urldefense.us/v3/__http:/mpie.de__;!!G_uCfscf7eWS!eslPb6ZEnby8M7NKRe_U_RT-95w2J8O2ngsc8rmrQVlMpEcB6ZTKlN2g65crXCWv7D2F9ubkzYI5nDpoeQ$" target="_blank">mpie.de</a> gültig sind. <u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">In Zweifelsfällen wenden Sie sich bitte an <a href="mailto:rco@mpie.de" target="_blank">rco@mpie.de</a><u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif">-------------------------------------------------<u></u><u></u></span></pre><pre><span style="font-size:13.5pt;font-family:Arial,sans-serif"><u></u> <u></u></span></pre></div></div></blockquote></div></div></blockquote><p class="MsoNormal"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif"><br style="font-variant-caps:normal;text-align:start;word-spacing:0px"><br></span><u></u><u></u></p><div class="MsoNormal" align="center" style="text-align:center"><hr size="2" width="100%" align="center"></div><p class="MsoNormal"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif">-------------------------------------------------<br>Stay up to date and follow us on LinkedIn, Twitter and YouTube.<br><br>Max-Planck-Institut für Eisenforschung GmbH<br>Max-Planck-Straße 1<br>D-40237 Düsseldorf<br> <br>Handelsregister B 2533 <br>Amtsgericht Düsseldorf<br> <br>Geschäftsführung<br>Prof. Dr. Gerhard Dehm<br>Prof. Dr. Jörg Neugebauer<br>Prof. Dr. Dierk Raabe<br>Dr. Kai de Weldige<br> <br>Ust.-Id.-Nr.: DE 11 93 58 514 <br>Steuernummer: 105 5891 1000<br><br><br>Please consider that invitations and e-mails of our institute are <br>only valid if they end with …@<a href="https://urldefense.us/v3/__http://mpie.de__;!!G_uCfscf7eWS!alXwgSQjxXmDUJsTS-lIe1vkm8LQxT514jCJ_Liu-cmapBf7mhL3YJ6kS0ncGzUOmFgRMYw_uqSHSbe5e_QDrnQ$" target="_blank">mpie.de</a>. <br>If you are not sure of the validity please contact </span><a href="mailto:rco@mpie.de" target="_blank"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif">rco@mpie.de</span></a><span style="font-size:13.5pt;font-family:Helvetica,sans-serif"><br><br>Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails<br>aus unserem Haus nur mit der Endung …@<a href="https://urldefense.us/v3/__http://mpie.de__;!!G_uCfscf7eWS!alXwgSQjxXmDUJsTS-lIe1vkm8LQxT514jCJ_Liu-cmapBf7mhL3YJ6kS0ncGzUOmFgRMYw_uqSHSbe5e_QDrnQ$" target="_blank">mpie.de</a> gültig sind. <br>In Zweifelsfällen wenden Sie sich bitte an </span><a href="mailto:rco@mpie.de" target="_blank"><span style="font-size:13.5pt;font-family:Helvetica,sans-serif">rco@mpie.de</span></a><span style="font-size:13.5pt;font-family:Helvetica,sans-serif"><br>-------------------------------------------------</span><u></u><u></u></p></div></blockquote></div><p class="MsoNormal"><u></u> <u></u></p></div><br>
<br>
<hr>
-------------------------------------------------<br>
Stay up to date and follow us on LinkedIn, Twitter and YouTube.<br>
<br>
Max-Planck-Institut für Eisenforschung GmbH<br>
Max-Planck-Straße 1<br>
D-40237 Düsseldorf<br>
<br>
Handelsregister B 2533 <br>
Amtsgericht Düsseldorf<br>
<br>
Geschäftsführung<br>
Prof. Dr. Gerhard Dehm<br>
Prof. Dr. Jörg Neugebauer<br>
Prof. Dr. Dierk Raabe<br>
Dr. Kai de Weldige<br>
<br>
Ust.-Id.-Nr.: DE 11 93 58 514 <br>
Steuernummer: 105 5891 1000<br>
<br>
<br>
Please consider that invitations and e-mails of our institute are <br>
only valid if they end with …@<a href="https://urldefense.us/v3/__http://mpie.de__;!!G_uCfscf7eWS!alXwgSQjxXmDUJsTS-lIe1vkm8LQxT514jCJ_Liu-cmapBf7mhL3YJ6kS0ncGzUOmFgRMYw_uqSHSbe5e_QDrnQ$" target="_blank">mpie.de</a>. <br>
If you are not sure of the validity please contact <a href="mailto:rco@mpie.de" target="_blank">rco@mpie.de</a><br>
<br>
Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails<br>
aus unserem Haus nur mit der Endung …@<a href="https://urldefense.us/v3/__http://mpie.de__;!!G_uCfscf7eWS!alXwgSQjxXmDUJsTS-lIe1vkm8LQxT514jCJ_Liu-cmapBf7mhL3YJ6kS0ncGzUOmFgRMYw_uqSHSbe5e_QDrnQ$" target="_blank">mpie.de</a> gültig sind. <br>
In Zweifelsfällen wenden Sie sich bitte an <a href="mailto:rco@mpie.de" target="_blank">rco@mpie.de</a><br>
-------------------------------------------------<br>
</div></div></blockquote></div>