<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.defaultfonthxmailstyle
{mso-style-name:defaultfonthxmailstyle;}
span.defaultfonthxmailstyle0
{mso-style-name:defaultfonthxmailstyle0;}
span.xapple-converted-space
{mso-style-name:xapple-converted-space;}
span.EmailStyle24
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt">Barry,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Thank you.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Yes, the DMDA is created with periodic BC. But it might not be the periodic BC since in azimuthal direction, the ghost grid -1 is actually grid Nmax, and Nmax+1 is grid 0.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Still the problem is how to properly pre-allocate matrix. Otherwise either “Augument out of range” error or code stuck at inserting values if MatSetOption() is called.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Jiannan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Barry Smith <bsmith@petsc.dev> <br>
<b>Sent:</b> Sunday, September 11, 2022 12:10 AM<br>
<b>To:</b> Tu, Jiannan <Jiannan_Tu@uml.edu><br>
<b>Cc:</b> petsc-users <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Using matrix-free with KSP<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none #FFCACA 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt;border-style:hidden">
<p class="MsoNormal" style="line-height:16.0pt;background:#FFCACA"><b><span style="font-size:12.0pt;color:black">CAUTION:</span></b><span style="font-size:12.0pt;color:black"> This email was sent from outside the UMass Lowell network.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> Have you created the DMDA with the periodic boundary conditions arguments? Or are your boundary conditions not simply periodic?
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Compare DMCreateMatrix_DA_3d_MPIAIJ() and DMCreateMatrix_DA_3d_MPIAIJ_Fill() in src/dm/impls/da/fdda.c I think they both attempt to handle the periodic case.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Sep 10, 2022, at 10:14 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Barry,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I think I understand how to set values for the intra-grid coupling. But I am not clear about the way to set values for inter-grid coupling. When a component couples to itself at<span class="apple-converted-space"> </span></span>above,
below, right, left, front and back grids, diagonal is set as 1 for the second array of DMDASetBlockFills(). How about if involving more grids? How does DMDA know where to allocate memory? In my case, a component couples to itself at the other end of the grid
because of periodic boundary condition. There is an error message “Augment out of range . Insert new nonzero at global row/column (26, 520526) into matrix” because of this far away coupling. The column index 520526 is related to the other end of the grid in
azimuthal direction.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Jiannan<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b>From:</b><span class="apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Saturday, September 10, 2022 1:10 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span>petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="line-height:16.0pt;background:#FFCACA"><b><span style="font-size:12.0pt;color:black">CAUTION:</span></b><span class="apple-converted-space"><span style="font-size:12.0pt;color:black"> </span></span><span style="font-size:12.0pt;color:black">This
email was sent from outside the UMass Lowell network.</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> If you are using DMCreateMatrix() then <span style="font-size:12.0pt">MatMPIAIJSetPreallocation is not needed, it is automatically handled by the DM.</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> It sounds like that the values you set in DMDASetBlockFills() did not include all the coupling, hence the preallocation was wrong, hence the time to insert values was too long.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> Barry</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">On Sep 10, 2022, at 8:16 AM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Just found MatMPIAIJSetPreallocation is needed. Now Jacobin insertion and assembly is fast.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<div>
<p class="MsoNormal"><b>From:</b><span class="apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Thursday, September 8, 2022 9:53 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="line-height:16.0pt;background:#FFCACA"><b><span style="font-size:12.0pt;color:black">CAUTION:</span></b><span class="apple-converted-space"><span style="font-size:12.0pt;color:black"> </span></span><span style="font-size:12.0pt;color:black">This
email was sent from outside the UMass Lowell network.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">On Sep 8, 2022, at 8:34 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>> wrote:<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Barry,</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thank you.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I set up these two arrays according to the non-zero pattern of the Jacobian. The DMDASetBlockFills() is then called after the call to DMCreateMatrix(da, &A).</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> Absolutely not! It must be called before DMCreateMatrix() so that the DM knows how to construct exactly the matrix you need.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Nevertheless, the program execution is still killed with exit code 9, which I assume is due to overuse of the memory. My desktop machine has 32 GB RAM.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I ran the code with a smaller mesh, 100x45x90, the result is the same. With that number of grids, 26 unknowns, and up to 10 non-zero at each matrix row, the estimated memory to store 100x45x90x26x10 elements
should be less than 1 GB (double precision). I am wondering why the matrix still takes too much memory. Maybe I have to use matrix-free?</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thank you,</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Jiannan</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<div>
<div>
<p class="MsoNormal"><b>From:</b><span class="apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Thursday, September 8, 2022 4:19 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="line-height:16.0pt;background:#FFCACA"><b><span style="font-size:12.0pt;color:black">CAUTION:</span></b><span class="apple-converted-space"><span style="font-size:12.0pt;color:black"> </span></span><span style="font-size:12.0pt;color:black">This
email was sent from outside the UMass Lowell network.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal">On Sep 8, 2022, at 3:39 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>> wrote:<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt">Barry,</span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt">Thank you very much for the detailed description.</span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt">So the first array, dfill, is for intra grids and the second one, ofill is for the inter-grid coupling?</span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt">In the case for inter-grid coupling, a component is only coupled to itself, say x in your example is coupled to itself on above, below, right, left, front, and back, how
to set values for the second array?</span></span><span class="apple-converted-space"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Then the off-diagonal one would just have values on its diagonal.<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Jiannan<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b>From:<span class="apple-converted-space"> </span></b><a href="mailto:bsmith@petsc.dev">Barry Smith</a><br>
<b>Sent:<span class="apple-converted-space"> </span></b>Thursday, September 8, 2022 2:12 PM<br>
<b>To:<span class="apple-converted-space"> </span></b><a href="mailto:Jiannan_Tu@uml.edu">Tu, Jiannan</a><br>
<b>Cc:<span class="apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
<b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] Using matrix-free with KSP<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="line-height:16.0pt;background:#FFCACA"><b><span style="font-size:12.0pt;color:black">CAUTION:</span></b><span class="apple-converted-space"><span style="font-size:12.0pt;color:black"> </span></span><span style="font-size:12.0pt;color:black">This
email was sent from outside the UMass Lowell network.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">On Sep 8, 2022, at 1:47 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>> wrote:<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle0"><span style="font-size:12.0pt">Barry,</span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle0"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle0"><span style="font-size:12.0pt">Thank you very much.</span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle0"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">DMDASetBlockFills() needs two input arrays specifying sparsity pattern. In my case, I have 26 components at each grid. Is it correct that I need to have two arrays of 26x26 elements with value 1 for coupling and 0 for no coupling?<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> Exactly<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Also how to determine when the value should be 1 or 0?<span class="apple-converted-space"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> That is problem dependent. Given paper-and-pencil you know for your stencil which components are coupled with other components at each grid point and which components are coupled with components at neighboring grid points. <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> Note the second array contains information about all the neighboring grid points. Above, below, right, left, front back. Normally the coupling is symmetric so the nonzero structures of those 6 blocks are identical. If, for you particular
problem, the nonzero structures of the six are not identical you use the union of the nonzero structures of the six of them.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> 2d example say you have x,y,z at each grid point and the inter-point coupling is x is connected to y and y is connect to x then the first array is<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">[ 1 1 0 ; 1 1; 0 ; 0 0 1]; the zeros are because x is not coupled to z and y is not coupled to z.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">For intra grid points, say x is coupled to z and itself, z is coupled to x and itself and y is not coupled to itself then<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">[1 0 1; 0 0 0; 1 0 1]; the middle row is zero because y is not coupled to anything and the zeros in the first and last are because x and z are not coupled to y.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Each component involves 7 stencils (i-1, j, k), (I, j, k), (i+1, j, k), (I, j-1, k), (I, j+1, k), (I, j, k-1), and (I, j, k+1), and couples with several other components.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Jiannan<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle0"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle0"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Sent from<span class="apple-converted-space"> </span><a href="https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986&data=05%7C01%7CJiannan_Tu%40uml.edu%7C68957aad881840dcf87f08da93ab77f6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637984661923188386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vWNBlC%2BrqWkX5Q0%2Bh6XVncZHaaqFAgShn2RthwcJem4%3D&reserved=0">Mail</a><span class="apple-converted-space"> </span>for
Windows<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle0"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b>From:<span class="apple-converted-space"> </span></b><a href="mailto:bsmith@petsc.dev">Barry Smith</a><br>
<b>Sent:<span class="apple-converted-space"> </span></b>Wednesday, September 7, 2022 11:53 AM<br>
<b>To:<span class="apple-converted-space"> </span></b><a href="mailto:Jiannan_Tu@uml.edu">Tu, Jiannan</a><br>
<b>Cc:<span class="apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
<b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] Using matrix-free with KSP<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="defaultfonthxmailstyle0"><span style="font-size:12.0pt"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="line-height:16.0pt;background:#FFCACA"><b><span style="font-size:12.0pt;color:black">CAUTION:</span></b><span class="apple-converted-space"><span style="font-size:12.0pt;color:black"> </span></span><span style="font-size:12.0pt;color:black">This
email was sent from outside the UMass Lowell network.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">On Sep 7, 2022, at 10:47 AM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>> wrote:<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Barry and Hong,</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thank you.<span class="apple-converted-space"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components.<span class="apple-converted-space"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can
reduce the memory usage.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Jiannan</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><E6340AADF8494DB7861EA4659E7713CE.png><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div id="divRplyFwdMsg">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b>From:</b><span class="apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday, September 6, 2022 11:33 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu">Jiannan_Tu@uml.edu</a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><span class="apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="line-height:16.0pt;background:#FFCACA"><b><span style="font-size:12.0pt;color:black">CAUTION:</span></b><span class="apple-converted-space"><span style="font-size:12.0pt;color:black"> </span></span><span style="font-size:12.0pt;color:black">This
email was sent from outside the UMass Lowell network.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
<br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif">On Sep 6, 2022, at 11:00 PM, Tu, Jiannan <</span><a href="mailto:Jiannan_Tu@uml.edu"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif">Jiannan_Tu@uml.edu</span></a><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif">>
wrote:</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part
is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to
global mapping?<span class="xapple-converted-space"> </span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> DMDA automatically sets up the </span><span style="font-size:12.0pt">ISLocaltoGlobalMapping() so you should not need to.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage
of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix.
The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that
live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate
the reduced coupling that actually exists for you model.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"> Barry</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica",sans-serif"><br>
<br>
<br>
<br>
<br>
<br>
<br>
</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thank you very much for your advice.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Jiannan</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>