<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div><div class=""><div class="">DMDASetBlockFills() or DMDASetBlockFillsSparse() (they are just different ways of providing the same information) will help you here enormously, your type of case is exactly what they were designed for.</div></div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><blockquote type="cite" class=""><div class="elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div></blockquote></div><div><br class=""><blockquote type="cite" class=""><div class="">On Sep 7, 2022, at 10:47 AM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Barry and Hong,<br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Thank you.<span class="Apple-converted-space"> </span><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">There are 26 components at each grid and there are not fully coupled in terms of stiff functions. Mutual coupling is among about 6 components.<span class="Apple-converted-space"> </span><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">I would prefer not using matrix-free since the Jacobina is not difficult to calculate and only up to 10 non-zeros at each row. I'll try DMDASetBlockFills() or DMDASetBlockFillsSparse() and see how they can reduce the memory usage.</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Jiannan<br class=""></div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 977.046875px;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Tuesday, September 6, 2022 11:33 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>><br class=""><b class="">Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Using matrix-free with KSP</font><div class=""> </div></div><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; line-break: after-white-space;"><p style="margin-top: 0px; margin-bottom: 0px;" class=""></p><div style="background-color: rgb(255, 202, 202); border: 1pt hidden rgb(255, 202, 202); padding: 2pt; font-size: 12pt; line-height: 16pt; font-family: Calibri; text-align: left;" class=""><span style="font-weight: bold;" class="">CAUTION:</span><span class="Apple-converted-space"> </span>This email was sent from outside the UMass Lowell network.</div><br class=""><p style="margin-top: 0px; margin-bottom: 0px;" class=""></p><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Sep 6, 2022, at 11:00 PM, Tu, Jiannan <<a href="mailto:Jiannan_Tu@uml.edu" class="">Jiannan_Tu@uml.edu</a>> wrote:</div><br class="x_Apple-interchange-newline"><div class=""><div class="x_WordSection1" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div class=""><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;">I am using TS IMEX to solve a large DAE system. The DAE is obtained by applying finite FV method to 3-D multi-specie ion/neutral fluids equations with magnetic induction equation. The Jacobian for stiff part is formed by using MatSetValuesStencil(). The Jacobian matrix is very sparse, no more than 10 non-zeros on each row. MatSetValuesStencil requires local to global mapping by calling ISLocaltoGlobalMapping(). Could you please instruct me how to use local to global mapping?<span class="x_Apple-converted-space"> </span></span></div></div></div></div></blockquote><div class=""><br class=""></div> DMDA automatically sets up the <span class="" style="font-size: 12pt; font-family: Calibri, sans-serif;">ISLocaltoGlobalMapping() so you should not need to.</span><blockquote type="cite" class=""><div class=""><div class="x_WordSection1" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div class=""><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;"></span></div><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;"> </span></div><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;">I also tried using DMCreateMatrix() to create the Jacobian. While local to global mapping is not necessary, the matrix takes too much memory and requires 64-bit indices. I would prefer to take the advantage of sparsity of the Jacobian, pre-allocate the matrix to use as less as possible memory so that the code can be run on a multi-core desktop.</span></div></div></div></div></blockquote><div class=""><br class=""></div><div class=""> If using matrix-free does not work for you because the linear solves do not converge or converge too slowly. Then you might be able to decrease the memory used in the matrix. The DMCreateMatrix() does take advantage of sparsity and tries to preallocate only what it needs. Often what it preallocates is the best possible, but for multicomponent problems it has to assume there is full coupling within all the degrees of freedom that live at each at each grid point. How many components live at each grid point in your model and are they not all coupled to each other in the equations? If they are not fully coupled you can use either DMDASetBlockFills() or DMDASetBlockFillsSparse() to indicate the reduced coupling that actually exists for you model.</div></div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div class="x_WordSection1" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div class=""><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;"></span></div><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;"> </span></div><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;">Thank you very much for your advice.</span></div><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;"> </span></div><div class="" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span class="" style="font-size: 12pt;">Jiannan</span></div></div></div></div></blockquote></div></div></div></div></blockquote></div><br class=""></body></html>