<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Varun,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Good to know it works. <span style="color:rgb(32, 31, 30);font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, "system-ui", Roboto, "Helvetica Neue", sans-serif;font-size:15px;background-color:rgb(255, 255, 255);display:inline !important">FactorSymbolic
function is still being called twice, but the 2nd call is a no-op, thus it still appears in '-log_view'. I made changes in the low level of mumps routine, not within PCSetUp() because I feel your use case is limited to mumps, not other matrix package solvers. </span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hong</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Varun Hiremath <varunhiremath@gmail.com><br>
<b>Sent:</b> Wednesday, January 19, 2022 2:44 AM<br>
<b>To:</b> Zhang, Hong <hzhang@mcs.anl.gov><br>
<b>Cc:</b> Peder Jørgensgaard Olesen via petsc-users <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] PETSc MUMPS interface</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi Hong,
<div><br>
</div>
<div>Thanks, I tested your branch and I think it is working fine. I don't see any increase in runtime, however with -log_view I see that the MatLUFactorSymbolic function is still being called twice, so is this expected? Is the second call a no-op?</div>
<div><br>
</div>
<div>$ ./ex52.o -use_mumps_lu -print_mumps_memory -log_view | grep MatLUFactorSym</div>
<div>MatLUFactorSym 2 1.0 4.4411e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0<br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Varun</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Mon, Jan 17, 2022 at 7:49 PM Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov">hzhang@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Varun,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I created a branch hzhang/feature-mumps-mem-estimate,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
see <a href="https://gitlab.com/petsc/petsc/-/merge_requests/4727" id="x_gmail-m_-2493930565457682428LPNoLPOWALinkPreview" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/4727</a></div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
You may give it a try <span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">and let me know if this is what you want.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">src/ksp/ksp/tutorials/ex52.c is an example. <br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hong</div>
<div id="x_gmail-m_-2493930565457682428appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-2493930565457682428divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Varun Hiremath <<a href="mailto:varunhiremath@gmail.com" target="_blank">varunhiremath@gmail.com</a>><br>
<b>Sent:</b> Monday, January 17, 2022 12:41 PM<br>
<b>To:</b> Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>><br>
<b>Cc:</b> Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>>; Peder Jørgensgaard Olesen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] PETSc MUMPS interface</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi Hong,<br>
<br>
Thanks for looking into this. Here is the workflow that I might use:<br>
<br>
MatLUFactorSymbolic(F,A,perm,iperm,&info);<br>
<br>
// get memory estimates from MUMPS e.g. INFO(3), INFOG(16), INFOG(17)<br>
// find available memory on the system e.g. RAM size<br>
if (estimated_memory > available_memory)<br>
{<br>
// inform and stop; or<br>
// switch MUMPS to out-of-core factorization<br>
ICNTL(22) = 1;<br>
}<br>
else<br>
{<br>
// set appropriate settings for in-core factorization<br>
}<br>
<br>
// Now we call the solve and inside if MatLUFactorSymbolic is already called then it should be skipped<br>
EPSSolve(eps);<br>
<br>
Thanks,<br>
Varun<br>
</div>
<br>
<div>
<div dir="ltr">On Mon, Jan 17, 2022 at 9:18 AM Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline">Varun,</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline">I am trying to find a way to enable you to <span style="color:rgb(0,0,0); font-size:14px; background-color:rgb(255,255,255); display:inline">switch options
after <span style="font-size:14.6667px; background-color:rgb(255,255,255); display:inline">MatLUFactorSymbolic(). A hack is modifying the flag 'mumps->matstruc' inside MatLUFactorSymbolic_AIJMUMPS() and MatFactorNumeric_MUMPS().</span></span></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline"><span style="color:rgb(0,0,0); font-size:14px; background-color:rgb(255,255,255); display:inline"><span style="font-size:14.6667px; background-color:rgb(255,255,255); display:inline"><br>
</span></span></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline"><span style="color:rgb(0,0,0); font-size:14px; background-color:rgb(255,255,255); display:inline"><span style="font-size:14.6667px; background-color:rgb(255,255,255); display:inline">My
understanding of what you want is:</span></span></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(32,31,30); font-size:15px; background-color:rgb(255,255,255); display:inline"><span style="color:rgb(0,0,0); font-size:14px; background-color:rgb(255,255,255); display:inline"><span style="font-size:14.6667px; background-color:rgb(255,255,255); display:inline">
// collect mumps memory info
<div> ...</div>
<div> MatLUFactorSymbolic(F,A,perm,iperm,&info);</div>
<div> printMumpsMemoryInfo(F);</div>
//---------<br>
</span></span></span></div>
<div style="color:rgb(0,0,0)"><span style="font-size:14.6667px">if (memory is available) {</span></div>
<div style="color:rgb(0,0,0)"><span style="font-size:14.6667px"> EPSSolve(eps); --> skip calling of <span style="background-color:rgb(255,255,255); display:inline">MatLUFactorSymbolic()</span></span></div>
<div style="color:rgb(0,0,0)"><span style="font-size:14.6667px"><span style="background-color:rgb(255,255,255); display:inline">} else {</span></span></div>
<div style="color:rgb(0,0,0)"><span style="font-size:14.6667px"><span style="background-color:rgb(255,255,255); display:inline"> //<span style="font-size:14px; background-color:rgb(255,255,255); display:inline">out-of-core (OOC) option in MUMPS</span></span></span></div>
<div style="color:rgb(0,0,0)"><span style="font-size:14.6667px"><span style="background-color:rgb(255,255,255); display:inline"><span style="font-size:14px; background-color:rgb(255,255,255); display:inline">}</span></span></span></div>
<div style="color:rgb(0,0,0)"><span style="font-size:14.6667px"><span style="background-color:rgb(255,255,255); display:inline"><span style="font-size:14px; background-color:rgb(255,255,255); display:inline"><br>
</span></span></span></div>
<div style="color:rgb(0,0,0)"><span style="font-size:14.6667px"><span style="background-color:rgb(255,255,255); display:inline"><span style="font-size:14px; background-color:rgb(255,255,255); display:inline">Am I correct? I'll let you know once I work out a
solution.</span></span></span></div>
<div style="color:rgb(0,0,0)"><span style="font-size:14.6667px"><span style="background-color:rgb(255,255,255); display:inline"><span style="font-size:14px; background-color:rgb(255,255,255); display:inline">Hong</span></span></span></div>
<div id="x_gmail-m_-2493930565457682428x_gmail-m_-6920542854740158255appendonsend">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-2493930565457682428x_gmail-m_-6920542854740158255divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Varun Hiremath <<a href="mailto:varunhiremath@gmail.com" target="_blank">varunhiremath@gmail.com</a>><br>
<b>Sent:</b> Sunday, January 16, 2022 10:10 PM<br>
<b>To:</b> Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>><br>
<b>Cc:</b> Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>>; Peder Jørgensgaard Olesen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] PETSc MUMPS interface</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi Jose, Hong,<br>
<br>
Thanks for the explanation. I have verified using -log_view that <font face="arial, sans-serif">
MatLUFactorSymbolic </font>is indeed getting called twice.<br>
<br>
Hong, we use MUMPS solver for other things, and we typically run the symbolic analysis first and get memory estimates to ensure that we have enough memory available to run the case. If the available memory is not enough, we can stop or switch to the out-of-core
(OOC) option in MUMPS. We wanted to do the same when using MUMPS via SLEPc/PETSc. Please let me know if there are other ways of getting these memory stats and switching options during runtime with PETSc.
<div>Appreciate your help!<br>
<br>
Thanks,<br>
Varun<br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Sun, Jan 16, 2022 at 4:01 PM Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:14.6667px; background-color:rgb(255,255,255); display:inline">Varun,</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:14.6667px; background-color:rgb(255,255,255); display:inline">I believe Jose is correct. You may verify it by running your code with option '-log_view', then check the number of calls to MatLUFactorSym. </span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:14.6667px; background-color:rgb(255,255,255); display:inline"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:14.6667px; background-color:rgb(255,255,255); display:inline">I guess I can add a flag in <span style="background-color:rgb(255,255,255); display:inline">PCSetUp() to check if <span style="background-color:rgb(255,255,255); display:inline">user
has already called MatLUFactorSymbolic() and wants to skip it. Normally, users simply allocate sufficient memory in the symbolic factorization. Why do you want to check it?</span></span></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:14.6667px; background-color:rgb(255,255,255); display:inline"><span style="background-color:rgb(255,255,255); display:inline"><span style="background-color:rgb(255,255,255); display:inline">Hong</span></span></span></div>
<div id="x_gmail-m_-2493930565457682428x_gmail-m_-6920542854740158255x_gmail-m_-8706958557292409506appendonsend">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-2493930565457682428x_gmail-m_-6920542854740158255x_gmail-m_-8706958557292409506divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>><br>
<b>Sent:</b> Sunday, January 16, 2022 5:11 AM<br>
<b>To:</b> Varun Hiremath <<a href="mailto:varunhiremath@gmail.com" target="_blank">varunhiremath@gmail.com</a>><br>
<b>Cc:</b> Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>>; Peder Jørgensgaard Olesen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] PETSc MUMPS interface</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>Hong may give a better answer, but if you look at PCSetUp_LU() <a href="https://petsc.org/main/src/ksp/pc/impls/factor/lu/lu.c.html#PCSetUp_LU" target="_blank">
https://petsc.org/main/src/ksp/pc/impls/factor/lu/lu.c.html#PCSetUp_LU</a> you will see that MatLUFactorSymbolic() is called unconditionally during the first PCSetUp(). Currently there is no way to check if the user has already called MatLUFactorSymbolic().<br>
<br>
Jose<br>
<br>
<br>
> El 16 ene 2022, a las 10:40, Varun Hiremath <<a href="mailto:varunhiremath@gmail.com" target="_blank">varunhiremath@gmail.com</a>> escribió:<br>
> <br>
> Hi Hong,<br>
> <br>
> Thank you, this is very helpful! <br>
> <br>
> Using this method I am able to get the memory estimates before the actual solve, however, I think my code may be causing the symbolic factorization to be run twice. Attached is my code where I am using SLEPc to compute eigenvalues, and I use MUMPS for factorization.
I have commented above the code that computes the memory estimates, could you please check and tell me if this would cause the symbolic factor to be computed twice (a second time inside EPSSolve?), as I am seeing a slight increase in the overall computation
time?<br>
> <br>
> Regards,<br>
> Varun<br>
> <br>
> On Wed, Jan 12, 2022 at 7:58 AM Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:<br>
> PCFactorSetMatSolverType(pc,MATSOLVERMUMPS);<br>
> PCFactorSetUpMatSolverType(pc); <br>
> PCFactorGetMatrix(pc,&F);<br>
> <br>
> MatLUFactorSymbolic(F,A,...)<br>
> You must provide row and column permutations etc, petsc/src/mat/tests/ex125.c may give you a clue on how to get these inputs.<br>
> <br>
> Hong<br>
> <br>
> <br>
> From: petsc-users <<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>> on behalf of Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
> Sent: Wednesday, January 12, 2022 9:03 AM<br>
> To: Varun Hiremath <<a href="mailto:varunhiremath@gmail.com" target="_blank">varunhiremath@gmail.com</a>><br>
> Cc: Peder Jørgensgaard Olesen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
> Subject: Re: [petsc-users] PETSc MUMPS interface<br>
> <br>
> Calling PCSetUp() before KSPSetUp()?<br>
> <br>
> --Junchao Zhang<br>
> <br>
> <br>
> On Wed, Jan 12, 2022 at 3:00 AM Varun Hiremath <<a href="mailto:varunhiremath@gmail.com" target="_blank">varunhiremath@gmail.com</a>> wrote:<br>
> Hi All,<br>
> <br>
> I want to collect MUMPS memory estimates based on the initial symbolic factorization analysis before the actual numerical factorization starts to check if the estimated memory requirements fit the available memory.<br>
> <br>
> I am following the steps from <a href="https://petsc.org/main/src/ksp/ksp/tutorials/ex52.c.html" target="_blank">
https://petsc.org/main/src/ksp/ksp/tutorials/ex52.c.html</a><br>
> <br>
> PCFactorSetMatSolverType(pc,MATSOLVERMUMPS);<br>
> PCFactorSetUpMatSolverType(pc); <br>
> PCFactorGetMatrix(pc,&F);<br>
> <br>
> KSPSetUp(ksp);<br>
> MatMumpsGetInfog(F,...)<br>
> <br>
> But it appears KSPSetUp calls both symbolic and numerical factorization. So is there some other way to get these statistics before the actual factorization starts?<br>
> <br>
> Thanks,<br>
> Varun<br>
> <slepc_eps_mumps_test.cpp><br>
<br>
</div>
</span></font></div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>