<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">
<div dir="ltr">Myriam:<br>
</div>
<div>Thank you very much for providing these results!</div>
<div>I have put effort to accelerate execution time and avoid using global sizes in PtAP, for which the algorithm of transpose of P_local and P_other likely doubles the memory usage. I'll try to investigate why it becomes unscalable.</div>
<div>Hong</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi,</p>
<p>you'll find the new scaling attached (green line). I used the version 3.11 and the four scalability options :<br>
-matptap_via scalable<br>
-inner_diag_matmatmult_via scalable<br>
-inner_offdiag_matmatmult_via scalable<br>
-mat_freeintermediatedatastructures</p>
<p>The scaling is much better! The code even uses less memory for the smallest cases. There is still an increase for the larger one.
<br>
</p>
<p>With regard to the time scaling, I used KSPView and LogView on the two previous scalings (blue and yellow lines) but not on the last one (green line). So we can't really compare them, am I right? However, we can see that the new time scaling looks quite
good. It slightly increases from ~8s to ~27s. <br>
</p>
<p>Unfortunately, the computations are expensive so I would like to avoid re-run them if possible. How relevant would be a proper time scaling for you?
<br>
</p>
<p>Myriam<br>
</p>
<br>
<div class="m_-4364359315279719822gmail-m_-6245019727744503832moz-cite-prefix">Le 04/12/19 à 18:18, Zhang, Hong a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Myriam :<br>
</div>
<div>Thanks for your effort. It will help us improve PETSc.</div>
<div>Hong</div>
<div><br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi all,<br>
<br>
I used the wrong script, that's why it diverged... Sorry about that. <br>
I tried again with the right script applied on a tiny problem (~200<br>
elements). I can see a small difference in memory usage (gain ~ 1mB).<br>
when adding the -mat_freeintermediatestructures option. I still have to<br>
execute larger cases to plot the scaling. The supercomputer I am used to<br>
run my jobs on is really busy at the moment so it takes a while. I hope<br>
I'll send you the results on Monday.<br>
<br>
Thanks everyone,<br>
<br>
Myriam<br>
<br>
<br>
Le 04/11/19 à 06:01, Jed Brown a écrit :<br>
> "Zhang, Hong" <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> writes:<br>
><br>
>> Jed:<br>
>>>> Myriam,<br>
>>>> Thanks for the plot. '-mat_freeintermediatedatastructures' should not affect solution. It releases almost half of memory in C=PtAP if C is not reused.<br>
>>> And yet if turning it on causes divergence, that would imply a bug.<br>
>>> Hong, are you able to reproduce the experiment to see the memory<br>
>>> scaling?<br>
>> I like to test his code using an alcf machine, but my hands are full now. I'll try it as soon as I find time, hopefully next week.<br>
> I have now compiled and run her code locally.<br>
><br>
> Myriam, thanks for your last mail adding configuration and removing the<br>
> MemManager.h dependency. I ran with and without<br>
> -mat_freeintermediatedatastructures and don't see a difference in<br>
> convergence. What commands did you run to observe that difference?<br>
<br>
-- <br>
Myriam Peyrounette<br>
CNRS/IDRIS - HLST<br>
--<br>
<br>
<br>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre class="m_-4364359315279719822gmail-m_-6245019727744503832moz-signature" cols="72">--
Myriam Peyrounette
CNRS/IDRIS - HLST
--
</pre>
</div>
</blockquote>
</div>
</div>
</body>
</html>