<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Dear Matt and Hong,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thank you for your quick replies!
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">In answer to your question Matt, the application fails in the same way as with 5 checkpoints. I dont believe the RAM capacity to be a problem though because we are running this case on a cluster
with 64GB RAM per node, and we anticipate 0.1GB storage requirements for the 4 checkpoints.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">The case is being run in MPMD mode with the following command:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">aprun -n 72 /work/e01/e01/chri4903/bin/cascade-ng/checkpoints_gradients ../data/nl_adj_0-chkpts.ini -adjoint -vr "/
/
/
/
/sims/aaX-testcases/10-RotorStator/test/data/ic_0_chkpts.h5:/0000000000/field"
-vTarg "/
/
/
/
/sims/aaX-testcases/10-RotorStator/test/data/targ_0_chkpts.h5:/targ/field" -vMet "/
/
/
/
/sims/aaX-testcases/10-RotorStator/test/data/targ_0_chkpts.h5:/metric_diag/field" -ts_trajectory_dirname ./test_directory_0 -ts_trajectory_type memory
-ts_trajectory_max_cps_ram 4 -ts_trajectory_max_cps_disk 5000 -ts_trajectory_monitor : -n 80 /
/
/
/
/bin/cascade-ng/checkpoints_gradients ../data/nl_adj_1-chkpts.ini -adjoint -vr "/
/
/
/
/sims/aaX-testcases/10-RotorStator/test/data/ic_1_chkpts.h5:/0000000000/field"
-vTarg "/
/
/
/
/sims/aaX-testcases/10-RotorStator/test/data/targ_1_chkpts.h5:/targ/field" -vMet "/
/
/
/
/sims/aaX-testcases/10-RotorStator/test/data/targ_1_chkpts.h5:/metric_diag/field" -ts_trajectory_dirname ./test_directory_1 -ts_trajectory_type memory
-ts_trajectory_max_cps_ram 4 -ts_trajectory_max_cps_disk 5000 -ts_trajectory_monitor > log.txt 2> error.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I have attached the log.txt and error.txt to this email so that you can have a look at these. It seems to look ok until the OOM killer kills the job.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Best wishes,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Anton <o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Matthew Knepley <knepley@gmail.com><br>
<b>Date: </b>Wednesday, 9 December 2020 at 01:38<br>
<b>To: </b>Zhang, Hong <hongzhang@anl.gov><br>
<b>Cc: </b>Anton Glazkov <anton.glazkov@chch.ox.ac.uk>, petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-users] TSAdjoint multilevel checkpointing running out of memory<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal">On Tue, Dec 8, 2020 at 6:47 PM Zhang, Hong via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Anton, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">TSAdjoint should manage checkpointing automatically, and the number of checkpoints in RAM and disk should not exceed the user-specified values. Can you send us the output for -ts_trajectory_monitor in your case?<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One other thing. It is always possible to miscalculate RAM a little. If you set it to 4 checkpoints, does it complete?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Matt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal">Hong (Mr.)<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Dec 8, 2020, at 3:37 PM, Anton Glazkov <<a href="mailto:anton.glazkov@chch.ox.ac.uk" target="_blank">anton.glazkov@chch.ox.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Good evening, </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Im attempting to run a multi-level checkpointing code on a cluster (ie RAM+disk storage with download-revolve as a configure option) with the options -ts_trajectory_type memory -ts_trajectory_max_cps_ram 5 -ts_trajectory_max_cps_disk
5000, for example. My question is, if I have 100,000 time points, for example, that need to be evaluated during the forward and adjoint run, does TSAdjoint automatically optimize the checkpointing so that the number of checkpoints in RAM and disk do not
exceed these values, or is one of the options ignored. I ask because I have a case that runs correctly with -ts_trajectory_type basic, but runs out of memory when attempting to fill the checkpoints in RAM when running the adjoint (I have verified that 5 checkpoints
will actually fit into the available memory). This makes me think that maybe the -ts_trajectory_max_cps_ram 5 option is being ignored?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Best wishes,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Anton </span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>