<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Anton,
<div class=""><br class="">
</div>
<div class="">There is a memory leak bug in TSTrajectory for this particular case. It has already been fixed. Can you update your PETSc and retry? Thank you for reporting the issue to us.</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Hong (Mr.)<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 10, 2020, at 5:19 PM, Anton Glazkov <<a href="mailto:anton.glazkov@chch.ox.ac.uk" class="">anton.glazkov@chch.ox.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class="">Dear Matt and Hong,<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class="">Thank you for your quick replies!<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class="">In answer to your question Matt, the application fails in the same way as with 5 checkpoints. I don’t believe the RAM capacity to be a problem though because we are running this case on a cluster with 64GB RAM per node, and we anticipate 0.1GB
storage requirements for the 4 checkpoints.<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class="">The case is being run in MPMD mode with the following command:<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class="">aprun -n 72 /work/e01/e01/chri4903/bin/cascade-ng/checkpoints_gradients ../data/nl_adj_0-chkpts.ini -adjoint -vr "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/ic_0_chkpts.h5:/0000000000/field" -vTarg "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/targ_0_chkpts.h5:/targ/field"
-vMet "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/targ_0_chkpts.h5:/metric_diag/field" -ts_trajectory_dirname ./test_directory_0 -ts_trajectory_type memory -ts_trajectory_max_cps_ram 4 -ts_trajectory_max_cps_disk 5000 -ts_trajectory_monitor : -n
80 /…/…/…/…/bin/cascade-ng/checkpoints_gradients ../data/nl_adj_1-chkpts.ini -adjoint -vr "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/ic_1_chkpts.h5:/0000000000/field" -vTarg "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/targ_1_chkpts.h5:/targ/field"
-vMet "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/targ_1_chkpts.h5:/metric_diag/field" -ts_trajectory_dirname ./test_directory_1 -ts_trajectory_type memory -ts_trajectory_max_cps_ram 4 -ts_trajectory_max_cps_disk 5000 -ts_trajectory_monitor > log.txt
2> error.txt<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class="">I have attached the log.txt and error.txt to this email so that you can have a look at these. It seems to look ok until the OOM killer kills the job.<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class="">Best wishes,<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class="">Anton<span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;" class="">
<p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 11pt; font-family: Calibri, sans-serif;">
<b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Wednesday, 9 December 2020 at 01:38<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>Zhang, Hong <<a href="mailto:hongzhang@anl.gov" class="">hongzhang@anl.gov</a>><br class="">
<b class="">Cc:<span class="Apple-converted-space"> </span></b>Anton Glazkov <<a href="mailto:anton.glazkov@chch.ox.ac.uk" class="">anton.glazkov@chch.ox.ac.uk</a>>,
<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] TSAdjoint multilevel checkpointing running out of memory<o:p class=""></o:p></span></p>
</div>
<div class="">
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Tue, Dec 8, 2020 at 6:47 PM Zhang, Hong via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users@mcs.anl.gov</a>> wrote:<o:p class=""></o:p></div>
</div>
<div class="">
<blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm;" class="">
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Anton,<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
TSAdjoint should manage checkpointing automatically, and the number of checkpoints in RAM and disk should not exceed the user-specified values. Can you send us the output for -ts_trajectory_monitor in your case?<o:p class=""></o:p></div>
</div>
</div>
</blockquote>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
One other thing. It is always possible to miscalculate RAM a little. If you set it to 4 checkpoints, does it complete?<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Thanks,<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Matt<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm;" class="">
<div class="">
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hong (Mr.)<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br class="">
<br class="">
<o:p class=""></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Dec 8, 2020, at 3:37 PM, Anton Glazkov <<a href="mailto:anton.glazkov@chch.ox.ac.uk" target="_blank" style="color: blue; text-decoration: underline;" class="">anton.glazkov@chch.ox.ac.uk</a>> wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">Good evening, </span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">I’m attempting to run a multi-level checkpointing code on a cluster (ie RAM+disk storage with –download-revolve as a configure option) with the options “-ts_trajectory_type memory -ts_trajectory_max_cps_ram 5 -ts_trajectory_max_cps_disk
5000”, for example. My question is, if I have 100,000 time points, for example, that need to be evaluated during the forward and adjoint run, does TSAdjoint automatically optimize the checkpointing so that the number of checkpoints in RAM and disk do not
exceed these values, or is one of the options ignored. I ask because I have a case that runs correctly with -ts_trajectory_type basic, but runs out of memory when attempting to fill the checkpoints in RAM when running the adjoint (I have verified that 5 checkpoints
will actually fit into the available memory). This makes me think that maybe the -ts_trajectory_max_cps_ram 5 option is being ignored?</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class=""> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">Best wishes,</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<span lang="EN-US" class="">Anton </span><o:p class=""></o:p></div>
</div>
</div>
</div>
</blockquote>
</div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
</div>
</blockquote>
</div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br clear="all" class="">
<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
--<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" style="color: blue; text-decoration: underline;" class="">https://www.cse.buffalo.edu/~knepley/</a><o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<span id="cid:7A7A0A94-EAA0-4781-A0AC-50F0E7C8B5BD@hsd1.il.comcast.net"><error.txt></span><span id="cid:78A71B86-288D-4685-9052-410229AA0322@hsd1.il.comcast.net"><log.txt></span></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>