[petsc-users] TSAdjoint multilevel checkpointing running out of memory

Thu Dec 10 21:20:15 CST 2020

  Anton,

   You can try running a smaller problem that does not run out of memory with -log_view -log_view_memory. This should show in the right most columns how much memory is being used and added for each event in the computation and might help you track down where the memory is being gobbled up.  Also running with -info may provide additional information during the run of the current status.  In addition, valgrind has options for tracking memory usage which can show where there are big jumps in the memory usage and could tell you when the memory usage jumps dramatically.

   We are eager to insure that the TSAdjoint does not use excess memory so are happy to work with you to figure out where the excessive memory is being used.

   As always using the latest version of PETSc is best to track down the problems that arise.

  Barry

[NID 01928] 2020-12-10 21:58:46 Apid 47584119: initiated application termination
[NID 01928] 2020-12-10 21:58:47

Application 47584119 exit signals: Killed
Application 47584119 resources: utime ~0s, stime ~25s, Rss ~11012, inblocks ~46654, outblocks ~154922

> On Dec 10, 2020, at 5:19 PM, Anton Glazkov <anton.glazkov at chch.ox.ac.uk> wrote:
> 
> Dear Matt and Hong,
>  
> Thank you for your quick replies!
> In answer to your question Matt, the application fails in the same way as with 5 checkpoints. I don’t believe the RAM capacity to be a problem though because we are running this case on a cluster with 64GB RAM per node, and we anticipate 0.1GB storage requirements for the 4 checkpoints.
> The case is being run in MPMD mode with the following command:
>  
> aprun -n 72 /work/e01/e01/chri4903/bin/cascade-ng/checkpoints_gradients ../data/nl_adj_0-chkpts.ini -adjoint -vr "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/ic_0_chkpts.h5:/0000000000/field" -vTarg "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/targ_0_chkpts.h5:/targ/field" -vMet "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/targ_0_chkpts.h5:/metric_diag/field" -ts_trajectory_dirname ./test_directory_0 -ts_trajectory_type memory -ts_trajectory_max_cps_ram 4 -ts_trajectory_max_cps_disk 5000 -ts_trajectory_monitor : -n 80 /…/…/…/…/bin/cascade-ng/checkpoints_gradients ../data/nl_adj_1-chkpts.ini -adjoint -vr "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/ic_1_chkpts.h5:/0000000000/field" -vTarg "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/targ_1_chkpts.h5:/targ/field" -vMet "/…/…/…/…/sims/aaX-testcases/10-RotorStator/test/data/targ_1_chkpts.h5:/metric_diag/field" -ts_trajectory_dirname ./test_directory_1 -ts_trajectory_type memory -ts_trajectory_max_cps_ram 4 -ts_trajectory_max_cps_disk 5000 -ts_trajectory_monitor > log.txt 2> error.txt
>  
> I have attached the log.txt and error.txt to this email so that you can have a look at these. It seems to look ok until the OOM killer kills the job.
>  
> Best wishes,
> Anton 
>  
> From: Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>>
> Date: Wednesday, 9 December 2020 at 01:38
> To: Zhang, Hong <hongzhang at anl.gov <mailto:hongzhang at anl.gov>>
> Cc: Anton Glazkov <anton.glazkov at chch.ox.ac.uk <mailto:anton.glazkov at chch.ox.ac.uk>>, petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>>
> Subject: Re: [petsc-users] TSAdjoint multilevel checkpointing running out of memory
> 
> On Tue, Dec 8, 2020 at 6:47 PM Zhang, Hong via petsc-users <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
> Anton, 
>  
> TSAdjoint should manage checkpointing automatically, and the number of checkpoints in RAM and disk should not exceed the user-specified values. Can you send us the output for -ts_trajectory_monitor in your case?
>  
> One other thing. It is always possible to miscalculate RAM a little. If you set it to 4 checkpoints, does it complete?
>  
>   Thanks,
>  
>       Matt
>  
> Hong (Mr.)
> 
> 
> On Dec 8, 2020, at 3:37 PM, Anton Glazkov <anton.glazkov at chch.ox.ac.uk <mailto:anton.glazkov at chch.ox.ac.uk>> wrote:
>  
> Good evening, 
>  
> I’m attempting to run a multi-level checkpointing code on a cluster (ie RAM+disk storage with –download-revolve as a configure option) with the options “-ts_trajectory_type memory -ts_trajectory_max_cps_ram 5 -ts_trajectory_max_cps_disk 5000”, for example. My question is, if I have 100,000 time points, for example, that need to be evaluated  during the forward and adjoint run, does TSAdjoint automatically optimize the checkpointing so that the number of checkpoints in RAM and disk do not exceed these values, or is one of the options ignored. I ask because I have a case that runs correctly with -ts_trajectory_type basic, but runs out of memory when attempting to fill the checkpoints in RAM when running the adjoint (I have verified that 5 checkpoints will actually fit into the available memory). This makes me think that maybe the -ts_trajectory_max_cps_ram 5 option is being ignored?
>  
> Best wishes,
> Anton 
>  
> 
>  
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>  
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/><error.txt><log.txt>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20201210/1713eaac/attachment-0001.html>