<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Dear All,</p>
<p>I have a question regarding the maximum memory usage for the
scaling test. My code is written in Fortran with support for both
structured grid (DM) and unstructured grid (DMPlex). It looks like
memory consumption is much larger when DMPlex is used and finally
causew out_of_memory problem. <br>
</p>
<p>Below are some test using both structured grid and unstructured
grid. The memory consumption by the code is estimated based on all
allocated arrays and PETSc memory consumption is estimated based
on PetscMemoryGetMaximumUsage. <br>
</p>
<p>I just wonder why the PETSc memory consumption does not decrease
when number of processors increases. For structured grid (scenario
7-9), the memory consumption decreases as number of processors
increases. However, for unstructured grid case (scenario 14-16),
the memory for PETSc part
remains unchanged. When I run a larger case, the code crashes
because memory is ran out. The same case works on another cluster
with 480GB memory per node. Does this make sense?<br>
</p>
<p> </p>
<table cellspacing="0" border="0">
<colgroup width="85" span="5"></colgroup> <colgroup width="83"></colgroup>
<colgroup width="125"></colgroup> <colgroup width="211"></colgroup>
<colgroup width="125"></colgroup> <colgroup width="175"></colgroup>
<colgroup width="119"></colgroup> <tbody>
<tr>
<td height="17" align="right">scenario</td>
<td align="right">no. points</td>
<td align="right">cell type</td>
<td align="right">DMPLex</td>
<td align="right">nprocs</td>
<td align="right">no. nodes</td>
<td align="right">mem per node GB</td>
<td align="right">solver</td>
<td align="right">Rank 0 memory MB</td>
<td align="right">Rank 0 petsc memory MB</td>
<td align="right">Runtime (sec)</td>
</tr>
<tr>
<td height="17" align="right">1</td>
<td align="right">2121</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">0.21</td>
<td align="right">41.6</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">2</td>
<td align="right">8241</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">0.59</td>
<td align="right">51.84</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">3</td>
<td align="right">32481</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">1.95</td>
<td align="right">59.1</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">4</td>
<td align="right">128961</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">7.05</td>
<td align="right">89.71</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">5</td>
<td align="right">513921</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">26.76</td>
<td align="right">110.58</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">6</td>
<td align="right">2051841</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">104.21</td>
<td align="right">232.05</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>7</b></td>
<td align="right"><b>8199681</b></td>
<td align="right"><b>rectangle</b></td>
<td align="right"><b>no</b></td>
<td align="right"><b>40</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>411.26</b></td>
<td align="right"><b>703.27</b></td>
<td align="right"><b>140.29</b></td>
</tr>
<tr>
<td height="17" align="right"><b>8</b></td>
<td align="right"><b>8199681</b></td>
<td align="right"><b>rectangle</b></td>
<td align="right"><b>no</b></td>
<td align="right"><b>80</b></td>
<td align="right"><b>2</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>206.6</b></td>
<td align="right"><b>387.25</b></td>
<td align="right"><b>62.04</b></td>
</tr>
<tr>
<td height="17" align="right"><b>9</b></td>
<td align="right"><b>8199681</b></td>
<td align="right"><b>rectangle</b></td>
<td align="right"><b>no</b></td>
<td align="right"><b>160</b></td>
<td align="right"><b>4</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>104.28</b></td>
<td align="right"><b>245.3</b></td>
<td align="right"><b>32.76</b></td>
</tr>
<tr>
<td height="17" align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">10</td>
<td align="right">2121</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">0.49</td>
<td align="right">61.78</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">11</td>
<td align="right">15090</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">2.32</td>
<td align="right">96.61</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">12</td>
<td align="right">59847</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">8.28</td>
<td align="right">176.14</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">13</td>
<td align="right">238568</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">31.89</td>
<td align="right">573.73</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>14</b></td>
<td align="right"><b>953433</b></td>
<td align="right"><b>triangle</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>40</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>119.23</b></td>
<td align="right"><b>2102.54</b></td>
<td align="right"><b>44.11</b></td>
</tr>
<tr>
<td height="17" align="right"><b>15</b></td>
<td align="right"><b>953433</b></td>
<td align="right"><b>triangle</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>80</b></td>
<td align="right"><b>2</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>72.99</b></td>
<td align="right"><b>2123.8</b></td>
<td align="right"><b>24.36</b></td>
</tr>
<tr>
<td height="17" align="right"><b>16</b></td>
<td align="right"><b>953433</b></td>
<td align="right"><b>triangle</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>160</b></td>
<td align="right"><b>4</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>48.65</b></td>
<td align="right"><b>2076.25</b></td>
<td align="right"><b>14.87</b></td>
</tr>
<tr>
<td height="17" align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">17</td>
<td align="right">55770</td>
<td align="right">prism</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">18.46</td>
<td align="right">219.39</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">18</td>
<td align="right">749814</td>
<td align="right">prism</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">149.86</td>
<td align="right">2412.39</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">19</td>
<td align="right">7000050</td>
<td align="right">prism</td>
<td align="right">yes</td>
<td align="right">40 to 640</td>
<td align="right">1 to 16</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right"><br>
</td>
<td align="right">out_of_memory</td>
<td align="left"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>20</b></td>
<td align="right"><b>7000050</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>64</b></td>
<td align="right"><b>2</b></td>
<td align="right"><b>480</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>890.92</b></td>
<td align="right"><b>17214.41</b></td>
<td align="right"><br>
</td>
</tr>
</tbody>
</table>
<p>
<style type="text/css">
body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }
a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em; }
a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em; }
comment { display:none; }
</style></p>
<p>The error information of scenario 19 is shown below:<br>
</p>
<p>kernel messages produced during job executions:<br>
[Oct 9 10:41] mpiexec.hydra invoked oom-killer: gfp_mask=0x200da,
order=0, oom_score_adj=0<br>
[ +0.010274] mpiexec.hydra cpuset=/ mems_allowed=0-1<br>
[ +0.006680] CPU: 2 PID: 144904 Comm: mpiexec.hydra Tainted:
G OE ------------ 3.10.0-862.14.4.el7.x86_64 #1<br>
[ +0.013365] Hardware name: Lenovo ThinkSystem SD530
-[7X21CTO1WW]-/-[7X21CTO1WW]-, BIOS -[TEE124N-1.40]- 06/12/2018<br>
[ +0.012866] Call Trace:<br>
[ +0.003945] [<ffffffffb3313754>] dump_stack+0x19/0x1b<br>
[ +0.006995] [<ffffffffb330e91f>] dump_header+0x90/0x229<br>
[ +0.007121] [<ffffffffb2cfa982>] ?
ktime_get_ts64+0x52/0xf0<br>
[ +0.007451] [<ffffffffb2d5141f>] ?
delayacct_end+0x8f/0xb0<br>
[ +0.007393] [<ffffffffb2d9ac94>]
oom_kill_process+0x254/0x3d0<br>
[ +0.007592] [<ffffffffb2d9a73d>] ?
oom_unkillable_task+0xcd/0x120<br>
[ +0.007978] [<ffffffffb2d9a7e6>] ?
find_lock_task_mm+0x56/0xc0<br>
[ +0.007729] [<ffffffffb2d9b4d6>] <b>out_of_memory+0x4b6/0x4f0</b><br>
[ +0.007358] [<ffffffffb330f423>]
__alloc_pages_slowpath+0x5d6/0x724<br>
[ +0.008190] [<ffffffffb2da18b5>]
__alloc_pages_nodemask+0x405/0x420</p>
<p>Thanks,</p>
<p>Danyang<br>
</p>
</body>
</html>