<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I test a small case with number of processors from 1 to 40
(scenario 17-22 below). The memory consumption by PETSc decreases
a little as number of processors increases, but not as significant
as the structured grid code. Is there any way to further reduce
the memory consumption by DMPlex?<br>
</p>
<p> </p>
<table cellspacing="0" border="0">
<colgroup width="85" span="5"></colgroup> <colgroup width="83"></colgroup>
<colgroup width="125" span="2"></colgroup> <colgroup width="175"></colgroup>
<colgroup width="119"></colgroup> <tbody>
<tr>
<td height="17" align="right">scenario</td>
<td align="right">no. points</td>
<td align="right">cell type</td>
<td align="right">DMPLex</td>
<td align="right">nprocs</td>
<td align="right">no. nodes</td>
<td align="right">mem per node GB</td>
<td align="right">Rank 0 memory MB</td>
<td align="right">Rank 0 petsc memory MB</td>
<td align="right">Runtime (sec)</td>
</tr>
<tr>
<td height="17" align="right">1</td>
<td align="right">2121</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">0.21</td>
<td align="right">41.6</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">2</td>
<td align="right">8241</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">0.59</td>
<td align="right">51.84</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">3</td>
<td align="right">32481</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">1.95</td>
<td align="right">59.1</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">4</td>
<td align="right">128961</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">7.05</td>
<td align="right">89.71</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">5</td>
<td align="right">513921</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">26.76</td>
<td align="right">110.58</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">6</td>
<td align="right">2051841</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">104.21</td>
<td align="right">232.05</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>7</b></td>
<td align="right"><b>8199681</b></td>
<td align="right"><b>rectangle</b></td>
<td align="right"><b>no</b></td>
<td align="right"><b>40</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>411.26</b></td>
<td align="right"><b>703.27</b></td>
<td align="right"><b>140.29</b></td>
</tr>
<tr>
<td height="17" align="right"><b>8</b></td>
<td align="right"><b>8199681</b></td>
<td align="right"><b>rectangle</b></td>
<td align="right"><b>no</b></td>
<td align="right"><b>80</b></td>
<td align="right"><b>2</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>206.6</b></td>
<td align="right"><b>387.25</b></td>
<td align="right"><b>62.04</b></td>
</tr>
<tr>
<td height="17" align="right"><b>9</b></td>
<td align="right"><b>8199681</b></td>
<td align="right"><b>rectangle</b></td>
<td align="right"><b>no</b></td>
<td align="right"><b>160</b></td>
<td align="right"><b>4</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>104.28</b></td>
<td align="right"><b>245.3</b></td>
<td align="right"><b>32.76</b></td>
</tr>
<tr>
<td height="17" align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">10</td>
<td align="right">2121</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">0.49</td>
<td align="right">61.78</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">11</td>
<td align="right">15090</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">2.32</td>
<td align="right">96.61</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">12</td>
<td align="right">59847</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">8.28</td>
<td align="right">176.14</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">13</td>
<td align="right">238568</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">31.89</td>
<td align="right">573.73</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>14</b></td>
<td align="right"><b>953433</b></td>
<td align="right"><b>triangle</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>40</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>119.23</b></td>
<td align="right"><b>2102.54</b></td>
<td align="right"><b>44.11</b></td>
</tr>
<tr>
<td height="17" align="right"><b>15</b></td>
<td align="right"><b>953433</b></td>
<td align="right"><b>triangle</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>80</b></td>
<td align="right"><b>2</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>72.99</b></td>
<td align="right"><b>2123.8</b></td>
<td align="right"><b>24.36</b></td>
</tr>
<tr>
<td height="17" align="right"><b>16</b></td>
<td align="right"><b>953433</b></td>
<td align="right"><b>triangle</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>160</b></td>
<td align="right"><b>4</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>48.65</b></td>
<td align="right"><b>2076.25</b></td>
<td align="right"><b>14.87</b></td>
</tr>
<tr>
<td height="17" align="right"><b><br>
</b></td>
<td align="right"><b><br>
</b></td>
<td align="right"><b><br>
</b></td>
<td align="right"><b><br>
</b></td>
<td align="right"><b><br>
</b></td>
<td align="right"><b><br>
</b></td>
<td align="right"><b><br>
</b></td>
<td align="right"><b><br>
</b></td>
<td align="right"><b><br>
</b></td>
<td align="left"><b><br>
</b></td>
</tr>
<tr>
<td height="17" align="right"><b>17</b></td>
<td align="right"><b>55770</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>340.98</b></td>
<td align="right"><b>545.33</b></td>
<td align="left"><b><br>
</b></td>
</tr>
<tr>
<td height="17" align="right"><b>18</b></td>
<td align="right"><b>55770</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>4</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>107.72</b></td>
<td align="right"><b>339.78</b></td>
<td align="left"><b><br>
</b></td>
</tr>
<tr>
<td height="17" align="right"><b>19</b></td>
<td align="right"><b>55770</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>8</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>61.39</b></td>
<td align="right"><b>272.03</b></td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>20</b></td>
<td align="right"><b>55770</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>16</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>34.64</b></td>
<td align="right"><b>236.58</b></td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>21</b></td>
<td align="right"><b>55770</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>32</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>23.41</b></td>
<td align="right"><b>225.66</b></td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>22</b></td>
<td align="right"><b>55770</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>40</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>18.46</b></td>
<td align="right"><b>219.39</b></td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">23</td>
<td align="right">749814</td>
<td align="right">prism</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">149.86</td>
<td align="right">2412.39</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">24</td>
<td align="right">7000050</td>
<td align="right">prism</td>
<td align="right">yes</td>
<td align="right">40 to 640</td>
<td align="right">1 to 16</td>
<td align="right">200</td>
<td align="right"><br>
</td>
<td align="right">out_of_memory</td>
<td align="left"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>25</b></td>
<td align="right"><b>7000050</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>32</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>480</b></td>
<td align="right"><b>1655.1</b></td>
<td align="right"><b>17380.12</b></td>
<td align="right"><b>2712.11</b></td>
</tr>
<tr>
<td height="17" align="right"><b>26</b></td>
<td align="right"><b>7000050</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>64</b></td>
<td align="right"><b>2</b></td>
<td align="right"><b>480</b></td>
<td align="right"><b>890.92</b></td>
<td align="right"><b>17214.41</b></td>
<td align="right"><b>1497.55</b></td>
</tr>
<tr>
<td height="17" align="right"><b>27</b></td>
<td align="right"><b>7000050</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>128</b></td>
<td align="right"><b>4</b></td>
<td align="right"><b>480</b></td>
<td align="right"><b>451.23</b></td>
<td align="right"><b>17159.34</b></td>
<td align="right"><b>941.52</b></td>
</tr>
</tbody>
</table>
<p>
<style type="text/css">
body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }
a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em; }
a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em; }
comment { display:none; }
</style></p>
<p>Thanks,</p>
<p>Danyang<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 2019-10-09 2:08 p.m., Danyang Su
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:70e93d54-6712-7fa3-cf8c-aa616425c359@gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<p>Dear All,</p>
<p>I have a question regarding the maximum memory usage for the
scaling test. My code is written in Fortran with support for
both structured grid (DM) and unstructured grid (DMPlex). It
looks like memory consumption is much larger when DMPlex is used
and finally causew out_of_memory problem. <br>
</p>
<p>Below are some test using both structured grid and unstructured
grid. The memory consumption by the code is estimated based on
all allocated arrays and PETSc memory consumption is estimated
based on PetscMemoryGetMaximumUsage. <br>
</p>
<p>I just wonder why the PETSc memory consumption does not
decrease when number of processors increases. For structured
grid (scenario 7-9), the memory consumption decreases as number
of processors increases. However, for unstructured grid case
(scenario 14-16), the memory for PETSc part remains unchanged.
When I run a larger case, the code crashes because memory is ran
out. The same case works on another cluster with 480GB memory
per node. Does this make sense?<br>
</p>
<p> </p>
<table cellspacing="0" border="0">
<colgroup width="85" span="5"></colgroup> <colgroup width="83"></colgroup>
<colgroup width="125"></colgroup> <colgroup width="211"></colgroup>
<colgroup width="125"></colgroup> <colgroup width="175"></colgroup>
<colgroup width="119"></colgroup> <tbody>
<tr>
<td height="17" align="right">scenario</td>
<td align="right">no. points</td>
<td align="right">cell type</td>
<td align="right">DMPLex</td>
<td align="right">nprocs</td>
<td align="right">no. nodes</td>
<td align="right">mem per node GB</td>
<td align="right">solver</td>
<td align="right">Rank 0 memory MB</td>
<td align="right">Rank 0 petsc memory MB</td>
<td align="right">Runtime (sec)</td>
</tr>
<tr>
<td height="17" align="right">1</td>
<td align="right">2121</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">0.21</td>
<td align="right">41.6</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">2</td>
<td align="right">8241</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">0.59</td>
<td align="right">51.84</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">3</td>
<td align="right">32481</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">1.95</td>
<td align="right">59.1</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">4</td>
<td align="right">128961</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">7.05</td>
<td align="right">89.71</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">5</td>
<td align="right">513921</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">26.76</td>
<td align="right">110.58</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">6</td>
<td align="right">2051841</td>
<td align="right">rectangle</td>
<td align="right">no</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">104.21</td>
<td align="right">232.05</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>7</b></td>
<td align="right"><b>8199681</b></td>
<td align="right"><b>rectangle</b></td>
<td align="right"><b>no</b></td>
<td align="right"><b>40</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>411.26</b></td>
<td align="right"><b>703.27</b></td>
<td align="right"><b>140.29</b></td>
</tr>
<tr>
<td height="17" align="right"><b>8</b></td>
<td align="right"><b>8199681</b></td>
<td align="right"><b>rectangle</b></td>
<td align="right"><b>no</b></td>
<td align="right"><b>80</b></td>
<td align="right"><b>2</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>206.6</b></td>
<td align="right"><b>387.25</b></td>
<td align="right"><b>62.04</b></td>
</tr>
<tr>
<td height="17" align="right"><b>9</b></td>
<td align="right"><b>8199681</b></td>
<td align="right"><b>rectangle</b></td>
<td align="right"><b>no</b></td>
<td align="right"><b>160</b></td>
<td align="right"><b>4</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>104.28</b></td>
<td align="right"><b>245.3</b></td>
<td align="right"><b>32.76</b></td>
</tr>
<tr>
<td height="17" align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">10</td>
<td align="right">2121</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">0.49</td>
<td align="right">61.78</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">11</td>
<td align="right">15090</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">2.32</td>
<td align="right">96.61</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">12</td>
<td align="right">59847</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">8.28</td>
<td align="right">176.14</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">13</td>
<td align="right">238568</td>
<td align="right">triangle</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">31.89</td>
<td align="right">573.73</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>14</b></td>
<td align="right"><b>953433</b></td>
<td align="right"><b>triangle</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>40</b></td>
<td align="right"><b>1</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>119.23</b></td>
<td align="right"><b>2102.54</b></td>
<td align="right"><b>44.11</b></td>
</tr>
<tr>
<td height="17" align="right"><b>15</b></td>
<td align="right"><b>953433</b></td>
<td align="right"><b>triangle</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>80</b></td>
<td align="right"><b>2</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>72.99</b></td>
<td align="right"><b>2123.8</b></td>
<td align="right"><b>24.36</b></td>
</tr>
<tr>
<td height="17" align="right"><b>16</b></td>
<td align="right"><b>953433</b></td>
<td align="right"><b>triangle</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>160</b></td>
<td align="right"><b>4</b></td>
<td align="right"><b>200</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>48.65</b></td>
<td align="right"><b>2076.25</b></td>
<td align="right"><b>14.87</b></td>
</tr>
<tr>
<td height="17" align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">17</td>
<td align="right">55770</td>
<td align="right">prism</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">18.46</td>
<td align="right">219.39</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">18</td>
<td align="right">749814</td>
<td align="right">prism</td>
<td align="right">yes</td>
<td align="right">40</td>
<td align="right">1</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right">149.86</td>
<td align="right">2412.39</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right">19</td>
<td align="right">7000050</td>
<td align="right">prism</td>
<td align="right">yes</td>
<td align="right">40 to 640</td>
<td align="right">1 to 16</td>
<td align="right">200</td>
<td align="right">GMRES,Hypre preconditioner</td>
<td align="right"><br>
</td>
<td align="right">out_of_memory</td>
<td align="left"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
<td align="right"><br>
</td>
</tr>
<tr>
<td height="17" align="right"><b>20</b></td>
<td align="right"><b>7000050</b></td>
<td align="right"><b>prism</b></td>
<td align="right"><b>yes</b></td>
<td align="right"><b>64</b></td>
<td align="right"><b>2</b></td>
<td align="right"><b>480</b></td>
<td align="right"><b>GMRES,Hypre preconditioner</b></td>
<td align="right"><b>890.92</b></td>
<td align="right"><b>17214.41</b></td>
<td align="right"><br>
</td>
</tr>
</tbody>
</table>
<p>
<style type="text/css">
body,div,table,thead,tbody,tfoot,tr,th,td,p { font-family:"Liberation Sans"; font-size:x-small }
a.comment-indicator:hover + comment { background:#ffd; position:absolute; display:block; border:1px solid black; padding:0.5em; }
a.comment-indicator { background:red; display:inline-block; border:1px solid black; width:0.5em; height:0.5em; }
comment { display:none; }
</style></p>
<p>The error information of scenario 19 is shown below:<br>
</p>
<p>kernel messages produced during job executions:<br>
[Oct 9 10:41] mpiexec.hydra invoked oom-killer:
gfp_mask=0x200da, order=0, oom_score_adj=0<br>
[ +0.010274] mpiexec.hydra cpuset=/ mems_allowed=0-1<br>
[ +0.006680] CPU: 2 PID: 144904 Comm: mpiexec.hydra Tainted:
G OE ------------ 3.10.0-862.14.4.el7.x86_64 #1<br>
[ +0.013365] Hardware name: Lenovo ThinkSystem SD530
-[7X21CTO1WW]-/-[7X21CTO1WW]-, BIOS -[TEE124N-1.40]- 06/12/2018<br>
[ +0.012866] Call Trace:<br>
[ +0.003945] [<ffffffffb3313754>] dump_stack+0x19/0x1b<br>
[ +0.006995] [<ffffffffb330e91f>] dump_header+0x90/0x229<br>
[ +0.007121] [<ffffffffb2cfa982>] ?
ktime_get_ts64+0x52/0xf0<br>
[ +0.007451] [<ffffffffb2d5141f>] ?
delayacct_end+0x8f/0xb0<br>
[ +0.007393] [<ffffffffb2d9ac94>]
oom_kill_process+0x254/0x3d0<br>
[ +0.007592] [<ffffffffb2d9a73d>] ?
oom_unkillable_task+0xcd/0x120<br>
[ +0.007978] [<ffffffffb2d9a7e6>] ?
find_lock_task_mm+0x56/0xc0<br>
[ +0.007729] [<ffffffffb2d9b4d6>] <b>out_of_memory+0x4b6/0x4f0</b><br>
[ +0.007358] [<ffffffffb330f423>]
__alloc_pages_slowpath+0x5d6/0x724<br>
[ +0.008190] [<ffffffffb2da18b5>]
__alloc_pages_nodemask+0x405/0x420</p>
<p>Thanks,</p>
<p>Danyang<br>
</p>
</blockquote>
</body>
</html>