<div dir="ltr"><div dir="ltr"><div>Thanks for your reply, Marks,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 11, 2019 at 8:06 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Interesting, nice work.<div><br></div><div>It would be interesting to get the flop counters working.</div><div><br></div><div>This looks like GMG, I assume 3D.</div><div><br></div><div>The degree of parallelism is not very realistic. </div></div></blockquote><div><br></div><div>We are actually working on real physics simulations. Not a benchmark at all.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>You should probably run a 10x smaller problem, at least,</div></div></blockquote><div><br></div><div>This is the coarsest mesh that can resolve the geometry completely, and so I can not make the problem smaller.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div> or use 10x more processes. </div></div></blockquote><div><br></div><div>I do not have such a machine :(.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I guess it does not matter. This basically like a one node run because the subdomains are so large.</div></div></blockquote><div><br></div><div>I guess you are interested in the performance of the new algorithms on small problems. I will try to test a petsc example such as mat/examples/tests/ex96.c. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>And are you sure the numerics are the same with and without hypre? Hypre is 15x slower. Any ideas what is going on?</div></div></blockquote><div><br></div><div>Hypre performs pretty good when the number of processor core is small ( a couple of hundreds). I guess the issue is related to how they handle the communications. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>It might be interesting to scale this test down to a node to see if this is from communication.</div></div></blockquote><div><br></div><div>Hypre preforms similarly as petsc on a single compute node.</div><div><br></div><div><br></div><div>Fande,</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Again, nice work,</div><div>Mark</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 11, 2019 at 7:08 PM Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Developers,<br><div><br></div><div>I just want to share a good news. It is known PETSc-ptap-scalable is taking too much memory for some applications because it needs to build intermediate data structures. According to Mark's suggestions, I implemented the all-at-once algorithm that does not cache any intermediate data. </div><div><br></div><div>I did some comparison, the new implementation is actually scalable in terms of the memory usage and the compute time even though it is still slower than "ptap-scalable". There are some memory profiling results (see the attachments). The new all-at-once implementation use the similar amount of memory as hypre, but it way faster than hypre.</div><div><br></div><div>For example, for a problem with 14,893,346,880 unknowns using 10,000 processor cores, There are timing results:</div><div><br></div><div>Hypre algorithm:</div><div><br></div><div><div>MatPtAP 50 1.0 3.5353e+03 1.0 0.00e+00 0.0 1.9e+07 3.3e+04 6.0e+02 33 0 1 0 17 33 0 1 0 17 0</div><div>MatPtAPSymbolic 50 1.0 2.3969e-0213.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</div><div>MatPtAPNumeric 50 1.0 3.5353e+03 1.0 0.00e+00 0.0 1.9e+07 3.3e+04 6.0e+02 33 0 1 0 17 33 0 1 0 17 0</div></div><div><br></div><div>PETSc scalable PtAP:</div><div><br></div><div>MatPtAP 50 1.0 1.1453e+02 1.0 2.07e+09 3.8 6.6e+07 2.0e+05 7.5e+02 2 1 4 6 20 2 1 4 6 20 129418</div><div>MatPtAPSymbolic 50 1.0 5.1562e+01 1.0 0.00e+00 0.0 4.1e+07 1.4e+05 3.5e+02 1 0 3 3 9 1 0 3 3 9 0</div><div>MatPtAPNumeric 50 1.0 6.3072e+01 1.0 2.07e+09 3.8 2.4e+07 3.1e+05 4.0e+02 1 1 2 4 11 1 1 2 4 11 235011</div><div><br></div><div>New implementation of the all-at-once algorithm:</div><div><br></div><div><div>MatPtAP 50 1.0 2.2153e+02 1.0 0.00e+00 0.0 1.0e+08 1.4e+05 6.0e+02 4 0 7 7 17 4 0 7 7 17 0</div><div>MatPtAPSymbolic 50 1.0 1.1055e+02 1.0 0.00e+00 0.0 7.9e+07 1.2e+05 2.0e+02 2 0 5 4 6 2 0 5 4 6 0</div><div>MatPtAPNumeric 50 1.0 1.1102e+02 1.0 0.00e+00 0.0 2.6e+07 2.0e+05 4.0e+02 2 0 2 3 11 2 0 2 3 11 0</div></div><div><br></div><div><br></div><div>You can see here the all-at-once is a bit slower than ptap-scalable, but it uses only much less memory. </div><div><br></div><div><br></div><div>Fande</div><div> </div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div></div></div>