<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 7, 2020, at 1:25 AM, Nidish <<a href="mailto:nb25@rice.edu" class="">nb25@rice.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div class=""><p class="">Indeed - I was just using the default solver (GMRES with ILU).</p><p class="">Using just standard LU (direct solve with "-pc_type lu -ksp_type
preonly"), I find elemental to be extremely slow even for a
1000x1000 matrix. </p></div></div></blockquote><div><br class=""></div>What about on one process? </div><div><br class=""></div><div>Elemental generally won't be competitive for such tiny matrices. <br class=""><blockquote type="cite" class=""><div class=""><div class=""><p class="">For MPIaij it's throwing me an error if I tried
"-pc_type lu".</p></div></div></blockquote><div><br class=""></div> Yes, there is no PETSc code for sparse parallel direct solver, this is expected.</div><div><br class=""></div><div> What about ?</div><div><br class=""></div><div><blockquote type="cite" class=""><div class=""><blockquote class=""><p class="">mpirun -n 1 ./ksps -N 1000 -mat_type mpidense -pc_type jacobi</p><div class="">mpirun -n 4 ./ksps -N 1000 -mat_type mpidense -pc_type jacobi</div></blockquote></div></blockquote></div><div>Where will your dense matrices be coming from and how big will they be in practice? This will help determine if an iterative solver is appropriate. If they will be 100,000 for example then testing with 1000 will tell you nothing useful, you need to test with the problem size you care about.</div><div><br class=""></div><div>Barry</div><div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><p class=""> I'm attaching the code here, in case you'd like to
have a look at what I've been trying to do. <br class="">
</p><p class="">The two configurations of interest are,</p>
<blockquote class=""><p class="">$> mpirun -n 4 ./ksps -N 1000 -mat_type mpiaij<br class="">
$> mpirun -n 4 ./ksps -N 1000 -mat_type elemental</p>
</blockquote><p class="">(for the GMRES with ILU) and,</p>
<blockquote class=""><p class="">$> mpirun -n 4 ./ksps -N 1000 -mat_type mpiaij -pc_type lu
-ksp_type preonly<br class="">
$> mpirun -n 4 ./ksps -N 1000 -mat_type elemental -pc_type lu
-ksp_type preonly</p>
</blockquote><p class="">elemental seems to perform poorly in both cases.</p><p class="">Nidish<br class="">
</p>
<div class="moz-cite-prefix">On 8/7/20 12:50 AM, Barry Smith wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:85F9F817-2754-4F55-9222-3E23003E79FD@petsc.dev" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div class=""><br class="">
</div>
<div class=""> What is the output of -ksp_view for the two case?</div>
<div class=""><br class="">
</div>
<div class=""> It is not only the matrix format but also the
matrix solver that matters. For example if you are using an
iterative solver the elemental format won't be faster, you
should use the PETSc MPIDENSE format. The elemental format is
really intended when you use a direct LU solver for the matrix.
For tiny matrices like this an iterative solver could easily be
faster than the direct solver, it depends on the conditioning
(eigenstructure) of the dense matrix. Also the default PETSc
solver uses block Jacobi with ILU on each process if using a
sparse format, ILU applied to a dense matrix is actually LU so
your solver is probably different also between the MPIAIJ and
the elemental. </div>
<div class=""><br class="">
</div>
<div class=""> Barry</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""> <br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 7, 2020, at 12:30 AM, Nidish <<a href="mailto:nb25@rice.edu" class="" moz-do-not-send="true">nb25@rice.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="zoom: 0%;" class="">
<div dir="auto" class="">Thank you for the response.<br class="">
<br class="">
</div>
<div dir="auto" class="">I've just been running some
tests with matrices up to 2e4 dimensions (dense). When
I compared the solution times for "-mat_type
elemental" and "-mat_type mpiaij" running with 4
cores, I found the mpidense versions running way
faster than elemental. I have not been able to make
the elemental version finish up for 2e4 so far (my
patience runs out faster). <br class="">
<br class="">
</div>
<div dir="auto" class="">What's going on here? I thought
elemental was supposed to be superior for dense
matrices.<br class="">
<br class="">
</div>
<div dir="auto" class="">I can share the code if that's
appropriate for this forum (sorry, I'm new here). <br class="">
<br class="">
</div>
<div dir="auto" class="">Nidish</div>
<div class="gmail_quote">On Aug 6, 2020, at 23:01, Barry
Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="" moz-do-not-send="true">bsmith@petsc.dev</a>>
wrote:
<blockquote class="gmail_quote" style="margin: 0pt 0pt
0pt 0.8ex; border-left: 1px solid rgb(204, 204,
204); padding-left: 1ex;">
<pre class="blue">
<blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> On Aug 6, 2020, at 7:32 PM, Nidish <<a href="mailto:nb25@rice.edu" class="" moz-do-not-send="true">nb25@rice.edu</a>> wrote:
I'm relatively new to PETSc, and my applications involve (for the most part) dense matrix solves.
I read in the documentation that this is an area PETSc does not specialize in but instead recommends external libraries such as Elemental. I'm wondering if there are any "best" practices in this regard. Some questions I'd like answered are:
1. Can I just declare my dense matrix as a sparse one and fill the whole matrix up? Do any of the others go this route? What're possible pitfalls/unfavorable outcomes for this? I understand the memory overhead probably shoots up.
</blockquote>
No, this isn't practical, the performance will be terrible.
<blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> 2. Are there any specific guidelines on when I can expect elemental to perform better in parallel than in serial?
</blockquote>
Because the computation to communication ratio for dense matrices is higher than for sparse you will see better parallel performance for dense problems of a given size than sparse problems of a similar size. In other words parallelism can help for dense matrices for relatively small problems, of course the specifics of your machine hardware and software also play a role.
Barry
<blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;">
Of course, I'm interesting in any other details that may be important in this regard.
Thank you,
Nidish
</blockquote>
</pre>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<div class="moz-signature">-- <br class="">
Nidish</div>
</div>
<span id="cid:FF8DD1F1-CA48-405B-8E23-364936BB6B64"><ksps.cpp></span></div></blockquote></div><br class=""></body></html>