<div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div>Hello,<br></div>I apologize for asking the very similar question again. I have a problem in which the greatest bottleneck is the Trace(A\B) function. Thus I am trying to run this function in parallel. If I understand correctly, MatMatSolve is a sequential operation, am I right? <br>
</div><div>I have tried the following approach - I divide the matrix B by columns, so basically I solve a bunch of linear systems of type A\b where A is a matrix and b is a vector (column of matrix B). Since these A\b operations are independent, I would like to run them all in parallel. <br>
In my experiment I have:<br><br></div>1) loaded the whole matrix A on each process<br></div>2) loaded matrix B as shared on all processes<br></div>3) transposed matrix B<br></div>On each separate process in parallel:<br></div>
- run Cholesky factorization (only for the first A\b solve)<br></div> - Solve linear systems A\b_i for each row i of matrix B that is stored on that process<br></div> - Send the partial trace result to the main process<br>
</div>Please find attached the complete code.<br><br></div>My question is about scalability of this solution. I see the improvement when running the code on 20 nodes as opposed to 10 nodes, but if I run the code on 40 nodes, my calculation time is even worse then calculation time on 20 nodes. Please find attached my log summary for 10 nodes, 20 nodes and 40 nodes. I should maybe point out that I am only interested in speeding up the Trace(A\B) calculation time (noted with "Trace calculations took: " in my output).<br>
Could you please help me figure out what is causing my performance to worsen when using more nodes? I am sorry if this is a very basic question, I am fairly new to PETSc.<br></div>Grateful in advance,<br></div>Jelena<br><div>
<div><div><br> <br><div><div><div><div><div><br><div><br><br><br></div></div></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Dec 3, 2012 at 2:21 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#computers" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#computers</a><br>
<div class="HOEnZb"><div class="h5"><br>
On Dec 3, 2012, at 1:08 PM, Jelena Slivka <<a href="mailto:slivkaje@gmail.com">slivkaje@gmail.com</a>> wrote:<br>
<br>
> Thank you very much!<br>
> However, I have another question. I have a cluster of 4 nodes and each node has 6 cores. If I run my code using 6 cores on one node (using the command "mpiexec -n 6") it is much faster than running it on just one process (which is expected). However, if I try running the code on multiple nodes (using "mpiexec -f machinefile -ppn 4", where machinefile is the file which contains the node names), it runs much slower than on just one process. This also happens with tutorial examples. I have checked the number of iteration for KSP solver when spread on multiple processors and it doesn't seem to be the problem. Do you have any suggestions on what am I doing wrong? Are the commands I am using wrong?<br>
><br>
><br>
> On Sat, Dec 1, 2012 at 6:03 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> We recommend following the directions <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#schurcomplement" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#schurcomplement</a> for computing a Schur complement; just skip the unneeded step. MUMPS supports a parallel Cholesky but you can also use a parallel LU with MUMPS, PaSTIX or SuperLU_Dist and those will work fine also. With current software Cholesky in parallel is not tons better than LU so generally not worth monkeying with.<br>
><br>
> Barry<br>
><br>
><br>
> On Dec 1, 2012, at 12:05 PM, Jelena Slivka <<a href="mailto:slivkaje@gmail.com">slivkaje@gmail.com</a>> wrote:<br>
><br>
> > Hello!<br>
> > I am trying to solve A*X = B where A and B are matrices, and then find trace of the resulting matrix X. My approach has been to partition matrix B in column vectors bi and then solve each system A*xi = bi. Then, for all vectors xi I would extract i-th element xi(i) and sum those elements in order to get Trace(X).<br>
> > Pseudo-code:<br>
> > 1) load matrices A and B<br>
> > 2) transpose matrix B (so that each right-hand side bi is in the row, as operation MatGetColumnVector is slow)<br>
> > 3) set up KSPSolve<br>
> > 4) create vector diagonal (in which xi(i) elements will be stored)<br>
> > 5) for each row i of matrix B owned by current process:<br>
> > - create vector bi by extracting row i from matrix B<br>
> > - apply KSPsolve to get xi<br>
> > - insert value xi(i) in diagonal vector (only the process which<br>
> > holds the ith value of vector x(i) should do so)<br>
> > 6) sum vector diagonal to get the trace.<br>
> > However, my code (attached, along with the test case) runs fine on one process, but hangs if started on multiple processes. Could you please help me figure out what am I doing wrong?<br>
> > Also, could you please tell me is it possible to use Cholesky factorization when running on multiple processes (I see that I cannot use it when I set the format of matrix A to MPIAIJ)?<br>
> ><br>
> > <Experiment.c><Abin><Bbin><br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div>