<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
<html><body>
<p>Hi.</p>
<p><br />I managed to finish the re-implementation. I ran the program with 1,2,3,4,5,6 machines and saved the summary. I send each of them in this email.<br />In these executions, the program performs Matrix-Vector (MatMult, MatMultAdd) products and Vector-Vector operations. From what I understand while reading the logs, the program takes most of the time in "VecScatterEnd". <br />In this example, the matrix taking part on the Matrix-Vector products is not "much diagonal heavy". <br />The following numbers are the percentages of nnz values on the matrix diagonal block for each machine, and each execution time.<br />NMachines %NNZ ExecTime <br />1 machine0 100%; 16min08sec<br /><br />2 machine0 91.1%; 24min58sec <br /> machine1 69.2%; <br /><br />3 machine0 90.9% 25min42sec<br /> machine1 82.8%<br /> machine2 51.6%<br /><br />4 machine0 91.9% 26min27sec <br /> machine1 82.4%<br /> machine2 73.1%<br /> machine3 39.9%<br /><br />5 machine0 93.2% 39min23sec<br /> machine1 82.8%<br /> machine2 74.4%<br /> machine3 64.6%<br /> machine4 31.6%<br /><br />6 machine0 94.2% 54min54sec<br /> machine1 82.6%<br /> machine2 73.1%<br /> machine3 65.2%<br /> machine4 55.9% <br /> machine5 25.4%</p>
<p>In this implementation I'm using MatCreate and VecCreate. I'm also leaving the partition sizes in PETSC_DECIDE.</p>
<p>Finally, to run the application, I'm using mpirun.hydra from mpich, downloaded by PETSc configure script.<br />I'm checking the process assignment as suggested on the last email.<br /><br />Am I missing anything?<br /><br />Regards,<br />Nelson</p>
<p>Em 2015-08-20 16:17, Matthew Knepley escreveu:</p>
<blockquote type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Aug 20, 2015 at 6:30 AM, Nelson Filipe Lopes da Silva <span><<a href="mailto:nelsonflsilva@ist.utl.pt">nelsonflsilva@ist.utl.pt</a>></span> wrote:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">Hello.<br /><br /> I am sorry for the long time without response. I decided to rewrite my application in a different way and will send the log_summary output when done reimplementing.<br /><br /> As for the machine, I am using mpirun to run jobs in a 8 node cluster. I modified the makefile on the steams folder so it would run using my hostfile.<br /> The output is attached to this email. It seems reasonable for a cluster with 8 machines. From "lscpu", each machine cpu has 4 cores and 1 socket.</blockquote>
<div>1) You launcher is placing processes haphazardly. I would figure out how to assign them to certain nodes</div>
<div>2) Each node has enough bandwidth for 1 core, so it does not make much sense to use more than 1.</div>
<div> Thanks,</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">Cheers,<br /> Nelson<br /><br /><br /> Em 2015-07-24 16:50, Barry Smith escreveu:<br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">It would be very helpful if you ran the code on say 1, 2, 4, 8, 16<br /> ... processes with the option -log_summary and send (as attachments)<br /> the log summary information.<br /><br /> Also on the same machine run the streams benchmark; with recent<br /> releases of PETSc you only need to do<br /><br /> cd $PETSC_DIR<br /> make streams NPMAX=16 (or whatever your largest process count is)<br /><br /> and send the output.<br /><br /> I suspect that you are doing everything fine and it is more an issue<br /> with the configuration of your machine. Also read the information at<br /><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#computers">http://www.mcs.anl.gov/petsc/documentation/faq.html#computers</a> on<br /> "binding"<br /><br /> Barry<br /><br />
<blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">On Jul 24, 2015, at 10:41 AM, Nelson Filipe Lopes da Silva <<a href="mailto:nelsonflsilva@ist.utl.pt">nelsonflsilva@ist.utl.pt</a>> wrote:<br /><br /> Hello,<br /><br /> I have been using PETSc for a few months now, and it truly is fantastic piece of software.<br /><br /> In my particular example I am working with a large, sparse distributed (MPI AIJ) matrix we can refer as 'G'.<br /> G is a horizontal - retangular matrix (for example, 1,1 Million rows per 2,1 Million columns). This matrix is commonly very sparse and not diagonal 'heavy' (for example 5,2 Million nnz in which ~50% are on the diagonal block of MPI AIJ representation).<br /> To work with this matrix, I also have a few parallel vectors (created using MatCreate Vec), we can refer as 'm' and 'k'.<br /> I am trying to parallelize an iterative algorithm in which the most computational heavy operations are:<br /><br /> ->Matrix-Vector Multiplication, more precisely G * m + k = b (MatMultAdd). From what I have been reading, to achive a good speedup in this operation, G should be as much diagonal as possible, due to overlapping communication and computation. But even when using a G matrix in which the diagonal block has ~95% of the nnz, I cannot get a decent speedup. Most of the times, the performance even gets worse.<br /><br /> ->Matrix-Matrix Multiplication, in this case I need to perform G * G' = A, where A is later used on the linear solver and G' is transpose of G. The speedup in this operation is not worse, although is not very good.<br /><br /> ->Linear problem solving. Lastly, In this operation I compute "Ax=b" from the last two operations. I tried to apply a RCM permutation to A to make it more diagonal, for better performance. However, the problem I faced was that, the permutation is performed locally in each processor and thus, the final result is different with different number of processors. I assume this was intended to reduce communication. The solution I found was<br /> 1-calculate A<br /> 2-calculate, localy to 1 machine, the RCM permutation IS using A<br /> 3-apply this permutation to the lines of G.<br /> This works well, and A is generated as if RCM permuted. It is fine to do this operation in one machine because it is only done once while reading the input. The nnz of G become more spread and less diagonal, causing problems when calculating G * m + k = b.<br /><br /> These 3 operations (except the permutation) are performed in each iteration of my algorithm.<br /><br /> So, my questions are.<br /> -What are the characteristics of G that lead to a good speedup in the operations I described? Am I missing something and too much obsessed with the diagonal block?<br /><br /> -Is there a better way to permute A without permute G and still get the same result using 1 or N machines?<br /><br /><br /> I have been avoiding asking for help for a while. I'm very sorry for the long email.<br /> Thank you very much for your time.<br /> Best Regards,<br /> Nelson</blockquote>
</blockquote>
</blockquote>
</div>
<br /><br clear="all" />-- <br />
<div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br />-- Norbert Wiener</div>
</div>
</div>
</blockquote>
<div> </div>
</body></html>