<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<title></title>
</head>
<body style="font-family:Arial;font-size:14px">
<p>Hi Junchao,<br>
<br>
I tried it out, but unfortunately, this does not seem to give any imporvements, the code is still much slower when starting more processes.<br>
<br>
<br>
----- Message from Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> ---------<br>
Date: Fri, 12 Jan 2024 09:41:39 -0600<br>
From: Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>><br>
Subject: Re: [petsc-users] Parallel processes run significantly slower<br>
To: Steffen Wilksen | Universitaet Bremen <<a href="mailto:swilksen@itp.uni-bremen.de">swilksen@itp.uni-bremen.de</a>><br>
Cc: Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>>, PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>></p>
<blockquote style="border-left:2px solid blue;margin-left:2px;padding-left:12px;" type="cite">
<div dir="ltr">Hi, Steffen,
<div> Would it be an MPI process binding issue? Could you try running with </div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div>mpiexec --bind-to core -n N python parallel_example.py</div>
</blockquote>
<div><br clear="all">
<div>
<div class="gmail_signature" data-smartmail="gmail_signature" dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div class="gmail_attr" dir="ltr">On Fri, Jan 12, 2024 at 8:52 AM Steffen Wilksen | Universitaet Bremen <<a href="mailto:swilksen@itp.uni-bremen.de">swilksen@itp.uni-bremen.de</a>> wrote:</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="font-family:Arial;font-size:14px">
<p><u>Thank you for your feedback.<br>
@Stefano: the use of my communicator was intentional, since I later intend to distribute M independent calculations to N processes, each process then only needing to do M/N calculations. Of course I don't expect speed up in my example since the number of calculations is constant and not dependent on N, but I would hope that the time each process takes does not increase too drastically with N.<br>
@Barry: I tried to do the STREAMS benchmark, these are my results:<br>
1 23467.9961 Rate (MB/s) 1<br>
2 26852.0536 Rate (MB/s) 1.1442<br>
3 29715.4762 Rate (MB/s) 1.26621<br>
4 34132.2490 Rate (MB/s) 1.45442<br>
5 34924.3020 Rate (MB/s) 1.48817<br>
6 34315.5290 Rate (MB/s) 1.46223<br>
7 33134.9545 Rate (MB/s) 1.41192<br>
8 33234.9141 Rate (MB/s) 1.41618<br>
9 32584.3349 Rate (MB/s) 1.38846<br>
10 32582.3962 Rate (MB/s) 1.38838<br>
11 32098.2903 Rate (MB/s) 1.36775<br>
12 32064.8779 Rate (MB/s) 1.36632<br>
13 31692.0541 Rate (MB/s) 1.35044<br>
14 31274.2421 Rate (MB/s) 1.33263<br>
15 31574.0196 Rate (MB/s) 1.34541<br>
16 30906.7773 Rate (MB/s) 1.31698<br>
<br>
I also attached the resulting plot. As it seems, I get very bad MPI speedup (red curve, right?), even decreasing if I use too many threads. I don't fully understand the reasons given in the discussion you linked since this is all very new to me, but I take that this is a problem with my computer which I can't easily fix, right?<br>
<br>
<br>
----- Message from Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> ---------<br>
Date: Thu, 11 Jan 2024 11:56:24 -0500<br>
From: Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>><br>
Subject: Re: [petsc-users] Parallel processes run significantly slower<br>
To: Steffen Wilksen | Universitaet Bremen <<a href="mailto:swilksen@itp.uni-bremen.de" target="_blank">swilksen@itp.uni-bremen.de</a>><br>
Cc: PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>></u></p>
<blockquote style="border-left:2px solid blue;margin-left:2px;padding-left:12px" type="cite">
<div><u> </u></div>
<u> Take a look at the discussion in <a href="https://petsc.gitlab.io/-/petsc/-/jobs/5814862879/artifacts/public/html/manual/streams.html" target="_blank">https://petsc.gitlab.io/-/petsc/-/jobs/5814862879/artifacts/public/html/manual/streams.html</a> and I suggest you run the streams benchmark from the branch <span style="font-size:large;color:rgb(51,50,56)">barry/2023-09-15/fix-log-pcmpi on your machine to get a baseline for what kind of speedup you can expect. </span></u>
<div><u> </u></div>
<div><u><font color="#333238" size="4"><span> Then let us know your thoughts.</span></font></u></div>
<div><u> </u></div>
<div><u><font color="#333238" size="4"><span> Barry</span></font></u></div>
<div><br>
<div><br id="m_6194710400128315596lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div><u>On Jan 11, 2024, at 11:37 AM, Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>> wrote:</u></div>
<br>
<div>
<div dir="auto"><u>You are creating the matrix on the wrong communicator if you want it parallel. You are using PETSc.COMM_SELF</u></div>
<br>
<div class="gmail_quote">
<div class="gmail_attr" dir="ltr"><u>On Thu, Jan 11, 2024, 19:28 Steffen Wilksen | Universitaet Bremen <<a href="mailto:swilksen@itp.uni-bremen.de" target="_blank">swilksen@itp.uni-bremen.de</a>> wrote:</u></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="font-family:Arial;font-size:14px">
<p><u><u>Hi all,<br>
<br>
I'm trying to do repeated matrix-vector-multiplication of large sparse matrices in python using petsc4py. Even the most simple method of parallelization, dividing up the calculation to run on multiple processes indenpendtly, does not seem to give a singnificant speed up for large matrices. I constructed a minimal working example, which I run using<br>
<br>
mpiexec -n N python parallel_example.py,<br>
<br>
where N is the number of processes. Instead of taking approximately the same time irrespective of the number of processes used, the calculation is much slower when starting more MPI processes. This translates to little to no speed up when splitting up a fixed number of calculations over N processes. As an example, running with N=1 takes 9s, while running with N=4 takes 34s. When running with smaller matrices, the problem is not as severe (only slower by a factor of 1.5 when setting MATSIZE=1e+5 instead of MATSIZE=1e+6). I get the same problems when just starting the script four times manually without using MPI.<br>
I attached both the script and the log file for running the script with N=4. Any help would be greatly appreciated. Calculations are done on my laptop, arch linux version 6.6.8 and PETSc version 3.20.2.<br>
<br>
Kind Regards<br>
Steffen</u></u></p>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<p><br>
<br>
<br>
<u><u>----- End message from Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> -----</u></u><br>
<br>
</p>
</div>
</blockquote>
</div>
</blockquote>
<p><br>
<br>
<br>
<u>----- End message from Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> -----</u><br>
<br>
<br type="_moz"></p>
</body>
</html>