<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Apr 6, 2018 at 9:17 AM, Yongxiang Wu <span dir="ltr"><<a href="mailto:yongxiang27@gmail.com" target="_blank">yongxiang27@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello Matt, <div>thanks for your quick reply. My situation is a bit different. What I am solving is a generalized eigenvalue problem. I only recently switched from scipy.eigs to slepc, because slepc support parallel eigenvalue problem. In other word, I am adapting my old code to utilizing slepc/petsc parallel support. To have a first test, I saved the matrix from eigenvalue problem with old code. Then read in this matrix for slepc/petsc validation. That's why I need to scatter it to processors. Do you have any idea how can I achieve this? </div><div></div></div></blockquote><div><br></div><div>Ah, the best way to do this is to first convert your matrix from the format it is in to PETSc binary format. Then you</div><div>can load it in parallel automatically. So, I would</div><div><br></div><div> a) Load my matrix</div><div> b) Create a serial PETSc matrix from that using petc4py</div><div> c) Save that matrix using a binary viewer</div><div><br></div><div>Then in your parallel code you can jsut MatLoad() that matrix, and it will automatically be</div><div>distributed.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>with regards</div><div>Yongxiang</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 6 April 2018 at 15:00, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Fri, Apr 6, 2018 at 8:44 AM, Yongxiang Wu <span dir="ltr"><<a href="mailto:yongxiang27@gmail.com" target="_blank">yongxiang27@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p style="margin:0px;padding:0px;word-wrap:break-word;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:-0.07px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Hello, everyone,</p><p style="margin:0px;padding:0px;word-wrap:break-word;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:-0.07px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">I already have a scipy sparse square matrix<span> </span><code style="font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Courier,monospace;font-size:12px;line-height:1.4;padding:1px 3px;border:0px;border-radius:3px;background:rgb(244,245,247);box-sizing:border-box;display:inline-block;max-width:100%;overflow-x:auto;vertical-align:bottom;white-space:nowrap">L0</code><span> </span>. Since my problem is large, parallel run is preferred. My Question is, how can I scatter my<span> </span><code style="font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Courier,monospace;font-size:12px;line-height:1.4;padding:1px 3px;border:0px;border-radius:3px;background:rgb(244,245,247);box-sizing:border-box;display:inline-block;max-width:100%;overflow-x:auto;vertical-align:bottom;white-space:nowrap">L0</code><span> </span>to each of the processors? In the following code, I can get the indices of the localized part of the matrix. In the tutorial, the matrix element are directly assign with value, but in my case, the matrix are so large, assign each element in loop (commented code) is not efficient. So if any function would do the mpi scatter work? </p></div></blockquote><div><br></div></span><div>Hi Yongxiang,</div><div><br></div><div>It would be really anomalous for what you propose to results in any speedup at all. If the matrix is large,</div><div>it will not fit on one process. Any speedup from using more processes will be eaten up by the time to</div><div>communicate the matrix. I would focus on generating the matrix in parallel.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p style="margin:0px;padding:0px;word-wrap:break-word;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:-0.07px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">With regards and <span style="letter-spacing:-0.07px">Thanks.</span></p><div class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-codehilite m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-language-python" style="margin:10px 0px 0px;padding:0px;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:-0.07px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><pre style="margin:0px;padding:5px 10px;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Courier,monospace;font-size:12px;line-height:1.4;background:rgb(244,245,247);border:0px;border-radius:3px;overflow-x:auto;word-wrap:normal;letter-spacing:0px"><span></span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-kn" style="font-weight:700">import</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-nn" style="color:rgb(85,85,85)">sys</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">,</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-nn" style="color:rgb(85,85,85)">slepc4py</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">slepc4py</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">init</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">(</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">sys</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">argv</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">)</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-kn" style="font-weight:700">from</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-nn" style="color:rgb(85,85,85)">petsc4py</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-kn" style="font-weight:700">import</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">PETSc</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-kn" style="font-weight:700">from</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-nn" style="color:rgb(85,85,85)">slepc4py</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-kn" style="font-weight:700">import</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">SLEPc</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">opts</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">=</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">PETSc</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">Options</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">()</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">opts</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">setValue</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">(</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-s1" style="color:rgb(187,136,68)">'-st_pc_factor_m<wbr>at_solver_package'</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">,</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-s1" style="color:rgb(187,136,68)">'mumps'</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">)</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">A</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">=</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">PETSc</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">Mat</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">()</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">createAIJ</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">(</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">size</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">=</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">L0</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n"><wbr>shape</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">,</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">comm</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">=</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">PETSc</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">COMM_WORLD</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">)</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">A</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">setUp</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">()</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">Istart</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">,</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">Iend</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">=</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">A</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">getOwnershipRange</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">()</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-c1" style="font-style:italic;color:rgb(153,153,136)"># for I in range(Istart,Iend):</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-c1" style="font-style:italic;color:rgb(153,153,136)"># for J in range(0,L0.shape[0]):</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-c1" style="font-style:italic;color:rgb(153,153,136)"># A[I,J] = L0[I,J]</span>
</pre></div><p style="margin:12px 0px 0px;padding:0px;word-wrap:break-word;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:-0.07px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">The flowing code, would make the assignment from the scipy sparse matrix<span> </span><code style="font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Courier,monospace;font-size:12px;line-height:1.4;padding:1px 3px;border:0px;border-radius:3px;background:rgb(244,245,247);box-sizing:border-box;display:inline-block;max-width:100%;overflow-x:auto;vertical-align:bottom;white-space:nowrap">L0</code><span> </span>to PETSc matrix A. But this would only work for one process.</p><div class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-codehilite m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-language-python" style="margin:10px 0px 0px;padding:0px;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:-0.07px;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><pre style="margin:0px;padding:5px 10px;font-family:SFMono-Medium,"SF Mono","Segoe UI Mono","Roboto Mono","Ubuntu Mono",Menlo,Courier,monospace;font-size:12px;line-height:1.4;background:rgb(244,245,247);border:0px;border-radius:3px;overflow-x:auto;word-wrap:normal;letter-spacing:0px"><span></span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">A</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">=</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">PETSc</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">Mat</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">()</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">createAIJ</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">(</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">size</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">=</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">L0</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n"><wbr>shape</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">,</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">csr</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">=</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">(</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">L0</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">indptr</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">,</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">L0</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">indices</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">,</span>
<span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">L0</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">data</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">),</span> <span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">comm</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">=</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">PETSc</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-o" style="font-weight:700">.</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-n">COMM_WORLD</span><span class="m_3954583165961064864m_-3129048948513139848m_-6351947006644004745gmail-p">)</span></pre></div><br></div>
</blockquote></span></div><span class="m_3954583165961064864HOEnZb"><font color="#888888"><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_3954583165961064864m_-3129048948513139848gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</font></span></font></span></div></div>
</blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>