<div dir="ltr"><div><div>I'm so apologize for the ambiguity. Let me clarify it. <br></div><br>I'm trying to simulation interactions among different bodies. Now I have calculated the interaction between two of them and stored in the sub-matrix m_ij. What I want to do is to consider the whole interaction and construct all sub-matrices m_ij into a big matrix M, just like this, imaging the problem contain 3 bodies, <br><br> [ m11 m12 m13 ]<br> M = | m21 m22 m23 | ,<br> [ m31 m32 m33 ]<br><br></div>The system is huge that I have to use MPI and a lot of cups. A mcve code is showing below, and I'm using a python wrap of PETSc, however, their grammar is similar. <br><br><pre style="background-color:rgb(248,248,255);color:rgb(0,0,0);font-family:"dejavu sans mono";font-size:9.1pt"><span style="font-weight:bold">import </span>numpy <span style="font-weight:bold">as </span>np<br><span style="font-weight:bold">from </span>petsc4py <span style="font-weight:bold">import </span>PETSc<br><br>mSizes <span style="font-weight:bold">= </span>(<span style="color:rgb(0,153,153)">5</span>, <span style="color:rgb(0,153,153)">8</span>, <span style="color:rgb(0,153,153)">6</span>)<br>mij <span style="font-weight:bold">= </span>[]<br><br><span style="color:rgb(153,153,136);font-style:italic"># create sub-matrices mij<br></span><span style="font-weight:bold">for </span>i <span style="font-weight:bold">in </span><span style="color:rgb(0,134,179)">range</span>(<span style="color:rgb(0,134,179)">len</span>(mSizes))<span style="font-weight:bold">:<br></span><span style="font-weight:bold"> for </span>j <span style="font-weight:bold">in </span><span style="color:rgb(0,134,179)">range</span>(<span style="color:rgb(0,134,179)">len</span>(mSizes))<span style="font-weight:bold">:<br></span><span style="font-weight:bold"> </span>temp_m <span style="font-weight:bold">= </span>PETSc.Mat().create(<span style="color:rgb(102,0,153)">comm</span><span style="font-weight:bold">=</span>PETSc.COMM_WORLD)<br> temp_m.setSizes(((<span style="font-weight:bold">None</span>, mSizes[i]), (<span style="font-weight:bold">None</span>, mSizes[j])))<br> temp_m.setType(<span style="color:rgb(0,128,128);font-weight:bold">'mpidense'</span>)<br> temp_m.setFromOptions()<br> temp_m.setUp()<br> temp_m[<span style="font-weight:bold">:</span>, <span style="font-weight:bold">:</span>] <span style="font-weight:bold">= </span>np.random.random_sample((mSizes[i], mSizes[j]))<br> temp_m.assemble()<br> mij.append(temp_m)<br><br><span style="color:rgb(153,153,136);font-style:italic"># Now we have four sub-matrices. I would like to construct them into a big matrix M.<br></span>M <span style="font-weight:bold">= </span>PETSc.Mat().create(<span style="color:rgb(102,0,153)">comm</span><span style="font-weight:bold">=</span>PETSc.COMM_WORLD)<br>M.setSizes(((<span style="font-weight:bold">None</span>, np.sum(mSizes)), (<span style="font-weight:bold">None</span>, np.sum(mSizes))))<br>M.setType(<span style="color:rgb(0,128,128);font-weight:bold">'mpidense'</span>)<br>M.setFromOptions()<br>M.setUp()<br>mLocations <span style="font-weight:bold">= </span>np.insert(np.cumsum(mSizes), <span style="color:rgb(0,153,153)">0</span>, <span style="color:rgb(0,153,153)">0</span>) <span style="color:rgb(153,153,136);font-style:italic"># mLocations = [0, mSizes]<br></span><span style="font-weight:bold">for </span>i <span style="font-weight:bold">in </span><span style="color:rgb(0,134,179)">range</span>(<span style="color:rgb(0,134,179)">len</span>(mSizes))<span style="font-weight:bold">:<br></span><span style="font-weight:bold"> for </span>j <span style="font-weight:bold">in </span><span style="color:rgb(0,134,179)">range</span>(<span style="color:rgb(0,134,179)">len</span>(mSizes))<span style="font-weight:bold">:<br></span><span style="font-weight:bold"> </span>M[mLocations[i]<span style="font-weight:bold">:</span>mLocations[i<span style="font-weight:bold">+</span><span style="color:rgb(0,153,153)">1</span>], mLocations[j]<span style="font-weight:bold">:</span>mLocations[j<span style="font-weight:bold">+</span><span style="color:rgb(0,153,153)">1</span>]] <span style="font-weight:bold">= </span>mij[i<span style="font-weight:bold">*</span><span style="color:rgb(0,134,179)">len</span>(mSizes)<span style="font-weight:bold">+</span>j][<span style="font-weight:bold">:</span>, <span style="font-weight:bold">:</span>]<br>M.assemble()</pre><br><div>Thanks. <br><br></div><br><div>2016-09-16</div><span><span><span><div>Best, </div><div>Regards,</div><div>Zhang Ji </div><div>Beijing Computational Science Research Center </div><div>E-mail: <a target="_blank" href="mailto:gotofd@gmail.com">gotofd@gmail.com</a><br><br><br><br></div></span></span></span><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature">Wayne</div></div>
<br><div class="gmail_quote">On Thu, Sep 15, 2016 at 8:58 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Thu, Sep 15, 2016 at 4:23 AM, Ji Zhang <span dir="ltr"><<a href="mailto:gotofd@gmail.com" target="_blank">gotofd@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks Matt. It works well for signal core. But is there any solution if I need a MPI program?<br></div></div></blockquote><div><br></div></span><div>It unclear what the stuff below would mean in parallel.</div><div><br></div><div>If you want to assemble several blocks of a parallel matrix that looks like serial matrices, then use</div><div><br></div><div> <a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html" target="_blank">http://www.mcs.anl.gov/<wbr>petsc/petsc-current/docs/<wbr>manualpages/Mat/<wbr>MatGetLocalSubMatrix.html</a></div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div>Thanks. <br></div><div class="gmail_extra"><br clear="all"><div><div>Wayne</div></div>
<br><div class="gmail_quote">On Tue, Sep 13, 2016 at 9:30 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Mon, Sep 12, 2016 at 8:24 PM, Ji Zhang <span dir="ltr"><<a href="mailto:gotofd@gmail.com" target="_blank">gotofd@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Dear all, <br><br>I'm using petsc4py and now face some problems.<br>I have a number of small petsc dense matrices mij, and I want to construct them to a big matrix M like this:<br><br> [ m11 m12 m13 ]<br> M = | m21 m22 m23 | ,<br> [ m31 m32 m33 ]<br>How could I do it effectively?<br><br>Now I'm using the code below:<br><br> # get indexes of matrix mij<br> index1_begin, index1_end = getindex_i( )<br> index2_begin, index2_end = getindex_j( )<br> M[index1_begin:index1_end, index2_begin:index2_end] = mij[:, :]<br>which report such error messages:<br><br> petsc4py.PETSc.Error: error code 56<br> [0] MatGetValues() line 1818 in /home/zhangji/PycharmProjects/<wbr>petsc-petsc-31a1859eaff6/src/m<wbr>at/interface/matrix.c<br> [0] MatGetValues_MPIDense() line 154 in /home/zhangji/PycharmProjects/<wbr>petsc-petsc-31a1859eaff6/src/m<wbr>at/impls/dense/mpi/mpidense.c<br></div></div></div></blockquote><div><br></div></span><div>Make M a sequential dense matrix.</div><div><br></div><div> Matt</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div> [0] No support for this operation for this object type<br> [0] Only local values currently supported<br><br></div>Thanks. <br><br></div><br><div>2016-09-13</div><span><span><span><div>Best, </div><div>Regards,</div><div>Zhang Ji </div><div>Beijing Computational Science Research Center </div><div>E-mail: <a href="mailto:gotofd@gmail.com" target="_blank">gotofd@gmail.com</a><br><br><br></div></span></span></span></div>
</blockquote></span></div><span><font color="#888888"><br><br clear="all"><span><font color="#888888"><div><br></div>-- <br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></font></span></div></div>
</blockquote></div><br></div>
</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div></div></div>
</blockquote></div><br></div>