<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2016-10-27 11:11 GMT+03:00 Ji Zhang <span dir="ltr"><<a href="mailto:gotofd@gmail.com" target="_blank">gotofd@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Dear all, <br><br></div>I'm using petsc as a solver for my project. However, the solver in parallel mode creates much more process then my expectation. <br><br>The code using python and petsc4py. The machine have 4 cores. <br>(a).  If I run it directly, petsc uses only 1 process to assemble the matrix, and creates 4 process to solve the equations,<br>(b).  If I use comment 'mpirun -n 4', petsc uses 4 process to assemble the matrix, but creates 16 process to solve the equations,<br></div></div></blockquote><div><br></div><div>What do you mean by "PETSc creates 16 processes"? PETSc does not create processes.</div><div>What's the output of PETSc.COMM_WORLD.getSize()?</div><div><br></div><div>My feeling is that you have some python component (numpy?) or the BLAS/LAPACK library that is multithreaded. Rerun using OMP_NUM_THREADS=1 (or MKL_NUM_THREADS=1) </div><div>If this does not fix your issues, try running under strace </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>I have checked my own python code,, the main component associates with matrix create is as follow:<br><pre style="background-color:rgb(248,248,255);color:rgb(0,0,0);font-family:"dejavu sans mono";font-size:9.1pt">m <span style="font-weight:bold">= </span>PETSc.Mat().create(<span style="color:rgb(102,0,153)">comm</span><span style="font-weight:bold">=</span>PETSc.<wbr>COMM_WORLD)<br>m.setSizes(((<span style="font-weight:bold">None</span>, n_vnode[<span style="color:rgb(0,153,153)">0</span>]<span style="font-weight:bold">*</span><span style="color:rgb(0,153,153)">3</span>), (<span style="font-weight:bold">None</span>, n_fnode[<span style="color:rgb(0,153,153)">0</span>]<span style="font-weight:bold">*</span><span style="color:rgb(0,153,153)">3</span>)))<br>m.setType(<span style="color:rgb(0,128,128);font-weight:bold">'dense'</span>)<br>m.setFromOptions()<br>m.setUp()<br>m_start, m_end <span style="font-weight:bold">= </span>m.getOwnershipRange()<br><span style="font-weight:bold">for </span>i0 <span style="font-weight:bold">in </span><span style="color:rgb(0,134,179)">range</span>(m_start, m_end)<span style="font-weight:bold">:<br></span><span style="font-weight:bold">    </span>delta_xi <span style="font-weight:bold">= </span>fnodes <span style="font-weight:bold">- </span>vnodes[i0<span style="font-weight:bold">//</span><span style="color:rgb(0,153,153)">3</span>]<br>    temp1 <span style="font-weight:bold">= </span>delta_xi <span style="font-weight:bold">** </span><span style="color:rgb(0,153,153)">2<br></span><span style="color:rgb(0,153,153)">    </span>delta_2 <span style="font-weight:bold">= </span>np.square(delta)  <span style="color:rgb(153,153,136);font-style:italic"># delta_2 = e^2<br></span><span style="color:rgb(153,153,136);font-style:italic">    </span>delta_r2 <span style="font-weight:bold">= </span>temp1.sum(<span style="color:rgb(102,0,153)">axis</span><span style="font-weight:bold">=</span><span style="color:rgb(0,153,153)">1</span>) <span style="font-weight:bold">+ </span>delta_2  <span style="color:rgb(153,153,136);font-style:italic"># delta_r2 = r^2+e^2<br></span><span style="color:rgb(153,153,136);font-style:italic">    </span>delta_r3 <span style="font-weight:bold">= </span>delta_r2 <span style="font-weight:bold">* </span>np.sqrt(delta_r2)  <span style="color:rgb(153,153,136);font-style:italic"># delta_r3 = (r^2+e^2)^1.5<br></span><span style="color:rgb(153,153,136);font-style:italic">    </span>temp2 <span style="font-weight:bold">= </span>(delta_r2 <span style="font-weight:bold">+ </span>delta_2) <span style="font-weight:bold">/ </span>delta_r3  <span style="color:rgb(153,153,136);font-style:italic"># temp2 = (r^2+2*e^2)/(r^2+e^2)^1.5<br></span><span style="color:rgb(153,153,136);font-style:italic">    </span><span style="font-weight:bold">if </span>i0 <span style="font-weight:bold">% </span><span style="color:rgb(0,153,153)">3 </span><span style="font-weight:bold">== </span><span style="color:rgb(0,153,153)">0</span><span style="font-weight:bold">:       </span><span style="color:rgb(153,153,136);font-style:italic"># x axis<br></span><span style="color:rgb(153,153,136);font-style:italic">        </span>m[i0, <span style="color:rgb(0,153,153)">0</span><span style="font-weight:bold">::</span><span style="color:rgb(0,153,153)">3</span>] <span style="font-weight:bold">= </span>( temp2 <span style="font-weight:bold">+ </span>np.square(delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">0</span>]) <span style="font-weight:bold">/ </span>delta_r3 ) <span style="font-weight:bold">/ </span>(<span style="color:rgb(0,153,153)">8 </span><span style="font-weight:bold">* </span>np.pi)  <span style="color:rgb(153,153,136);font-style:italic"># Mxx<br></span><span style="color:rgb(153,153,136);font-style:italic">        </span>m[i0, <span style="color:rgb(0,153,153)">1</span><span style="font-weight:bold">::</span><span style="color:rgb(0,153,153)">3</span>] <span style="font-weight:bold">= </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">0</span>] <span style="font-weight:bold">* </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">1</span>] <span style="font-weight:bold">/ </span>delta_r3 <span style="font-weight:bold">/ </span>(<span style="color:rgb(0,153,153)">8 </span><span style="font-weight:bold">* </span>np.pi)    <span style="color:rgb(153,153,136);font-style:italic"># Mxy<br></span><span style="color:rgb(153,153,136);font-style:italic">        </span>m[i0, <span style="color:rgb(0,153,153)">2</span><span style="font-weight:bold">::</span><span style="color:rgb(0,153,153)">3</span>] <span style="font-weight:bold">= </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">0</span>] <span style="font-weight:bold">* </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">2</span>] <span style="font-weight:bold">/ </span>delta_r3 <span style="font-weight:bold">/ </span>(<span style="color:rgb(0,153,153)">8 </span><span style="font-weight:bold">* </span>np.pi)    <span style="color:rgb(153,153,136);font-style:italic"># Mxz<br></span><span style="color:rgb(153,153,136);font-style:italic">    </span><span style="font-weight:bold">elif </span>i0 <span style="font-weight:bold">% </span><span style="color:rgb(0,153,153)">3 </span><span style="font-weight:bold">== </span><span style="color:rgb(0,153,153)">1</span><span style="font-weight:bold">:     </span><span style="color:rgb(153,153,136);font-style:italic"># y axis<br></span><span style="color:rgb(153,153,136);font-style:italic">        </span>m[i0, <span style="color:rgb(0,153,153)">0</span><span style="font-weight:bold">::</span><span style="color:rgb(0,153,153)">3</span>] <span style="font-weight:bold">= </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">0</span>] <span style="font-weight:bold">* </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">1</span>] <span style="font-weight:bold">/ </span>delta_r3 <span style="font-weight:bold">/ </span>(<span style="color:rgb(0,153,153)">8 </span><span style="font-weight:bold">* </span>np.pi)    <span style="color:rgb(153,153,136);font-style:italic"># Mxy<br></span><span style="color:rgb(153,153,136);font-style:italic">        </span>m[i0, <span style="color:rgb(0,153,153)">1</span><span style="font-weight:bold">::</span><span style="color:rgb(0,153,153)">3</span>] <span style="font-weight:bold">= </span>( temp2 <span style="font-weight:bold">+ </span>np.square(delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">1</span>]) <span style="font-weight:bold">/ </span>delta_r3 ) <span style="font-weight:bold">/ </span>(<span style="color:rgb(0,153,153)">8 </span><span style="font-weight:bold">* </span>np.pi)  <span style="color:rgb(153,153,136);font-style:italic"># Myy<br></span><span style="color:rgb(153,153,136);font-style:italic">        </span>m[i0, <span style="color:rgb(0,153,153)">2</span><span style="font-weight:bold">::</span><span style="color:rgb(0,153,153)">3</span>] <span style="font-weight:bold">= </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">1</span>] <span style="font-weight:bold">* </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">2</span>] <span style="font-weight:bold">/ </span>delta_r3 <span style="font-weight:bold">/ </span>(<span style="color:rgb(0,153,153)">8 </span><span style="font-weight:bold">* </span>np.pi)    <span style="color:rgb(153,153,136);font-style:italic"># Myz<br></span><span style="color:rgb(153,153,136);font-style:italic">    </span><span style="font-weight:bold">else:     </span><span style="color:rgb(153,153,136);font-style:italic"># z axis<br></span><span style="color:rgb(153,153,136);font-style:italic">        </span>m[i0, <span style="color:rgb(0,153,153)">0</span><span style="font-weight:bold">::</span><span style="color:rgb(0,153,153)">3</span>] <span style="font-weight:bold">= </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">0</span>] <span style="font-weight:bold">* </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">2</span>] <span style="font-weight:bold">/ </span>delta_r3 <span style="font-weight:bold">/ </span>(<span style="color:rgb(0,153,153)">8 </span><span style="font-weight:bold">* </span>np.pi)    <span style="color:rgb(153,153,136);font-style:italic"># Mxz<br></span><span style="color:rgb(153,153,136);font-style:italic">        </span>m[i0, <span style="color:rgb(0,153,153)">1</span><span style="font-weight:bold">::</span><span style="color:rgb(0,153,153)">3</span>] <span style="font-weight:bold">= </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">1</span>] <span style="font-weight:bold">* </span>delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">2</span>] <span style="font-weight:bold">/ </span>delta_r3 <span style="font-weight:bold">/ </span>(<span style="color:rgb(0,153,153)">8 </span><span style="font-weight:bold">* </span>np.pi)    <span style="color:rgb(153,153,136);font-style:italic"># Myz<br></span><span style="color:rgb(153,153,136);font-style:italic">        </span>m[i0, <span style="color:rgb(0,153,153)">2</span><span style="font-weight:bold">::</span><span style="color:rgb(0,153,153)">3</span>] <span style="font-weight:bold">= </span>( temp2 <span style="font-weight:bold">+ </span>np.square(delta_xi[<span style="font-weight:bold">:</span>, <span style="color:rgb(0,153,153)">2</span>]) <span style="font-weight:bold">/ </span>delta_r3 ) <span style="font-weight:bold">/ </span>(<span style="color:rgb(0,153,153)">8 </span><span style="font-weight:bold">* </span>np.pi)  <span style="color:rgb(153,153,136);font-style:italic"># Mzz<br></span>m.assemble()</pre><br><br>the main component associates to petsc solver is as follow:<pre style="background-color:rgb(248,248,255);color:rgb(0,0,0);font-family:"dejavu sans mono";font-size:9.1pt">ksp <span style="font-weight:bold">= </span>PETSc.KSP()<br>ksp.create(<span style="color:rgb(102,0,153)">comm</span><span style="font-weight:bold">=</span>PETSc.COMM_<wbr>WORLD)<br>ksp.setType(<span style="font-style:italic">solve_method</span>)<br>ksp.getPC().setType(<span style="font-style:italic">preconditi<wbr>on_method</span>)<br>ksp.setOperators(<span style="color:rgb(148,85,141)">self</span>._M_<wbr>petsc)<br>ksp.setFromOptions()<br>ksp.solve(velocity_petsc, <span style="background-color:rgb(228,228,255)">force_petsc</span>)</pre>Is there any one could give me some suggestion? Thanks. <br></div><div><div><div><div><div class="m_-2564840228686091141gmail_signature"><div dir="ltr"><font size="2" face="System"><span><span>此致<br>    敬礼<br>张骥（</span></span></font><font size="2" face="System"><span><span><font face="System"><span><span>博士研究生）</span></span></font><br>北京计算科学研究中心
<br><span style="background-color:transparent">北京市海淀区西北旺东路10号院东区9号楼 </span>（100193）<br><br></span></span></font><div><font size="2" face="System">Best, </font></div><div><font size="2" face="System">Regards, </font></div><div><font size="2" face="System"><span style="background-color:transparent;color:rgb(80,0,80)">Zhang Ji, PhD student</span><br></font></div><div><font size="2" face="System"><span style="color:rgb(80,0,80);background-color:transparent">Beijing Computational Science Research Center <br>Zhongguancun Software Park II, No. 10 Dongbeiwang West Road, Haidian District, Beijing 100193, China <br></span></font></div><font size="2"></font></div></div></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Stefano</div>
</div></div>