[petsc-users] how to run petsc in MPI mode correct?

[Stefano Zampini]

2016-10-27 11:11 GMT+03:00 Ji Zhang <gotofd at gmail.com>:

> Dear all,
>
> I'm using petsc as a solver for my project. However, the solver in
> parallel mode creates much more process then my expectation.
>
> The code using python and petsc4py. The machine have 4 cores.
> (a).  If I run it directly, petsc uses only 1 process to assemble the
> matrix, and creates 4 process to solve the equations,
> (b).  If I use comment 'mpirun -n 4', petsc uses 4 process to assemble the
> matrix, but creates 16 process to solve the equations,
>

What do you mean by "PETSc creates 16 processes"? PETSc does not create
processes.
What's the output of PETSc.COMM_WORLD.getSize()?

My feeling is that you have some python component (numpy?) or the
BLAS/LAPACK library that is multithreaded. Rerun using OMP_NUM_THREADS=1
(or MKL_NUM_THREADS=1)
If this does not fix your issues, try running under strace


> I have checked my own python code,, the main component associates with
> matrix create is as follow:
>
> m = PETSc.Mat().create(comm=PETSc.COMM_WORLD)
> m.setSizes(((None, n_vnode[0]*3), (None, n_fnode[0]*3)))
> m.setType('dense')
> m.setFromOptions()
> m.setUp()
> m_start, m_end = m.getOwnershipRange()
> for i0 in range(m_start, m_end):
>     delta_xi = fnodes - vnodes[i0//3]
>     temp1 = delta_xi ** 2
>     delta_2 = np.square(delta)  # delta_2 = e^2
>     delta_r2 = temp1.sum(axis=1) + delta_2  # delta_r2 = r^2+e^2
>     delta_r3 = delta_r2 * np.sqrt(delta_r2)  # delta_r3 = (r^2+e^2)^1.5
>     temp2 = (delta_r2 + delta_2) / delta_r3  # temp2 = (r^2+2*e^2)/(r^2+e^2)^1.5
>     if i0 % 3 == 0:       # x axis
>         m[i0, 0::3] = ( temp2 + np.square(delta_xi[:, 0]) / delta_r3 ) / (8 * np.pi)  # Mxx
>         m[i0, 1::3] = delta_xi[:, 0] * delta_xi[:, 1] / delta_r3 / (8 * np.pi)    # Mxy
>         m[i0, 2::3] = delta_xi[:, 0] * delta_xi[:, 2] / delta_r3 / (8 * np.pi)    # Mxz
>     elif i0 % 3 == 1:     # y axis
>         m[i0, 0::3] = delta_xi[:, 0] * delta_xi[:, 1] / delta_r3 / (8 * np.pi)    # Mxy
>         m[i0, 1::3] = ( temp2 + np.square(delta_xi[:, 1]) / delta_r3 ) / (8 * np.pi)  # Myy
>         m[i0, 2::3] = delta_xi[:, 1] * delta_xi[:, 2] / delta_r3 / (8 * np.pi)    # Myz
>     else:     # z axis
>         m[i0, 0::3] = delta_xi[:, 0] * delta_xi[:, 2] / delta_r3 / (8 * np.pi)    # Mxz
>         m[i0, 1::3] = delta_xi[:, 1] * delta_xi[:, 2] / delta_r3 / (8 * np.pi)    # Myz
>         m[i0, 2::3] = ( temp2 + np.square(delta_xi[:, 2]) / delta_r3 ) / (8 * np.pi)  # Mzz
> m.assemble()
>
>
>
> the main component associates to petsc solver is as follow:
>
> ksp = PETSc.KSP()
> ksp.create(comm=PETSc.COMM_WORLD)
> ksp.setType(solve_method)
> ksp.getPC().setType(precondition_method)
> ksp.setOperators(self._M_petsc)
> ksp.setFromOptions()
> ksp.solve(velocity_petsc, force_petsc)
>
> Is there any one could give me some suggestion? Thanks.
> 此致
>     敬礼
> 张骥（博士研究生）
> 北京计算科学研究中心
> 北京市海淀区西北旺东路10号院东区9号楼 （100193）
>
> Best,
> Regards,
> Zhang Ji, PhD student
> Beijing Computational Science Research Center
> Zhongguancun Software Park II, No. 10 Dongbeiwang West Road, Haidian
> District, Beijing 100193, China
>



-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20161027/d28d320e/attachment-0001.html>