<html><head><title>Re: [petsc-users] Is OpenMP still available for PETSc?</title>
</head>
<body>
<span style=" font-family:'courier new'; font-size: 9pt;">HI,<br>
<br>
W liście datowanym 3 lipca 2017 (15:50:17) napisano:<br>
<br>
</span><table>
<tr>
<td width=2 bgcolor= #0000ff><br>
</td>
<td><span style=" font-family:'tahoma'; font-size: 10pt;">On Mon, Jul 3, 2017 at 8:47 AM, Damian Kaliszan <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><span style=" font-family:'tahoma'; font-size: 10pt;">> wrote:<br>
<span style=" font-size: 9pt;">Hi,<br>
I use MPI.COMM_WORLD as communicator because I use bcast to send around some other info to all processes before reading in the matrices and running solver.<br>
(I can't call bcast on PETSc.COMM_WORLD).<br>
<br>
<span style=" font-size: 10pt;">1) You can call Bcast on PETSC_COMM_WORLD<br>
<br>
2) If you are using WORLD, the number of iterates will be the same on each process since iteration is collective.<br>
<br>
<br>
<br>
<br>
Thanks,<br>
<br>
Matt<br>
<br>
<span style=" font-size: 9pt;">Best,<br>
Damian<br>
<br>
W liście datowanym 3 lipca 2017 (15:14:10) napisano:<br>
<br>
</span></span></span></span><table>
<tr>
<td width=2 bgcolor= #0000ff><br>
</td>
<td><span style=" font-family:'tahoma'; font-size: 10pt;">On Mon, Jul 3, 2017 at 4:18 AM, Damian Kaliszan <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><span style=" font-family:'tahoma'; font-size: 10pt;">> wrote:<br>
Hi,<br>
<br>
OK. So this is clear now.<br>
Maybe you will be able to help the answer the question I raised<br>
some time ago, why, when<br>
submitting a slurm job and setting different number of<br>
--cpus-per-task, the execution times of ksp solver<br>
(gmres type) vary when max number of iterations set to 100.000<br>
and ksp, when finished, returns ksp.getIterationNumber() (in petsc4py)<br>
equal to 900 (for some cases) and 100.000 for other.<br>
It looks like number of cpus per task influence the number of steps<br>
the solver solves the equation. Is it possible at all?<br>
<br>
If you have one solver executing on the whole communicator, PETSC_COMM_WORLD,<br>
all processes will report the same number of iterates. If you have independent, serial<br>
solvers on each process, PETSC_COMM_SELF, they can report different numbers of iterates<br>
depending on the convergence criteria.<br>
<br>
Matt<br>
<br>
Best,<br>
Damian<br>
<br>
W liście datowanym 1 lipca 2017 (22:23:19) napisano:<br>
<br>
<br>
>> On Jul 1, 2017, at 3:15 PM, Damian Kaliszan <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><span style=" font-family:'tahoma'; font-size: 10pt;">> wrote:<br>
>><br>
>> Hi,<br>
>> So... - - with-openmp=0/1 configuration option seems to be useless?...<br>
<br>
> It merely enables the compiler flags for compiling with OpenMP;<br>
> for example if your code has OpenMP in it.<br>
<br>
>> In one of my previous messages I wrote that, when openmp enabled, and OMP_NUM_THREADS set, I notice different timings for ksp solver. Strange....?<br>
<br>
> For the PETSc part yes number of threads shouldn't matter since PETSc would only use 1.<br>
>><br>
>><br>
>> Best,<br>
>> Damian<br>
>> W dniu 1 lip 2017, o 00:50, użytkownik Barry Smith <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a><span style=" font-family:'tahoma'; font-size: 10pt;">> napisał:<br>
>><br>
>> The current version of PETSc does not use OpenMP, you are free to use OpenMP in your portions of the code of course. If you want PETSc using OpenMP you have to use the old, unsupported version of PETSc. We never found any benefit to using OpenMP.<br>
>><br>
>> Barry<br>
>><br>
>> On Jun 30, 2017, at 5:40 PM, Danyang Su <</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="mailto:danyang.su@gmail.com">danyang.su@gmail.com</a><span style=" font-family:'tahoma'; font-size: 10pt;">> wrote:<br>
>> <br>
>> Dear All,<br>
>> <br>
>> I recalled there was OpenMP available for PETSc for the old development version. When google "petsc hybrid mpi openmp", there returned some papers about this feature. My code was first parallelized using OpenMP and then redeveloped using PETSc, with OpenMP kept but not used together with MPI. Before retesting the code using hybrid mpi-openmp, I picked one PETSc example ex10 by adding "omp_set_num_threads(max_threads);" under PetscInitialize.<br>
>> <br>
>> The PETSc is the current development version configured as follows<br>
>> <br>
>> --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-debugging=0 --CFLAGS=-fopenmp --CXXFLAGS=-fopenmp --FFLAGS=-fopenmp COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-large-file-io=1 --download-cmake=yes --download-mumps --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-fblaslapack --download-mpich --download-hypre --download-superlu_dist --download-hdf5=yes --with-openmp --with-threadcomm --with-pthreadclasses --with-openmpclasses<br>
>> <br>
>> The code can be successfully compiled. However, when I run the code with OpenMP, it does not work, the time shows no change in performance if 1 or 2 threads per processor is used. Also, the CPU/Threads usage indicates that no thread is used.<br>
>> <br>
>> I just wonder if OpenMP is still available in the latest version, though it is not recommended to use.<br>
>> <br>
>> mpiexec -n 2 ./ex10 -f0 mat_rhs_pc_nonzero/a_react_in_2.bin -rhs mat_rhs_pc_nonzero/b_react_in_2.bin -ksp_rtol 1.0e-20 -ksp_monitor -ksp_error_if_not_converged -sub_pc_factor_shift_type nonzero -mat_view ascii::ascii_info -log_view -max_threads 1 -threadcomm_type openmp -threadcomm_nthreads 1<br>
>> <br>
>> KSPSolve 1 1.0 8.9934e-01 1.0 1.03e+09 1.0 7.8e+01 3.6e+04 7.8e+01 69 97 89 6 76 89 97 98 98 96 2290<br>
>> PCSetUp 2 1.0 8.9590e-02 1.0 2.91e+07 1.0 0.0e+00 0.0e+00 0.0e+00 7 3 0 0 0 9 3 0 0 0 648<br>
>> PCSetUpOnBlocks 2 1.0 8.9465e-02 1.0 2.91e+07 1.0 0.0e+00 0.0e+00 0.0e+00 7 3 0 0 0 9 3 0 0 0 649<br>
>> PCApply 40 1.0 3.1993e-01 1.0 2.70e+08 1.0 0.0e+00 0.0e+00 0.0e+00 24 25 0 0 0 32 25 0 0 0 1686<br>
>> <br>
>> mpiexec -n 2 ./ex10 -f0 mat_rhs_pc_nonzero/a_react_in_2.bin -rhs mat_rhs_pc_nonzero/b_react_in_2.bin -ksp_rtol 1.0e-20 -ksp_monitor -ksp_error_if_not_converged -sub_pc_factor_shift_type nonzero -mat_view ascii::ascii_info -log_view -max_threads 2 -threadcomm_type openmp -threadcomm_nthreads 2<br>
>> <br>
>> KSPSolve 1 1.0 8.9701e-01 1.0 1.03e+09 1.0 7.8e+01 3.6e+04 7.8e+01 69 97 89 6 76 89 97 98 98 96 2296<br>
>> PCSetUp 2 1.0 8.7635e-02 1.0 2.91e+07 1.0 0.0e+00 0.0e+00 0.0e+00 7 3 0 0 0 9 3 0 0 0 663<br>
>> PCSetUpOnBlocks 2 1.0 8.7511e-02 1.0 2.91e+07 1.0 0.0e+00 0.0e+00 0.0e+00 7 3 0 0 0 9 3 0 0 0 664<br>
>> PCApply 40 1.0 3.1878e-01 1.0 2.70e+08 1.0 0.0e+00 0.0e+00 0.0e+00 24 25 0 0 0 32 25 0 0 0 1692<br>
>> <br>
>> Thanks and regards,<br>
>> <br>
>> Danyang<br>
>> <br>
>> <br>
>> <ex10.c><makefile.txt><br>
>><br>
<br>
<br>
<br>
<br>
-</td>
</tr>
</table>
<br><br>
<br>
<br>
<br>
<br>
<span style=" font-family:'tahoma'; font-size: 10pt;">-- <br>
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
<br>
</span><a style=" font-family:'tahoma'; font-size: 10pt;" href="http://www.caam.rice.edu/~mk51/">http://www.caam.rice.edu/~mk51/</a></td>
</tr>
</table>
<br><br>
<br>
<br>
<span style=" font-family:'Tahoma'; font-size: 10pt;">-------------------------------------------------------<br>
Damian Kaliszan<br>
<br>
Poznan Supercomputing and Networking Center<br>
HPC and Data Centres Technologies<br>
ul. Jana Pawła II 10<br>
61-139 Poznan<br>
POLAND<br>
<br>
phone (+48 61) 858 5109<br>
e-mail </span><a style=" font-family:'courier new'; font-size: 9pt;" href="mailto:damian@man.poznan.pl">damian@man.poznan.pl</a><br>
<span style=" font-family:'courier new'; font-size: 9pt;">www - </span><a style=" font-family:'courier new'; font-size: 9pt;" href="http://www.man.poznan.pl/">http://www.man.poznan.pl/</a><br>
<span style=" font-family:'courier new'; font-size: 9pt;">------------------------------------------------------- </body></html>