[petsc-users] Is OpenMP still available for PETSc?
Matthew Knepley
knepley at gmail.com
Mon Jul 3 08:14:10 CDT 2017
On Mon, Jul 3, 2017 at 4:18 AM, Damian Kaliszan <damian at man.poznan.pl>
wrote:
> Hi,
>
> OK. So this is clear now.
> Maybe you will be able to help the answer the question I raised
> some time ago, why, when
> submitting a slurm job and setting different number of
> --cpus-per-task, the execution times of ksp solver
> (gmres type) vary when max number of iterations set to 100.000
> and ksp, when finished, returns ksp.getIterationNumber() (in petsc4py)
> equal to 900 (for some cases) and 100.000 for other.
> It looks like number of cpus per task influence the number of steps
> the solver solves the equation. Is it possible at all?
>
If you have one solver executing on the whole communicator,
PETSC_COMM_WORLD,
all processes will report the same number of iterates. If you have
independent, serial
solvers on each process, PETSC_COMM_SELF, they can report different numbers
of iterates
depending on the convergence criteria.
Matt
> Best,
> Damian
>
> W liście datowanym 1 lipca 2017 (22:23:19) napisano:
>
>
> >> On Jul 1, 2017, at 3:15 PM, Damian Kaliszan <damian at man.poznan.pl>
> wrote:
> >>
> >> Hi,
> >> So... - - with-openmp=0/1 configuration option seems to be useless?...
>
> > It merely enables the compiler flags for compiling with OpenMP;
> > for example if your code has OpenMP in it.
>
> >> In one of my previous messages I wrote that, when openmp enabled, and
> OMP_NUM_THREADS set, I notice different timings for ksp solver. Strange....?
>
> > For the PETSc part yes number of threads shouldn't matter since PETSc
> would only use 1.
> >>
> >>
> >> Best,
> >> Damian
> >> W dniu 1 lip 2017, o 00:50, użytkownik Barry Smith <bsmith at mcs.anl.gov>
> napisał:
> >>
> >> The current version of PETSc does not use OpenMP, you are free to use
> OpenMP in your portions of the code of course. If you want PETSc using
> OpenMP you have to use the old, unsupported version of PETSc. We never
> found any benefit to using OpenMP.
> >>
> >> Barry
> >>
> >> On Jun 30, 2017, at 5:40 PM, Danyang Su <danyang.su at gmail.com> wrote:
> >>
> >> Dear All,
> >>
> >> I recalled there was OpenMP available for PETSc for the old
> development version. When google "petsc hybrid mpi openmp", there returned
> some papers about this feature. My code was first parallelized using OpenMP
> and then redeveloped using PETSc, with OpenMP kept but not used together
> with MPI. Before retesting the code using hybrid mpi-openmp, I picked one
> PETSc example ex10 by adding "omp_set_num_threads(max_threads);" under
> PetscInitialize.
> >>
> >> The PETSc is the current development version configured as follows
> >>
> >> --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-debugging=0
> --CFLAGS=-fopenmp --CXXFLAGS=-fopenmp --FFLAGS=-fopenmp COPTFLAGS="-O3
> -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native"
> FOPTFLAGS="-O3 -march=native -mtune=native" --with-large-file-io=1
> --download-cmake=yes --download-mumps --download-scalapack
> --download-parmetis --download-metis --download-ptscotch
> --download-fblaslapack --download-mpich --download-hypre
> --download-superlu_dist --download-hdf5=yes --with-openmp --with-threadcomm
> --with-pthreadclasses --with-openmpclasses
> >>
> >> The code can be successfully compiled. However, when I run the code
> with OpenMP, it does not work, the time shows no change in performance if 1
> or 2 threads per processor is used. Also, the CPU/Threads usage indicates
> that no thread is used.
> >>
> >> I just wonder if OpenMP is still available in the latest version,
> though it is not recommended to use.
> >>
> >> mpiexec -n 2 ./ex10 -f0 mat_rhs_pc_nonzero/a_react_in_2.bin -rhs
> mat_rhs_pc_nonzero/b_react_in_2.bin -ksp_rtol 1.0e-20 -ksp_monitor
> -ksp_error_if_not_converged -sub_pc_factor_shift_type nonzero -mat_view
> ascii::ascii_info -log_view -max_threads 1 -threadcomm_type openmp
> -threadcomm_nthreads 1
> >>
> >> KSPSolve 1 1.0 8.9934e-01 1.0 1.03e+09 1.0 7.8e+01
> 3.6e+04 7.8e+01 69 97 89 6 76 89 97 98 98 96 2290
> >> PCSetUp 2 1.0 8.9590e-02 1.0 2.91e+07 1.0 0.0e+00
> 0.0e+00 0.0e+00 7 3 0 0 0 9 3 0 0 0 648
> >> PCSetUpOnBlocks 2 1.0 8.9465e-02 1.0 2.91e+07 1.0 0.0e+00
> 0.0e+00 0.0e+00 7 3 0 0 0 9 3 0 0 0 649
> >> PCApply 40 1.0 3.1993e-01 1.0 2.70e+08 1.0 0.0e+00
> 0.0e+00 0.0e+00 24 25 0 0 0 32 25 0 0 0 1686
> >>
> >> mpiexec -n 2 ./ex10 -f0 mat_rhs_pc_nonzero/a_react_in_2.bin -rhs
> mat_rhs_pc_nonzero/b_react_in_2.bin -ksp_rtol 1.0e-20 -ksp_monitor
> -ksp_error_if_not_converged -sub_pc_factor_shift_type nonzero -mat_view
> ascii::ascii_info -log_view -max_threads 2 -threadcomm_type openmp
> -threadcomm_nthreads 2
> >>
> >> KSPSolve 1 1.0 8.9701e-01 1.0 1.03e+09 1.0 7.8e+01
> 3.6e+04 7.8e+01 69 97 89 6 76 89 97 98 98 96 2296
> >> PCSetUp 2 1.0 8.7635e-02 1.0 2.91e+07 1.0 0.0e+00
> 0.0e+00 0.0e+00 7 3 0 0 0 9 3 0 0 0 663
> >> PCSetUpOnBlocks 2 1.0 8.7511e-02 1.0 2.91e+07 1.0 0.0e+00
> 0.0e+00 0.0e+00 7 3 0 0 0 9 3 0 0 0 664
> >> PCApply 40 1.0 3.1878e-01 1.0 2.70e+08 1.0 0.0e+00
> 0.0e+00 0.0e+00 24 25 0 0 0 32 25 0 0 0 1692
> >>
> >> Thanks and regards,
> >>
> >> Danyang
> >>
> >>
> >> <ex10.c><makefile.txt>
> >>
>
>
>
>
> -------------------------------------------------------
> Damian Kaliszan
>
> Poznan Supercomputing and Networking Center
> HPC and Data Centres Technologies
> ul. Jana Pawła II 10
> 61-139 Poznan
> POLAND
>
> phone (+48 61) 858 5109
> e-mail damian at man.poznan.pl
> www - http://www.man.poznan.pl/
> -------------------------------------------------------
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
http://www.caam.rice.edu/~mk51/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20170703/8cc98f2f/attachment.html>
More information about the petsc-users
mailing list