<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi Shri,<br>
<br>
It seems that the problem does not result from the affinities
setting for threads. I have tried several settings, the threads
are set to different cores, but there is no improvement. <br>
<br>
Here is the information of package, core and thread maps<br>
<br>
OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids.<br>
OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid
leaf 11 info<br>
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected:
{0,1,2,3,4,5,6,7,8,9,10,11}<br>
OMP: Info #156: KMP_AFFINITY: 12 available OS procs<br>
OMP: Info #157: KMP_AFFINITY: Uniform topology<br>
OMP: Info #179: KMP_AFFINITY: 1 packages x 6 cores/pkg x 2
threads/core (6 total cores)<br>
OMP: Info #206: KMP_AFFINITY: OS proc to physical thread map:<br>
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0
thread 0<br>
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0
thread 1<br>
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1
thread 0<br>
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1
thread 1<br>
OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 2
thread 0<br>
OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 2
thread 1<br>
OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 3
thread 0<br>
OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3
thread 1<br>
OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 4
thread 0<br>
OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 4
thread 1<br>
OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 5
thread 0<br>
OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 5
thread 1<br>
OMP: Info #144: KMP_AFFINITY: Threads may migrate across 1
innermost levels of machine<br>
<br>
<br>
And here is the internal thread bounding with different
kmp_affinity settings:<br>
<br>
1. KMP_AFFINITY=verbose,granularity=thread,compact<br>
<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc
set {0}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc
set {1}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc
set {2}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc
set {3}<br>
<br>
2. KMP_AFFINITY=verbose,granularity=fine,compact<br>
<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc
set {0}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc
set {1}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc
set {2}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc
set {3}<br>
<br>
3. KMP_AFFINITY=verbose,granularity=fine,compact,1,0<br>
<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc
set {0}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc
set {2}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc
set {4}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc
set {6}<br>
<br>
4. KMP_AFFINITY=verbose,scatter<br>
<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc
set {0,1}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc
set {2,3}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc
set {4,5}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc
set {6,7}<br>
<br>
5. KMP_AFFINITY=verbose,compact (For this setting, two threads are
assigned to the same core)<br>
<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc
set {0,1}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc
set {0,1}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc
set {2,3}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc
set {2,3}<br>
<br>
6. KMP_AFFINITY=verbose,granularity=core,compact (For this
setting, two threads are assigned to the same core)<br>
<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc
set {0,1}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc
set {0,1}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc
set {2,3}<br>
OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc
set {2,3}<br>
<br>
The first 4 settings can assign threads to a distinct core, but
the problem is not solved.<br>
<br>
Thanks,<br>
<br>
Danyang<br>
<br>
<br>
<br>
On 22/09/2013 8:00 PM, Shri wrote:<br>
</div>
<blockquote
cite="mid:B7FB4C6D-7D0E-47BE-B0FF-C8C1A1F5976B@mcs.anl.gov"
type="cite">I think this is definitely an issue with setting the
affinities for threads, i.e., the assignment of threads to cores.
Ideally each thread should be assigned to a distinct core but in
your case all the 4 threads are getting pinned to the same core
resulting in such a massive slowdown. Unfortunately, the thread
affinities for OpenMP are set through environment variables. For
Intel's OpenMP one needs to define the thread affinities through
the environment variable KMP_AFFINITY. See this document here <a
moz-do-not-send="true"
href="http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm">http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011Update/compiler_c/optaps/common/optaps_openmp_thread_affinity.htm</a>.
Try setting the affinities via KMP_AFFINITY and let us know if it
works.
<div><br>
</div>
<div>Shri<br>
<div>
<div>On Sep 21, 2013, at 11:06 PM, Danyang Su wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi Shri,<br>
<br>
Thanks for your info. It can work with the option
-threadcomm_type openmp. But another problem arises, as
described as follows.<br>
<br>
The sparse matrix is 53760*53760 with 1067392 non-zero
entries. If the codes is compiled using PETSc-3.4.2, it
works fine, the equations can be solved quickly and I
can see the speedup. But if the code is compiled using
PETSc-dev with OpenMP option, it takes a long time in
solving the equations and I cannot see any speedup when
more processors are used.<br>
<br>
For PETSc-3.4.2, run by "mpiexec -n 4 ksp_inhm_d
-log_summary log_mpi4_petsc3.4.2.log", the iteration and
runtime are:<br>
Iterations 6 <b><font color="#ff0000">time_assembly
0.4137E-01</font></b> time_ksp 0.9296E-01<br>
<br>
For PETSc-dev, run by "mpiexec -n 1 ksp_inhm_d
-threadcomm_type openmp -threadcomm_nthreads 4
-log_summary log_openmp_petsc_dev.log", the iteration
and runtime are:<br>
Iterations 6 <b><font color="#ff0000">time_assembly
0.3595E+03</font></b> time_ksp 0.2907E+00<br>
<br>
Most of the time <b><font color="#ff0000">'time_assembly
0.3595E+03</font></b>' is spent on the following
codes<br>
do i = istart, iend - 1<br>
ii = ia_in(i+1)<br>
jj = ia_in(i+2)<br>
call MatSetValues(a, ione, i, jj-ii,
ja_in(ii:jj-1)-1, a_in(ii:jj-1), Insert_Values, ierr)<br>
end do <br>
<br>
The log files for both PETSc-3.4.2 and PETSc-dev are
attached.<br>
<br>
Is there anything wrong with my codes or with running
option? The above codes works fine when using MPICH.<br>
<br>
Thanks and regards,<br>
<br>
Danyang<br>
<br>
On 21/09/2013 2:09 PM, Shri wrote:<br>
</div>
<blockquote
cite="mid:933EF7F4-598A-4F88-8902-0613E27F0679@mcs.anl.gov"
type="cite">
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<div>There are three thread communicator types in PETSc.
The default is "no thread" which is basically a
non-threaded version. The other two types are "openmp"
and "pthread". If you want to use OpenMP then use the
option -threadcomm_type openmp.</div>
<div><br>
</div>
<div>Shri</div>
<div><br>
On Sep 21, 2013, at 3:46 PM, Danyang Su <<a
moz-do-not-send="true"
href="mailto:danyang.su@gmail.com">danyang.su@gmail.com</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Hi Barry,<br>
<br>
Thanks for the quick reply.<br>
<br>
After changing<br>
<font color="#ff0000"><i>#if
defined(PETSC_HAVE_PTHREADCLASSES) || defined
(PETSC_HAVE_OPENMP) </i></font><br>
to <br>
<font color="#ff0000"><i>#if
defined(PETSC_HAVE_PTHREADCLASSES)</i></font><br>
and comment out<br>
<font color="#ff0000"><i>#elif
defined(PETSC_HAVE_OPENMP)</i><i><br>
</i><i>PETSC_EXTERN PetscStack *petscstack;</i></font><br>
<br>
It can be compiled and validated with "make test".<br>
<br>
But I still have questions on running the
examples. After rebuild the codes (e.g.,
ksp_ex2f.f), I can run it with "mpiexec -n 1
ksp_ex2f", or "mpiexec -n 4 ksp_ex2f", or "mpiexec
-n 1 ksp_ex2f -threadcomm_nthreads 1", but if I
run it with "mpiexec -n 1 ksp_ex2f
-threadcomm_nthreads 4", there will be a lot of
error information (attached).<br>
<br>
The codes is not modified and there is no OpenMP
routines in it. For the current development in my
project, I want to keep the OpenMP codes in
calculating matrix values, but want to solve it
with PETSc (OpenMP). Is it possible? <br>
<br>
Thanks and regards,<br>
<br>
Danyang<br>
<br>
<br>
<br>
On 21/09/2013 7:26 AM, Barry Smith wrote:<br>
</div>
<blockquote
cite="mid:101244E8-FFFA-4F3F-A4BC-CC47F68D93B7@mcs.anl.gov"
type="cite">
<pre wrap=""> Danyang,
I don't think the || defined (PETSC_HAVE_OPENMP) belongs in the code below.
/* Linux functions CPU_SET and others don't work if sched.h is not included before
including pthread.h. Also, these functions are active only if either _GNU_SOURCE
or __USE_GNU is not set (see /usr/include/sched.h and /usr/include/features.h), hence
set these first.
*/
#if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP)
Edit include/petscerror.h and locate these lines and remove that part and then rerun make all. Let us know if it works or not.
Barry
i.e. replace
#if defined(PETSC_HAVE_PTHREADCLASSES) || defined (PETSC_HAVE_OPENMP)
with
#if defined(PETSC_HAVE_PTHREADCLASSES)
On Sep 21, 2013, at 6:53 AM, Matthew Knepley <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:petsc-maint@mcs.anl.gov"><petsc-maint@mcs.anl.gov></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Sat, Sep 21, 2013 at 12:18 AM, Danyang Su <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:danyang.su@gmail.com"><danyang.su@gmail.com></a> wrote:
Hi All,
I got error information in compiling petsc-dev with openmp in cygwin. Before, I have successfully compiled petsc-3.4.2 and it works fine.
The log files have been attached.
The OpenMP configure test is wrong. It clearly fails to find pthread.h, but the test passes. Then in petscerror.h
we guard pthread.h using PETSC_HAVE_OPENMP. Can someone who knows OpenMP fix this?
Matt
Thanks,
Danyang
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
</pre>
</blockquote>
</blockquote>
<br>
</div>
</blockquote>
<blockquote type="cite">
<div><error.txt></div>
</blockquote>
</blockquote>
<br>
</div>
<span><log_mpi4_petsc3.4.2.log></span><span><log_openmp_petsc_dev.log></span></blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>