<div dir="ltr">Thanks Pierre,<div><br></div><div>This page should be linked on the 'google petsc threads' page.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 9, 2015 at 8:15 AM, Pierre Jolivet <span dir="ltr"><<a href="mailto:pierre.jolivet@ljll.math.upmc.fr" target="_blank">pierre.jolivet@ljll.math.upmc.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Maybe you should try to re-configure --with-threadcomm --with-pthreadclasses and/or --with-openmp, cf. <a href="http://www.mcs.anl.gov/petsc/documentation/installation.html#threads" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/installation.<u></u>html#threads</a>.<br>
<br>
Pierre<span class=""><br>
<br>
On 2015-01-09 14:05, Mark Adams wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Humm, pthread does not work either:<br>
<br>
[0]PETSC ERROR: --------------------- Error Message<br>
------------------------------<u></u>------------------------------<u></u>--<br>
[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing<br>
package:<br>
<a href="http://www.mcs.anl.gov/petsc/documentation/installation.html#external" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/installation.<u></u>html#external</a><br></span>
[1]<span class=""><br>
[0]PETSC ERROR: Unable to find requested PetscThreadComm type pthread<br>
[0]PETSC ERROR: See<br>
</span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/faq.html</a> [2] for trouble<div><div class="h5"><br>
shooting.<br>
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.2-1345-g927ffcc<br>
GIT Date: 2015-01-08 16:04:39 -0700<br>
[0]PETSC ERROR: ./xgc2 on a arch-titan-opt-pgi named nid07141 by adams<br>
Fri Jan  9 08:04:32 2015<br>
[0]PETSC ERROR: Configure options --COPTFLAGS="-mp -fast"<br>
--CXXOPTFLAGS="-mp -fast" --FOPTFLAGS="-mp -fast" --download-hypre<br>
--download-metis --download-parmetis --with-cc=cc<br>
--with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0<br>
--with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0<br>
--known-mpi-shared-libraries=1 --with-x=0 --with-debugging=0<br>
PETSC_ARCH=arch-titan-opt-pgi<br>
PETSC_DIR=/lustre/atlas2/<u></u>env003/scratch/adams/petsc2<br>
[0]PETSC ERROR: #1 PetscThreadCommSetType() line 512 in<br>
/lustre/atlas2/env003/scratch/<u></u>adams/petsc2/src/sys/<u></u>threadcomm/interface/<u></u>threadcomm.c<br>
[0]PETSC ERROR: #2 PetscThreadCommWorldInitialize<u></u>() line 1250 in<br>
/lustre/atlas2/env003/scratch/<u></u>adams/petsc2/src/sys/<u></u>threadcomm/interface/<u></u>threadcomm.c<br>
[0]PETSC ERROR: #3 PetscGetThreadCommWorld() line 82 in<br>
/lustre/atlas2/env003/scratch/<u></u>adams/petsc2/src/sys/<u></u>threadcomm/interface/<u></u>threadcomm.c<br>
[0]PETSC ERROR: #4 PetscCommGetThreadComm() line 117 in<br>
/lustre/atlas2/env003/scratch/<u></u>adams/petsc2/src/sys/<u></u>threadcomm/interface/<u></u>threadcomm.c<br>
[0]PETSC ERROR: #5 PetscCommDuplicate() line 195 in<br>
/lustre/atlas2/env003/scratch/<u></u>adams/petsc2/src/sys/objects/<u></u>tagm.c<br>
<br>
On Fri, Jan 9, 2015 at 7:42 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
And is openmp not implemented? I get this:<br>
<br>
[0]PETSC ERROR: --------------------- Error Message<br>
------------------------------<u></u>------------------------------<u></u>--<br>
[0]PETSC ERROR: Unknown type. Check for miss-spelling or missing<br>
package:<br>
<br>
</blockquote>
<a href="http://www.mcs.anl.gov/petsc/documentation/installation.html#external" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/installation.<u></u>html#external</a><br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[1]<span class=""><br>
[0]PETSC ERROR: Unable to find requested PetscThreadComm type openmp<br>
[0]PETSC ERROR: See<br>
</span><a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/faq.html</a> [2] for trouble<div><div class="h5"><br>
shooting.<br>
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.2-1345-g927ffcc<br>
GIT Date: 2015-01-08 16:04:39 -0700<br>
[0]PETSC ERROR: ./xgc2 on a arch-titan-opt-pgi named nid09736 by<br>
adams Fri Jan 9 07:39:56 2015<br>
[0]PETSC ERROR: Configure options --COPTFLAGS="-mp -fast"<br>
--CXXOPTFLAGS="-mp -fast" --FOPTFLAGS="-mp -fast" --download-hypre<br>
--download-metis --download-parmetis --with-cc=cc<br>
--with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0<br>
--with-fc=ftn --with-fortranlib-autodetect=0<br>
--with-shared-libraries=0 --known-mpi-shared-libraries=1 --with-x=0<br>
--with-debugging=0 PETSC_ARCH=arch-titan-opt-pgi<br>
PETSC_DIR=/lustre/atlas2/<u></u>env003/scratch/adams/petsc2<br>
[0]PETSC ERROR: #1 PetscThreadCommSetType() line 512 in<br>
<br>
</div></div></blockquote><div><div class="h5">
/lustre/atlas2/env003/scratch/<u></u>adams/petsc2/src/sys/<u></u>threadcomm/interface/<u></u>threadcomm.c<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[0]PETSC ERROR: #2 PetscThreadCommWorldInitialize<u></u>() line 1250 in<br>
<br>
</blockquote>
/lustre/atlas2/env003/scratch/<u></u>adams/petsc2/src/sys/<u></u>threadcomm/interface/<u></u>threadcomm.c<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
<br>
On Fri, Jan 9, 2015 at 12:25 AM, Abhyankar, Shrirang G.<br>
<<a href="mailto:abhyshr@mcs.anl.gov" target="_blank">abhyshr@mcs.anl.gov</a>> wrote:<br>
<br>
Mark,<br>
The input for -threadcomm_affinities are the processor numbers<br>
<br>
So -threadcomm_nthreads 4<br>
-threadcomm_affinities 0 1 2 3<br>
<br>
will pin the 4 threads to processors 0,1,2,3. Unfortunately, there<br>
is no standardization of processor number mapping on physical and/or<br>
logical cores (it is decided by the OS I think). For example, on one<br>
node with two quad-core CPUs (total 8 processors, no<br>
hyperthreading), the 1st CPU may have processor numbers 0,1,3,5,<br>
while the other 2,4,6,8. On another node with similar hardware, the<br>
processor numbers may be 0,1,2,3 on the 1st CPU and 4,5,6,7 on the<br>
second. Hence, tools like likwid or hwloc are very helpful for<br>
getting the hardware layout. You may also obtain this info by<br>
looking at /proc/cupinfo on linux.<br>
<br>
Shri<br>
From: Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
Date: Thu, 8 Jan 2015 21:43:30 -0500<br>
To: barry smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>><br>
Cc: petsc-dev mailing list <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br>
Subject: Re: [petsc-dev] configuring hypre on batch system<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-threadcomm_affinities 0 1 2 3 4 5 6 7 ?????<br>
</blockquote>
<br>
I don't know what the flag is here<br>
<br>
Neither do I. The web page<br>
</div></div><span class=""><a href="http://www.mcs.anl.gov/petsc/features/threads.html" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>features/threads.html</a> [3] says:<br>
<br>
*<br>
-threadcomm_affinities <list_of_affinities>: Sets the core<br>
affinities of threads<br>
<br>
I'm not sure what to put here ...<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-threadcomm_type openmp<br>
<br>
Then would I get threaded MatVec and other CG + MG stuff? I know<br>
</blockquote>
this will not be faster but I just need data to corroborate what we<br>
all know. And I don't care about setup.<br>
<br>
Depends on the smoother, we don't have any threaded SOR, if you<br>
using Jacobi + Cheyby it will be threaded.<br>
<br>
Oh right, thanks,<br>
</span></blockquote>
<br>
<br>
<br>
Links:<br>
------<br>
[1] <a href="http://www.mcs.anl.gov/petsc/documentation/installation.html#external" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/installation.<u></u>html#external</a><br>
[2] <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>documentation/faq.html</a><br>
[3] <a href="http://www.mcs.anl.gov/petsc/features/threads.html" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>features/threads.html</a><br>
</blockquote>
<br>
</blockquote></div><br></div>