[petsc-dev] configuring hypre on batch system

Fri Jan 9 16:00:47 CST 2015

I've have a test up and running but hypre and GAMG are running very very
slow. The test only has about 100 equation per core.  Jed mentioned 20K
cycles to start OMP parallel (really?) which would explain a lot.  Do I
understand that correctly Jed?

On Fri, Jan 9, 2015 at 1:44 PM, Abhyankar, Shrirang G. <abhyshr at mcs.anl.gov>
wrote:

>  The values need to be comma separated. You can specify the affinities in
> different ways as described in the PetscOptionsGetIntArray docs.
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscOptionsGetIntArray.html
> .
>
>  I'll fix the docs for threadcomm.
>
>   From: Mark Adams <mfadams at lbl.gov>
> Date: Fri, 9 Jan 2015 13:28:48 -0500
> To: Shri <abhyshr at mcs.anl.gov>
> Cc: barry smith <bsmith at mcs.anl.gov>, petsc-dev mailing list <
> petsc-dev at mcs.anl.gov>
>
> Subject: Re: [petsc-dev] configuring hypre on batch system
>
>  OK, threads seem to be working as advertised but I still get this error
> with
>
>  -threadcomm_nthreads 8
> -threadcomm_affinities 0 1 2 3 4 5 6 7
>
>  I'm guessing it must do the right thing.
>
>  Thanks again,
> Mark
>
>  SOLVER_INIT: make partitioning with    2872/   3705 real vertices
> [10]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [10]PETSC ERROR: Nonconforming object sizes
> [10]PETSC ERROR: Must set affinities for all threads, Threads = 8, Core
> affinities set = 1
> [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [10]PETSC ERROR: Petsc Development GIT revision: v3.5.2-1345-g927ffcc  GIT
> Date: 2015-01-08 16:04:39 -0700
> [10]PETSC ERROR: ./xgc2 on a arch-titan-opt-pgi named nid02295 by adams
> Fri Jan  9 13:20:52 2015
> [10]PETSC ERROR: Configure options --COPTFLAGS="-mp -fast"
> --CXXOPTFLAGS="-mp -fast" --FOPTFLAGS="-mp -fast" --with-threadcomm
> --with-pthreadclasses --with-openmp --download-hypre --download-metis
> --download-parmetis --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
> --with-cxxlib-autodetect=0 --with-fc=ftn --with-fortranlib-autodetect=0
> --with-shared-libraries=0 --known-mpi-shared-libraries=1 --with-x=0
> --with-debugging=0 PETSC_ARCH=arch-titan-opt-pgi
> PETSC_DIR=/lustre/atlas2/env003/scratch/adams/petsc2
> [10]PETSC ERROR: #1 PetscThreadCommSetAffinities() line 431 in
> /lustre/atlas2/env003/scratch/adams/petsc2/src/sys/threadcomm/interface/threadcomm.c
> [10]PETSC ERROR: #2 PetscThreadCommWorldInitialize() line 1231 in
> /lustre/atlas2/env003/scratch/adams/petsc2/src/sys/threadcomm/interface/threadcomm.c
> [10]PETSC ERROR: #3 PetscGetThreadCommWorld() line 82 in
> /lustre/atlas2/env003/scratch/adams/petsc2/src/sys/threadcomm/interface/threadcomm.c
> [10]PETSC ERROR: #4 PetscCommGetThreadComm() line 117 in
> /lustre/atlas2/env003/scratch/adams/petsc2/src/sys/threadcomm/interface/threadcomm.c
> [10]PETSC ERROR: #5 PetscCommDuplicate() line 195 in
> /lustre/atlas2/env003/scratch/adams/petsc2/src/sys/objects/tagm.c
> [10]PETSC ERROR: #6 PetscHeaderCreate_Private() line 59 in
> /lustre/atlas2/env003/scratch/adams/petsc2/src/sys/objects/inherit.c
>
>
> On Fri, Jan 9, 2015 at 12:25 AM, Abhyankar, Shrirang G. <
> abhyshr at mcs.anl.gov> wrote:
>
>>  Mark,
>>   The input for -threadcomm_affinities are the processor numbers
>>
>>  So -threadcomm_nthreads 4
>>       -threadcomm_affinities 0 1 2 3
>>
>>  will pin the 4 threads to processors 0,1,2,3. Unfortunately, there is
>> no standardization of processor number mapping on physical and/or logical
>> cores (it is decided by the OS I think). For example, on one node with two
>> quad-core CPUs (total 8 processors, no hyperthreading), the 1st CPU may
>> have processor numbers 0,1,3,5, while the other 2,4,6,8. On another node
>> with similar hardware, the processor numbers may be 0,1,2,3 on the 1st CPU
>> and 4,5,6,7 on the second. Hence, tools like likwid or hwloc are very
>> helpful for getting the hardware layout. You may also obtain this info by
>> looking at /proc/cupinfo on linux.
>>
>>  Shri
>>  From: Mark Adams <mfadams at lbl.gov>
>> Date: Thu, 8 Jan 2015 21:43:30 -0500
>> To: barry smith <bsmith at mcs.anl.gov>
>> Cc: petsc-dev mailing list <petsc-dev at mcs.anl.gov>
>> Subject: Re: [petsc-dev] configuring hypre on batch system
>>
>>
>>> > -threadcomm_affinities 0 1 2 3 4 5 6 7  ?????
>>>
>>> I    don't know what the flag is here
>>>
>>>
>>  Neither do I.  The web page
>> http://www.mcs.anl.gov/petsc/features/threads.html says:
>>
>>
>>    -
>>    -threadcomm_affinities <list_of_affinities>: Sets the core affinities
>>    of threads
>>
>>  I'm not sure what to put here ...
>>
>>
>>
>>> > -threadcomm_type openmp
>>> >
>>> > Then would I get threaded MatVec and other CG + MG stuff?  I know this
>>> will not be faster but I just need data to corroborate what we all know.
>>> And I don't care about setup.
>>>
>>>   Depends on the smoother, we don't have any threaded SOR, if you using
>>> Jacobi + Cheyby it will be threaded.
>>>
>>>
>>  Oh right, thanks,
>>
>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150109/10af8d91/attachment.html>