[petsc-dev] CUDA + OMP make error

[Mark Adams]

I was looking into assembling matrices with threads. I have a coloring to
avoid conflicts.

Turning off all the logging seems way overkill and for methods that can get
called in a thread then we could use PETSC_HAVE_THREADSAFTEY (thingy) to
protect logging functions. So one can still get timings for the whole
assembly process, just not for MatSetValues. Few people are going to do
this. I don't think it will be a time sink, and if it is we just revert
back to saying 'turn logging off'. I don't see a good argument for
insisting on turning off logging, it is pretty important, if we just say
that we are going to protect methods as needed.

It is not a big deal, I am just exploring this idea. It is such a basic
concept in shared memory sparse linear algebra that it seems like a good
thing to be able to support and have in an example to say we can assemble
matrices in threads (not that it is a great idea). We have all the tools
(eg, coloring methods) that it is just a matter of protecting code a few
methods. I use DMPlex MatClosure instead of MatSetValues and this is where
I die now with non-thread safe code. We have an idea, from Jed, on how to
fix it.

Anyway, thanks for your help, but I think we should hold off on doing
anything until we have some consensus that this would be a good idea to put
some effort into getting a thread safe PETSc that can support OMP matrix
assembly with a nice compact example.

Thanks again,
Mark

On Mon, Apr 13, 2020 at 5:44 PM Junchao Zhang <junchao.zhang at gmail.com>
wrote:

> Mark,
>  I saw you had "--with-threadsaftey --with-log=0".  Do you really want to
> call petsc from multiple threads (in contrast to letting petsc call
> other libraries, e.g., BLAS, doing multithreading)?  If not, you can
> drop --with-threadsaftey.
>  I have https://gitlab.com/petsc/petsc/-/merge_requests/2714 that should
> fix your original compilation errors.
>
> --Junchao Zhang
>
> On Mon, Apr 13, 2020 at 2:07 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> https://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html
>>
>> and I see this on my Mac:
>>
>> 14:23 1 mark/feature-xgc-interface-rebase *= ~/Codes/petsc$
>> ../arch-macosx-gnu-O-omp.py
>>
>>
>>
>> ===============================================================================
>>              Configuring PETSc to compile on your system
>>
>>
>> ===============================================================================
>> ===============================================================================
>>
>>
>>                Warning: PETSC_ARCH from environment does not match
>> command-line or name of script.
>>
>>                                      Warning: Using from command-line or
>> name of script: arch-macosx-gnu-O-omp, ignoring environment:
>> arch-macosx-gnu-g
>>
>> ===============================================================================
>>
>>
>>          TESTING: configureLibraryOptions from
>> PETSc.options.libraryOptions(config/PETSc/options/libraryOptions.py:37)
>>
>>
>>  *******************************************************************************
>>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for
>> details):
>>
>> -------------------------------------------------------------------------------
>> Must use --with-log=0 with --with-threadsafety
>>
>> *******************************************************************************
>>
>>
>> On Mon, Apr 13, 2020 at 2:54 PM Junchao Zhang <junchao.zhang at gmail.com>
>> wrote:
>>
>>>
>>>
>>>
>>> On Mon, Apr 13, 2020 at 12:06 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> BTW, I can build on SUMMIT with logging and OMP, apparently. I also
>>>> seem to be able to build with debugging. Both of which are not allowed
>>>> according the the docs. I am puzzled.
>>>>
>>>  What are "the docs"?
>>>
>>>>
>>>> On Mon, Apr 13, 2020 at 12:05 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>> I think the problem is that you have to turn off logging with openmp
>>>>> and the (newish) GPU timers did not protect their timers.
>>>>>
>>>>> I don't see a good reason to require logging be turned off with OMP.
>>>>> We could use PETSC_HAVE_THREADSAFETY to protect logs that we care about
>>>>> (eg, in MatSetValues) and as users discover more things that they want to
>>>>> call in an OMP thread block, then tell them to turn logging off and we will
>>>>> fix it when we can.
>>>>>
>>>>> Any thoughts on the idea of letting users keep logging with openmp?
>>>>>
>>>>> On Mon, Apr 13, 2020 at 11:40 AM Junchao Zhang <
>>>>> junchao.zhang at gmail.com> wrote:
>>>>>
>>>>>> Yes. Looks we need to include petsclog.h. Don't know why OMP
>>>>>> triggered the error.
>>>>>> --Junchao Zhang
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 13, 2020 at 9:59 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>
>>>>>>> Should I do an MR to fix this?
>>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20200413/7938d5c5/attachment.html>