[petsc-users] PETSc and AMPI

[Barry Smith]

> On Feb 1, 2015, at 10:15 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>> On Feb 1, 2015, at 9:33 PM, Jed Brown <jed at jedbrown.org> wrote:
>>> 
>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>>  We could possibly "cheat" with AMPI to essentially have
>>>>  PetscInitialize()/Finalize() run through most their code only on
>>>>  thread 0 (assuming we have a way of determining thread 0) 
>>> 
>>> Just guard it with a lock.
>> 
>>  Not sure what you mean here. We want that code to be only run
>>  through once, we don't want or need it to be run by each thread. It
>>  makes no sense for each thread to call TSInitializePackage() for
>>  example.
> 
> Yes, as long as the threads see TSPackageInitialized as true, it's safe
> to call.  So the only problem is that we have a race condition.  One way
> to do this is to make TSPackageInitialized an int.  The code looks
> something like this (depending on the primitives)
> 
>  if (AtomicCompareAndSwap(&TSPackageInitialized,0,1)) {
>    do the initialization
>    TSPackageInitialized = 2;
>    MemoryFenceWrite();
>  } else {
>    while (AccessOnce(TSPackageInitialized) != 2) CPURelax();
>  }

   Way more complicated then needed :-)

> 
>>> What about debugging and profiling?
>> 
>>   This is the same issue for "thread safety"* as well as AMPI. I
>>   don't think AMPI introduces any particular additional hitches.
>> 
>>  Barry
>> 
>> * in the sense that it is currently implemented meaning each thread
>> works on each own objects so doesn't need to lock "MatSetValues"
>> etc. This other "thread safety" has its own can of worms.
> 
> If AMPI creates threads dynamically,

  How could it possibly create threads dynamically and still be running in the MPI 1 model of a consistent number of MPI "processes" at all times? 

> we don't have the luxury of having
> hooks that can run when threads are spawned or finish.  How do we ensure
> that profiling information has been propagated into the parent
> structure?