[petsc-users] PETSc and AMPI

[Dan Negrut]

Barry - I'll make an attempt to translate what you guys discussed in this
thread.
Long story short, I imagine that current thread safety issues in PETSc
combined with the philosophy of Charm, which asynchronously works with
chares, will probably prevent me in the short term from using the
PETSc-charm combo.
Have I got this right?

Thank you guys very much, I really appreciate that you spent the time on
this issue.  
Dan


-----Original Message-----
From: Barry Smith [mailto:bsmith at mcs.anl.gov] 
Sent: Monday, February 02, 2015 7:49 AM
To: Jed Brown
Cc: Matthew Knepley; petsc-users at mcs.anl.gov; Dan Negrut
Subject: Re: [petsc-users] PETSc and AMPI


> On Feb 1, 2015, at 10:15 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Barry Smith <bsmith at mcs.anl.gov> writes:
> 
>>> On Feb 1, 2015, at 9:33 PM, Jed Brown <jed at jedbrown.org> wrote:
>>> 
>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>>  We could possibly "cheat" with AMPI to essentially have
>>>>  PetscInitialize()/Finalize() run through most their code only on  
>>>> thread 0 (assuming we have a way of determining thread 0)
>>> 
>>> Just guard it with a lock.
>> 
>>  Not sure what you mean here. We want that code to be only run  
>> through once, we don't want or need it to be run by each thread. It  
>> makes no sense for each thread to call TSInitializePackage() for  
>> example.
> 
> Yes, as long as the threads see TSPackageInitialized as true, it's 
> safe to call.  So the only problem is that we have a race condition.  
> One way to do this is to make TSPackageInitialized an int.  The code 
> looks something like this (depending on the primitives)
> 
>  if (AtomicCompareAndSwap(&TSPackageInitialized,0,1)) {
>    do the initialization
>    TSPackageInitialized = 2;
>    MemoryFenceWrite();
>  } else {
>    while (AccessOnce(TSPackageInitialized) != 2) CPURelax();  }

   Way more complicated then needed :-)

> 
>>> What about debugging and profiling?
>> 
>>   This is the same issue for "thread safety"* as well as AMPI. I
>>   don't think AMPI introduces any particular additional hitches.
>> 
>>  Barry
>> 
>> * in the sense that it is currently implemented meaning each thread 
>> works on each own objects so doesn't need to lock "MatSetValues"
>> etc. This other "thread safety" has its own can of worms.
> 
> If AMPI creates threads dynamically,

  How could it possibly create threads dynamically and still be running in
the MPI 1 model of a consistent number of MPI "processes" at all times? 

> we don't have the luxury of having
> hooks that can run when threads are spawned or finish.  How do we 
> ensure that profiling information has been propagated into the parent 
> structure?