[petsc-dev] Expectations for lock usage in PETSc

[Barry Smith]

  Paul,

    See also https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/issues/1809__;!!G_uCfscf7eWS!fOmcgx78I1C9JX-dXyYn83sMwfJapFPQWZcgvchHaGGnDdclfA5m53D_cPBeOlxxb48IknculVkmT5Uk-IxpIw$ 

    We wanted to have systematic use of read and write locks of Vec memory in PETSc to prevent programmer error of reading from or writing to Vec memory that someone else (not another thread, just another chunk of code) was supposed to have control over; so Rust-like.

    We found that just pushing locks on VecGetArrayXXX() and popping them on VecRestoreArrayXXX() opened a can of worms, and we decided it wasn't doable. This was unfortunate since a big advantage of
having the locks in VecGetYYY() would be that, since we use the get/restore routines throughout PETSc, we would immediately have a lot of new Rust-like memory safety without needing to add extra lines of code to our code base manually.

     We instead adopted the VecLockReadPush/Pop(), VecLockWriteSet(),  and VecSetErrorIfLocked(). To be useful, this requires manually adding these function calls at appropriate places throughout the code base. A bunch of VecLockReadPush/Pop() were added (frankly, because as Jed notes, they can be nested, so it is easy to know where one can safely put them). However, only one VecLockWriteSet() was ever added to the source code, as it is delicate exactly where it can be placed. 

  It would be very cool if your code analysis tools could help us know where we are missing potential VecLockReadPush/Pop() and VecLockWriteSet() calls—allowing us to improve our simple PETSc/C Rust-like memory ownership model.


 Barry

Here is an example of how VecLockReadPush/Pop() helps us find user errors. SNESComputeFunction() calls a user-provided function (callback) that they wish to find a zero of. It is obvious to us (but not to all users) that they should not change the values in the input vector, and some people did, resulting in buggy, confusing results. 

SNESComputeFunction() is a bit messy but here are the guts

    PetscCall(VecLockReadPush(x));
    /* ensure domainerror is false prior to computefunction evaluation (may not have been reset) */
    snes->domainerror = PETSC_FALSE;
    {
      void           *ctx;
      SNESFunctionFn *computefunction;
      PetscCall(DMSNESGetFunction(dm, &computefunction, &ctx));
      PetscCallBack("SNES callback function", (*computefunction)(snes, x, y, ctx));
    }
    PetscCall(VecLockReadPop(x));

Now, if the user calls VecGetArray() or VecGetArrayWrite() on the input x or tries to change its values in other ways, the code immediately stops with a very helpful error message.











> On Sep 8, 2025, at 7:18 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Read locks can nest, but writable locks cannot.
> 
> The code here could have called VecLockWriteSet(y, PETSC_TRUE) and then released it. Arguably that would be more correct, though it does require a matching line of code and the writable access does not outlive the function. It also would the x == y logic to be reworked because the simpler functions can't be called when a write lock is held.
> 
> IMO, it would be great if the compiler could enforce that arguments are not aliased, but C doesn't provide a way to do that and forbidding it would have ripple effects.
> 
> Satish Balay <balay.anl at fastmail.org> writes:
> 
>> I see the initial change was at:
>> 
>> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/commit/d9ca1df42eda25bc875149d6d493aad6e204c9ff__;!!G_uCfscf7eWS!axSTcfB_iW7kKJe0hb14ApkNvXzHZBDqzONfz8eP0TVZNPqaBKoS2fxZhoUWUBKQk8RQpEVrly41e_Pt-04eizfoQw$ 
>> 
>> And then refined at:
>> 
>> https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/commit/8860a1345bb13588f3290163e72ce904901dbfb9__;!!G_uCfscf7eWS!axSTcfB_iW7kKJe0hb14ApkNvXzHZBDqzONfz8eP0TVZNPqaBKoS2fxZhoUWUBKQk8RQpEVrly41e_Pt-04ysS1gxw$ 
>> 
>> Satish
>> 
>> On Mon, 8 Sep 2025, Hovland, Paul via petsc-dev wrote:
>> 
>>> Dear PETSc developers:
>>> 
>>> We have a question about the (expected) use of read and write locks in the PETSc implementation of vectors. If one looks at the implementation of VecAXPY in rvector.c (or, really, the implementation of VecAXPYAsync_Private, which is called by VexAXPY), it first calls
>>>       PetscCall(VecSetErrorIfLocked(y, 1));
>>> then calls
>>>       PetscCall(VecLockReadPush(x));
>>> then dispatches the AXPY method, followed by
>>>       PetscCall(VecLockReadPop(x));
>>> 
>>> We’re curious why it just checks whether y is locked but goes through the whole locking and unlocking process for x. It seems like one should either also lock y or one could simply check whether there is a write lock on x, without having to go through the whole push/pop sequence.
>>> 
>>> Is it written this way because the easiest way to check whether x has a write lock on it (and throw an error if it does)  is to call VecLockReadPush? Is this pattern a relic of when PETSc implemented locks differently? Is there some other reason for the “inconsistent” (to our minds) handling of x and y?
>>> 
>>> Thanks,
>>> 
>>> Paul & Steve
>>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20250908/8954e74d/attachment.html>