[petsc-users] Mat created by DMStag cannot access ghost points

[Patrick Sanan]

Sorry about the long delay on this.
https://gitlab.com/petsc/petsc/-/merge_requests/5329




Am Do., 2. Juni 2022 um 15:01 Uhr schrieb Matthew Knepley <knepley at gmail.com
>:

> On Thu, Jun 2, 2022 at 8:59 AM Patrick Sanan <patrick.sanan at gmail.com>
> wrote:
>
>> Thanks, Barry and Changqing! That seems reasonable to me, so I'll make an
>> MR with that change.
>>
>
> Hi Patrick,
>
> In the MR, could you add that option to all places we internally use
> Preallocator? I think we mean it for those.
>
>   Thanks,
>
>      Matt
>
>
>> Am Mi., 1. Juni 2022 um 20:06 Uhr schrieb Barry Smith <bsmith at petsc.dev>:
>>
>>>
>>>   This appears to be a bug in the DMStag/Mat preallocator code. If you
>>> add after the DMCreateMatrix() line in your code
>>>
>>> PetscCall(MatSetOption(A, MAT_NO_OFF_PROC_ENTRIES, PETSC_FALSE));
>>>
>>> Your code will run correctly.
>>>
>>>   Patrick and Matt,
>>>
>>>   MatPreallocatorPreallocate_Preallocator() has
>>>
>>> PetscCall(MatSetOption(A, MAT_NO_OFF_PROC_ENTRIES, p->nooffproc));
>>>
>>> to make the assembly of the stag matrix from the preallocator matrix a
>>> little faster,
>>>
>>> but then it never "undoes" this call. Hence the matrix is left in the
>>> state where it will error if someone sets values from a different rank
>>> (which they certainly can using DMStagMatSetValuesStencil().
>>>
>>>  I think you need to clear the NO_OFF_PROC at the end
>>> of MatPreallocatorPreallocate_Preallocator() because just because the
>>> preallocation process never needed communication does not mean that when
>>> someone puts real values in the matrix they will never use communication;
>>> they can put in values any dang way they please.
>>>
>>> I don't know why this bug has not come up before.
>>>
>>>   Barry
>>>
>>>
>>> On May 31, 2022, at 11:08 PM, Ye Changqing <Ye_Changqing at outlook.com>
>>> wrote:
>>>
>>> Dear all,
>>>
>>> [BugReport.c] is a sample code, [BugReportParallel.output] is the output
>>> when execute BugReport with mpiexec, [BugReportSerial.output] is the output
>>> in serial execution.
>>>
>>> Best,
>>> Changqing
>>>
>>> ------------------------------
>>> *发件人:* Dave May <dave.mayhem23 at gmail.com>
>>> *发送时间:* 2022年5月31日 22:55
>>> *收件人:* Ye Changqing <Ye_Changqing at outlook.com>
>>> *抄送:* petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
>>> *主题:* Re: [petsc-users] Mat created by DMStag cannot access ghost points
>>>
>>>
>>>
>>> On Tue 31. May 2022 at 16:28, Ye Changqing <Ye_Changqing at outlook.com>
>>> wrote:
>>>
>>> Dear developers of PETSc,
>>>
>>> I encountered a problem when using the DMStag module. The program could
>>> be executed perfectly in serial, while errors are thrown out in parallel
>>> (using mpiexec). Some rows in Mat cannot be accessed in local processes
>>> when looping all elements in DMStag. The DM object I used only has one DOF
>>> in each element. Hence, I could switch to the DMDA module easily, and the
>>> program now is back to normal.
>>>
>>> Some snippets are below.
>>>
>>> Initialise a DMStag object:
>>> PetscCall(DMStagCreate2d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE,
>>> DM_BOUNDARY_NONE, M, N, PETSC_DECIDE, PETSC_DECIDE, 0, 0, 1,
>>> DMSTAG_STENCIL_BOX, 1, NULL, NULL, &(s_ctx->dm_P)));
>>> Created a Mat:
>>> PetscCall(DMCreateMatrix(s_ctx->dm_P, A));
>>> Loop:
>>> PetscCall(DMStagGetCorners(s_ctx->dm_V, &startx, &starty, &startz, &nx,
>>> &ny, &nz, &extrax, &extray, &extraz));
>>> for (ey = starty; ey < starty + ny; ++ey)
>>> for (ex = startx; ex < startx + nx; ++ex)
>>> {
>>> ...
>>> PetscCall(DMStagMatSetValuesStencil(s_ctx->dm_P, *A, 2, &row[0], 2,
>>> &col[0], &val_A[0][0], ADD_VALUES));  // The traceback shows the problem is
>>> in here.
>>> }
>>>
>>>
>>> In addition to the code or MWE, please forward us the complete stack
>>> trace / error thrown to stdout.
>>>
>>> Thanks,
>>> Dave
>>>
>>>
>>>
>>> Best,
>>> Changqing
>>>
>>> <BugReport.c><BugReportParallel.output><BugReportSerial.output>
>>>
>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220610/706ce051/attachment-0001.html>