<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> This appears to be a bug in the DMStag/Mat preallocator code. If you add after the DMCreateMatrix() line in your code <div class=""><br class=""></div><div class="">PetscCall(MatSetOption(A, MAT_NO_OFF_PROC_ENTRIES, PETSC_FALSE));</div><div class=""><br class=""></div><div class="">Your code will run correctly.</div><div class=""><br class=""></div><div class=""> Patrick and Matt,</div><div class=""><br class=""></div><div class=""> MatPreallocatorPreallocate_Preallocator() has </div><div class=""><br class=""></div><div class="">PetscCall(MatSetOption(A, MAT_NO_OFF_PROC_ENTRIES, p->nooffproc));</div><div class=""><br class=""></div><div class="">to make the assembly of the stag matrix from the preallocator matrix a little faster,</div><div class=""><br class=""></div><div class="">but then it never "undoes" this call. Hence the matrix is left in the state where it will error if someone sets values from a different rank (which they certainly can using DMStagMatSetValuesStencil(). </div><div class=""><br class=""></div><div class=""> I think you need to clear the NO_OFF_PROC at the end of MatPreallocatorPreallocate_Preallocator() because just because the preallocation process never needed communication does not mean that when someone puts real values in the matrix they will never use communication; they can put in values any dang way they please.</div><div class=""><br class=""></div><div class="">I don't know why this bug has not come up before.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On May 31, 2022, at 11:08 PM, Ye Changqing <<a href="mailto:Ye_Changqing@outlook.com" class="">Ye_Changqing@outlook.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Dear all,</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">[BugReport.c] is a sample code, [BugReportParallel.output] is the output when execute BugReport with mpiexec, [BugReportSerial.output] is the output in serial execution.</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br class=""></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Best,</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Changqing</div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div><div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1075.046875px;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">发件人:</b><span class="Apple-converted-space"> </span>Dave May <<a href="mailto:dave.mayhem23@gmail.com" class="">dave.mayhem23@gmail.com</a>><br class=""><b class="">发送时间:</b><span class="Apple-converted-space"> </span>2022年5月31日 22:55<br class=""><b class="">收件人:</b><span class="Apple-converted-space"> </span>Ye Changqing <<a href="mailto:Ye_Changqing@outlook.com" class="">Ye_Changqing@outlook.com</a>><br class=""><b class="">抄送:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>><br class=""><b class="">主题:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Mat created by DMStag cannot access ghost points</font><div class=""> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""><br class=""></div><div class=""><br class=""><div class="x_gmail_quote"><div dir="ltr" class="">On Tue 31. May 2022 at 16:28, Ye Changqing <<a href="mailto:Ye_Changqing@outlook.com" data-auth="NotApplicable" class="">Ye_Changqing@outlook.com</a>> wrote:<br class=""></div><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">Dear developers of PETSc,</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">I encountered a problem when using the DMStag module. The program could be executed perfectly in serial, while errors are thrown out in parallel (using mpiexec). Some rows in Mat cannot be accessed in local processes when looping all elements in DMStag. The DM object I used only has one DOF in each element. Hence, I could switch to the DMDA module easily, and the program now is back to normal.</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">Some snippets are below.</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">Initialise a DMStag object:</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">PetscCall(DMStagCreate2d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE, DM_BOUNDARY_NONE, M, N, PETSC_DECIDE, PETSC_DECIDE, 0, 0, 1, DMSTAG_STENCIL_BOX, 1, NULL, NULL, &(s_ctx->dm_P)));<br class=""></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">Created a Mat:</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">PetscCall(DMCreateMatrix(s_ctx->dm_P, A));<br class=""></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">Loop:</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">PetscCall(DMStagGetCorners(s_ctx->dm_V, &startx, &starty, &startz, &nx, &ny, &nz, &extrax, &extray, &extraz));</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">for (ey = starty; ey < starty + ny; ++ey)</span></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""><div class="">for (ex = startx; ex < startx + nx; ++ex)</div>{<br class=""></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">...</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">PetscCall(DMStagMatSetValuesStencil(s_ctx->dm_P, *A, 2, &row[0], 2, &col[0], &val_A[0][0], ADD_VALUES)); // The traceback shows the problem is in here.<br class=""></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">}</div></div></blockquote><div dir="auto" class=""><br class=""></div><div dir="auto" class="">In addition to the code or MWE, please forward us the complete stack trace / error thrown to stdout.</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Thanks,</div><div dir="auto" class="">Dave</div><div dir="auto" class=""><br class=""></div><div dir="auto" class=""><br class=""></div><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">Best,</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class="">Changqing</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div></div></blockquote></div></div></div><span id="cid:A74F9B90-FA44-4E0C-AD24-8C4CEEA1607C"><BugReport.c></span><span id="cid:A5F0C74A-B57C-4784-9AA8-E0D07E2190C0"><BugReportParallel.output></span><span id="cid:6E964459-DB49-4014-85A5-09B37E4F6B71"><BugReportSerial.output></span></div></blockquote></div><br class=""></div></body></html>