<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 3 May 2022, at 7:33 PM, Randall Mackie <<a href="mailto:rlmackie862@gmail.com" class="">rlmackie862@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Pierre,<div class=""><br class=""></div><div class="">I do not discount that I might not be doing something right.</div><div class=""><br class=""></div><div class="">However, this code has not changed and worked fine for versions up to and including 3.16.</div></div></div></blockquote><div><br class=""></div><div>What are you implying?</div><div>It is not because a code works that it is not bugged, that would be too simple.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Can you please explain what changed in 3.17 that is causing this?</div></div></div></blockquote><div><br class=""></div><div>First, let me tell you that now, MatSetBlockSize() and ISSetBlockSize() work in rather different ways. The latter is not permissive and performs lots of checks, while the former is a little bit looser and does not check the pattern of the matrix.</div><div>What you are seeing is a side effect of two commits.</div><div>1) <a href="https://gitlab.com/petsc/petsc/-/commit/30f30de87881d4fa2944681a7d91a8e1d429a77c" class="">https://gitlab.com/petsc/petsc/-/commit/30f30de87881d4fa2944681a7d91a8e1d429a77c</a> check for valid block size of IS</div><div>2) <a href="https://gitlab.com/petsc/petsc/-/commit/9dd3beda480d3fbab2ab495f81e1e366d04021c6" class="">https://gitlab.com/petsc/petsc/-/commit/9dd3beda480d3fbab2ab495f81e1e366d04021c6</a> propagation of Mat bs to overlapping IS</div><div>PCASM uses MatIncreaseOverlap(). You use this PC with a Mat that you tell PETSc has a bs of 3.</div><div>We don’t check this for you, but then when setting the bs of the IS, we do check this, and this errors out.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Are you saying that now you have to explicitly set each 3x3 dense block, even if they are not used and that was not the case before?</div></div></div></blockquote><div><br class=""></div><div>That was always the case before, you may have misinterpreted the meaning of a Mat block size?</div><div>Again, with version 3.XY (XY < 17), try MatConvert(A,MATBAIJ,MAT_INITIAL_MATRIX,C,ierr) and you will still get an error.</div><div>You have two options: either do not set a block size or set a block size but make sure that the pattern of the Mat is indeed only made of blocks of size bs x bs.</div><div><br class=""></div><div>Thanks,</div><div>Pierre</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Randy</div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 3, 2022, at 10:09 AM, Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 3 May 2022, at 6:54 PM, Randall Mackie <<a href="mailto:rlmackie862@gmail.com" class="">rlmackie862@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Pierre,<div class=""><br class=""></div><div class="">Here is how I create the matrix, and how I’ve done it for many years:</div><div class=""><br class=""></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> ! Create global matrix to hold system of equations resulting from finite discretization<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> ! of the Maxwell equations. <o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> ngrow3=ginfo%nx*ginfo%ny*ginfo%nz*3<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call MatCreate(comm,A,ierr)<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call MatSetSizes(A,mloc3,mloc3,ngrow3,ngrow3,ierr)<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call MatSetBlockSize(A,i3,ierr)</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">I don’t know enough about your discretization stencil, but again, since a simple call to MatConvert(A,MATBAIJ,MAT_INITIAL_MATRIX,C,ierr) fails in your MWE, I doubt that this line is correct.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call MatSetType(A,MATAIJ,ierr)<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call MatSeqAIJSetPreallocation(A,i15,PETSC_NULL_INTEGER,ierr)<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call MatMPIAIJSetPreallocation(A,i15,PETSC_NULL_INTEGER,i7,PETSC_NULL_INTEGER,ierr)<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> call MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE,ierr)</div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">This is a staggered grid formulation (but derived many years before PETSc had a DMStag class) and so I use regular DMDA….for those locations that are not involved, I put a 1 on the diagonal.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">If you want to use a block size of dimension 3, you also need to explicitly MatSetValue (or MatSetValues or MatSetValuesBlocked) zeros for all 3x3 dense blocks.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pierre </div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Randy</div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 3, 2022, at 9:37 AM, Pierre Jolivet <<a href="mailto:pierre@joliv.et" class="">pierre@joliv.et</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thanks for the reproducer.<div class="">My guess is that your AIJ matrix has not a block size of 3.</div><div class="">A simple call such as: call MatConvert(A,MATBAIJ,MAT_INITIAL_MATRIX,C,ierr) is also failing, while it shouldn’t if your AIJ Mat is truly made of 3x3 dense blocks.</div><div class="">How did you determine the block size of your Mat?</div><div class="">Are you allocating 3x3 dense blocks everywhere or are you skipping zero coefficients in your AIJ Mat?</div><div class="">In the meantime, you can bypass the issue by not setting a block size of 3 on your Mat, or by setting different block size for the column and row distributions, see MatSetBlockSizes().</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pierre</div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 3 May 2022, at 5:39 PM, Randall Mackie <<a href="mailto:rlmackie862@gmail.com" class="">rlmackie862@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Dear PETSc team:<div class=""><br class=""></div><div class="">A part of our code that has worked for years and previous versions is now failing with the latest version 3.17.1, on the KSP solve with the following error:</div><div class=""><br class=""></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: Invalid argument<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: Block size 3 is incompatible with the indices: non consecutive indices 153055 153124<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: See <a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Frelease%2Ffaq%2F&data=05%7C01%7CRandall.Mackie%40CGG.com%7C323fb230ca4f45d45c7408da2d18c44e%7C307ea68275e14701a1146c42f9ff0d24%7C0%7C0%7C637871881639501468%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3iAxFsi8wDXPcxKRBVFotn7DpoVNc9%2Fq60r%2F57nn7Lw%3D&reserved=0" style="color: rgb(5, 99, 193);" class="">https://petsc.org/release/faq/</a> for trouble shooting.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: Petsc Release Version 3.17.1, Apr 28, 2022 <o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: ./test on a linux-gfortran-complex-debug named rmackie-VirtualBox by rmackie Tue May 3 08:12:15 2022<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex --with-debugging=1 --with-fortran=1 --download-mpich=../external/mpich-4.0.1.tar.gz<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: #1 ISSetBlockSize() at /home/rmackie/PETSc/petsc-3.17.1/src/vec/is/is/interface/index.c:1898<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: #2 MatIncreaseOverlap() at /home/rmackie/PETSc/petsc-3.17.1/src/mat/interface/matrix.c:7086<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: #3 PCSetUp_ASM() at /home/rmackie/PETSc/petsc-3.17.1/src/ksp/pc/impls/asm/asm.c:238<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: #4 PCSetUp() at /home/rmackie/PETSc/petsc-3.17.1/src/ksp/pc/interface/precon.c:990<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: #5 KSPSetUp() at /home/rmackie/PETSc/petsc-3.17.1/src/ksp/ksp/interface/itfunc.c:407<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: #6 KSPSolve_Private() at /home/rmackie/PETSc/petsc-3.17.1/src/ksp/ksp/interface/itfunc.c:843<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: #7 KSPSolve() at /home/rmackie/PETSc/petsc-3.17.1/src/ksp/ksp/interface/itfunc.c:1078</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">I have a small test program with binary matrix and right hand side that will show the problem and I can send it as a zip file, please advise what email address to use or where to send it.</div><div class=""><br class=""></div><div class="">Thanks, </div><div class=""><br class=""></div><div class="">Randy M.</div></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></body></html>