<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Sorry, I had a mistake in my thinking, PCREDISTRIBUTE supports completely empty rows but MatZero* does not.<div><br></div><div> When you put values into the matrix you will need to insert a 0.0 on the diagonal of each "inactive" row; all of this will be eliminated during the linear solve process. It would be a major project to change the MatZero* functions to handle empty rows. </div><div><br></div><div> Barry</div><div><br></div><div><div><br><blockquote type="cite"><div>On Feb 4, 2023, at 12:06 PM, Karthikeyan Chockalingam - STFC UKRI <karthikeyan.chockalingam@stfc.ac.uk> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Thank you very much for offering to debug.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I built PETSc along with AMReX, so I tried to extract the PETSc code alone which would reproduce the same error on the smallest sized problem possible.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I have attached three files:<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">petsc_amrex_error_redistribute.txt – The error message from amrex/petsc interface, but THE linear system solves and converges to a solution.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">problem.c – A simple stand-alone petsc code, which produces almost the same error message.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">petsc_ error_redistribute.txt – The error message from problem.c but strangely it does NOT solve – I am not sure why?<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Please use problem.c to debug the issue.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Kind regards,<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Karthik.<o:p></o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;">Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><br><b>Date:<span class="Apple-converted-space"> </span></b>Saturday, 4 February 2023 at 00:22<br><b>To:<span class="Apple-converted-space"> </span></b>Chockalingam, Karthikeyan (STFC,DL,HC) <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;">karthikeyan.chockalingam@stfc.ac.uk</a>><br><b>Cc:<span class="Apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] Eliminating rows and columns which are zeros<o:p></o:p></span></p></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> If you can help me reproduce the problem with a simple code I can debug the problem and fix it.<o:p></o:p></span></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Barry<o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><br><br><o:p></o:p></span></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">On Feb 3, 2023, at 6:42 PM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></span></div></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I updated the main branch to the below commit but the same problem persists.</span><o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><b><span style="font-size: 8.5pt; font-family: Menlo;">[0]PETSC ERROR: Petsc Development GIT revision: v3.18.4-529-g995ec06f92<span class="apple-converted-space"> </span>GIT Date: 2023-02-03 18:41:48 +0000</span></b><span style="font-size: 8.5pt; font-family: Menlo;"><o:p></o:p></span></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 12pt;">Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><br><b>Date:<span class="apple-converted-space"> </span></b>Friday, 3 February 2023 at 18:51<br><b>To:<span class="apple-converted-space"> </span></b>Chockalingam, Karthikeyan (STFC,DL,HC) <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;">karthikeyan.chockalingam@stfc.ac.uk</a>><br><b>Cc:<span class="apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span class="apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] Eliminating rows and columns which are zeros</span><o:p></o:p></p></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> If you switch to use the main branch of petsc <a href="https://petsc.org/release/install/download/#advanced-obtain-petsc-development-version-with-git" style="color: blue; text-decoration: underline;">https://petsc.org/release/install/download/#advanced-obtain-petsc-development-version-with-git</a> you will not have the problem below (previously we required that a row exist before we zeroed it but now we allow the row to initially have no entries and still be zeroed.</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Barry</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div><div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">On Feb 3, 2023, at 1:04 PM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:</span><o:p></o:p></div></div></div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Thank you. The entire error output was an attachment in my previous email. I am pasting here for your reference.</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[1;31m[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0;39m[0;49m[0]PETSC ERROR: Object is in wrong state</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: Matrix is missing diagonal entry in row 0 (65792)</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: Option left: name:-options_left (no value)</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: See<span class="apple-converted-space"> </span><a href="https://petsc.org/release/faq/" style="color: blue; text-decoration: underline;">https://petsc.org/release/faq/</a><span class="apple-converted-space"> </span>for trouble shooting.</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: Petsc Development GIT revision: v3.18.1-127-ga207d08eda GIT Date: 2022-10-30 11:03:25 -0500</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: /Users/karthikeyan.chockalingam/AMReX/amrFEM/build/Debug/amrFEM on a named HC20210312 by karthikeyan.chockalingam Fri Feb 3 11:10:01 2023</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: Configure options --with-debugging=0 --prefix=/Users/karthikeyan.chockalingam/AMReX/petsc --download-fblaslapack=yes --download-scalapack=yes --download-mumps=yes --with-hypre-dir=/Users/karthikeyan.chockalingam/AMReX/hypre/src/hypre</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: #1 MatZeroRowsColumns_SeqAIJ() at /Users/karthikeyan.chockalingam/AMReX/SRC_PKG/petsc/src/mat/impls/aij/seq/aij.c:2218</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: #2 MatZeroRowsColumns() at /Users/karthikeyan.chockalingam/AMReX/SRC_PKG/petsc/src/mat/interface/matrix.c:6085</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: #3 MatZeroRowsColumns_MPIAIJ() at /Users/karthikeyan.chockalingam/AMReX/SRC_PKG/petsc/src/mat/impls/aij/mpi/mpiaij.c:879</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: #4 MatZeroRowsColumns() at /Users/karthikeyan.chockalingam/AMReX/SRC_PKG/petsc/src/mat/interface/matrix.c:6085</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: #5 MatZeroRowsColumnsIS() at /Users/karthikeyan.chockalingam/AMReX/SRC_PKG/petsc/src/mat/interface/matrix.c:6124</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: #6 localAssembly() at /Users/karthikeyan.chockalingam/AMReX/amrFEM/src/FENodalPoisson.cpp:435</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Residual norms for redistribute_ solve.</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 0 KSP preconditioned resid norm 5.182603110407e+00 true resid norm 1.382027496109e+01 ||r(i)||/||b|| 1.000000000000e+00</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 1 KSP preconditioned resid norm 1.862430383976e+00 true resid norm 4.966481023937e+00 ||r(i)||/||b|| 3.593619546588e-01</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 2 KSP preconditioned resid norm 2.132803507689e-01 true resid norm 5.687476020503e-01 ||r(i)||/||b|| 4.115313216645e-02</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 3 KSP preconditioned resid norm 5.499797533437e-02 true resid norm 1.466612675583e-01 ||r(i)||/||b|| 1.061203687852e-02</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 4 KSP preconditioned resid norm 2.829814271435e-02 true resid norm 7.546171390493e-02 ||r(i)||/||b|| 5.460217985345e-03</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 5 KSP preconditioned resid norm 7.431048995318e-03 true resid norm 1.981613065418e-02 ||r(i)||/||b|| 1.433844891652e-03</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 6 KSP preconditioned resid norm 3.182040728972e-03 true resid norm 8.485441943932e-03 ||r(i)||/||b|| 6.139850305312e-04</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 7 KSP preconditioned resid norm 1.030867020459e-03 true resid norm 2.748978721225e-03 ||r(i)||/||b|| 1.989091193167e-04</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 8 KSP preconditioned resid norm 4.469429300003e-04 true resid norm 1.191847813335e-03 ||r(i)||/||b|| 8.623908111021e-05</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 9 KSP preconditioned resid norm 1.237303313796e-04 true resid norm 3.299475503456e-04 ||r(i)||/||b|| 2.387416685085e-05</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 10 KSP preconditioned resid norm 5.822094326756e-05 true resid norm 1.552558487134e-04 ||r(i)||/||b|| 1.123391894522e-05</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> 11 KSP preconditioned resid norm 1.735776150969e-05 true resid norm 4.628736402585e-05 ||r(i)||/||b|| 3.349236115503e-06</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Linear redistribute_ solve converged due to CONVERGED_RTOL iterations 11</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">KSP Object: (redistribute_) 1 MPI process</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> type: cg</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> maximum iterations=10000, initial guess is zero</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> left preconditioning</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> using PRECONDITIONED norm type for convergence test</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">PC Object: (redistribute_) 1 MPI process</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> type: jacobi</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> type DIAGONAL</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> linear system matrix = precond matrix:</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Mat Object: 1 MPI process</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> type: mpiaij</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> rows=48896, cols=48896</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> total: nonzeros=307976, allocated nonzeros=307976</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> total number of mallocs used during MatSetValues calls=0</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> not using I-node (on process 0) routines</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">End of program</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> solve time 0.564714744 seconds</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Starting max value is: 0</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Min value of level 0 is: 0</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Interpolated min value is: 741.978761</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Unused ParmParse Variables:</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> [TOP]::model.type(nvals = 1) :: [3]</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> [TOP]::ref_ratio(nvals = 1) :: [2]</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">AMReX (22.10-20-g3082028e4287) finalized</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">#PETSc Option Table entries:</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-ksp_type preonly</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-options_left</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-pc_type redistribute</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_ksp_converged_reason</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_ksp_monitor_true_residual</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_ksp_type cg</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_ksp_view</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_pc_type jacobi</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">#End of PETSc Option Table entries</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">There are no unused options.</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Program ended with exit code: 0</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Best,</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Karthik.</span><o:p></o:p></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 12pt;">Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><br><b>Date:<span class="apple-converted-space"> </span></b>Friday, 3 February 2023 at 17:41<br><b>To:<span class="apple-converted-space"> </span></b>Chockalingam, Karthikeyan (STFC,DL,HC) <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;">karthikeyan.chockalingam@stfc.ac.uk</a>><br><b>Cc:<span class="apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span class="apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] Eliminating rows and columns which are zeros</span><o:p></o:p></p></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> We need all the error output for the errors you got below to understand why the errors are happening.</span><o:p></o:p></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><br><br><br><br></span><o:p></o:p></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">On Feb 3, 2023, at 11:41 AM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:</span><o:p></o:p></div></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Hello Barry,</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I would like to better understand<span class="apple-converted-space"> </span>pc_type redistribute usage.<span class="apple-converted-space"> </span></span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I am plan to use pc_type<span class="apple-converted-space"> </span><u>redistribute</u><span class="apple-converted-space"> </span>in the context of adaptive mesh refinement on a structured grid in 2D. My base mesh (level 0) is indexed from 0 to N-1 elements and refined mesh (level 1) is indexed from 0 to 4(N-1) elements. When I construct system matrix A on (level 1); I probably only use 20% of 4(N-1) elements, however the indexes are scattered in the range of 0 to 4(N-1). That leaves 80% of the rows and columns of the system matrix A on (level 1) to be zero. From your earlier response, I believe this would be a use case for petsc_type redistribute.</span><o:p></o:p></div></div></div></div></div></blockquote><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Indeed the linear solve will be more efficient if you use the redistribute solver. </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> But I don't understand your plan. With adaptive refinement I would just create the two matrices, one for the initial grid on which you solve the system, this will be a smaller matrix and then create a new larger matrix for the refined grid (and discard the previous matrix).<br><br><br><br></span><o:p></o:p></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Question (1)</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">If N is really large, I would have to allocate memory of size 4(N-1) for the system matrix A on (level 1). How does pc_type redistribute<span class="apple-converted-space"><u> </u></span>help? Because, I did end up allocating memory for a large system, where most of the rows and columns are zeros. Is most of the allotted memory not wasted?<span class="apple-converted-space"><u> </u></span><u>Is this the correct usage?</u></span><o:p></o:p></div></div></div></div></div></blockquote><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> See above<br><br><br><br></span><o:p></o:p></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Question (2)</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I tried using pc_type redistribute for a two level system.</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I have<span class="apple-converted-space"> </span><b><u>attached</u></b><span class="apple-converted-space"> </span>the output only for (level 1)</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">The solution converges to right solution but still petsc outputs some error messages.</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[0]PETSC ERROR: Option left: name:-options_left (no value)</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">But the there were no unused options</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">#PETSc Option Table entries:</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-ksp_type preonly</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-options_left</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-pc_type redistribute</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_ksp_converged_reason</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_ksp_monitor_true_residual</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_ksp_type cg</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_ksp_view</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-redistribute_pc_type jacobi</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">#End of PETSc Option Table entries</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 11pt;">There are no unused options.</span></b><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Program ended with exit code: 0</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div></div></blockquote><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I cannot explain this<br><br><br><br></span><o:p></o:p></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><br>Question (2)</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 11pt;">[0;39m[0;49m[0]PETSC ERROR: Object is in wrong state</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 11pt;">[0]PETSC ERROR: Matrix is missing diagonal entry in row 0 (65792)</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">What does this error message imply? Given I only use 20% of 4(N-1) indexes, I can imagine most of the diagonal entrees are zero.<span class="apple-converted-space"> </span><u>Is my understanding correct?</u></span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Question (3)<br><br><br><br><br></span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 8.5pt; font-family: Menlo;">[0]PETSC ERROR: #5 MatZeroRowsColumnsIS() at /Users/karthikeyan.chockalingam/AMReX/SRC_PKG/petsc/src/mat/interface/matrix.c:6124</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 8.5pt; font-family: Menlo;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I am using MatZeroRowsColumnsIS to set the homogenous Dirichelet boundary. I don’t follow why I get this error message as the linear system converges to the right solution.</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Thank you for your help.</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Kind regards,</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Karthik.</span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 8.5pt; font-family: Menlo;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 12pt;">Barry Smith <<a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;">bsmith@petsc.dev</a>><br><b>Date:<span class="apple-converted-space"> </span></b>Tuesday, 10 January 2023 at 18:50<br><b>To:<span class="apple-converted-space"> </span></b>Chockalingam, Karthikeyan (STFC,DL,HC) <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;">karthikeyan.chockalingam@stfc.ac.uk</a>><br><b>Cc:<span class="apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a><span class="apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;">petsc-users@mcs.anl.gov</a>><br><b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] Eliminating rows and columns which are zeros</span><o:p></o:p></p></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Yes, after the solve the x will contain correct values for ALL the locations including the (zeroed out rows). You use case is exactly what redistribute it for.</span><o:p></o:p></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> Barry</span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><br><br><br><br><br></span><o:p></o:p></div></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">On Jan 10, 2023, at 11:25 AM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:</span><o:p></o:p></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Thank you Barry. This is great!</span><o:p></o:p></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I plan to solve using ‘-pc_type redistribute’ after applying the Dirichlet bc using</span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span class="s1"><span style="font-size: 9pt; font-family: Menlo; color: rgb(108, 54, 169);">MatZeroRowsColumnsIS</span></span><span style="font-size: 9pt; font-family: Menlo;">(</span><span class="s2"><span style="font-size: 9pt; font-family: Menlo; color: rgb(50, 109, 116);">A</span></span><span style="font-size: 9pt; font-family: Menlo;">,<span class="apple-converted-space"> </span></span><span class="s2"><span style="font-size: 9pt; font-family: Menlo; color: rgb(50, 109, 116);">isout</span></span><span style="font-size: 9pt; font-family: Menlo;">,<span class="apple-converted-space"> </span></span><span class="s3"><span style="font-size: 9pt; font-family: Menlo; color: rgb(28, 0, 207);">1</span></span><span style="font-size: 9pt; font-family: Menlo;">,<span class="apple-converted-space"> </span></span><span class="s2"><span style="font-size: 9pt; font-family: Menlo; color: rgb(50, 109, 116);">x</span></span><span style="font-size: 9pt; font-family: Menlo;">,<span class="apple-converted-space"> </span></span><span class="s2"><span style="font-size: 9pt; font-family: Menlo; color: rgb(50, 109, 116);">b)</span></span><span style="font-size: 9pt; font-family: Menlo;">;<span class="apple-converted-space"> </span></span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 9pt; font-family: Menlo;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">While I retrieve the solution data from x (after the solve) – can I index them using the original ordering (if I may say that)?</span><o:p></o:p></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Kind regards,</span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Karthik.</span><o:p></o:p></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;"><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 12pt;">Barry Smith <</span><a href="mailto:bsmith@petsc.dev" style="color: blue; text-decoration: underline;"><span style="font-size: 12pt;">bsmith@petsc.dev</span></a><span style="font-size: 12pt;">><br><b>Date:<span class="apple-converted-space"> </span></b>Tuesday, 10 January 2023 at 16:04<br><b>To:<span class="apple-converted-space"> </span></b>Chockalingam, Karthikeyan (STFC,DL,HC) <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" style="color: blue; text-decoration: underline;"><span style="font-size: 12pt;">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size: 12pt;">><br><b>Cc:<span class="apple-converted-space"> </span></b></span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;"><span style="font-size: 12pt;">petsc-users@mcs.anl.gov</span></a><span class="apple-converted-space"><span style="font-size: 12pt;"> </span></span><span style="font-size: 12pt;"><</span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;"><span style="font-size: 12pt;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 12pt;">><br><b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] Eliminating rows and columns which are zeros</span><o:p></o:p></p></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><a href="https://petsc.org/release/docs/manualpages/PC/PCREDISTRIBUTE/#pcredistribute" style="color: blue; text-decoration: underline;"><span style="font-size: 11pt;">https://petsc.org/release/docs/manualpages/PC/PCREDISTRIBUTE/#pcredistribute</span></a><span style="font-size: 11pt;"> -pc_type redistribute</span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">It does everything for you. Note that if the right hand side for any of the "zero" rows is nonzero then the system is inconsistent and the system does not have a solution.</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Barry</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><br><br><br><br></span><o:p></o:p></p></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;"><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">On Jan 10, 2023, at 10:30 AM, Karthikeyan Chockalingam - STFC UKRI via petsc-users <</span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;"><span style="font-size: 11pt;">petsc-users@mcs.anl.gov</span></a><span style="font-size: 11pt;">> wrote:</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div><div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Hello,</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I am assembling a MATIJ of size N, where a very large number of rows (and corresponding columns), are zeros. I would like to potentially eliminate them before the solve.</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">For instance say N=7</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 9pt; font-family: Menlo; color: rgb(93, 108, 121);">0 0<span class="apple-converted-space"> </span>0<span class="apple-converted-space"> </span>0 0 0 0</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 9pt; font-family: Menlo; color: rgb(93, 108, 121);">0 1 -1<span class="apple-converted-space"> </span>0 0 0 0</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 9pt; font-family: Menlo; color: rgb(93, 108, 121);">0 -1 2<span class="apple-converted-space"> </span>0 0 0 -1</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 9pt; font-family: Menlo; color: rgb(93, 108, 121);">0 0<span class="apple-converted-space"> </span>0<span class="apple-converted-space"> </span>0 0 0 0</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 9pt; font-family: Menlo; color: rgb(93, 108, 121);">0 0<span class="apple-converted-space"> </span>0<span class="apple-converted-space"> </span>0 0 0 0</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 9pt; font-family: Menlo; color: rgb(93, 108, 121);">0 0<span class="apple-converted-space"> </span>0<span class="apple-converted-space"> </span>0 0 0 0</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif; background-color: white;"><span style="font-size: 9pt; font-family: Menlo; color: rgb(93, 108, 121);">0 0<span class="apple-converted-space"> </span>-1 0 0 0 1</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I would like to reduce it to a 3x3</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">1 -1 0</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">-1 2 -1</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">0 -1 1</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I do know the size N.</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Q1) How do I do it?</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Q2) Is it better to eliminate them as it would save a lot of memory?</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Q3) At the moment, I don’t know which rows (and columns) have the zero entries but with some effort I probably can find them. Should I know which rows (and columns) I am eliminating?</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Thank you.</span><o:p></o:p></div></div></div></div></div></div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Karthik.</span><o:p></o:p></div></div></div></div></div></div><p style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0px;"><span style="font-size: 6pt; font-family: Helvetica;">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments<span class="apple-converted-space"> </span><span class="Apple-converted-space"> </span>and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. </span><span style="font-size: 13.5pt; font-family: Helvetica;"><o:p></o:p></span></p></div></blockquote></div></div></div></blockquote></div><div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"> </span><o:p></o:p></div></div></div></div></div><div><div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><petsc_redistribute.txt></span><o:p></o:p></div></div></div></div></blockquote></div></div></blockquote></div></div></blockquote></div><div style="margin: 0cm; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div></div></div><span id="cid:03CEEB10-96F4-4C1F-BBE6-0D69558B4CD1"><petsc_error_redistribute.txt></span><span id="cid:EC581AB2-6D33-4D91-9D8C-F5E9E64A5698"><petsc_amrex_error_redistribute.txt></span><span id="cid:91122B9D-0CC0-4DBF-92AF-C2125134D2B2"><problem.c></span></div></blockquote></div><br></div></body></html>