<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">It’s indeed very suspicious (to me) that we are using rmap to change a column index.<div>Switching to cmap gets your code running, but I’ll need to see if this triggers regressions.</div><div><br></div><div>Thanks for the report,</div><div>Pierre</div><div><br></div><div><div>diff --git a/src/mat/impls/aij/mpi/mpiov.c b/src/mat/impls/aij/mpi/mpiov.c</div><div>index d1037d7d817..051981ebe9a 100644</div><div>--- a/src/mat/impls/aij/mpi/mpiov.c</div><div>+++ b/src/mat/impls/aij/mpi/mpiov.c</div><div>@@ -2948,3 +2948,3 @@ PetscErrorCode MatSetSeqMats_MPIAIJ(Mat C, IS rowemb, IS dcolemb, IS ocolemb, Ma</div><div> </div><div>- PetscCall(PetscLayoutGetRange(C->rmap, &rstart, &rend));</div><div>+ PetscCall(PetscLayoutGetRange(C->cmap, &rstart, &rend));</div><div> shift = rend - rstart;</div><div><br></div><div>$ cat proc_0_output.txt</div><div>rstart 0 rend 4</div><div>Mat Object: 3 MPI processes</div><div> type: mpiaij</div><div> row 0: (0, 101.) (3, 104.) (6, 107.) (9, 110.) </div><div> row 1: (2, 203.) (5, 206.) (8, 209.) (11, 212.) </div><div> row 2: (1, 302.) (4, 305.) (7, 308.) (10, 311.) </div><div> row 3: (0, 401.) (3, 404.) (6, 407.) (9, 410.) </div><div> row 4: (2, 503.) (5, 506.) (8, 509.) (11, 512.) </div><div> row 5: (1, 602.) (4, 605.) (7, 608.) (10, 611.) </div><div> row 6: (0, 701.) (3, 704.) (6, 707.) (9, 710.) </div><div> row 7: (2, 803.) (5, 806.) (8, 809.) (11, 812.) </div><div> row 8: (1, 902.) (4, 905.) (7, 908.) (10, 911.) </div><div> row 9: (0, 1001.) (3, 1004.) (6, 1007.) (9, 1010.) </div><div> row 10: (2, 1103.) (5, 1106.) (8, 1109.) (11, 1112.) </div><div> row 11: (1, 1202.) (4, 1205.) (7, 1208.) (10, 1211.) </div><div>idxr proc</div><div>IS Object: 2 MPI processes</div><div> type: general</div><div>[0] Number of indices in set 4</div><div>[0] 0 0</div><div>[0] 1 1</div><div>[0] 2 2</div><div>[0] 3 3</div><div>[1] Number of indices in set 4</div><div>[1] 0 4</div><div>[1] 1 5</div><div>[1] 2 6</div><div>[1] 3 7</div><div>idxc proc</div><div>IS Object: 2 MPI processes</div><div> type: general</div><div>[0] Number of indices in set 2</div><div>[0] 0 0</div><div>[0] 1 1</div><div>[1] Number of indices in set 2</div><div>[1] 0 6</div><div>[1] 1 7</div><div>Mat Object: 2 MPI processes</div><div> type: mpiaij</div><div> row 0: (0, 101.) (2, 107.) </div><div> row 1: </div><div> row 2: (1, 302.) (3, 308.) </div><div> row 3: (0, 401.) (2, 407.) </div><div> row 4: </div><div> row 5: (1, 602.) (3, 608.) </div><div> row 6: (0, 701.) (2, 707.) </div><div> row 7: </div><div>rstart 0 rend 4</div><div>local row 0: ( 0 , 1.010000e+02) ( 2 , 1.070000e+02)</div><div>local row 1:</div><div>local row 2: ( 1 , 3.020000e+02) ( 3 , 3.080000e+02)</div><div>local row 3: ( 0 , 4.010000e+02) ( 2 , 4.070000e+02)</div><div><br><blockquote type="cite"><div>On 26 Aug 2025, at 3:18 PM, Pierre Jolivet <pierre@joliv.et> wrote:</div><br class="Apple-interchange-newline"><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><br><div><blockquote type="cite"><div>On 26 Aug 2025, at 12:50 PM, Alexis SALZMAN <alexis.salzman@ec-nantes.fr> wrote:</div><br class="Apple-interchange-newline"><div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div><p>Mark, you were right and I was wrong about the dense matrix.
Adding explicit zeros to the distributed matrix used to extract
the sub-matrices (making it dense) in my test does not change the
behaviour: there is still an error.</p><p>I am finding it increasingly difficult to understand the logic of
the row and column 'IS' creation. I ran many tests to achieve the
desired result: a rectangular sub-matrix (so a rectangular or
square sub-matrix appears to be possible). However, many others
resulted in the same kind of error.</p></div></div></blockquote><div>This may be a PETSc bug in MatSetSeqMats_MPIAIJ().</div><div><div>-> 2965<span class="Apple-tab-span" style="white-space:pre"> </span> PetscCall(MatSetValues(aij->B, 1, &row, 1, &col, &v, INSERT_VALUES));</div><div>col has a value of 4, which doesn’t make sense since the output Mat has 4 columns (thus, has the error message suggests, the value should be lower than or equal to 3).</div><div><br></div><div>Thanks,</div><div>Pierre</div></div><blockquote type="cite"><div><div><p>From what I observed, the test only works if the column selection
contribution (size_c in the test) has a specific value related to
the row selection contribution (size_r in the test) for proc 0
(rank for both communicator and sub-communicator):</p>
<ul>
<li>if size_r==2 then if size_c<=2 it works.</li>
<li>if size_r>=3 and size_r<=5 then size_c==size_r is the
only working case.</li>
</ul><p>This occurs "regardless" of what is requested in proc 1 and in
selr/selc (It can't be a dummy setting, though). In any case, it's
certainly not an exhaustive analysis.<br>
</p><p>Many thanks to anyone who can explain to me the logic behind the
construction of row and column 'IS'.</p><p>Regards</p><p>A.S.<br>
</p><p><br>
</p>
<div class="moz-cite-prefix">Le 25/08/2025 à 20:00, Alexis SALZMAN a
écrit :<br>
</div>
<blockquote type="cite" cite="mid:113ece7d-25cf-4b16-b29b-bdd1ad2a004a@ec-nantes.fr">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><p>Thanks Mark for your attention.</p><p>The uncleaned error message, compared to my post in July, is as
follows:</p><p> <span style="font-family:monospace"><span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: --------------------- Error Message
--------------------------------------------------------------</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: Argument out of range</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: Column too large: col 4 max 3</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: See <a class="moz-txt-link-freetext" href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dWBkCu100EMuxu8ooVUnqSFN7OhzOBoNHAiwDYEQ5cJ921sU5hdFb-G24ounZFeUQgZkfWqGRX4iIHyQ-xLQElJst5RbKa2pGnk$" moz-do-not-send="true">https://petsc.org/release/faq/</a>
for trouble shooting.</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: Petsc Release Version 3.22.2, unknown </span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: subnb with 3 MPI process(es) and PETSC_ARCH on
pc-str97.ec-nantes.fr by salzman Mon Aug 25 19:11:37 2025</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: Configure options:
PETSC_ARCH=real_fc41_Release_gcc_i4
PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc
--doCleanup=1 --with-scalar-type=real --known-level1-dcach</span><br>
e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC "
--CC_LINKER_FLAGS=-fopenmp --with-cxx=g++
--with-cxx-dialect=c++20 --CXXFLAGS="-fPIC "
--CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran --FFLAGS=<br>
"-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0
--with-fortran-bindings=0 --with-fortran-kernels=1
--with-mpi-compilers=0
--with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-li<br>
b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]"
--with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i<br>
ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]"
--with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi<br>
/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]"
--with-mumps=1
--with-mumps-include=/home/salzman/local/i4_gcc/include
--with-mumps-lib="[/home/salzma<br>
n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]"
--with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_<br>
scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]"
--with-mkl_pardiso=1
--with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include
--with-mkl_pardiso-lib<br>
="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]"
--with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64
--with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with<br>
-pastix=0 --download-pastix=no --with-hwloc=1
--with-hwloc-dir=/home/salzman/local/i4_gcc
--download-hwloc=no
--with-ptscotch-include=/home/salzman/local/i4_gcc/include
--with-ptscotch-lib=<br>
"[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a<br>
,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]"
--with-hypre=1 --download-hypre=yes --with-suitesparse=1
--with-suitesparse-include=/home/<br>
salzman/local/i4_gcc/include
--with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk<br>
lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s<br>
alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]"
--download-suitesparse=no --with-python-exec=python3.12
--have-numpy<br>
=1 ---with-petsc4py=1 ---with-petsc4py-test-np=4
---with-mpi4py=1
--prefix=/home/salzman/local/i4_gcc/real_arithmetic
COPTFLAGS="-O3 -g " CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g "<span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: #1 MatSetValues_SeqAIJ() at
/home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: #2 MatSetValues() at
/home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: #3 MatSetSeqMats_MPIAIJ() at
/home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at
/home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at
/home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: #6 MatCreateSubMatricesMPI() at
/home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: #7 main() at subnb.c:181</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: No PETSc Option Table entries</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">[0]PETSC
ERROR: ----------------End of Error Message -------send
entire error message to <a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:petsc-maint@mcs.anl.gov" moz-do-not-send="true">petsc-maint@mcs.anl.gov</a>----------</span><span style="background-color: rgb(255, 255, 255);"> </span><br>
<span style="background-color: rgb(255, 255, 255);">--------------------------------------------------------------------------</span><span style="background-color: rgb(255, 255, 255);"></span></span></p><p>This message comes from executing the attached test (I
simplified the test by removing the block size from the matrix
used for extraction, compared to the July test). In
proc_xx_output.txt, you will find the output from the code
execution with the -ok option (i.e. irow/idxr and icol/idxc are
the same, i.e. a square sub-block for colour 0 distributed
across the first two processes). <br>
</p><p>Has expected in this case we obtain the 0,3,6,9 sub-block
terms, which are distributed across processes 0 and 1 (two rows
per proc).</p><p>When asking for rectangular sub-block (i.e. with no option) it
crash with column to large on process 0: 4 col max 3 ??? I ask
for 4 rows and 2 columns in this process ???</p><p>Otherwise, I mention the dense aspect of the matrix in ex183.c,
because, in this case, no matter what selection is requested,
all terms are non-null. If there is an issue with the way the
selection is coded in the user program, I think it will be
masked thanks to the full graph representation. However, this
may not be the case — I should test it.</p><p>I'll take a look at ex23.c.</p><p>Thanks,</p><p>A.S.<br>
</p><p><br>
</p><p><br>
</p>
<div class="moz-cite-prefix">Le 25/08/2025 à 17:55, Mark Adams a
écrit :<br>
</div>
<blockquote type="cite" cite="mid:CADOhEh6=fxmWHM5wLgB1QUuSL=p74QRAt2UTUEGp7aR10UyuXg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Ah, OK, never say never.
<div><br>
</div>
<div>MatCreateSubMatrices seems to support creating a new
matrix with the communicator of the IS.</div>
<div>It just needs to read from the input matrix and does not
use it for communication, so it can do that.</div>
<div><br>
</div>
<div>As far as rectangular matrices, there is no reason not to
support that (the row IS and column IS can be distinct).</div>
<div>Can you send the whole error message?</div>
<div>There may not be a test that does this,
but src/mat/tests/ex23.c looks like it may be a
rectangular matrix output.</div>
<div><br>
</div>
<div>And, it should not matter if the input matrix has a 100%
full sparse matrix. It is still MatAIJ.</div>
<div>The semantics and API is the same for sparse or dense
matrices.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Mark</div>
</div>
<br>
<div class="gmail_quote gmail_quote_container">
<div dir="ltr" class="gmail_attr">On Mon, Aug 25, 2025 at
7:31 AM Alexis SALZMAN <<a href="mailto:alexis.salzman@ec-nantes.fr" moz-do-not-send="true" class="moz-txt-link-freetext">alexis.salzman@ec-nantes.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> Hi,<br>
<br>
Thanks for your answer, Mark. Perhaps
MatCreateSubMatricesMPI is the only PETSc function that
acts on a sub-communicator — I'm not sure — but it's clear
that there's no ambiguity on that point. The first line of
the documentation for that function states that it 'may
live on subcomms'. This is confirmed by the
'src/mat/tests/ex183.c' test case. I used this test case
to understand the function, which helped me with my code
and the example I provided in my initial post.
Unfortunately, in this example, the matrix from which the
sub-matrices are extracted is dense, even though it uses a
sparse structure. This does not clarify how to define
sub-matrices when extracting from a sparse distributed
matrix. Since my initial post, I have discovered that
having more columns than rows can also result in the same
error message.
<p>So, my questions boil down to:</p><p>Can MatCreateSubMatricesMPI extract rectangular
matrices from a square distributed sparse matrix?</p><p>If not, the fact that only square matrices can be
extracted in this context should perhaps be mentioned in
the documentation.<br>
</p><p>If so, I would be very grateful for any assistance in
defining an IS pair in this context.<br>
</p><p>Regards</p><p>A.S.<br>
</p>
<div>Le 27/07/2025 à 00:15, Mark Adams a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">First, you can not mix communicators in
PETSc calls in general (ever?), but this error looks
like you might be asking for a row from the matrix
that does not exist.
<div>You should start with a PETSc example code. Test
it and modify it to suit your needs.</div>
<div><br>
</div>
<div>Good luck,</div>
<div>Mark</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jul 25, 2025
at 9:31 AM Alexis SALZMAN <<a href="mailto:alexis.salzman@ec-nantes.fr" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">alexis.salzman@ec-nantes.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
As I am relatively new to Petsc, I may have
misunderstood how to use the <br>
MatCreateSubMatricesMPI function. The attached code
is tuned for three <br>
processes and extracts one matrix for each colour of
a subcommunicator <br>
that has been created using the MPI_Comm_split
function from an MPIAij <br>
matrix. The following error message appears when the
code is set to its <br>
default configuration (i.e. when a rectangular
matrix is extracted with <br>
more rows than columns for colour 0):<br>
<br>
[0]PETSC ERROR: --------------------- Error Message
<br>
--------------------------------------------------------------<br>
[0]PETSC ERROR: Argument out of range<br>
[0]PETSC ERROR: Column too large: col 4 max 3<br>
[0]PETSC ERROR: See <a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$" rel="noreferrer" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$</a>
for trouble shooting.<br>
[0]PETSC ERROR: Petsc Release Version 3.22.2,
unknown<br>
<br>
... petsc git hash 2a89477b25f compiled on a dell i9
computer with Gcc <br>
14.3, mkl 2025.2, .....<br>
[0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at <br>
...petsc/src/mat/impls/aij/seq/aij.c:426<br>
[0]PETSC ERROR: #2 MatSetValues() at <br>
...petsc/src/mat/interface/matrix.c:1543<br>
[0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at <br>
.../petsc/src/mat/impls/aij/mpi/mpiov.c:2965<br>
[0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ()
at <br>
.../petsc/src/mat/impls/aij/mpi/mpiov.c:3163<br>
[0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ()
at <br>
.../petsc/src/mat/impls/aij/mpi/mpiov.c:3196<br>
[0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at <br>
.../petsc/src/mat/interface/matrix.c:7293<br>
[0]PETSC ERROR: #7 main() at sub.c:169<br>
<br>
When the '-ok' option is selected, the code extracts
a square matrix for <br>
colour 0, which runs smoothly in this case.
Selecting the '-trans' <br>
option swaps the row and column selection indices,
providing a <br>
transposed submatrix smoothly. For colour 1, which
uses only one process <br>
and is therefore sequential, rectangular extraction
is OK regardless of <br>
the shape.<br>
<br>
Is this dependency on the shape expected? Have I
missed an important <br>
tuning step somewhere?<br>
<br>
Thank you in advance for any clarification.<br>
<br>
Regards<br>
<br>
A.S.<br>
<br>
P.S.: I'm sorry, but as I'm leaving my office for
the following weeks <br>
this evening, I won't be very responsive during this
period.<br>
<br>
<br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</blockquote>
</div>
</div></blockquote></div><br></div></div></blockquote></div><br></div></body></html>