<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Thanks Mark for your attention.</p>
<p>The uncleaned error message, compared to my post in July, is as
follows:</p>
<p> <span style="font-family:monospace"><span
style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: --------------------- Error Message
--------------------------------------------------------------</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: Argument out of range</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: Column too large: col 4 max 3</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: See <a class="moz-txt-link-freetext" href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!dFZ4KSyqnoKD_8HJEOBBrLiK5TUCQmZbw09Dxau1D-3pxswHNP1D3HpEP-nXcrUdppQnRXo6rVLtt26Bd50bK6i7-_w38dW0qu4$">https://petsc.org/release/faq/</a>
for trouble shooting.</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: Petsc Release Version 3.22.2, unknown </span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: subnb with 3 MPI process(es) and PETSC_ARCH on
pc-str97.ec-nantes.fr by salzman Mon Aug 25 19:11:37 2025</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: Configure options: PETSC_ARCH=real_fc41_Release_gcc_i4
PETSC_DIR=/home/salzman/devel/ExternalLib/build/PETSC/petsc
--doCleanup=1 --with-scalar-type=real --known-level1-dcach</span><br>
e-linesize=64 --with-cc=gcc --CFLAGS="-fPIC "
--CC_LINKER_FLAGS=-fopenmp --with-cxx=g++
--with-cxx-dialect=c++20 --CXXFLAGS="-fPIC "
--CXX_LINKER_FLAGS=-fopenmp --with-fc=gfortran --FFLAGS=<br>
"-fPIC " --FC_LINKER_FLAGS=-fopenmp --with-debugging=0
--with-fortran-bindings=0 --with-fortran-kernels=1
--with-mpi-compilers=0
--with-mpi-include=/usr/include/openmpi-x86_64 --with-mpi-li<br>
b="[/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi.so,/usr/lib64/openmpi/lib/libmpi_mpifh.so]"
--with-blas-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/i<br>
ntel/oneapi/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]"
--with-lapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_intel_lp64.so,/opt/intel/oneapi<br>
/mkl/latest/lib/libmkl_gnu_thread.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_core.so]"
--with-mumps=1
--with-mumps-include=/home/salzman/local/i4_gcc/include
--with-mumps-lib="[/home/salzma<br>
n/local/i4_gcc/lib/libdmumps.so,/home/salzman/local/i4_gcc/lib/libmumps_common.so,/home/salzman/local/i4_gcc/lib/libpord.so]"
--with-scalapack-lib="[/opt/intel/oneapi/mkl/latest/lib/libmkl_<br>
scalapack_lp64.so,/opt/intel/oneapi/mkl/latest/lib/libmkl_blacs_openmpi_lp64.so]"
--with-mkl_pardiso=1
--with-mkl_pardiso-include=/opt/intel/oneapi/mkl/latest/include
--with-mkl_pardiso-lib<br>
="[/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_lp64.so]"
--with-hdf5=1 --with-hdf5-include=/usr/include/openmpi-x86_64
--with-hdf5-lib="[/usr/lib64/openmpi/lib/libhdf5.so]" --with<br>
-pastix=0 --download-pastix=no --with-hwloc=1
--with-hwloc-dir=/home/salzman/local/i4_gcc --download-hwloc=no
--with-ptscotch-include=/home/salzman/local/i4_gcc/include
--with-ptscotch-lib=<br>
"[/home/salzman/local/i4_gcc/lib/libptscotch.a,/home/salzman/local/i4_gcc/lib/libptscotcherr.a,/home/salzman/local/i4_gcc/lib/libptscotcherrexit.a,/home/salzman/local/i4_gcc/lib/libscotch.a<br>
,/home/salzman/local/i4_gcc/lib/libscotcherr.a,/home/salzman/local/i4_gcc/lib/libscotcherrexit.a]"
--with-hypre=1 --download-hypre=yes --with-suitesparse=1
--with-suitesparse-include=/home/<br>
salzman/local/i4_gcc/include
--with-suitesparse-lib="[/home/salzman/local/i4_gcc/lib/libsuitesparseconfig.so,/home/salzman/local/i4_gcc/lib/libumfpack.so,/home/salzman/local/i4_gcc/lib/libk<br>
lu.so,/home/salzman/local/i4_gcc/lib/libcholmod.so,/home/salzman/local/i4_gcc/lib/libspqr.so,/home/salzman/local/i4_gcc/lib/libcolamd.so,/home/salzman/local/i4_gcc/lib/libccolamd.so,/home/s<br>
alzman/local/i4_gcc/lib/libcamd.so,/home/salzman/local/i4_gcc/lib/libamd.so,/home/salzman/local/i4_gcc/lib/libmetis.so]"
--download-suitesparse=no --with-python-exec=python3.12
--have-numpy<br>
=1 ---with-petsc4py=1 ---with-petsc4py-test-np=4
---with-mpi4py=1
--prefix=/home/salzman/local/i4_gcc/real_arithmetic
COPTFLAGS="-O3 -g " CXXOPTFLAGS="-O3 -g " FOPTFLAGS="-O3 -g "<span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: #1 MatSetValues_SeqAIJ() at
/home/salzman/devel/PETSc/petsc/src/mat/impls/aij/seq/aij.c:426</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: #2 MatSetValues() at
/home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:1543</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: #3 MatSetSeqMats_MPIAIJ() at
/home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:2965</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ() at
/home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3163</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at
/home/salzman/devel/PETSc/petsc/src/mat/impls/aij/mpi/mpiov.c:3196</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: #6 MatCreateSubMatricesMPI() at
/home/salzman/devel/PETSc/petsc/src/mat/interface/matrix.c:7293</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: #7 main() at subnb.c:181</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: No PETSc Option Table entries</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">[0]PETSC
ERROR: ----------------End of Error Message -------send entire
error message to <a
class="moz-txt-link-abbreviated moz-txt-link-freetext"
href="mailto:petsc-maint@mcs.anl.gov">petsc-maint@mcs.anl.gov</a>----------</span><span
style="color:#000000;background-color:#ffffff;"> </span><br>
<span style="color:#000000;background-color:#ffffff;">--------------------------------------------------------------------------</span><span
style="color:#000000;background-color:#ffffff;"></span></span></p>
<p>This message comes from executing the attached test (I simplified
the test by removing the block size from the matrix used for
extraction, compared to the July test). In proc_xx_output.txt, you
will find the output from the code execution with the -ok option
(i.e. irow/idxr and icol/idxc are the same, i.e. a square
sub-block for colour 0 distributed across the first two
processes). <br>
</p>
<p>Has expected in this case we obtain the 0,3,6,9 sub-block terms,
which are distributed across processes 0 and 1 (two rows per
proc).</p>
<p>When asking for rectangular sub-block (i.e. with no option) it
crash with column to large on process 0: 4 col max 3 ??? I ask for
4 rows and 2 columns in this process ???</p>
<p>Otherwise, I mention the dense aspect of the matrix in ex183.c,
because, in this case, no matter what selection is requested, all
terms are non-null. If there is an issue with the way the
selection is coded in the user program, I think it will be masked
thanks to the full graph representation. However, this may not be
the case — I should test it.</p>
<p>I'll take a look at ex23.c.</p>
<p>Thanks,</p>
<p>A.S.<br>
</p>
<p><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">Le 25/08/2025 à 17:55, Mark Adams a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CADOhEh6=fxmWHM5wLgB1QUuSL=p74QRAt2UTUEGp7aR10UyuXg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Ah, OK, never say never.
<div><br>
</div>
<div>MatCreateSubMatrices seems to support creating a new matrix
with the communicator of the IS.</div>
<div>It just needs to read from the input matrix and does not
use it for communication, so it can do that.</div>
<div><br>
</div>
<div>As far as rectangular matrices, there is no reason not to
support that (the row IS and column IS can be distinct).</div>
<div>Can you send the whole error message?</div>
<div>There may not be a test that does this,
but src/mat/tests/ex23.c looks like it may be a
rectangular matrix output.</div>
<div><br>
</div>
<div>And, it should not matter if the input matrix has a 100%
full sparse matrix. It is still MatAIJ.</div>
<div>The semantics and API is the same for sparse or dense
matrices.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Mark</div>
</div>
<br>
<div class="gmail_quote gmail_quote_container">
<div dir="ltr" class="gmail_attr">On Mon, Aug 25, 2025 at
7:31 AM Alexis SALZMAN <<a
href="mailto:alexis.salzman@ec-nantes.fr"
moz-do-not-send="true" class="moz-txt-link-freetext">alexis.salzman@ec-nantes.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> Hi,<br>
<br>
Thanks for your answer, Mark. Perhaps
MatCreateSubMatricesMPI is the only PETSc function that acts
on a sub-communicator — I'm not sure — but it's clear that
there's no ambiguity on that point. The first line of the
documentation for that function states that it 'may live on
subcomms'. This is confirmed by the 'src/mat/tests/ex183.c'
test case. I used this test case to understand the function,
which helped me with my code and the example I provided in
my initial post. Unfortunately, in this example, the matrix
from which the sub-matrices are extracted is dense, even
though it uses a sparse structure. This does not clarify how
to define sub-matrices when extracting from a sparse
distributed matrix. Since my initial post, I have discovered
that having more columns than rows can also result in the
same error message.
<p>So, my questions boil down to:</p>
<p>Can MatCreateSubMatricesMPI extract rectangular matrices
from a square distributed sparse matrix?</p>
<p>If not, the fact that only square matrices can be
extracted in this context should perhaps be mentioned in
the documentation.<br>
</p>
<p>If so, I would be very grateful for any assistance in
defining an IS pair in this context.<br>
</p>
<p>Regards</p>
<p>A.S.<br>
</p>
<div>Le 27/07/2025 à 00:15, Mark Adams a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">First, you can not mix communicators in
PETSc calls in general (ever?), but this error looks
like you might be asking for a row from the matrix
that does not exist.
<div>You should start with a PETSc example code. Test it
and modify it to suit your needs.</div>
<div><br>
</div>
<div>Good luck,</div>
<div>Mark</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jul 25, 2025
at 9:31 AM Alexis SALZMAN <<a
href="mailto:alexis.salzman@ec-nantes.fr"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">alexis.salzman@ec-nantes.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
As I am relatively new to Petsc, I may have
misunderstood how to use the <br>
MatCreateSubMatricesMPI function. The attached code is
tuned for three <br>
processes and extracts one matrix for each colour of a
subcommunicator <br>
that has been created using the MPI_Comm_split
function from an MPIAij <br>
matrix. The following error message appears when the
code is set to its <br>
default configuration (i.e. when a rectangular matrix
is extracted with <br>
more rows than columns for colour 0):<br>
<br>
[0]PETSC ERROR: --------------------- Error Message <br>
--------------------------------------------------------------<br>
[0]PETSC ERROR: Argument out of range<br>
[0]PETSC ERROR: Column too large: col 4 max 3<br>
[0]PETSC ERROR: See <a
href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$"
rel="noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ZqH097BZ0G0O3WI7RWrwIKFNpyk0czSWEqfusAeTlgEygAffwpgBUzsLw1TIoGkjZ3mYG-NRQxxFoxU4y8EyY0ofiz9I43Qwe0w$</a>
for trouble shooting.<br>
[0]PETSC ERROR: Petsc Release Version 3.22.2, unknown<br>
<br>
... petsc git hash 2a89477b25f compiled on a dell i9
computer with Gcc <br>
14.3, mkl 2025.2, .....<br>
[0]PETSC ERROR: #1 MatSetValues_SeqAIJ() at <br>
...petsc/src/mat/impls/aij/seq/aij.c:426<br>
[0]PETSC ERROR: #2 MatSetValues() at <br>
...petsc/src/mat/interface/matrix.c:1543<br>
[0]PETSC ERROR: #3 MatSetSeqMats_MPIAIJ() at <br>
.../petsc/src/mat/impls/aij/mpi/mpiov.c:2965<br>
[0]PETSC ERROR: #4 MatCreateSubMatricesMPI_MPIXAIJ()
at <br>
.../petsc/src/mat/impls/aij/mpi/mpiov.c:3163<br>
[0]PETSC ERROR: #5 MatCreateSubMatricesMPI_MPIAIJ() at
<br>
.../petsc/src/mat/impls/aij/mpi/mpiov.c:3196<br>
[0]PETSC ERROR: #6 MatCreateSubMatricesMPI() at <br>
.../petsc/src/mat/interface/matrix.c:7293<br>
[0]PETSC ERROR: #7 main() at sub.c:169<br>
<br>
When the '-ok' option is selected, the code extracts a
square matrix for <br>
colour 0, which runs smoothly in this case. Selecting
the '-trans' <br>
option swaps the row and column selection indices,
providing a <br>
transposed submatrix smoothly. For colour 1, which
uses only one process <br>
and is therefore sequential, rectangular extraction is
OK regardless of <br>
the shape.<br>
<br>
Is this dependency on the shape expected? Have I
missed an important <br>
tuning step somewhere?<br>
<br>
Thank you in advance for any clarification.<br>
<br>
Regards<br>
<br>
A.S.<br>
<br>
P.S.: I'm sorry, but as I'm leaving my office for the
following weeks <br>
this evening, I won't be very responsive during this
period.<br>
<br>
<br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</body>
</html>