<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
I have tracked down the problem further and it basically boils down
to the following question: Is it possible to use MatSetValue(s) to
set values which are owned by other processes?<br>
<br>
If I create a matrix with<br>
<br>
for(int i = 0; i < 4 * size; ++i){<br>
CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));<br>
}<br>
<br>
for n=m=4, on four processes, one would expect each entry to be 1 +
2 + 3 + 4 = 10, however, PETSc prints<br>
<blockquote>Matrix Object: 1 MPI processes<br>
type: mpiaij<br>
row 0: (0, 1) <br>
row 1: (1, 1)<br>
row 2: (2, 1)<br>
row 3: (3, 1)<br>
row 4: (4, 2)<br>
row 5: (5, 2)<br>
row 6: (6, 2)<br>
row 7: (7, 2)<br>
row 8: (8, 3)<br>
row 9: (9, 3)<br>
row 10: (10, 3)<br>
row 11: (11, 3)<br>
row 12: (12, 4)<br>
row 13: (13, 4)<br>
row 14: (14, 4)<br>
row 15: (15, 4)<br>
</blockquote>
which is exactly, what <br>
<br>
<blockquote>CHKERRXX(VecGetOwnershipRange(x, &ownership_start,
&ownership_end));<br>
for(int i = ownership_start; i < ownership_end; ++i){<br>
CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));<br>
}<br>
</blockquote>
would give us.<br>
<br>
Kind regards,<br>
Niklas Fischer<br>
<br>
<div class="moz-cite-prefix">Am 22.04.2014 14:59, schrieb Niklas
Fischer:<br>
</div>
<blockquote cite="mid:535667BC.10607@niklasfi.de" type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
I should probably note that everything is fine if I run the serial
version of this (with the exact same matrix + right hand side).<br>
<br>
PETSc KSPSolve done, residual norm: 3.13459e-13, it took 6
iterations.<br>
<br>
<div class="moz-cite-prefix">Am 22.04.2014 14:12, schrieb Niklas
Fischer:<br>
</div>
<blockquote cite="mid:53565CB3.8030604@niklasfi.de" type="cite">
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
<br>
<div class="moz-cite-prefix">Am 22.04.2014 13:57, schrieb
Matthew Knepley:<br>
</div>
<blockquote
cite="mid:CAMYG4GkXGE5Vx6qTncw1z0xw+yvzxUCJS5Y6co_3K3UUZotRcw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Apr 22, 2014 at 6:48 AM,
Niklas Fischer <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:niklas@niklasfi.de" target="_blank">niklas@niklasfi.de</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Am
22.04.2014 13:08, schrieb Jed Brown:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Niklas Fischer <<a moz-do-not-send="true"
href="mailto:niklas@niklasfi.de" target="_blank">niklas@niklasfi.de</a>>
writes:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
I have attached a small test case for a problem I
am experiencing. What<br>
this dummy program does is it reads a vector and a
matrix from a text<br>
file and then solves Ax=b. The same data is
available in two forms:<br>
- everything is in one file (matops.s.0 and
vops.s.0)<br>
- the matrix and vector are split between
processes (matops.0,<br>
matops.1, vops.0, vops.1)<br>
<br>
The serial version of the program works perfectly
fine but unfortunately<br>
errors occure, when running the parallel version:<br>
<br>
make && mpirun -n 2 a.out matops vops<br>
<br>
mpic++ -DPETSC_CLANGUAGE_CXX -isystem<br>
/home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/include
-isystem<br>
/home/data/fischer/libs/petsc-3.4.3/include
petsctest.cpp -Werror -Wall<br>
-Wpedantic -std=c++11 -L<br>
/home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib
-lpetsc<br>
/usr/bin/ld: warning: libmpi_cxx.so.0, needed by<br>
/home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,<br>
may conflict with libmpi_cxx.so.1<br>
/usr/bin/ld: warning: libmpi.so.0, needed by<br>
/home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,<br>
may conflict with libmpi.so.1<br>
librdmacm: couldn't read ABI version.<br>
librdmacm: assuming: 4<br>
CMA: unable to get RDMA device list<br>
--------------------------------------------------------------------------<br>
[[43019,1],0]: A high-performance Open MPI
point-to-point messaging module<br>
was unable to find any relevant network
interfaces:<br>
<br>
Module: OpenFabrics (openib)<br>
Host: <a moz-do-not-send="true"
href="http://dornroeschen.igpm.rwth-aachen.de"
target="_blank">dornroeschen.igpm.rwth-aachen.de</a><br>
CMA: unable to get RDMA device list<br>
</blockquote>
It looks like your MPI is either broken or some of
the code linked into<br>
your application was compiled with a different MPI
or different version.<br>
Make sure you can compile and run simple MPI
programs in parallel.<br>
</blockquote>
Hello Jed,<br>
<br>
thank you for your inputs. Unfortunately MPI does not
seem to be the issue here. The attachment contains a
simple MPI hello world program which runs flawlessly
(I will append the output to this mail) and I have not
encountered any problems with other MPI programs. My
question still stands.<br>
</blockquote>
<div><br>
</div>
<div>This is a simple error. You created the matrix A
using PETSC_COMM_WORLD, but you try to view it</div>
<div>using PETSC_VIEWER_STDOUT_SELF. You need to use
PETSC_VIEWER_STDOUT_WORLD in</div>
<div>order to match.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Greetings,<br>
Niklas Fischer<br>
<br>
mpirun -np 2 ./mpitest<br>
<br>
librdmacm: couldn't read ABI version.<br>
librdmacm: assuming: 4<br>
CMA: unable to get RDMA device list<br>
--------------------------------------------------------------------------<br>
[[44086,1],0]: A high-performance Open MPI
point-to-point messaging module<br>
was unable to find any relevant network interfaces:<br>
<br>
Module: OpenFabrics (openib)<br>
Host: <a moz-do-not-send="true"
href="http://dornroeschen.igpm.rwth-aachen.de"
target="_blank">dornroeschen.igpm.rwth-aachen.de</a><br>
<br>
Another transport will be used instead, although this
may result in<br>
lower performance.<br>
--------------------------------------------------------------------------<br>
librdmacm: couldn't read ABI version.<br>
librdmacm: assuming: 4<br>
CMA: unable to get RDMA device list<br>
Hello world from processor <a moz-do-not-send="true"
href="http://dornroeschen.igpm.rwth-aachen.de"
target="_blank">dornroeschen.igpm.rwth-aachen.de</a>,
rank 0 out of 2 processors<br>
Hello world from processor <a moz-do-not-send="true"
href="http://dornroeschen.igpm.rwth-aachen.de"
target="_blank">dornroeschen.igpm.rwth-aachen.de</a>,
rank 1 out of 2 processors<br>
[<a moz-do-not-send="true"
href="http://dornroeschen.igpm.rwth-aachen.de:128141"
target="_blank">dornroeschen.igpm.rwth-aachen.de:128141</a>]
1 more process has sent help message
help-mpi-btl-base.txt / btl:no-nics<br>
[<a moz-do-not-send="true"
href="http://dornroeschen.igpm.rwth-aachen.de:128141"
target="_blank">dornroeschen.igpm.rwth-aachen.de:128141</a>]
Set MCA parameter "orte_base_help_aggregate" to 0 to
see all help / error messages<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
Thank you, Matthew, this solves my viewing problem. Am I doing
something wrong when initializing the matrices as well? The
matrix' viewing output starts with "Matrix Object: 1 MPI
processes" and the Krylov solver does not converge.<br>
<br>
Your help is really appreciated,<br>
Niklas Fischer<br>
<blockquote
cite="mid:CAMYG4GkXGE5Vx6qTncw1z0xw+yvzxUCJS5Y6co_3K3UUZotRcw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra"><br clear="all">
<div><br>
</div>
-- <br>
What most experimenters take for granted before they begin
their experiments is infinitely more interesting than any
results to which their experiments lead.<br>
-- Norbert Wiener </div>
</div>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>