[petsc-users] [beginner question] Different communicators in the two objects: Argument # 1 and 2 flag 3!
Niklas Fischer
niklas at niklasfi.de
Tue Apr 22 10:23:00 CDT 2014
I have tracked down the problem further and it basically boils down to
the following question: Is it possible to use MatSetValue(s) to set
values which are owned by other processes?
If I create a matrix with
for(int i = 0; i < 4 * size; ++i){
CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
}
for n=m=4, on four processes, one would expect each entry to be 1 + 2 +
3 + 4 = 10, however, PETSc prints
Matrix Object: 1 MPI processes
type: mpiaij
row 0: (0, 1)
row 1: (1, 1)
row 2: (2, 1)
row 3: (3, 1)
row 4: (4, 2)
row 5: (5, 2)
row 6: (6, 2)
row 7: (7, 2)
row 8: (8, 3)
row 9: (9, 3)
row 10: (10, 3)
row 11: (11, 3)
row 12: (12, 4)
row 13: (13, 4)
row 14: (14, 4)
row 15: (15, 4)
which is exactly, what
CHKERRXX(VecGetOwnershipRange(x, &ownership_start, &ownership_end));
for(int i = ownership_start; i < ownership_end; ++i){
CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
}
would give us.
Kind regards,
Niklas Fischer
Am 22.04.2014 14:59, schrieb Niklas Fischer:
> I should probably note that everything is fine if I run the serial
> version of this (with the exact same matrix + right hand side).
>
> PETSc KSPSolve done, residual norm: 3.13459e-13, it took 6 iterations.
>
> Am 22.04.2014 14:12, schrieb Niklas Fischer:
>>
>> Am 22.04.2014 13:57, schrieb Matthew Knepley:
>>> On Tue, Apr 22, 2014 at 6:48 AM, Niklas Fischer <niklas at niklasfi.de
>>> <mailto:niklas at niklasfi.de>> wrote:
>>>
>>> Am 22.04.2014 13:08, schrieb Jed Brown:
>>>
>>> Niklas Fischer <niklas at niklasfi.de
>>> <mailto:niklas at niklasfi.de>> writes:
>>>
>>> Hello,
>>>
>>> I have attached a small test case for a problem I am
>>> experiencing. What
>>> this dummy program does is it reads a vector and a
>>> matrix from a text
>>> file and then solves Ax=b. The same data is available in
>>> two forms:
>>> - everything is in one file (matops.s.0 and vops.s.0)
>>> - the matrix and vector are split between processes
>>> (matops.0,
>>> matops.1, vops.0, vops.1)
>>>
>>> The serial version of the program works perfectly fine
>>> but unfortunately
>>> errors occure, when running the parallel version:
>>>
>>> make && mpirun -n 2 a.out matops vops
>>>
>>> mpic++ -DPETSC_CLANGUAGE_CXX -isystem
>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/include
>>> -isystem
>>> /home/data/fischer/libs/petsc-3.4.3/include
>>> petsctest.cpp -Werror -Wall
>>> -Wpedantic -std=c++11 -L
>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib
>>> -lpetsc
>>> /usr/bin/ld: warning: libmpi_cxx.so.0, needed by
>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,
>>> may conflict with libmpi_cxx.so.1
>>> /usr/bin/ld: warning: libmpi.so.0, needed by
>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,
>>> may conflict with libmpi.so.1
>>> librdmacm: couldn't read ABI version.
>>> librdmacm: assuming: 4
>>> CMA: unable to get RDMA device list
>>> --------------------------------------------------------------------------
>>> [[43019,1],0]: A high-performance Open MPI
>>> point-to-point messaging module
>>> was unable to find any relevant network interfaces:
>>>
>>> Module: OpenFabrics (openib)
>>> Host: dornroeschen.igpm.rwth-aachen.de
>>> <http://dornroeschen.igpm.rwth-aachen.de>
>>> CMA: unable to get RDMA device list
>>>
>>> It looks like your MPI is either broken or some of the code
>>> linked into
>>> your application was compiled with a different MPI or
>>> different version.
>>> Make sure you can compile and run simple MPI programs in
>>> parallel.
>>>
>>> Hello Jed,
>>>
>>> thank you for your inputs. Unfortunately MPI does not seem to be
>>> the issue here. The attachment contains a simple MPI hello world
>>> program which runs flawlessly (I will append the output to this
>>> mail) and I have not encountered any problems with other MPI
>>> programs. My question still stands.
>>>
>>>
>>> This is a simple error. You created the matrix A using
>>> PETSC_COMM_WORLD, but you try to view it
>>> using PETSC_VIEWER_STDOUT_SELF. You need to use
>>> PETSC_VIEWER_STDOUT_WORLD in
>>> order to match.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>> Greetings,
>>> Niklas Fischer
>>>
>>> mpirun -np 2 ./mpitest
>>>
>>> librdmacm: couldn't read ABI version.
>>> librdmacm: assuming: 4
>>> CMA: unable to get RDMA device list
>>> --------------------------------------------------------------------------
>>> [[44086,1],0]: A high-performance Open MPI point-to-point
>>> messaging module
>>> was unable to find any relevant network interfaces:
>>>
>>> Module: OpenFabrics (openib)
>>> Host: dornroeschen.igpm.rwth-aachen.de
>>> <http://dornroeschen.igpm.rwth-aachen.de>
>>>
>>> Another transport will be used instead, although this may result in
>>> lower performance.
>>> --------------------------------------------------------------------------
>>> librdmacm: couldn't read ABI version.
>>> librdmacm: assuming: 4
>>> CMA: unable to get RDMA device list
>>> Hello world from processor dornroeschen.igpm.rwth-aachen.de
>>> <http://dornroeschen.igpm.rwth-aachen.de>, rank 0 out of 2
>>> processors
>>> Hello world from processor dornroeschen.igpm.rwth-aachen.de
>>> <http://dornroeschen.igpm.rwth-aachen.de>, rank 1 out of 2
>>> processors
>>> [dornroeschen.igpm.rwth-aachen.de:128141
>>> <http://dornroeschen.igpm.rwth-aachen.de:128141>] 1 more process
>>> has sent help message help-mpi-btl-base.txt / btl:no-nics
>>> [dornroeschen.igpm.rwth-aachen.de:128141
>>> <http://dornroeschen.igpm.rwth-aachen.de:128141>] Set MCA
>>> parameter "orte_base_help_aggregate" to 0 to see all help /
>>> error messages
>>>
>>>
>> Thank you, Matthew, this solves my viewing problem. Am I doing
>> something wrong when initializing the matrices as well? The matrix'
>> viewing output starts with "Matrix Object: 1 MPI processes" and the
>> Krylov solver does not converge.
>>
>> Your help is really appreciated,
>> Niklas Fischer
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which
>>> their experiments lead.
>>> -- Norbert Wiener
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20140422/058748bf/attachment.html>
More information about the petsc-users
mailing list