[petsc-users] [beginner question] Different communicators in the two objects: Argument # 1 and 2 flag 3!

Tue Apr 22 11:08:45 CDT 2014

On Apr 22, 2014, at 10:23 AM, Niklas Fischer <niklas at niklasfi.de> wrote:

> I have tracked down the problem further and it basically boils down to the following question: Is it possible to use MatSetValue(s) to set values which are owned by other processes?

   Yes it is certainly possible. You should not set a large percent of the values on the “wrong” process but setting some is fine. The values will also be added together if you use ADD_VALUES.

   Below have you called the MatAssemblyBegin/End after setting all the values?

> 
> If I create a matrix with
> 
>     for(int i = 0; i < 4 * size; ++i){
>       CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
>     }
> 
> for n=m=4, on four processes, one would expect each entry to be 1 + 2 + 3 + 4 = 10, however, PETSc prints
> Matrix Object: 1 MPI processes
>   type: mpiaij
> row 0: (0, 1) 
> row 1: (1, 1)
> row 2: (2, 1)
> row 3: (3, 1)
> row 4: (4, 2)
> row 5: (5, 2)
> row 6: (6, 2)
> row 7: (7, 2)
> row 8: (8, 3)
> row 9: (9, 3)
> row 10: (10, 3)
> row 11: (11, 3)
> row 12: (12, 4)
> row 13: (13, 4)
> row 14: (14, 4)
> row 15: (15, 4)
> which is exactly, what 
> 
> CHKERRXX(VecGetOwnershipRange(x, &ownership_start, &ownership_end));
> for(int i = ownership_start; i < ownership_end; ++i){
>    CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
> }
> would give us.
> 
> Kind regards,
> Niklas Fischer
> 
> Am 22.04.2014 14:59, schrieb Niklas Fischer:
>> I should probably note that everything is fine if I run the serial version of this (with the exact same matrix + right hand side).
>> 
>> PETSc KSPSolve done, residual norm: 3.13459e-13, it took 6 iterations.
>> 
>> Am 22.04.2014 14:12, schrieb Niklas Fischer:
>>> 
>>> Am 22.04.2014 13:57, schrieb Matthew Knepley:
>>>> On Tue, Apr 22, 2014 at 6:48 AM, Niklas Fischer <niklas at niklasfi.de> wrote:
>>>> Am 22.04.2014 13:08, schrieb Jed Brown:
>>>> Niklas Fischer <niklas at niklasfi.de> writes:
>>>> 
>>>> Hello,
>>>> 
>>>> I have attached a small test case for a problem I am experiencing. What
>>>> this dummy program does is it reads a vector and a matrix from a text
>>>> file and then solves Ax=b. The same data is available in two forms:
>>>>   - everything is in one file (matops.s.0 and vops.s.0)
>>>>   - the matrix and vector are split between processes (matops.0,
>>>> matops.1, vops.0, vops.1)
>>>> 
>>>> The serial version of the program works perfectly fine but unfortunately
>>>> errors occure, when running the parallel version:
>>>> 
>>>> make && mpirun -n 2 a.out matops vops
>>>> 
>>>> mpic++ -DPETSC_CLANGUAGE_CXX -isystem
>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/include -isystem
>>>> /home/data/fischer/libs/petsc-3.4.3/include petsctest.cpp -Werror -Wall
>>>> -Wpedantic -std=c++11 -L
>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib -lpetsc
>>>> /usr/bin/ld: warning: libmpi_cxx.so.0, needed by
>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,
>>>> may conflict with libmpi_cxx.so.1
>>>> /usr/bin/ld: warning: libmpi.so.0, needed by
>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,
>>>> may conflict with libmpi.so.1
>>>> librdmacm: couldn't read ABI version.
>>>> librdmacm: assuming: 4
>>>> CMA: unable to get RDMA device list
>>>> --------------------------------------------------------------------------
>>>> [[43019,1],0]: A high-performance Open MPI point-to-point messaging module
>>>> was unable to find any relevant network interfaces:
>>>> 
>>>> Module: OpenFabrics (openib)
>>>>    Host: dornroeschen.igpm.rwth-aachen.de
>>>> CMA: unable to get RDMA device list
>>>> It looks like your MPI is either broken or some of the code linked into
>>>> your application was compiled with a different MPI or different version.
>>>> Make sure you can compile and run simple MPI programs in parallel.
>>>> Hello Jed,
>>>> 
>>>> thank you for your inputs. Unfortunately MPI does not seem to be the issue here. The attachment contains a simple MPI hello world program which runs flawlessly (I will append the output to this mail) and I have not encountered any problems with other MPI programs. My question still stands.
>>>> 
>>>> This is a simple error. You created the matrix A using PETSC_COMM_WORLD, but you try to view it
>>>> using PETSC_VIEWER_STDOUT_SELF. You need to use PETSC_VIEWER_STDOUT_WORLD in
>>>> order to match.
>>>> 
>>>>   Thanks,
>>>> 
>>>>      Matt
>>>>  
>>>> Greetings,
>>>> Niklas Fischer
>>>> 
>>>> mpirun -np 2 ./mpitest
>>>> 
>>>> librdmacm: couldn't read ABI version.
>>>> librdmacm: assuming: 4
>>>> CMA: unable to get RDMA device list
>>>> --------------------------------------------------------------------------
>>>> [[44086,1],0]: A high-performance Open MPI point-to-point messaging module
>>>> was unable to find any relevant network interfaces:
>>>> 
>>>> Module: OpenFabrics (openib)
>>>>   Host: dornroeschen.igpm.rwth-aachen.de
>>>> 
>>>> Another transport will be used instead, although this may result in
>>>> lower performance.
>>>> --------------------------------------------------------------------------
>>>> librdmacm: couldn't read ABI version.
>>>> librdmacm: assuming: 4
>>>> CMA: unable to get RDMA device list
>>>> Hello world from processor dornroeschen.igpm.rwth-aachen.de, rank 0 out of 2 processors
>>>> Hello world from processor dornroeschen.igpm.rwth-aachen.de, rank 1 out of 2 processors
>>>> [dornroeschen.igpm.rwth-aachen.de:128141] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
>>>> [dornroeschen.igpm.rwth-aachen.de:128141] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
>>>> 
>>> Thank you, Matthew, this solves my viewing problem. Am I doing something wrong when initializing the matrices as well? The matrix' viewing output starts with "Matrix Object: 1 MPI processes" and the Krylov solver does not converge.
>>> 
>>> Your help is really appreciated,
>>> Niklas Fischer
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>> 
>> 
> 