[petsc-users] [beginner question] Different communicators in the two objects: Argument # 1 and 2 flag 3!

Tue Apr 22 11:32:40 CDT 2014

Hello Barry,

Am 22.04.2014 18:08, schrieb Barry Smith:
> On Apr 22, 2014, at 10:23 AM, Niklas Fischer <niklas at niklasfi.de> wrote:
>
>> I have tracked down the problem further and it basically boils down to the following question: Is it possible to use MatSetValue(s) to set values which are owned by other processes?
>     Yes it is certainly possible. You should not set a large percent of the values on the “wrong” process but setting some is fine. The values will also be added together if you use ADD_VALUES.
>
>     Below have you called the MatAssemblyBegin/End after setting all the values?
It certainly is AssemblyBegin first, then set values, then AssemblyEnd

     CHKERRXX(MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY));
     CHKERRXX(VecAssemblyBegin(b));

     for(int i = 0; i < 4 * size; ++i){
       CHKERRXX(VecSetValue(b, i, 1, ADD_VALUES));
     }

     for(int i = 0; i < 4 * size; ++i){
       CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
     }

     CHKERRXX(MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY));
     CHKERRXX(VecAssemblyEnd(b));

My observation, setting the values does not work, also ties in with the 
solution given by the solver which is the result of

Diag [1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4]x=Constant(1,16)

Process [0]
1
1
1
1
Process [1]
0.5
0.5
0.5
0.5
Process [2]
0.333333
0.333333
0.333333
0.333333
Process [3]
0.25
0.25
0.25
0.25
>     
>> If I create a matrix with
>>
>>      for(int i = 0; i < 4 * size; ++i){
>>        CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
>>      }
>>
>> for n=m=4, on four processes, one would expect each entry to be 1 + 2 + 3 + 4 = 10, however, PETSc prints
>> Matrix Object: 1 MPI processes
>>    type: mpiaij
>> row 0: (0, 1)
>> row 1: (1, 1)
>> row 2: (2, 1)
>> row 3: (3, 1)
>> row 4: (4, 2)
>> row 5: (5, 2)
>> row 6: (6, 2)
>> row 7: (7, 2)
>> row 8: (8, 3)
>> row 9: (9, 3)
>> row 10: (10, 3)
>> row 11: (11, 3)
>> row 12: (12, 4)
>> row 13: (13, 4)
>> row 14: (14, 4)
>> row 15: (15, 4)
>> which is exactly, what
>>
>> CHKERRXX(VecGetOwnershipRange(x, &ownership_start, &ownership_end));
>> for(int i = ownership_start; i < ownership_end; ++i){
>>     CHKERRXX(MatSetValue(A, i, i, rank+1, ADD_VALUES));
>> }
>> would give us.
>>
>> Kind regards,
>> Niklas Fischer
>>
>> Am 22.04.2014 14:59, schrieb Niklas Fischer:
>>> I should probably note that everything is fine if I run the serial version of this (with the exact same matrix + right hand side).
>>>
>>> PETSc KSPSolve done, residual norm: 3.13459e-13, it took 6 iterations.
>>>
>>> Am 22.04.2014 14:12, schrieb Niklas Fischer:
>>>> Am 22.04.2014 13:57, schrieb Matthew Knepley:
>>>>> On Tue, Apr 22, 2014 at 6:48 AM, Niklas Fischer <niklas at niklasfi.de> wrote:
>>>>> Am 22.04.2014 13:08, schrieb Jed Brown:
>>>>> Niklas Fischer <niklas at niklasfi.de> writes:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have attached a small test case for a problem I am experiencing. What
>>>>> this dummy program does is it reads a vector and a matrix from a text
>>>>> file and then solves Ax=b. The same data is available in two forms:
>>>>>    - everything is in one file (matops.s.0 and vops.s.0)
>>>>>    - the matrix and vector are split between processes (matops.0,
>>>>> matops.1, vops.0, vops.1)
>>>>>
>>>>> The serial version of the program works perfectly fine but unfortunately
>>>>> errors occure, when running the parallel version:
>>>>>
>>>>> make && mpirun -n 2 a.out matops vops
>>>>>
>>>>> mpic++ -DPETSC_CLANGUAGE_CXX -isystem
>>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/include -isystem
>>>>> /home/data/fischer/libs/petsc-3.4.3/include petsctest.cpp -Werror -Wall
>>>>> -Wpedantic -std=c++11 -L
>>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib -lpetsc
>>>>> /usr/bin/ld: warning: libmpi_cxx.so.0, needed by
>>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,
>>>>> may conflict with libmpi_cxx.so.1
>>>>> /usr/bin/ld: warning: libmpi.so.0, needed by
>>>>> /home/data/fischer/libs/petsc-3.4.3/arch-linux2-c-debug/lib/libpetsc.so,
>>>>> may conflict with libmpi.so.1
>>>>> librdmacm: couldn't read ABI version.
>>>>> librdmacm: assuming: 4
>>>>> CMA: unable to get RDMA device list
>>>>> --------------------------------------------------------------------------
>>>>> [[43019,1],0]: A high-performance Open MPI point-to-point messaging module
>>>>> was unable to find any relevant network interfaces:
>>>>>
>>>>> Module: OpenFabrics (openib)
>>>>>     Host: dornroeschen.igpm.rwth-aachen.de
>>>>> CMA: unable to get RDMA device list
>>>>> It looks like your MPI is either broken or some of the code linked into
>>>>> your application was compiled with a different MPI or different version.
>>>>> Make sure you can compile and run simple MPI programs in parallel.
>>>>> Hello Jed,
>>>>>
>>>>> thank you for your inputs. Unfortunately MPI does not seem to be the issue here. The attachment contains a simple MPI hello world program which runs flawlessly (I will append the output to this mail) and I have not encountered any problems with other MPI programs. My question still stands.
>>>>>
>>>>> This is a simple error. You created the matrix A using PETSC_COMM_WORLD, but you try to view it
>>>>> using PETSC_VIEWER_STDOUT_SELF. You need to use PETSC_VIEWER_STDOUT_WORLD in
>>>>> order to match.
>>>>>
>>>>>    Thanks,
>>>>>
>>>>>       Matt
>>>>>   
>>>>> Greetings,
>>>>> Niklas Fischer
>>>>>
>>>>> mpirun -np 2 ./mpitest
>>>>>
>>>>> librdmacm: couldn't read ABI version.
>>>>> librdmacm: assuming: 4
>>>>> CMA: unable to get RDMA device list
>>>>> --------------------------------------------------------------------------
>>>>> [[44086,1],0]: A high-performance Open MPI point-to-point messaging module
>>>>> was unable to find any relevant network interfaces:
>>>>>
>>>>> Module: OpenFabrics (openib)
>>>>>    Host: dornroeschen.igpm.rwth-aachen.de
>>>>>
>>>>> Another transport will be used instead, although this may result in
>>>>> lower performance.
>>>>> --------------------------------------------------------------------------
>>>>> librdmacm: couldn't read ABI version.
>>>>> librdmacm: assuming: 4
>>>>> CMA: unable to get RDMA device list
>>>>> Hello world from processor dornroeschen.igpm.rwth-aachen.de, rank 0 out of 2 processors
>>>>> Hello world from processor dornroeschen.igpm.rwth-aachen.de, rank 1 out of 2 processors
>>>>> [dornroeschen.igpm.rwth-aachen.de:128141] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
>>>>> [dornroeschen.igpm.rwth-aachen.de:128141] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
>>>>>
>>>> Thank you, Matthew, this solves my viewing problem. Am I doing something wrong when initializing the matrices as well? The matrix' viewing output starts with "Matrix Object: 1 MPI processes" and the Krylov solver does not converge.
>>>>
>>>> Your help is really appreciated,
>>>> Niklas Fischer
>>>>>
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener