[petsc-users] How to have a local copy (sequential) of a parallel matrix
ehsan sadrfaridpour
it.sadr at gmail.com
Tue Jul 5 17:13:36 CDT 2016
I faced a problem with my code. The problem is related to MatCreateSeqAIJ().
I comment the rest of my code and just keeping the below lines cause me the
error.
*Code:*
Mat * m_WA_nt_local;
MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points, pre_init_size,
NULL, m_WA_nt_local);
PetscPrintf(PETSC_COMM_SELF, "[CS][pCalc_P] rank:%d, num_points:%d,
p_init:%d\n", rank, num_points, pre_init_size);
exit(1);
*Error:*
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Null argument, when expecting valid pointer
> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Null argument, when expecting valid pointer
> [1]PETSC ERROR: Null Pointer: Parameter # 2
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [2]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [2]PETSC ERROR: Null argument, when expecting valid pointer
> [2]PETSC ERROR: Null Pointer: Parameter # 2
> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [2]PETSC ERROR: Petsc Release Version 3.6.3, unknown
> [2]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp Tue
> Jul 5 18:05:15 2016
> [2]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug --with-cc=gcc
> --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1
> --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1
> --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1
> --download-superlu=1 --download-metis=1 --download-parmetis=1
> --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/
> [2]PETSC ERROR: #1 MatCreate() line 79 in
> /home/esfp/tools/libraries/petsc/src/mat/utils/gcreate.c
> [2]PETSC ERROR: #2 MatCreateSeqAIJ() line 3471 in
> /home/esfp/tools/libraries/petsc/src/mat/impls/aij/seq/aij.c
> Null Pointer: Parameter # 2
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown
> [0]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp Tue
> Jul 5 18:05:15 2016
> [0]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug --with-cc=gcc
> --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1
> --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1
> --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1
> --download-superlu=1 --download-metis=1 --download-parmetis=1
> --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/
> [0]PETSC ERROR: #1 MatCreate() line 79 in
> /home/esfp/tools/libraries/petsc/src/mat/utils/gcreate.c
> [0]PETSC ERROR: #2 MatCreateSeqAIJ() line 3471 in
> /home/esfp/tools/libraries/petsc/src/mat/impls/aij/seq/aij.c
> [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown
> [1]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp Tue
> Jul 5 18:05:15 2016
> [1]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug --with-cc=gcc
> --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1
> --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1
> --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1
> --download-superlu=1 --download-metis=1 --download-parmetis=1
> --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/
> [1]PETSC ERROR: #1 MatCreate() line 79 in
> /home/esfp/tools/libraries/petsc/src/mat/utils/gcreate.c
> [1]PETSC ERROR: #2 MatCreateSeqAIJ() line 3471 in
> /home/esfp/tools/libraries/petsc/src/mat/impls/aij/seq/aij.c
> [CS][pCalc_P] rank:1, num_points:10, p_init:300
> [CS][pCalc_P] rank:2, num_points:10, p_init:300
> [CS][pCalc_P] rank:0, num_points:10, p_init:300
>
As you can see nothing is NULL in my call to the MatCreateSeqAIJ.
I tried to debug it with -start_in_debugger, but I got another error.
> $ make ut_main && mpirun -n 3 ut_main -start_in_debugger
> /home/esfp/tools/libraries/petsc/linux-cxx-debug/bin/mpicxx -o ut_main.o
> -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0
> -fPIC -I/home/esfp/tools/libraries/petsc/include
> -I/home/esfp/tools/libraries/petsc/linux-cxx-debug/include
> -I/usr/local/hdf5/include -std=c++11 -g -O3 `pwd`/ut_main.cc
> /home/esfp/tools/libraries/petsc/linux-cxx-debug/bin/mpicxx -Wall
> -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 -I.
> svm.o solver.o model_selection.o ut_ms.o ut_common.o ut_kf.o
> ut_partitioning.o ds_node.o ds_graph.o coarsening.o ut_coarsening.o
> partitioning.o ut_mr.o pugixml.o config_params.o etimer.o common_funcs.o
> OptionParser.o loader.o ut_loader.o k_fold.o ut_main.o
> -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib
> -L/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -lpetsc
> -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib
> -lsuperlu_4.3 -lsuperlu_dist_4.1 -lf2clapack -lf2cblas -lm -lparmetis
> -lmetis -lX11 -Wl,-rpath,/usr/local/hdf5/lib -L/usr/local/hdf5/lib
> -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lhwloc -lm
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
> -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
> -L/lib/x86_64-linux-gnu -lmpi_usempi -lmpi_mpifh -lgfortran -lm -lgfortran
> -lm -lquadmath -lm -lmpi_cxx -lstdc++
> -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib
> -L/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
> -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
> -L/usr/lib/x86_64-linux-gnu -ldl
> -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -lmpi
> -lgcc_s -lpthread -ldl -o ut_main
> /bin/rm -f ut_main.o
> [0]PETSC ERROR: PETSC: Attaching gdb to ut_main of pid 2818 on display :0
> on machine grappelli
> [1]PETSC ERROR: PETSC: Attaching gdb to ut_main of pid 2819 on display :0
> on machine grappelli
> [2]PETSC ERROR: PETSC: Attaching gdb to ut_main of pid 2820 on display :0
> on machine grappelli
>
And I got below error in gdb GUI:
[image: Inline image 1]
I appreciate your support.
Best regards,
Ehsan
On Wed, Jun 29, 2016 at 4:31 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> On all other processes don't pass in 1 pass in 0 since all other
> processes want 0 sub matrices
>
>
> > On Jun 29, 2016, at 3:24 PM, ehsan sadrfaridpour <it.sadr at gmail.com>
> wrote:
> >
> > Thanks, the IS problem is solved.
> > But now I have another problem to compile the code.
> >
> > I use below code:
> > Mat m_WA_nt_local;
> > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points,
> Config_params::getInstance()->get_pre_init_loader_matrix(), NULL,
> &m_WA_nt_local);
> > IS set;
> > if(rank ==0){
> > ISCreateStride(PETSC_COMM_SELF, num_points, 0, 1, &set);
> > ISView(set, PETSC_VIEWER_STDOUT_SELF);
> > }
> > MatGetSubMatrices(m_WA_norm_T, 1, &set, &set, MAT_INITIAL_MATRIX,
> &m_WA_nt_local);
> >
> > The error I get is :
> > error: cannot convert ‘_p_Mat**’ to ‘_p_Mat***’ for argument ‘6’ to
> ‘PetscErrorCode MatGetSubMatrices(Mat, PetscInt, _p_IS* const*, _p_IS*
> const*, MatReuse, _p_Mat***)’
> > MatGetSubMatrices(m_WA_norm_T, 1, &set, &set, MAT_INITIAL_MATRIX,
> &m_WA_nt_local);
> >
> >
> > I tried to go around it by define a array of Matrices using "Mat *
> m_WA_nt_local"
> > So, the first 2 lines changed to below and I can compile the code.
> > Mat * m_WA_nt_local;
> > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points,
> Config_params::getInstance()->get_pre_init_loader_matrix(), NULL,
> m_WA_nt_local);
> >
> >
> >
> > However, I get errors like below when I run the code with 2 mpi process.
> > --------------------- Error Message
> --------------------------------------------------------------
> > [1]PETSC ERROR: Invalid argument
> > [1]PETSC ERROR: Wrong type of object: Parameter # 3
> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown
> > [1]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp Wed
> Jun 29 16:21:04 2016
> > [1]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug
> --with-cc=gcc --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1
> --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1
> --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1
> --download-superlu=1 --download-metis=1 --download-parmetis=1
> --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/
> > [1]PETSC ERROR: #1 MatGetSubMatrices() line 6605 in
> /home/esfp/tools/libraries/petsc/src/mat/interface/matrix.c
> >
> >
> > I think I need to do something for other processes, but I don't know
> what I need to do.
> >
> > Best,
> > Ehsan
> >
> >
> >
> > On Wed, Jun 29, 2016 at 4:03 PM, Dave May <dave.mayhem23 at gmail.com>
> wrote:
> >
> >
> > On Wednesday, 29 June 2016, ehsan sadrfaridpour <it.sadr at gmail.com>
> wrote:
> > I faced the below error during compiling my code for using
> MatGetSubMatrices.
> >
> > error: cannot convert ‘IS {aka _p_IS*}’ to ‘_p_IS* const*’ for argument
> ‘3’ to ‘PetscErrorCode MatGetSubMatrices(Mat, PetscInt, _p_IS* const*,
> _p_IS* const*, MatReuse, _p_Mat***)’
> > MatGetSubMatrices(m_WA_norm_T, 1, set, set, MAT_INITIAL_MATRIX,
> &m_local_W);
> >
> > My code :
> > PetscMPIInt rank;
> > MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
> >
> > if(rank ==0){
> > Mat m_local_W;
> > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points, num_nz,
> NULL,&m_local_W);// try to reserve space for only number of final non zero
> entries for each fine node (e.g. 4)
> > IS set;
> > ISCreateStride(PETSC_COMM_SELF, num_points, 0, 1, &set_row);
> > MatGetSubMatrices(m_WA_norm_T, 1, set_row, set_col,
> MAT_INITIAL_MATRIX, &m_local_W);
> >
> > }
> >
> > I followed below example:
> >
> http://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/is/examples/tutorials/ex2.c.html
> >
> > This code won't work in parallel.
> > The man page says this function is collective on Mat. You need to move
> the call to MatGetSubMatrices outside of the if(rank==0) loop.
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Jun 29, 2016 at 3:19 PM, ehsan sadrfaridpour <it.sadr at gmail.com>
> wrote:
> > Thanks a lot for great support.
> >
> > On Wed, Jun 29, 2016 at 3:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > MatGetSubmatrices() just have the first process request all the rows
> and columns and the others request none. You can use ISCreateStride() to
> create the ISs without having to make an array of all the indices.
> >
> >
> > > On Jun 29, 2016, at 1:43 PM, ehsan sadrfaridpour <it.sadr at gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > I need to have access to most of elements of a parallel MPIAIJ matrix
> only from 1 process (rank 0).
> > > I tried to copy or duplicate it to SEQAIJ, but I faced problems.
> > >
> > > How can I have a local copy of a matrix which is distributed on
> multiple process? I don't want to update the matrix, and the read-only
> version of it would be enough.
> > >
> > > Best,
> > > Ehsan
> > >
> > >
> >
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160705/9b5a4cf7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 3695 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160705/9b5a4cf7/attachment.png>
More information about the petsc-users
mailing list