<div dir="ltr"><div><div>Sorry, I think your suggestion needs something, since it doesn't compile.<br><br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div> error: expected initializer before ‘*’ token<br> Mat m_WA_nt_local *m_WA_nt_local;<br></div></blockquote><div> </div><br>Yes, this is the same problem that compiled and worked but it has a bug. <br></div>I faced this problem and I tried to define the array of Matrices to fix this 4 days ago.<br><br></div><div>However, my first email today is the problem that array of matrices caused me. <br></div><div>I get a little confused in the logic. <br></div><div><br>Let me review what is happening:<br></div><div>As this method is collective, all the processes needs to run it. <br>Therefore, I need to define a local matrix and create it for all of the processes.<br></div><div>Only for the process I want to have the local matrix, I request a matrix (matrices) and for the rest of them I pass 0 in the MatGetSubMatrices.<br></div><div>I am suspicious about creating only 1 matrix for any process, while I expect an array of matrices in the MatGetSubMatrices.<br><br><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 5, 2016 at 6:43 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
> On Jul 5, 2016, at 5:36 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br>
><br>
> On Tue, Jul 5, 2016 at 5:26 PM, ehsan sadrfaridpour <<a href="mailto:it.sadr@gmail.com">it.sadr@gmail.com</a>> wrote:<br>
> Thanks for your prompt reply. Using & solve this problem, but then I have another problem.<br>
><br>
> Rest of the Code:<br>
> Mat m_WA_nt_local;<br>
> MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points, pre_init_size, NULL, &m_WA_nt_local);<br>
> PetscPrintf(PETSC_COMM_SELF, "[CS][pCalc_P] rank:%d, num_points:%d, p_init:%d\n", rank, num_points, pre_init_size);<br>
><br>
> IS set;<br>
> if(rank ==0){<br>
> // - - - - - create local matrix - - - - -<br>
> PetscPrintf(PETSC_COMM_SELF, "[CS][pCalc_P] rank:%d, num_points:%d\n", rank, num_points);<br>
> ISCreateStride(PETSC_COMM_SELF, num_points, 0, 1, &set);<br>
> ISView(set, PETSC_VIEWER_STDOUT_SELF);<br>
> MatGetSubMatrices(m_WA_norm_T, 1, &set, &set, MAT_INITIAL_MATRIX, &m_WA_nt_local);<br>
> }else{<br>
> MatGetSubMatrices(m_WA_norm_T, 0, &set, &set, MAT_INITIAL_MATRIX, &m_WA_nt_local);<br>
> }<br>
><br>
> This returns an ARRAY of Mat objects, not just one.<br>
<br>
</span> Didn't we just do this email a couple of days ago?<br>
<br>
You need<br>
<br>
Mat m_WA_nt_local *m_WA_nt_local;<br>
<span class="im HOEnZb">> MatGetSubMatrices(m_WA_norm_T, 0, &set, &set, MAT_INITIAL_MATRIX, &m_WA_nt_local);<br>
<br>
<br>
<br>
<br>
<br>
><br>
</span><div class="HOEnZb"><div class="h5">> Matt<br>
><br>
><br>
> Error in compile:<br>
> /home/esfp/dev/ws_qt/mlsvm/coarsening.cc: In member function ‘_p_Mat* Coarsening::pCalc_P(_p_Mat*&, _p_Vec*&, std::vector<long unsigned int>&, cs_info&)’:<br>
> /home/esfp/dev/ws_qt/mlsvm/coarsening.cc:113:89: error: cannot convert ‘_p_Mat**’ to ‘_p_Mat***’ for argument ‘6’ to ‘PetscErrorCode MatGetSubMatrices(Mat, PetscInt, _p_IS* const*, _p_IS* const*, MatReuse, _p_Mat***)’<br>
> MatGetSubMatrices(m_WA_norm_T, 1, &set, &set, MAT_INITIAL_MATRIX, &m_WA_nt_local);<br>
> ^<br>
> /home/esfp/dev/ws_qt/mlsvm/coarsening.cc:115:89: error: cannot convert ‘_p_Mat**’ to ‘_p_Mat***’ for argument ‘6’ to ‘PetscErrorCode MatGetSubMatrices(Mat, PetscInt, _p_IS* const*, _p_IS* const*, MatReuse, _p_Mat***)’<br>
> MatGetSubMatrices(m_WA_norm_T, 0, &set, &set, MAT_INITIAL_MATRIX, &m_WA_nt_local);<br>
> ^<br>
><br>
><br>
> On Tue, Jul 5, 2016 at 6:21 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> It should be<br>
><br>
> Mat m_WA_nt_local;<br>
><br>
> > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points, pre_init_size, NULL, &m_WA_nt_local);<br>
> ^^^^^^^^^^^^ note the &<br>
><br>
><br>
><br>
> > On Jul 5, 2016, at 5:13 PM, ehsan sadrfaridpour <<a href="mailto:it.sadr@gmail.com">it.sadr@gmail.com</a>> wrote:<br>
> ><br>
> > I faced a problem with my code. The problem is related to MatCreateSeqAIJ().<br>
> > I comment the rest of my code and just keeping the below lines cause me the error.<br>
> > Code:<br>
> > Mat * m_WA_nt_local;<br>
> > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points, pre_init_size, NULL, m_WA_nt_local);<br>
> > PetscPrintf(PETSC_COMM_SELF, "[CS][pCalc_P] rank:%d, num_points:%d, p_init:%d\n", rank, num_points, pre_init_size);<br>
> ><br>
> > exit(1);<br>
> ><br>
> > Error:<br>
> > [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
> > [0]PETSC ERROR: Null argument, when expecting valid pointer<br>
> > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
> > [1]PETSC ERROR: Null argument, when expecting valid pointer<br>
> > [1]PETSC ERROR: Null Pointer: Parameter # 2<br>
> > [1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
> > [2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
> > [2]PETSC ERROR: Null argument, when expecting valid pointer<br>
> > [2]PETSC ERROR: Null Pointer: Parameter # 2<br>
> > [2]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
> > [2]PETSC ERROR: Petsc Release Version 3.6.3, unknown<br>
> > [2]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp Tue Jul 5 18:05:15 2016<br>
> > [2]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug --with-cc=gcc --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1 --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1 --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1 --download-superlu=1 --download-metis=1 --download-parmetis=1 --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/<br>
> > [2]PETSC ERROR: #1 MatCreate() line 79 in /home/esfp/tools/libraries/petsc/src/mat/utils/gcreate.c<br>
> > [2]PETSC ERROR: #2 MatCreateSeqAIJ() line 3471 in /home/esfp/tools/libraries/petsc/src/mat/impls/aij/seq/aij.c<br>
> > Null Pointer: Parameter # 2<br>
> > [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
> > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown<br>
> > [0]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp Tue Jul 5 18:05:15 2016<br>
> > [0]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug --with-cc=gcc --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1 --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1 --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1 --download-superlu=1 --download-metis=1 --download-parmetis=1 --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/<br>
> > [0]PETSC ERROR: #1 MatCreate() line 79 in /home/esfp/tools/libraries/petsc/src/mat/utils/gcreate.c<br>
> > [0]PETSC ERROR: #2 MatCreateSeqAIJ() line 3471 in /home/esfp/tools/libraries/petsc/src/mat/impls/aij/seq/aij.c<br>
> > [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown<br>
> > [1]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp Tue Jul 5 18:05:15 2016<br>
> > [1]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug --with-cc=gcc --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1 --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1 --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1 --download-superlu=1 --download-metis=1 --download-parmetis=1 --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/<br>
> > [1]PETSC ERROR: #1 MatCreate() line 79 in /home/esfp/tools/libraries/petsc/src/mat/utils/gcreate.c<br>
> > [1]PETSC ERROR: #2 MatCreateSeqAIJ() line 3471 in /home/esfp/tools/libraries/petsc/src/mat/impls/aij/seq/aij.c<br>
> > [CS][pCalc_P] rank:1, num_points:10, p_init:300<br>
> > [CS][pCalc_P] rank:2, num_points:10, p_init:300<br>
> > [CS][pCalc_P] rank:0, num_points:10, p_init:300<br>
> ><br>
> > As you can see nothing is NULL in my call to the MatCreateSeqAIJ.<br>
> ><br>
> > I tried to debug it with -start_in_debugger, but I got another error.<br>
> > $ make ut_main && mpirun -n 3 ut_main -start_in_debugger<br>
> > /home/esfp/tools/libraries/petsc/linux-cxx-debug/bin/mpicxx -o ut_main.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 -fPIC -I/home/esfp/tools/libraries/petsc/include -I/home/esfp/tools/libraries/petsc/linux-cxx-debug/include -I/usr/local/hdf5/include -std=c++11 -g -O3 `pwd`/ut_main.cc<br>
> > /home/esfp/tools/libraries/petsc/linux-cxx-debug/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 -I. svm.o solver.o model_selection.o ut_ms.o ut_common.o ut_kf.o ut_partitioning.o ds_node.o ds_graph.o coarsening.o ut_coarsening.o partitioning.o ut_mr.o pugixml.o config_params.o etimer.o common_funcs.o OptionParser.o loader.o ut_loader.o k_fold.o ut_main.o -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -L/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -lpetsc -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -lsuperlu_4.3 -lsuperlu_dist_4.1 -lf2clapack -lf2cblas -lm -lparmetis -lmetis -lX11 -Wl,-rpath,/usr/local/hdf5/lib -L/usr/local/hdf5/lib -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lhwloc -lm -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lmpi_usempi -lmpi_mpifh -lgfortran -lm -lgfortran -lm -lquadmath -lm -lmpi_cxx -lstdc++ -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -L/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -lmpi -lgcc_s -lpthread -ldl -o ut_main<br>
> > /bin/rm -f ut_main.o<br>
> > [0]PETSC ERROR: PETSC: Attaching gdb to ut_main of pid 2818 on display :0 on machine grappelli<br>
> > [1]PETSC ERROR: PETSC: Attaching gdb to ut_main of pid 2819 on display :0 on machine grappelli<br>
> > [2]PETSC ERROR: PETSC: Attaching gdb to ut_main of pid 2820 on display :0 on machine grappelli<br>
> ><br>
> ><br>
> > And I got below error in gdb GUI:<br>
> > <image.png><br>
> ><br>
> > I appreciate your support.<br>
> ><br>
> > Best regards,<br>
> > Ehsan<br>
> ><br>
> > On Wed, Jun 29, 2016 at 4:31 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
> ><br>
> > On all other processes don't pass in 1 pass in 0 since all other processes want 0 sub matrices<br>
> ><br>
> ><br>
> > > On Jun 29, 2016, at 3:24 PM, ehsan sadrfaridpour <<a href="mailto:it.sadr@gmail.com">it.sadr@gmail.com</a>> wrote:<br>
> > ><br>
> > > Thanks, the IS problem is solved.<br>
> > > But now I have another problem to compile the code.<br>
> > ><br>
> > > I use below code:<br>
> > > Mat m_WA_nt_local;<br>
> > > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points, Config_params::getInstance()->get_pre_init_loader_matrix(), NULL, &m_WA_nt_local);<br>
> > > IS set;<br>
> > > if(rank ==0){<br>
> > > ISCreateStride(PETSC_COMM_SELF, num_points, 0, 1, &set);<br>
> > > ISView(set, PETSC_VIEWER_STDOUT_SELF);<br>
> > > }<br>
> > > MatGetSubMatrices(m_WA_norm_T, 1, &set, &set, MAT_INITIAL_MATRIX, &m_WA_nt_local);<br>
> > ><br>
> > > The error I get is :<br>
> > > error: cannot convert ‘_p_Mat**’ to ‘_p_Mat***’ for argument ‘6’ to ‘PetscErrorCode MatGetSubMatrices(Mat, PetscInt, _p_IS* const*, _p_IS* const*, MatReuse, _p_Mat***)’<br>
> > > MatGetSubMatrices(m_WA_norm_T, 1, &set, &set, MAT_INITIAL_MATRIX, &m_WA_nt_local);<br>
> > ><br>
> > ><br>
> > > I tried to go around it by define a array of Matrices using "Mat * m_WA_nt_local"<br>
> > > So, the first 2 lines changed to below and I can compile the code.<br>
> > > Mat * m_WA_nt_local;<br>
> > > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points, Config_params::getInstance()->get_pre_init_loader_matrix(), NULL, m_WA_nt_local);<br>
> > ><br>
> > ><br>
> > ><br>
> > > However, I get errors like below when I run the code with 2 mpi process.<br>
> > > --------------------- Error Message --------------------------------------------------------------<br>
> > > [1]PETSC ERROR: Invalid argument<br>
> > > [1]PETSC ERROR: Wrong type of object: Parameter # 3<br>
> > > [1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
> > > [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown<br>
> > > [1]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp Wed Jun 29 16:21:04 2016<br>
> > > [1]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug --with-cc=gcc --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1 --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1 --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1 --download-superlu=1 --download-metis=1 --download-parmetis=1 --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/<br>
> > > [1]PETSC ERROR: #1 MatGetSubMatrices() line 6605 in /home/esfp/tools/libraries/petsc/src/mat/interface/matrix.c<br>
> > ><br>
> > ><br>
> > > I think I need to do something for other processes, but I don't know what I need to do.<br>
> > ><br>
> > > Best,<br>
> > > Ehsan<br>
> > ><br>
> > ><br>
> > ><br>
> > > On Wed, Jun 29, 2016 at 4:03 PM, Dave May <<a href="mailto:dave.mayhem23@gmail.com">dave.mayhem23@gmail.com</a>> wrote:<br>
> > ><br>
> > ><br>
> > > On Wednesday, 29 June 2016, ehsan sadrfaridpour <<a href="mailto:it.sadr@gmail.com">it.sadr@gmail.com</a>> wrote:<br>
> > > I faced the below error during compiling my code for using MatGetSubMatrices.<br>
> > ><br>
> > > error: cannot convert ‘IS {aka _p_IS*}’ to ‘_p_IS* const*’ for argument ‘3’ to ‘PetscErrorCode MatGetSubMatrices(Mat, PetscInt, _p_IS* const*, _p_IS* const*, MatReuse, _p_Mat***)’<br>
> > > MatGetSubMatrices(m_WA_norm_T, 1, set, set, MAT_INITIAL_MATRIX, &m_local_W);<br>
> > ><br>
> > > My code :<br>
> > > PetscMPIInt rank;<br>
> > > MPI_Comm_rank(PETSC_COMM_WORLD, &rank);<br>
> > ><br>
> > > if(rank ==0){<br>
> > > Mat m_local_W;<br>
> > > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points, num_nz, NULL,&m_local_W);// try to reserve space for only number of final non zero entries for each fine node (e.g. 4)<br>
> > > IS set;<br>
> > > ISCreateStride(PETSC_COMM_SELF, num_points, 0, 1, &set_row);<br>
> > > MatGetSubMatrices(m_WA_norm_T, 1, set_row, set_col, MAT_INITIAL_MATRIX, &m_local_W);<br>
> > ><br>
> > > }<br>
> > ><br>
> > > I followed below example:<br>
> > > <a href="http://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/is/examples/tutorials/ex2.c.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/is/examples/tutorials/ex2.c.html</a><br>
> > ><br>
> > > This code won't work in parallel.<br>
> > > The man page says this function is collective on Mat. You need to move the call to MatGetSubMatrices outside of the if(rank==0) loop.<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On Wed, Jun 29, 2016 at 3:19 PM, ehsan sadrfaridpour <<a href="mailto:it.sadr@gmail.com">it.sadr@gmail.com</a>> wrote:<br>
> > > Thanks a lot for great support.<br>
> > ><br>
> > > On Wed, Jun 29, 2016 at 3:11 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
> > ><br>
> > > MatGetSubmatrices() just have the first process request all the rows and columns and the others request none. You can use ISCreateStride() to create the ISs without having to make an array of all the indices.<br>
> > ><br>
> > ><br>
> > > > On Jun 29, 2016, at 1:43 PM, ehsan sadrfaridpour <<a href="mailto:it.sadr@gmail.com">it.sadr@gmail.com</a>> wrote:<br>
> > > ><br>
> > > > Hi,<br>
> > > ><br>
> > > > I need to have access to most of elements of a parallel MPIAIJ matrix only from 1 process (rank 0).<br>
> > > > I tried to copy or duplicate it to SEQAIJ, but I faced problems.<br>
> > > ><br>
> > > > How can I have a local copy of a matrix which is distributed on multiple process? I don't want to update the matrix, and the read-only version of it would be enough.<br>
> > > ><br>
> > > > Best,<br>
> > > > Ehsan<br>
> > > ><br>
> > > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<br>
<br>
</div></div></blockquote></div><br></div>