[petsc-users] How to have a local copy (sequential) of a parallel matrix
ehsan sadrfaridpour
it.sadr at gmail.com
Wed Jul 6 08:12:28 CDT 2016
Thanks all,
Sorry for lots of questions. Thanks to your advice, I didn't create the
local matrix and it seems the problem is solved.
I mean it seems that I shouldn't create the local matrix at all.
And this is my final working code.
Mat *m_WA_nt_local;
IS set;
if(rank ==0){
ISCreateStride(PETSC_COMM_SELF, num_points, 0, 1, &set);
MatGetSubMatrices(m_WA_norm_T, 1, &set, &set, MAT_INITIAL_MATRIX,
&m_WA_nt_local);
}else{
MatGetSubMatrices(m_WA_norm_T, 0, &set, &set, MAT_INITIAL_MATRIX,
&m_WA_nt_local);
}
if(rank ==0){
PetscInt m_WA_nt_local_start, m_WA_nt_local_end;
MatGetOwnershipRange( (*m_WA_nt_local), &m_WA_nt_local_start,
&m_WA_nt_local_end);
PetscPrintf(PETSC_COMM_SELF, "[CS][pCalc_P] rank:%d, m_WA_nt_local
start:%d, end:%d\n", rank, m_WA_nt_local_start,m_WA_nt_local_end);
MatView((*m_WA_nt_local), PETSC_VIEWER_STDOUT_SELF);
}
It compiled and run without any problem.
Best regards,
Ehsan
On Tue, Jul 5, 2016 at 9:02 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Jul 5, 2016 at 5:58 PM, ehsan sadrfaridpour <it.sadr at gmail.com>
> wrote:
>
>> Sorry, I think your suggestion needs something, since it doesn't compile.
>>
>> error: expected initializer before ‘*’ token
>>> Mat m_WA_nt_local *m_WA_nt_local;
>>>
>>
>>
>> Yes, this is the same problem that compiled and worked but it has a bug.
>> I faced this problem and I tried to define the array of Matrices to fix
>> this 4 days ago.
>>
>> However, my first email today is the problem that array of matrices
>> caused me.
>> I get a little confused in the logic.
>>
>> Let me review what is happening:
>> As this method is collective, all the processes needs to run it.
>> Therefore, I need to define a local matrix and create it for all of the
>> processes.
>>
>
> No no no. Each process extracts a SET of SEQUENTIAL matrices. Each proc
> choose how many
> it will extract (could be 0).
>
> Thanks,
>
> Matt
>
>
>> Only for the process I want to have the local matrix, I request a matrix
>> (matrices) and for the rest of them I pass 0 in the MatGetSubMatrices.
>> I am suspicious about creating only 1 matrix for any process, while I
>> expect an array of matrices in the MatGetSubMatrices.
>>
>>
>>
>>
>>
>> On Tue, Jul 5, 2016 at 6:43 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>> > On Jul 5, 2016, at 5:36 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>> >
>>> > On Tue, Jul 5, 2016 at 5:26 PM, ehsan sadrfaridpour <it.sadr at gmail.com>
>>> wrote:
>>> > Thanks for your prompt reply. Using & solve this problem, but then I
>>> have another problem.
>>> >
>>> > Rest of the Code:
>>> > Mat m_WA_nt_local;
>>> > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points,
>>> pre_init_size, NULL, &m_WA_nt_local);
>>> > PetscPrintf(PETSC_COMM_SELF, "[CS][pCalc_P] rank:%d,
>>> num_points:%d, p_init:%d\n", rank, num_points, pre_init_size);
>>> >
>>> > IS set;
>>> > if(rank ==0){
>>> > // - - - - - create local matrix - - - - -
>>> > PetscPrintf(PETSC_COMM_SELF, "[CS][pCalc_P] rank:%d,
>>> num_points:%d\n", rank, num_points);
>>> > ISCreateStride(PETSC_COMM_SELF, num_points, 0, 1, &set);
>>> > ISView(set, PETSC_VIEWER_STDOUT_SELF);
>>> > MatGetSubMatrices(m_WA_norm_T, 1, &set, &set,
>>> MAT_INITIAL_MATRIX, &m_WA_nt_local);
>>> > }else{
>>> > MatGetSubMatrices(m_WA_norm_T, 0, &set, &set,
>>> MAT_INITIAL_MATRIX, &m_WA_nt_local);
>>> > }
>>> >
>>> > This returns an ARRAY of Mat objects, not just one.
>>>
>>> Didn't we just do this email a couple of days ago?
>>>
>>> You need
>>>
>>> Mat m_WA_nt_local *m_WA_nt_local;
>>> > MatGetSubMatrices(m_WA_norm_T, 0, &set, &set, MAT_INITIAL_MATRIX,
>>> &m_WA_nt_local);
>>>
>>>
>>>
>>>
>>>
>>> >
>>> > Matt
>>> >
>>> >
>>> > Error in compile:
>>> > /home/esfp/dev/ws_qt/mlsvm/coarsening.cc: In member function ‘_p_Mat*
>>> Coarsening::pCalc_P(_p_Mat*&, _p_Vec*&, std::vector<long unsigned int>&,
>>> cs_info&)’:
>>> > /home/esfp/dev/ws_qt/mlsvm/coarsening.cc:113:89: error: cannot convert
>>> ‘_p_Mat**’ to ‘_p_Mat***’ for argument ‘6’ to ‘PetscErrorCode
>>> MatGetSubMatrices(Mat, PetscInt, _p_IS* const*, _p_IS* const*, MatReuse,
>>> _p_Mat***)’
>>> > MatGetSubMatrices(m_WA_norm_T, 1, &set, &set,
>>> MAT_INITIAL_MATRIX, &m_WA_nt_local);
>>> >
>>> ^
>>> > /home/esfp/dev/ws_qt/mlsvm/coarsening.cc:115:89: error: cannot convert
>>> ‘_p_Mat**’ to ‘_p_Mat***’ for argument ‘6’ to ‘PetscErrorCode
>>> MatGetSubMatrices(Mat, PetscInt, _p_IS* const*, _p_IS* const*, MatReuse,
>>> _p_Mat***)’
>>> > MatGetSubMatrices(m_WA_norm_T, 0, &set, &set,
>>> MAT_INITIAL_MATRIX, &m_WA_nt_local);
>>> >
>>> ^
>>> >
>>> >
>>> > On Tue, Jul 5, 2016 at 6:21 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> >
>>> > It should be
>>> >
>>> > Mat m_WA_nt_local;
>>> >
>>> > > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points,
>>> pre_init_size, NULL, &m_WA_nt_local);
>>> >
>>> ^^^^^^^^^^^^
>>> note the &
>>> >
>>> >
>>> >
>>> > > On Jul 5, 2016, at 5:13 PM, ehsan sadrfaridpour <it.sadr at gmail.com>
>>> wrote:
>>> > >
>>> > > I faced a problem with my code. The problem is related to
>>> MatCreateSeqAIJ().
>>> > > I comment the rest of my code and just keeping the below lines cause
>>> me the error.
>>> > > Code:
>>> > > Mat * m_WA_nt_local;
>>> > > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points,
>>> pre_init_size, NULL, m_WA_nt_local);
>>> > > PetscPrintf(PETSC_COMM_SELF, "[CS][pCalc_P] rank:%d,
>>> num_points:%d, p_init:%d\n", rank, num_points, pre_init_size);
>>> > >
>>> > > exit(1);
>>> > >
>>> > > Error:
>>> > > [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> > > [0]PETSC ERROR: Null argument, when expecting valid pointer
>>> > > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> > > [1]PETSC ERROR: Null argument, when expecting valid pointer
>>> > > [1]PETSC ERROR: Null Pointer: Parameter # 2
>>> > > [1]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>> shooting.
>>> > > [2]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> > > [2]PETSC ERROR: Null argument, when expecting valid pointer
>>> > > [2]PETSC ERROR: Null Pointer: Parameter # 2
>>> > > [2]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>> shooting.
>>> > > [2]PETSC ERROR: Petsc Release Version 3.6.3, unknown
>>> > > [2]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp
>>> Tue Jul 5 18:05:15 2016
>>> > > [2]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug
>>> --with-cc=gcc --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1
>>> --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1
>>> --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1
>>> --download-superlu=1 --download-metis=1 --download-parmetis=1
>>> --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/
>>> > > [2]PETSC ERROR: #1 MatCreate() line 79 in
>>> /home/esfp/tools/libraries/petsc/src/mat/utils/gcreate.c
>>> > > [2]PETSC ERROR: #2 MatCreateSeqAIJ() line 3471 in
>>> /home/esfp/tools/libraries/petsc/src/mat/impls/aij/seq/aij.c
>>> > > Null Pointer: Parameter # 2
>>> > > [0]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>> shooting.
>>> > > [0]PETSC ERROR: Petsc Release Version 3.6.3, unknown
>>> > > [0]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp
>>> Tue Jul 5 18:05:15 2016
>>> > > [0]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug
>>> --with-cc=gcc --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1
>>> --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1
>>> --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1
>>> --download-superlu=1 --download-metis=1 --download-parmetis=1
>>> --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/
>>> > > [0]PETSC ERROR: #1 MatCreate() line 79 in
>>> /home/esfp/tools/libraries/petsc/src/mat/utils/gcreate.c
>>> > > [0]PETSC ERROR: #2 MatCreateSeqAIJ() line 3471 in
>>> /home/esfp/tools/libraries/petsc/src/mat/impls/aij/seq/aij.c
>>> > > [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown
>>> > > [1]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by esfp
>>> Tue Jul 5 18:05:15 2016
>>> > > [1]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug
>>> --with-cc=gcc --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1
>>> --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1
>>> --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1
>>> --download-superlu=1 --download-metis=1 --download-parmetis=1
>>> --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/
>>> > > [1]PETSC ERROR: #1 MatCreate() line 79 in
>>> /home/esfp/tools/libraries/petsc/src/mat/utils/gcreate.c
>>> > > [1]PETSC ERROR: #2 MatCreateSeqAIJ() line 3471 in
>>> /home/esfp/tools/libraries/petsc/src/mat/impls/aij/seq/aij.c
>>> > > [CS][pCalc_P] rank:1, num_points:10, p_init:300
>>> > > [CS][pCalc_P] rank:2, num_points:10, p_init:300
>>> > > [CS][pCalc_P] rank:0, num_points:10, p_init:300
>>> > >
>>> > > As you can see nothing is NULL in my call to the MatCreateSeqAIJ.
>>> > >
>>> > > I tried to debug it with -start_in_debugger, but I got another error.
>>> > > $ make ut_main && mpirun -n 3 ut_main -start_in_debugger
>>> > > /home/esfp/tools/libraries/petsc/linux-cxx-debug/bin/mpicxx -o
>>> ut_main.o -c -Wall -Wwrite-strings -Wno-strict-aliasing
>>> -Wno-unknown-pragmas -g -O0 -fPIC
>>> -I/home/esfp/tools/libraries/petsc/include
>>> -I/home/esfp/tools/libraries/petsc/linux-cxx-debug/include
>>> -I/usr/local/hdf5/include -std=c++11 -g -O3 `pwd`/ut_main.cc
>>> > > /home/esfp/tools/libraries/petsc/linux-cxx-debug/bin/mpicxx -Wall
>>> -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O0 -I.
>>> svm.o solver.o model_selection.o ut_ms.o ut_common.o ut_kf.o
>>> ut_partitioning.o ds_node.o ds_graph.o coarsening.o ut_coarsening.o
>>> partitioning.o ut_mr.o pugixml.o config_params.o etimer.o common_funcs.o
>>> OptionParser.o loader.o ut_loader.o k_fold.o ut_main.o
>>> -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib
>>> -L/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -lpetsc
>>> -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib
>>> -lsuperlu_4.3 -lsuperlu_dist_4.1 -lf2clapack -lf2cblas -lm -lparmetis
>>> -lmetis -lX11 -Wl,-rpath,/usr/local/hdf5/lib -L/usr/local/hdf5/lib
>>> -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lhwloc -lm
>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
>>> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
>>> -L/lib/x86_64-linux-gnu -lmpi_usempi -lmpi_mpifh -lgfortran -lm -lgfortran
>>> -lm -lquadmath -lm -lmpi_cxx -lstdc++
>>> -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib
>>> -L/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib
>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.8
>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -Wl,-rpath,/usr/lib/x86_64-linux-gnu
>>> -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu
>>> -L/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/x86_64-linux-gnu
>>> -L/usr/lib/x86_64-linux-gnu -ldl
>>> -Wl,-rpath,/home/esfp/tools/libraries/petsc/linux-cxx-debug/lib -lmpi
>>> -lgcc_s -lpthread -ldl -o ut_main
>>> > > /bin/rm -f ut_main.o
>>> > > [0]PETSC ERROR: PETSC: Attaching gdb to ut_main of pid 2818 on
>>> display :0 on machine grappelli
>>> > > [1]PETSC ERROR: PETSC: Attaching gdb to ut_main of pid 2819 on
>>> display :0 on machine grappelli
>>> > > [2]PETSC ERROR: PETSC: Attaching gdb to ut_main of pid 2820 on
>>> display :0 on machine grappelli
>>> > >
>>> > >
>>> > > And I got below error in gdb GUI:
>>> > > <image.png>
>>> > >
>>> > > I appreciate your support.
>>> > >
>>> > > Best regards,
>>> > > Ehsan
>>> > >
>>> > > On Wed, Jun 29, 2016 at 4:31 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> > >
>>> > > On all other processes don't pass in 1 pass in 0 since all other
>>> processes want 0 sub matrices
>>> > >
>>> > >
>>> > > > On Jun 29, 2016, at 3:24 PM, ehsan sadrfaridpour <
>>> it.sadr at gmail.com> wrote:
>>> > > >
>>> > > > Thanks, the IS problem is solved.
>>> > > > But now I have another problem to compile the code.
>>> > > >
>>> > > > I use below code:
>>> > > > Mat m_WA_nt_local;
>>> > > > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points,
>>> Config_params::getInstance()->get_pre_init_loader_matrix(), NULL,
>>> &m_WA_nt_local);
>>> > > > IS set;
>>> > > > if(rank ==0){
>>> > > > ISCreateStride(PETSC_COMM_SELF, num_points, 0, 1, &set);
>>> > > > ISView(set, PETSC_VIEWER_STDOUT_SELF);
>>> > > > }
>>> > > > MatGetSubMatrices(m_WA_norm_T, 1, &set, &set,
>>> MAT_INITIAL_MATRIX, &m_WA_nt_local);
>>> > > >
>>> > > > The error I get is :
>>> > > > error: cannot convert ‘_p_Mat**’ to ‘_p_Mat***’ for argument ‘6’
>>> to ‘PetscErrorCode MatGetSubMatrices(Mat, PetscInt, _p_IS* const*, _p_IS*
>>> const*, MatReuse, _p_Mat***)’
>>> > > > MatGetSubMatrices(m_WA_norm_T, 1, &set, &set,
>>> MAT_INITIAL_MATRIX, &m_WA_nt_local);
>>> > > >
>>> > > >
>>> > > > I tried to go around it by define a array of Matrices using "Mat *
>>> m_WA_nt_local"
>>> > > > So, the first 2 lines changed to below and I can compile the code.
>>> > > > Mat * m_WA_nt_local;
>>> > > > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points,
>>> Config_params::getInstance()->get_pre_init_loader_matrix(), NULL,
>>> m_WA_nt_local);
>>> > > >
>>> > > >
>>> > > >
>>> > > > However, I get errors like below when I run the code with 2 mpi
>>> process.
>>> > > > --------------------- Error Message
>>> --------------------------------------------------------------
>>> > > > [1]PETSC ERROR: Invalid argument
>>> > > > [1]PETSC ERROR: Wrong type of object: Parameter # 3
>>> > > > [1]PETSC ERROR: See
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>> shooting.
>>> > > > [1]PETSC ERROR: Petsc Release Version 3.6.3, unknown
>>> > > > [1]PETSC ERROR: ut_main on a linux-cxx-debug named grappelli by
>>> esfp Wed Jun 29 16:21:04 2016
>>> > > > [1]PETSC ERROR: Configure options PETSC_ARCH=linux-cxx-debug
>>> --with-cc=gcc --with-cxx=g++ --with-clanguage=c++ --with-gnu-compilers=1
>>> --with-mpi-compilers=1 --with-debugging=1 --with-shared-libraries=1
>>> --download-openmpi=1 --download-f2cblaslapack --download-superlu_dist=1
>>> --download-superlu=1 --download-metis=1 --download-parmetis=1
>>> --download-blacs=1 --with-hdf5 --with-hdf5-dir=/usr/local/hdf5/
>>> > > > [1]PETSC ERROR: #1 MatGetSubMatrices() line 6605 in
>>> /home/esfp/tools/libraries/petsc/src/mat/interface/matrix.c
>>> > > >
>>> > > >
>>> > > > I think I need to do something for other processes, but I don't
>>> know what I need to do.
>>> > > >
>>> > > > Best,
>>> > > > Ehsan
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Wed, Jun 29, 2016 at 4:03 PM, Dave May <dave.mayhem23 at gmail.com>
>>> wrote:
>>> > > >
>>> > > >
>>> > > > On Wednesday, 29 June 2016, ehsan sadrfaridpour <it.sadr at gmail.com>
>>> wrote:
>>> > > > I faced the below error during compiling my code for using
>>> MatGetSubMatrices.
>>> > > >
>>> > > > error: cannot convert ‘IS {aka _p_IS*}’ to ‘_p_IS* const*’ for
>>> argument ‘3’ to ‘PetscErrorCode MatGetSubMatrices(Mat, PetscInt, _p_IS*
>>> const*, _p_IS* const*, MatReuse, _p_Mat***)’
>>> > > > MatGetSubMatrices(m_WA_norm_T, 1, set, set,
>>> MAT_INITIAL_MATRIX, &m_local_W);
>>> > > >
>>> > > > My code :
>>> > > > PetscMPIInt rank;
>>> > > > MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
>>> > > >
>>> > > > if(rank ==0){
>>> > > > Mat m_local_W;
>>> > > > MatCreateSeqAIJ(PETSC_COMM_SELF,num_points,num_points,
>>> num_nz, NULL,&m_local_W);// try to reserve space for only number of final
>>> non zero entries for each fine node (e.g. 4)
>>> > > > IS set;
>>> > > > ISCreateStride(PETSC_COMM_SELF, num_points, 0, 1,
>>> &set_row);
>>> > > > MatGetSubMatrices(m_WA_norm_T, 1, set_row, set_col,
>>> MAT_INITIAL_MATRIX, &m_local_W);
>>> > > >
>>> > > > }
>>> > > >
>>> > > > I followed below example:
>>> > > >
>>> http://www.mcs.anl.gov/petsc/petsc-current/src/vec/is/is/examples/tutorials/ex2.c.html
>>> > > >
>>> > > > This code won't work in parallel.
>>> > > > The man page says this function is collective on Mat. You need to
>>> move the call to MatGetSubMatrices outside of the if(rank==0) loop.
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Wed, Jun 29, 2016 at 3:19 PM, ehsan sadrfaridpour <
>>> it.sadr at gmail.com> wrote:
>>> > > > Thanks a lot for great support.
>>> > > >
>>> > > > On Wed, Jun 29, 2016 at 3:11 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> > > >
>>> > > > MatGetSubmatrices() just have the first process request all the
>>> rows and columns and the others request none. You can use ISCreateStride()
>>> to create the ISs without having to make an array of all the indices.
>>> > > >
>>> > > >
>>> > > > > On Jun 29, 2016, at 1:43 PM, ehsan sadrfaridpour <
>>> it.sadr at gmail.com> wrote:
>>> > > > >
>>> > > > > Hi,
>>> > > > >
>>> > > > > I need to have access to most of elements of a parallel MPIAIJ
>>> matrix only from 1 process (rank 0).
>>> > > > > I tried to copy or duplicate it to SEQAIJ, but I faced problems.
>>> > > > >
>>> > > > > How can I have a local copy of a matrix which is distributed on
>>> multiple process? I don't want to update the matrix, and the read-only
>>> version of it would be enough.
>>> > > > >
>>> > > > > Best,
>>> > > > > Ehsan
>>> > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > -- Norbert Wiener
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160706/99e26b3b/attachment-0001.html>
More information about the petsc-users
mailing list