<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Apr 9, 2018 at 7:55 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera-w@sdsu.edu" target="_blank">mvalera-w@sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Apr 9, 2018 at 4:53 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-5148423001550827731gmail-">On Mon, Apr 9, 2018 at 7:52 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera-w@sdsu.edu" target="_blank">mvalera-w@sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Ok thanks, i'm learning more every day,<div><br></div><div>I still get the same error, i am running with -dm_vec_type viennacl -dm_mat_type aijviennacl -pc_type saviennacl </div></div></blockquote><div><br></div></span><div>1) Are you calling DMSetFromOptions()?</div></div></div></div></blockquote><div><br></div><div>Yes, i get:</div></div></div></div></blockquote><div><br></div><div>Let me be more precise. For the DMDA you are using, you must call DMSetFromOptions() before you call</div><div>DMCreateMatrix (or before its called automatically), in order to get the Mat type you want from the command line.</div><div>If you do not get the right type, you know this has not happened.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div>[valera@node50 Src]$ grep DMSetFromOptions *.f90</div><div>DMDAmetrics.f90:call DMSetFromOptions(daDerivs,<wbr>ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daGrid,ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daSingle,<wbr>ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daConstants,<wbr>ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daSgs,ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daLaplacian,<wbr>ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daMetrics,<wbr>ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daCenters,<wbr>ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(<wbr>daPressureCoeffs,ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daDivSgs,<wbr>ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daDummy,ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daVelocities,<wbr>ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daScalars,<wbr>ierr)</div><div>dmdaobjs.f90: call DMSetFromOptions(daDensity,<wbr>ierr)</div><div>fileIO.f90: call DMSetFromOptions(<wbr>daWriteCenters,ierr)</div><div>fileIO.f90: call DMSetFromOptions(daWriteUgrid,<wbr>ierr)</div><div>fileIO.f90: call DMSetFromOptions(daWriteVgrid,<wbr>ierr)</div><div>fileIO.f90: call DMSetFromOptions(daWriteWgrid,<wbr>ierr)</div><div>romsmodule.f90: call DMSetFromOptions(<wbr>daSeqEastWest_u,ierrp)</div><div>romsmodule.f90: call DMSetFromOptions(<wbr>daSeqEastWest_v,ierrp)</div><div>romsmodule.f90: call DMSetFromOptions(<wbr>daSeqEastWest_w,ierrp)</div><div>seamountbeamroms.f90: call DMSetFromOptions(daMinPlaneZ,<wbr>ierr)</div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>2) Does your DM have a prefix?</div></div></div></div></blockquote><div><br></div><div>What does this mean? the one used in the KSPSolve is daDummy</div><div><br></div><div>Thanks,</div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div> Matt</div><div><div class="m_-5148423001550827731gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The error: </div><div><br></div><div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #2 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #5 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div> Finished setting up matrix objects</div><div> Exiting PrepareNetCDF</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #6 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #7 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #8 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #9 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.7096996307373047E-002</div><div> PETSC_Solve_SSG: 1.7321825027465820E-002</div><div> -.-.-.-.-</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #10 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #11 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #12 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #13 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.2702941894531250E-002</div><div> PETSC_Solve_SSG: 1.2929439544677734E-002</div><div> -.-.-.-.-</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #14 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #15 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #16 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #17 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.2219429016113281E-002</div><div> PETSC_Solve_SSG: 1.2480735778808594E-002</div><div> -.-.-.-.-</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #18 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #19 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #20 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #21 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.2894392013549805E-002</div><div> PETSC_Solve_SSG: 1.3124227523803711E-002</div><div> -.-.-.-.-</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #22 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #23 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #24 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #25 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.2183427810668945E-002</div><div> PETSC_Solve_SSG: 1.2410402297973633E-002</div><div> -.-.-.-.-</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #26 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #27 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #28 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #29 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.2546300888061523E-002</div><div> PETSC_Solve_SSG: 1.2778043746948242E-002</div><div> -.-.-.-.-</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #30 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #31 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #32 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #33 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.2255668640136719E-002</div><div> PETSC_Solve_SSG: 1.2504816055297852E-002</div><div> -.-.-.-.-</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #34 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #35 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #36 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #37 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.3138055801391602E-002</div><div> PETSC_Solve_SSG: 1.3362646102905273E-002</div><div> -.-.-.-.-</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #38 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #39 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #40 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #41 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.2072801589965820E-002</div><div> PETSC_Solve_SSG: 1.2319087982177734E-002</div><div> -.-.-.-.-</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.9-13-g05d412b GIT Date: 2018-04-09 08:39:52 -0500</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda-debug named node50 by valera Mon Apr 9 16:50:21 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug --download-mpich --with-blaslapack-dir=/usr/lib<wbr>64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp --download-cusp-commit=116b090</div><div>[0]PETSC ERROR: #42 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #43 PCSetUp() line 923 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #44 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: #45 KSPSolve() line 612 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div> PETSC_Solve_Sol: 1.2498617172241211E-002</div><div> PETSC_Solve_SSG: 1.2726306915283203E-002</div><div> -.-.-.-.-</div><div> -.-.-.-.-.-</div><div> TMain_Loop: 1.1020996570587158 </div><div> -.-.-.-.-.-</div><div>WARNING! There are options you set that were not used!</div><div>WARNING! could be spelling mistake, etc!</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 9, 2018 at 4:45 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Mon, Apr 9, 2018 at 7:27 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera-w@sdsu.edu" target="_blank">mvalera-w@sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Apr 9, 2018 at 4:09 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218m_4346941778822010806m_1832576363965756588gmail-">On Mon, Apr 9, 2018 at 6:12 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera-w@sdsu.edu" target="_blank">mvalera-w@sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello guys,<div><br></div><div>I've made advances in my CUDA acceleration project, as you remember i have a CFD model in need of better execution times. </div><div><br></div><div>So far i have been able to solve the pressure system in the GPU and the rest in serial, using PETSc only for this pressure solve, the library i got to work was ViennaCL. First question, do i still have to switch installations to use either CUDA library? this was a suggestion before, so in order to use CUSP instead of ViennaCL, for example, i currently have to change installations, is this still the case?</div></div></blockquote><div><br></div></span><div>I am not sure what that means exactly. However, you can build a PETSc with CUDA and ViennaCL support. The type of Vec/Mat is selected at runtime.</div></div></div></div></blockquote><div><br></div><div>Karl Rupp wrote in a previous email:</div><div><br></div><div><i><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> * Right now only one of {native CUDA, </span><span class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218m_4346941778822010806m_1832576363965756588gmail-il" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">CUSP</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">, ViennaCL} can be activated at configure time. This will be fixed later this month.</span></i><br></div><div><i><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></i></div><div><span style="font-size:12.8px">I was asking if this was already solved in 3.9, </span></div></div></div></div></blockquote><div><br></div></span><div>Karl knows better than I do. I thought that was fixed, but maybe not in this release.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218m_4346941778822010806m_1832576363965756588gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Now, i started working in a fully parallelized version of the model, which uses the DMs and DMDAs to distribute the arrays, if i try the same flags as before i get an error saying "Currently only handles ViennaCL matrices" when trying to solve for pressure, i get this is a feature still not implemented? what options do i have to solve pressure, or assign a DMDA array update to be done specifically in a GPU device?</div></div></blockquote><div><br></div></span><div>If we can't see the error, we are just guessing. Please send the entire error message.</div></div></div></div></blockquote><div><br></div><div>Got it, I will paste the error at the end of this email</div></div></div></div></blockquote><div><br></div></span><div>It is asking for a ViennaCL matrix. You must tell the DM to create one:</div><div><br></div><div> -dm_mat_type aijviennacl</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Note, we only do linear algebra on the GPU, so none of the FormFunction/FormJacobian stuff for DMDA would be on the GPU.</div></div></div></div></blockquote><div><br></div><div>Yes, we only use it for linear algebra, e.g. solving a linear system and updating an array with a problematic algorithm.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218m_4346941778822010806m_1832576363965756588gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I was thinking of using the VecScatterCreateToZero for a regular vector,</div></div></blockquote><div><br></div></span><div>Why do you want a serial vector?</div></div></div></div></blockquote><div><br></div><div>Because it looks live ViennaCL doesn't handle arrays created with DMDAVec, it was just an idea</div></div></div></div></blockquote><div><br></div></span><div>No, it just needs the right type.</div><span><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218m_4346941778822010806m_1832576363965756588gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div> but then i would have to create a vector and copy the DMDAVec into it,</div></div></blockquote><div><br></div></span><div>I do not understand what it means to copy the DM into the Vec.</div></div></div></div></blockquote><div><br></div><div>I meant copying a DMDAVec into a Vec object, the first is created with a DMDA object for it's mapping across processors,</div></div></div></div></blockquote><div><br></div></span><div>There is no such thing as a DMDAVec. Everything is just a Vec.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><span class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218m_4346941778822010806m_1832576363965756588gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div> is this accomplished with DMDAVecGetArrayReadF90 and then just copy? do you think this will generate too much overhead? </div><div><br></div><div>Thanks so much for your input,</div><div><br></div><div>Manuel </div></div>
</blockquote></span></div><span class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218m_4346941778822010806m_1832576363965756588gmail-HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218m_4346941778822010806m_1832576363965756588gmail-m_545928891280725234gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a></div></div></div></div></div></font></span></div></div></blockquote><div><br></div><div>The error happens when trying to use KSPSolve() for a vector made with DMDAVec routines, the matrix is created without any DMDA routines</div><div><br></div><div>Error:</div><div><br></div><div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 16:24:26 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=no --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp</div><div>[0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #2 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div>[0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ksp<wbr>/interface/itfunc.c</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: No support for this operation for this object type</div><div>[0]PETSC ERROR: Currently only handles ViennaCL matrices</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 16:24:26 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=no --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp</div><div>[0]PETSC ERROR: #4 PCSetUp_SAVIENNACL() line 47 in /home/valera/petsc/src/ksp/pc/<wbr>impls/saviennaclcuda/<a href="http://saviennacl.cu" target="_blank">saviennac<wbr>l.cu</a></div><div>[0]PETSC ERROR: #5 PCSetUp() line 924 in /home/valera/petsc/src/ksp/pc/<wbr>interface/precon.c</div><div> Finished setting up matrix objects</div><div> Exiting PrepareNetCDF</div><div>[0]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------</div><div>[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</div><div>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#valgrind</a></div><div>[0]PETSC ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</div><div>[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run </div><div>[0]PETSC ERROR: to get more information on the crash.</div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--</div><div>[0]PETSC ERROR: Signal received</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Development GIT revision: v3.8.4-2418-gd9c423b GIT Date: 2018-04-02 11:59:41 +0200</div><div>[0]PETSC ERROR: ./gcmBEAM on a cuda named node50 by valera Mon Apr 9 16:24:26 2018</div><div>[0]PETSC ERROR: Configure options PETSC_ARCH=cuda --download-mpich --download-fblaslapack COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --download-hypre --with-debugging=no --with-cuda=1 --CUDAFLAGS=-arch=sm_60 --download-hypre --download-viennacl --download-cusp</div><div>[0]PETSC ERROR: #6 User provided function() line 0 in unknown file</div><div>application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0</div><div>[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div></div><br></div></div>
</blockquote></div></div></div><div><div class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218h5"><br><br clear="all"><span class="HOEnZb"><font color="#888888"><span class="m_-5148423001550827731gmail-m_-5209244255995237640HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_-5148423001550827731gmail-m_-5209244255995237640m_-1125057346576645218m_4346941778822010806gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</font></span></font></span></div></div></div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><div><div class="m_-5148423001550827731gmail-h5"><br><br clear="all"><div><br></div>-- <br><div class="m_-5148423001550827731gmail-m_-5209244255995237640gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</div></div></font></span></div></div>
</blockquote></div><br></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>