<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 3, 2017 at 1:49 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Well that is strange, the PETSc tests work. <div><br></div><div>Wenbo, could you please:</div><div><br>> git clone -b gamg-fix-eig-err <a href="https://bitbucket.org/petsc/petsc" target="_blank">https://bitbucket.org/petsc/<wbr>petsc</a> petsc2<br>> cd petsc2<br><br>and reconfigure, make, and then run your test without the -st_gamg_est_ksp_error_if_not_<wbr>converged 0 fix, and see if this fixes the problem.<br><br>Don't for<span style="font-size:12.8px">get to set PETSC_DIR=..../petsc2</span><br></div><div><span style="font-size:12.8px"><br></span></div></div></blockquote><div>>git clone -b mark/gamg-fix-eig-err <a href="https://bitbucket.org/petsc/petsc" target="_blank">https://bitbucket.org/petsc/<wbr>petsc</a> petsc2</div><div><br></div><div>It works well without the -st_gamg_est_ksp_error_if_not_<wbr>converged 0.</div><div>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres -st_ksp_view -st_ksp_monitor \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 1 \<br> -mata AMAT.dat -matb BMAT.dat \<br> -st_gamg_est_ksp_view -st_gamg_est_ksp_monitor \<br> -st_gamg_est_ksp_converged_reason \<br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_smooth 2>&1<br></div><div><br></div><div>Thanks,</div><div>Wenbo<br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span style="font-size:12.8px"></span></div><div><span style="font-size:12.8px">If you have time and this works, you could do a 'git checkout master' and remake, and retest. You should not have to reconfigure. I have tested master on petsc tests. I don't understand how this happened.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks,</span></div><div>Mark</div><div><br></div></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 2, 2017 at 12:32 PM, Wenbo Zhao <span dir="ltr"><<a href="mailto:zhaowenbo.npic@gmail.com" target="_blank">zhaowenbo.npic@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-m_8757369113618564603h5">On Tue, Oct 3, 2017 at 12:23 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-h5">On Mon, Oct 2, 2017 at 11:56 AM, Wenbo Zhao <span dir="ltr"><<a href="mailto:zhaowenbo.npic@gmail.com" target="_blank">zhaowenbo.npic@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 2, 2017 at 11:49 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Whenbo, do you build your PETSc?</div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-HOEnZb"><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-h5"><div class="gmail_extra"><br></div></div></div></blockquote><div>Yes.</div><div>My configure option is listed below<br></div><div>./configure --with-mpi=1 --with-shared-libraries=1 \<br> --with-64-bit-indices=1 --with-debugging=1</div><div><br></div><div>And I set PETSC_DIR, PETSC_ARCH and SLEPC_DIR in my ~/.bashrc.</div><div><br></div><div><br></div><div>The Makefile for my problem is listed below,</div><div><br></div><div>PETSC_ARCH = arch-linux2-c-debug<br>PETSC_DIR = /home/zhaowenbo/research/petsc<wbr>/petsc_git<br>SLEPC_DIR = /home/zhaowenbo/research/slepc<wbr>/slepc_git<br>#PETSC_DIR = /home/zhaowenbo/research/petsc<wbr>/petsc-3.7.4<br>#SLEPC_DIR = /home/zhaowenbo/research/slepc<wbr>/slepc-3.7.3<br>HYPRE_DIR = /usr/local/hypre<br># <br>DEBUG_OPT = -g<br>COMP_FLAGS = -fPIC -Wall \<br> -I${SLEPC_DIR}/include -I${SLEPC_DIR}/${PETSC_ARCH}/i<wbr>nclude \<br> -I${PETSC_DIR}/include -I${PETSC_DIR}/${PETSC_ARCH}/i<wbr>nclude \<br> -Isrc<br><br>LINK_FLAGS = -fPIC -Wall \<br> -Wl,-rpath,${SLEPC_DIR}/${PETS<wbr>C_ARCH}/lib -L${SLEPC_DIR}/${PETSC_ARCH}/l<wbr>ib -lslepc \<br> -Wl,-rpath,${PETSC_DIR}/${PETS<wbr>C_ARCH}/lib -L${PETSC_DIR}/${PETSC_ARCH}/l<wbr>ib -lpetsc \<br> -llapack -lblas -lhwloc -lm -lgfortran -lquadmath<br><br>step-41: src/main.o src/readinp.o src/base.o src/sp3.o src/diffu.o <br> mpicxx -o step-41 $^ ${LINK_FLAGS} ${DEBUG_OPT} <br><br>src/main.o: src/main.c <br> mpicxx -o src/main.o -c $^ ${COMP_FLAGS} ${DEBUG_OPT} <br><br>src/readinp.o: src/readinp.c <br> mpicxx -o src/readinp.o -c $^ ${COMP_FLAGS} ${DEBUG_OPT} <br><br>src/sp3.o: src/sp3.c <br> mpicxx -o src/sp3.o -c $^ ${COMP_FLAGS} ${DEBUG_OPT} <br><br>src/diffu.o: src/diffu.c <br> mpicxx -o src/diffu.o -c $^ ${COMP_FLAGS} ${DEBUG_OPT} <br><br>src/base.o: src/base.c<br> mpicxx -o src/base.o -c $^ ${COMP_FLAGS} ${DEBUG_OPT}<br><br><br>clean:<br> rm step-41 src/main.o src/readinp.o src/sp3.o src/diffu.o src/base.o<br><br>runkr_smooth:<br> mpirun -n ${NCORE} ./step-41 \<br> -st_ksp_type gmres -st_ksp_view -st_ksp_monitor \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 1 \<br> -mata AMAT.dat -matb BMAT.dat \<br> -st_gamg_est_ksp_view -st_gamg_est_ksp_monitor \<br> -st_gamg_est_ksp_converged_rea<wbr>son \<br></div></div></div></div></blockquote><div><br></div></div></div><div>Add -st_gamg_est_ksp_error_if_not_<wbr>converged 0<br></div><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-h5"><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div></div></div></div></div></div></blockquote><div><br></div></div></div><div>It works after adding -st_gamg_est_ksp_error_if_not_<wbr>converged 0.</div><div><br></div><div>Thanks,</div><div>Wenbo<br></div><div><div class="gmail-m_8757369113618564603h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_smooth 2>&1<br><br>runkr_nonsmooth:<br> mpirun -n ${NCORE} ./step-41 \<br> -st_ksp_type gmres -st_ksp_view -st_ksp_monitor \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 0 \<br> -mata AMAT.dat -matb BMAT.dat \<br> -st_gamg_est_ksp_view -st_gamg_est_ksp_monitor \<br> -st_gamg_est_ksp_converged_rea<wbr>son \<br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_nonsmooth 2>&1<br></div></div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote">Thanks,</div><div class="gmail_quote">Wenbo<br></div><div class="gmail_quote"><br></div><div class="gmail_quote"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-HOEnZb"><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-h5"><div class="gmail_extra"><div class="gmail_quote">On Mon, Oct 2, 2017 at 11:45 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span style="font-size:12.8px">This is normal:</span></div><div><span style="font-size:12.8px"><br></span></div><div><div><span style="font-size:12.8px">Linear st_gamg_est_ solve did not converge due to DIVERGED_ITS iterations 10</span></div><div style="font-size:12.8px"><br></div></div><span style="font-size:12.8px"><div>It looks like <span style="font-size:12.8px">ksp->errorifnotconverged got set somehow. If the default changed in KSP then (SAGG) GAMG would not ever work.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I assume you don't have a .petscrc file with more (crazy) options in it ... </span></div><div><span style="font-size:12.8px"><br></span></div></span></div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093HOEnZb"><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 2, 2017 at 11:39 AM, Wenbo Zhao <span dir="ltr"><<a href="mailto:zhaowenbo.npic@gmail.com" target="_blank">zhaowenbo.npic@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, Oct 2, 2017 at 11:30 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-">On Mon, Oct 2, 2017 at 11:15 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">non-smoothed aggregation is converging very fast. smoothed fails in the eigen estimator.<div><br></div><div>Run this again with -st_gamg_est_ksp_view and -st_gamg_est_ksp_monitor, and see if you get more output (I'm not 100% sure about these args).</div></div></blockquote><div><br></div></span><div>I also want -st_gamg_est_ksp_converged_rea<wbr>son</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-h5"><div></div></div></div></div></div></div></blockquote></span><div>$make NCORE=1 runkr_smooth <br><span>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres -st_ksp_view -st_ksp_monitor \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 1 \<br> -mata AMAT.dat -matb BMAT.dat \<br></span> -st_gamg_est_ksp_view -st_gamg_est_ksp_monitor \<br> -st_gamg_est_ksp_converged_rea<wbr>son \<span><br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_smooth 2>&1<br>makefile:43: recipe for target 'runkr_smooth' failed<br>make: *** [runkr_smooth] Error 91<br></span></div><div><br></div><div>Thanks</div><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217h5"><div>Wenbo<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 2, 2017 at 11:06 AM, Wenbo Zhao <span dir="ltr"><<a href="mailto:zhaowenbo.npic@gmail.com" target="_blank">zhaowenbo.npic@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Matt,<br><br>Test 1 nonsmooth<span><br>zhaowenbo@ubuntu:~/test_slepc/<wbr>SPARK/spark$ make NCORE=1 runkr_nonsmooth <br>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres -st_ksp_view -st_ksp_monitor \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 0 \<br></span> -mata AMAT.dat -matb BMAT.dat \<span><br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_nonsmooth 2>&1<br><br>Test 2 smooth<br></span><span>zhaowenbo@ubuntu:~/test_slepc/<wbr>SPARK/spark$ make NCORE=1 runkr_smooth <br>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres -st_ksp_view -st_ksp_monitor \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 1 \<br></span> -mata AMAT.dat -matb BMAT.dat \<span><br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_smooth 2>&1<br>makefile:43: recipe for target 'runkr_smooth' failed<br>make: *** [runkr_smooth] Error 91<br><br><br></span></div>Thanks,</div><div><br></div>Wenbo<br></div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397HOEnZb"><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 2, 2017 at 10:48 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000h5">On Mon, Oct 2, 2017 at 10:43 AM, Wenbo Zhao <span dir="ltr"><<a href="mailto:zhaowenbo.npic@gmail.com" target="_blank">zhaowenbo.npic@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Mark,<br></div><br>Thanks for your reply.<br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 2, 2017 at 9:51 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Please send the output with -st_ksp_view and -st_ksp_monitor and we can start to debug it.<div><br></div></div></blockquote><div>Test 1 with nonsmooth and preonly is OK<br>zhaowenbo@ubuntu:~/test_slepc/<wbr>SPARK/spark$ make NCORE=1 runkr_nonsmooth <br>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres -st_ksp_view -st_ksp_monitor \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 0 \<br> -st_ksp_view -mata AMAT.dat -matb BMAT.dat \<br> -st_mg_coarse_ksp_type preonly -st_mg_coarse_ksp_monitor \<br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_nonsmooth 2>&1<br><br>Test 2 smooth and preonly is not OK<br>zhaowenbo@ubuntu:~/test_slepc/<wbr>SPARK/spark$ make NCORE=1 runkr_smooth <br>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres -st_ksp_view -st_ksp_monitor \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 1 \<br> -st_ksp_view -mata AMAT.dat -matb BMAT.dat \<br> -st_mg_coarse_ksp_type preonly -st_mg_coarse_ksp_monitor \<br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_smooth 2>&1<br>makefile:43: recipe for target 'runkr_smooth' failed<br>make: *** [runkr_smooth] Error 91<br><br>Test 3 nonsmooth and gmres is not OK<br>zhaowenbo@ubuntu:~/test_slepc/<wbr>SPARK/spark$ make NCORE=1 runkr_gmres <br>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres -st_ksp_view -st_ksp_monitor \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 0 \<br> -st_ksp_view -mata AMAT.dat -matb BMAT.dat \<br> -st_mg_coarse_ksp_type gmres -st_mg_coarse_ksp_monitor -st_mg_coarse_ksp_rtol 1.0e-6 \<br></div></div></div></div></blockquote><div><br></div></div></div><div>DO NOT DO THIS. Please send the output where you do NOTHING to the coarse solver.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_gmres 2>&1<br>makefile:59: recipe for target 'runkr_gmres' failed<br>make: *** [runkr_gmres] Error 91</div><div><br></div><div>log-files is attached.<br> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>You mentioned that B is not symmetric. I assume it is elliptic (diffusion). Where does the asymmetry come from?</div></div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-HOEnZb"><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-h5"><div class="gmail_extra"><br></div></div></div></blockquote><div><br></div><div>It is a two-group diffusion equations, where group denotes neutron enegry discretisation. <br>Matrix B consists of neutron diffusion/leakage term, removal term and minus neutron scatter source term between different energies, when matrix A denotes neutron fission source.<br><br>Diffusion term(Laplace operator) is elliptic and symmetric. Removal term is diagonal only. However scatter term is asymmetry since scatter term from high energy to low energy is far greater than the term from low to high. <br></div><div><br></div><div><br></div><div>Wenbo<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-HOEnZb"><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-h5"><div class="gmail_extra"><div class="gmail_quote">On Mon, Oct 2, 2017 at 9:39 AM, Wenbo Zhao <span dir="ltr"><<a href="mailto:zhaowenbo.npic@gmail.com" target="_blank">zhaowenbo.npic@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span><div>Matt,<br></div>Thanks for your reply.<br></span><div class="gmail_extra">For the defalt option doesnt work firstly( -st_ksp_type gmres -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 1), I tried to test those options.</div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815HOEnZb"><font color="#888888"><div class="gmail_extra"><br></div><div class="gmail_extra">Wenbo<br></div></font></span><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 2, 2017 at 9:08 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-">On Mon, Oct 2, 2017 at 8:30 AM, Wenbo Zhao <span dir="ltr"><<a href="mailto:zhaowenbo.npic@gmail.com" target="_blank">zhaowenbo.npic@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Matt</div><div><br></div><div>Because I am not clear about what will happen using 'preonly' for large scale problem. <br></div></div></div></blockquote><div><br></div></span><div>The size of the problem has nothing to do with 'preonly'. All it means is to apply a preconditioner without a Krylov solver.</div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>It seems to use a direct solver from below, <br><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPPREONLY.html" target="_blank">http://www.mcs.anl.gov/petsc/p<wbr>etsc-current/docs/manualpages/<wbr>KSP/KSPPREONLY.html</a></div></div></div></blockquote><div><br></div></span><div>However, I still cannot understand why you would change the default?</div><div><br></div><div> Matt</div><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div>Thanks!<br></div>Wenbo<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 2, 2017 at 5:09 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Sun, Oct 1, 2017 at 9:53 PM, Wenbo Zhao <span dir="ltr"><<a href="mailto:zhaowenbo.npic@gmail.com" target="_blank">zhaowenbo.npic@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Matt,<br>Thanks for your reply.<br>It DOES make no sense for this problem.<br>But I am not clear about the 'preonly' option. Which solver is used in preonly? I wonder if 'preonly' is suitable for large scale problem such as 400,000,000 unknowns.<br>So I tried 'gmres' option and found these error messages.<br></div></blockquote><div><br></div></span><div>I mean, why are you setting this at all. Just do not set the coarse solver. The default should work fine.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278m_-3221509347772759455h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Could you give me some suggestions?<br><br>Thanks.<br><br>Wenbo<br><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 2, 2017 at 12:34 AM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Sun, Oct 1, 2017 at 6:49 AM, Wenbo Zhao <span dir="ltr"><<a href="mailto:zhaowenbo.npic@gmail.com" target="_blank">zhaowenbo.npic@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,<br><br>I met some questions when I use PETSC/SLEPC to solve two-group neutron diffusion equations with finite difference method. The grid is 3*3*3, when DOF on each points is 2. So the matrix size is 54*54.<br>It is generalized eigenvalue problem Ax=\lamda Bx, where B is diagonally dominant matrix but not symmetry.<br>EPS is set as below,<br> ierr = EPSSetProblemType(eps,EPS_GNHE<wbr>P);CHKERRQ(ierr);¬<br> ierr = EPSSetWhichEigenpairs(eps,EPS_<wbr>LARGEST_REAL);CHKERRQ(ierr);¬<br><br>Krylovschur is used as eps sovler. GAMG is used as PC.<br>I tried agg_nsmooths and mg_coarse_ksp_type. Only non-smooths and preonly is OK.<br></div></div></blockquote><div><br></div></span><div>Why are you setting the coarse solver. This makes no sense.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278m_-3221509347772759455m_-8305666638070075194m_3153904134462254503h5"><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br>Test 1<br>$ make NCORE=1 runkr_nonsmooth <br>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 0 \<br> -st_ksp_view -mata AMAT.dat -matb BMAT.dat \<br> -st_mg_coarse_ksp_type preonly -st_mg_coarse_ksp_monitor \<br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_nonsmooth 2>&1<br> <br>Test 2<br>$ make NCORE=1 runkr_smooth <br>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 1 \<br> -st_ksp_view -mata AMAT.dat -matb BMAT.dat \<br> -st_mg_coarse_ksp_type preonly -st_mg_coarse_ksp_monitor \<br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_smooth 2>&1<br>makefile:43: recipe for target 'runkr_smooth' failed<br>make: *** [runkr_smooth] Error 91<br><br>Test 3<br>$ make NCORE=1 runkr_gmres <br>mpirun -n 1 ./step-41 \<br> -st_ksp_type gmres \<br> -st_pc_type gamg -st_pc_gamg_type agg -st_pc_gamg_agg_nsmooths 0 \<br> -st_ksp_view -mata AMAT.dat -matb BMAT.dat \<br> -st_mg_coarse_ksp_type gmres -st_mg_coarse_ksp_monitor -st_mg_coarse_ksp_rtol 1.0e-6 \<br> -eps_nev 1 -eps_ncv 10 -eps_monitor -log_view > log_gmres 2>&1<br>makefile:59: recipe for target 'runkr_gmres' failed<br>make: *** [runkr_gmres] Error 91<br><br>Log files were attched.<br>The matrix file were also attched as AMAT.dat and BMAT.dat.<br><br>Is it correct? Or something wrong with my code or commad-line?<br><br>Thanks!<span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278m_-3221509347772759455m_-8305666638070075194m_3153904134462254503m_5044481091402600956HOEnZb"><font color="#888888"><br><br></font></span></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278m_-3221509347772759455m_-8305666638070075194m_3153904134462254503m_5044481091402600956HOEnZb"><font color="#888888">Wenbo<br></font></span></div>
</blockquote></div></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278m_-3221509347772759455m_-8305666638070075194m_3153904134462254503HOEnZb"><font color="#888888"><br><br clear="all"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278HOEnZb"><font color="#888888"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278m_-3221509347772759455m_-8305666638070075194HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278m_-3221509347772759455m_-8305666638070075194m_3153904134462254503m_5044481091402600956gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</font></span></font></span></font></span></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278HOEnZb"><font color="#888888"><br></font></span></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278HOEnZb"><font color="#888888"><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278m_-3221509347772759455h5"><br><br clear="all"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278m_-3221509347772759455m_-8305666638070075194gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</font></span></div></div></font></span></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237HOEnZb"><font color="#888888"><br></font></span></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237HOEnZb"><font color="#888888"><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-h5"><br><br clear="all"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237m_51802801014489142gmail-m_-5744122032765884815m_1885042015433017079gmail-m_-7838439549008716278gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</font></span></div></div></font></span></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237HOEnZb"><font color="#888888">
</font></span></font></span></blockquote></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237HOEnZb"><font color="#888888"><br></font></span></font></span></div></div></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237HOEnZb"><font color="#888888">
</font></span></font></span></blockquote></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237HOEnZb"><font color="#888888"><br></font></span></font></span></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888">
</font></span></div></div></blockquote></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888"><br></font></span></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888"><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000h5"><br><br clear="all"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227m_3012646543920758397m_6396678028311612000m_-4426497621381592237gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</font></span></div></div></font></span></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888">
</font></span></font></span></blockquote></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888"><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227HOEnZb"><font color="#888888"><br></font></span></font></span></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888">
</font></span></div></div></blockquote></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888"><br></font></span></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888"><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-h5"><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-m_-7080410290322438150gmail-m_-464527774853619093m_8111321455483730217m_3280775451942266663gmail-m_-8526603498554076227gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</div></div></font></span></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888"><br></font></span></div></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888"><br></font></span></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888">
</font></span></div></div></blockquote></div><span class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail-HOEnZb"><font color="#888888"><br></font></span></div>
</div></div></blockquote></div><br></div></div>
</blockquote></div></div></div><div><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-h5"><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_8757369113618564603m_-7190124374887765436gmail-m_7039699781864570666gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~k<wbr>nepley/</a><br></div></div></div></div></div>
</div></div></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>