On Tue, Oct 23, 2012 at 6:15 PM, Jinquan Zhong <span dir="ltr"><<a href="mailto:jzhong@scsolutions.com" target="_blank">jzhong@scsolutions.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hong and Jed,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">For KSPSolver, what kind of PC is the most proper ? I tested
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:1.0in;text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas">ierr = PCSetType(pc,PCREDUNDANT);CHKERRQ(ierr);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It worked for small complex double matrix but not for the big ones. Here is my set up</span></p></div>
</div></blockquote><div><br></div><div>Did you read the documentation on this PC? At all? That it solves the entire system on each proces?</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas"> KSP ksp;
<span style="color:green">/* linear solver context */</span><u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas"> PC pc;<u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas"> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);<u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas"> ierr = KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr);<u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas">
<u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas"> ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);<u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas"> ierr = PCSetType(pc,PCREDUNDANT);CHKERRQ(ierr);<u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas"> KSPSetTolerances(ksp,1.e-12,1.e-12,PETSC_DEFAULT,PETSC_DEFAULT);<u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas">
<u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:11.0pt;font-family:Consolas"> ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in;text-autospace:none">
<span style="font-size:11.0pt;font-family:Consolas">ierr = KSPSolve(ksp,b,*x);CHKERRQ(ierr);<u></u><u></u></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:12.5pt;font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Do you see any problem?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Jinquan<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a> [mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a>]
<b>On Behalf Of </b>Hong Zhang<br>
<b>Sent:</b> Tuesday, October 23, 2012 12:03 PM<br>
<b>To:</b> PETSc users list<br>
<b>Subject:</b> Re: [petsc-users] Setting up MUMPS in PETSc<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Jinquan:<u></u><u></u></p>
<div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I have a question on how to use mumps properly in PETSc. It appears that I didn’t set up mumps right.
I followed the example in </span><u></u><u></u></p>
<p class="MsoNormal" style="text-indent:.5in">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><a href="http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html</a></span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This example is for our internal testing, not intended for production runs.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Suggest using PETSc high level KSP solver which provides more flexibility.<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">to set up my program. Here is my situation on using the default setting
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<pre> <a href="http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscInt.html#PetscInt" target="_blank">PetscInt</a> icntl_7 = 5;<u></u><u></u></pre>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">
<a href="http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMumpsSetIcntl.html#MatMumpsSetIcntl" target="_blank">
MatMumpsSetIcntl</a>(F,7,icntl_7);</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">in the example ex125.c:</span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Using KSP sover, the mumps options can be chosen at runtime,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">e.g. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">~petsc/src/ksp/ksp/examples/tutorials/ex2.c:<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_7 5<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Norm of error 1.49777e-15 iterations 1<u></u><u></u></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">1.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The program work fine on all
<b>small</b> models (sparse matrices at the order of m= 894, 1097, 31k with a dense matrix included in the sparse matrix). The residuals are at the magnitude of 10^-3.</span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> ^^^^<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">With a direct solver, residual = <span style="font-size:11.5pt;font-family:"Calibri","sans-serif";color:#1f497d">10^-3 indicates your matrix might be very ill-conditioned or close to singular. What do you get for |R|/|rhs| = ?</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Calibri","sans-serif";color:#1f497d">Is this the reason you want to use a direct solver instead of iterative one?</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Calibri","sans-serif";color:#1f497d">What do you mean "31k with a dense matrix included in the sparse matrix"?</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Calibri","sans-serif";color:#1f497d">How sparse is your matrix, e.g., nnz(A)/(m*m)=?</span><u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">2.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The program has some issues on
<b>medium</b> size problem (m=460k with a dense matrix at the order of n=30k included in the sparse matrix). The full sparse matrix is sized at 17GB.</span><u></u><u></u></p>
<p style="margin-left:1.0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">a.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We used another software to generate sparse matrix by using 144 cores:
</span><u></u><u></u></p>
<p style="margin-left:1.5in"><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">i.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 144 cores (12 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation.</span><u></u><u></u></p>
<p style="margin-left:1.5in"><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">ii.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 432 cores (36 nodes with 48GB/node), it provided the solution. </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Direct solvers are notoriously memory consuming. It seems your matrix is quite dense, requiring more memory than 144 cores could provide. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">What is "<span style="font-size:11.5pt;font-family:"Calibri","sans-serif";color:#1f497d">another software "?</span><u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p style="margin-left:1.0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">b.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We used another software to generate the same sparse matrix by using 576 cores:
</span><u></u><u></u></p>
<p style="margin-left:1.5in"><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">i.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 576 cores (48 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation.</span><u></u><u></u></p>
<p style="margin-left:1.5in"><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">ii.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 1152 cores (96 nodes with 48GB/node), it provided the solution. </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Both a and b seem indicate that, you can use small num of cores to generate original matrix A, but need more cores (resource) to solve A x =b.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This is because A = LU, the factored matrices L and U require far more memory than original A. Run your code using KSP with your matrix data and option -ksp_view<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">e.g., petsc/src/ksp/ksp/examples/tutorials/ex10.c<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">mpiexec -n 2 ./ex10 -f <matrix binary data file> -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">...<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">then you'll see memory info provided by mumps.<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">3.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The program could not solve the
<b>large</b> size problem (m=640k with a dense matrix at the order of n=178k included in the sparse matrix). The full sparse matrix is sized at 511GB.</span><u></u><u></u></p>
<p style="margin-left:1.0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">a.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We used another software to generate sparse matrix by using 900 cores:
</span><u></u><u></u></p>
<p style="margin-left:1.5in"><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">i.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 900 cores (75 nodes with 48GB/node), it could not provide the solution. There was a complain on the memory violation.</span><u></u><u></u></p>
<p style="margin-left:1.5in"><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">ii.</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 2400 cores (200 nodes with 48GB/node), it STILL COULD NOT provide the solution. </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Your computer system and software have limits. Find the answers to your 'medium size' problems first. <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">My confusion starts from the medium size problem:</span><u></u><u></u></p>
<p style="margin-left:41.45pt"><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It seems something was not right in the default setting in ex125.c for these problems.
</span><u></u><u></u></p>
<p style="margin-left:41.45pt"><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I got the info that METIS was used instead of ParMETIS in solving these problems. </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">By default, petsc-mumps interface uses sequential symbolic factorization (analysis phase). Use '-mat_mumps_icntl_28 2' to switch to parallel.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I tested it, but seems parmetis is still not used. Check mumps manual <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">or contact mumps developer on how to use parmetis.<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p style="margin-left:41.45pt"><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Furthermore, it appears that there was unreasonable demand on the solver even on the medium size problem. </span><u></u><u></u></p>
<p style="margin-left:41.45pt"><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I suspect one rank was trying to collect all data from other ranks. </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Yes, analysis is sequential, and rhs vector must be in the host :-( <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In general, direct solvers cannot be scaled to very large num of cores.<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">What other addition setting is needed for mumps such that it could deal with medium and large size
problems?</span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Run your code with option '-help |grep mumps' and experiment with<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">various options, e.g., matrix orderings, nonzero-fills etc. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">You may also try superlu_dist. Good luck!<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Do you guys have similar experience on that?</span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">I personally never used mumps or superlu_dist for such large matrices.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Consult mumps developers.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Hong<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Jinquan</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>