<div dir="ltr">Noel,<div><br></div><div>I am cc'ing the PETSc mailing list. BTW, this question is very normal and, in hindsight, you could have sent us this earlier.</div><div><br></div><div>Plasticity can be very hard, but you can do a lot better than ILU. Your examples so far do not look terrible, but plasticity can get really hard (or stiff to be more precise). First, your couple thermal problems will probably want to use FieldSplit, but let's work on solid mechanics part first.</div><div><br></div><div>Are these symmetric positive definite (SPD, normal finite element discretizations)? I assume they are, or at least are close enough. If they are then you want to use -ksp_type cg. I've appended a decent parameter list to start; this will get tweaked with optimizations, depending on your problems. You need to use AMG and you need to give AMG the null space of the operator. See petsc/src/ksp/ksp/example/tutorial/ex56.c for an example of a 3D elasticity problem. You need to give us coordinates of the vertices basically.</div><div><br></div><div>Mark</div><div><br></div><div><div>-ksp_type cg</div><div>-ksp_rtol 1.e-6</div><div>-pc_type gamg</div><div>-pc_gamg_agg_nsmooths 1 </div><div>-pc_gamg_threshold -0.02</div><div>-gamg_est_ksp_type cg</div><div>#-mg_levels_ksp_type richardson</div><div>-mg_levels_ksp_type chebyshev </div><div>-mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05</div><div>-mg_levels_pc_type sor</div><div>-mg_levels_ksp_max_it 2</div><div>-ksp_converged_reason </div><div>#-ksp_view</div><div>-pc_gamg_repartition true</div><div>-pc_gamg_mat_partitioning_type parmetis</div><div>-pc_gamg_process_eq_limit 200</div><div>#-pc_mg_log</div><div>-ksp_monitor</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 2, 2015 at 7:02 PM, Noel Keen <span dir="ltr"><<a href="mailto:ndkeen@lbl.gov" target="_blank">ndkeen@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Mark,<br>
<br>
I don't mind admitting that I'm not very familiar with the physical problem and the equations,<br>
so I had to ask someone else. I was hoping it would be a simple matter of replacing the serial<br>
Ax=b with a parallel one, but it's probably never that easy for the interesting problems. :)<br>
<br>
>From others (T+M stands for Tough+Mechanics or the Tough fluid code coupled with a geomechanics code [named rocmech])<br>
1. We have both 2D and 3D models for geomechanics (solid model).<br>
2. Plasticity is there. We have several publications for plasticity of T+M. But, regarding the hydrate case, we have not tried yet, although it certainly works.<br>
(I can verify that I see this in the code and we do *not* use it for these inputs I'm trying to work with)<br>
3. We have poroelasticity. Precisely, we have rigorously two-way coupled thermo-poro-elasto-plasticity. T+M is the most advanced simulator in coupled flow and geomechanics. People in this community already know this, I believe.<br>
4. The most incompressible fluid is water (almost cw=4.4e-10 Pa^{-1}). Although civil engineering people approximate water compressibility to zero, the accurate one is TOUGH. Also, if we deal with multiphase flow, flow is certainly compressible. e.g., for oil-water phase, ct(total fluid compressibility)=Soco+Swcw, sum of each saturation times compressibility.<br>
(That is, depending on the EOS, the fluid could be compressible)<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
Noel<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Oct 2, 2015, at 12:52 PM, Mark Adams wrote:<br>
<br>
> Noel,<br>
><br>
> It sounds like you are doing 3D solids. Geomechanics problems use plasticity a lot, which is hard. I don't know what you doing.<br>
><br>
> Are these Stokes problems or are they compressible?<br>
><br>
> Mark<br>
><br>
><br>
> On Fri, Oct 2, 2015 at 12:32 PM, noel keen LBL <<a href="mailto:ndkeen@lbl.gov">ndkeen@lbl.gov</a>> wrote:<br>
><br>
><br>
> Begin forwarded message:<br>
><br>
> > From: noel keen LBL <<a href="mailto:ndkeen@lbl.gov">ndkeen@lbl.gov</a>><br>
> > Date: October 2, 2015 10:31:22 AM PDT<br>
> > To: "Mark F. Adams" <<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>>, noel keen <<a href="mailto:ndkeen@lbl.gov">ndkeen@lbl.gov</a>><br>
> > Subject: KSPsolve<br>
> ><br>
> > Hello Mark,<br>
> ><br>
> > At on point I asked if you could help with PETSc. I was able to figure it out myself following examples, etc.<br>
> > And then sidetracked with other tasks, etc.<br>
> ><br>
> > I've finally reached a point where the bottleneck is the PETSc parallel solve in our code (tough+rocmech <a href="http://tinyurl.com/nskrgf8" rel="noreferrer" target="_blank">http://tinyurl.com/nskrgf8</a>).<br>
> > There are essentially 2 "types" of solves, one is the standard tough "flow" and the other is the newer geomechanical solve (aka the rocmech solve).<br>
> > I'm having trouble with both, but certainly the rocmech solve is the slowest. I created a stand-alone PETSc code (written in F90,<br>
> > unfortunately), that simply reads in a Jacobian (which is output from an actual simulation run) and then solves the system of equations<br>
> > using KSPsolve. I can then solve the system again using their serial solver and compare the performance and results directly.<br>
> ><br>
> > The serial solver has been surprisingly robust and efficient for them <a href="http://tinyurl.com/opgjgw5" rel="noreferrer" target="_blank">http://tinyurl.com/opgjgw5</a>. It uses a version of ILU and CGS.<br>
> > I was convinced (and still am), that once I got PETSc setup properly, I would be able to obtain a large speedup. I can just beat it in<br>
> > serial, but when I add procs, PETSc really struggles. I've tried hundreds if not thousands of different permutations of PC's, PC options,<br>
> > KSP options, and numbers of processors. I _can_ get a speedup, but am just baffled at why the performance (and stability) is not better.<br>
> ><br>
> > Can you offer any help? I can show/give you my small stand-alone code and the data.<br>
> > Could you recommend someone else to help?<br>
> > Should I even expect a parallel solver to significantly outperform an efficient serial solve?<br>
> > Are there any obvious "tricks" the serial solve is doing that would help us in determining the best way to choose parallel solver?<br>
> ><br>
> > Some basic stats and sample of results:<br>
> ><br>
> > nrows= 261144 nnzero= 19879452<br>
> ><br>
> > I have the preallocation done correctly (I know this will destroy parallel perf) . For a parallel run, all procs report:<br>
> > MatGetInfo says nnz= 19879452. nlocal= 21762 nrows= 261144 Mallocs= 0. NZalloc= 19879452. NZunneeded= 0.<br>
> ><br>
> > For this system, the serial solver can reach a tolerance of 1.0e-6 in about 22 seconds on edison. So that's the time to beat.<br>
> ><br>
> > Some of the problems I'm seeing: poor performance, method divergence as I decrease tolerance or increase number of procs,<br>
> > num iterations/performance don't always make sense when changing options/parameters, number of iterations often<br>
> > *increasing* as I add procs (always bothersome).<br>
> ><br>
> > I typically test with a KSP tolerance of 1.0e-4 as I try options. Here are some good results:<br>
> ><br>
> > -ksp_type bcgs -ksp_rtol 1.e-4 -pc_type bjacobi -sub_pc_type ilu<br>
> > kspsolve np= 48 num_iterations= 562 rnorm= 4.0257E-10 reason= 2 wcsolve= 2.56<br>
> > kspsolve np= 24 num_iterations= 470 rnorm= 6.3274E-10 reason= 2 wcsolve= 2.42<br>
> > kspsolve np= 12 num_iterations= 238 rnorm= 7.4453E-10 reason= 2 wcsolve= 2.08<br>
> ><br>
> > -ksp_view ascii -ksp_type bcgs -ksp_rtol 1.e-4 -pc_type hypre -pc_hypre_type pilut -pc_hypre_pilut_maxiter 200 -pc_hypre_pilut_tol 1.0e-4<br>
> > kspsolve np= 96 num_iterations= 1053 rnorm= 2.7155E-10 reason= 2 wcsolve= 15.16<br>
> > kspsolve np= 48 num_iterations= 913 rnorm= 2.8016E-10 reason= 2 wcsolve= 1.64<br>
> > kspsolve np= 24 num_iterations= 794 rnorm= 2.7149E-10 reason= 2 wcsolve= 2.75<br>
> > kspsolve np= 4 num_iterations= 2375 rnorm= 3.0389E-10 reason= 2 wcsolve= 38.19<br>
> ><br>
> ><br>
> > But then going to 1.0e-6 KSP tolerance (a fair comparison to serial DLUST solver) I have trouble:<br>
> ><br>
> > -ksp_view ascii -ksp_type bcgs -ksp_rtol 1.e-6 -pc_type bjacobi -sub_pc_type ilu<br>
> > kspsolve np= 48 num_iterations= 2179 rnorm= 3.0013E-12 reason= 2 wcsolve= 9.26<br>
> > kspsolve np= 24 num_iterations= 1100 rnorm= 2.2589E-10 reason= -5 wcsolve= 5.63 <-- diverges<br>
> > kspsolve np= 12 num_iterations= 759 rnorm= 4.9629E-07 reason= -5 wcsolve= 6.38 <-- diverges<br>
> ><br>
> > -ksp_view ascii -ksp_type bcgs -ksp_rtol 1.e-6 -pc_type hypre -pc_hypre_type pilut -pc_hypre_pilut_maxitr 1000 -pc_hypre_pilut_tol 1.0e-10<br>
> > kspsolve np= 48 num_iterations= 7579 rnorm= 5.7098E-12 reason= 2 wcsolve= 43.81 <-- slow, lots of iterations<br>
> ><br>
> ><br>
> ><br>
> > Thanks<br>
> ><br>
> > Noel<br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div>