<div dir="ltr">I basically used 'runex56' and set '-ne' be compatible with np. <div>Then I used option </div><div>'-matptap_via scalable'</div><div>'-matptap_via hypre'</div><div>'-matptap_via nonscalable'</div><div><br></div><div>I attached a job script below.</div><div><br></div><div>In master branch, I set default as 'nonscalable' for small - medium size matrices, and automatically switch to 'scalable' when matrix size gets larger.</div><div><br></div><div>Petsc solver uses MatPtAP, which does local RAP to reduce communication and accelerate computation. </div><div>I suggest you simply use default setting. Let me know if you encounter trouble.<br></div><div><br></div><div>Hong</div><div><br></div><div><a href="http://job.ne174.n8.np125.sh">job.ne174.n8.np125.sh</a>:<br></div><div><div>runjob --np 125 -p 16 --block $COBALT_PARTNAME --verbose=INFO : ./ex56 -ne 174 -alpha 1.e-3 -ksp_type cg -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_reuse_interpolation true -ksp_converged_reason -use_mat_nearnullspace -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -pc_gamg_square_graph 1 -mg_levels_ksp_max_it 1 -mg_levels_ksp_type chebyshev -mg_levels_ksp_chebyshev_esteig 0,0.2,0,1.05 -gamg_est_ksp_type cg -gamg_est_ksp_max_it 10 -pc_gamg_asm_use_agg true -mg_levels_sub_pc_type lu -mg_levels_pc_asm_overlap 0 -pc_gamg_threshold -0.01 -pc_gamg_coarse_eq_limit 200 -pc_gamg_process_eq_limit 30 -pc_gamg_repartition false -pc_mg_cycle_type v -pc_gamg_use_parallel_coarse_grid_solver -mg_coarse_pc_type jacobi -mg_coarse_ksp_type cg -ksp_monitor -log_view -matptap_via scalable > log.ne174.n8.np125.scalable</div><div><br></div><div>runjob --np 125 -p 16 --block $COBALT_PARTNAME --verbose=INFO : ./ex56 -ne 174 -alpha 1.e-3 -ksp_type cg -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_reuse_interpolation true -ksp_converged_reason -use_mat_nearnullspace -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -pc_gamg_square_graph 1 -mg_levels_ksp_max_it 1 -mg_levels_ksp_type chebyshev -mg_levels_ksp_chebyshev_esteig 0,0.2,0,1.05 -gamg_est_ksp_type cg -gamg_est_ksp_max_it 10 -pc_gamg_asm_use_agg true -mg_levels_sub_pc_type lu -mg_levels_pc_asm_overlap 0 -pc_gamg_threshold -0.01 -pc_gamg_coarse_eq_limit 200 -pc_gamg_process_eq_limit 30 -pc_gamg_repartition false -pc_mg_cycle_type v -pc_gamg_use_parallel_coarse_grid_solver -mg_coarse_pc_type jacobi -mg_coarse_ksp_type cg -ksp_monitor -log_view -matptap_via hypre > log.ne174.n8.np125.hypre</div><div><br></div><div>runjob --np 125 -p 16 --block $COBALT_PARTNAME --verbose=INFO : ./ex56 -ne 174 -alpha 1.e-3 -ksp_type cg -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_reuse_interpolation true -ksp_converged_reason -use_mat_nearnullspace -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -pc_gamg_square_graph 1 -mg_levels_ksp_max_it 1 -mg_levels_ksp_type chebyshev -mg_levels_ksp_chebyshev_esteig 0,0.2,0,1.05 -gamg_est_ksp_type cg -gamg_est_ksp_max_it 10 -pc_gamg_asm_use_agg true -mg_levels_sub_pc_type lu -mg_levels_pc_asm_overlap 0 -pc_gamg_threshold -0.01 -pc_gamg_coarse_eq_limit 200 -pc_gamg_process_eq_limit 30 -pc_gamg_repartition false -pc_mg_cycle_type v -pc_gamg_use_parallel_coarse_grid_solver -mg_coarse_pc_type jacobi -mg_coarse_ksp_type cg -ksp_monitor -log_view -matptap_via nonscalable > log.ne174.n8.np125.nonscalable</div><div><br></div><div>runjob --np 125 -p 16 --block $COBALT_PARTNAME --verbose=INFO : ./ex56 -ne 174 -alpha 1.e-3 -ksp_type cg -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_reuse_interpolation true -ksp_converged_reason -use_mat_nearnullspace -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -pc_gamg_square_graph 1 -mg_levels_ksp_max_it 1 -mg_levels_ksp_type chebyshev -mg_levels_ksp_chebyshev_esteig 0,0.2,0,1.05 -gamg_est_ksp_type cg -gamg_est_ksp_max_it 10 -pc_gamg_asm_use_agg true -mg_levels_sub_pc_type lu -mg_levels_pc_asm_overlap 0 -pc_gamg_threshold -0.01 -pc_gamg_coarse_eq_limit 200 -pc_gamg_process_eq_limit 30 -pc_gamg_repartition false -pc_mg_cycle_type v -pc_gamg_use_parallel_coarse_grid_solver -mg_coarse_pc_type jacobi -mg_coarse_ksp_type cg -ksp_monitor -log_view > log.ne174.n8.np125</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 3, 2017 at 2:08 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hong,the input files do not seem to be accessible. What are the command line option? (I don't see a "rap" or "scale" in the source).<div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 3, 2017 at 12:17 PM, Hong <span dir="ltr"><<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Mark,<div>Below is the copy of my email sent to you on Feb 27:</div><div><br></div><div><span style="font-size:12.8px">I implemented scalable MatPtAP and did comparisons of three implementations using </span><span class="m_-7628900733860927440m_-1229466717095455600gmail-il" style="font-size:12.8px;background-color:rgb(255,255,255)">ex56</span><span style="font-size:12.8px">.c on alcf </span><span class="m_-7628900733860927440m_-1229466717095455600gmail-il" style="font-size:12.8px;background-color:rgb(255,255,255)">cetus</span><span style="font-size:12.8px"> machine (this machine has small memory, 1GB/core):</span><div style="font-size:12.8px">- nonscalable PtAP: use an array of length PN to do dense axpy</div><div style="font-size:12.8px">- scalable PtAP: do sparse axpy without use of PN array</div><div style="font-size:12.8px">- hypre PtAP.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">The results are attached. Summary:</div><div style="font-size:12.8px">- nonscalable PtAP is 2x faster than scalable, 8x faster than hypre PtAP</div><div style="font-size:12.8px">- scalable PtAP is 4x faster than hypre PtAP</div><div style="font-size:12.8px">- hypre uses less memory (see <a href="http://job.ne399.n63.np1000.sh/" target="_blank">job.ne399.n63.np1000.sh</a>)</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Based on above observation, I set the default PtAP algorithm as 'nonscalable'. </div><div style="font-size:12.8px">When PN > local estimated nonzero of C=PtAP, then switch default to 'scalable'.</div><div style="font-size:12.8px">User can overwrite default.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">For the case of np=8000, ne=599 (see <a href="http://job.ne599.n500.np8000.sh/" target="_blank">job.ne599.n500.np8000.sh</a>)<wbr>, I get</div><div style="font-size:12.8px">MatPtAP 3.6224e+01 (nonscalable for small mats, scalable for larger ones)<br></div><div style="font-size:12.8px">scalable MatPtAP 4.6129e+01<br></div><div style="font-size:12.8px">hypre 1.9389e+02 </div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">This work in on petsc-master. Give it a try. If you encounter any problem, let me know.</div><span class="m_-7628900733860927440HOEnZb"><font color="#888888"><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Hong</div></font></span></div><div class="m_-7628900733860927440HOEnZb"><div class="m_-7628900733860927440h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 3, 2017 at 10:01 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">(Hong), what is the current state of optimizing RAP for scaling?<div><br></div><div>Nate, is driving 3D elasticity problems at scaling with GAMG and we are working out performance problems. They are hitting problems at ~1.5B dof problems on a basic Cray (XC30 I think).</div><div><br></div><div>Thanks,</div><div>Mark</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>