<div dir="ltr">Just an update. This is not deterministic. I am thinking that Permutters MPI might be an issue.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 16, 2021 at 4:47 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I am running snes/ex13 on Perlmutter and doing a scaling study in a script. The first case runs fine:</div><div><br></div><div>+ srun -G 32 -n 256 --cpu-bind=cores --ntasks-per-core=1 /global/homes/m/madams/mps-wrapper.sh ../ex13 -dm_plex_box_faces 4,8,8 -petscpartitioner_simple_node_grid 2,2,2 <b>-dm_refine 3 </b>-dm_mat_type aijcusparse -dm_vec_type cuda -dm_view -ksp_max_it 15 -log_view<br></div><div><br></div><div>but the next case <b>with another levels of refinement</b>:</div><div><br></div><div>+ srun -G 32 -n 256 --cpu-bind=cores --ntasks-per-core=1 /global/homes/m/madams/mps-wrapper.sh ../ex13 -dm_plex_box_faces 4,8,8 -petscpartitioner_simple_node_grid 2,2,2 <b>-dm_refine 4 </b>-dm_mat_type aijcusparse -dm_vec_type cuda -dm_view -ksp_max_it 15 -log_<br></div><div><br></div><div>hangs in BuildTwoSided. With log_trace I see this (grepping on 177). Args appended.</div><div><br></div><div>Any ideas?</div><div>Thanks</div><div><br></div><div><br></div> [177] 21.4441 Event begin: MatGetBrAoCol<br> [177] 21.4441 Event begin: SFSetUp<br> [177] 21.4441 Event begin: BuildTwoSided<br> [177] 21.5443 Event end: BuildTwoSided<br> [177] 21.5443 Event end: SFSetUp<br> [177] 21.5443 Event begin: MatAssemblyBegin<br> [177] 21.5444 Event end: MatAssemblyBegin<br> [177] 21.5444 Event begin: MatAssemblyEnd<br> [177] 21.5444 Event end: MatAssemblyEnd<br> [177] 21.5444 Event end: MatGetBrAoCol<br> [177] 21.5444 Event begin: MatGetLocalMat<br> [177] 21.5569 Event begin: MatCUSPARSCopyTo<br> [177] 21.557 Event end: MatCUSPARSCopyTo<br> [177] 21.557 Event begin: MatCUSPARSCopyTo<br> [177] 21.5571 Event end: MatCUSPARSCopyTo<br> [177] 21.5571 Event end: MatGetLocalMat<br> [177] 21.5571 Event begin: MatCUSPARSCopyTo<br> [177] 21.5571 Event end: MatCUSPARSCopyTo<br> [177] 21.5698 Event begin: MatCUSPARSCopyTo<br> [177] 21.5698 Event end: MatCUSPARSCopyTo<br> [177] 21.5827 Event begin: MatConvert<br> [177] 21.5954 Event end: MatConvert<br> [177] 21.5954 Event begin: MatCUSPARSCopyTo<br> [177] 21.5954 Event end: MatCUSPARSCopyTo<br> [177] 21.5954 Event begin: MatCUSPARSCopyTo<br> [177] 21.5955 Event end: MatCUSPARSCopyTo<br> [177] 21.6208 Event begin: SFSetGraph<br> [177] 21.6208 Event end: SFSetGraph<br> [177] 21.6208 Event begin: SFSetUp<br> [177] 21.6208 Event begin: BuildTwoSided<br><div><br></div><div><br></div><div><br></div><div>#PETSc Option Table entries:<br>-benchmark_it 10<br>-dm_distribute<br>-dm_mat_type aijcusparse<br>-dm_plex_box_faces 4,8,8<br>-dm_plex_box_lower 0,0,0<br>-dm_plex_box_upper 2,4,4<br>-dm_plex_dim 3<br>-dm_plex_simplex 0<br>-dm_refine 3<br>-dm_vec_type cuda<br>-dm_view<br>-ksp_max_it 15<br>-ksp_monitor_short<br>-ksp_norm_type unpreconditioned<br>-ksp_rtol 1.e-12<br>-ksp_type cg<br>-log_view<br>-matptap_via scalable<br>-mg_levels_esteig_ksp_max_it 5<br>-mg_levels_esteig_ksp_type cg<br>-mg_levels_ksp_max_it 2<br>-mg_levels_ksp_type richardson<br>-mg_levels_pc_type jacobi<br>-options_left<br>-pc_gamg_coarse_eq_limit 100<br>-pc_gamg_coarse_grid_layout_type spread<br>-pc_gamg_esteig_ksp_max_it 5<br>-pc_gamg_esteig_ksp_type cg<br>-pc_gamg_process_eq_limit 100<br>-pc_gamg_repartition false<br>-pc_gamg_reuse_interpolation true<br>-pc_gamg_square_graph 1<br>-pc_gamg_threshold 0.01<br>-pc_gamg_threshold_scale .5<br>-pc_type gamg<br>-petscpartitioner_simple_node_grid 2,2,2<br>-petscpartitioner_simple_process_grid 2,4,4<br>-petscpartitioner_type simple<br>-potential_petscspace_degree 2<br>-snes_max_it 1<br>-snes_rtol 1.e-8<br>-snes_type ksponly<br>-use_gpu_aware_mpi 0<br>#End of PETSc Option Table entries<br></div><div>Compiled without FORTRAN kernels<br>Compiled with full precision matrices (default)<br>sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4<br>Configure options: --CFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CXXFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_<br>Q=4" --with-cc=cc --with-cxx=CC --with-fc=ftn --with-cudac=/global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc --FFLAGS=" -g " --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with<br>-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda<br>-----------------------------------------<br>Libraries compiled on 2021-10-16 18:33:45 on login02</div><div> <br></div></div>
</blockquote></div>