[petsc-users] Help with input construction hang on 2-GPU CG Solve

[Jed Brown]

I ran your code successfully with and without GPU-aware MPI. I see a bit of time in MatSetValue -- you can make it a bit faster using one MatSetValues call per row, but it's typical that assembling a matrix like this (sequentially on the host) will be more expensive than some unpreconditioned CG iterations (that don't come close to solving the problem -- use multigrid if you want to actually solve this problem).

Rohan Yadav <rohany at alumni.cmu.edu> writes:

> Hi,
>
> I'm developing a microbenchmark that runs a CG solve using PETSc on a mesh
> using a 5-point stencil matrix. My code (linked here:
> https://github.com/rohany/petsc-pde-benchmark/blob/main/main.cpp, only 120
> lines) works on 1 GPU and has great performance. When I move to 2 GPUs, the
> program appears to get stuck in the input generation. I've literred the
> code with print statements and have found out the following clues:
>
> * The first rank progresses through this loop:
> https://github.com/rohany/petsc-pde-benchmark/blob/main/main.cpp#L44, but
> then does not exit (it seems to get stuck right before rowStart == rowEnd)
> * The second rank makes very few iterations through the loop for its
> allotted rows.
>
> Therefore, neither rank makes it to the call to MatAssemblyBegin.
>
> I'm running the code using the following command line on the Summit
> supercomputer:
> ```
> jsrun -n 2 -g 1 -c 1 -b rs -r 2
> /gpfs/alpine/scratch/rohany/csc335/petsc-pde-benchmark/main -ksp_max_it 200
> -ksp_type cg -pc_type none -ksp_atol 1e-10 -ksp_rtol 1e-10 -vec_type cuda
> -mat_type aijcusparse -use_gpu_aware_mpi 0 -nx 8485 -ny 8485
> ```
>
> Any suggestions will be appreciated! I feel that I have applied many of the
> common petsc optimizations of preallocating my matrix row counts, so I'm
> not sure what's going on with this input generation.
>
> Thanks,
>
> Rohan Yadav