[petsc-dev] [GPU] Crash on ex19 with mpirun -np 2 (optimized build)

[Karl Rupp]

Hi Pierre,

 > I try running in parallel the ex19 test case on CPU and GPU:
>
> OPT="_opt"
> dir=/ccc/cont002/home/den/triou/Version_test_airain-hybrid/Trio_U/lib/src/LIBPETSC/petsc/linux$OPT/src/snes/examples/tutorials
>
> option="-pc_type none -ksp_type fgmres -snes_monitor_short -snes_rtol
> 1.e-5 -log_summary -ksp_view -cuda_show_devices"
> mpirun -np 2 $dir/ex19 $option 1>cpu$OPT.log 2>&1
> mpirun -np 2 $dir/ex19 -dm_vec_type cusp -dm_mat_type aijcusp $option
> 1>gpu$OPT.log 2>&1
>
> With OPT="", PETSc optimized library is used, parallel calculation runs
> well on CPU and GPU.
> With OPT="_opt", PETSc non optimized library is used, parallel
> calculation crashes on GPU (it is OK on CPU).
>
> I join the log files. The only difference seems that PETSc-dev is built
> with -O3 intead of -g...
> I could try to rebuild PETSc with -O2 but do you have any idea of the
> problem ?

I could reproduce the problem and also get some uninitialized variable 
warnings in Valgrind. The debug version detects these errors, hence you 
only see the errors in the debug build. For the optimized build, chances 
are good that the computed values are either wrong or may become wrong 
in other environments. I'll see what I can do when I'm again at GPU 
machine tomorrow (parallel GPU debugging via SSH is not great...)

Best regards,
Karli