[petsc-dev] [GPU] src/ksp/ksp/examples/tutorials/ex9 fails with mpirun -np 2

[Karl Rupp]

Hi Pierre,

 > My parallel code is really near to run using GPU device
> thanks to PETSc but I am struggling with the best way
> to fill the PETSc vectors in order in runs on CPU & GPU.
>
> I was using in the past, VecCreateMPIWithArray, to
> create vector but I had trouble with it on GPU.
>
> So I followed the example in the src/ksp/ksp/examples/tutorials/ex2.c
> and create now the vectors like this:
>
>    // Build x
>    ierr = VecCreate(PETSC_COMM_WORLD,&SecondMembrePetsc_); check(ierr);
>    ierr = VecSetSizes(SecondMembrePetsc_, nb_rows, nb_rows_tot);
> check(ierr);
>    ierr = VecSetFromOptions(SecondMembrePetsc_); check(ierr);
>    // Build b
>    ierr = VecDuplicate(SecondMembrePetsc_,&SolutionPetsc_);check(ierr);
>
> And fills it with VecSetValues function. It runs well on CPU and
> GPU but crashed only in parallel on GPU. It I use VecSet instead of
> VecSetValues, it didn't crash (but of course VecSet is not enough
> for me :-)

This sounds like exactly the problem we are now fighting for some time 
now. The reason VecSet() works is that there is no MPI communication 
involved, while with VecSetValues() there is MPI communication involved 
to transfer values to remote ranks. Some parts of these data 
distributions are not fully functional, Dominic just sent a new pull 
request a few days back:
https://bitbucket.org/petsc/petsc/pull-request/138/adding-checks-for-vecscattertype/diff



> I tried to find an example to reproduce for you the problem, and I
> think src/ksp/ksp/examples/tutorials/ex9 (it is usingVecSetValues,)
> is a good one.

Thanks, I'll try to reproduce tomorrow on my CUDA-enabled machine and 
will report back.


> Or did I miss something (I also try VecPlaceArray/VecRestoreArray
> but without success on GPU) ?
>
> Thanks, and yes you are right,  "WARNING: Using GPUs effectively is
> difficult!" :-)

One thing is to get it running - the second is to get good performance 
out of it. particularly in terms of performance/Watt ;-)

Best regards,
Karli