[petsc-dev] VecScatterInitializeForGPU

[Karl Rupp]

Hi guys,

 > I see. I figured there was some logic in place to make sure that this
> function only gets called in cases where the transfer type is
> MPI_General. I'm getting segfaults in this function where the todata and
> fromdata are of a different type. This could easily be user error but
> I'm not sure. Here is an example valgrind error:
>
> ==27781== Invalid read of size 8
> ==27781== at 0x1188080: VecScatterInitializeForGPU (vscatcusp.c:46)
> ==27781== by 0xEEAE5D: MatMult_MPIAIJCUSPARSE(_p_Mat*, _p_Vec*, _p_Vec*)
> (mpiaijcusparse.cu:108)
> ==27781== by 0xA20CC3: MatMult (matrix.c:2242)
> ==27781== by 0x4645E4: main (ex7.c:93)
> ==27781== Address 0x286305e0 is 1,616 bytes inside a block of size 1,620
> alloc'd
> ==27781== at 0x4C26548: memalign (vg_replace_malloc.c:727)
> ==27781== by 0x4654F9: PetscMallocAlign(unsigned long, int, char const*,
> char const*, void**) (mal.c:27)
> ==27781== by 0xCAEECC: PetscTrMallocDefault(unsigned long, int, char
> const*, char const*, void**) (mtr.c:186)
> ==27781== by 0x5A5296: VecScatterCreate (vscat.c:1168)
> ==27781== by 0x9AF3C5: MatSetUpMultiply_MPIAIJ (mmaij.c:116)
> ==27781== by 0x96F0F0: MatAssemblyEnd_MPIAIJ(_p_Mat*, MatAssemblyType)
> (mpiaij.c:706)
> ==27781== by 0xA45358: MatAssemblyEnd (matrix.c:4959)
> ==27781== by 0x464301: main (ex7.c:78)

Okay, so this is another problem related to VecScatter* in addition to 
the ones reported for mpiaijcusp matrices these days. It seems like 
something is fundamentally broken with vector scatters for GPU data 
types right now. :-(

Best regards,
Karli