[petsc-dev] presentation of cuda on PETSc

[Shri]

Thanks for the updates victor. 
I got my code working with petsccuda but there is a lot of time spent in copying to/from GPU. I guess i don't understand correctly the mechanism of GPU<->CPU copying so i have some questions on it. 
1) Do these flags mean the following 
PETSC_CUDA_CPU = Data on CPU 
PETSC_CUDA_GPU = Data on GPU 
PETSC_CUDA_BOTH = Both on CPU and GPU 
PETSC_CUDA_UNALLOCATED = Data not allocated on GPU. 
2) Do any of these petsc vector functions do the copying from GPU to CPU, VecGetArray, VecGetArrayRead, VecCopy, VecRestoreArray,any others ? 
3) Do any of these petsc vector functions do the copying from CPU to GPU, VecGetArray, VecGetArrayRead, VecCopy,VecRestoreArray,any others ? 
4) What do the functions VecPlaceArray and VecResetArray do when the data is on CPU/GPU? 
5) I have a piece of code which does the following (all the vectors here are of type seqcuda) 
VecGetArrayRead(X,&x); 
VecGetArray(F,&f); 
VecPlaceArray(f1,f); 
VecPlaceArray(x1,x); 
VecPlaceArray(f2,f+n1); 
VecPlaceArray(x2,x+n1); 
.... 
VecCopy(x1,f1); 
Bunch of VecAXPY to update f1 
Bunch of MatMult,MatMultAdd to update f2 
.... 
VecResetArray(f1); 
VecResetArray(x1); 
VecResetArray(f2); 
VecResetArray(x2); 
VecRestoreArrayRead(X,&x); 
VecRestoreArray(F,&f); 


Can you please explain what happens for each of these calls in terms of data copying to/from GPU to CPU. Does F get updated on CPU? 
6) Do VecAXPY and MatMult update the data on CPU? 
7) Any updates on direct linear solve on GPUs? Anything in CUBLAS? I think CUBLAS provides triangular solves, anything about LU factorization? 


Thanks, 
Shri 
----- Original Message -----


No worries, 


In that case, the presentation is a little out of date so I'll just mention a few things 


relevant files: aijcuda.cu/mpiaijcuda.cu , veccuda.cu/mpicuda.cu , sacuda.cu , cudavecimpl.h, cudamatimpl.h 


-vectors can be type "cuda" to auto-select seq/mpi depending on number of processes (and matrices "aijcuda" similarly) 


-sacuda preconditioner I believe works now (though not as richardson). As of last time I checked it only worked with symmetric matrices, but the NVIDIA guys were working on that so it's possible the new version of cusp lets it work for all matrices 


Those should be the only real changes that affect how you use petsccuda, there are some other changes under the hood (scatter with cuda vecs doesn't do redundant copying, etc). 


Let me know if you have questions and hopefully I'll have answers. 

--- 
Victor L. Minden 

Tufts University 
School of Engineering 
Class of 2012 



On Sun, Dec 19, 2010 at 1:39 PM, Shrirang Abhyankar < abhyshr at mcs.anl.gov > wrote: 


Thanks victor. I am trying to implement a project using petsccuda so I might bug you a little if I have questions. 

Sent from my iPhone 




On Dec 19, 2010, at 2:13 PM, Victor Minden < victorminden at gmail.com > wrote: 

> Shri, 
> 
> For sure, attached. 
> 
> Cheers, 
> 
> Victor 
> --- 
> Victor L. Minden 
> 
> Tufts University 
> School of Engineering 
> Class of 2012 
> 
> 
> On Sun, Dec 19, 2010 at 11:14 AM, Shri < abhyshr at mcs.anl.gov > wrote: 
> 
>> Hey Victor, 
>> Can you please me a copy of the presentation. 
>> 
>> Thanks, 
>> Shri 
>> 
>> ------------------------------ 
>> 
>> All, 
>> 
>> If anyone has a mini DisplayPort (MacBook video out) adapter they could 
>> bring, I would be greatly indebted. 
>> 
>> -Victor 
>> --- 
>> Victor L. Minden 
>> 
>> Tufts University 
>> School of Engineering 
>> Class of 2012 
>> 
>> 
>> On Tue, Aug 3, 2010 at 9:26 AM, Hong Zhang < hzhang at mcs.anl.gov > wrote: 
>> 
>>> Reminder: Victor's talk at 1:30pm, Room 1178, today. 
>>> 
>>> Hong 
>>> 
>>> On Tue, Jul 27, 2010 at 11:36 AM, Satish Balay < balay at mcs.anl.gov > wrote: 
>>>> I've reserved 'Room 1178' [capacity for 15] and a projector for 
>>> aug-3/1:30pm. 
>>>> 
>>>> Satish 
>>>> 
>>>> On Tue, 27 Jul 2010, Barry Smith wrote: 
>>>> 
>>>>> 
>>>>> Ok, let's make it Tuesday August 3rd at 1:30 pm. 
>>>>> 
>>>>> Satish could you please reserve a room and a projector? 
>>>>> 
>>>>> Thanks 
>>>>> 
>>>>> Barry 
>>>>> 
>>>>> 
>>>>> On Jul 26, 2010, at 2:18 PM, Barry Smith wrote: 
>>>>> 
>>>>>> 
>>>>>> Victor who has been working on adding GPU CUDA support to PETSc will 
>>> be leaving on August 6th. Hong has asked for an overview of what PETSc can 
>>> do now with CUDA. 
>>>>>> 
>>>>>> When would be a good time for everyone to attend? I'm thinking 
>>> Tuesday August 3rd in afternoon. Please let me know who cannot make it at 
>>> that time. Would Monday August 2nd work better for most people? 
>>>>>> 
>>>>>> Barry 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
> <minden - anl presentation.pdf> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20101220/a6ca4389/attachment.html>