<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Times New Roman; font-size: 12pt; color: #000000'><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">Thanks for the updates victor. </div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">I got my code working with petsccuda but there is a lot of time spent in copying to/from GPU. I guess i don't understand correctly the mechanism of GPU<->CPU copying so i have some questions on it.</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">1) Do these flags mean the following</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> PETSC_CUDA_CPU = Data on CPU</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> PETSC_CUDA_GPU = Data on GPU</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> PETSC_CUDA_BOTH = Both on CPU and GPU</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> PETSC_CUDA_UNALLOCATED = Data not allocated on GPU.</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">2) Do any of these petsc vector functions do the copying from GPU to CPU, VecGetArray, VecGetArrayRead, VecCopy,VecRestoreArray,any others?</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">3) Do any of these petsc vector functions do the copying from CPU to GPU, VecGetArray, VecGetArrayRead, VecCopy,VecRestoreArray,any others?</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">4) What do the functions VecPlaceArray and VecResetArray do when the data is on CPU/GPU?</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">5) I have a piece of code which does the following (all the vectors here are of type seqcuda)</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecGetArrayRead(X,&x);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecGetArray(F,&f); </div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecPlaceArray(f1,f);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecPlaceArray(x1,x);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecPlaceArray(f2,f+n1);</div><div style="font-family: 'Times New Roman'; font-size: 12pt; color: rgb(0, 0, 0); "> VecPlaceArray(x2,x+n1);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> ....</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecCopy(x1,f1);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> Bunch of VecAXPY to update f1</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> Bunch of MatMult,MatMultAdd to update f2</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> ....</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecResetArray(f1);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecResetArray(x1);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecResetArray(f2);</div><div style="font-family: 'Times New Roman'; font-size: 12pt; color: rgb(0, 0, 0); "> VecResetArray(x2);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecRestoreArrayRead(X,&x);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"> VecRestoreArray(F,&f);</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"><br></div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">Can you please explain what happens for each of these calls in terms of data copying to/from GPU to CPU. Does F get updated on CPU?</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">6) Do VecAXPY and MatMult update the data on CPU?</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">7) Any updates on direct linear solve on GPUs? Anything in CUBLAS? I think CUBLAS provides triangular solves, anything about LU factorization?</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"><br></div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">Thanks,</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000">Shri</div><div style="font-family: Times New Roman; font-size: 12pt; color: #000000"><hr id="zwchr"><blockquote style="border-left:2px solid rgb(16, 16, 255);margin-left:5px;padding-left:5px;">No worries,<div><br></div><div>In that case, the presentation is a little out of date so I'll just mention a few things</div><div><br></div><div>relevant files: <a href="http://aijcuda.cu/mpiaijcuda.cu" target="_blank">aijcuda.cu/mpiaijcuda.cu</a>, <a href="http://veccuda.cu/mpicuda.cu" target="_blank">veccuda.cu/mpicuda.cu</a>, <a href="http://sacuda.cu" target="_blank">sacuda.cu</a>, cudavecimpl.h, cudamatimpl.h</div>
<div><br></div><div>-vectors can be type "cuda" to auto-select seq/mpi depending on number of processes (and matrices "aijcuda" similarly)</div><div><br></div><div>-sacuda preconditioner I believe works now (though not as richardson). As of last time I checked it only worked with symmetric matrices, but the NVIDIA guys were working on that so it's possible the new version of cusp lets it work for all matrices</div>
<div><br></div><div>Those should be the only real changes that affect how you use petsccuda, there are some other changes under the hood (scatter with cuda vecs doesn't do redundant copying, etc).</div><div><br></div>
<div>Let me know if you have questions and hopefully I'll have answers.</div><div><br clear="all">---<br>Victor L. Minden<br><br>Tufts University<br>School of Engineering<br>Class of 2012<br>
<br><br><div class="gmail_quote">On Sun, Dec 19, 2010 at 1:39 PM, Shrirang Abhyankar <span dir="ltr"><<a href="mailto:abhyshr@mcs.anl.gov" target="_blank">abhyshr@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Thanks victor. I am trying to implement a project using petsccuda so I might bug you a little if I have questions.<br>
<br>
Sent from my iPhone<br>
<div><div></div><div class="h5"><br>
On Dec 19, 2010, at 2:13 PM, Victor Minden <<a href="mailto:victorminden@gmail.com" target="_blank">victorminden@gmail.com</a>> wrote:<br>
<br>
> Shri,<br>
><br>
> For sure, attached.<br>
><br>
> Cheers,<br>
><br>
> Victor<br>
> ---<br>
> Victor L. Minden<br>
><br>
> Tufts University<br>
> School of Engineering<br>
> Class of 2012<br>
><br>
><br>
> On Sun, Dec 19, 2010 at 11:14 AM, Shri <<a href="mailto:abhyshr@mcs.anl.gov" target="_blank">abhyshr@mcs.anl.gov</a>> wrote:<br>
><br>
>> Hey Victor,<br>
>> Can you please me a copy of the presentation.<br>
>><br>
>> Thanks,<br>
>> Shri<br>
>><br>
>> ------------------------------<br>
>><br>
>> All,<br>
>><br>
>> If anyone has a mini DisplayPort (MacBook video out) adapter they could<br>
>> bring, I would be greatly indebted.<br>
>><br>
>> -Victor<br>
>> ---<br>
>> Victor L. Minden<br>
>><br>
>> Tufts University<br>
>> School of Engineering<br>
>> Class of 2012<br>
>><br>
>><br>
>> On Tue, Aug 3, 2010 at 9:26 AM, Hong Zhang <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:<br>
>><br>
>>> Reminder: Victor's talk at 1:30pm, Room 1178, today.<br>
>>><br>
>>> Hong<br>
>>><br>
>>> On Tue, Jul 27, 2010 at 11:36 AM, Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
>>>> I've reserved 'Room 1178' [capacity for 15] and a projector for<br>
>>> aug-3/1:30pm.<br>
>>>><br>
>>>> Satish<br>
>>>><br>
>>>> On Tue, 27 Jul 2010, Barry Smith wrote:<br>
>>>><br>
>>>>><br>
>>>>> Ok, let's make it Tuesday August 3rd at 1:30 pm.<br>
>>>>><br>
>>>>> Satish could you please reserve a room and a projector?<br>
>>>>><br>
>>>>> Thanks<br>
>>>>><br>
>>>>> Barry<br>
>>>>><br>
>>>>><br>
>>>>> On Jul 26, 2010, at 2:18 PM, Barry Smith wrote:<br>
>>>>><br>
>>>>>><br>
>>>>>> Victor who has been working on adding GPU CUDA support to PETSc will<br>
>>> be leaving on August 6th. Hong has asked for an overview of what PETSc can<br>
>>> do now with CUDA.<br>
>>>>>><br>
>>>>>> When would be a good time for everyone to attend? I'm thinking<br>
>>> Tuesday August 3rd in afternoon. Please let me know who cannot make it at<br>
>>> that time. Would Monday August 2nd work better for most people?<br>
>>>>>><br>
>>>>>> Barry<br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>><br>
>>>><br>
>>><br>
>><br>
>><br>
>><br>
</div></div>> <minden - anl presentation.pdf><br>
</blockquote></div><br></div>
</blockquote><br></div></div></body></html>