<div dir="ltr"><div>Hi, Chang,</div><div> Do you have the error stack message? And yes, petsc gpu code is tricky, since we have to carefully sync data on GPU and CPU. </div><div> Thanks.</div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 20, 2021 at 11:55 AM Chang Liu <<a href="mailto:cliu@pppl.gov">cliu@pppl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Junchao,<br>
<br>
Thank you for the suggestion. I did some more tests and found that <br>
MatConvert does not always work. In one of my tests, I did MatConvert to <br>
convert the matrix to aijcusparse, then did a preonly ksp solver and it <br>
works well. But then I tried a fgmres solver and it gave an error. It <br>
only happen when the matrix is mpiaijcusparse and for seqaijcusparse it <br>
works.<br>
<br>
So I tried to create a new aijcusparse matrix and copy the data line by <br>
line, then both solvers works. So I guess there are some tricky things <br>
with MatConvert.<br>
<br>
Chang<br>
<br>
On 10/18/21 9:23 PM, Junchao Zhang wrote:<br>
> MatSetOptionsPrefix(A,"mymat")<br>
> VecSetOptionsPrefix(v,"myvec")<br>
> <br>
> --Junchao Zhang<br>
> <br>
> <br>
> On Mon, Oct 18, 2021 at 8:04 PM Chang Liu <<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <br>
> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>> wrote:<br>
> <br>
> Hi Junchao,<br>
> <br>
> Thank you for your answer. I tried MatConvert and it works. I didn't<br>
> make it before because I forgot to convert a vector from mpi to mpicuda<br>
> previously.<br>
> <br>
> For vector, there is no VecConvert to use, so I have to do<br>
> VecDuplicate,<br>
> VecSetType and VecCopy. Is there an easier option?<br>
> <br>
> As Matt suggested, you could single out the matrix and vector with <br>
> options prefix and set their type on command line<br>
> <br>
> MatSetOptionsPrefix(A,"mymat");<br>
> VecSetOptionsPrefix(v,"myvec");<br>
> <br>
> Then, -mymat_mat_type aijcusparse -myvec_vec_type cuda<br>
> A simpler code is to have the vector type automatically set by <br>
> MatCreateVecs(A,&v,NULL)<br>
> <br>
> <br>
> Chang<br>
> <br>
> On 10/18/21 5:23 PM, Junchao Zhang wrote:<br>
> ><br>
> ><br>
> > On Mon, Oct 18, 2021 at 3:42 PM Chang Liu via petsc-users<br>
> > <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>>><br>
> wrote:<br>
> ><br>
> > Hi Matt,<br>
> ><br>
> > I have a related question. In my code I have many matrices<br>
> and I only<br>
> > want to have one living on GPU, the others still staying on<br>
> CPU mem.<br>
> ><br>
> > I wonder if there is an easier way to copy a mpiaij matrix to<br>
> > mpiaijcusparse (in other words, copy data to GPUs). I can<br>
> think of<br>
> > creating a new mpiaijcusparse matrix, and copying the data<br>
> line by<br>
> > line.<br>
> > But I wonder if there is a better option.<br>
> ><br>
> > I have tried MatCopy and MatConvert but neither work.<br>
> ><br>
> > Did you use MatConvert(mat,matype,MAT_INPLACE_MATRIX,&mat)?<br>
> ><br>
> ><br>
> > Chang<br>
> ><br>
> > On 10/17/21 7:50 PM, Matthew Knepley wrote:<br>
> > > On Sun, Oct 17, 2021 at 7:12 PM Swarnava Ghosh<br>
> > <<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>><br>
> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>>><br>
> > > <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>><br>
> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>>>>> wrote:<br>
> > ><br>
> > > Do I need convert the MATSEQBAIJ to a cuda matrix in code?<br>
> > ><br>
> > ><br>
> > > You would need a call to MatSetFromOptions() to take that type<br>
> > from the<br>
> > > command line, and not have<br>
> > > the type hard-coded in your application. It is generally a bad<br>
> > idea to<br>
> > > hard code the implementation type.<br>
> > ><br>
> > > If I do it from command line, then are the other<br>
> MatVec calls are<br>
> > > ported onto CUDA? I have many MatVec calls in my code,<br>
> but I<br>
> > > specifically want to port just one call.<br>
> > ><br>
> > ><br>
> > > You can give that one matrix an options prefix to isolate it.<br>
> > ><br>
> > > Thanks,<br>
> > ><br>
> > > Matt<br>
> > ><br>
> > > Sincerely,<br>
> > > Swarnava<br>
> > ><br>
> > > On Sun, Oct 17, 2021 at 7:07 PM Junchao Zhang<br>
> > > <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a><br>
> <mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> <mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a><br>
> <mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>>><br>
> > <mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a><br>
> <mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> <mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a><br>
> <mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>>>>><br>
> > wrote:<br>
> > ><br>
> > > You can do that with command line options -mat_type<br>
> > aijcusparse<br>
> > > -vec_type cuda<br>
> > ><br>
> > > On Sun, Oct 17, 2021, 5:32 PM Swarnava Ghosh<br>
> > > <<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a><br>
> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a><br>
> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>>><br>
> > <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>><br>
> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a> <mailto:<a href="mailto:swarnava89@gmail.com" target="_blank">swarnava89@gmail.com</a>>>>> wrote:<br>
> > ><br>
> > > Dear Petsc team,<br>
> > ><br>
> > > I had a query regarding using CUDA to<br>
> accelerate a matrix<br>
> > > vector product.<br>
> > > I have a sequential sparse matrix<br>
> (MATSEQBAIJ type).<br>
> > I want<br>
> > > to port a MatVec call onto GPUs. Is there any<br>
> > code/example I<br>
> > > can look at?<br>
> > ><br>
> > > Sincerely,<br>
> > > SG<br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > What most experimenters take for granted before they begin<br>
> their<br>
> > > experiments is infinitely more interesting than any<br>
> results to which<br>
> > > their experiments lead.<br>
> > > -- Norbert Wiener<br>
> > ><br>
> > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> ><br>
> > --<br>
> > Chang Liu<br>
> > Staff Research Physicist<br>
> > +1 609 243 3438<br>
> > <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a><br>
> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>>><br>
> > Princeton Plasma Physics Laboratory<br>
> > 100 Stellarator Rd, Princeton NJ 08540, USA<br>
> ><br>
> <br>
> -- <br>
> Chang Liu<br>
> Staff Research Physicist<br>
> +1 609 243 3438<br>
> <a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a> <mailto:<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a>><br>
> Princeton Plasma Physics Laboratory<br>
> 100 Stellarator Rd, Princeton NJ 08540, USA<br>
> <br>
<br>
-- <br>
Chang Liu<br>
Staff Research Physicist<br>
+1 609 243 3438<br>
<a href="mailto:cliu@pppl.gov" target="_blank">cliu@pppl.gov</a><br>
Princeton Plasma Physics Laboratory<br>
100 Stellarator Rd, Princeton NJ 08540, USA<br>
</blockquote></div></div>