[petsc-users] Questions about setting values for GPU based matrices

[Barry Smith]

On Oct 31, 2011, at 11:14 AM, Matthew Knepley wrote:

> On Mon, Oct 31, 2011 at 3:54 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>  Matt,
> 
>    Won't you always want ADD_VALUES support? After all this is how finite element matrix assembly is done. Each element stiffness becomes one block in the batch and then they get added into the global matrix? How would one use the current batch routine with finite elements?
> 
> Of course ADD_VALUES works. What does not work is calling it multiple times on the same matrix, for the reasons I stated.

   Maybe you should change the manual page to make this whole business clear. Currently one could interpret it in a variety of ways.

   For example "Inserts many blocks of values into a matrix at once."   "Inserts" isn't really right

   and "In the future, we may extend this routine to handle rectangular blocks, and additive mode."   What the hey is "additive mode"? I thought ADD_VALUES as opposed to INSERT_VALUES.

    and just say that it cannot be called multiply times before MatAssemblyBegin/End ....



    Barry

> 
>    Matt
>  
> 
>    Barry
> 
> On Oct 31, 2011, at 10:48 AM, Matthew Knepley wrote:
> 
> > On Mon, Oct 31, 2011 at 12:31 PM, Fredrik Heffer Valdmanis <fredva at ifi.uio.no> wrote:
> > 2011/10/30 Matthew Knepley <knepley at gmail.com>
> > On Sun, Oct 30, 2011 at 12:22 PM, Fredrik Heffer Valdmanis <fredva at ifi.uio.no> wrote:
> > 2011/10/28 Matthew Knepley <knepley at gmail.com>
> > On Fri, Oct 28, 2011 at 10:24 AM, Fredrik Heffer Valdmanis <fredva at ifi.uio.no> wrote:
> > Hi,
> >
> > I am working on integrating the new GPU based vectors and matrices into FEniCS. Now, I'm looking at the possibility for getting some speedup during finite element assembly, specifically when inserting the local element matrix into the global element matrix. In that regard, I have a few questions I hope you can help me out with:
> >
> > - When calling MatSetValues with a MATSEQAIJCUSP matrix as parameter, what exactly is it that happens? As far as I can see, MatSetValues is not implemented for GPU based matrices, neither is the mat->ops->setvalues set to point at any function for this Mat type.
> >
> > Yes, MatSetValues always operates on the CPU side. It would not make sense to do individual operations on the GPU.
> >
> > I have written batched of assembly for element matrices that are all the same size:
> >
> >   http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/manualpages/Mat/MatSetValuesBatch.html
> >
> > Thanks. I assume that the best way to use the batch function is to batch up all element matrices and insert all with one function call? Or is it recommended to split it up into several smaller batches?
> >
> > Right now, several batches does not work.For insertion to be efficient, you should keep the matrices in COO
> > format, or convert them back. We do not do either right now. The idea is to see if it ever matters for applications.
> >
> >
> > OK, thanks.
> >
> > Any estimate on when additive mode will be added to MatSetValuesBatch? As it is now, this batch function is of limited use to us, as it forces us to maintain an extra internal data structure to handle accumulation of numbers that are inserted at the same indices in the matrix.
> >
> > I cannot understand what you need this for. All you need is the complete list of element matrices.
> >
> > Any particular reason you chose not to support additive mode in this first implementation? Are there any considerations I should be aware of?
> >
> > I said why above. It would require data structure changes and code support, and we would need some level of user request for that.
> >
> >    Matt
> >
> > Thanks,
> >
> > Fredrik
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener