<div dir="ltr"><div dir="ltr">On Thu, Oct 19, 2023 at 10:51 AM Enrico <<a href="mailto:degregori@dkrz.de">degregori@dkrz.de</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">In the application the storage is contiguous but the global indexing is <br>
not. I would like to use AO as a translation layer but I don't <br>
understand it.<br></blockquote><div><br></div><div>Why would you choose to index differently from your storage?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
My case is actually simple even if it is in a large application, I have<br>
<br>
Mat A, Vec b and Vec x<br>
<br>
After calling KSPSolve, I use VecGetArrayReadF90 to get a pointer to the <br>
data and they are in the wrong ordering, so for example the first <br>
element of the solution array on process 0 belongs to process 1 in the <br>
application.<br></blockquote><div><br></div><div>Again, this seems to be a poor choice of layout. What we typically do is to partition</div><div>the data into chunks owned by each process first.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Is it at this point that I should use the AO translation layer? This <br>
would be quite bad, it means to build Mat A and Vec b there is MPI <br>
communication and also to get the data of Vec x back in the application.<br></blockquote><div><br></div><div>If you want to store data that process i updates on process j, this will need communication.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Anyway, I've tried to use AOPetscToApplicationPermuteReal on the <br>
solution array but it doesn't work as I would like. Is this function <br>
suppose to do MPI communication between processes and fetch the values <br>
of the application ordering?<br></blockquote><div><br></div><div>There is no communication here. That function call just changes one integer into another.</div><div>If you want to update values on another process, we recommend using VecScatter() or</div><div>MatSetValues(), both of which take global indices and do communication if necessary.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Cheers,<br>
Enrico<br>
<br>
On 19/10/2023 15:25, Matthew Knepley wrote:<br>
> On Thu, Oct 19, 2023 at 8:57 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>> wrote:<br>
> <br>
> Maybe I wasn't clear enough. I would like to completely get rid of<br>
> Petsc<br>
> ordering because I don't want extra communication between processes to<br>
> construct the vector and the matrix (since I have to fill them every<br>
> time step because I'm just using the linear solver with a Mat and a Vec<br>
> data structure). I don't understand how I can do that.<br>
> <br>
> <br>
> Any program you write to do linear algebra will have contiguous storage <br>
> because it<br>
> is so much faster. Contiguous indexing makes sense for contiguous <br>
> storage. If you<br>
> want to use non-contiguous indexing for contiguous storage, you would <br>
> need some<br>
> translation layer. The AO is such a translation, but you could do this <br>
> any way you want.<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> My initial idea was to create another global index ordering within my<br>
> application to use only for the Petsc interface but then I think that<br>
> the ghost cells are wrong.<br>
> <br>
> On 19/10/2023 14:50, Matthew Knepley wrote:<br>
> > On Thu, Oct 19, 2023 at 6:51 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>> wrote:<br>
> ><br>
> > Hello,<br>
> ><br>
> > if I create an application ordering using AOCreateBasic, should I<br>
> > provide the same array for const PetscInt myapp[] and const<br>
> PetscInt<br>
> > mypetsc[] in order to get the same ordering of the application<br>
> > within PETSC?<br>
> ><br>
> ><br>
> > Are you asking if the identity permutation can be constructed<br>
> using the<br>
> > same array twice? Yes.<br>
> ><br>
> > And once I define the ordering so that the local vector and<br>
> matrix are<br>
> > defined in PETSC as in my application, how can I use it to<br>
> create the<br>
> > actual vector and matrix?<br>
> ><br>
> ><br>
> > The vectors and matrices do not change. The AO is a permutation.<br>
> You can<br>
> > use it to permute<br>
> > a vector into another order, or to convert on index to another.<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Matt<br>
> ><br>
> > Thanks in advance for the help.<br>
> ><br>
> > Cheers,<br>
> > Enrico<br>
> ><br>
> > On 18/10/2023 13:39, Matthew Knepley wrote:<br>
> > > On Wed, Oct 18, 2023 at 5:55 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>> wrote:<br>
> > ><br>
> > > Hello,<br>
> > ><br>
> > > I'm trying to use Petsc to solve a linear system in an<br>
> > application. I'm<br>
> > > using the coordinate format to define the matrix and the<br>
> > vector (it<br>
> > > should work better on GPU but at the moment every test<br>
> is on<br>
> > CPU).<br>
> > > After<br>
> > > the call to VecSetValuesCOO, I've noticed that the<br>
> vector is<br>
> > storing<br>
> > > the<br>
> > > data in a different way from my application. For<br>
> example with two<br>
> > > processes in the application<br>
> > ><br>
> > > process 0 owns cells 2, 3, 4<br>
> > ><br>
> > > process 1 owns cells 0, 1, 5<br>
> > ><br>
> > > But in the vector data structure of Petsc<br>
> > ><br>
> > > process 0 owns cells 0, 1, 2<br>
> > ><br>
> > > process 1 owns cells 3, 4, 5<br>
> > ><br>
> > > This is in principle not a big issue, but after<br>
> solving the<br>
> > linear<br>
> > > system I get the solution vector x and I want to get the<br>
> > values in the<br>
> > > correct processes. Is there a way to get vector values<br>
> from other<br>
> > > processes or to get a mapping so that I can do it myself?<br>
> > ><br>
> > ><br>
> > > By definition, PETSc vectors and matrices own contiguous row<br>
> > blocks. If<br>
> > > you want to have another,<br>
> > > global ordering, we support that with<br>
> > > <a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>><br>
> > ><br>
> > > Thanks,<br>
> > ><br>
> > > Matt<br>
> > ><br>
> > > Cheers,<br>
> > > Enrico Degregori<br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > What most experimenters take for granted before they begin<br>
> their<br>
> > > experiments is infinitely more interesting than any<br>
> results to which<br>
> > > their experiments lead.<br>
> > > -- Norbert Wiener<br>
> > ><br>
> > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > What most experimenters take for granted before they begin their<br>
> > experiments is infinitely more interesting than any results to which<br>
> > their experiments lead.<br>
> > -- Norbert Wiener<br>
> ><br>
> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> <br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their <br>
> experiments is infinitely more interesting than any results to which <br>
> their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>