<div dir="ltr"><div dir="ltr">On Thu, Oct 19, 2023 at 11:33 AM Enrico <<a href="mailto:degregori@dkrz.de">degregori@dkrz.de</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The layout is not poor, just the global indices are not contiguous,this <br>
has nothing to do with the local memory layout which is extremely <br>
optimized for different architectures. I can not change the layout <br>
anyway because it's a climate model with a million lines of code.<br>
<br>
I don't understand why Petsc is doing all this MPI communication under <br>
the hood. </blockquote><div><br></div><div>I don't think we are communicating under the hood.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I mean, it is changing the layout of the application and doing <br>
a lot of communication.</blockquote><div><br></div><div>We do not create the layout. The user creates the data layout when they create a vector or matrix.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Is there no way to force the same layout and <br>
provide info about how to do the halo exchange? In this way I can have <br>
the same memory layout and there is no communication when I fill or <br>
fetch the vectors and the matrix.<br></blockquote><div><br></div><div>Yes, you tell the vector/matrix your data layout when you create it.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Cheers,<br>
Enrico<br>
<br>
On 19/10/2023 17:21, Matthew Knepley wrote:<br>
> On Thu, Oct 19, 2023 at 10:51 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>> wrote:<br>
> <br>
> In the application the storage is contiguous but the global indexing is<br>
> not. I would like to use AO as a translation layer but I don't<br>
> understand it.<br>
> <br>
> <br>
> Why would you choose to index differently from your storage?<br>
> <br>
> My case is actually simple even if it is in a large application, I have<br>
> <br>
> Mat A, Vec b and Vec x<br>
> <br>
> After calling KSPSolve, I use VecGetArrayReadF90 to get a pointer to<br>
> the<br>
> data and they are in the wrong ordering, so for example the first<br>
> element of the solution array on process 0 belongs to process 1 in the<br>
> application.<br>
> <br>
> <br>
> Again, this seems to be a poor choice of layout. What we typically do is <br>
> to partition<br>
> the data into chunks owned by each process first.<br>
> <br>
> Is it at this point that I should use the AO translation layer? This<br>
> would be quite bad, it means to build Mat A and Vec b there is MPI<br>
> communication and also to get the data of Vec x back in the application.<br>
> <br>
> <br>
> If you want to store data that process i updates on process j, this will <br>
> need communication.<br>
> <br>
> Anyway, I've tried to use AOPetscToApplicationPermuteReal on the<br>
> solution array but it doesn't work as I would like. Is this function<br>
> suppose to do MPI communication between processes and fetch the values<br>
> of the application ordering?<br>
> <br>
> <br>
> There is no communication here. That function call just changes one <br>
> integer into another.<br>
> If you want to update values on another process, we recommend using <br>
> VecScatter() or<br>
> MatSetValues(), both of which take global indices and do communication <br>
> if necessary.<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> Cheers,<br>
> Enrico<br>
> <br>
> On 19/10/2023 15:25, Matthew Knepley wrote:<br>
> > On Thu, Oct 19, 2023 at 8:57 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>> wrote:<br>
> ><br>
> > Maybe I wasn't clear enough. I would like to completely get<br>
> rid of<br>
> > Petsc<br>
> > ordering because I don't want extra communication between<br>
> processes to<br>
> > construct the vector and the matrix (since I have to fill<br>
> them every<br>
> > time step because I'm just using the linear solver with a Mat<br>
> and a Vec<br>
> > data structure). I don't understand how I can do that.<br>
> ><br>
> ><br>
> > Any program you write to do linear algebra will have contiguous<br>
> storage<br>
> > because it<br>
> > is so much faster. Contiguous indexing makes sense for contiguous<br>
> > storage. If you<br>
> > want to use non-contiguous indexing for contiguous storage, you<br>
> would<br>
> > need some<br>
> > translation layer. The AO is such a translation, but you could do<br>
> this<br>
> > any way you want.<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Matt<br>
> ><br>
> > My initial idea was to create another global index ordering<br>
> within my<br>
> > application to use only for the Petsc interface but then I<br>
> think that<br>
> > the ghost cells are wrong.<br>
> ><br>
> > On 19/10/2023 14:50, Matthew Knepley wrote:<br>
> > > On Thu, Oct 19, 2023 at 6:51 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>> wrote:<br>
> > ><br>
> > > Hello,<br>
> > ><br>
> > > if I create an application ordering using<br>
> AOCreateBasic, should I<br>
> > > provide the same array for const PetscInt myapp[] and<br>
> const<br>
> > PetscInt<br>
> > > mypetsc[] in order to get the same ordering of the<br>
> application<br>
> > > within PETSC?<br>
> > ><br>
> > ><br>
> > > Are you asking if the identity permutation can be constructed<br>
> > using the<br>
> > > same array twice? Yes.<br>
> > ><br>
> > > And once I define the ordering so that the local<br>
> vector and<br>
> > matrix are<br>
> > > defined in PETSC as in my application, how can I use it to<br>
> > create the<br>
> > > actual vector and matrix?<br>
> > ><br>
> > ><br>
> > > The vectors and matrices do not change. The AO is a<br>
> permutation.<br>
> > You can<br>
> > > use it to permute<br>
> > > a vector into another order, or to convert on index to<br>
> another.<br>
> > ><br>
> > > Thanks,<br>
> > ><br>
> > > Matt<br>
> > ><br>
> > > Thanks in advance for the help.<br>
> > ><br>
> > > Cheers,<br>
> > > Enrico<br>
> > ><br>
> > > On 18/10/2023 13:39, Matthew Knepley wrote:<br>
> > > > On Wed, Oct 18, 2023 at 5:55 AM Enrico<br>
> <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>>> wrote:<br>
> > > ><br>
> > > > Hello,<br>
> > > ><br>
> > > > I'm trying to use Petsc to solve a linear<br>
> system in an<br>
> > > application. I'm<br>
> > > > using the coordinate format to define the<br>
> matrix and the<br>
> > > vector (it<br>
> > > > should work better on GPU but at the moment<br>
> every test<br>
> > is on<br>
> > > CPU).<br>
> > > > After<br>
> > > > the call to VecSetValuesCOO, I've noticed that the<br>
> > vector is<br>
> > > storing<br>
> > > > the<br>
> > > > data in a different way from my application. For<br>
> > example with two<br>
> > > > processes in the application<br>
> > > ><br>
> > > > process 0 owns cells 2, 3, 4<br>
> > > ><br>
> > > > process 1 owns cells 0, 1, 5<br>
> > > ><br>
> > > > But in the vector data structure of Petsc<br>
> > > ><br>
> > > > process 0 owns cells 0, 1, 2<br>
> > > ><br>
> > > > process 1 owns cells 3, 4, 5<br>
> > > ><br>
> > > > This is in principle not a big issue, but after<br>
> > solving the<br>
> > > linear<br>
> > > > system I get the solution vector x and I want<br>
> to get the<br>
> > > values in the<br>
> > > > correct processes. Is there a way to get vector<br>
> values<br>
> > from other<br>
> > > > processes or to get a mapping so that I can do<br>
> it myself?<br>
> > > ><br>
> > > ><br>
> > > > By definition, PETSc vectors and matrices own<br>
> contiguous row<br>
> > > blocks. If<br>
> > > > you want to have another,<br>
> > > > global ordering, we support that with<br>
> > > > <a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>><br>
> > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>>><br>
> > > ><br>
> > > > Thanks,<br>
> > > ><br>
> > > > Matt<br>
> > > ><br>
> > > > Cheers,<br>
> > > > Enrico Degregori<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > What most experimenters take for granted before<br>
> they begin<br>
> > their<br>
> > > > experiments is infinitely more interesting than any<br>
> > results to which<br>
> > > > their experiments lead.<br>
> > > > -- Norbert Wiener<br>
> > > ><br>
> > > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>>><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > What most experimenters take for granted before they begin<br>
> their<br>
> > > experiments is infinitely more interesting than any<br>
> results to which<br>
> > > their experiments lead.<br>
> > > -- Norbert Wiener<br>
> > ><br>
> > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > What most experimenters take for granted before they begin their<br>
> > experiments is infinitely more interesting than any results to which<br>
> > their experiments lead.<br>
> > -- Norbert Wiener<br>
> ><br>
> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> <br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their <br>
> experiments is infinitely more interesting than any results to which <br>
> their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>