<div dir="ltr"><div dir="ltr">On Thu, Oct 19, 2023 at 1:46 PM Enrico <<a href="mailto:degregori@dkrz.de">degregori@dkrz.de</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Sorry but I don't want another partition, Petsc internally is changing <br>
the partition. I would like to have the same partition that I have in <br>
the application. Is the example not clear?<br></blockquote><div><br></div><div>"Petsc internally is changing the partition" This is not correct. PETSc does</div><div>not prescribe partitions. I refer to the documentation for the creation of Vec:</div><div><br></div><div> <a href="https://petsc.org/main/manualpages/Vec/VecCreateMPI/">https://petsc.org/main/manualpages/Vec/VecCreateMPI/</a></div><div><br></div><div>Here the user sets the local and global sizes. Since data is contiguous, these</div><div>completely define the vector, and are under user control.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
On 19/10/2023 19:43, Matthew Knepley wrote:<br>
> On Thu, Oct 19, 2023 at 1:00 PM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>> wrote:<br>
> <br>
> Here is a very very simple reproducer of my problem. It is a fortran<br>
> program and it has to run with 2 processes.<br>
> <br>
> <br>
> You seem to be saying that you start with one partition of your data, <br>
> but you would like<br>
> another partition. For this, you have to initially communicate. For this <br>
> I would use VecScatter.<br>
> However, since most data is generated, I would consider not generating <br>
> my data in that initial<br>
> distribution.<br>
> <br>
> There are many examples in the repository. In the discretization of a <br>
> PDE, we first divide the domain,<br>
> then number up each piece, then assemble the linear algebra objects.<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> The output is:<br>
> <br>
> process 0 : xx_v( 1 ) = 0.000000000000000<br>
> process 0 : xx_v( 2 ) = 1.000000000000000<br>
> process 0 : xx_v( 3 ) = 2.000000000000000<br>
> process 1 : xx_v( 1 ) = 3.000000000000000<br>
> process 1 : xx_v( 2 ) = 4.000000000000000<br>
> process 1 : xx_v( 3 ) = 5.000000000000000<br>
> <br>
> and I would like to have:<br>
> <br>
> process 0 : xx_v( 1 ) = 2.000000000000000<br>
> process 0 : xx_v( 2 ) = 3.000000000000000<br>
> process 0 : xx_v( 3 ) = 4.000000000000000<br>
> process 1 : xx_v( 1 ) = 0.000000000000000<br>
> process 1 : xx_v( 2 ) = 1.000000000000000<br>
> process 1 : xx_v( 3 ) = 5.000000000000000<br>
> <br>
> How can I do that?<br>
> <br>
> program main<br>
> #include <petsc/finclude/petscksp.h><br>
> use petscksp<br>
> implicit none<br>
> <br>
> PetscErrorCode ierr<br>
> PetscInt :: Psize = 6<br>
> integer :: Lsize<br>
> PetscInt :: work_size<br>
> PetscInt :: work_rank<br>
> Vec :: b<br>
> integer, allocatable, dimension(:) :: glb_index<br>
> double precision, allocatable, dimension(:) :: array<br>
> PetscScalar, pointer :: xx_v(:)<br>
> integer :: i<br>
> PetscCount :: csize<br>
> <br>
> CALL PetscInitialize(ierr)<br>
> <br>
> Lsize = 3<br>
> csize = Lsize<br>
> <br>
> allocate(glb_index(0:Lsize-1), array(0:Lsize-1))<br>
> <br>
> CALL MPI_Comm_size(PETSC_COMM_WORLD, work_size, ierr);<br>
> CALL MPI_Comm_rank(PETSC_COMM_WORLD, work_rank, ierr);<br>
> if (work_rank == 0) then<br>
> glb_index(0) = 2<br>
> glb_index(1) = 3<br>
> glb_index(2) = 4<br>
> array(0) = 2<br>
> array(1) = 3<br>
> array(2) = 4<br>
> else if (work_rank == 1) then<br>
> glb_index(0) = 0<br>
> glb_index(1) = 1<br>
> glb_index(2) = 5<br>
> array(0) = 0<br>
> array(1) = 1<br>
> array(2) = 5<br>
> end if<br>
> <br>
> ! Create and fill rhs vector<br>
> CALL VecCreate(PETSC_COMM_WORLD, b, ierr);<br>
> CALL VecSetSizes(b, Lsize, Psize, ierr);<br>
> CALL VecSetType(b, VECMPI, ierr);<br>
> <br>
> CALL VecSetPreallocationCOO(b, csize, glb_index, ierr)<br>
> CALL VecSetValuesCOO(b, array, INSERT_VALUES, ierr)<br>
> <br>
> CALL VecGetArrayReadF90(b, xx_v, ierr)<br>
> <br>
> do i=1,Lsize<br>
> write(*,*) 'process ', work_rank, ': xx_v(',i,') = ', xx_v(i)<br>
> end do<br>
> <br>
> CALL VecRestoreArrayReadF90(b, xx_v, ierr)<br>
> <br>
> deallocate(glb_index, array)<br>
> CALL VecDestroy(b,ierr)<br>
> <br>
> CALL PetscFinalize(ierr)<br>
> <br>
> end program main<br>
> <br>
> <br>
> On 19/10/2023 17:36, Matthew Knepley wrote:<br>
> > On Thu, Oct 19, 2023 at 11:33 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>> wrote:<br>
> ><br>
> > The layout is not poor, just the global indices are not<br>
> contiguous,this<br>
> > has nothing to do with the local memory layout which is extremely<br>
> > optimized for different architectures. I can not change the<br>
> layout<br>
> > anyway because it's a climate model with a million lines of code.<br>
> ><br>
> > I don't understand why Petsc is doing all this MPI<br>
> communication under<br>
> > the hood.<br>
> ><br>
> ><br>
> > I don't think we are communicating under the hood.<br>
> ><br>
> > I mean, it is changing the layout of the application and doing<br>
> > a lot of communication.<br>
> ><br>
> ><br>
> > We do not create the layout. The user creates the data layout<br>
> when they<br>
> > create a vector or matrix.<br>
> ><br>
> > Is there no way to force the same layout and<br>
> > provide info about how to do the halo exchange? In this way I<br>
> can have<br>
> > the same memory layout and there is no communication when I<br>
> fill or<br>
> > fetch the vectors and the matrix.<br>
> ><br>
> ><br>
> > Yes, you tell the vector/matrix your data layout when you create it.<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Matt<br>
> ><br>
> > Cheers,<br>
> > Enrico<br>
> ><br>
> > On 19/10/2023 17:21, Matthew Knepley wrote:<br>
> > > On Thu, Oct 19, 2023 at 10:51 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>> wrote:<br>
> > ><br>
> > > In the application the storage is contiguous but the<br>
> global<br>
> > indexing is<br>
> > > not. I would like to use AO as a translation layer but<br>
> I don't<br>
> > > understand it.<br>
> > ><br>
> > ><br>
> > > Why would you choose to index differently from your storage?<br>
> > ><br>
> > > My case is actually simple even if it is in a large<br>
> > application, I have<br>
> > ><br>
> > > Mat A, Vec b and Vec x<br>
> > ><br>
> > > After calling KSPSolve, I use VecGetArrayReadF90 to get a<br>
> > pointer to<br>
> > > the<br>
> > > data and they are in the wrong ordering, so for<br>
> example the first<br>
> > > element of the solution array on process 0 belongs to<br>
> process<br>
> > 1 in the<br>
> > > application.<br>
> > ><br>
> > ><br>
> > > Again, this seems to be a poor choice of layout. What we<br>
> > typically do is<br>
> > > to partition<br>
> > > the data into chunks owned by each process first.<br>
> > ><br>
> > > Is it at this point that I should use the AO translation<br>
> > layer? This<br>
> > > would be quite bad, it means to build Mat A and Vec b<br>
> there<br>
> > is MPI<br>
> > > communication and also to get the data of Vec x back<br>
> in the<br>
> > application.<br>
> > ><br>
> > ><br>
> > > If you want to store data that process i updates on process j,<br>
> > this will<br>
> > > need communication.<br>
> > ><br>
> > > Anyway, I've tried to use<br>
> AOPetscToApplicationPermuteReal on the<br>
> > > solution array but it doesn't work as I would like. Is<br>
> this<br>
> > function<br>
> > > suppose to do MPI communication between processes and<br>
> fetch<br>
> > the values<br>
> > > of the application ordering?<br>
> > ><br>
> > ><br>
> > > There is no communication here. That function call just<br>
> changes one<br>
> > > integer into another.<br>
> > > If you want to update values on another process, we<br>
> recommend using<br>
> > > VecScatter() or<br>
> > > MatSetValues(), both of which take global indices and do<br>
> > communication<br>
> > > if necessary.<br>
> > ><br>
> > > Thanks,<br>
> > ><br>
> > > Matt<br>
> > ><br>
> > > Cheers,<br>
> > > Enrico<br>
> > ><br>
> > > On 19/10/2023 15:25, Matthew Knepley wrote:<br>
> > > > On Thu, Oct 19, 2023 at 8:57 AM Enrico<br>
> <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>>> wrote:<br>
> > > ><br>
> > > > Maybe I wasn't clear enough. I would like to<br>
> > completely get<br>
> > > rid of<br>
> > > > Petsc<br>
> > > > ordering because I don't want extra<br>
> communication between<br>
> > > processes to<br>
> > > > construct the vector and the matrix (since I<br>
> have to fill<br>
> > > them every<br>
> > > > time step because I'm just using the linear solver<br>
> > with a Mat<br>
> > > and a Vec<br>
> > > > data structure). I don't understand how I can<br>
> do that.<br>
> > > ><br>
> > > ><br>
> > > > Any program you write to do linear algebra will have<br>
> > contiguous<br>
> > > storage<br>
> > > > because it<br>
> > > > is so much faster. Contiguous indexing makes sense for<br>
> > contiguous<br>
> > > > storage. If you<br>
> > > > want to use non-contiguous indexing for contiguous<br>
> > storage, you<br>
> > > would<br>
> > > > need some<br>
> > > > translation layer. The AO is such a translation,<br>
> but you<br>
> > could do<br>
> > > this<br>
> > > > any way you want.<br>
> > > ><br>
> > > > Thanks,<br>
> > > ><br>
> > > > Matt<br>
> > > ><br>
> > > > My initial idea was to create another global index<br>
> > ordering<br>
> > > within my<br>
> > > > application to use only for the Petsc interface<br>
> but then I<br>
> > > think that<br>
> > > > the ghost cells are wrong.<br>
> > > ><br>
> > > > On 19/10/2023 14:50, Matthew Knepley wrote:<br>
> > > > > On Thu, Oct 19, 2023 at 6:51 AM Enrico<br>
> > <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>><br>
> > > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>>>> wrote:<br>
> > > > ><br>
> > > > > Hello,<br>
> > > > ><br>
> > > > > if I create an application ordering using<br>
> > > AOCreateBasic, should I<br>
> > > > > provide the same array for const PetscInt<br>
> > myapp[] and<br>
> > > const<br>
> > > > PetscInt<br>
> > > > > mypetsc[] in order to get the same<br>
> ordering of the<br>
> > > application<br>
> > > > > within PETSC?<br>
> > > > ><br>
> > > > ><br>
> > > > > Are you asking if the identity permutation<br>
> can be<br>
> > constructed<br>
> > > > using the<br>
> > > > > same array twice? Yes.<br>
> > > > ><br>
> > > > > And once I define the ordering so that<br>
> the local<br>
> > > vector and<br>
> > > > matrix are<br>
> > > > > defined in PETSC as in my application,<br>
> how can<br>
> > I use it to<br>
> > > > create the<br>
> > > > > actual vector and matrix?<br>
> > > > ><br>
> > > > ><br>
> > > > > The vectors and matrices do not change. The<br>
> AO is a<br>
> > > permutation.<br>
> > > > You can<br>
> > > > > use it to permute<br>
> > > > > a vector into another order, or to convert<br>
> on index to<br>
> > > another.<br>
> > > > ><br>
> > > > > Thanks,<br>
> > > > ><br>
> > > > > Matt<br>
> > > > ><br>
> > > > > Thanks in advance for the help.<br>
> > > > ><br>
> > > > > Cheers,<br>
> > > > > Enrico<br>
> > > > ><br>
> > > > > On 18/10/2023 13:39, Matthew Knepley wrote:<br>
> > > > > > On Wed, Oct 18, 2023 at 5:55 AM Enrico<br>
> > > <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>><br>
> > > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>>><br>
> > > > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>><br>
> > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>>>>> wrote:<br>
> > > > > ><br>
> > > > > > Hello,<br>
> > > > > ><br>
> > > > > > I'm trying to use Petsc to solve<br>
> a linear<br>
> > > system in an<br>
> > > > > application. I'm<br>
> > > > > > using the coordinate format to<br>
> define the<br>
> > > matrix and the<br>
> > > > > vector (it<br>
> > > > > > should work better on GPU but at<br>
> the moment<br>
> > > every test<br>
> > > > is on<br>
> > > > > CPU).<br>
> > > > > > After<br>
> > > > > > the call to VecSetValuesCOO, I've<br>
> > noticed that the<br>
> > > > vector is<br>
> > > > > storing<br>
> > > > > > the<br>
> > > > > > data in a different way from my<br>
> > application. For<br>
> > > > example with two<br>
> > > > > > processes in the application<br>
> > > > > ><br>
> > > > > > process 0 owns cells 2, 3, 4<br>
> > > > > ><br>
> > > > > > process 1 owns cells 0, 1, 5<br>
> > > > > ><br>
> > > > > > But in the vector data structure<br>
> of Petsc<br>
> > > > > ><br>
> > > > > > process 0 owns cells 0, 1, 2<br>
> > > > > ><br>
> > > > > > process 1 owns cells 3, 4, 5<br>
> > > > > ><br>
> > > > > > This is in principle not a big issue,<br>
> > but after<br>
> > > > solving the<br>
> > > > > linear<br>
> > > > > > system I get the solution vector<br>
> x and I<br>
> > want<br>
> > > to get the<br>
> > > > > values in the<br>
> > > > > > correct processes. Is there a way<br>
> to get<br>
> > vector<br>
> > > values<br>
> > > > from other<br>
> > > > > > processes or to get a mapping so<br>
> that I<br>
> > can do<br>
> > > it myself?<br>
> > > > > ><br>
> > > > > ><br>
> > > > > > By definition, PETSc vectors and<br>
> matrices own<br>
> > > contiguous row<br>
> > > > > blocks. If<br>
> > > > > > you want to have another,<br>
> > > > > > global ordering, we support that with<br>
> > > > > ><br>
> <a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>><br>
> > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>>><br>
> > > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>><br>
> > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>>>><br>
> > > > > ><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>><br>
> > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>>><br>
> > > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>><br>
> > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>>>>><br>
> > > > > ><br>
> > > > > > Thanks,<br>
> > > > > ><br>
> > > > > > Matt<br>
> > > > > ><br>
> > > > > > Cheers,<br>
> > > > > > Enrico Degregori<br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > --<br>
> > > > > > What most experimenters take for<br>
> granted before<br>
> > > they begin<br>
> > > > their<br>
> > > > > > experiments is infinitely more<br>
> interesting<br>
> > than any<br>
> > > > results to which<br>
> > > > > > their experiments lead.<br>
> > > > > > -- Norbert Wiener<br>
> > > > > ><br>
> > > > > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>>><br>
> > > > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>>>><br>
> > > > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>>><br>
> > > > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>>>>><br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > --<br>
> > > > > What most experimenters take for granted before<br>
> > they begin<br>
> > > their<br>
> > > > > experiments is infinitely more interesting<br>
> than any<br>
> > > results to which<br>
> > > > > their experiments lead.<br>
> > > > > -- Norbert Wiener<br>
> > > > ><br>
> > > > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>>><br>
> > > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>>>><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > What most experimenters take for granted before<br>
> they begin<br>
> > their<br>
> > > > experiments is infinitely more interesting than any<br>
> > results to which<br>
> > > > their experiments lead.<br>
> > > > -- Norbert Wiener<br>
> > > ><br>
> > > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>>><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > What most experimenters take for granted before they begin<br>
> their<br>
> > > experiments is infinitely more interesting than any<br>
> results to which<br>
> > > their experiments lead.<br>
> > > -- Norbert Wiener<br>
> > ><br>
> > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > What most experimenters take for granted before they begin their<br>
> > experiments is infinitely more interesting than any results to which<br>
> > their experiments lead.<br>
> > -- Norbert Wiener<br>
> ><br>
> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> <br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their <br>
> experiments is infinitely more interesting than any results to which <br>
> their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>