<div dir="ltr"><div dir="ltr">On Thu, Oct 19, 2023 at 1:00 PM Enrico <<a href="mailto:degregori@dkrz.de">degregori@dkrz.de</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Here is a very very simple reproducer of my problem. It is a fortran <br>
program and it has to run with 2 processes.<br></blockquote><div><br></div><div>You seem to be saying that you start with one partition of your data, but you would like</div><div>another partition. For this, you have to initially communicate. For this I would use VecScatter.</div><div>However, since most data is generated, I would consider not generating my data in that initial</div><div>distribution.</div><div><br></div><div>There are many examples in the repository. In the discretization of a PDE, we first divide the domain,</div><div>then number up each piece, then assemble the linear algebra objects.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
The output is:<br>
<br>
process 0 : xx_v( 1 ) = 0.000000000000000<br>
process 0 : xx_v( 2 ) = 1.000000000000000<br>
process 0 : xx_v( 3 ) = 2.000000000000000<br>
process 1 : xx_v( 1 ) = 3.000000000000000<br>
process 1 : xx_v( 2 ) = 4.000000000000000<br>
process 1 : xx_v( 3 ) = 5.000000000000000<br>
<br>
and I would like to have:<br>
<br>
process 0 : xx_v( 1 ) = 2.000000000000000<br>
process 0 : xx_v( 2 ) = 3.000000000000000<br>
process 0 : xx_v( 3 ) = 4.000000000000000<br>
process 1 : xx_v( 1 ) = 0.000000000000000<br>
process 1 : xx_v( 2 ) = 1.000000000000000<br>
process 1 : xx_v( 3 ) = 5.000000000000000<br>
<br>
How can I do that?<br>
<br>
program main<br>
#include <petsc/finclude/petscksp.h><br>
use petscksp<br>
implicit none<br>
<br>
PetscErrorCode ierr<br>
PetscInt :: Psize = 6<br>
integer :: Lsize<br>
PetscInt :: work_size<br>
PetscInt :: work_rank<br>
Vec :: b<br>
integer, allocatable, dimension(:) :: glb_index<br>
double precision, allocatable, dimension(:) :: array<br>
PetscScalar, pointer :: xx_v(:)<br>
integer :: i<br>
PetscCount :: csize<br>
<br>
CALL PetscInitialize(ierr)<br>
<br>
Lsize = 3<br>
csize = Lsize<br>
<br>
allocate(glb_index(0:Lsize-1), array(0:Lsize-1))<br>
<br>
CALL MPI_Comm_size(PETSC_COMM_WORLD, work_size, ierr);<br>
CALL MPI_Comm_rank(PETSC_COMM_WORLD, work_rank, ierr);<br>
if (work_rank == 0) then<br>
glb_index(0) = 2<br>
glb_index(1) = 3<br>
glb_index(2) = 4<br>
array(0) = 2<br>
array(1) = 3<br>
array(2) = 4<br>
else if (work_rank == 1) then<br>
glb_index(0) = 0<br>
glb_index(1) = 1<br>
glb_index(2) = 5<br>
array(0) = 0<br>
array(1) = 1<br>
array(2) = 5<br>
end if<br>
<br>
! Create and fill rhs vector<br>
CALL VecCreate(PETSC_COMM_WORLD, b, ierr);<br>
CALL VecSetSizes(b, Lsize, Psize, ierr);<br>
CALL VecSetType(b, VECMPI, ierr);<br>
<br>
CALL VecSetPreallocationCOO(b, csize, glb_index, ierr)<br>
CALL VecSetValuesCOO(b, array, INSERT_VALUES, ierr)<br>
<br>
CALL VecGetArrayReadF90(b, xx_v, ierr)<br>
<br>
do i=1,Lsize<br>
write(*,*) 'process ', work_rank, ': xx_v(',i,') = ', xx_v(i)<br>
end do<br>
<br>
CALL VecRestoreArrayReadF90(b, xx_v, ierr)<br>
<br>
deallocate(glb_index, array)<br>
CALL VecDestroy(b,ierr)<br>
<br>
CALL PetscFinalize(ierr)<br>
<br>
end program main<br>
<br>
<br>
On 19/10/2023 17:36, Matthew Knepley wrote:<br>
> On Thu, Oct 19, 2023 at 11:33 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>> wrote:<br>
> <br>
> The layout is not poor, just the global indices are not contiguous,this<br>
> has nothing to do with the local memory layout which is extremely<br>
> optimized for different architectures. I can not change the layout<br>
> anyway because it's a climate model with a million lines of code.<br>
> <br>
> I don't understand why Petsc is doing all this MPI communication under<br>
> the hood. <br>
> <br>
> <br>
> I don't think we are communicating under the hood.<br>
> <br>
> I mean, it is changing the layout of the application and doing<br>
> a lot of communication.<br>
> <br>
> <br>
> We do not create the layout. The user creates the data layout when they <br>
> create a vector or matrix.<br>
> <br>
> Is there no way to force the same layout and<br>
> provide info about how to do the halo exchange? In this way I can have<br>
> the same memory layout and there is no communication when I fill or<br>
> fetch the vectors and the matrix.<br>
> <br>
> <br>
> Yes, you tell the vector/matrix your data layout when you create it.<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> Cheers,<br>
> Enrico<br>
> <br>
> On 19/10/2023 17:21, Matthew Knepley wrote:<br>
> > On Thu, Oct 19, 2023 at 10:51 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>> wrote:<br>
> ><br>
> > In the application the storage is contiguous but the global<br>
> indexing is<br>
> > not. I would like to use AO as a translation layer but I don't<br>
> > understand it.<br>
> ><br>
> ><br>
> > Why would you choose to index differently from your storage?<br>
> ><br>
> > My case is actually simple even if it is in a large<br>
> application, I have<br>
> ><br>
> > Mat A, Vec b and Vec x<br>
> ><br>
> > After calling KSPSolve, I use VecGetArrayReadF90 to get a<br>
> pointer to<br>
> > the<br>
> > data and they are in the wrong ordering, so for example the first<br>
> > element of the solution array on process 0 belongs to process<br>
> 1 in the<br>
> > application.<br>
> ><br>
> ><br>
> > Again, this seems to be a poor choice of layout. What we<br>
> typically do is<br>
> > to partition<br>
> > the data into chunks owned by each process first.<br>
> ><br>
> > Is it at this point that I should use the AO translation<br>
> layer? This<br>
> > would be quite bad, it means to build Mat A and Vec b there<br>
> is MPI<br>
> > communication and also to get the data of Vec x back in the<br>
> application.<br>
> ><br>
> ><br>
> > If you want to store data that process i updates on process j,<br>
> this will<br>
> > need communication.<br>
> ><br>
> > Anyway, I've tried to use AOPetscToApplicationPermuteReal on the<br>
> > solution array but it doesn't work as I would like. Is this<br>
> function<br>
> > suppose to do MPI communication between processes and fetch<br>
> the values<br>
> > of the application ordering?<br>
> ><br>
> ><br>
> > There is no communication here. That function call just changes one<br>
> > integer into another.<br>
> > If you want to update values on another process, we recommend using<br>
> > VecScatter() or<br>
> > MatSetValues(), both of which take global indices and do<br>
> communication<br>
> > if necessary.<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Matt<br>
> ><br>
> > Cheers,<br>
> > Enrico<br>
> ><br>
> > On 19/10/2023 15:25, Matthew Knepley wrote:<br>
> > > On Thu, Oct 19, 2023 at 8:57 AM Enrico <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>> wrote:<br>
> > ><br>
> > > Maybe I wasn't clear enough. I would like to<br>
> completely get<br>
> > rid of<br>
> > > Petsc<br>
> > > ordering because I don't want extra communication between<br>
> > processes to<br>
> > > construct the vector and the matrix (since I have to fill<br>
> > them every<br>
> > > time step because I'm just using the linear solver<br>
> with a Mat<br>
> > and a Vec<br>
> > > data structure). I don't understand how I can do that.<br>
> > ><br>
> > ><br>
> > > Any program you write to do linear algebra will have<br>
> contiguous<br>
> > storage<br>
> > > because it<br>
> > > is so much faster. Contiguous indexing makes sense for<br>
> contiguous<br>
> > > storage. If you<br>
> > > want to use non-contiguous indexing for contiguous<br>
> storage, you<br>
> > would<br>
> > > need some<br>
> > > translation layer. The AO is such a translation, but you<br>
> could do<br>
> > this<br>
> > > any way you want.<br>
> > ><br>
> > > Thanks,<br>
> > ><br>
> > > Matt<br>
> > ><br>
> > > My initial idea was to create another global index<br>
> ordering<br>
> > within my<br>
> > > application to use only for the Petsc interface but then I<br>
> > think that<br>
> > > the ghost cells are wrong.<br>
> > ><br>
> > > On 19/10/2023 14:50, Matthew Knepley wrote:<br>
> > > > On Thu, Oct 19, 2023 at 6:51 AM Enrico<br>
> <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>>> wrote:<br>
> > > ><br>
> > > > Hello,<br>
> > > ><br>
> > > > if I create an application ordering using<br>
> > AOCreateBasic, should I<br>
> > > > provide the same array for const PetscInt<br>
> myapp[] and<br>
> > const<br>
> > > PetscInt<br>
> > > > mypetsc[] in order to get the same ordering of the<br>
> > application<br>
> > > > within PETSC?<br>
> > > ><br>
> > > ><br>
> > > > Are you asking if the identity permutation can be<br>
> constructed<br>
> > > using the<br>
> > > > same array twice? Yes.<br>
> > > ><br>
> > > > And once I define the ordering so that the local<br>
> > vector and<br>
> > > matrix are<br>
> > > > defined in PETSC as in my application, how can<br>
> I use it to<br>
> > > create the<br>
> > > > actual vector and matrix?<br>
> > > ><br>
> > > ><br>
> > > > The vectors and matrices do not change. The AO is a<br>
> > permutation.<br>
> > > You can<br>
> > > > use it to permute<br>
> > > > a vector into another order, or to convert on index to<br>
> > another.<br>
> > > ><br>
> > > > Thanks,<br>
> > > ><br>
> > > > Matt<br>
> > > ><br>
> > > > Thanks in advance for the help.<br>
> > > ><br>
> > > > Cheers,<br>
> > > > Enrico<br>
> > > ><br>
> > > > On 18/10/2023 13:39, Matthew Knepley wrote:<br>
> > > > > On Wed, Oct 18, 2023 at 5:55 AM Enrico<br>
> > <<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>><br>
> > > > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>><br>
> > > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>><br>
> > <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>><br>
> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a> <mailto:<a href="mailto:degregori@dkrz.de" target="_blank">degregori@dkrz.de</a>>>>>>> wrote:<br>
> > > > ><br>
> > > > > Hello,<br>
> > > > ><br>
> > > > > I'm trying to use Petsc to solve a linear<br>
> > system in an<br>
> > > > application. I'm<br>
> > > > > using the coordinate format to define the<br>
> > matrix and the<br>
> > > > vector (it<br>
> > > > > should work better on GPU but at the moment<br>
> > every test<br>
> > > is on<br>
> > > > CPU).<br>
> > > > > After<br>
> > > > > the call to VecSetValuesCOO, I've<br>
> noticed that the<br>
> > > vector is<br>
> > > > storing<br>
> > > > > the<br>
> > > > > data in a different way from my<br>
> application. For<br>
> > > example with two<br>
> > > > > processes in the application<br>
> > > > ><br>
> > > > > process 0 owns cells 2, 3, 4<br>
> > > > ><br>
> > > > > process 1 owns cells 0, 1, 5<br>
> > > > ><br>
> > > > > But in the vector data structure of Petsc<br>
> > > > ><br>
> > > > > process 0 owns cells 0, 1, 2<br>
> > > > ><br>
> > > > > process 1 owns cells 3, 4, 5<br>
> > > > ><br>
> > > > > This is in principle not a big issue,<br>
> but after<br>
> > > solving the<br>
> > > > linear<br>
> > > > > system I get the solution vector x and I<br>
> want<br>
> > to get the<br>
> > > > values in the<br>
> > > > > correct processes. Is there a way to get<br>
> vector<br>
> > values<br>
> > > from other<br>
> > > > > processes or to get a mapping so that I<br>
> can do<br>
> > it myself?<br>
> > > > ><br>
> > > > ><br>
> > > > > By definition, PETSc vectors and matrices own<br>
> > contiguous row<br>
> > > > blocks. If<br>
> > > > > you want to have another,<br>
> > > > > global ordering, we support that with<br>
> > > > > <a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>><br>
> > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>>><br>
> > > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>><br>
> > > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>><br>
> > > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>><br>
> > <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a><br>
> <<a href="https://petsc.org/main/manualpages/AO/" rel="noreferrer" target="_blank">https://petsc.org/main/manualpages/AO/</a>>>>>><br>
> > > > ><br>
> > > > > Thanks,<br>
> > > > ><br>
> > > > > Matt<br>
> > > > ><br>
> > > > > Cheers,<br>
> > > > > Enrico Degregori<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > --<br>
> > > > > What most experimenters take for granted before<br>
> > they begin<br>
> > > their<br>
> > > > > experiments is infinitely more interesting<br>
> than any<br>
> > > results to which<br>
> > > > > their experiments lead.<br>
> > > > > -- Norbert Wiener<br>
> > > > ><br>
> > > > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>>><br>
> > > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>>>><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > What most experimenters take for granted before<br>
> they begin<br>
> > their<br>
> > > > experiments is infinitely more interesting than any<br>
> > results to which<br>
> > > > their experiments lead.<br>
> > > > -- Norbert Wiener<br>
> > > ><br>
> > > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> > > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>>><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > What most experimenters take for granted before they begin<br>
> their<br>
> > > experiments is infinitely more interesting than any<br>
> results to which<br>
> > > their experiments lead.<br>
> > > -- Norbert Wiener<br>
> > ><br>
> > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > What most experimenters take for granted before they begin their<br>
> > experiments is infinitely more interesting than any results to which<br>
> > their experiments lead.<br>
> > -- Norbert Wiener<br>
> ><br>
> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> <br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their <br>
> experiments is infinitely more interesting than any results to which <br>
> their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>