Combine ghost updates

[Barry Smith]

   Billy,

    Since the three vectors are independent there is currently
no way to do this. We would need to add additional support to the
scatter operations to allow packing several scatters together
(actually not a bad idea).

    This problem does not usually come up because we recommend
under most circumstances to "interlace" field variables rather than
keep them in seperate vectors. For example, in your case you would
have a single U vector that looked like (ux_0,uy_0,uz_0,ux_1,uy_1,uz_1,...)
The reason to interlace is for efficiency; in most codes when ux_i is loaded
the uy_i and uz_i are also needed "at the same time"; by combining them
you can get less cache thrashing and less tlb misses: see
for example http://www-fp.mcs.anl.gov/petsc-fun3d/Papers/manke.pdf table 3
where performance is more than doubled by simply interlacing. There are
many other papers that discuss this at 
http://www-fp.mcs.anl.gov/petsc-fun3d/Papers/papers.html

   It is likely that switching to interlaced variables would give much
more of a performance boost then combining the scatters. But you could
much through the code in src/vec/vec/utils/vpscat.c you see how it would
be possible to manage multiple vectors with some hacking.


    Barry


On Tue, 4 Apr 2006, billy at dem.uminho.pt wrote:

>
> Hi,
>
> I read in a paper that if you combine all updates in a single vector, you can
> speed up the comunication. They combined gradient x, y, z values in one vector.
> Does anyone know how this can be done?
>
> For example vectors ux, uy, and uz:
>
> VecGhostUpdateBegin(ux,INSERT_VALUES,SCATTER_FORWARD);
> VecGhostUpdateEnd(ux,INSERT_VALUES,SCATTER_FORWARD);
>
> VecGhostUpdateBegin(uy,INSERT_VALUES,SCATTER_FORWARD);
> VecGhostUpdateEnd(uy,INSERT_VALUES,SCATTER_FORWARD);
>
> VecGhostUpdateBegin(uz,INSERT_VALUES,SCATTER_FORWARD);
> VecGhostUpdateEnd(uz,INSERT_VALUES,SCATTER_FORWARD);
>
> Could they be combined in a vector U:
>
> VecGhostUpdateBegin(U,INSERT_VALUES,SCATTER_FORWARD);
> VecGhostUpdateEnd(U,INSERT_VALUES,SCATTER_FORWARD);
>
> and would it be faster?
>
>
> Billy.
>
>