2 Questions about DAs

[Milad Fatenejad]

Hello:
I've attached the result of two calculations. The file "log-multi-da"
uses 1 DA for each vector (322 in all) and the file "log-single-da"
using 1 DA for the entire calculation. When using 322 DA's, about 10x
more time is spent in VecScatterBegin and VecScatterEnd. Both were
running using two processes

I should mention that the source code for these two runs was exactly
the same, I didn't reorder the scatters differently. The only
difference was the number of DAs

Any suggestions? Do you think this is related to the number of DA's,
or something else?

Thanks for your help
Milad

On Mon, May 12, 2008 at 1:56 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Mon, May 12, 2008 at 11:02 AM, Milad Fatenejad <mfatenejad at wisc.edu> wrote:
>  > Hello:
>  >  I have two separate DA questions:
>  >
>  >  1) I am writing a large finite difference code and would like to be
>  >  able to represent an array of vectors. I am currently doing this by
>  >  creating a single DA and calling DACreateGlobalVector several times,
>  >  but the manual also states that:
>  >
>  >  "PETSc currently provides no container for multiple arrays sharing the
>  >  same distributed array communication; note, however, that the dof
>  >  parameter handles many cases of interest."
>  >
>  >  I also found the following mailing list thread which describes how to
>  >  use the dof parameter to represent several vectors:
>  >
>  >
>  >  http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html
>  >
>  >  Where the following solution is proposed:
>  >  """
>  >  The easiest thing to do in C is to declare a struct:
>  >
>  >  typedef struct {
>  >   PetscScalar v[3];
>  >   PetscScalar p;
>  >  } Space;
>  >
>  >  and then cast pointers
>  >
>  >   Space ***array;
>  >
>  >   DAVecGetArray(da, u, (void *) &array);
>  >
>  >      array[k][j][i].v *= -1.0;
>  >  """
>  >
>  >  The problem with the proposed solution, is that they use a struct to
>  >  get the individual values, but what if you don't know the number of
>  >  degrees of freedom at compile time?
>
>  It would be nice to get variable structs in C. However, you can just deference
>  the object directly. For example, for 50 degrees of freedom, you can do
>
>    array[k][j][i][47] *= -1.0;
>
>
>  >  So my question is two fold:
>  >  a) Is there a problem with just having a single DA and calling
>  >  DACreateGlobalVector multiple times? Does this affect performance at
>  >  all (I have many different vectors)?
>
>  These are all independent objects. Thus, by itself, creating any number of
>  Vecs does nothing to performance (unless you start to run out of memory).
>
>
>  >  b) Is there a way to use the dof parameter when creating a DA when the
>  >  number of degrees of freedom is not known at compile time?
>  >  Specifically, I would like to be able to access the individual values
>  >  of the vector, just like the example shows...
>
>
> see above.
>
>  >  2) The code I am writing has a lot of different parts which present a
>  >  lot of opportunities to overlap communication an computation when
>  >  scattering vectors to update values in the ghost points. Right now,
>  >  all of my vectors (there are ~50 of them) share a single DA because
>  >  they all have the same shape. However, by sharing a single DA, I can
>  >  only scatter one vector at a time. It would be nice to be able to
>  >  start scattering each vector right after I'm done computing it, and
>  >  finish scattering it right before I need it again but I can't because
>  >  other vectors might need to be scattered in between. I then re-wrote
>  >  part of my code so that each vector had its own DA object, but this
>  >  ended up being incredibly slow (I assume this is because I have so
>  >  many vectors).
>
>  The problem here is that buffering will have to be done for each outstanding
>  scatter. Thus I see two resolutions:
>
>   1) Duplicate the DA scatter for as many Vecs as you wish to scatter at once.
>       This is essentially what you accomplish with separate DAs.
>
>   2) You the dof method. However, this scatter ALL the vectors every time.
>
>  I do not understand what performance problem you would have with multiple
>  DAs. With any performance questions, we suggest sending the output of
>  -log_summary so we have data to look at.
>
>   Matt
>
>
>
>  >  My question is, is there a way to scatter multiple vectors
>  >  simultaneously without affecting the performance of the code? Does it
>  >  make sense to do this?
>  >
>  >
>  >  I'd really appreciate any help...
>  >
>  >  Thanks
>  >  Milad Fatenejad
>  >
>  >
>
>
>
>  --
>  What most experimenters take for granted before they begin their
>  experiments is infinitely more interesting than any results to which
>  their experiments lead.
>  -- Norbert Wiener
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log-multi-da
Type: application/octet-stream
Size: 11372 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080512/b45b0f2e/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log-single-da
Type: application/octet-stream
Size: 11372 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080512/b45b0f2e/attachment-0001.obj>