2 Questions about DAs

[Milad Fatenejad]

Hello:
I have two separate DA questions:

1) I am writing a large finite difference code and would like to be
able to represent an array of vectors. I am currently doing this by
creating a single DA and calling DACreateGlobalVector several times,
but the manual also states that:

"PETSc currently provides no container for multiple arrays sharing the
same distributed array communication; note, however, that the dof
parameter handles many cases of interest."

I also found the following mailing list thread which describes how to
use the dof parameter to represent several vectors:

http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html

Where the following solution is proposed:
"""
The easiest thing to do in C is to declare a struct:

typedef struct {
  PetscScalar v[3];
  PetscScalar p;
} Space;

and then cast pointers

  Space ***array;

  DAVecGetArray(da, u, (void *) &array);

     array[k][j][i].v *= -1.0;
"""

The problem with the proposed solution, is that they use a struct to
get the individual values, but what if you don't know the number of
degrees of freedom at compile time?

So my question is two fold:
a) Is there a problem with just having a single DA and calling
DACreateGlobalVector multiple times? Does this affect performance at
all (I have many different vectors)?
b) Is there a way to use the dof parameter when creating a DA when the
number of degrees of freedom is not known at compile time?
Specifically, I would like to be able to access the individual values
of the vector, just like the example shows...


2) The code I am writing has a lot of different parts which present a
lot of opportunities to overlap communication an computation when
scattering vectors to update values in the ghost points. Right now,
all of my vectors (there are ~50 of them) share a single DA because
they all have the same shape. However, by sharing a single DA, I can
only scatter one vector at a time. It would be nice to be able to
start scattering each vector right after I'm done computing it, and
finish scattering it right before I need it again but I can't because
other vectors might need to be scattered in between. I then re-wrote
part of my code so that each vector had its own DA object, but this
ended up being incredibly slow (I assume this is because I have so
many vectors).

My question is, is there a way to scatter multiple vectors
simultaneously without affecting the performance of the code? Does it
make sense to do this?


I'd really appreciate any help...

Thanks
Milad Fatenejad