2 Questions about DAs

[Milad Fatenejad]

Hello:
First, I'm having some email problems, so sorry if this shows up twice...

I am using PETSc to write a large multi-physics finite difference code
with a lot of opportunity for overlapping computation and
communication. Right now, I have created ~100 petsc vectors for
storing various quantities, which currently all share a single DA. The
problem with this system is that I can only scatter one quantity at a
time to update the values of the ghost points. If I try to scatter
more than one object at a time, I get the following error:

[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR:  Scatter ctx already in use!

It would be really nice to be able to start scattering a vector
whenever I am done with a computation, and just finish the scatter
whenever I need the vector again. Again, this is impossible because
all of the vectors share the same DA.

I then reorganized my code, so that each vector had its own DA,
however, this led to the program running significantly more slowly (I
assume this is just because I have so many vectors).

So my first question is: Is there a way to organize the code so I can
overlap the scattering of vectors without having a significant
performance hit?


And on a related note, many times I need to create arrays of vectors.
I just discovered the function "VecDuplicateVecs" (and related
functions),  which look like performs this operation. Is this the best
way to create arrays of vectors? Is there a way to directly get the
array from the DA without having to create a vector and duplicate it
(I don't see a "DACreateGlobalVectorS")?

I know it is also possible to do something like this using the DOF
parameter in the DACreate call as shown in:
http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html

Are there any advantages to using dof as opposed to VecDuplicateVecs, etc.?

I'd appreciate any help

Thank You
Milad Fatenejad