2 Questions about DAs
mfatenejad at wisc.edu
Mon May 12 13:10:29 CDT 2008
First, I'm having some email problems, so sorry if this shows up twice...
I am using PETSc to write a large multi-physics finite difference code
with a lot of opportunity for overlapping computation and
communication. Right now, I have created ~100 petsc vectors for
storing various quantities, which currently all share a single DA. The
problem with this system is that I can only scatter one quantity at a
time to update the values of the ghost points. If I try to scatter
more than one object at a time, I get the following error:
PETSC ERROR: Object is in wrong state!
PETSC ERROR: Scatter ctx already in use!
It would be really nice to be able to start scattering a vector
whenever I am done with a computation, and just finish the scatter
whenever I need the vector again. Again, this is impossible because
all of the vectors share the same DA.
I then reorganized my code, so that each vector had its own DA,
however, this led to the program running significantly more slowly (I
assume this is just because I have so many vectors).
So my first question is: Is there a way to organize the code so I can
overlap the scattering of vectors without having a significant
And on a related note, many times I need to create arrays of vectors.
I just discovered the function "VecDuplicateVecs" (and related
functions), which look like performs this operation. Is this the best
way to create arrays of vectors? Is there a way to directly get the
array from the DA without having to create a vector and duplicate it
(I don't see a "DACreateGlobalVectorS")?
I know it is also possible to do something like this using the DOF
parameter in the DACreate call as shown in:
Are there any advantages to using dof as opposed to VecDuplicateVecs, etc.?
I'd appreciate any help
More information about the petsc-users