2 Questions about DAs

Milad Fatenejad mfatenejad at wisc.edu
Mon May 12 11:02:33 CDT 2008

I have two separate DA questions:

1) I am writing a large finite difference code and would like to be
able to represent an array of vectors. I am currently doing this by
creating a single DA and calling DACreateGlobalVector several times,
but the manual also states that:

"PETSc currently provides no container for multiple arrays sharing the
same distributed array communication; note, however, that the dof
parameter handles many cases of interest."

I also found the following mailing list thread which describes how to
use the dof parameter to represent several vectors:


Where the following solution is proposed:
The easiest thing to do in C is to declare a struct:

typedef struct {
  PetscScalar v[3];
  PetscScalar p;
} Space;

and then cast pointers

  Space ***array;

  DAVecGetArray(da, u, (void *) &array);

     array[k][j][i].v *= -1.0;

The problem with the proposed solution, is that they use a struct to
get the individual values, but what if you don't know the number of
degrees of freedom at compile time?

So my question is two fold:
a) Is there a problem with just having a single DA and calling
DACreateGlobalVector multiple times? Does this affect performance at
all (I have many different vectors)?
b) Is there a way to use the dof parameter when creating a DA when the
number of degrees of freedom is not known at compile time?
Specifically, I would like to be able to access the individual values
of the vector, just like the example shows...

2) The code I am writing has a lot of different parts which present a
lot of opportunities to overlap communication an computation when
scattering vectors to update values in the ghost points. Right now,
all of my vectors (there are ~50 of them) share a single DA because
they all have the same shape. However, by sharing a single DA, I can
only scatter one vector at a time. It would be nice to be able to
start scattering each vector right after I'm done computing it, and
finish scattering it right before I need it again but I can't because
other vectors might need to be scattered in between. I then re-wrote
part of my code so that each vector had its own DA object, but this
ended up being incredibly slow (I assume this is because I have so
many vectors).

My question is, is there a way to scatter multiple vectors
simultaneously without affecting the performance of the code? Does it
make sense to do this?

I'd really appreciate any help...

Milad Fatenejad

More information about the petsc-users mailing list