2 Questions about DAs
Milad Fatenejad
icksa1 at gmail.com
Mon May 12 15:18:34 CDT 2008
Hi Amit:
I was thinking of the following situation.
There are two vectors global/local1 and global/local2.
Function f1 modifies the values of global1 and global2.
Function f3 requires the updated ghost point values.
Function f2 doesn't depend on either vector at all, but is really
computationally intensive
I would like to do the following:
f1(global1, global2) // Both vectors modified, I need to
scatter before calling f3()
DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2);
f2() // function that takes a really long time
DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2);
f3(local1, local2) // Needs the ghost values
If I don't overlap the scattering, I end up with something like this:
f1(global1, global2) // Both vectors modified, I need to
scatter before calling f3()
DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1);
f2() // function that takes a really long time
DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2);
// Nothing left to do, just wait for scattering to end
DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2);
f3(local1, local2) // Needs the ghost values
In the second case, there is nothing left to do while the second
vector scatters and it seems like I just have to wait for this to
occur. Ideally, I would like to scatter both vectors while waiting for
f2() to finish...
I'm a little new to all of this, so let me know if my understanding is
just wrong...
Thanks
Milad
On Mon, May 12, 2008 at 2:51 PM, <Amit.Itagi at seagate.com> wrote:
> I don't understand your motivation for trying to have two consecutive
> scatterBegin before the scatterEnd of the first. I think that the
> scatterBegin and the corresponding scatterEnd need to thought of as a
> single scatter operation. Infact, I don't understand the concept of
> "overlapping" the scattering.
>
>
> Thanks
>
> Rgds,
> Amit
>
>
>
>
> "Milad Fatenejad"
> <icksa1 at gmail.com
>
> > To
> Sent by: petsc-users at mcs.anl.gov
> owner-petsc-users cc
> @mcs.anl.gov
> No Phone Info Subject
> Available Re: 2 Questions about DAs
>
>
> 05/12/2008 02:58
>
>
> PM
>
>
> Please respond to
> petsc-users at mcs.a
> nl.gov
>
>
>
>
>
>
> Hi,
> Thanks for the reply.
>
> If I have two global vectors from the same DA, say global1 and global2
> and I try to scatter to local vectors local1 and local2 using the
> command DAGlobalToLocalBegin/End in the following manner:
>
> DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1);
> DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1);
>
> DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2);
> DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2);
>
> Everything is fine. If instead I do:
>
> DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1);
> DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2);
>
> DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1);
> DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2);
>
> I get the error:
> [0]PETSC ERROR: Object is in wrong state!
> [0]PETSC ERROR: Scatter ctx already in use!
>
> What I would like to be able to do is overlap the scattering and that
> produces the error.
>
> Thanks
> Milad
>
> On Mon, May 12, 2008 at 1:43 PM, <Amit.Itagi at seagate.com> wrote:
> > Milad,
> >
> > I have a DA with 6 vectors sharing the DA structure. I defined the DA to
> > have just 1 DOF. I generated the 6 vectors from that DA. Also, I scatter
> > the vectors separately. Things seem to work fine.
> >
> > Thanks
> >
> > Rgds,
> > Amit
> >
> >
> >
> >
> > "Milad Fatenejad"
> > <mfatenejad at wisc.
> > edu>
> To
> > Sent by: petsc-users at mcs.anl.gov
> > owner-petsc-users
> cc
> > @mcs.anl.gov
> > No Phone Info
> Subject
> > Available 2 Questions about DAs
> >
> >
> > 05/12/2008 12:02
> > PM
> >
> >
> > Please respond to
> > petsc-users at mcs.a
> > nl.gov
> >
> >
> >
> >
> >
> >
> >
> >
> > Hello:
> > I have two separate DA questions:
> >
> > 1) I am writing a large finite difference code and would like to be
> > able to represent an array of vectors. I am currently doing this by
> > creating a single DA and calling DACreateGlobalVector several times,
> > but the manual also states that:
> >
> > "PETSc currently provides no container for multiple arrays sharing the
> > same distributed array communication; note, however, that the dof
> > parameter handles many cases of interest."
> >
> > I also found the following mailing list thread which describes how to
> > use the dof parameter to represent several vectors:
> >
> >
> http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html
>
> >
> >
> > Where the following solution is proposed:
> > """
> > The easiest thing to do in C is to declare a struct:
> >
> > typedef struct {
> > PetscScalar v[3];
> > PetscScalar p;
> > } Space;
> >
> > and then cast pointers
> >
> > Space ***array;
> >
> > DAVecGetArray(da, u, (void *) &array);
> >
> > array[k][j][i].v *= -1.0;
> > """
> >
> > The problem with the proposed solution, is that they use a struct to
> > get the individual values, but what if you don't know the number of
> > degrees of freedom at compile time?
> >
> > So my question is two fold:
> > a) Is there a problem with just having a single DA and calling
> > DACreateGlobalVector multiple times? Does this affect performance at
> > all (I have many different vectors)?
> > b) Is there a way to use the dof parameter when creating a DA when the
> > number of degrees of freedom is not known at compile time?
> > Specifically, I would like to be able to access the individual values
> > of the vector, just like the example shows...
> >
> >
> > 2) The code I am writing has a lot of different parts which present a
> > lot of opportunities to overlap communication an computation when
> > scattering vectors to update values in the ghost points. Right now,
> > all of my vectors (there are ~50 of them) share a single DA because
> > they all have the same shape. However, by sharing a single DA, I can
> > only scatter one vector at a time. It would be nice to be able to
> > start scattering each vector right after I'm done computing it, and
> > finish scattering it right before I need it again but I can't because
> > other vectors might need to be scattered in between. I then re-wrote
> > part of my code so that each vector had its own DA object, but this
> > ended up being incredibly slow (I assume this is because I have so
> > many vectors).
> >
> > My question is, is there a way to scatter multiple vectors
> > simultaneously without affecting the performance of the code? Does it
> > make sense to do this?
> >
> >
> > I'd really appreciate any help...
> >
> > Thanks
> > Milad Fatenejad
> >
> >
> >
> >
>
>
>
>
More information about the petsc-users
mailing list