2 Questions about DAs
Milad Fatenejad
icksa1 at gmail.com
Mon May 12 18:21:07 CDT 2008
Hi:
I created a simple test problem that demonstrates the issue. In the
test problem, 100 vectors are created using:
single.cpp: a single distributed array and
multi.cpp: 100 distributed arrays
Some math is performed on the vectors, then they are scattered to
local vectors..
The log summary (running 2 processes) shows that multi.cpp uses more
memory and performs more reductions than single.cpp, which is similar
to the experience I had with my program...
I hope this helps
Milad
On Mon, May 12, 2008 at 3:15 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Mon, May 12, 2008 at 3:01 PM, Milad Fatenejad <icksa1 at gmail.com> wrote:
> > Hello:
> > I've attached the result of two calculations. The file "log-multi-da"
> > uses 1 DA for each vector (322 in all) and the file "log-single-da"
> > using 1 DA for the entire calculation. When using 322 DA's, about 10x
> > more time is spent in VecScatterBegin and VecScatterEnd. Both were
> > running using two processes
> >
> > I should mention that the source code for these two runs was exactly
> > the same, I didn't reorder the scatters differently. The only
> > difference was the number of DAs
> >
> > Any suggestions? Do you think this is related to the number of DA's,
> > or something else?
>
> There are vastly different numbers of reductions and much bigger memory usage.
> Please send the code and I will look at it.
>
> Matt
>
>
>
> > Thanks for your help
> > Milad
> >
> > On Mon, May 12, 2008 at 1:56 PM, Matthew Knepley <knepley at gmail.com> wrote:
> > >
> > > On Mon, May 12, 2008 at 11:02 AM, Milad Fatenejad <mfatenejad at wisc.edu> wrote:
> > > > Hello:
> > > > I have two separate DA questions:
> > > >
> > > > 1) I am writing a large finite difference code and would like to be
> > > > able to represent an array of vectors. I am currently doing this by
> > > > creating a single DA and calling DACreateGlobalVector several times,
> > > > but the manual also states that:
> > > >
> > > > "PETSc currently provides no container for multiple arrays sharing the
> > > > same distributed array communication; note, however, that the dof
> > > > parameter handles many cases of interest."
> > > >
> > > > I also found the following mailing list thread which describes how to
> > > > use the dof parameter to represent several vectors:
> > > >
> > > >
> > > > http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html
> > > >
> > > > Where the following solution is proposed:
> > > > """
> > > > The easiest thing to do in C is to declare a struct:
> > > >
> > > > typedef struct {
> > > > PetscScalar v[3];
> > > > PetscScalar p;
> > > > } Space;
> > > >
> > > > and then cast pointers
> > > >
> > > > Space ***array;
> > > >
> > > > DAVecGetArray(da, u, (void *) &array);
> > > >
> > > > array[k][j][i].v *= -1.0;
> > > > """
> > > >
> > > > The problem with the proposed solution, is that they use a struct to
> > > > get the individual values, but what if you don't know the number of
> > > > degrees of freedom at compile time?
> > >
> > > It would be nice to get variable structs in C. However, you can just deference
> > > the object directly. For example, for 50 degrees of freedom, you can do
> > >
> > > array[k][j][i][47] *= -1.0;
> > >
> > >
> > > > So my question is two fold:
> > > > a) Is there a problem with just having a single DA and calling
> > > > DACreateGlobalVector multiple times? Does this affect performance at
> > > > all (I have many different vectors)?
> > >
> > > These are all independent objects. Thus, by itself, creating any number of
> > > Vecs does nothing to performance (unless you start to run out of memory).
> > >
> > >
> > > > b) Is there a way to use the dof parameter when creating a DA when the
> > > > number of degrees of freedom is not known at compile time?
> > > > Specifically, I would like to be able to access the individual values
> > > > of the vector, just like the example shows...
> > >
> > >
> > > see above.
> > >
> > > > 2) The code I am writing has a lot of different parts which present a
> > > > lot of opportunities to overlap communication an computation when
> > > > scattering vectors to update values in the ghost points. Right now,
> > > > all of my vectors (there are ~50 of them) share a single DA because
> > > > they all have the same shape. However, by sharing a single DA, I can
> > > > only scatter one vector at a time. It would be nice to be able to
> > > > start scattering each vector right after I'm done computing it, and
> > > > finish scattering it right before I need it again but I can't because
> > > > other vectors might need to be scattered in between. I then re-wrote
> > > > part of my code so that each vector had its own DA object, but this
> > > > ended up being incredibly slow (I assume this is because I have so
> > > > many vectors).
> > >
> > > The problem here is that buffering will have to be done for each outstanding
> > > scatter. Thus I see two resolutions:
> > >
> > > 1) Duplicate the DA scatter for as many Vecs as you wish to scatter at once.
> > > This is essentially what you accomplish with separate DAs.
> > >
> > > 2) You the dof method. However, this scatter ALL the vectors every time.
> > >
> > > I do not understand what performance problem you would have with multiple
> > > DAs. With any performance questions, we suggest sending the output of
> > > -log_summary so we have data to look at.
> > >
> > > Matt
> > >
> > >
> > >
> > > > My question is, is there a way to scatter multiple vectors
> > > > simultaneously without affecting the performance of the code? Does it
> > > > make sense to do this?
> > > >
> > > >
> > > > I'd really appreciate any help...
> > > >
> > > > Thanks
> > > > Milad Fatenejad
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to which
> > > their experiments lead.
> > > -- Norbert Wiener
> > >
> > >
> >
>
>
>
> --
>
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log-multi
Type: application/octet-stream
Size: 10666 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080512/ffd5be5c/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log-single
Type: application/octet-stream
Size: 10666 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080512/ffd5be5c/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: multi.cpp
Type: text/x-c++src
Size: 1557 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080512/ffd5be5c/attachment.cpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: single.cpp
Type: text/x-c++src
Size: 1449 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080512/ffd5be5c/attachment-0001.cpp>
More information about the petsc-users
mailing list