2 Questions about DAs

Mon May 12 14:51:38 CDT 2008

I don't understand your motivation for trying to have two consecutive
scatterBegin before the scatterEnd of the first. I think that the
scatterBegin and the corresponding scatterEnd need to thought of as a
single scatter operation. Infact, I don't understand the concept of
"overlapping" the scattering.

Thanks

Rgds,
Amit

             "Milad Fatenejad"                                             
             <icksa1 at gmail.com                                             
             >                                                          To 
             Sent by:                  petsc-users at mcs.anl.gov             
             owner-petsc-users                                          cc 
             @mcs.anl.gov                                                  
             No Phone Info                                         Subject 
             Available                 Re: 2 Questions about DAs           

             05/12/2008 02:58                                              
             PM                                                            

             Please respond to                                             
             petsc-users at mcs.a                                             
                  nl.gov                                                   

Hi,
Thanks for the reply.

If I have two global vectors from the same DA, say global1 and global2
and I try to scatter to local vectors local1 and local2 using the
command DAGlobalToLocalBegin/End in the following manner:

DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1);

DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2);
DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2);

Everything is fine. If instead I do:

DAGlobalToLocalBegin(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalBegin(da, global2, INSERT_VALUES, local2);

DAGlobalToLocalEnd(da, global1, INSERT_VALUES, local1);
DAGlobalToLocalEnd(da, global2, INSERT_VALUES, local2);

I get the error:
[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR:  Scatter ctx already in use!

What I would like to be able to do is overlap the scattering and that
produces the error.

Thanks
Milad

On Mon, May 12, 2008 at 1:43 PM,  <Amit.Itagi at seagate.com> wrote:
> Milad,
>
>  I have a DA with 6 vectors sharing the DA structure. I defined the DA to
>  have just 1 DOF. I generated the 6 vectors from that DA. Also, I scatter
>  the vectors separately. Things seem to work fine.
>
>  Thanks
>
>  Rgds,
>  Amit
>
>
>
>
>              "Milad Fatenejad"
>              <mfatenejad at wisc.
>              edu>
To
>              Sent by:                  petsc-users at mcs.anl.gov
>              owner-petsc-users
cc
>              @mcs.anl.gov
>              No Phone Info
Subject
>              Available                 2 Questions about DAs
>
>
>              05/12/2008 12:02
>              PM
>
>
>              Please respond to
>              petsc-users at mcs.a
>                   nl.gov
>
>
>
>
>
>
>
>
>  Hello:
>  I have two separate DA questions:
>
>  1) I am writing a large finite difference code and would like to be
>  able to represent an array of vectors. I am currently doing this by
>  creating a single DA and calling DACreateGlobalVector several times,
>  but the manual also states that:
>
>  "PETSc currently provides no container for multiple arrays sharing the
>  same distributed array communication; note, however, that the dof
>  parameter handles many cases of interest."
>
>  I also found the following mailing list thread which describes how to
>  use the dof parameter to represent several vectors:
>
>
http://www-unix.mcs.anl.gov/web-mail-archive/lists/petsc-users/2008/02/msg00040.html

>
>
>  Where the following solution is proposed:
>  """
>  The easiest thing to do in C is to declare a struct:
>
>  typedef struct {
>   PetscScalar v[3];
>   PetscScalar p;
>  } Space;
>
>  and then cast pointers
>
>   Space ***array;
>
>   DAVecGetArray(da, u, (void *) &array);
>
>      array[k][j][i].v *= -1.0;
>  """
>
>  The problem with the proposed solution, is that they use a struct to
>  get the individual values, but what if you don't know the number of
>  degrees of freedom at compile time?
>
>  So my question is two fold:
>  a) Is there a problem with just having a single DA and calling
>  DACreateGlobalVector multiple times? Does this affect performance at
>  all (I have many different vectors)?
>  b) Is there a way to use the dof parameter when creating a DA when the
>  number of degrees of freedom is not known at compile time?
>  Specifically, I would like to be able to access the individual values
>  of the vector, just like the example shows...
>
>
>  2) The code I am writing has a lot of different parts which present a
>  lot of opportunities to overlap communication an computation when
>  scattering vectors to update values in the ghost points. Right now,
>  all of my vectors (there are ~50 of them) share a single DA because
>  they all have the same shape. However, by sharing a single DA, I can
>  only scatter one vector at a time. It would be nice to be able to
>  start scattering each vector right after I'm done computing it, and
>  finish scattering it right before I need it again but I can't because
>  other vectors might need to be scattered in between. I then re-wrote
>  part of my code so that each vector had its own DA object, but this
>  ended up being incredibly slow (I assume this is because I have so
>  many vectors).
>
>  My question is, is there a way to scatter multiple vectors
>  simultaneously without affecting the performance of the code? Does it
>  make sense to do this?
>
>
>  I'd really appreciate any help...
>
>  Thanks
>  Milad Fatenejad
>
>
>
>