common nodla points on neighbouring subdomains

[Thomas Geenen]

On Wednesday 17 May 2006 14:53, Barry Smith wrote:
>    Thomas,
>
>      I think you might misunderstand the function of XXXSetValuesLocal().
> What happens with them is the index is mapped to the global index and
> the value is then put into the "correct" global location, so some values
> will be shipped off to the "owning" process.
this happens in the assemble phase right?
>
> 1) If two processes that
> "share" a vertex both generate the entire stiffness for that vertex
> (that is, each process loops over all the elements containing the
> vertex summing the stiffness for that vertex) then the result will
> be exactly twice the expected value (since one process will send the
> result to the "owning" process). I THINK this is what you are doing.
I intercept the matrix and rhs just before the "old" call to the solver 
routine just before the call the value for the rhs of common nodalpoints are 
calculated globally and broadcasted to the owning processes. In this setup 
more than one process can own a nodal point. I construct a local2global 
mapping and apply it to the matrix and vector

ISLocalToGlobalMapping ltog;
ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, nc, local2global, &ltog);
MatSetLocalToGlobalMapping(A, ltog);

>
> 2) if, on the other hand, each process generates only the part of the
> stiffness for the vertex from elements that it "owns" then the global
> matrix will be correct, BUT the communication of matrix elements to the
> "owning" process is done. If you DO NOT want the communication to the
> owning process but want to keep part of the stiffness on each process (for
> a shared vertex) I would call this the unassembled form. This is done by
> some people but is not really supported by PETSc.
>
> I think you may want to consider NOT having "shared" vertices (that is
> either 1 if you are using that or 2). This may not require much change to
> what you do not want to change but will allow you to easily use all of the
> PETSc solvers. The problem with "unassembled" approaches is that they save
> a bit of communication in generating the stiffness but then (since the
> matrix is never assembled) cannot form much of anything in terms of
> preconditioners, thus you do lots of linear solver iterations with
> communication. Communicating the stiffness initially (if you have a good
> partitioning of elements) is really not much communication and IMHO is the
> wrong place to optimize.
basically what happens in my code is that each process owns all nodalpoints of 
the element it owns. I tell petsc the global numbering of the matrix and 
vector entries so I can use all the preconditioners and solvers etc. However 
in the original setup the program expects on return to have the solution 
vector for all nodes of the elements it owns . Petsc gives the solution to 
the "owner process" of the nodal point. To correct this I would have to send 
parts of the solution to processes that don't "own" that part of the solution 
but do expect it to be there. I thought It would be elegant if Petsc could 
return a solution vector that corresponds to the local filling of the vector 
not to the "owners" of the nodal point.

thanks for you fast response

Thomas Geenen

>
>    I hope my response is not too confusing, I may have misunderstood
> parts of your question.
>
>     Barry
>
> On Wed, 17 May 2006, Thomas Geenen wrote:
> > Dear Petsc users,
> >
> > I am solving a system of equations Ax=b resulting from a Finite element
> > package.  The parallelization strategy for this program is to minimize
> > communication. therefore each subdomain has its own copy of common nodal
> > points between subdomains (local numbering). This strategy is already
> > implemented and I don't want to change that
> >
> > I fill the matrix and right hand side with a call to MatSetValuesLocal
> > and VecSetValuesLocal. Some entries are filled on different subdomains
> > (of course with the same value)
> >
> > The solution for the first subdomain is correct. however the solution
> > vector on the second subdomain does not contain the solution on the
> > common nodal points it shares with subdomain 1.
> > This is probably a feature but it is not consistent with my program
> > setup. The question is:
> > how can i tell petsc to return a solution for all the positions i filled
> > with VecSetValuesLocal?
> >
> > Thanks in advance
> > Thomas Geenen
> > ps I got the impression that the assemble routines also exchange common
> > nodal point info.  It would be nice if this could be skipped as well.
> > Of course some extra communication could solve my problem but I would
> > prefer a more elegant solution if one exists