[petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results!

[Analabha Roy]

On Wed, Sep 25, 2013 at 12:12 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> Analabha Roy <hariseldon99 at gmail.com> writes:
>
> > There is one thing
> >
> > In the code, the evaluation of each element of AVG_BIBJ requires a
> > read-only matrix U_parallel that I input from another program, and a
> > writeable sequential vector BDB_AA that is different for each element.
> >
> > I sequentiate U_parallel to U_seq by using MatCopy here in lines
> > 242+<
> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#242>
>
>   /*
>      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> - - - - - - - - - - - -
>      Create a sequential matrix and copy U_parallel to it
>      so that each processor has complete local copy of U
>      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> - - - - - - - - - - - -
>    */
>   ierr = MatCreateSeqAIJ (PETSC_COMM_SELF, dim, dim, 0, PETSC_NULL,
> &U_seq);
>   CHKERRQ (ierr);
>   ierr = MatSetOption (U_seq, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);
>   CHKERRQ (ierr);
>   ierr = MatCopy (U_parallel, U_seq, DIFFERENT_NONZERO_PATTERN);
>
> This is wrong and should raise an error (definitely in a debugging
> build).  Are you intending to create a redundant matrix containing all
> of U_parallel replicated onto each process?  If so, you may as well
> assemble the whole thing redundantly, but if you'd rather distribute the
> computation of entries, you can use MatGetRedundantMatrix().  If you
> want something else, what is it?
>
>

Thanks for the informative post. You're right in that I need to distribute
the computation of entries, but need the data scattered/gathered to all
procs afterwards. I thought MatCopy would do this. I'm not surprised that
this is wrong. This part is probably what's wrong with the code.

I'm looking at MatGetRedundantMatrix() docs and using it to fix the
problem. Will post results as soon as I'm done...



>   CHKERRQ (ierr);
>   ierr = MatAssemblyBegin (U_seq, MAT_FINAL_ASSEMBLY);
>   CHKERRQ (ierr);
>   ierr = MatAssemblyEnd (U_seq, MAT_FINAL_ASSEMBLY);
>   CHKERRQ (ierr);
>
>
> > and each process is supposed to update its copy of BDB_AA at every loop
> > iteration here in line
> > 347+<
> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#347>
> >
> >
> > Is this right? Or are sequential vectors/matrices handled by the root
> > process only?
>
> PETSC_COMM_SELF means the current process, so every process does that
> stuff.
>
> > I know how to scatter a parallel vector to all processes using PetSc
> > scatter contexts but don't see any way to do so to a matrix other than
> > MatCopy. How do I ensure that each process has its own private
> > writeable copy of a sequential vector?
>
> Is the last sentence supposed to say "vector"?  If so, see
> VecScatterCreateToAll.
>



-- 
---
*Analabha Roy*
C.S.I.R <http://www.csir.res.in>  Senior Research
Associate<http://csirhrdg.res.in/poolsra.htm>
Saha Institute of Nuclear Physics <http://www.saha.ac.in>
Section 1, Block AF
Bidhannagar, Calcutta 700064
India
*Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com
*Webpage*: http://www.ph.utexas.edu/~daneel/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130925/c51a8213/attachment.html>