[petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results!

[Analabha Roy]

Progress Report:

 So I modified my code to remove all the wrong usages of MatCopy where I
was trying to AllScatter/AllGather a parallel matrix, and replaced with
usage of MatGetRedundantMatrix(). The highlighted diff is
here<http://goo.gl/yyJVcV>
.

Now, running it with 1 proc crashes, I guess because the documentation
says<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetRedundantMatrix.html>that
only parallel matrices are supported by MatGetRedundantMatrix(), and
running with 1 proc means the parallel matrices are sequantial (from what I
read in the PetSc users manual).

However, running it with 2,3,4,5+ procs now yields identical and consistent
results (unlike before), so the problem appears to have been solved.

I would be very grateful if an extra pair of eyes could take a quick gander
at the diff and confirm if my changes are correct.

Thanks a bunch and Regards,





On Wed, Sep 25, 2013 at 12:55 PM, Analabha Roy <hariseldon99 at gmail.com>wrote:

>
>
>
> On Wed, Sep 25, 2013 at 12:12 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Analabha Roy <hariseldon99 at gmail.com> writes:
>>
>> > There is one thing
>> >
>> > In the code, the evaluation of each element of AVG_BIBJ requires a
>> > read-only matrix U_parallel that I input from another program, and a
>> > writeable sequential vector BDB_AA that is different for each element.
>> >
>> > I sequentiate U_parallel to U_seq by using MatCopy here in lines
>> > 242+<
>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#242
>> >
>>
>>   /*
>>      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>> - - - - - - - - - - - - -
>>      Create a sequential matrix and copy U_parallel to it
>>      so that each processor has complete local copy of U
>>      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>> - - - - - - - - - - - - -
>>    */
>>   ierr = MatCreateSeqAIJ (PETSC_COMM_SELF, dim, dim, 0, PETSC_NULL,
>> &U_seq);
>>   CHKERRQ (ierr);
>>   ierr = MatSetOption (U_seq, MAT_NEW_NONZERO_ALLOCATION_ERR,
>> PETSC_FALSE);
>>   CHKERRQ (ierr);
>>   ierr = MatCopy (U_parallel, U_seq, DIFFERENT_NONZERO_PATTERN);
>>
>> This is wrong and should raise an error (definitely in a debugging
>> build).  Are you intending to create a redundant matrix containing all
>> of U_parallel replicated onto each process?  If so, you may as well
>> assemble the whole thing redundantly, but if you'd rather distribute the
>> computation of entries, you can use MatGetRedundantMatrix().  If you
>> want something else, what is it?
>>
>>
>
> Thanks for the informative post. You're right in that I need to distribute
> the computation of entries, but need the data scattered/gathered to all
> procs afterwards. I thought MatCopy would do this. I'm not surprised that
> this is wrong. This part is probably what's wrong with the code.
>
> I'm looking at MatGetRedundantMatrix() docs and using it to fix the
> problem. Will post results as soon as I'm done...
>
>
>
>>   CHKERRQ (ierr);
>>   ierr = MatAssemblyBegin (U_seq, MAT_FINAL_ASSEMBLY);
>>   CHKERRQ (ierr);
>>   ierr = MatAssemblyEnd (U_seq, MAT_FINAL_ASSEMBLY);
>>   CHKERRQ (ierr);
>>
>>
>> > and each process is supposed to update its copy of BDB_AA at every loop
>> > iteration here in line
>> > 347+<
>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#347
>> >
>> >
>> >
>> > Is this right? Or are sequential vectors/matrices handled by the root
>> > process only?
>>
>> PETSC_COMM_SELF means the current process, so every process does that
>> stuff.
>>
>> > I know how to scatter a parallel vector to all processes using PetSc
>> > scatter contexts but don't see any way to do so to a matrix other than
>> > MatCopy. How do I ensure that each process has its own private
>> > writeable copy of a sequential vector?
>>
>> Is the last sentence supposed to say "vector"?  If so, see
>> VecScatterCreateToAll.
>>
>
>
>
> --
> ---
> *Analabha Roy*
> C.S.I.R <http://www.csir.res.in>  Senior Research Associate<http://csirhrdg.res.in/poolsra.htm>
> Saha Institute of Nuclear Physics <http://www.saha.ac.in>
> Section 1, Block AF
> Bidhannagar, Calcutta 700064
> India
> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com
> *Webpage*: http://www.ph.utexas.edu/~daneel/
>



-- 
---
*Analabha Roy*
C.S.I.R <http://www.csir.res.in>  Senior Research
Associate<http://csirhrdg.res.in/poolsra.htm>
Saha Institute of Nuclear Physics <http://www.saha.ac.in>
Section 1, Block AF
Bidhannagar, Calcutta 700064
India
*Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com
*Webpage*: http://www.ph.utexas.edu/~daneel/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130925/b088f4f2/attachment-0001.html>