[petsc-users] Fwd: Building the same petsc matrix with different numprocs gives different results!

[Matthew Knepley]

On Tue, Sep 24, 2013 at 8:08 AM, Analabha Roy <hariseldon99 at gmail.com>wrote:

>
> On Tue, Sep 24, 2013 at 1:42 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>
>> Analabha Roy <hariseldon99 at gmail.com> writes:
>>
>> > Hi all,
>> >
>> >
>> > Compiling and running this
>> > code<
>> https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c
>> >that
>> > builds a petsc matrix gives different results when run with different
>> > number of processors.
>>
>>
> Thanks for the reply.
>
>
>>  Uh, if you call rand() on different processors, why would you expect it
>> to give the same results?
>>
>> Right, I get that. The rand() was a placeholder.
>
> This original much larger code<https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c>replicates the same loop structure and runs the same Petsc subroutines, but
> running it by
>
> mpirun -np $N ./eth -lattice_size 5 -vector_size 1 -repulsion 0.0
> -draw_out -draw_pause -1
>
> with N=1,2,3,4 gives different results for the matrix dumped out by lines
> 514-519<https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#514>.
> The matrix itself is evaluated in parallel, created in lines 263-275
> <https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#263>and
> evaluated in lines 294-356<https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#294>
>
> (you can click on the line numbers above to navigate directly to them)
>
> Here is a sample <http://i43.tinypic.com/zyhf2f.jpg> of the output of
> lines 514-519<https://code.google.com/p/daneelrepo/source/browse/eth_question/eth.c#514>  for N=1,2,3,4 procs left to right.
>
> Thty're different for different procs. They should be the same, since none
> of my input parameters are numprocs dependent, and I don't explicitly use
> the size or rank anywhere in the code.
>

You are likely not dividing the rows you loop over so you are redundantly
computing.

   Matt


>
>
>> for (sitestride1 = Istart; sitestride1 < Iend; sitestride1++)
>>     {
>>       for (sitestride2 = 0; sitestride2 < matsize; sitestride2++)
>>         {
>>           for (alpha = 0; alpha < dim; alpha++)
>>             {
>>               for (mu = 0; mu < dim; mu++)
>>                 for (lambda = 0; lambda < dim; lambda++)
>>                   {
>>                     vecval = rand () / rand ();
>>                   }
>>
>>               VecSetValue (BDB_AA, alpha, vecval, INSERT_VALUES);
>>
>>             }
>>           VecAssemblyBegin (BDB_AA);
>>           VecAssemblyEnd (BDB_AA);
>>           VecSum (BDB_AA, &element);
>>           colvalues[sitestride2] = element;
>>
>>         }
>>       //Insert the array of colvalues to the sitestride1^th row of H
>>       MatSetValues (AVG_BDIBJ, 1, &sitestride1, matsize, idx, colvalues,
>>                     INSERT_VALUES);
>>
>>     }
>>
>> > The code is large and complex, so I have created a smaller program
>> > with the same
>> > loop structure here. <http://pastebin.ca/2457643>
>> >
>> > Compile it and run it with "mpirun -np $N ./test -draw_pause -1" gives
>> > different results for different values of N even though it's not
>> supposed
>> > to.
>>
>> What do you expect to see?
>>
>> > Here is a sample output <http://i42.tinypic.com/2s16ccw.jpg> for
>> N=1,2,3,4
>> > from left to right.
>> >
>> > Can anyone guide me as to what I'm doing wrong? Are any of the petssc
>> > routines used not parallelizable?
>> >
>> > Thanks in advance,
>> >
>> > Regards.
>> >
>> > --
>> > ---
>> > *Analabha Roy*
>> > C.S.I.R <http://www.csir.res.in>  Senior Research
>> > Associate<http://csirhrdg.res.in/poolsra.htm>
>> > Saha Institute of Nuclear Physics <http://www.saha.ac.in>
>> > Section 1, Block AF
>> > Bidhannagar, Calcutta 700064
>> > India
>> > *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com
>> > *Webpage*: http://www.ph.utexas.edu/~daneel/
>>
>
>
>
> --
> ---
> *Analabha Roy*
> C.S.I.R <http://www.csir.res.in>  Senior Research Associate<http://csirhrdg.res.in/poolsra.htm>
> Saha Institute of Nuclear Physics <http://www.saha.ac.in>
> Section 1, Block AF
> Bidhannagar, Calcutta 700064
> India
> *Emails*: daneel at physics.utexas.edu, hariseldon99 at gmail.com
> *Webpage*: http://www.ph.utexas.edu/~daneel/
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130924/e9b25493/attachment.html>