Parallel matrix assembly - SetValues Problem?

[Barry Smith]

    Send the code to petsc-maint at mcs.anl.gov and we'll take a look
at it.

    Barry

On Feb 19, 2008, at 4:12 AM, Bernhard Kubicek wrote:

> Dear List,
>
> sorry to bother you  but I just finished reading the whole archive and
> couldn't find a solution to a problem of mine that keeps on  
> bothering me
> now for 7+ days.
>
> The problem is that my code produces different matrices if run in
> parallel or single cpu.
>
> I do a manual partitioning of the mesh by using metis by hand.
> Thereafter, there is a list of finite-volume elements that I want to  
> be
> stored on the individual cpu and a renumbering that is manged somehow.
> I create my matrix with
>
> MatCreateMPIAIJ(PETSC_COMM_WORLD,mycount,mycount,
> PETSC_DETERMINE,PETSC_DETERMINE,50,PETSC_NULL,50,PETSC_NULL,&A),
>
> where mycount is different on each cpu, and is the mentioned number of
> elements I wish to have there locally.
>
>
> for each local row/element at the same time let a user calculate the
> matrix elements and column positions, and the right hand side values  
> for
> this row. I output those for debugging. Within the loop for the local
> rows, I call
> MatSetValues(A,1,&i,nrEntries,entries,v,INSERT_VALUES)
> VecSetValue(rhs,i,rhsval,INSERT_VALUES)
> in this order
>
> When I run on one cpu, everything works nicely. A 3d mesh of a
> 10-element long bar with each element having volume 1, creates the
> following matrix:
> -2.  1.  0.  0.  0.  0.  0.  0.  0.  1.
> 1.  -2.  1.  0.  0.  0.  0.  0.  0.  0.
> 0.  1.  -2.  1.  0.  0.  0.  0.  0.  0.
> 0.  0.  1.  -2.  1.  0.  0.  0.  0.  0.
> 0.  0.  0.  1.  -3.  0.  0.  0.  0.  0.
> 0.  0.  0.  0.  0.  -3.  1.  0.  0.  0.
> 0.  0.  0.  0.  0.  1.  -2.  1.  0.  0.
> 0.  0.  0.  0.  0.  0.  1.  -2.  1.  0.
> 0.  0.  0.  0.  0.  0.  0.  1.  -2.  1.
> 1.  0.  0.  0.  0.  0.  0.  0.  1.  -2.
>
> rhs
> 0
> 0
> 0
> 0
> 0
> -2
> 0
> 0
> 0
> 0
> The CPU sets the matrix and rhs like this ( global matrix row:
> column/value ...column/value | rhs-value )
> Row 0 cols:0/-2	9/1 1/1		|	0
> Row 1 cols:1/-2	0/1 2/1		|	0
> Row 2 cols:2/-2	1/1 3/1		|	0
> Row 3 cols:3/-2	2/1 4/1		|	0
> Row 4 cols:4/-3	3/1	|	0
> Row 5 cols:5/-3	6/1	|	-2
> Row 6 cols:6/-2	5/1 7/1		|	0
> Row 7 cols:7/-2	6/1 8/1		|	0
> Row 8 cols:8/-2	7/1 9/1		|	0
> Row 9 cols:9/-2	0/1 8/1		|	0
> because of the meshing the central rows in the matrix are the most
> exterior elements, on which wall boundary condition 0 and 1 are set
> (laplace equation).
>
> one 2 cpus,
> the matrix looses is different, although the global-local element
> renumbering is defacto nonexisting (cpu 0: rows 0-4, cpu 1: rows 5-9):
> 1.  1.  0.  0.  0.  0.  0.  0.  0.  0.
> 1.  -2.  1.  0.  0.  0.  0.  0.  0.  0.
> 0.  1.  -2.  1.  0.  0.  0.  0.  0.  0.
> 0.  0.  1.  -2.  1.  0.  0.  0.  0.  0.
> 0.  0.  0.  1.  -3.  0.  0.  0.  0.  0.
> 0.  0.  0.  0.  0.  -3.  1.  0.  0.  0.
> 0.  0.  0.  0.  0.  1.  -2.  1.  0.  0.
> 0.  0.  0.  0.  0.  0.  1.  -2.  1.  0.
> 0.  0.  0.  0.  0.  0.  0.  1.  -2.  1.
> 1.  0.  0.  0.  0.  0.  0.  0.  1.  -2.
> Process [0]
> 0
> 0
> 0
> 0
> 0
> Process [1]
> -2
> 0
> 0
> 0
> 0
> here cpu 0 sets:
> Row 0 cols:0/-2	9/1 1/1	|0
> Row 1 cols:1/-2	0/1 2/1	|0
> Row 2 cols:2/-2	1/1 3/1	|0
> Row 3 cols:3/-2	2/1 4/1	|0
> Row 4 cols:4/-3	3/1 |0
>
> and cpu 1:
> Row 5 cols:5/-3	6/1|-2
> Row 6 cols:6/-2	5/1 7/1	|0
> Row 7 cols:7/-2	6/1 8/1	|0
> Row 8 cols:8/-2	7/1 9/1	|0
> Row 9 cols:9/-2	0/1 8/1	|0
> I triple verified that ***SetValues is called with the exactly same
> values as on one cpu, and that nothing is set twice, and that every  
> cpu
> sets it's correct columns. Also for more sophisticated renumberings
>
> Attached are the outputs when run with with -info.
>
> My current guess is that I create the matrix falsely, or that I cannot
> mix the setting of Vec and Mat values before their
> respective ???AssemblyBegin/Ends.
>
> If anyone has any idea where the problem is, is would be extremely  
> nice
> to help me here.
>
> Thank you very much, even for the slightest help
> Bernhard Kubicek
>
> ------
> Physics Doctorate Student Techn. University of Vienna, Austria
> Freelancer arsenal research, Vienna Austria
>
>
>
>
>
> <par.txt><seq.txt>