[petsc-users] Unequal sparse matrix row distribution for MPI MatMult

[Barry Smith]

> On Mar 29, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com> wrote:
> 
> Thanks Matt. I used PETSC_DETERMINE but I'm now getting an allocation-based error:
> 
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (2,18) caused a malloc!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> 
> I tried preallocating on each rank for the diagonal and off diagonal section of the matrix as the next step  My current approximations for preallocation
> 
> CHKERRQ( MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT));  

   These arguments where you pass PETSC_DEFAULT are expecting a pointer not an integer. You can pass NULL in those locations. Though it is better to provide the correct preallocation rather than some defaults.

  Barry

> 
> are throwing segmentation errors.
> 
> [0]PETSC ERROR:  Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> 
> Any insights into what I'm doing wrong?
> 
> Thanks,
> Steena
> 
> 
> 
> On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com> wrote:
> 
> Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
> To: "Steena M" <stm8086 at yahoo.com>
> Cc: "Barry Smith" <bsmith at mcs.anl.gov>, petsc-users at mcs.anl.gov
> Date: Sunday, March 29, 2015, 10:02 PM
> 
> On Sun, Mar 29, 2015 at
> 9:56 PM, Steena M <stm8086 at yahoo.com>
> wrote:
> Hi
> Barry,
> 
> 
> 
> I am trying to partition a 20 row and 20 col sparse matrix
> between two procs such that proc 0 has 15 rows and 20 cols
> and proc 1 has 5 rows and 20 cols. The code snippet:
> 
> 
> 
> 
> 
> 
> 
>         CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A)); //
> at runtime: -matload_block_size 1
> 
> 
> 
> 
> 
>         if (rank ==0)
> 
>         {
> 
>                 CHKERRQ( MatSetSizes(A, 15, 20, 20,
> 20) ); //rank 0 gets 75% of the rows
> 
>                 CHKERRQ( MatSetType(A, MATMPIBAIJ)
> );
> 
>                 CHKERRQ( MatLoad(A,fd) );
> 
>          }
> 
> 
> 
>         else
> 
>         {
> 
>                 CHKERRQ( MatSetSizes(A, 5, 20, 20,
> 20) ); //rank 1 gets 25% of the rows
> 
>                 CHKERRQ( MatSetType(A, MATMPIBAIJ)
> );
> 
>                 CHKERRQ( MatLoad(A,fd) );
> 
>       }
> 
> 
> 
> This throws the following error (probably from psplit.c):
> 
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> 
> [1]PETSC ERROR: Nonconforming object sizes!
> 
> [1]PETSC ERROR: Sum of local lengths 40 does not equal
> global length 20, my local length 20
> 
>   likely a call to VecSetSizes() or MatSetSizes() is
> wrong.
> 
> See http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
> 
> 
> 
> This error printout doesn't quite make sense to me.
> I'm trying to specify a total matrix size of 20x20... I
> haven't yet figured out where the '40' comes
> from in the error message.
> 
> 
> 
> Any thoughts on what might be going wrong?
> 
> Its the column specification. Just
> use PETSC_DETERMINE for the local columns since all our
> sparse matrixformats are row divisions
> anyway.
>  
> Thanks,
>    
>  Matt 
> Thanks in advance,
> 
> Steena
> 
> 
> 
> 
> 
> 
> 
> --------------------------------------------
> 
> On Sun, 3/22/15, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> 
> 
> 
>  Subject: Re: [petsc-users] Unequal sparse matrix row
> distribution for MPI MatMult
> 
>  To: "Steena M" <stm8086 at yahoo.com>
> 
>  Cc: petsc-users at mcs.anl.gov
> 
>  Date: Sunday, March 22, 2015, 3:58 PM
> 
> 
> 
> 
> 
>   
> 
>  Steena,
> 
> 
> 
>     I am
> 
>  a little unsure of your question. 
> 
> 
> 
>     1) you can create a MPIBAIJ
> 
>  matrix with any distribution of block rows per process
> you
> 
>  want, just set the local row size for each process to
> be
> 
>  what you like.  Use MatCreateVecs() to get
> correspondingly
> 
>  laid out vectors.
> 
> 
> 
>     or 2) if you have a MPIBAIJ
> 
>  matrix with "equal" row layout and you want a
> new
> 
>  one with uneven row layout you can simply use
> 
>  MatGetSubMatrix() to create that new matrix.
> 
> 
> 
>    Barry
> 
> 
> 
>  Unless you have another reason to have the
> 
>  matrix with an equal number row layout I would just
> generate
> 
>  the matrix with the layout you want.
> 
> 
> 
> 
> 
>  > On Mar 22, 2015, at 5:50 PM, Steena M
> 
>  <stm8086 at yahoo.com>
> 
>  wrote:
> 
>  >
> 
>  > Hello,
> 
>  >
> 
>  > I need to distribute
> 
>  a sparse matrix such that each proc owns an unequal
> number
> 
>  of blocked rows before I proceed with MPI MatMult. My
> 
>  initial thoughts on doing this:
> 
>  >
> 
>  > 1) Use  MatGetSubMatrices() on the test
> 
>  MATMPIBAIJ matrix to produce a new matrix where each
> proc
> 
>  has an unequal number of rows.
> 
>  >
> 
>  > 2) Provide scatter context for vector X
> 
>  (for MatMult )using IS iscol from MatGetSubMatrices()
> while
> 
>  creating the vector X.
> 
>  >
> 
>  > 3) Call MatMult()
> 
>  >
> 
>  > Will MatMult_MPIBAIJ continue to scatter
> 
>  this matrix and vector such that each proc will own an
> equal
> 
>  number of matrix rows and corresponding diagonal vector
> 
>  elements? Should I write my own MPIMatMult function to
> 
>  retain my redistribution of the matrix and vector?
> 
>  >
> 
>  > Thanks in
> 
>  advance,
> 
>  > Steena
> 
> 
> 
> 
> 
> 
> -- 
> What most experimenters
> take for granted before they begin their experiments is
> infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert
> Wiener
>