[petsc-users] Unequal sparse matrix row distribution for MPI MatMult

[Steena M]

Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step. 
--------------------------------------------
On Sun, 3/29/15, Barry Smith <bsmith at mcs.anl.gov> wrote:

 Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
 To: "Steena M" <stm8086 at yahoo.com>
 Cc: "Matthew Knepley" <knepley at gmail.com>, petsc-users at mcs.anl.gov
 Date: Sunday, March 29, 2015, 9:26 PM
 
 
 > On
 Mar 29, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com>
 wrote:
 > 
 > Thanks
 Matt. I used PETSC_DETERMINE but I'm now getting an
 allocation-based error:
 > 
 > [0]PETSC ERROR: ---------------------
 Error Message ------------------------------------
 > [0]PETSC ERROR: Argument out of range!
 > [0]PETSC ERROR: New nonzero at (2,18)
 caused a malloc!
 > [0]PETSC ERROR:
 ------------------------------------------------------------------------
 > 
 > I tried
 preallocating on each rank for the diagonal and off diagonal
 section of the matrix as the next step  My current
 approximations for preallocation
 > 
 > CHKERRQ(
 MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT)); 
 
 
    These
 arguments where you pass PETSC_DEFAULT are expecting a
 pointer not an integer. You can pass NULL in those
 locations. Though it is better to provide the correct
 preallocation rather than some defaults.
 
   Barry
 
 > 
 > are throwing segmentation errors.
 > 
 > [0]PETSC ERROR: 
 Caught signal number 11 SEGV: Segmentation Violation,
 probably memory access out of range
 > 
 > Any insights into what I'm doing
 wrong?
 > 
 > Thanks,
 > Steena
 > 
 > 
 > 
 > On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com>
 wrote:
 > 
 > Subject:
 Re: [petsc-users] Unequal sparse matrix row distribution for
 MPI MatMult
 > To: "Steena M"
 <stm8086 at yahoo.com>
 > Cc: "Barry Smith" <bsmith at mcs.anl.gov>,
 petsc-users at mcs.anl.gov
 > Date: Sunday, March 29, 2015, 10:02 PM
 > 
 > On Sun, Mar 29, 2015
 at
 > 9:56 PM, Steena M <stm8086 at yahoo.com>
 > wrote:
 > Hi
 > Barry,
 > 
 > 
 > 
 > I am trying to partition a 20 row and 20
 col sparse matrix
 > between two procs
 such that proc 0 has 15 rows and 20 cols
 > and proc 1 has 5 rows and 20 cols. The
 code snippet:
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 >     
    CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A));
 //
 > at runtime: -matload_block_size 1
 > 
 > 
 > 
 > 
 > 
 >     
    if (rank ==0)
 > 
 >         {
 > 
 >             
    CHKERRQ( MatSetSizes(A, 15, 20, 20,
 > 20) ); //rank 0 gets 75% of the rows
 > 
 >             
    CHKERRQ( MatSetType(A, MATMPIBAIJ)
 > );
 > 
 >                 CHKERRQ(
 MatLoad(A,fd) );
 > 
 >          }
 > 
 > 
 > 
 >         else
 > 
 >     
    {
 > 
 > 
                CHKERRQ( MatSetSizes(A, 5,
 20, 20,
 > 20) ); //rank 1 gets 25% of the
 rows
 > 
 >       
          CHKERRQ( MatSetType(A, MATMPIBAIJ)
 > );
 > 
 >                 CHKERRQ(
 MatLoad(A,fd) );
 > 
 >       }
 >
 
 > 
 > 
 > This throws the following error (probably
 from psplit.c):
 > 
 >
 [1]PETSC ERROR: --------------------- Error Message
 > ------------------------------------
 > 
 > [1]PETSC ERROR:
 Nonconforming object sizes!
 > 
 > [1]PETSC ERROR: Sum of local lengths 40
 does not equal
 > global length 20, my
 local length 20
 > 
 >   likely a call to
 VecSetSizes() or MatSetSizes() is
 >
 wrong.
 > 
 > See
 http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
 > 
 > 
 > 
 > This error printout
 doesn't quite make sense to me.
 >
 I'm trying to specify a total matrix size of 20x20...
 I
 > haven't yet figured out where the
 '40' comes
 > from in the error
 message.
 > 
 > 
 > 
 > Any thoughts on what
 might be going wrong?
 > 
 > Its the column specification. Just
 > use PETSC_DETERMINE for the local columns
 since all our
 > sparse matrixformats are
 row divisions
 > anyway.
 >  
 > Thanks,
 >    
 >  Matt 
 > Thanks in advance,
 >
 
 > Steena
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 >
 --------------------------------------------
 > 
 > On Sun, 3/22/15,
 Barry Smith <bsmith at mcs.anl.gov>
 > wrote:
 > 
 > 
 > 
 >  Subject: Re: [petsc-users] Unequal
 sparse matrix row
 > distribution for MPI
 MatMult
 > 
 >  To:
 "Steena M" <stm8086 at yahoo.com>
 > 
 >  Cc: petsc-users at mcs.anl.gov
 > 
 >  Date: Sunday,
 March 22, 2015, 3:58 PM
 > 
 > 
 > 
 > 
 > 
 >   
 > 
 >  Steena,
 > 
 > 
 > 
 >     I am
 >
 
 >  a little unsure of your question.
 
 > 
 > 
 > 
 >     1)
 you can create a MPIBAIJ
 > 
 >  matrix with any distribution of block
 rows per process
 > you
 > 
 >  want, just set the
 local row size for each process to
 >
 be
 > 
 >  what you
 like.  Use MatCreateVecs() to get
 >
 correspondingly
 > 
 > 
 laid out vectors.
 > 
 >
 
 > 
 > 
    or 2) if you have a MPIBAIJ
 > 
 >  matrix with
 "equal" row layout and you want a
 > new
 > 
 >  one with uneven row layout you can
 simply use
 > 
 > 
 MatGetSubMatrix() to create that new matrix.
 > 
 > 
 > 
 >    Barry
 > 
 > 
 > 
 >  Unless you have
 another reason to have the
 > 
 >  matrix with an equal number row layout I
 would just
 > generate
 > 
 >  the matrix with
 the layout you want.
 > 
 > 
 > 
 > 
 > 
 >  > On Mar 22, 2015, at 5:50 PM, Steena
 M
 > 
 >  <stm8086 at yahoo.com>
 > 
 >  wrote:
 > 
 >  >
 > 
 >  > Hello,
 > 
 >  >
 > 
 >  > I need to
 distribute
 > 
 >  a
 sparse matrix such that each proc owns an unequal
 > number
 > 
 >  of blocked rows before I proceed with
 MPI MatMult. My
 > 
 > 
 initial thoughts on doing this:
 > 
 >  >
 > 
 >  > 1) Use  MatGetSubMatrices() on the
 test
 > 
 >  MATMPIBAIJ
 matrix to produce a new matrix where each
 > proc
 > 
 >  has an unequal number of rows.
 > 
 >  >
 > 
 >  > 2) Provide
 scatter context for vector X
 > 
 >  (for MatMult )using IS iscol from
 MatGetSubMatrices()
 > while
 > 
 >  creating the
 vector X.
 > 
 > 
 >
 > 
 >  > 3)
 Call MatMult()
 > 
 > 
 >
 > 
 >  > Will
 MatMult_MPIBAIJ continue to scatter
 > 
 >  this matrix and vector such that each
 proc will own an
 > equal
 > 
 >  number of matrix
 rows and corresponding diagonal vector
 >
 
 >  elements? Should I write my own
 MPIMatMult function to
 > 
 >  retain my redistribution of the matrix
 and vector?
 > 
 > 
 >
 > 
 >  >
 Thanks in
 > 
 > 
 advance,
 > 
 >  >
 Steena
 > 
 > 
 > 
 > 
 > 
 > 
 > -- 
 > What most
 experimenters
 > take for granted before
 they begin their experiments is
 >
 infinitely more interesting than any results to which
 their
 > experiments lead.
 > -- Norbert
 > Wiener
 >