[petsc-users] Unequal sparse matrix row distribution for MPI MatMult
Barry Smith
bsmith at mcs.anl.gov
Tue Mar 31 23:15:00 CDT 2015
You should not need to call any preallocation routines when using MatLoad()
How did you generate the file? Are you sure it has the correct information for the matrix?
Barry
> On Mar 31, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com> wrote:
>
> Thanks Matt. I'm still getting the malloc error
>
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (2,18) caused a malloc!
>
> and
>
> a new incorrect matrix file error:
>
> [0]PETSC ERROR: Unexpected data in file!
> [0]PETSC ERROR: not matrix object!
>
> Maybe the order of calls is mixed up. This is the code snippet:
>
> if (rank ==0)
> {
> PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank);
>
> CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20));
> CHKERRQ(MatSetType(A, MATMPIBAIJ));
> CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL));
> CHKERRQ( MatLoad(A,fd));
> CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE));
> }
>
> else
> {
> PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank);
>
> CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) );
> CHKERRQ(MatSetType(A, MATMPIBAIJ));
> CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL));
> CHKERRQ(MatLoad(A,fd));
> CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE));
> }
>
> Is there something I'm missing?
>
> Thanks,
> Steena
>
>
>
>
>
> On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
>
> On Tue, Mar 31, 2015 at 6:51 PM, Steena M <stm8086 at yahoo.com> wrote:
> Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step.
>
> 1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE)
>
> 2) Note that this is never ever ever more efficient than making another pass and preallocating
>
> Thanks,
>
> Matt
> --------------------------------------------
> On Sun, 3/29/15, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
> To: "Steena M" <stm8086 at yahoo.com>
> Cc: "Matthew Knepley" <knepley at gmail.com>, petsc-users at mcs.anl.gov
> Date: Sunday, March 29, 2015, 9:26 PM
>
>
> > On
> Mar 29, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com>
> wrote:
> >
> > Thanks
> Matt. I used PETSC_DETERMINE but I'm now getting an
> allocation-based error:
> >
> > [0]PETSC ERROR: ---------------------
> Error Message ------------------------------------
> > [0]PETSC ERROR: Argument out of range!
> > [0]PETSC ERROR: New nonzero at (2,18)
> caused a malloc!
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> >
> > I tried
> preallocating on each rank for the diagonal and off diagonal
> section of the matrix as the next step My current
> approximations for preallocation
> >
> > CHKERRQ(
> MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT));
>
>
> These
> arguments where you pass PETSC_DEFAULT are expecting a
> pointer not an integer. You can pass NULL in those
> locations. Though it is better to provide the correct
> preallocation rather than some defaults.
>
> Barry
>
> >
> > are throwing segmentation errors.
> >
> > [0]PETSC ERROR:
> Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> >
> > Any insights into what I'm doing
> wrong?
> >
> > Thanks,
> > Steena
> >
> >
> >
> > On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com>
> wrote:
> >
> > Subject:
> Re: [petsc-users] Unequal sparse matrix row distribution for
> MPI MatMult
> > To: "Steena M"
> <stm8086 at yahoo.com>
> > Cc: "Barry Smith" <bsmith at mcs.anl.gov>,
> petsc-users at mcs.anl.gov
> > Date: Sunday, March 29, 2015, 10:02 PM
> >
> > On Sun, Mar 29, 2015
> at
> > 9:56 PM, Steena M <stm8086 at yahoo.com>
> > wrote:
> > Hi
> > Barry,
> >
> >
> >
> > I am trying to partition a 20 row and 20
> col sparse matrix
> > between two procs
> such that proc 0 has 15 rows and 20 cols
> > and proc 1 has 5 rows and 20 cols. The
> code snippet:
> >
> >
> >
> >
> >
> >
> >
> >
> CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A));
> //
> > at runtime: -matload_block_size 1
> >
> >
> >
> >
> >
> >
> if (rank ==0)
> >
> > {
> >
> >
> CHKERRQ( MatSetSizes(A, 15, 20, 20,
> > 20) ); //rank 0 gets 75% of the rows
> >
> >
> CHKERRQ( MatSetType(A, MATMPIBAIJ)
> > );
> >
> > CHKERRQ(
> MatLoad(A,fd) );
> >
> > }
> >
> >
> >
> > else
> >
> >
> {
> >
> >
> CHKERRQ( MatSetSizes(A, 5,
> 20, 20,
> > 20) ); //rank 1 gets 25% of the
> rows
> >
> >
> CHKERRQ( MatSetType(A, MATMPIBAIJ)
> > );
> >
> > CHKERRQ(
> MatLoad(A,fd) );
> >
> > }
> >
>
> >
> >
> > This throws the following error (probably
> from psplit.c):
> >
> >
> [1]PETSC ERROR: --------------------- Error Message
> > ------------------------------------
> >
> > [1]PETSC ERROR:
> Nonconforming object sizes!
> >
> > [1]PETSC ERROR: Sum of local lengths 40
> does not equal
> > global length 20, my
> local length 20
> >
> > likely a call to
> VecSetSizes() or MatSetSizes() is
> >
> wrong.
> >
> > See
> http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
> >
> >
> >
> > This error printout
> doesn't quite make sense to me.
> >
> I'm trying to specify a total matrix size of 20x20...
> I
> > haven't yet figured out where the
> '40' comes
> > from in the error
> message.
> >
> >
> >
> > Any thoughts on what
> might be going wrong?
> >
> > Its the column specification. Just
> > use PETSC_DETERMINE for the local columns
> since all our
> > sparse matrixformats are
> row divisions
> > anyway.
> >
> > Thanks,
> >
> > Matt
> > Thanks in advance,
> >
>
> > Steena
> >
> >
> >
> >
> >
> >
> >
> >
> --------------------------------------------
> >
> > On Sun, 3/22/15,
> Barry Smith <bsmith at mcs.anl.gov>
> > wrote:
> >
> >
> >
> > Subject: Re: [petsc-users] Unequal
> sparse matrix row
> > distribution for MPI
> MatMult
> >
> > To:
> "Steena M" <stm8086 at yahoo.com>
> >
> > Cc: petsc-users at mcs.anl.gov
> >
> > Date: Sunday,
> March 22, 2015, 3:58 PM
> >
> >
> >
> >
> >
> >
> >
> > Steena,
> >
> >
> >
> > I am
> >
>
> > a little unsure of your question.
>
> >
> >
> >
> > 1)
> you can create a MPIBAIJ
> >
> > matrix with any distribution of block
> rows per process
> > you
> >
> > want, just set the
> local row size for each process to
> >
> be
> >
> > what you
> like. Use MatCreateVecs() to get
> >
> correspondingly
> >
> >
> laid out vectors.
> >
> >
>
> >
> >
> or 2) if you have a MPIBAIJ
> >
> > matrix with
> "equal" row layout and you want a
> > new
> >
> > one with uneven row layout you can
> simply use
> >
> >
> MatGetSubMatrix() to create that new matrix.
> >
> >
> >
> > Barry
> >
> >
> >
> > Unless you have
> another reason to have the
> >
> > matrix with an equal number row layout I
> would just
> > generate
> >
> > the matrix with
> the layout you want.
> >
> >
> >
> >
> >
> > > On Mar 22, 2015, at 5:50 PM, Steena
> M
> >
> > <stm8086 at yahoo.com>
> >
> > wrote:
> >
> > >
> >
> > > Hello,
> >
> > >
> >
> > > I need to
> distribute
> >
> > a
> sparse matrix such that each proc owns an unequal
> > number
> >
> > of blocked rows before I proceed with
> MPI MatMult. My
> >
> >
> initial thoughts on doing this:
> >
> > >
> >
> > > 1) Use MatGetSubMatrices() on the
> test
> >
> > MATMPIBAIJ
> matrix to produce a new matrix where each
> > proc
> >
> > has an unequal number of rows.
> >
> > >
> >
> > > 2) Provide
> scatter context for vector X
> >
> > (for MatMult )using IS iscol from
> MatGetSubMatrices()
> > while
> >
> > creating the
> vector X.
> >
> >
> >
> >
> > > 3)
> Call MatMult()
> >
> >
> >
> >
> > > Will
> MatMult_MPIBAIJ continue to scatter
> >
> > this matrix and vector such that each
> proc will own an
> > equal
> >
> > number of matrix
> rows and corresponding diagonal vector
> >
>
> > elements? Should I write my own
> MPIMatMult function to
> >
> > retain my redistribution of the matrix
> and vector?
> >
> >
> >
> >
> > >
> Thanks in
> >
> >
> advance,
> >
> > >
> Steena
> >
> >
> >
> >
> >
> >
> > --
> > What most
> experimenters
> > take for granted before
> they begin their experiments is
> >
> infinitely more interesting than any results to which
> their
> > experiments lead.
> > -- Norbert
> > Wiener
>
> >
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
>
More information about the petsc-users
mailing list