[petsc-users] Unequal sparse matrix row distribution for MPI MatMult
Steena M
stm8086 at yahoo.com
Tue Mar 31 23:05:06 CDT 2015
Thanks Matt. I'm still getting the malloc error
[0]PETSC ERROR: Argument out of range![0]PETSC ERROR: New nonzero at (2,18) caused a malloc!
and
a new incorrect matrix file error:
[0]PETSC ERROR: Unexpected data in file![0]PETSC ERROR: not matrix object!
Maybe the order of calls is mixed up. This is the code snippet:
if (rank ==0) { PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); CHKERRQ(MatSetSizes(A, 15, PETSC_DETERMINE, 20, 20)); CHKERRQ(MatSetType(A, MATMPIBAIJ)); CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); CHKERRQ( MatLoad(A,fd)); CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); } else { PetscPrintf (PETSC_COMM_WORLD,"\n On rank %d ", rank); CHKERRQ( MatSetSizes(A, 5, PETSC_DETERMINE, 20, 20) ); CHKERRQ(MatSetType(A, MATMPIBAIJ)); CHKERRQ( MatMPIBAIJSetPreallocation(A,1,1,NULL,1,NULL)); CHKERRQ(MatLoad(A,fd)); CHKERRQ(MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE)); }
Is there something I'm missing?
Thanks,Steena
On Tuesday, March 31, 2015 6:10 PM, Matthew Knepley <knepley at gmail.com> wrote:
On Tue, Mar 31, 2015 at 6:51 PM, Steena M <stm8086 at yahoo.com> wrote:
Thanks Barry. I'm still getting the malloc error with NULL. Is there a way to distribute the matrix without explicit preallocation? Different matrices will be loaded during runtime and assigning preallocation parameters would mean an additional preprocessing step.
1) MatSetOption(MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) 2) Note that this is never ever ever more efficient than making another pass and preallocating
Thanks,
Matt
--------------------------------------------
On Sun, 3/29/15, Barry Smith <bsmith at mcs.anl.gov> wrote:
Subject: Re: [petsc-users] Unequal sparse matrix row distribution for MPI MatMult
To: "Steena M" <stm8086 at yahoo.com>
Cc: "Matthew Knepley" <knepley at gmail.com>, petsc-users at mcs.anl.gov
Date: Sunday, March 29, 2015, 9:26 PM
> On
Mar 29, 2015, at 11:05 PM, Steena M <stm8086 at yahoo.com>
wrote:
>
> Thanks
Matt. I used PETSC_DETERMINE but I'm now getting an
allocation-based error:
>
> [0]PETSC ERROR: ---------------------
Error Message ------------------------------------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: New nonzero at (2,18)
caused a malloc!
> [0]PETSC ERROR:
------------------------------------------------------------------------
>
> I tried
preallocating on each rank for the diagonal and off diagonal
section of the matrix as the next step My current
approximations for preallocation
>
> CHKERRQ(
MatMPIBAIJSetPreallocation(A,1,5,PETSC_DEFAULT,5,PETSC_DEFAULT));
These
arguments where you pass PETSC_DEFAULT are expecting a
pointer not an integer. You can pass NULL in those
locations. Though it is better to provide the correct
preallocation rather than some defaults.
Barry
>
> are throwing segmentation errors.
>
> [0]PETSC ERROR:
Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
>
> Any insights into what I'm doing
wrong?
>
> Thanks,
> Steena
>
>
>
> On Sun, 3/29/15, Matthew Knepley <knepley at gmail.com>
wrote:
>
> Subject:
Re: [petsc-users] Unequal sparse matrix row distribution for
MPI MatMult
> To: "Steena M"
<stm8086 at yahoo.com>
> Cc: "Barry Smith" <bsmith at mcs.anl.gov>,
petsc-users at mcs.anl.gov
> Date: Sunday, March 29, 2015, 10:02 PM
>
> On Sun, Mar 29, 2015
at
> 9:56 PM, Steena M <stm8086 at yahoo.com>
> wrote:
> Hi
> Barry,
>
>
>
> I am trying to partition a 20 row and 20
col sparse matrix
> between two procs
such that proc 0 has 15 rows and 20 cols
> and proc 1 has 5 rows and 20 cols. The
code snippet:
>
>
>
>
>
>
>
>
CHKERRQ(MatCreate(PETSC_COMM_WORLD,&A));
//
> at runtime: -matload_block_size 1
>
>
>
>
>
>
if (rank ==0)
>
> {
>
>
CHKERRQ( MatSetSizes(A, 15, 20, 20,
> 20) ); //rank 0 gets 75% of the rows
>
>
CHKERRQ( MatSetType(A, MATMPIBAIJ)
> );
>
> CHKERRQ(
MatLoad(A,fd) );
>
> }
>
>
>
> else
>
>
{
>
>
CHKERRQ( MatSetSizes(A, 5,
20, 20,
> 20) ); //rank 1 gets 25% of the
rows
>
>
CHKERRQ( MatSetType(A, MATMPIBAIJ)
> );
>
> CHKERRQ(
MatLoad(A,fd) );
>
> }
>
>
>
> This throws the following error (probably
from psplit.c):
>
>
[1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
>
> [1]PETSC ERROR:
Nonconforming object sizes!
>
> [1]PETSC ERROR: Sum of local lengths 40
does not equal
> global length 20, my
local length 20
>
> likely a call to
VecSetSizes() or MatSetSizes() is
>
wrong.
>
> See
http://www.mcs.anl.gov/petsc/documentation/faq.html#split!
>
>
>
> This error printout
doesn't quite make sense to me.
>
I'm trying to specify a total matrix size of 20x20...
I
> haven't yet figured out where the
'40' comes
> from in the error
message.
>
>
>
> Any thoughts on what
might be going wrong?
>
> Its the column specification. Just
> use PETSC_DETERMINE for the local columns
since all our
> sparse matrixformats are
row divisions
> anyway.
>
> Thanks,
>
> Matt
> Thanks in advance,
>
> Steena
>
>
>
>
>
>
>
>
--------------------------------------------
>
> On Sun, 3/22/15,
Barry Smith <bsmith at mcs.anl.gov>
> wrote:
>
>
>
> Subject: Re: [petsc-users] Unequal
sparse matrix row
> distribution for MPI
MatMult
>
> To:
"Steena M" <stm8086 at yahoo.com>
>
> Cc: petsc-users at mcs.anl.gov
>
> Date: Sunday,
March 22, 2015, 3:58 PM
>
>
>
>
>
>
>
> Steena,
>
>
>
> I am
>
> a little unsure of your question.
>
>
>
> 1)
you can create a MPIBAIJ
>
> matrix with any distribution of block
rows per process
> you
>
> want, just set the
local row size for each process to
>
be
>
> what you
like. Use MatCreateVecs() to get
>
correspondingly
>
>
laid out vectors.
>
>
>
>
or 2) if you have a MPIBAIJ
>
> matrix with
"equal" row layout and you want a
> new
>
> one with uneven row layout you can
simply use
>
>
MatGetSubMatrix() to create that new matrix.
>
>
>
> Barry
>
>
>
> Unless you have
another reason to have the
>
> matrix with an equal number row layout I
would just
> generate
>
> the matrix with
the layout you want.
>
>
>
>
>
> > On Mar 22, 2015, at 5:50 PM, Steena
M
>
> <stm8086 at yahoo.com>
>
> wrote:
>
> >
>
> > Hello,
>
> >
>
> > I need to
distribute
>
> a
sparse matrix such that each proc owns an unequal
> number
>
> of blocked rows before I proceed with
MPI MatMult. My
>
>
initial thoughts on doing this:
>
> >
>
> > 1) Use MatGetSubMatrices() on the
test
>
> MATMPIBAIJ
matrix to produce a new matrix where each
> proc
>
> has an unequal number of rows.
>
> >
>
> > 2) Provide
scatter context for vector X
>
> (for MatMult )using IS iscol from
MatGetSubMatrices()
> while
>
> creating the
vector X.
>
>
>
>
> > 3)
Call MatMult()
>
>
>
>
> > Will
MatMult_MPIBAIJ continue to scatter
>
> this matrix and vector such that each
proc will own an
> equal
>
> number of matrix
rows and corresponding diagonal vector
>
> elements? Should I write my own
MPIMatMult function to
>
> retain my redistribution of the matrix
and vector?
>
>
>
>
> >
Thanks in
>
>
advance,
>
> >
Steena
>
>
>
>
>
>
> --
> What most
experimenters
> take for granted before
they begin their experiments is
>
infinitely more interesting than any results to which
their
> experiments lead.
> -- Norbert
> Wiener
>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150401/90d2c473/attachment-0001.html>
More information about the petsc-users
mailing list