Problem with saving the binary matrix via Matview

[Matthew Knepley]

On Tue, Sep 8, 2009 at 3:21 PM, Nguyen, Hung V ERDC-ITL-MS <
Hung.V.Nguyen at usace.army.mil> wrote:

> Hello Keita,
>
>  Are you running a simple program to test MatView function on CrayXT4 or a
> *big* application codes? I have a simple program that reads its portion of
> matrix and save a matrix with nrows=2124108, nnzeros=43024032,
> max-nnzeros-per-row=27 on 16 cores. However, I have a problem with MatView
> function in the ADaptive Hydraulics Modeling (ADH)code with petsc interface
> for a large matrix application as indicated in previous email.
>
> By the way, setting MPI environment variables is not help.
>

If MatView() with a large matrix works in your example, but is slow in your
application
(for the same size matrix), doesn't that lead you to believe that PETSc is
not responsible
for the slow down, but something else in your application is?

  Matt


> Thank you,
>
> -hung
>
>
>
>
>
> -----Original Message-----
> From: petsc-users-bounces at mcs.anl.gov
> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Keita Teranishi
> Sent: Tuesday, September 08, 2009 10:56 AM
> To: PETSc users list
> Subject: RE: Problem with saving the binary matrix via Matview
>
> Hung,
>
> I did not see any problem calling MatView on 8 nodes of XT4 (32 cores). I
> was
> able to save a 900Kx900K (40 nonzeros per row) sparse matrix within a few
> seconds.
>
> Thanks,
>
>  ierr =
> PetscViewerBinaryOpen(PETSC_COMM_WORLD,"matrix.dat",FILE_MODE_WRITE,&view);
>  CHKERRQ(ierr);
>  ierr = MatView(A,view);CHKERRQ(ierr);
>  ierr = PetscViewerDestroy(view);CHKERRQ(ierr);
>
> ================================
>  Keita Teranishi
>  Scientific Library Group
>  Cray, Inc.
>  keita at cray.com
> ================================
>
> -----Original Message-----
> From: petsc-users-bounces at mcs.anl.gov
> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Nguyen, Hung V
> ERDC-ITL-MS
> Sent: Friday, September 04, 2009 3:59 PM
> To: PETSc users list
> Subject: RE: Problem with saving the binary matrix via Matview
>
> Hello,
>
> Yes, It is a CrayXT4. I just rerun with setting the variables below. I will
> let you know if it helps.
>
> hvnguyen:jade20% setenv MPICH_PTL_SEND_CREDITS -1 hvnguyen:jade20% setenv
> MPICH_MAX_SHORT_MSG_SIZE 64000 hvnguyen:jade20% setenv
> MPICH_UNEX_BUFFER_SIZE
> 240M hvnguyen:jade20% setenv MPICH_PTL_UNEX_EVENTS 60000
>
> >The problems will be worse if you don't have the latest Cray software
> >on the
> system.
>
> What do you mean exactly about the latest Cray software?
>
> >Note we generally frown up saving huge matrices to disk except for
> >debugging
> testing purposes. But as Matt notes this is not a particularly huge matrix.
> With any reasonable configuration it should take very little time to write
> the file.
>
> We need to dump out some matrices required large number of iterations for
> testing.
> As for small application case (nrows-12856,nnzeros=675744), it took a
> minute
> to write binary matrix to a file on CrayXT4 system so I don't know why it
> has
> a deadlock in this case.
>
> Thank you.
>
> -hung
>
>
>
>
> -----Original Message-----
> From: petsc-users-bounces at mcs.anl.gov
> [mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith
> Sent: Friday, September 04, 2009 3:24 PM
> To: PETSc users list
> Subject: Re: Problem with saving the binary matrix via Matview
>
>
>  Is this a Cray? You may need to set the environmental variables MPI UNEX
> BUFFER SIZE and/or MPICH PTL MATCH OFF and/or MPICH PTL OTHER EVENTS and/or
> MPICH MSGS PER PROC and/or MPICH PTL SEND CREDITS you will likely need to
> hunt through Cray documentation to find the meaning of all this stuff. The
> problems will be worse if you don't have the latest Cray software on the
> system.
>
>   Note we generally frown up saving huge matrices to disk except for
> debugging testing purposes. But as Matt notes this is not a particularly
> huge
> matrix. With any reasonable configuration it should take very little time
> to
> write the file.
>
>    Good luck,
>
>    Barry
>
>
>
>
> On Sep 4, 2009, at 2:25 PM, Nguyen, Hung V ERDC-ITL-MS wrote:
>
> >
> > Hello,
> >
> > I have a problem with saving a matrix with the binary viewer
> > PetscBinaryViewerOpen() for the matrix with nrow=853564,
> > nnzeros=47191472.
> > The application ran with 32 pes for more than 3 hours without writing
> > any into file.
> > However, the same executable ran for small size of application
> > (nrows-12856,nnzeros=675744) and were able to write successfully
> > binary files.
> >
> > Note: using Totalview I found that it hang in the line of
> > Matview(matrix->petsc, fd);
> >
> > Thank you,
> >
> > -hung
> >
> > --- code:
> >      petsc_analyst_mat(matrix->petsc);
> >      sprintf(file[LOAD_MATRIX],"Matrix.at%f",t_prev);
> >      PetscViewerBinaryOpen(PETSC_COMM_WORLD,
> > file[LOAD_MATRIX],FILE_MODE_WRITE,&fd);
> >      MatView(matrix->petsc, fd);
> >      PetscViewerDestroy(fd);
> >
> > ---- Matrix info using salsa/AnaMod module:
> >
> > Computed <simple:trace> as <9.362113e+03> Computed <simple:trace-abs>
> > as <9.362623e+03> Computed <simple:norm1> as <2.800000e+01> Computed
> > <simple:normInf> as <2.800000e+01> Computed <simple:normF> as
> > <2.404511e+02> Computed <simple:diagonal-dominance> as <-3.064463e-02>
> > Could not compute <simple:symmetry-snorm> Could not compute
> > <simple:symmetry-anorm> Could not compute <simple:symmetry-fsnorm>
> > Could not compute <simple:symmetry-fanorm> Could not compute
> > <structure:n-struct-unsymm> Computed <structure:nrows> as <853564>
> > Could not compute <structure:symmetry> Computed <structure:nnzeros> as
> > <47191472> Computed <structure:max-nnzeros-per-row> as <112> Computed
> > <structure:min-nnzeros-per-row> as <16>
>
>


-- 
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20090908/e661f76b/attachment.htm>