Problem with saving the binary matrix via Matview

[Nguyen, Hung V ERDC-ITL-MS]

Hello Keita,

 Are you running a simple program to test MatView function on CrayXT4 or a
*big* application codes? I have a simple program that reads its portion of
matrix and save a matrix with nrows=2124108, nnzeros=43024032,
max-nnzeros-per-row=27 on 16 cores. However, I have a problem with MatView
function in the ADaptive Hydraulics Modeling (ADH)code with petsc interface
for a large matrix application as indicated in previous email.

By the way, setting MPI environment variables is not help.

Thank you,

-hung





-----Original Message-----
From: petsc-users-bounces at mcs.anl.gov
[mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Keita Teranishi
Sent: Tuesday, September 08, 2009 10:56 AM
To: PETSc users list
Subject: RE: Problem with saving the binary matrix via Matview

Hung,

I did not see any problem calling MatView on 8 nodes of XT4 (32 cores). I was
able to save a 900Kx900K (40 nonzeros per row) sparse matrix within a few
seconds.

Thanks,

 ierr =
PetscViewerBinaryOpen(PETSC_COMM_WORLD,"matrix.dat",FILE_MODE_WRITE,&view);
 CHKERRQ(ierr);
 ierr = MatView(A,view);CHKERRQ(ierr);
 ierr = PetscViewerDestroy(view);CHKERRQ(ierr);

================================
 Keita Teranishi
 Scientific Library Group
 Cray, Inc.
 keita at cray.com
================================

-----Original Message-----
From: petsc-users-bounces at mcs.anl.gov
[mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Nguyen, Hung V
ERDC-ITL-MS
Sent: Friday, September 04, 2009 3:59 PM
To: PETSc users list
Subject: RE: Problem with saving the binary matrix via Matview

Hello,

Yes, It is a CrayXT4. I just rerun with setting the variables below. I will
let you know if it helps. 

hvnguyen:jade20% setenv MPICH_PTL_SEND_CREDITS -1 hvnguyen:jade20% setenv
MPICH_MAX_SHORT_MSG_SIZE 64000 hvnguyen:jade20% setenv MPICH_UNEX_BUFFER_SIZE
240M hvnguyen:jade20% setenv MPICH_PTL_UNEX_EVENTS 60000

>The problems will be worse if you don't have the latest Cray software 
>on the
system.

What do you mean exactly about the latest Cray software?

>Note we generally frown up saving huge matrices to disk except for 
>debugging
testing purposes. But as Matt notes this is not a particularly huge matrix.
With any reasonable configuration it should take very little time to write
the file.

We need to dump out some matrices required large number of iterations for
testing. 
As for small application case (nrows-12856,nnzeros=675744), it took a minute
to write binary matrix to a file on CrayXT4 system so I don't know why it has
a deadlock in this case.

Thank you.

-hung




-----Original Message-----
From: petsc-users-bounces at mcs.anl.gov
[mailto:petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith
Sent: Friday, September 04, 2009 3:24 PM
To: PETSc users list
Subject: Re: Problem with saving the binary matrix via Matview


  Is this a Cray? You may need to set the environmental variables MPI UNEX
BUFFER SIZE and/or MPICH PTL MATCH OFF and/or MPICH PTL OTHER EVENTS and/or
MPICH MSGS PER PROC and/or MPICH PTL SEND CREDITS you will likely need to
hunt through Cray documentation to find the meaning of all this stuff. The
problems will be worse if you don't have the latest Cray software on the
system.

   Note we generally frown up saving huge matrices to disk except for
debugging testing purposes. But as Matt notes this is not a particularly huge
matrix. With any reasonable configuration it should take very little time to
write the file.

    Good luck,

    Barry




On Sep 4, 2009, at 2:25 PM, Nguyen, Hung V ERDC-ITL-MS wrote:

>
> Hello,
>
> I have a problem with saving a matrix with the binary viewer
> PetscBinaryViewerOpen() for the matrix with nrow=853564, 
> nnzeros=47191472.
> The application ran with 32 pes for more than 3 hours without writing 
> any into file.
> However, the same executable ran for small size of application
> (nrows-12856,nnzeros=675744) and were able to write successfully 
> binary files.
>
> Note: using Totalview I found that it hang in the line of 
> Matview(matrix->petsc, fd);
>
> Thank you,
>
> -hung
>
> --- code:
>      petsc_analyst_mat(matrix->petsc);
>      sprintf(file[LOAD_MATRIX],"Matrix.at%f",t_prev);
>      PetscViewerBinaryOpen(PETSC_COMM_WORLD,
> file[LOAD_MATRIX],FILE_MODE_WRITE,&fd);
>      MatView(matrix->petsc, fd);
>      PetscViewerDestroy(fd);
>
> ---- Matrix info using salsa/AnaMod module:
>
> Computed <simple:trace> as <9.362113e+03> Computed <simple:trace-abs> 
> as <9.362623e+03> Computed <simple:norm1> as <2.800000e+01> Computed 
> <simple:normInf> as <2.800000e+01> Computed <simple:normF> as 
> <2.404511e+02> Computed <simple:diagonal-dominance> as <-3.064463e-02> 
> Could not compute <simple:symmetry-snorm> Could not compute 
> <simple:symmetry-anorm> Could not compute <simple:symmetry-fsnorm> 
> Could not compute <simple:symmetry-fanorm> Could not compute 
> <structure:n-struct-unsymm> Computed <structure:nrows> as <853564> 
> Could not compute <structure:symmetry> Computed <structure:nnzeros> as 
> <47191472> Computed <structure:max-nnzeros-per-row> as <112> Computed 
> <structure:min-nnzeros-per-row> as <16>