On Tue, Sep 8, 2009 at 3:21 PM, Nguyen, Hung V ERDC-ITL-MS <span dir="ltr"><<a href="mailto:Hung.V.Nguyen@usace.army.mil">Hung.V.Nguyen@usace.army.mil</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hello Keita,<br>
<br>
Are you running a simple program to test MatView function on CrayXT4 or a<br>
*big* application codes? I have a simple program that reads its portion of<br>
matrix and save a matrix with nrows=2124108, nnzeros=43024032,<br>
max-nnzeros-per-row=27 on 16 cores. However, I have a problem with MatView<br>
function in the ADaptive Hydraulics Modeling (ADH)code with petsc interface<br>
for a large matrix application as indicated in previous email.<br>
<br>
By the way, setting MPI environment variables is not help.<br></blockquote><div><br>If MatView() with a large matrix works in your example, but is slow in your application<br>(for the same size matrix), doesn't that lead you to believe that PETSc is not responsible<br>
for the slow down, but something else in your application is?<br><br> Matt<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Thank you,<br>
<br>
-hung<br>
<br>
<br>
<br>
<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a><br>
[mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a>] On Behalf Of Keita Teranishi<br>
Sent: Tuesday, September 08, 2009 10:56 AM<br>
To: PETSc users list<br>
Subject: RE: Problem with saving the binary matrix via Matview<br>
<br>
Hung,<br>
<br>
I did not see any problem calling MatView on 8 nodes of XT4 (32 cores). I was<br>
able to save a 900Kx900K (40 nonzeros per row) sparse matrix within a few<br>
seconds.<br>
<br>
Thanks,<br>
<br>
ierr =<br>
PetscViewerBinaryOpen(PETSC_COMM_WORLD,"matrix.dat",FILE_MODE_WRITE,&view);<br>
CHKERRQ(ierr);<br>
ierr = MatView(A,view);CHKERRQ(ierr);<br>
ierr = PetscViewerDestroy(view);CHKERRQ(ierr);<br>
<br>
================================<br>
Keita Teranishi<br>
Scientific Library Group<br>
Cray, Inc.<br>
<a href="mailto:keita@cray.com">keita@cray.com</a><br>
================================<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a><br>
[mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a>] On Behalf Of Nguyen, Hung V<br>
ERDC-ITL-MS<br>
Sent: Friday, September 04, 2009 3:59 PM<br>
To: PETSc users list<br>
Subject: RE: Problem with saving the binary matrix via Matview<br>
<br>
Hello,<br>
<br>
Yes, It is a CrayXT4. I just rerun with setting the variables below. I will<br>
let you know if it helps.<br>
<br>
hvnguyen:jade20% setenv MPICH_PTL_SEND_CREDITS -1 hvnguyen:jade20% setenv<br>
MPICH_MAX_SHORT_MSG_SIZE 64000 hvnguyen:jade20% setenv MPICH_UNEX_BUFFER_SIZE<br>
240M hvnguyen:jade20% setenv MPICH_PTL_UNEX_EVENTS 60000<br>
<br>
>The problems will be worse if you don't have the latest Cray software<br>
>on the<br>
system.<br>
<br>
What do you mean exactly about the latest Cray software?<br>
<br>
>Note we generally frown up saving huge matrices to disk except for<br>
>debugging<br>
testing purposes. But as Matt notes this is not a particularly huge matrix.<br>
With any reasonable configuration it should take very little time to write<br>
the file.<br>
<br>
We need to dump out some matrices required large number of iterations for<br>
testing.<br>
As for small application case (nrows-12856,nnzeros=675744), it took a minute<br>
to write binary matrix to a file on CrayXT4 system so I don't know why it has<br>
a deadlock in this case.<br>
<br>
Thank you.<br>
<br>
-hung<br>
<br>
<br>
<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a><br>
[mailto:<a href="mailto:petsc-users-bounces@mcs.anl.gov">petsc-users-bounces@mcs.anl.gov</a>] On Behalf Of Barry Smith<br>
Sent: Friday, September 04, 2009 3:24 PM<br>
To: PETSc users list<br>
Subject: Re: Problem with saving the binary matrix via Matview<br>
<br>
<br>
Is this a Cray? You may need to set the environmental variables MPI UNEX<br>
BUFFER SIZE and/or MPICH PTL MATCH OFF and/or MPICH PTL OTHER EVENTS and/or<br>
MPICH MSGS PER PROC and/or MPICH PTL SEND CREDITS you will likely need to<br>
hunt through Cray documentation to find the meaning of all this stuff. The<br>
problems will be worse if you don't have the latest Cray software on the<br>
system.<br>
<br>
Note we generally frown up saving huge matrices to disk except for<br>
debugging testing purposes. But as Matt notes this is not a particularly huge<br>
matrix. With any reasonable configuration it should take very little time to<br>
write the file.<br>
<br>
Good luck,<br>
<br>
Barry<br>
<br>
<br>
<br>
<br>
On Sep 4, 2009, at 2:25 PM, Nguyen, Hung V ERDC-ITL-MS wrote:<br>
<br>
><br>
> Hello,<br>
><br>
> I have a problem with saving a matrix with the binary viewer<br>
> PetscBinaryViewerOpen() for the matrix with nrow=853564,<br>
> nnzeros=47191472.<br>
> The application ran with 32 pes for more than 3 hours without writing<br>
> any into file.<br>
> However, the same executable ran for small size of application<br>
> (nrows-12856,nnzeros=675744) and were able to write successfully<br>
> binary files.<br>
><br>
> Note: using Totalview I found that it hang in the line of<br>
> Matview(matrix->petsc, fd);<br>
><br>
> Thank you,<br>
><br>
> -hung<br>
><br>
> --- code:<br>
> petsc_analyst_mat(matrix->petsc);<br>
> sprintf(file[LOAD_MATRIX],"Matrix.at%f",t_prev);<br>
> PetscViewerBinaryOpen(PETSC_COMM_WORLD,<br>
> file[LOAD_MATRIX],FILE_MODE_WRITE,&fd);<br>
> MatView(matrix->petsc, fd);<br>
> PetscViewerDestroy(fd);<br>
><br>
> ---- Matrix info using salsa/AnaMod module:<br>
><br>
> Computed <simple:trace> as <9.362113e+03> Computed <simple:trace-abs><br>
> as <9.362623e+03> Computed <simple:norm1> as <2.800000e+01> Computed<br>
> <simple:normInf> as <2.800000e+01> Computed <simple:normF> as<br>
> <2.404511e+02> Computed <simple:diagonal-dominance> as <-3.064463e-02><br>
> Could not compute <simple:symmetry-snorm> Could not compute<br>
> <simple:symmetry-anorm> Could not compute <simple:symmetry-fsnorm><br>
> Could not compute <simple:symmetry-fanorm> Could not compute<br>
> <structure:n-struct-unsymm> Computed <structure:nrows> as <853564><br>
> Could not compute <structure:symmetry> Computed <structure:nnzeros> as<br>
> <47191472> Computed <structure:max-nnzeros-per-row> as <112> Computed<br>
> <structure:min-nnzeros-per-row> as <16><br>
<br>
</blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>