[petsc-users] Customizeing MatSetValuesBlocked(...)
Jinquan Zhong
jzhong at scsolutions.com
Thu Aug 9 10:19:00 CDT 2012
Has this problem or similar problem been solved using scalapack?
* Yes. We have solved A^-1 from A using scalapack.
Your problem is virtually a dense problem, why choose petsc library?
* Because the A' is a sparse matrix at a much larger order.
What type of computer, number of cores, to be used for this problem?
* Supercomputer. We are looking at about 3~5 milllion cpu hours.
Jinquan
First question:
Are your matrices sparse or dense?
* A is a dense matrix at the order of 225,000 of complex*16;
How large?
* 800GB on disk
What type of math equations, e.g., solve Ax=b or eigenvalue problems?
* A'x=b, where A'=[A^-1 U B]= sparse matrix at the order of 2 million. B = sparse matrix to be mixed and mingled with A^-1.
Hong
Jinquan
On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong <jzhong at scsolutions.com<mailto:jzhong at scsolutions.com>> wrote:
Thanks, Prof. Zhang.
I could have used PETSc directly for this application. However, I have to use this procedure for QA (quality assurance) purposes since this parallelizing process for a program I am working on is a standard program widely used in industry. That is, QA purposes have driven us to use ScaLAPACK for inversion first then PETSc for solution. I wish I could a big move now to eliminate ScaLAPACK.
Jinquan
From: petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov> [mailto:petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>] On Behalf Of Hong Zhang
Sent: Wednesday, August 08, 2012 3:29 PM
To: PETSc users list
Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...)
Jinquan:
You need understand some basic concepts in numerical computations, for example:
- dense matrix vs. sparse matrix and their data structures
- algorithms for solve Ax=b
- available software packages, their targeted applications
See www.mcs.anl.gov/~hzhang/teach/cs595<http://www.mcs.anl.gov/~hzhang/teach/cs595> for a list of references.
Hong
What I am trying to say is that
* allocated 1D array A in PETSc
* passed 1D A into ScaLAPACK
* obtained the indices of A^-1 computed from ScaLAPACK and stored them in a 1D array
* indexed A^-1 as a 1D array in PETSc using the indices obtained from ScaLAPACK
* constructed a sparse matrix, including A^-1 and others, using a new set of indices for A^-1.
I can have the last step done in using
ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr);
ierr = MatSetFromOptions(A);CHKERRQ(ierr);
ierr = MatSetUp(A);CHKERRQ(ierr);
ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);
It just took too long. This is the performance issue I am having.
To improve the performance, it is necessary for preallocation. The dilemma is: MatCreateAIJ and MatXCreateAIJ complained the setting of d_nz, d_nnz,o_nz, o_nnz since there are no diagonal and off-diagonal parts defined in A^-1.
Jinquan
From: petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov> [mailto:petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>] On Behalf Of Jed Brown
Sent: Wednesday, August 08, 2012 3:04 PM
To: PETSc users list
Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...)
On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong <jzhong at scsolutions.com<mailto:jzhong at scsolutions.com>> wrote:
It is a 1D matrix in PETSc. I just passed it as a 2D matrix in ScaLAPACK. It is still a 1D matrix when get back into PETSc.
What are you trying to say?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120809/73418b6a/attachment.html>
More information about the petsc-users
mailing list