[petsc-users] Customizeing MatSetValuesBlocked(...)

[Jinquan Zhong]

First question:
Are your matrices sparse or dense?

*         A is a dense matrix at the order of 225,000 of complex*16;

How large?

*         800GB on disk

What type of math equations, e.g., solve Ax=b or eigenvalue problems?

*         A'x=b, where A'=[A^-1 U B]= sparse matrix at the order of 2 million.  B = sparse matrix to be mixed and mingled with A^-1.

Hong

Jinquan
On Wed, Aug 8, 2012 at 5:36 PM, Jinquan Zhong <jzhong at scsolutions.com<mailto:jzhong at scsolutions.com>> wrote:
Thanks, Prof. Zhang.

I could have used PETSc directly for this application. However, I have to use this procedure for QA (quality assurance) purposes since this parallelizing process for a program I am working on is a standard program widely used in industry.  That is, QA purposes have driven us to use ScaLAPACK for inversion first then PETSc for solution.  I wish I could a big move now to eliminate ScaLAPACK.

Jinquan

From: petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov> [mailto:petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>] On Behalf Of Hong Zhang
Sent: Wednesday, August 08, 2012 3:29 PM

To: PETSc users list
Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...)

Jinquan:

You need understand some basic concepts in numerical computations, for example:
- dense matrix vs. sparse matrix and their data structures
- algorithms for solve Ax=b
- available software packages, their targeted applications

See www.mcs.anl.gov/~hzhang/teach/cs595<http://www.mcs.anl.gov/~hzhang/teach/cs595> for a list of references.

Hong

What I am trying to say is that

*         allocated 1D array A in PETSc

*         passed 1D A into ScaLAPACK

*         obtained the indices of A^-1 computed from ScaLAPACK and stored them in a 1D array

*         indexed A^-1 as a 1D array in PETSc using the indices obtained from ScaLAPACK

*         constructed a sparse matrix, including A^-1 and others, using a new set of indices for A^-1.
I can have  the last step done in using

                       ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
                       ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,N,N);CHKERRQ(ierr);
                       ierr = MatSetFromOptions(A);CHKERRQ(ierr);
                       ierr = MatSetUp(A);CHKERRQ(ierr);

  ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);

It just took too long.  This is the performance issue I am having.

To improve the performance, it is necessary for preallocation.  The dilemma is: MatCreateAIJ and MatXCreateAIJ complained the setting of d_nz, d_nnz,o_nz, o_nnz  since there are no diagonal and off-diagonal parts defined in A^-1.

Jinquan


From: petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov> [mailto:petsc-users-bounces at mcs.anl.gov<mailto:petsc-users-bounces at mcs.anl.gov>] On Behalf Of Jed Brown
Sent: Wednesday, August 08, 2012 3:04 PM

To: PETSc users list
Subject: Re: [petsc-users] Customizeing MatSetValuesBlocked(...)

On Wed, Aug 8, 2012 at 3:45 PM, Jinquan Zhong <jzhong at scsolutions.com<mailto:jzhong at scsolutions.com>> wrote:
It is a 1D matrix in PETSc.  I just passed it as a 2D matrix in ScaLAPACK.  It is still a 1D matrix when get back into PETSc.

What are you trying to say?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120808/de356d20/attachment-0001.html>