[petsc-users] reuse a real matrix for a second linear system with complex numbers

feng wang snailsoar at hotmail.com
Mon May 17 09:30:49 CDT 2021 

Thanks for your reply.

Ax=B
(A + i*w) Z = C

I have been using the matrix A for my real-number system, it is ok, so I believe the matrix "A" should not be corrupted.   It is a bit strange that MatDuplicate and MatCopy complain. I am quite puzzled by this. my code is a big library, it would take some effort to make a stripped-down version, I can have a try.

Maybe I don't actually need to make a copy of A. I can directly work on it to add the diagonal contribution i*w and then call KSPSolve.  For a different "w", I can undo the previous diagonal contribution and add the new one and so on. so I only need the memory space of "A" in the end for both systems.  I've tried this for one "w" and it seems working for me.

I am relying on the fact that "A" (its values and structure) is untouched by "KSPSolve" , am I correct on this or I am being lucky?

Thanks,
feng

________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: 17 May 2021 11:37
To: feng wang <snailsoar at hotmail.com>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] reuse a real matrix for a second linear system with complex numbers

On Mon, May 17, 2021 at 5:18 AM feng wang <snailsoar at hotmail.com<mailto:snailsoar at hotmail.com>> wrote:
Hi Mat,

I have tried the 1st approach you suggested, which re-configures petsc with complex numbers. For my real-number system, Ax=B, it still works fine. I would like to save a copy a A, because for my complex system (A + i*w) Z = C, the value of "w" could be a list of values,  so I only need to do a MatShift for my saved copy of A to build the LHS for each "w".

I am trying to use MatDuplicate to create the copy of "A" and MatCopy to copy the values of "A". the following is what I do.  "petsc_A_pre" is "A" and "petsc_A_pre_copy" is its copy.

      ierr = MatCreateBAIJ(*A_COMM_WORLD, blocksize, nlocal*blocksize, nlocal*blocksize, PETSC_DETERMINE, PETSC_DETERMINE,
                            maxneig, NULL, maxneig, NULL, &petsc_A_pre); CHKERRQ(ierr);
      ierr = MatSetOption(petsc_A_pre, MAT_STRUCTURALLY_SYMMETRIC, PETSC_TRUE); CHKERRQ(ierr);
      ierr = MatDuplicate(petsc_A_pre, MAT_SHARE_NONZERO_PATTERN, &petsc_A_pre_copy); CHKERRQ(ierr);   //line 69

.....some operations.....

         ierr = MatAssemblyBegin(petsc_A_pre,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
         ierr = MatAssemblyEnd(petsc_A_pre,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);

         ierr = MatCopy(petsc_A_pre, petsc_A_pre_copy, SAME_NONZERO_PATTERN);
         ierr = MatAssemblyBegin(petsc_A_pre_copy,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
         ierr = MatAssemblyEnd(petsc_A_pre_copy,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);


I get the following error for MatDuplicate and MatCopy:

[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Corrupt matrix
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.14.4, unknown
[0]PETSC ERROR: cfd on a arch-linux2-c-debug named snail by snail Mon May 17 09:34:46 2021
[0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-scalar-type=complex
[0]PETSC ERROR: #1 MatDuplicateNoCreate_SeqBAIJ() line 3103 in /home/feng/cfd/petsc.complex/src/mat/impls/baij/seq/baij.c
[0]PETSC ERROR: #2 MatDuplicate_SeqBAIJ() line 3215 in /home/feng/cfd/petsc.complex/src/mat/impls/baij/seq/baij.c
[0]PETSC ERROR: #3 MatDuplicate() line 4663 in /home/feng/cfd/petsc.complex/src/mat/interface/matrix.c
[0]PETSC ERROR: #4 petsc_nk_init() line 69 in domain/cfd/petsc_nk.cpp

Your matrix is actually corrupt. We have to fix this. It is either a PETSc bug or a bug in your code. Would you be able to send us a stripped down
version of your code (say one that puts in all zeros, or similarly simplified), so we can track down where this is?

Alternatively, you can run under valgrind, since this looks like a memory overwrite.

  Thanks,

     Matt

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 2 "B" before MatCopy()
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.14.4, unknown
[0]PETSC ERROR: cfd on a arch-linux2-c-debug named feng by feng Mon May 17 09:34:46 2021
[0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-scalar-type=complex
[0]PETSC ERROR: #5 MatCopy() line 4087 in /home/feng/cfd/petsc.complex/src/mat/interface/matrix.c

 Is there a better way to do this? could you give any comments?

Thanks,
Feng

________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: 14 May 2021 20:26
To: feng wang <snailsoar at hotmail.com<mailto:snailsoar at hotmail.com>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] reuse a real matrix for a second linear system with complex numbers

On Fri, May 14, 2021 at 1:36 PM feng wang <snailsoar at hotmail.com<mailto:snailsoar at hotmail.com>> wrote:
Yes, you are right. I can do row permutations to make them continuous. I will try this.

could I re-use my KSP object from the 1st linear system in my 2nd system by simply changing the operators and setting new parameters? or I need a separate KSP object for the 2nd system?

I tink you want 2 KSP objects. You could reuse the settings of the first, but since the system is a different size, all storage would have to be deleted and recreated anyway.

  Thanks,

     Matt

Thanks,
Feng

________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: 14 May 2021 15:20
To: feng wang <snailsoar at hotmail.com<mailto:snailsoar at hotmail.com>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] reuse a real matrix for a second linear system with complex numbers

On Fri, May 14, 2021 at 10:36 AM feng wang <snailsoar at hotmail.com<mailto:snailsoar at hotmail.com>> wrote:
Thanks for your comments. It is very helpful!

I might try the 1st approach first.   For the 2nd approach which uses an equivalent real-number system, I see potential issues when running in parallel. I have re-ordered my cells to allow each rank hold continuous rows in the first real system Ax=B. For the equivalent real-number system,  each rank now holds (or can assign values to) two patches of continuous rows, which are separated by N rows, N is the size of square matrix A. I can't see a straightforward way to allow each rank hold continuous rows in this case. or petsc can handle these two patches of continuous rows with fixed row index difference in this case?

I just wrote it that way for ease of typing. You can imagine permuting into 2x2 blocks with

  /a  w\
  \-w a/

for each entry.

  Thanks,

     Matt

By the way, could I still re-use my KSP object in my second system by simply changing the operators and setting new parameters?

Thanks,
Feng

________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: 14 May 2021 10:00
To: feng wang <snailsoar at hotmail.com<mailto:snailsoar at hotmail.com>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] reuse a real matrix for a second linear system with complex numbers

On Fri, May 14, 2021 at 4:23 AM feng wang <snailsoar at hotmail.com<mailto:snailsoar at hotmail.com>> wrote:
Dear All,

I am solving a coupled system. One system is AX=B. A, X and B are all real numbers and it is solved with GMRES in petsc. Now I need to solve a second linear system, it can be represented as (A+i*w)*Z=C. i is the imaginary unit.  Z and C are also complex numbers.

So the Jacobian matrix of the second system is just A plus a diagonal contribution i*w.  I would like solve the second system with GMRES, could petsc handle this? any comments are welcome.

Mixing real and complex numbers in the same code is somewhat difficult now. You have two obvious choices:

1) Configure for complex numbers and solve your first system as complex but with 0 imaginary part. This will work fine, but uses more memory for that system. However, since you will already
     use that much memory for the second system, it does not seem like a big deal to me.

2) You could solve the second system in its equivalent real form

  / A  w \ /Zr\ = /Cr\
 \ -w  A / \Zi/     \Ci/

   This uses more memory for the second system, but does not require reconfiguring.

  THanks,

     Matt

Thanks,
Feng
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210517/2e6ccc07/attachment-0001.html>

More information about the petsc-users mailing list