<div dir="ltr"><div dir="ltr">On Mon, May 17, 2021 at 5:18 AM feng wang <<a href="mailto:snailsoar@hotmail.com">snailsoar@hotmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Mat,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div>
<div id="gmail-m_6517126346834551521appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I have tried the 1st approach you suggested, which re-configures petsc with complex numbers. For my real-number system, Ax=B, it still works fine. I would like to save a copy a A, because for my complex system (A + i*w) Z = C, the value of "w" could be a list
of values, so I only need to do a MatShift for my saved copy of A to build the LHS for each "w".<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I am trying to use <span>MatDuplicate to create the copy</span> of "A" and MatCopy to copy the values of "A". the following is what I do. "petsc_A_pre" is "A" and "<span>petsc_A_pre_copy</span>" is its copy.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
ierr = MatCreateBAIJ(*A_COMM_WORLD, blocksize, nlocal*blocksize, nlocal*blocksize, PETSC_DETERMINE, PETSC_DETERMINE,
<div> maxneig, NULL, maxneig, NULL, &petsc_A_pre); CHKERRQ(ierr);</div>
<div> ierr = MatSetOption(petsc_A_pre, MAT_STRUCTURALLY_SYMMETRIC, PETSC_TRUE); CHKERRQ(ierr);</div>
<span> ierr = MatDuplicate(petsc_A_pre, MAT_SHARE_NONZERO_PATTERN, &petsc_A_pre_copy); CHKERRQ(ierr);
<u>//line 69</u><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span>.....some operations.....</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span> ierr = MatAssemblyBegin(petsc_A_pre,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
<div> ierr = MatAssemblyEnd(petsc_A_pre,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);</div>
<div><br>
</div>
<div> ierr = MatCopy(petsc_A_pre, petsc_A_pre_copy, SAME_NONZERO_PATTERN);</div>
<div> ierr = MatAssemblyBegin(petsc_A_pre_copy,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);</div>
<div> ierr = MatAssemblyEnd(petsc_A_pre_copy,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);</div>
<br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span>I get the following error for MatDuplicate and MatCopy:</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span>[0]PETSC ERROR: Petsc has generated inconsistent data
<div>[0]PETSC ERROR: Corrupt matrix</div>
<div>[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div>
<div>[0]PETSC ERROR: Petsc Release Version 3.14.4, unknown </div>
<div>[0]PETSC ERROR: cfd on a arch-linux2-c-debug named snail by snail Mon May 17 09:34:46 2021</div>
<div>[0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-scalar-type=complex</div>
<div>[0]PETSC ERROR: #1 MatDuplicateNoCreate_SeqBAIJ() line 3103 in /home/feng/cfd/petsc.complex/src/mat/impls/baij/seq/baij.c</div>
<div>[0]PETSC ERROR: #2 MatDuplicate_SeqBAIJ() line 3215 in /home/feng/cfd/petsc.complex/src/mat/impls/baij/seq/baij.c</div>
<div>[0]PETSC ERROR: #3 MatDuplicate() line 4663 in /home/feng/cfd/petsc.complex/src/mat/interface/matrix.c</div>
<span>[0]PETSC ERROR: #4 petsc_nk_init() line 69 in domain/cfd/petsc_nk.cpp</span></span></div></div></div></blockquote><div><br></div><div>Your matrix is actually corrupt. We have to fix this. It is either a PETSc bug or a bug in your code. Would you be able to send us a stripped down</div><div>version of your code (say one that puts in all zeros, or similarly simplified), so we can track down where this is?</div><div><br></div><div>Alternatively, you can run under valgrind, since this looks like a memory overwrite.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
<div>[0]PETSC ERROR: Object is in wrong state</div>
<div>[0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 2 "B" before MatCopy()</div>
<div>[0]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div>
<div>[0]PETSC ERROR: Petsc Release Version 3.14.4, unknown </div>
<div>[0]PETSC ERROR: cfd on a arch-linux2-c-debug named feng by feng Mon May 17 09:34:46 2021</div>
<div>[0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-scalar-type=complex</div>
<span>[0]PETSC ERROR: #5 MatCopy() line 4087 in /home/feng/cfd/petsc.complex/src/mat/interface/matrix.c</span><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Is there a better way to do this? could you give any comments?<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Feng<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_6517126346834551521divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> 14 May 2021 20:26<br>
<b>To:</b> feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] reuse a real matrix for a second linear system with complex numbers</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, May 14, 2021 at 1:36 PM feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Yes, you are right. I can do row permutations to make them continuous. I will try this.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
could I re-use my KSP object from the 1st linear system in my 2nd system by simply changing the operators and setting new parameters? or I need a separate KSP object for the 2nd system?</div>
</div>
</blockquote>
<div><br>
</div>
<div>I tink you want 2 KSP objects. You could reuse the settings of the first, but since the system is a different size, all storage would have to be deleted and recreated anyway.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Feng<br>
</div>
<br>
<div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_6517126346834551521x_gmail-m_-9145296352994440899divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> 14 May 2021 15:20<br>
<b>To:</b> feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] reuse a real matrix for a second linear system with complex numbers</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, May 14, 2021 at 10:36 AM feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks for your comments. It is very helpful!</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I might try the 1st approach first. For the 2nd approach which uses an equivalent real-number system, I see potential issues when running in parallel. I have re-ordered my cells to allow each rank hold continuous rows in the first real system Ax=B. For the
equivalent real-number system, each rank now holds (or can assign values to) two patches of continuous rows, which are separated by N rows, N is the size of square matrix A. I can't see a straightforward way to allow each rank hold continuous rows in this
case. or petsc can handle these two patches of continuous rows with fixed row index difference in this case?</div>
</div>
</blockquote>
<div><br>
</div>
<div>I just wrote it that way for ease of typing. You can imagine permuting into 2x2 blocks with</div>
<div><br>
</div>
<div> /a w\</div>
<div> \-w a/</div>
<div><br>
</div>
<div>for each entry.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
By the way, could I still re-use my KSP object in my second system by simply changing the operators and setting new parameters?</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Feng<br>
</div>
<div>
<div id="gmail-m_6517126346834551521x_gmail-m_-9145296352994440899x_gmail-m_6060122375414069395appendonsend">
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_6517126346834551521x_gmail-m_-9145296352994440899x_gmail-m_6060122375414069395divRplyFwdMsg" dir="ltr">
<font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> 14 May 2021 10:00<br>
<b>To:</b> feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] reuse a real matrix for a second linear system with complex numbers</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, May 14, 2021 at 4:23 AM feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Dear All,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I am solving a coupled system. One system is AX=B. A, X and B are all real numbers and it is solved with GMRES in petsc. Now I need to solve a second linear system, it can be represented as (A+i*w)*Z=C. i is the imaginary unit. Z and C are also complex numbers.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
So the Jacobian matrix of the second system is just A plus a diagonal contribution i*w. I would like solve the second system with GMRES, could petsc handle this? any comments are welcome.</div>
</div>
</blockquote>
<div><br>
</div>
<div>Mixing real and complex numbers in the same code is somewhat difficult now. You have two obvious choices:</div>
<div><br>
</div>
<div>1) Configure for complex numbers and solve your first system as complex but with 0 imaginary part. This will work fine, but uses more memory for that system. However, since you will already</div>
<div> use that much memory for the second system, it does not seem like a big deal to me.</div>
<div><br>
</div>
<div>2) You could solve the second system in its equivalent real form </div>
<div><br>
</div>
<div> / A w \ /Zr\ = /Cr\</div>
<div> \ -w A / \Zi/ \Ci/</div>
<div> </div>
<div> This uses more memory for the second system, but does not require reconfiguring.</div>
<div><br>
</div>
<div> THanks,</div>
<div><br>
</div>
<div> Matt</div>
<div><br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Feng</div>
</div>
</blockquote>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>