<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">
<div>Run your code with option '-ksp_error_if_not_converged' to get more info.</div>
<div>Hong</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Nov 7, 2019 at 5:45 AM s.a.hack--- via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_277588947448858389WordSection1">
<p class="MsoNormal"><span style="color:black">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">I am doing calculations with version 3.12.0 of PETSc.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">Using the finite-element method, I solve the Maxwell equations on the interior of a 3D domain, coupled with boundary condition auxiliary equations on the boundary of the domain. The auxiliary equations employ auxiliary
variables g.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">For ease of implementation of element matrix assembly, the auxiliary variables g are defined on the entire domain. However, only the basis functions for g with nonzero value at the boundary give nonzero entries
in the system matrix.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">The element matrices hence have the structure<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">[ A B; C D]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">at the boundary.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">In the interior the element matrices have the structure<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">[A 0; 0 0].<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">The degrees of freedom in the system matrix can be ordered by element [u_e1 g_e1 u_e2 g_e2 …] or by parallel process [u_p1 g_p1 u_p2 g_p2 …].<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">To solve the system matrix, I need to filter out zero rows and columns:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">error = MatFindNonzeroRows(stiffnessMatrix, &nonzeroRows);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">CHKERRABORT(PETSC_COMM_WORLD, error);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">error = MatCreateSubMatrix(stiffnessMatrix, nonzeroRows, nonzeroRows, MAT_INITIAL_MATRIX, &stiffnessMatrixSubMatrix);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">CHKERRABORT(PETSC_COMM_WORLD, error);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">I solve the system matrix in parallel on multiple nodes connected with InfiniBand.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">The problem is that the MUMPS solver frequently (nondeterministically) hangs during KSPSolve() (after KSPSetUp() is completed).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">Running with the options -ksp_view and -info the last printed statement is:<u></u><u></u></span></p>
<p class="gmail-m_277588947448858389p1"><span class="gmail-m_277588947448858389s1">[0] VecScatterCreate_SF(): Using StarForest for vector scatter</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:black">In the calculations where the program does not hang, the calculated solution is correct. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:black">The problem doesn’t occur for calculations on a single node, or for calculations with the SuperLU solver (but SuperLU will not allow calculations that are as large).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">The problem also doesn’t seem to occur for small problems.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">The problem doesn’t occur either when I put ones on the diagonal, but this is computationally expensive:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">error = MatFindZeroRows(stiffnessMatrix, &zeroRows);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">CHKERRABORT(PETSC_COMM_WORLD, error);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">error = MatZeroRowsColumnsIS(stiffnessMatrix, zeroRows, diagEntry, PETSC_IGNORE, PETSC_IGNORE);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">CHKERRABORT(PETSC_COMM_WORLD, error);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">Would you have any ideas on what I could check?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">Sjoerd<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>