<div dir="ltr"><div dir="ltr">The error messages may have nothing to do with PETSc and MOOSE. <div><br></div><div>It might be from a package for MPI communication <a href="https://github.com/openucx/ucx">https://github.com/openucx/ucx</a>. I have no experiences on such things. It may be helpful to contact your HPC administer.</div><div><br></div><div>Thanks,</div><div><br></div><div>Fande,</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 2, 2018 at 9:24 AM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Tue, Oct 2, 2018 at 11:16 AM Y. Yang <<a href="mailto:yangyiwei.yang@mfm.tu-darmstadt.de" target="_blank">yangyiwei.yang@mfm.tu-darmstadt.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear PETSc team<br>
<br>
Recently I'm using MOOSE (<a href="http://www.mooseframework.org/" rel="noreferrer" target="_blank">http://www.mooseframework.org/</a>) which is built <br>
with PETSc and, Unfortunately, I encountered some problems with <br>
following PETSc options:<br></blockquote><div><br></div><div>I do not know what problem you are reporting.I don't know what package knem_ep.c is</div><div>part of, but its not PETSc.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
petsc_options_iname = '-pc_type -ksp_gmres_restart -sub_ksp_type <br>
-sub_pc_type -pc_asm_overlap -pc_factor_mat_solver_package'<br>
<br>
petsc_options_value = 'asm 1201 preonly ilu <br>
4 superlu_dist'<br>
<br>
<br>
the error message is:<br>
<br>
Time Step 1, time = 1<br>
dt = 1<br>
<br>
|residual|_2 of individual variables:<br>
c: 779.034<br>
w: 0<br>
T: 6.57948e+07<br>
gr0: 211.617<br>
gr1: 206.973<br>
gr2: 209.382<br>
gr3: 191.089<br>
gr4: 185.242<br>
gr5: 157.361<br>
gr6: 128.473<br>
gr7: 87.6029<br>
<br>
0 Nonlinear |R| = [32m6.579482e+07 [39m<br>
[1538482623.976180] [hpb0085:22501:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482605.111342] [hpb0085:22502:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482606.761138] [hpb0085:22502:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482607.107478] [hpb0085:22502:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482605.882817] [hpb0085:22503:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482607.133543] [hpb0085:22503:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482621.905475] [hpb0085:22510:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482626.531234] [hpb0085:22510:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482627.613343] [hpb0085:22515:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482627.830489] [hpb0085:22515:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482629.852351] [hpb0085:22515:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482630.194620] [hpb0085:22515:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482630.280636] [hpb0085:22515:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482600.219314] [hpb0085:22516:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482658.960350] [hpb0085:22516:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482622.949471] [hpb0085:22517:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482612.502017] [hpb0085:22500:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482613.231970] [hpb0085:22500:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482621.417530] [hpb0085:22520:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482622.020998] [hpb0085:22520:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482606.221292] [hpb0085:22521:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482606.676987] [hpb0085:22521:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482606.896865] [hpb0085:22521:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482639.611427] [hpb0085:22522:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482631.435277] [hpb0085:22523:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482658.278343] [hpb0085:22512:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482658.396945] [hpb0085:22512:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482659.917476] [hpb0085:22512:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
[1538482660.162064] [hpb0085:22512:0] knem_ep.c:84 UCX ERROR <br>
KNEM inline copy failed, err = -1 Invalid argument<br>
2 total processes killed (some possibly by mpirun during cleanup)<br>
<br>
<br>
Here's the status of the simulation<br>
<br>
Parallelism:<br>
Num Processors: 100<br>
Num Threads: 1<br>
<br>
Mesh:<br>
Parallel Type: distributed<br>
Mesh Dimension: 3<br>
Spatial Dimension: 3<br>
Nodes:<br>
Total: 2065551<br>
Local: 22774<br>
Elems:<br>
Total: 2000000<br>
Local: 20006<br>
Num Subdomains: 1<br>
Num Partitions: 100<br>
Partitioner: parmetis<br>
<br>
Nonlinear System:<br>
Num DOFs: 18589959<br>
Num Local DOFs: 204966<br>
Variables: { "c" "w" "T" "gr0" "gr1" "gr2" "gr3" "gr4" <br>
"gr5" }<br>
Finite Element Types: "LAGRANGE"<br>
Approximation Orders: "FIRST"<br>
<br>
Auxiliary System:<br>
Num DOFs: 10065551<br>
Num Local DOFs: 102798<br>
Variables: "bnds" { "var_indices" "unique_grains" } { <br>
"M" "dM/dT" }<br>
Finite Element Types: "LAGRANGE" "MONOMIAL" "MONOMIAL"<br>
Approximation Orders: "FIRST" "CONSTANT" "CONSTANT"<br>
<br>
Relationship Managers:<br>
Geometric : GrainTrackerHaloRM (2 layers)<br>
<br>
Execution Information:<br>
Executioner: Transient<br>
TimeStepper: IterationAdaptiveDT<br>
Solver Mode: Preconditioned JFNK<br>
<br>
<br>
I tried modifying the parameters and other preconditioning option, the <br>
problem is much the same. So I don't know where I did wrong or there is <br>
actually suitable PETSc option to deal with such problem with large <br>
mesh. I would like to hear your response.<br>
<br>
Sincerely,<br>
Yang<br>
<br>
-- <br>
______________________________________________________<br>
<br>
Yangyiwei Yang<br>
Wissenschaftliche Hilfskraft<br>
<br>
TU Darmstadt<br>
Fachbereich 11 - Material- und Geowissenschaften<br>
Fachgebiet Mechanik funktionaler Materialien<br>
<br>
L1 | 08 402<br>
Otto Berndt Straße 3<br>
D-64287 Darmstadt<br>
<br>
Tel: +49 (0)6151-16-22923<br>
Email: <a href="mailto:yangyiwei.yang@mfm.tu-darmstadt.de" target="_blank">yangyiwei.yang@mfm.tu-darmstadt.de</a><br>
Homepage: <a href="http://www.mawi.tu-darmstadt.de/mfm" rel="noreferrer" target="_blank">http://www.mawi.tu-darmstadt.de/mfm</a><br>
ORCID: 0000-0001-5505-7117<br>
<br>
______________________________________________________<br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-1739292831653612772gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>