<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi Pierre,</p>
<div class="moz-cite-prefix">On 2021-03-02 3:50 p.m., Pierre Jolivet
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:FF313618-3465-474B-B13E-A7FDA1FB074D@joliv.et">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Hello Eric,
<div class="">
<div class=""><span style="white-space: pre-wrap;" class="">src/mat/tests/ex237.c is a recent test with some code paths that should be disabled for “old” MKL versions. It’s tricky to check directly in the source (we do check in BuildSystem) because there is no such thing as PETSC_PKG_MKL_VERSION_LT, but I guess we can change if defined(PETSC_HAVE_MKL) to if defined(PETSC_HAVE_MKL) && defined(PETSC_HAVE_MKL_SPARSE_OPTIMIZE), I’ll make a MR, thanks for reporting this.</span></div>
</div>
</blockquote>
Just saw your MR, thanks for the fix! :)<br>
<blockquote type="cite"
cite="mid:FF313618-3465-474B-B13E-A7FDA1FB074D@joliv.et">
<div class="">
<div class="">
<div><span style="white-space: pre-wrap;" class="">
</span></div>
<div>
<div class="">For the other issues, I’m sensing this is a
problem with gomp + intel_gnu_thread, but this is pure
speculation… sorry.</div>
<div class="">I’ll try to reproduce some of these problems
if you are not given a more meaningful answer.</div>
</div>
</div>
</div>
</blockquote>
<p>ok, I feel like I am the first to have problems with this...</p>
<p>If it ends that there is a problem combining MKL + openMP that
relies on linking configuration for example, should it be a good
thing to have this (--with-openmp=1) tested into the pipelines
(with external packages of course)?</p>
<p>Thanks,</p>
<p>Eric<br>
</p>
<blockquote type="cite"
cite="mid:FF313618-3465-474B-B13E-A7FDA1FB074D@joliv.et">
<div class="">
<div class="">
<div><span style="white-space: pre-wrap;" class="">
</span></div>
<div><span style="white-space: pre-wrap;" class="">Thanks,</span></div>
<div><span style="white-space: pre-wrap;" class="">Pierre</span></div>
</div>
<div class="">
<div><span style="white-space: pre-wrap;" class="">
</span></div>
<div>
<blockquote type="cite" class="">
<div class="">On 2 Mar 2021, at 9:14 PM, Eric Chamberland
<<a href="mailto:Eric.Chamberland@giref.ulaval.ca"
class="" moz-do-not-send="true">Eric.Chamberland@giref.ulaval.ca</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8" class="">
<div class="">
<p class="">Hi,</p>
<p class="">It all started when I wanted to test
PETSC/CUDA compatibility for our code.</p>
<p class="">I had to activate --with-openmp to
configure with --with-cuda=1 successfully.</p>
<p class="">I then saw that PETSC_HAVE_OPENMP is used
at least in MUMPS (and some other places).</p>
<p class="">So, I configured and tested petsc with
openmp activated, without CUDA.<br class="">
</p>
<p class="">The first thing I see is that our code CI
pipelines now fails for many tests.</p>
<p class="">After looking deeper, it seems that PETSc
itself fails many tests when I activate openmp!</p>
Here are all the configurations I have results for,
after/before activating OpenMP for PETSc:<br class="">
<p class="">==============================================================================</p>
<p class="">==============================================================================</p>
<p class="">For petsc/master + OpenMPI 4.0.4 + MKL
2019.4.243:</p>
<p class="">With OpenMP=1</p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_make_test.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_make_test.log</a></p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_configure.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_configure.log</a></p>
<pre style="font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; overflow-wrap: break-word; white-space: pre-wrap;" class=""># -------------
# Summary
# -------------
# FAILED snes_tutorials-ex12_quad_hpddm_reuse_baij diff-ksp_ksp_tests-ex33_superlu_dist_2 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 ksp_ksp_tutorials-ex50_tut_2 diff-ksp_ksp_tests-ex33_superlu_dist diff-snes_tutorials-ex56_hypre snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij ksp_ksp_tutorials-ex5_superlu_dist_3 ksp_ksp_tutorials-ex5f_superlu_dist snes_tutorials-ex12_tri_parmetis_hpddm_baij diff-snes_tutorials-ex19_tut_3 mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex19_superlu_dist diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-ksp_ksp_tutorials-ex49_hypre_nullspace ts_tutorials-ex18_p1p1_xper_ref ts_tutorials-ex18_p1p1_xyper_ref snes_tutorials-ex19_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist_2 diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex64_1 ksp_ksp_tutorials-ex5_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2
# success 8275/10003 tests (82.7%)
# <b class="">failed 33/10003</b> tests (0.3%)</pre>
<p class="">With OpenMP=0</p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_make_test.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_make_test.log</a></p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_configure.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_configure.log</a><br
class="">
</p>
<pre style="font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; overflow-wrap: break-word; white-space: pre-wrap;" class=""># -------------
# Summary
# -------------
# FAILED tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex17_3d_q3_trig_elas mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5
# success 8262/9983 tests (82.8%)
# <b class="">failed 6/9983</b> tests (0.1%)</pre>
<p class="">==============================================================================</p>
<p class="">==============================================================================</p>
<p class="">For OpenMPI 3.1.x/master:<br class="">
</p>
<p class="">With OpenMP=1:</p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_make_test.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_make_test.log</a></p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_configure.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_configure.log</a></p>
<pre style="font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; overflow-wrap: break-word; white-space: pre-wrap;" class=""># -------------
# Summary
# -------------
# FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex5_superlu_dist_3 diff-ksp_ksp_tutorials-ex49_hypre_nullspace ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex17_3d_q3_trig_vlap diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-snes_tutorials-ex19_tut_3 diff-snes_tutorials-ex56_hypre diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 tao_constrained_tutorials-tomographyADMM_6 diff-tao_constrained_tutorials-toyf_1
# success 8142/9765 tests (83.4%)
# <b class="">failed 16/9765</b> tests (0.2%)</pre>
<p class="">With OpenMP=0:</p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_make_test.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_make_test.log</a></p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_configure.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_configure.log</a><br
class="">
</p>
<pre style="font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; overflow-wrap: break-word; white-space: pre-wrap;" class=""># -------------
# Summary
# -------------
# FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex56_2 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 diff-tao_constrained_tutorials-toyf_1
# success 8151/9767 tests (83.5%)
# <b class="">failed 9/9767</b> tests (0.1%)
</pre>
<p class="">==============================================================================</p>
<p class="">==============================================================================</p>
<p class="">For OpenMPI 4.0.x/master:</p>
<p class="">With OpenMP=1:</p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_make_test.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_make_test.log</a></p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_configure.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_configure.log</a><br
class="">
</p>
<pre style="font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; overflow-wrap: break-word; white-space: pre-wrap;" class=""># FAILED snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex19_hypre ksp_ksp_tutorials-ex56_2 tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5 mat_tests-ex242_3 ksp_ksp_tutorials-ex55_hypre ksp_ksp_tutorials-ex5_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex56_hypre snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist_3 ksp_ksp_tutorials-ex34_hyprestruct diff-ksp_ksp_tutorials-ex49_hypre_nullspace snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist snes_tutorials-ex19_tut_3 snes_tutorials-ex19_superlu_dist ksp_ksp_tutorials-ex50_tut_2 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5_superlu_dist_3 snes_tutorials-ex19_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_4 ts_tutorials-ex26_2
# success 8125/9753 tests (83.3%)
# <b class="">failed 26/9753</b> tests (0.3%)</pre>
<p class="">With OpenMP=0</p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_make_test.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_make_test.log</a></p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_configure.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_configure.log</a><br
class="">
</p>
<pre style="font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; overflow-wrap: break-word; white-space: pre-wrap;" class=""># FAILED mat_tests-ex242_3
# success 8174/9777 tests (83.6%)
# <b class="">failed 1/9777</b> tests (0.0%)
</pre>
<p class="">==============================================================================</p>
<p class="">==============================================================================</p>
<p class="">Is that known and normal?</p>
<p class="">In all cases, I am using MKL and I suspect
it may come from there... :/</p>
<p class="">I also saw a second problem, "make test"
fails to compile petsc examples on older versions of
MKL (but that's less important for me, I just
upgraded to OneAPI to avoid this, but you may want
to know):<br class="">
</p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_make_test.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_make_test.log</a></p>
<p class=""><a class="moz-txt-link-freetext"
href="https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_configure.log"
moz-do-not-send="true">https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_configure.log</a><br
class="">
</p>
<p class=""> Thanks,</p>
<p class="">Eric<br class="">
</p>
<pre class="moz-signature" cols="72">--
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Eric Chamberland, ing., M. Ing
Professionnel de recherche
GIREF/Université Laval
(418) 656-2131 poste 41 22 42</pre>
</body>
</html>