<div dir="auto"><div>Can you try using -log_sync ? This should check every entry/exit points of logged Events and complain if something is not collectively called</div><div><br></div><div data-smartmail="gmail_signature">Stefano</div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Tue, Apr 29, 2025, 18:21 Randall Mackie <<a href="mailto:rlmackie862@gmail.com">rlmackie862@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-break:after-white-space">ah okay, I missed that this was found using openmpi.<div><br></div><div>then it’s probably not the same issue we had.</div><div><br></div><div>I can’t remember in which version it was fixed (I’m away from my work computer)….I do know in our case openmpi and the latest Intel One API work fine.</div><div><br></div><div>Randy<br id="m_-5104574022392558396lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Apr 29, 2025, at 8:58 AM, Klaij, Christiaan <<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>> wrote:</div><br><div><div><div>Well, the error below only shows-up thanks to openmpi and gnu compilers.<br>With the intel mpi and compilers it just hangs (tried oneapi 2023.1.0). In which version was that bug fixed?<br><br>Chris<br><br>________________________________________<div dir="ltr" style="font-size:1px;direction:ltr"><table dir="ltr" cellpadding="0" cellspacing="0" border="0" style="width:100%;direction:ltr;border-collapse:collapse;font-size:1px"><tbody><tr style="font-size:0"><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0;line-height:normal"><tbody><tr style="font-size:0"><td align="left" style="padding:10px 0;vertical-align:top"><span id="m_-5104574022392558396cid:image374583.png@2AC1B0A1.BB763CF8"><image374583.png></span></td></tr></tbody></table></td></tr><tr style="font-size:0"><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0"><tbody><tr style="font-size:0"><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0;color:#000001;font-style:normal;font-weight:400;white-space:nowrap"><tbody><tr style="font-size:14.67px"><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif">dr. ir.<span style="font-family:remialcxesans;font-size:1px;color:#ffffff;line-height:1px">​<span style="font-family:'template-zjzHWwipEfCqpwAiSIGong'">​</span><span style="font-family:'zone-1'">​</span><span style="font-family:'zones-AQ'">​</span></span> </td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif">Christiaan</td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif"> Klaij</td></tr></tbody></table></td><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0;color:#000001;font-style:normal;font-weight:400;white-space:nowrap"><tbody><tr style="font-size:14.67px"><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif"> | </td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif">senior researcher</td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif"> | </td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif">Research & Development</td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif"> | </td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif">CFD Development</td><td align="left" style="vertical-align:top;font-size:0"></td></tr></tbody></table></td></tr></tbody></table></td></tr><tr style="font-size:0"><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0"><tbody><tr style="font-size:0"><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0;color:#000001;font-style:normal;font-weight:400;white-space:nowrap"><tbody><tr style="font-size:14.67px"><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif">T <a href="tel:+31%20317%2049%2033%2044" id="m_-5104574022392558396LPlnk689713" style="text-decoration:none;color:#000001" target="_blank" rel="noreferrer">+31 317 49 33 44</a></td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif"> | </td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif"> <a href="mailto:C.Klaij@marin.nl" id="m_-5104574022392558396LPlnk689713" style="text-decoration:none;color:#000001" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a> </td><td align="left" style="vertical-align:top;font-family:Calibri,Arial,sans-serif">| <a href="https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cx-ci9iNg$" id="m_-5104574022392558396LPlnk689713" style="text-decoration:none;color:#000001" target="_blank" rel="noreferrer">www.marin.nl</a></td></tr></tbody></table></td></tr></tbody></table></td></tr><tr style="font-size:0"><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0"><tbody><tr style="font-size:0"><td align="left" style="padding:5px 0 0;vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0"><tbody><tr style="font-size:0"><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0;line-height:normal"><tbody><tr style="font-size:0"><td align="left" style="padding:0 3px 3px 0;vertical-align:top"><a href="https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cxl-rBbfA$" id="m_-5104574022392558396LPlnk689713" style="text-decoration:none" target="_blank" rel="noreferrer"><span id="m_-5104574022392558396cid:image322757.png@12DC6910.0160635C"><image322757.png></span></a></td></tr></tbody></table></td><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0;line-height:normal"><tbody><tr style="font-size:0"><td align="left" style="padding:0 3px 3px 0;vertical-align:top"><a href="https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6Cw7FNelIQ$" id="m_-5104574022392558396LPlnk689713" style="text-decoration:none" target="_blank" rel="noreferrer"><span id="m_-5104574022392558396cid:image512282.png@E0CA701E.3C38E121"><image512282.png></span></a></td></tr></tbody></table></td><td align="left" style="vertical-align:top"><table cellpadding="0" cellspacing="0" border="0" style="border-collapse:collapse;font-size:0;line-height:normal"><tbody><tr style="font-size:0"><td align="left" style="padding:0 3px 3px 0;vertical-align:top"><a href="https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwmfHQRog$" id="m_-5104574022392558396LPlnk689713" style="text-decoration:none" target="_blank" rel="noreferrer"><span id="m_-5104574022392558396cid:image090879.png@328196D0.49EF02FF"><image090879.png></span></a></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></div><br>From: Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" rel="noreferrer">rlmackie862@gmail.com</a>><br>Sent: Tuesday, April 29, 2025 3:33 PM<br>To: Klaij, Christiaan<br>Cc: Matthew Knepley; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>; Isaac, Toby<br>Subject: Re: [petsc-users] problem with nested logging<br><br>You don't often get email from <a href="mailto:rlmackie862@gmail.com" target="_blank" rel="noreferrer">rlmackie862@gmail.com</a>. Learn why this is important<<a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!bW2X4VgowGOLrAASgbFOR_Mh6HW4HtWqrdtpsvpnpiFrIwki34JOGyih-h-1bvgb-Bh4EdWRUoVqQW7s6CwgJDUS5w$" target="_blank" rel="noreferrer">https://aka.ms/LearnAboutSenderIdentification</a>><br>We had a similar issue last year that we eventually tracked down to a bug in Intel MPI AllReduce, which was around the same version you are using.<br><br>Can you try a different MPI or the latest Intel One API and see if your error clears?<br><br>Randy<br><br>On Tue, Apr 29, 2025 at 8:17 AM Klaij, Christiaan via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>> wrote:<br>I don't think so, we have tracing in place to detect mismatches. But as soon as I switch the tracing on, the error disappears... Same if I add a counter or print statements before and after EventBegin/End. Looks like a memory corruption problem, maybe nothing to do with petsc despite the error message.<br><br>Chris<br><br>________________________________________<br>From: Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a><mailto:<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>>><br>Sent: Tuesday, April 29, 2025 1:50 PM<br>To: Klaij, Christiaan<br>Cc: Junchao Zhang; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>; Isaac, Toby<br>Subject: Re: [petsc-users] problem with nested logging<br><br>On Tue, Apr 29, 2025 at 6:50 AM Klaij, Christiaan <<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>>>> wrote:<br>Here's a slightly better error message, obtained --with-debugging=1<br><br>Is it possible that you have a mismatched EventBegin()/EventEnd() in your code? That could be why we cannot reproduce it here.<br><br>  Thanks,<br><br>      Matt<br><br>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[0]PETSC ERROR: Petsc has generated inconsistent data<br>[0]PETSC ERROR: MPIU_Allreduce() called in different locations (code lines) on different processors<br>[0]PETSC ERROR: See <a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjggvQAzPU$</a>  for trouble shooting.<br>[0]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025<br>[0]PETSC ERROR: ./refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by cklaij Tue Apr 29 12:43:54 2025<br>[0]PETSC ERROR: Configure options: --prefix=/home/cklaij/ReFRESCO/trunk/install/extLibs --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=1 --download-superlu_dist=<a href="https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgVgVAJPM$</a>  --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=<a href="https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgo2JWTO4$</a>  --download-metis=<a href="https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgX9ZMYJA$</a>  --with-packages-build-dir=/home/cklaij/ReFRESCO/trunk/build-libs/superbuild --with-ssl=0 --with-shared-libraries=1<br>[0]PETSC ERROR: #1 PetscLogNestedTreePrintLine() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:289<br>[0]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:379<br>[0]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384<br>[0]PETSC ERROR: #4 PetscLogNestedTreePrint() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384<br>[0]PETSC ERROR: #5 PetscLogNestedTreePrintTop() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420<br>[0]PETSC ERROR: #6 PetscLogHandlerView_Nested_XML() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443<br>[0]PETSC ERROR: #7 PetscLogHandlerView_Nested() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405<br>[0]PETSC ERROR: #8 PetscLogHandlerView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342<br>[0]PETSC ERROR: #9 PetscLogView() at /home/cklaij/ReFRESCO/trunk/build-libs/superbuild/petsc/src/src/sys/logging/plog.c:2040<br>[0]PETSC ERROR: #10 /home/cklaij/ReFRESCO/trunk/Code/src/petsc_include_impl.F90:130<br><br>________________________________________<br>[cid:ii_19681617e7812ff9cfc1]<br>dr. ir.         Christiaan       Klaij<br> |      senior researcher        |      Research & Development   |      CFD Development<br>T +31 317 49 33 44<tel:+31%20317%2049%2033%2044>         |       <a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>>>      | <a href="https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgJooxJhg$</a> <<a href="https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgpOXjQSM$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgpOXjQSM$</a> ><br>[Facebook]<<a href="https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg_3Zt0Pw$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg_3Zt0Pw$</a> ><br>[LinkedIn]<<a href="https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcpgQdSE$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcpgQdSE$</a> ><br>[YouTube]<<a href="https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgn8qP6_I$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgn8qP6_I$</a> ><br><br>From: Klaij, Christiaan <<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>>>><br>Sent: Monday, April 28, 2025 3:53 PM<br>To: Matthew Knepley<br>Cc: Junchao Zhang; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>>; Isaac, Toby<br>Subject: Re: [petsc-users] problem with nested logging<br><br>Bisecting would be quite hard, it's not just the petsc version that changed, also other libs, compilers, even os components.<br><br>Chris<br><br>________________________________________<br>From: Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a><mailto:<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>><mailto:<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a><mailto:<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>>>><br>Sent: Monday, April 28, 2025 3:06 PM<br>To: Klaij, Christiaan<br>Cc: Junchao Zhang; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>>; Isaac, Toby<br>Subject: Re: [petsc-users] problem with nested logging<br><br>You don't often get email from <a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a><mailto:<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>><mailto:<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a><mailto:<a href="mailto:knepley@gmail.com" target="_blank" rel="noreferrer">knepley@gmail.com</a>>>. Learn why this is important<<a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcKEJXRA$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgcKEJXRA$</a> ><br>On Mon, Apr 28, 2025 at 8:45 AM Klaij, Christiaan via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>>>> wrote:<br>I've tried adding a nested log viewer to src/snes/tutorials/ex70.c,<br>but it does not replicate the problem and works fine.<br><br>Perhaps it is related to fortran, since the manualpage of<br>PetscLogNestedBegin says "No fortran support" (why? we've been<br>using it in fortran ever since).<br><br>Therefore I've tried adding it to src/snes/ex5f90.F90 and that<br>also works fine. It seems I cannot replicate the problem in a<br>small example, unfortunately.<br><br>We cannot replicate it here. Is there a chance you could bisect to see what change is responsible?<br><br>Thanks,<br><br>Matt<br><br>Chris<br><br>________________________________________<br>From: Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a>><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a>>><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a>><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a>>>>><br>Sent: Saturday, April 26, 2025 3:51 PM<br>To: Klaij, Christiaan<br>Cc: <a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>>>; Isaac, Toby<br>Subject: Re: [petsc-users] problem with nested logging<br><br>You don't often get email from <a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a>><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a>>><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a>><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a><mailto:<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a>>>>. Learn why this is important<<a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gVgt1dmE$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gVgt1dmE$</a> ><br>Toby (Cc'ed) might know it. Or could you provide an example?<br><br>--Junchao Zhang<br><br><br>On Fri, Apr 25, 2025 at 3:31 AM Klaij, Christiaan via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><mailto:<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>>>>> wrote:<br>We recently upgraded from 3.19.4 to 3.22.4 but face the problem below with the nested logging. Any ideas?<br><br>Chris<br><br><br>[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[1]PETSC ERROR: General MPI error<br>[1]PETSC ERROR: MPI error 1 MPI_ERR_BUFFER: invalid buffer pointer<br>[1]PETSC ERROR: See <a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gIT68pbk$</a> <<a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM3UaqHzc$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM3UaqHzc$</a>> for trouble shooting.<br>[1]PETSC ERROR: Petsc Release Version 3.22.4, Mar 01, 2025<br>[1]PETSC ERROR: refresco with 2 MPI process(es) and PETSC_ARCH on marclus3login2 by jwindt Fri Apr 25 08:52:30 2025<br>[1]PETSC ERROR: Configure options: --prefix=/home/jwindt/cmake_builds/refresco/install-libs-gnu --with-mpi-dir=/cm/shared/apps/openmpi/gcc/4.0.2 --with-x=0 --with-mpe=0 --with-debugging=0 --download-superlu_dist=<a href="https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6grH5BbeU$</a> <<a href="https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM21-2D-o$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/superlu_dist-8.1.2.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM21-2D-o$</a>> --with-blaslapack-dir=/cm/shared/apps/intel/oneapi/mkl/2021.4.0 --download-parmetis=<a href="https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gw4-tEtY$</a> <<a href="https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMW0lYHko$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/parmetis-4.0.3-p9.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMW0lYHko$</a>> --download-metis=<a href="https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6gHq4uYiY$</a> <<a href="https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMbSrIiUg$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://updates.marin.nl/refresco/libs/metis-5.1.0-p11.tar.gz__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMbSrIiUg$</a>> --with-packages-build-dir=/home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild --with-ssl=0 --with-shared-libraries=1 CFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " COPTFLAGS="-std=gnu11 -Wall -funroll-all-loops -O3 -DNDEBUG" CXXOPTFLAGS="-std=gnu++14 -Wall -funroll-all-loops -O3 -DNDEBUG " FCFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" F90FLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG" FOPTFLAGS="-Wall -funroll-all-loops -ffree-line-length-0 -Wno-maybe-uninitialized -Wno-target-lifetime -Wno-unused-function -O3 -DNDEBUG"<br>[1]PETSC ERROR: #1 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:330<br>[1]PETSC ERROR: #2 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384<br>[1]PETSC ERROR: #3 PetscLogNestedTreePrint() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:384<br>[1]PETSC ERROR: #4 PetscLogNestedTreePrintTop() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:420<br>[1]PETSC ERROR: #5 PetscLogHandlerView_Nested_XML() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/xmlviewer.c:443<br>[1]PETSC ERROR: #6 PetscLogHandlerView_Nested() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/impls/nested/lognested.c:405<br>[1]PETSC ERROR: #7 PetscLogHandlerView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/handler/interface/loghandler.c:342<br>[1]PETSC ERROR: #8 PetscLogView() at /home/jwindt/cmake_builds/refresco/build-libs-gnu/superbuild/petsc/src/src/sys/logging/plog.c:2040<br>--------------------------------------------------------------------------<br>MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD<br>with errorcode 98.<br><br>NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>You may or may not see output from other processes, depending on<br>exactly when Open MPI kills them.<br>--------------------------------------------------------------------------<br>[cid:ii_196725d1e2a809852191]<br>dr. ir. Christiaan Klaij<br>| senior researcher | Research & Development | CFD Development<br>T +31 317 49 33 44<tel:+31%20317%2049%2033%2044> | <a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>>>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a><mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank" rel="noreferrer">C.Klaij@marin.nl</a>>>>> | <a href="https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!cmLENZvO_Uydoa8ciUmsyX-F-QiJt9a2ZfQRUvQnRibGm7VE6PED7S_BDsUgjOzvPZIJyiIoJ8bLJk6g8TwMMcw$</a> <<a href="https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM8tSNH1g$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGM8tSNH1g$</a>><br>[Facebook]<<a href="https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMcVCZ9hk$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMcVCZ9hk$</a>><br>[LinkedIn]<<a href="https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMIDBZW7k$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMIDBZW7k$</a>><br>[YouTube]<<a href="https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMVKWos24$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!biVlk6PKXoJvq5oVlmWdVJfW9tXv-JlwuWr3zg3jI5u1_jo8rvtZpEYnHO5RjdBqQEoqpqlJ3nusrFGMVKWos24$</a>><br><br><br>--<br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br><br><a href="https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$</a> <<a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$</a> ><br><br><br>--<br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br><br><a href="https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjg539kFLg$</a> <<a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$" target="_blank" rel="noreferrer">https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fFq1daAFFpKhBA_NWU3sd2QJe_S44rklqeRi0TB57XI0nQsh9jgy8iw3JNGpBbd21zqvO3QlGTLa7kjgMsu6hhA$</a> ><br></div></div></div></blockquote></div><br></div></div></blockquote></div>